Re: t-filter: no table created when Unicode characters are included

[Paul Menzel <paulepanter@users.sourceforge.net>]

[-- Attachment #1.1: Type: text/plain, Size: 6165 bytes --]

Am Samstag, den 30.04.2011, 11:10 -0400 schrieb Aditya Mahajan:
> On Sat, 30 Apr 2011, Paul Menzel wrote:
> 
> > Am Freitag, den 29.04.2011, 18:13 -0400 schrieb Aditya Mahajan:
> >> On Fri, 29 Apr 2011, Paul Menzel wrote:
> >
> >>> the filter module installed using
> >>> does not work with the following example, which is also attached.
> >>> 
> >>> The output file `minimal-temp-rest.tex` is just empty.
> >>>
> >>> Replacing »α« by »a« solves the problem.
> >>>
> >>> Running `filtercommand` in a terminal produces the *correct* output
> >>> though and no empty file.
> >>>
> >>> Is that a known problem?
> >>
> >> No. The content of the file should not affect whether the filter is run or
> >> not. Can you add \traceexternalfilters on the top and check the output
> >> (look for lines starting with t-filter).
> >
> >> The file works correctly on my machine.
> >
> > That is weired.
> >
> >> Which version of pandoc are you using?
> >
> >        $ pandoc --version
> >        pandoc 1.5.1.1
> 
> I am using 1.6 (the latest version is 1.8), but ...

Version 1.8 is not yet uploaded to the Debian repositories since some
Haskell packages are not yet available. See ticket #613256 [1] in the
Debian BTS.

> > As already written in my last mail. Executing the command directly works
> > fine.
> 
> I really have no idea why this is happening. Pandoc does not give too many 
> debugging options. A couple of things to check:
> 
> - What happens is you use pandoc -f rst -t native? (The updated example on 
> github)?

The output file is also empty. Running the command from console works
again.

> - If that fails, then try a simple rst file with a unicode letter (no 
> table)?

The output file is still empty.

        $ pandoc -f rst -t native -o rest-temp-rest.tex
        rest-temp-rest.tmp
        $ more rest-temp-rest.t{mp,ex}
        ::::::::::::::
        rest-temp-rest.tmp
        ::::::::::::::
        α
        ::::::::::::::
        rest-temp-rest.tex
        ::::::::::::::
        Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []})
        [ Para [Str "\945"] ]
        
        
        $

Is the α also encoded by `Str "\945"` in your setup?

> - If that also fails, then a simple markdown file with a unicode letter.

That fails too.

        $ more rest-temp-rest.tex
        $ pandoc -f markdown -t native -o rest-temp-rest.tex rest-temp-rest.tmp
        $ more rest-temp-rest.t{mp,ex}
        ::::::::::::::
        rest-temp-rest.tmp
        ::::::::::::::
        α
        ::::::::::::::
        rest-temp-rest.tex
        ::::::::::::::
        Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []})
        [ Para [Str "\945"] ]
        
        
        $

> - If that also fails, then some other program that reads a file and writes 
> its contents to another file.

Usinng the following

        \defineexternalfilter
          [rest]
          [filter={cat \externalfilterinputfile\space | tee \externalfilteroutputfile}]

works.

        $ more rest-temp-rest.t{mp,ex}
        ::::::::::::::
        rest-temp-rest.tmp
        ::::::::::::::
        α
        ::::::::::::::
        rest-temp-rest.tex
        ::::::::::::::
        α

> - Manually execute the shell command from inside ConTeXt using
> \ctxlua{os.execute("....")}.

This does not work either.

        $ more tests/rest.tex
        % LM does not appear to have Greek letters in unicode slots
        \setupbodyfont[times]
        
        \starttext
        Start.
        
        \ctxlua{os.execute("pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp")}
        \stoptext
        $ context tests/rest.tex
        $ more rest-temp-rest.t{mp,ex}
        ::::::::::::::
        rest-temp-rest.tmp
        ::::::::::::::
        α
        ::::::::::::::
        rest-temp-rest.tex
        ::::::::::::::
        $

> Also, what OS you are on?

I am using Debian Sid/unstable.

> >        $ pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp
> >        $ more rest-temp-rest.t*::::::::::::::
> >        rest-temp-rest.tex
> >        ::::::::::::::
> >        \placetable[here]{none}
> >        \starttable[|l|l|]
> >        \HL
> >        \NC test
> >        \NC table
> >        \NC\AR
> >        \HL
> >        \NC α
> >        \NC b
> >        \NC\AR
> >        \NC c
> >        \NC d
> >        \NC\AR
> >        \HL
> >        \stoptable
> >        ::::::::::::::
> >        rest-temp-rest.tmp
> >        ::::::::::::::
> >        =========       =========
> >          test            table
> >        =========       =========
> >        α                 b
> >        c                 d
> >        ========        ========
> 
> Since the .tex file is created, it means that the filter module is running 
> correctly. It is calling pandoc with the right options, and pandoc is 
> creating an empty rest-temp-rest.tex file. Now I have no idea why that 
> file is empty. IIRC, pandoc sometimes misbehaves due to locale settings 
> (but I don't see why locale should be different when a program is run from 
> inside context then when it is run from a shell).

I am using the following locale.

        $ locale
        LANG=de_DE.UTF-8
        LC_CTYPE="de_DE.UTF-8"
        LC_NUMERIC="de_DE.UTF-8"
        LC_TIME="de_DE.UTF-8"
        LC_COLLATE="de_DE.UTF-8"
        LC_MONETARY="de_DE.UTF-8"
        LC_MESSAGES="de_DE.UTF-8"
        LC_PAPER="de_DE.UTF-8"
        LC_NAME="de_DE.UTF-8"
        LC_ADDRESS="de_DE.UTF-8"
        LC_TELEPHONE="de_DE.UTF-8"
        LC_MEASUREMENT="de_DE.UTF-8"
        LC_IDENTIFICATION="de_DE.UTF-8"
        LC_ALL=

> I am stymied.

Well, I would say to not waste your time any longer and to suspect
Pandoc at fault here. If a newer Pandoc version is available in the
Debian archive I can retest to check the Pandoc version as the possible
culprit.


Thank you very much for your help,

Paul


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613256

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________