How to use Python filter with Pandoc to convert md with tikz to html on Windows 8.1

How to use Python filter with Pandoc to convert md with tikz to html on Windows 8.1

[Richard Herron]

[-- Attachment #1.1: Type: text/plain, Size: 5660 bytes --]

This is a cross-post from Stackoverflow. I asked the question 8 days ago, 
but it didn't receive much attention. 

Here's the question 
link: http://stackoverflow.com/questions/30696797/how-to-use-python-filter-with-pandoc-to-convert-md-with-tikz-to-html-on-windows

You may want to answer the question there (then paste here), because I have 
a 100 point bounty. Sorry for the cross-post.

---

I am trying to use a Pandoc filter to convert a markdown file with a tikz 
picture to html. I am on Win 8.1 (and I have all the dependencies -- 
pdflatex, Python 2.7, ImageMagick, and the pandocfilters Python package). I 
am using the tikz.py script that John MacFarlane provides on 
[github](https://github.com/jgm/pandocfilters).

I found a similar [question](https://github.com/jgm/pandoc/issues/1096) on 
the Pandoc Google Group and John MacFarlane suggests wrapping the filter in 
a Windows batch script (the filter must be an executable). Here is my 
command line input (I'll provide the file contents below).

    pandoc -o temp.html --filter .\tikz.bat -s temp.md

But I keep getting the following error.

    pandoc: Failed reading: satisfyElem

The script generates the "tikz-images" subfolder, but it is empty, as is 
the resulting output file temp.html.

If I run `python tikz.py temp.md` from the Windows command prompt it hangs. 
If I kill the command with `CTRL-C`, then I get the following.

    C:\Users\Richard\Desktop\temp>python tikz.py temp.md
    Traceback (most recent call last):
      File "tikz.py", line 70, in <module>
        toJSONFilter(tikz)

The same occurs with `python caps.py temp.md`. A hang, followed by:

    C:\Users\Richard\Desktop\temp>python caps.py temp.md
    Traceback (most recent call last):
      File "caps.py", line 15, in <module>
        toJSONFilter(caps)


How can I get this to work? FWIW, the bigger goal is for the input files to 
be [R Markdown](http://rmarkdown.rstudio.com/), but I want to understand 
the filters in Pandoc Markdown alone first.

Here are the file contents.

tikz.bat

    python tikz.py %*

temp.md

    \begin{tikzpicture}

    \draw [<->](-3,0)--(3,0);
    \draw (-2,-.2)--(-2,.2);
    \draw (-1,-.2)--(-1,.2);
    \draw(0,-.2)--(0,.2);
    \draw (1,-.2)--(1,.2);
    \draw (2,-.2)--(2,.2);
    \node[align=left,below] at (-4.5,-0.2) {Cash flow};
    \node[align=left,above] at (-4.5,0.2) {Time period};
    \node[align=left,above] at (-2,0.2) {-2};
    \node[align=left,above] at (-1,0.2) {-1};
    \node[align=left,above] at (0,0.2) {0};
    \node[align=left,above] at (1,0.2) {+1};
    \node[align=left,above] at (2,0.2) {+2};
    \node[align=left,below] at (1,-0.2) {\$100};
    \node[align=left,below] at (2,-0.2) {\$100};

    \end{tikzpicture}

    Can this work?

tikz.py

    #!/usr/bin/env python

    """
    Pandoc filter to process raw latex tikz environments into images.
    Assumes that pdflatex is in the path, and that the standalone
    package is available.  Also assumes that ImageMagick's convert
    is in the path. Images are put in the tikz-images directory.
    """

    import hashlib
    import re
    import os
    import sys
    import shutil
    from pandocfilters import toJSONFilter, Para, Image
    from subprocess import Popen, PIPE, call
    from tempfile import mkdtemp

    imagedir = "tikz-images"


    def sha1(x):
        return 
hashlib.sha1(x.encode(sys.getfilesystemencoding())).hexdigest()


    def tikz2image(tikz, filetype, outfile):
        tmpdir = mkdtemp()
        olddir = os.getcwd()
        os.chdir(tmpdir)
        f = open('tikz.tex', 'w')
        f.write("""\\documentclass{standalone}
                 \\usepackage{tikz}
                 \\begin{document}
                 """)
        f.write(tikz)
        f.write("\n\\end{document}\n")
        f.close()
        p = call(["pdflatex", 'tikz.tex'], stdout=sys.stderr)
        os.chdir(olddir)
        if filetype == 'pdf':
            shutil.copyfile(tmpdir + '/tikz.pdf', outfile + '.pdf')
        else:
            call(["convert", tmpdir + '/tikz.pdf', outfile + '.' + 
filetype])
        shutil.rmtree(tmpdir)


    def tikz(key, value, format, meta):
        if key == 'RawBlock':
            [fmt, code] = value
            if fmt == "latex" and re.match("\\\\begin{tikzpicture}", code):
                outfile = imagedir + '/' + sha1(code)
                if format == "html":
                    filetype = "png"
                elif format == "latex":
                    filetype = "pdf"
                else:
                    filetype = "png"
                src = outfile + '.' + filetype
                if not os.path.isfile(src):
                    try:
                        os.mkdir(imagedir)
                        sys.stderr.write('Created directory ' + imagedir + 
'\n')
                    except OSError:
                        pass
                    tikz2image(code, filetype, outfile)
                    sys.stderr.write('Created image ' + src + '\n')
                return Para([Image([], [src, ""])])

    if __name__ == "__main__":
        toJSONFilter(tikz)


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/33c5e06a-2bac-4c44-9da8-073ca2bf624b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 9611 bytes --]

Re: How to use Python filter with Pandoc to convert md with tikz to html on Windows 8.1

[John MacFarlane]

Can you try (with a relatively recent pandoc) with simply

    --filter tikz.py


+++ Richard Herron [Jun 15 15 09:17 ]:
>   This is a cross-post from Stackoverflow. I asked the question 8 days
>   ago, but it didn't receive much attention.
>   Here's the question
>   link: http://stackoverflow.com/questions/30696797/how-to-use-python-fil
>   ter-with-pandoc-to-convert-md-with-tikz-to-html-on-windows
>   You may want to answer the question there (then paste here), because I
>   have a 100 point bounty. Sorry for the cross-post.
>   ---
>   I am trying to use a Pandoc filter to convert a markdown file with a
>   tikz picture to html. I am on Win 8.1 (and I have all the dependencies
>   -- pdflatex, Python 2.7, ImageMagick, and the pandocfilters Python
>   package). I am using the tikz.py script that John MacFarlane provides
>   on [github](https://github.com/jgm/pandocfilters).
>   I found a similar [question](https://github.com/jgm/pandoc/issues/1096)
>   on the Pandoc Google Group and John MacFarlane suggests wrapping the
>   filter in a Windows batch script (the filter must be an executable).
>   Here is my command line input (I'll provide the file contents below).
>       pandoc -o temp.html --filter .\tikz.bat -s temp.md
>   But I keep getting the following error.
>       pandoc: Failed reading: satisfyElem
>   The script generates the "tikz-images" subfolder, but it is empty, as
>   is the resulting output file temp.html.
>   If I run `python tikz.py temp.md` from the Windows command prompt it
>   hangs. If I kill the command with `CTRL-C`, then I get the following.
>       C:\Users\Richard\Desktop\temp>python tikz.py temp.md
>       Traceback (most recent call last):
>         File "tikz.py", line 70, in <module>
>           toJSONFilter(tikz)
>   The same occurs with `python caps.py temp.md`. A hang, followed by:
>       C:\Users\Richard\Desktop\temp>python caps.py temp.md
>       Traceback (most recent call last):
>         File "caps.py", line 15, in <module>
>           toJSONFilter(caps)
>   How can I get this to work? FWIW, the bigger goal is for the input
>   files to be [R Markdown](http://rmarkdown.rstudio.com/), but I want to
>   understand the filters in Pandoc Markdown alone first.
>   Here are the file contents.
>   tikz.bat
>       python tikz.py %*
>   temp.md
>       \begin{tikzpicture}
>       \draw [<->](-3,0)--(3,0);
>       \draw (-2,-.2)--(-2,.2);
>       \draw (-1,-.2)--(-1,.2);
>       \draw(0,-.2)--(0,.2);
>       \draw (1,-.2)--(1,.2);
>       \draw (2,-.2)--(2,.2);
>       \node[align=left,below] at (-4.5,-0.2) {Cash flow};
>       \node[align=left,above] at (-4.5,0.2) {Time period};
>       \node[align=left,above] at (-2,0.2) {-2};
>       \node[align=left,above] at (-1,0.2) {-1};
>       \node[align=left,above] at (0,0.2) {0};
>       \node[align=left,above] at (1,0.2) {+1};
>       \node[align=left,above] at (2,0.2) {+2};
>       \node[align=left,below] at (1,-0.2) {\$100};
>       \node[align=left,below] at (2,-0.2) {\$100};
>       \end{tikzpicture}
>       Can this work?
>   tikz.py
>       #!/usr/bin/env python
>       """
>       Pandoc filter to process raw latex tikz environments into images.
>       Assumes that pdflatex is in the path, and that the standalone
>       package is available.  Also assumes that ImageMagick's convert
>       is in the path. Images are put in the tikz-images directory.
>       """
>       import hashlib
>       import re
>       import os
>       import sys
>       import shutil
>       from pandocfilters import toJSONFilter, Para, Image
>       from subprocess import Popen, PIPE, call
>       from tempfile import mkdtemp
>       imagedir = "tikz-images"
>       def sha1(x):
>           return
>   hashlib.sha1(x.encode(sys.getfilesystemencoding())).hexdigest()
>       def tikz2image(tikz, filetype, outfile):
>           tmpdir = mkdtemp()
>           olddir = os.getcwd()
>           os.chdir(tmpdir)
>           f = open('tikz.tex', 'w')
>           f.write("""\\documentclass{standalone}
>                    \\usepackage{tikz}
>                    \\begin{document}
>                    """)
>           f.write(tikz)
>           f.write("\n\\end{document}\n")
>           f.close()
>           p = call(["pdflatex", 'tikz.tex'], stdout=sys.stderr)
>           os.chdir(olddir)
>           if filetype == 'pdf':
>               shutil.copyfile(tmpdir + '/tikz.pdf', outfile + '.pdf')
>           else:
>               call(["convert", tmpdir + '/tikz.pdf', outfile + '.' +
>   filetype])
>           shutil.rmtree(tmpdir)
>       def tikz(key, value, format, meta):
>           if key == 'RawBlock':
>               [fmt, code] = value
>               if fmt == "latex" and re.match("\\\\begin{tikzpicture}",
>   code):
>                   outfile = imagedir + '/' + sha1(code)
>                   if format == "html":
>                       filetype = "png"
>                   elif format == "latex":
>                       filetype = "pdf"
>                   else:
>                       filetype = "png"
>                   src = outfile + '.' + filetype
>                   if not os.path.isfile(src):
>                       try:
>                           os.mkdir(imagedir)
>                           sys.stderr.write('Created directory ' +
>   imagedir + '\n')
>                       except OSError:
>                           pass
>                       tikz2image(code, filetype, outfile)
>                       sys.stderr.write('Created image ' + src + '\n')
>                   return Para([Image([], [src, ""])])
>       if __name__ == "__main__":
>           toJSONFilter(tikz)
>
>   --
>   You received this message because you are subscribed to the Google
>   Groups "pandoc-discuss" group.
>   To unsubscribe from this group and stop receiving emails from it, send
>   an email to [1]pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To post to this group, send email to
>   [2]pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To view this discussion on the web visit
>   [3]https://groups.google.com/d/msgid/pandoc-discuss/33c5e06a-2bac-4c44-
>   9da8-073ca2bf624b%40googlegroups.com.
>   For more options, visit [4]https://groups.google.com/d/optout.
>
>References
>
>   1. mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   2. mailto:pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   3. https://groups.google.com/d/msgid/pandoc-discuss/33c5e06a-2bac-4c44-9da8-073ca2bf624b-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org?utm_medium=email&utm_source=footer
>   4. https://groups.google.com/d/optout

Re: How to use Python filter with Pandoc to convert md with tikz to html on Windows 8.1

[Richard Herron]

[-- Attachment #1.1: Type: text/plain, Size: 8620 bytes --]

Thanks, John. I should have mentioned that I'm using 1.13.2

    C:\Users\Richard\Desktop\pandoc_question>pandoc --version
    pandoc 1.13.2
    Compiled with texmath 0.8.0.1, highlighting-kate 0.5.11.1.

If I run pandoc with only `--filter tikz.py`, then it hangs and prints just 
a "p" when a press CTRL-C.

    C:\Users\Richard\Desktop\pandoc_question>pandoc --filter tikz.py
    p

This is the same whether I use tikz.py or .\tikz.py.

If I run tikz.py with full arguments, but without the batch file wrapper, 
then I get "Error running filter tikz.py".

    C:\Users\Richard\Desktop\pandoc_question>pandoc -o temp.pdf --filter 
tikz.py temp.md
    pandoc: Error running filter tikz.py
    C:\Users\Richard\Desktop\pandoc_question\tikz.py: createProcess: 
invalid argument (Exec format error)

Do these cover all the cases you wanted to see?

On Tuesday, June 16, 2015 at 2:13:27 PM UTC-4, John MacFarlane wrote:
>
> Can you try (with a relatively recent pandoc) with simply 
>
>     --filter tikz.py 
>
>
> +++ Richard Herron [Jun 15 15 09:17 ]: 
> >   This is a cross-post from Stackoverflow. I asked the question 8 days 
> >   ago, but it didn't receive much attention. 
> >   Here's the question 
> >   link: 
> http://stackoverflow.com/questions/30696797/how-to-use-python-fil 
> >   ter-with-pandoc-to-convert-md-with-tikz-to-html-on-windows 
> >   You may want to answer the question there (then paste here), because I 
> >   have a 100 point bounty. Sorry for the cross-post. 
> >   --- 
> >   I am trying to use a Pandoc filter to convert a markdown file with a 
> >   tikz picture to html. I am on Win 8.1 (and I have all the dependencies 
> >   -- pdflatex, Python 2.7, ImageMagick, and the pandocfilters Python 
> >   package). I am using the tikz.py script that John MacFarlane provides 
> >   on [github](https://github.com/jgm/pandocfilters). 
> >   I found a similar [question](https://github.com/jgm/pandoc/issues/1096) 
>
> >   on the Pandoc Google Group and John MacFarlane suggests wrapping the 
> >   filter in a Windows batch script (the filter must be an executable). 
> >   Here is my command line input (I'll provide the file contents below). 
> >       pandoc -o temp.html --filter .\tikz.bat -s temp.md 
> >   But I keep getting the following error. 
> >       pandoc: Failed reading: satisfyElem 
> >   The script generates the "tikz-images" subfolder, but it is empty, as 
> >   is the resulting output file temp.html. 
> >   If I run `python tikz.py temp.md` from the Windows command prompt it 
> >   hangs. If I kill the command with `CTRL-C`, then I get the following. 
> >       C:\Users\Richard\Desktop\temp>python tikz.py temp.md 
> >       Traceback (most recent call last): 
> >         File "tikz.py", line 70, in <module> 
> >           toJSONFilter(tikz) 
> >   The same occurs with `python caps.py temp.md`. A hang, followed by: 
> >       C:\Users\Richard\Desktop\temp>python caps.py temp.md 
> >       Traceback (most recent call last): 
> >         File "caps.py", line 15, in <module> 
> >           toJSONFilter(caps) 
> >   How can I get this to work? FWIW, the bigger goal is for the input 
> >   files to be [R Markdown](http://rmarkdown.rstudio.com/), but I want 
> to 
> >   understand the filters in Pandoc Markdown alone first. 
> >   Here are the file contents. 
> >   tikz.bat 
> >       python tikz.py %* 
> >   temp.md 
> >       \begin{tikzpicture} 
> >       \draw [<->](-3,0)--(3,0); 
> >       \draw (-2,-.2)--(-2,.2); 
> >       \draw (-1,-.2)--(-1,.2); 
> >       \draw(0,-.2)--(0,.2); 
> >       \draw (1,-.2)--(1,.2); 
> >       \draw (2,-.2)--(2,.2); 
> >       \node[align=left,below] at (-4.5,-0.2) {Cash flow}; 
> >       \node[align=left,above] at (-4.5,0.2) {Time period}; 
> >       \node[align=left,above] at (-2,0.2) {-2}; 
> >       \node[align=left,above] at (-1,0.2) {-1}; 
> >       \node[align=left,above] at (0,0.2) {0}; 
> >       \node[align=left,above] at (1,0.2) {+1}; 
> >       \node[align=left,above] at (2,0.2) {+2}; 
> >       \node[align=left,below] at (1,-0.2) {\$100}; 
> >       \node[align=left,below] at (2,-0.2) {\$100}; 
> >       \end{tikzpicture} 
> >       Can this work? 
> >   tikz.py 
> >       #!/usr/bin/env python 
> >       """ 
> >       Pandoc filter to process raw latex tikz environments into images. 
> >       Assumes that pdflatex is in the path, and that the standalone 
> >       package is available.  Also assumes that ImageMagick's convert 
> >       is in the path. Images are put in the tikz-images directory. 
> >       """ 
> >       import hashlib 
> >       import re 
> >       import os 
> >       import sys 
> >       import shutil 
> >       from pandocfilters import toJSONFilter, Para, Image 
> >       from subprocess import Popen, PIPE, call 
> >       from tempfile import mkdtemp 
> >       imagedir = "tikz-images" 
> >       def sha1(x): 
> >           return 
> >   hashlib.sha1(x.encode(sys.getfilesystemencoding())).hexdigest() 
> >       def tikz2image(tikz, filetype, outfile): 
> >           tmpdir = mkdtemp() 
> >           olddir = os.getcwd() 
> >           os.chdir(tmpdir) 
> >           f = open('tikz.tex', 'w') 
> >           f.write("""\\documentclass{standalone} 
> >                    \\usepackage{tikz} 
> >                    \\begin{document} 
> >                    """) 
> >           f.write(tikz) 
> >           f.write("\n\\end{document}\n") 
> >           f.close() 
> >           p = call(["pdflatex", 'tikz.tex'], stdout=sys.stderr) 
> >           os.chdir(olddir) 
> >           if filetype == 'pdf': 
> >               shutil.copyfile(tmpdir + '/tikz.pdf', outfile + '.pdf') 
> >           else: 
> >               call(["convert", tmpdir + '/tikz.pdf', outfile + '.' + 
> >   filetype]) 
> >           shutil.rmtree(tmpdir) 
> >       def tikz(key, value, format, meta): 
> >           if key == 'RawBlock': 
> >               [fmt, code] = value 
> >               if fmt == "latex" and re.match("\\\\begin{tikzpicture}", 
> >   code): 
> >                   outfile = imagedir + '/' + sha1(code) 
> >                   if format == "html": 
> >                       filetype = "png" 
> >                   elif format == "latex": 
> >                       filetype = "pdf" 
> >                   else: 
> >                       filetype = "png" 
> >                   src = outfile + '.' + filetype 
> >                   if not os.path.isfile(src): 
> >                       try: 
> >                           os.mkdir(imagedir) 
> >                           sys.stderr.write('Created directory ' + 
> >   imagedir + '\n') 
> >                       except OSError: 
> >                           pass 
> >                       tikz2image(code, filetype, outfile) 
> >                       sys.stderr.write('Created image ' + src + '\n') 
> >                   return Para([Image([], [src, ""])]) 
> >       if __name__ == "__main__": 
> >           toJSONFilter(tikz) 
> > 
> >   -- 
> >   You received this message because you are subscribed to the Google 
> >   Groups "pandoc-discuss" group. 
> >   To unsubscribe from this group and stop receiving emails from it, send 
> >   an email to [1]pandoc-discus...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <javascript:>. 
> >   To post to this group, send email to 
> >   [2]pandoc-...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <javascript:>. 
> >   To view this discussion on the web visit 
> >   [3]
> https://groups.google.com/d/msgid/pandoc-discuss/33c5e06a-2bac-4c44- 
> >   9da8-073ca2bf624b%40googlegroups.com. 
> >   For more options, visit [4]https://groups.google.com/d/optout. 
> > 
> >References 
> > 
> >   1. mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <javascript:> 
> >   2. mailto:pandoc-...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <javascript:> 
> >   3. 
> https://groups.google.com/d/msgid/pandoc-discuss/33c5e06a-2bac-4c44-9da8-073ca2bf624b-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org?utm_medium=email&utm_source=footer 
> >   4. https://groups.google.com/d/optout 
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0168f345-1f44-40a4-bc6e-9fb9925044e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 19391 bytes --]

Re: How to use Python filter with Pandoc to convert md with tikz to html on Windows 8.1

[John MacFarlane]

+++ Richard Herron [Jun 16 15 14:06 ]:
>   Thanks, John. I should have mentioned that I'm using 1.13.2
>       C:\Users\Richard\Desktop\pandoc_question>pandoc --version
>       pandoc 1.13.2
>       Compiled with texmath 0.8.0.1, highlighting-kate 0.5.11.1.
>   If I run pandoc with only `--filter tikz.py`, then it hangs and prints
>   just a "p" when a press CTRL-C.

It's waiting for input from the terminal or a pipe.  You
should specify the name of the input file.

>   If I run tikz.py with full arguments, but without the batch file
>   wrapper, then I get "Error running filter tikz.py".
>       C:\Users\Richard\Desktop\pandoc_question>pandoc -o temp.pdf
>   --filter tikz.py temp.md
>       pandoc: Error running filter tikz.py
>       C:\Users\Richard\Desktop\pandoc_question\tikz.py: createProcess:
>   invalid argument (Exec format error)

I don't use Windows.  Can anyone out there who uses python filters
with Windows comment?