externalizing TikZ graphics

externalizing TikZ graphics

[Henning Hraban Ramm]

Hi,

did anyone succeed in externalizing TikZ graphics with ConTeXt?

It’s about a project with hundreds of diagrams that (in the current state) clutter the main directory with cryptically-named PDFs (at least sorted by component name) and that on each ConTeXt run take a lot of runtime (I guess also the many log lines slow the process).

In 2013, Aditya said it wasn’t possible to externalize pgf/TikZ graphics as documented in the pgf manual, and "perhaps" it would make sense to use his filter module:

https://mailman.ntg.nl/pipermail/ntg-context/2013/071162.html

Did anyone try to do that? I didn’t find anything in this list’s archives.

The original pgf way is two-pass, and you’d need to call LaTeX with the intended graphics name as --jobname; that’s far too much of a hassle for that many graphics.

I generally like how the filter module works, as you can also define the name of every buffer, but my only experience with that is my LilyPond setup.

If nobody has better advice, I’ll try to come up with a filter setup. That would mean we’d avoid the tikz module, every TikZ diagram will get written to a buffer of configurable name, and that (with preamble setup) will get run through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps we can also use LuaTeX in plain mode to avoid the LaTeX dependency.

Best, Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Henri Menke]

On Mon, 2021-01-25 at 16:42 +0100, Henning Hraban Ramm wrote:
> Hi,
> 
> did anyone succeed in externalizing TikZ graphics with ConTeXt?

The external library is LaTeX-only and that unfortunately won't change because
it heavily relies on implementation detail of LaTeX's output routine.

> It’s about a project with hundreds of diagrams that (in the current state)
> clutter the main directory with cryptically-named PDFs (at least sorted by
> component name) and that on each ConTeXt run take a lot of runtime (I guess
> also the many log lines slow the process).
> 
> In 2013, Aditya said it wasn’t possible to externalize pgf/TikZ graphics as
> documented in the pgf manual, and "perhaps" it would make sense to use his
> filter module:

filter is your best bet, in my opinion.

> https://mailman.ntg.nl/pipermail/ntg-context/2013/071162.html
> 
> Did anyone try to do that? I didn’t find anything in this list’s archives.
> 
> The original pgf way is two-pass, and you’d need to call LaTeX with the
> intended graphics name as --jobname; that’s far too much of a hassle for that
> many graphics.
> 
> I generally like how the filter module works, as you can also define the name
> of every buffer, but my only experience with that is my LilyPond setup.
> 
> If nobody has better advice, I’ll try to come up with a filter setup. That
> would mean we’d avoid the tikz module, every TikZ diagram will get written to
> a buffer of configurable name, and that (with preamble setup) will get run
> through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps
> we can also use LuaTeX in plain mode to avoid the LaTeX dependency.
> 
> Best, Hraban
> ______________________________________________________________________________
> _____
> If your question is of interest to others as well, please add an entry to the
> Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ______________________________________________________________________________
> _____

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Hans Hagen]

On 1/25/2021 4:42 PM, Henning Hraban Ramm wrote:
> Hi,
> 
> did anyone succeed in externalizing TikZ graphics with ConTeXt?
> 
> It’s about a project with hundreds of diagrams that (in the current state) clutter the main directory with cryptically-named PDFs (at least sorted by component name) and that on each ConTeXt run take a lot of runtime (I guess also the many log lines slow the process).
> 
> In 2013, Aditya said it wasn’t possible to externalize pgf/TikZ graphics as documented in the pgf manual, and "perhaps" it would make sense to use his filter module:
> 
> https://mailman.ntg.nl/pipermail/ntg-context/2013/071162.html
> 
> Did anyone try to do that? I didn’t find anything in this list’s archives.
> 
> The original pgf way is two-pass, and you’d need to call LaTeX with the intended graphics name as --jobname; that’s far too much of a hassle for that many graphics.
> 
> I generally like how the filter module works, as you can also define the name of every buffer, but my only experience with that is my LilyPond setup.
> 
> If nobody has better advice, I’ll try to come up with a filter setup. That would mean we’d avoid the tikz module, every TikZ diagram will get written to a buffer of configurable name, and that (with preamble setup) will get run through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps we can also use LuaTeX in plain mode to avoid the LaTeX dependency.
In the following case the buffer only gets typeset when it changes:

\starttext

\startbuffer[foo]
     % maybe include a style / enviroment
     \usemodule[tikz] \usetikzlibrary[patterns]
     \startTEXpage
         \starttikzpicture
            \draw[pattern color=red,pattern=bricks] (0,0) circle (1cm);
         \stoptikzpicture
     \stopTEXpage
\stopbuffer

\framed[offset=overlay]{\typesetbuffer[foo]}

\stoptext

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Henning Hraban Ramm]

> Am 25.01.2021 um 17:31 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> 
>> If nobody has better advice, I’ll try to come up with a filter setup. That would mean we’d avoid the tikz module, every TikZ diagram will get written to a buffer of configurable name, and that (with preamble setup) will get run through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps we can also use LuaTeX in plain mode to avoid the LaTeX dependency.
> In the following case the buffer only gets typeset when it changes:

Thank you (and Henri)! But we’d need to include the same setup header every time, so the filter module is probably really the best way.

There’s just a problem with labels, I guess, if I go the filter-LaTeX or filter-TeX route. So filter-ConTeXt might make sense.
I think I understand now why integrating MetaPost made a lot of sense.

Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Aditya Mahajan]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

On Mon, 25 Jan 2021, Henning Hraban Ramm wrote:

> 
> > Am 25.01.2021 um 17:31 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> > 
> >> If nobody has better advice, I’ll try to come up with a filter setup. That would mean we’d avoid the tikz module, every TikZ diagram will get written to a buffer of configurable name, and that (with preamble setup) will get run through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps we can also use LuaTeX in plain mode to avoid the LaTeX dependency.
> > In the following case the buffer only gets typeset when it changes:
> 
> Thank you (and Henri)! But we’d need to include the same setup header every time, so the filter module is probably really the best way.

It is not only the tikz setup; you also need to copy the font setup etc. 

\typesetbuffer[setups,figure]

or some variant of that should also work. But the filter module does make it easier to store the results in another directory so that they are out of sight.

> There’s just a problem with labels, I guess, if I go the filter-LaTeX or filter-TeX route. So filter-ConTeXt might make sense.
> I think I understand now why integrating MetaPost made a lot of sense.

Here is a simple example:


\usemodule[filter]

\startbuffer[tikz-before]
\usemodule[tikz]
\startTEXpage
\starttikzpicture
\stopbuffer

\startbuffer[tikz-after]
\stoptikzpicture
\stopTEXpage
\stopbuffer

\defineexternalfilter
  [externaltikz]
  [
    bufferbefore=tikz-before,
    bufferafter=tikz-after,
    cache=yes,
    filtercommand={context --batchmode \externalfilterinputfile\space --purgeall --result=\externalfilteroutputfile},
    output={\externalfilterbasefile.pdf},
    readcommand=\readPDFfile,
  ]

\define[1]\readPDFfile{\externalfigure[#1]} % width and height could be added

\starttext
\startexternaltikz
  \draw (0,0) -- (1,1);
\stopexternaltikz

\stoptext



[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Henning Hraban Ramm]

> Am 25.01.2021 um 18:58 schrieb Aditya Mahajan <adityam@umich.edu>:
> 
> or some variant of that should also work. But the filter module does make it easier to store the results in another directory so that they are out of sight.

No, you need additional file system commands.

> Here is a simple example:

Thank you!
Meanwhile, looking at my LilyPond module, I got as far as:

"""
\startmodule[tikzfilter]

\usemodule[filter]
% docs see https://github.com/adityam/filter

\def\TIKZTEMP{tikztemp} % name of folder for TikZ/buffer files

\startbuffer[tikz::before]
\environment env_world % contains font setup etc.
\environment env_TikZ % contains TikZ settings

\startpagefigure
\starttikzpicture[thick,domain=-90:90]% TODO: make this configurable
\stopbuffer

\startbuffer[tikz::after]
\stoptikzpicture
\stoppagefigure
\stopbuffer

\define[1]\ParseTikZFile% #1 is the name of the output file
  {\ctxlua{thirddata.parsetikzfile("#1")}}

\startluacode
 thirddata = thirddata or {}

 -- create temp folder if missing
 if not lfs.isdir("\TIKZTEMP") then
   lfs.mkdir("\TIKZTEMP")
 end

 function thirddata.parsetikzfile(name)
   -- name is like \TIKZTEMP/mainfile-temp-tikz-21.pdf
   logs.report("TIKZFILTER","name='" .. name .. "'")
   context("\\externalfigure[" .. name .. "]")
 end
\stopluacode

\defineexternalfilter[tikz][
  cache=yes,
  purge=no,
  directory=\TIKZTEMP,
  bufferbefore={tikz::before},
  bufferafter={tikz::after},
  readcommand=\ParseTikZFile,
  output={\externalfilterbasefile.pdf},
% ConTeXt never writes results to other directories
  filtercommand={context --batchmode --once --autopdf=auto --result=\externalfilterbasefile\space \externalfilterinputfile\space && mv \externalfilterbasefile.pdf\space \TIKZTEMP/},
]

\stopmodule


Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: externalizing TikZ graphics

[Henning Hraban Ramm]

> Am 25.01.2021 um 20:08 schrieb Henning Hraban Ramm <texml@fiee.net>:
> 
> """
> \startmodule[tikzfilter]
> 
> \usemodule[filter]
> % docs see https://github.com/adityam/filter
> 
> \def\TIKZTEMP{tikztemp} % name of folder for TikZ/buffer files
> 
> \startbuffer[tikz::before]
> \environment env_world % contains font setup etc.
> \environment env_TikZ % contains TikZ settings
> 
> \startpagefigure

Wrong, I need \startTEXpage

> \starttikzpicture[thick,domain=-90:90]% TODO: make this configurable

Now, how can I forward options from
\starttix[anything]
to this place?

> \stopbuffer
> 
> \startbuffer[tikz::after]
> \stoptikzpicture
> \stoppagefigure

\stopTEXpage

> \stopbuffer
> 
> \define[1]\ParseTikZFile% #1 is the name of the output file
>  {\ctxlua{thirddata.parsetikzfile("#1")}}
> 
> \startluacode
> thirddata = thirddata or {}
> 
> -- create temp folder if missing
> if not lfs.isdir("\TIKZTEMP") then
>   lfs.mkdir("\TIKZTEMP")
> end
> 
> function thirddata.parsetikzfile(name)
>   -- name is like \TIKZTEMP/mainfile-temp-tikz-21.pdf
>   logs.report("TIKZFILTER","name='" .. name .. "'")
>   context("\\externalfigure[" .. name .. "]")
> end
> \stopluacode
> 
> \defineexternalfilter[tikz][
>  cache=yes,
>  purge=no,
>  directory=\TIKZTEMP,
>  bufferbefore={tikz::before},
>  bufferafter={tikz::after},
>  readcommand=\ParseTikZFile,
>  output={\externalfilterbasefile.pdf},
> % ConTeXt never writes results to other directories
>  filtercommand={context --batchmode --once --autopdf=auto --result=\externalfilterbasefile\space \externalfilterinputfile\space && mv \externalfilterbasefile.pdf\space \TIKZTEMP/},
> ]
> 
> \stopmodule



___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: filter module / externalizing TikZ graphics

[Henning Hraban Ramm]

Still working on that.
I hope you can help me here, Aditya!
I’ll add these issues to the tracker of t-filter, if it makes sense.


Testing with: ConTeXt  ver: 2021.01.24 16:07 LMTX  fmt: 2021.1.25

The setup below works, but only with \starttikz ... \stoptikz so far, I don’t like the output file names, and using the working directory is tricky.


(1) \processFILTERfile

\processtikzfile{img/example.pgf}
can’t find the file:

t-filter        > cached output file 'tikztemp/example1.pgf.pdf' missing. Rerunning filter
t-filter        > file 'tikztemp/example1.pgf.pdf' cannot be found
t-filter        > current filter : tikzThick
t-filter        > base file : example1.pgf
t-filter        > input file : img/example1.pgf
t-filter        > output file : tikztemp/example1.pgf.pdf
...
system          >   13: filename=img/example1.pgf foundname=img/example1.pgf fullname=.../img/example1.pgf usedmethod=direct

Looks like ConTeXt can find the file, but the filter module cannot.

--trackers=resolvers.schemes (or \enabletrackers) doesn’t change anything WRT log/output.

Since this fails already, I don’t know if \processFILTERfile also uses bufferbefore/-after (it should).


(2) working directory

Since ConTeXt can’t put generated (temporary) files into a directory and --result also only renames afterwards, the directory key doesn’t help, I had to add "mv" and "cd" to my filtercommand.
I didn’t check yet if caching works.


(3) file names

Using \processFILTERfile the original file name base is preserved, according to the log entry above.

If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp".
I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp".
Is there a configuration hook in the module to change name generation?


(4) subprocess parameters

I’d like to forward some parameters from \starttikz to the temp buffer, e.g. 
\starttikz[name=example][mystyle=thick]
(using the same syntax as \startsection with a second pair of brackets for custom parameters).
And then have
\starttikzpicture[\filterusersetupvariable{mystyle}]
in the buffer.

My workaround so far is a copy of the filter setup (tikz/tikzThick), since we have only a few different configurations.



Best regards, Hraban


""" % file: t-tikzfilter.tex

\startmodule[tikzfilter]

\usemodule[filter] % docs see https://github.com/adityam/filter

\def\TIKZTEMP{tikztemp} % name of folder for TikZ/buffer files


% normal setup
\startbuffer[tikz::before]
\environment env_world
\environment env_TikZ

\startTEXpage
\starttikzpicture
\stopbuffer

% setup for graphics with thicker lines
\startbuffer[tikzThick::before]
\environment env_world
\environment env_TikZ

\startTEXpage
\starttikzpicture[thick] % only difference
\stopbuffer


\startbuffer[tikz::after]
\stoptikzpicture
\stopTEXpage
\stopbuffer


\define[1]\ReadPDFFile% #1 is the name of the output file
  {\ctxlua{thirddata.readpdffile("#1")}}

\startluacode
 thirddata = thirddata or {}

 local report_tikz = logs.reporter("tikzfilter")

 -- create temp folder if missing
 if not lfs.isdir("\TIKZTEMP") then
   lfs.mkdir("\TIKZTEMP")
 end

 function thirddata.readpdffile(name)
   -- name is like \TIKZTEMP/mainfile-temp-tikz-21.pdf
   report_tikz("name='%s'", name) % doesn’t work
   context("\\externalfigure[" .. name .. "]")
 end
\stopluacode

\defineexternalfilter[tikz][
  cache=yes,
  % directory=\TIKZTEMP, % doesn’t work with ConTeXt
  bufferbefore={tikz::before},
  bufferafter={tikz::after},
  readcommand=\ReadPDFFile,
  output={\TIKZTEMP/\externalfilterbasefile.pdf},
  filtercommand={mv \externalfilterbasefile.* \TIKZTEMP/ && cd \TIKZTEMP\space && context --batchmode --nodummy --once \externalfilterbasefile.tmp},
]

\defineexternalfilter[tikzThick][tikz][
  bufferbefore={tikzThick::before},
]

\stopmodule
"""


""" % file: tikztest.tex

\usemodule[t-tikzfilter]
\usemodule[visual]


\starttext

\placefigure[margin,here][fig:example]{A box.}
{\vskip1in\noindent%
\starttikzThick[name=example0]
	\draw[fill=black!10] (0,4.75) rectangle (1.5,-4.75);
\stoptikzThick
}

\fakewords{150}{200}

\processtikzThickfile{tests/example1.pgf}

\stoptext
"""

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: filter module / externalizing TikZ graphics

[Aditya Mahajan]

[-- Attachment #1: Type: text/plain, Size: 4491 bytes --]

On Thu, 28 Jan 2021, Henning Hraban Ramm wrote:

> Still working on that.
> I hope you can help me here, Aditya!
> I’ll add these issues to the tracker of t-filter, if it makes sense.
> 
> 
> Testing with: ConTeXt  ver: 2021.01.24 16:07 LMTX  fmt: 2021.1.25
> 
> The setup below works, but only with \starttikz ... \stoptikz so far, I don’t like the output file names, and using the working directory is tricky.
> 
> 
> (1) \processFILTERfile
> 
> \processtikzfile{img/example.pgf}
> can’t find the file:
> 
> t-filter        > cached output file 'tikztemp/example1.pgf.pdf' missing. Rerunning filter
> t-filter        > file 'tikztemp/example1.pgf.pdf' cannot be found
> t-filter        > current filter : tikzThick
> t-filter        > base file : example1.pgf
> t-filter        > input file : img/example1.pgf
> t-filter        > output file : tikztemp/example1.pgf.pdf
> ...
> system          >   13: filename=img/example1.pgf foundname=img/example1.pgf fullname=.../img/example1.pgf usedmethod=direct
> 
> Looks like ConTeXt can find the file, but the filter module cannot.
> 
> --trackers=resolvers.schemes (or \enabletrackers) doesn’t change anything WRT log/output.

For \processFILTER, t-filter does not even read the file... so it has to do with how the filtercommand is defined. 

> Since this fails already, I don’t know if \processFILTERfile also uses bufferbefore/-after (it should).

It does not. As I said, for processing external files, t-filter does not even read the file and just takes care of caching the result. It is assumed that the filter command can process the external file. 

I can see if it is possible to active before and after filters while processing external files. Another option is to write the external files as valid tex file (i.e., with a \startTEXpage .. \stopTEXpage wrapper), and when defining the filtercommand, use something like `context --environment=env filename`, where env is the environment file with all the settings for fonts etc. 

> (2) working directory
> 
> Since ConTeXt can’t put generated (temporary) files into a directory and --result also only renames afterwards, the directory key doesn’t help, I had to add "mv" and "cd" to my filtercommand.
> I didn’t check yet if caching works.

Caching should work if \definefilter is configured correctly. 

From the point of view of t-filter module, this is a bug in the external program and you should contact upstream for a fix :-) Most (all?) other programs are happy to write a file in any directory (modulo write permissions, etc.). I do not want to increase the complexity of the module to handle such corner cases. 

> (3) file names
> 
> Using \processFILTERfile the original file name base is preserved, according to the log entry above.
> 
> If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp".
> I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp".
> Is there a configuration hook in the module to change name generation?

No. I don't quite understand the concern. How do you plan to use a `name=value` option?

> (4) subprocess parameters
> 
> I’d like to forward some parameters from \starttikz to the temp buffer, e.g. 
> \starttikz[name=example][mystyle=thick]
> (using the same syntax as \startsection with a second pair of brackets for custom parameters).
> And then have
> \starttikzpicture[\filterusersetupvariable{mystyle}]
> in the buffer.
> 
> My workaround so far is a copy of the filter setup (tikz/tikzThick), since we have only a few different configurations.

The code that I had posted already handles this, provided that the options start on the line after \startfilter:


\usemodule[filter]

\startbuffer[tikz-before]
\usemodule[tikz]
\startTEXpage
\starttikzpicture
\stopbuffer

\startbuffer[tikz-after]
\stoptikzpicture
\stopTEXpage
\stopbuffer

\defineexternalfilter
  [externaltikz]
  [
    bufferbefore=tikz-before,
    bufferafter=tikz-after,
    cache=yes,
    filtercommand={context --batchmode \externalfilterinputfile\space --purgeall --result=\externalfilteroutputfile},
    output={\externalfilterbasefile.pdf},
    readcommand=\readPDFfile,
  ]

\define[1]\readPDFfile{\externalfigure[#1]} % width and height could be added

\starttext
\startexternaltikz
    [very thick, ->]
  \draw (0,0) -- (1,1);
\stopexternaltikz

\stoptext

Aditya

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: filter module / externalizing TikZ graphics

[Henning Hraban Ramm]

> Am 30.01.2021 um 00:13 schrieb Aditya Mahajan <adityam@umich.edu>:
> 
> For \processFILTER, t-filter does not even read the file... so it has to do with how the filtercommand is defined. 
> 
>> Since this fails already, I don’t know if \processFILTERfile also uses bufferbefore/-after (it should).
> 
> It does not. As I said, for processing external files, t-filter does not even read the file and just takes care of caching the result. It is assumed that the filter command can process the external file. 

A pity. I didn’t understand that from the documentation.
That means I need to setup different filters for internal and external graphics with the same content.

> I can see if it is possible to active before and after filters while processing external files.

That would solve this problem for me and would make the behavior of filter commands more consistent.
OTOH I can understand you don’t want to bloat your module.

> Another option is to write the external files as valid tex file (i.e., with a \startTEXpage .. \stopTEXpage wrapper), and when defining the filtercommand, use something like `context --environment=env filename`, where env is the environment file with all the settings for fonts etc. 

I don’t see this as an option, as we try to avoid code duplication over hundreds of graphics snippets.
The \start/stop environment, as well as external files, should just contain the TikZ commands. All common setup/teardown code belongs in extra files/buffers.

>> (2) working directory
>> 
>> Since ConTeXt can’t put generated (temporary) files into a directory and --result also only renames afterwards, the directory key doesn’t help, I had to add "mv" and "cd" to my filtercommand.
>> I didn’t check yet if caching works.
> 
> Caching should work if \definefilter is configured correctly. 

Thanks.

> From the point of view of t-filter module, this is a bug in the external program and you should contact upstream for a fix :-)

Yes.
There were already several requests about ConTeXt using work/temp directories and changing the behavior of "--result".

> Most (all?) other programs are happy to write a file in any directory (modulo write permissions, etc.).

> I do not want to increase the complexity of the module to handle such corner cases. 

I can understand that.
I should probably let filter call an external script, or fork t-filter for my needs.


>> If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp".
>> I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp".
>> Is there a configuration hook in the module to change name generation?
> 
> No. I don't quite understand the concern. How do you plan to use a `name=value` option?

The current auto-generated file names are uncomfortably long, and I’d like to know which contains what.

I’m not yet sure what would make the most sense for this project (at least sorting by chapter/component would), but I have an use case with LilyPond where I could save several LilyPond compile runs if I could influence the naming:

I got ~500 songs in their own component files, and I use them in different songbooklets with the same layout. E.g. there’s one containing all the songs and several with a subject (e.g. winter & christmas, children, medieval & fantasy, some author...).
They all use [name=\currentcomponent]. If I could avoid the JOBNAME part of the file name, all booklets would use the same note PDFs. Now, the temp directory contains literally thousands of files.
Since this is just for fun, the additional space and compile time is just a bit annoying and doesn’t affect others, but I like to streamline my workflows...


>> (4) subprocess parameters
>> 
>> I’d like to forward some parameters from \starttikz to the temp buffer, e.g. 
>> \starttikz[name=example][mystyle=thick]
>> (using the same syntax as \startsection with a second pair of brackets for custom parameters).
>> And then have
>> \starttikzpicture[\filterusersetupvariable{mystyle}]
>> in the buffer.
>> 
>> My workaround so far is a copy of the filter setup (tikz/tikzThick), since we have only a few different configurations.
> 
> The code that I had posted already handles this, provided that the options start on the line after \startfilter:

Ah, I didn’t look closely enough. Thank you, that’s an usable workaround.


Best, Hraban

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: filter module / externalizing TikZ graphics

[Aditya Mahajan]

[-- Attachment #1: Type: text/plain, Size: 2943 bytes --]

On Sat, 30 Jan 2021, Henning Hraban Ramm wrote:

> >> If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp".
> >> I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp".
> >> Is there a configuration hook in the module to change name generation?
> > 
> > No. I don't quite understand the concern. How do you plan to use a `name=value` option?
> 
> The current auto-generated file names are uncomfortably long, and I’d like to know which contains what.

The idea was to avoid clash of names. That is why the file name starts with \jobname. I also want to avoid clash if you use multiple filters in the same file (a common use case for my projects), hence there is filtername. The -temp- is there because at some point context used to delete \jobname-temp-.tmp files if you passed the --purge option, but I see that it is no longer the case. Maybe that can be removed from the default name. The last part are the numbers, which are needed when caching results. 

> I’m not yet sure what would make the most sense for this project (at least sorting by chapter/component would), but I have an use case with LilyPond where I could save several LilyPond compile runs if I could influence the naming:
> 
> I got ~500 songs in their own component files, and I use them in different songbooklets with the same layout. E.g. there’s one containing all the songs and several with a subject (e.g. winter & christmas, children, medieval & fantasy, some author...).
> They all use [name=\currentcomponent]. If I could avoid the JOBNAME part of the file name, all booklets would use the same note PDFs. Now, the temp directory contains literally thousands of files.
> Since this is just for fun, the additional space and compile time is just a bit annoying and doesn’t affect others, but I like to streamline my workflows...

This is a relatively simple feature to add. I have pushed a new branch with this feature: https://github.com/adityam/filter/tree/prefix

I have added a new `prefix` option: you can use 

\defineexternalfilter[....][prefix=whatever]

and then the filter module will write the temp name as `whateverfilter0.ext` `whateverfilter1.ext`. The default value of prefix is \jobname-temp- (note trailing dash), so there should be no difference in the default output, but now you can configure the prefix if you want.

I am also open to making filter name as part of the prefix, so that you can have the ability to remove it if you wish. But I am hesitant to do so because it can lead to hard to debug errors if there is a clash in the prefix of two different filters. 

I haven't tested it beyond simple tests to make sure that nothing breaks. 
You can copy t-filter.mkiv from https://raw.githubusercontent.com/adityam/filter/prefix/t-filter.mkiv to your working directory and test if works for your situation. 


Aditya


[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: filter module / externalizing TikZ graphics

[Henning Hraban Ramm]

Finally coming back to this.

> Am 31.01.2021 um 07:39 schrieb Aditya Mahajan <adityam@umich.edu>:
> 
>> I’m not yet sure what would make the most sense for this project (at least sorting by chapter/component would), but I have an use case with LilyPond where I could save several LilyPond compile runs if I could influence the naming:
>> 
>> I got ~500 songs in their own component files, and I use them in different songbooklets with the same layout. E.g. there’s one containing all the songs and several with a subject (e.g. winter & christmas, children, medieval & fantasy, some author...).
>> They all use [name=\currentcomponent]. If I could avoid the JOBNAME part of the file name, all booklets would use the same note PDFs. Now, the temp directory contains literally thousands of files.
>> Since this is just for fun, the additional space and compile time is just a bit annoying and doesn’t affect others, but I like to streamline my workflows...
> 
> This is a relatively simple feature to add. I have pushed a new branch with this feature: https://github.com/adityam/filter/tree/prefix
> 
> I have added a new `prefix` option: you can use 
> 
> \defineexternalfilter[....][prefix=whatever]
> 
> and then the filter module will write the temp name as `whateverfilter0.ext` `whateverfilter1.ext`. The default value of prefix is \jobname-temp- (note trailing dash), so there should be no difference in the default output, but now you can configure the prefix if you want.

That’s good, so if I set "prefix=,", I get "FILTER-COMPONENT.*", e.g. "lilypond-mysong-1.pdf" (LilyPond adds the number for each system). In the rare cases where component names collide or I need more than one filter environment per component, I can fix that manually.

> I am also open to making filter name as part of the prefix, so that you can have the ability to remove it if you wish. But I am hesitant to do so because it can lead to hard to debug errors if there is a clash in the prefix of two different filters. 

Not needed IMO.

> I haven't tested it beyond simple tests to make sure that nothing breaks. 
> You can copy t-filter.mkiv from https://raw.githubusercontent.com/adityam/filter/prefix/t-filter.mkiv to your working directory and test if works for your situation. 

Works for me.

Thank you very much!

For the TikZ setup I’ll copy some of your code and sacrifice most of the configurability.
Will report back...

Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________