Running mtxrun in a continuous batch mode

Running mtxrun in a continuous batch mode

[Ramkumar KB]

[-- Attachment #1.1: Type: text/plain, Size: 720 bytes --]

Hello,

I am trying to run mtxrun in a batch process and read the available
documentation here -
https://wiki.contextgarden.net/Running_ConTeXt_without_a_shell

My usecase is being able to create PDF docs based on the input data that
comes in a JSON format (in a continuous batch process). In ConTeXt, thanks
to the Lua engine, I am able to inject a JSON read from a file - and use
the JSON in the tex file (here variable `tab` holds the JSON).

How do I make this as a continuous webservice - in comes JSON and out comes
the PDF ?

\startluacode
require("test-json")
\stopluacode

The name of the document is \ctxlua{tex.print(tab['documentName'])}

Any tips or pointers would be much appreciated.

Thank you,
Ramkumar

[-- Attachment #1.2: Type: text/html, Size: 1974 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Hans Hagen]

On 11/3/2020 1:55 AM, Ramkumar KB wrote:
> Hello,
> 
> I am trying to run mtxrun in a batch process and read the available 
> documentation here - 
> https://wiki.contextgarden.net/Running_ConTeXt_without_a_shell
> 
> My usecase is being able to create PDF docs based on the input data that 
> comes in a JSON format (in a continuous batch process). In ConTeXt, 
> thanks to the Lua engine, I am able to inject a JSON read from a file - 
> and use the JSON in the tex file (here variable `tab` holds the JSON).
> 
> How do I make this as a continuous webservice - in comes JSON and out 
> comes the PDF ?
> 
> \startluacode
> require("test-json")
> \stopluacode
> 
> The name of the document is \ctxlua{tex.print(tab['documentName'])}
> 
> Any tips or pointers would be much appreciated.
foo.cld :

require("util-jsn")

-- local str = io.loaddata("somefile.json")

local str = [[ {
     "title": "Some JSON",
     "text" : "Just an example.",
     "data" : [
         { "a" : "first 1", "b" : "last 1" },
         { "b" : "last 2", "a" : "first 2" }
     ]
} ]]

local tmp = utilities.json.tolua(str)

context.starttext()

     context.starttitle { title = tmp.title }

         context(tmp.text) context.par()

         context.starttabulate { "|c|c|" }
             for i=1,#tmp.data do
                 local d = tmp.data[i]
                 context.NC()
                     context(d.a) context.NC()
                     context(d.b) context.NC()
                 context.NR()
             end
         context.stoptabulate()

     context.stoptitle()

context.stoptext()

and then

context foo.cld


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Ramkumar KB]

[-- Attachment #1.1: Type: text/plain, Size: 2986 bytes --]

Hans,

Thank you so much for the kind response.

Apologies that I was not very clear in my initial query. I have -

   1. MyStatement.tex, with all the necessary fonts, static text, images,
   header, footer etc
   2. MyStatement.tex reads from a JSON file for certain sections of the
   document for the dynamic content
   3. context MyStatement.tex produces MyStatement.pdf

How do I do the above steps 1 to 3 in a continuous fashion such that -

   - Output is MyStatement_1.pdf (reads from data_1.json), MyStatement_2.pdf
   (reads from data_2.json) and so on
   - The tex processing is fast as fonts etc are loaded once (as the Tex
   template is same for all the MyStatement_n.pdf)

From the sample code that you gave, I get a hint that this can be possibly
done using ConTeXt Lua Document approach. Is this correct?

Thank you once again,

Best,
Ramkumar

On Tue, Nov 3, 2020 at 7:02 PM Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 11/3/2020 1:55 AM, Ramkumar KB wrote:
> > Hello,
> >
> > I am trying to run mtxrun in a batch process and read the available
> > documentation here -
> > https://wiki.contextgarden.net/Running_ConTeXt_without_a_shell
> >
> > My usecase is being able to create PDF docs based on the input data that
> > comes in a JSON format (in a continuous batch process). In ConTeXt,
> > thanks to the Lua engine, I am able to inject a JSON read from a file -
> > and use the JSON in the tex file (here variable `tab` holds the JSON).
> >
> > How do I make this as a continuous webservice - in comes JSON and out
> > comes the PDF ?
> >
> > \startluacode
> > require("test-json")
> > \stopluacode
> >
> > The name of the document is \ctxlua{tex.print(tab['documentName'])}
> >
> > Any tips or pointers would be much appreciated.
> foo.cld :
>
> require("util-jsn")
>
> -- local str = io.loaddata("somefile.json")
>
> local str = [[ {
>      "title": "Some JSON",
>      "text" : "Just an example.",
>      "data" : [
>          { "a" : "first 1", "b" : "last 1" },
>          { "b" : "last 2", "a" : "first 2" }
>      ]
> } ]]
>
> local tmp = utilities.json.tolua(str)
>
> context.starttext()
>
>      context.starttitle { title = tmp.title }
>
>          context(tmp.text) context.par()
>
>          context.starttabulate { "|c|c|" }
>              for i=1,#tmp.data do
>                  local d = tmp.data[i]
>                  context.NC()
>                      context(d.a) context.NC()
>                      context(d.b) context.NC()
>                  context.NR()
>              end
>          context.stoptabulate()
>
>      context.stoptitle()
>
> context.stoptext()
>
> and then
>
> context foo.cld
>
>
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
>

[-- Attachment #1.2: Type: text/html, Size: 4476 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Hans Hagen]

On 11/4/2020 4:48 AM, Ramkumar KB wrote:
> Hans,
> 
> Thank you so much for the kind response.
> 
> Apologies that I was not very clear in my initial query. I have -
> 
>  1. MyStatement.tex, with all the necessary fonts, static text, images,
>     header, footer etc
>  2. MyStatement.tex reads from a JSON file for certain sections of the
>     document for the dynamic content
>  3. context MyStatement.tex produces MyStatement.pdf 
> 
> How do I do the above steps 1 to 3 in a continuous fashion such that -
> 
>   * Output is MyStatement_1.pdf (reads from data_1.json),
>     MyStatement_2.pdf (reads from data_2.json) and so on
>   * The tex processing is fast as fonts etc are loaded once (as the Tex
>     template is same for all the MyStatement_n.pdf)
> 
>  From the sample code that you gave, I get a hint that this can be 
> possibly done using ConTeXt Lua Document approach. Is this correct?
i'd just write a script that calls context like

context mystatement.tex --myjsonsection=1 --batch
context mystatement.tex --myjsonsection=2 --batch
context mystatement.tex --myjsonsection=3 --batch

etc .. maybe --once if no multipass is needed, or --runs=2 if you know 
how many runs are needed

to stay in a run and kind of restart is asking for troubles because what 
should be reset? of course i could context make do that buit it doesn't 
pay off

an alternative that you generate one document with all statements and 
use mutool to split of the pages

but anyway, nowadays machines are fast enough to have separate runs (and 
one can run them in parallel)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Ramkumar KB]

[-- Attachment #1.1: Type: text/plain, Size: 2204 bytes --]

Hans,

Thank you for the suggestions. Let me try them out and feedback back to you
(as the batch set-up involves > 10,000 docs).

Best,
Ramkumar

On Wed, Nov 4, 2020 at 7:24 PM Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 11/4/2020 4:48 AM, Ramkumar KB wrote:
> > Hans,
> >
> > Thank you so much for the kind response.
> >
> > Apologies that I was not very clear in my initial query. I have -
> >
> >  1. MyStatement.tex, with all the necessary fonts, static text, images,
> >     header, footer etc
> >  2. MyStatement.tex reads from a JSON file for certain sections of the
> >     document for the dynamic content
> >  3. context MyStatement.tex produces MyStatement.pdf
> >
> > How do I do the above steps 1 to 3 in a continuous fashion such that -
> >
> >   * Output is MyStatement_1.pdf (reads from data_1.json),
> >     MyStatement_2.pdf (reads from data_2.json) and so on
> >   * The tex processing is fast as fonts etc are loaded once (as the Tex
> >     template is same for all the MyStatement_n.pdf)
> >
> >  From the sample code that you gave, I get a hint that this can be
> > possibly done using ConTeXt Lua Document approach. Is this correct?
> i'd just write a script that calls context like
>
> context mystatement.tex --myjsonsection=1 --batch
> context mystatement.tex --myjsonsection=2 --batch
> context mystatement.tex --myjsonsection=3 --batch
>
> etc .. maybe --once if no multipass is needed, or --runs=2 if you know
> how many runs are needed
>
> to stay in a run and kind of restart is asking for troubles because what
> should be reset? of course i could context make do that buit it doesn't
> pay off
>
> an alternative that you generate one document with all statements and
> use mutool to split of the pages
>
> but anyway, nowadays machines are fast enough to have separate runs (and
> one can run them in parallel)
>
> Hans
>
>
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
>

[-- Attachment #1.2: Type: text/html, Size: 2979 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[thierry horsin]

[-- Attachment #1.1: Type: text/plain, Size: 3418 bytes --]

Hi Ramkumar,

Maybe you could have some scripts that check when the JSON file is 
modified and if so launches the compilation. I did that sort of script 
in order to have a continuous compilation process by checking every 
second if my current .tex file is modified.

Best

Thierry

On 06/11/2020 09:38, Ramkumar KB wrote:
> Hans,
>
> Thank you for the suggestions. Let me try them out and feedback back 
> to you (as the batch set-up involves > 10,000 docs).
>
> Best,
> Ramkumar
>
> On Wed, Nov 4, 2020 at 7:24 PM Hans Hagen <j.hagen@xs4all.nl 
> <mailto:j.hagen@xs4all.nl>> wrote:
>
>     On 11/4/2020 4:48 AM, Ramkumar KB wrote:
>     > Hans,
>     >
>     > Thank you so much for the kind response.
>     >
>     > Apologies that I was not very clear in my initial query. I have -
>     >
>     >  1. MyStatement.tex, with all the necessary fonts, static text,
>     images,
>     >     header, footer etc
>     >  2. MyStatement.tex reads from a JSON file for certain sections
>     of the
>     >     document for the dynamic content
>     >  3. context MyStatement.tex produces MyStatement.pdf
>     >
>     > How do I do the above steps 1 to 3 in a continuous fashion such
>     that -
>     >
>     >   * Output is MyStatement_1.pdf (reads from data_1.json),
>     >     MyStatement_2.pdf (reads from data_2.json) and so on
>     >   * The tex processing is fast as fonts etc are loaded once (as
>     the Tex
>     >     template is same for all the MyStatement_n.pdf)
>     >
>     >  From the sample code that you gave, I get a hint that this can be
>     > possibly done using ConTeXt Lua Document approach. Is this correct?
>     i'd just write a script that calls context like
>
>     context mystatement.tex --myjsonsection=1 --batch
>     context mystatement.tex --myjsonsection=2 --batch
>     context mystatement.tex --myjsonsection=3 --batch
>
>     etc .. maybe --once if no multipass is needed, or --runs=2 if you
>     know
>     how many runs are needed
>
>     to stay in a run and kind of restart is asking for troubles
>     because what
>     should be reset? of course i could context make do that buit it
>     doesn't
>     pay off
>
>     an alternative that you generate one document with all statements and
>     use mutool to split of the pages
>
>     but anyway, nowadays machines are fast enough to have separate
>     runs (and
>     one can run them in parallel)
>
>     Hans
>
>
>     -----------------------------------------------------------------
>                                                Hans Hagen | PRAGMA ADE
>                    Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>             tel: 038 477 53 69 | www.pragma-ade.nl
>     <http://www.pragma-ade.nl> | www.pragma-pod.nl
>     <http://www.pragma-pod.nl>
>     -----------------------------------------------------------------
>
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
>
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________

[-- Attachment #1.2: Type: text/html, Size: 5674 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Aditya Mahajan]

On Fri, 6 Nov 2020, thierry horsin wrote:

> Hi Ramkumar,
> 
> Maybe you could have some scripts that check when the JSON file is modified
> and if so launches the compilation. I did that sort of script in order to have
> a continuous compilation process by checking every second if my current .tex
> file is modified.

Here is a script which does this for general tasks:

http://users.fred.net/tds/lab/ftp/atchange

It is also possible to build something on top of inotifywait on linux.

Aditya
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Running mtxrun in a continuous batch mode

[Ramkumar KB]

[-- Attachment #1.1: Type: text/plain, Size: 1405 bytes --]

Hi,

Thank you for the tips. Yes, I will probably write something similar (but
probably in Go <https://github.com/fsnotify/fsnotify> or Python
<https://github.com/gorakhargosh/watchdog>, as they are easier to maintain
in the environments that I work).

Thank you very much!
Ramkumar

On Fri, Nov 6, 2020 at 10:52 PM Aditya Mahajan <adityam@umich.edu> wrote:

> On Fri, 6 Nov 2020, thierry horsin wrote:
>
> > Hi Ramkumar,
> >
> > Maybe you could have some scripts that check when the JSON file is
> modified
> > and if so launches the compilation. I did that sort of script in order
> to have
> > a continuous compilation process by checking every second if my current
> .tex
> > file is modified.
>
> Here is a script which does this for general tasks:
>
> http://users.fred.net/tds/lab/ftp/atchange
>
> It is also possible to build something on top of inotifywait on linux.
>
> Aditya
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to
> the Wiki!
>
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
>
> ___________________________________________________________________________________
>

[-- Attachment #1.2: Type: text/html, Size: 2404 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________