Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Stephen Gaito]

Hello,

I am slowly working on a Mathematical problem requiring underlying
computation.

As Mathematicians (myself included) are rather "conservative", I need
to discuss each "chunk" of code with the full set of Mathematical
notation.

A couple of years ago I started using ConTeXt-MKIV as a
Mathematically-Literate-Programming tool by using its excellent Lua
interface to capture the code and dump it to disk for external
compilation.

I am now revisiting my original design and want to redo my tools using
ConTeXt-LMTX.

I would *like* to be able to "stop" the ConTeXt typesetting at various
points for differing purposes:

1. After all macro expansions (and hence after *my* calls into Lua)
   but before line/paragraph/page layout begins.

2. After line/paragraph/page layout but before PDF generation.

3. After all PDF generated (ie. a "normal" "full" ConTeXt run).

Stopping after all macro expansions would allow my code generation
builds to proceed without the un-needed page setting or PDF generation.

Stopping after the line/paragraph/page layout would allow multiple
"faster(?)" ConTeXt runs while the "*.tuc" file converges to a complete
set of page numbers and cross references (etc). Then, once the "*.tuc"
file has converged, a full ConTeXt run with PDF output could be
done.

I am very aware that *internally* ConTeXt is probably structured as a
tight pipeline with each of the "traditional" TeX stages "Mouth",
"Stomach", "page setting", PDF generation.... tightly "chained"...
This means that there is no "one" place in the code where all macro
expansions have completed but before the page setting "starts", or
similarly, after the page setting has finished but before the PDF
generation "starts".

----
QUESTION: Is it possible to use the new LuaMetaTeX callbacks (found in
chapter 10 of the "LuaMetaTEX Reference Manual") to "suppress" any
further computation at various points in the ConTeXt pipeline?
----

For example, could I use one of the "*_linebreak_filter"s (or the
"append_to_vlist_filter") to "return" an empty value and hence reduce
further computation downstream in the pipeline?

Could I use the "pre_output_filter" to "return" an empty value and
hence "stop" PDF generation?

(I realize that these callbacks *are* a currently fast moving target. I
am happy to follow their changes, equally I would be testing their
usefulness and/or impact)

----
ALTERNATIVE QUESTION: Would it be possible to provide official
ConTeXt-LMTX "modes" which suppress further computation at these points?
----

This alternative, while some more work for the writing of
ConTeXt-LMTX, would ensure less direct external dependence on the
LuaMetaTeX callbacks, but would almost certainly be welcomed by the
ConTeXt community.

----
QUESTION: Are the "stages" I have identified major, computationally
expensive, "steps" in the overall ConTeXt "computation"?
----

Many thanks for an excellent tool!

Regards,

Stephen Gaito
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Hans Hagen]

On 11/30/2020 10:51 AM, Stephen Gaito wrote:
> Hello,
> 
> I am slowly working on a Mathematical problem requiring underlying
> computation.
> 
> As Mathematicians (myself included) are rather "conservative", I need
> to discuss each "chunk" of code with the full set of Mathematical
> notation.
> 
> A couple of years ago I started using ConTeXt-MKIV as a
> Mathematically-Literate-Programming tool by using its excellent Lua
> interface to capture the code and dump it to disk for external
> compilation.
> 
> I am now revisiting my original design and want to redo my tools using
> ConTeXt-LMTX.
> 
> I would *like* to be able to "stop" the ConTeXt typesetting at various
> points for differing purposes:
> 
> 1. After all macro expansions (and hence after *my* calls into Lua)
>     but before line/paragraph/page layout begins.

maybe something

\startmystuff

\stopmystuff

and then you can hook something into startmystuff and \stopmystuff

> 2. After line/paragraph/page layout but before PDF generation.

pdf is generated per page, if needed one can kick in a shipout overload

but keep in mind that multipass data is flushed as part of the shipout 
(because it is often location and order bound)

> 3. After all PDF generated (ie. a "normal" "full" ConTeXt run).
> 
> Stopping after all macro expansions would allow my code generation
> builds to proceed without the un-needed page setting or PDF generation.

hm, the problem is always in the 'state' of all kind of variables

> Stopping after the line/paragraph/page layout would allow multiple
> "faster(?)" ConTeXt runs while the "*.tuc" file converges to a complete
> set of page numbers and cross references (etc). Then, once the "*.tuc"
> file has converged, a full ConTeXt run with PDF output could be
> done.

not sure what you mean here ... what is fast? or: how slow is it now? 
what is the bottleneck? can you cache data that didn't change?

a large document is normally split up in sections that can be processed 
independent

\starttext
     \dorecurse{10000}{\samplefile{ward}\par}
\stoptext

runs on my 2013 laptop at over 65 pages per second

quite often performance is hit by inefficient styling and such .. it's 
no problem to bring a tex system a grinding halt

> I am very aware that *internally* ConTeXt is probably structured as a
> tight pipeline with each of the "traditional" TeX stages "Mouth",
> "Stomach", "page setting", PDF generation.... tightly "chained"...
> This means that there is no "one" place in the code where all macro
> expansions have completed but before the page setting "starts", or
> similarly, after the page setting has finished but before the PDF
> generation "starts".

yes and often something is left over for a next page so it's kind of fluid

> ----
> QUESTION: Is it possible to use the new LuaMetaTeX callbacks (found in
> chapter 10 of the "LuaMetaTEX Reference Manual") to "suppress" any
> further computation at various points in the ConTeXt pipeline?
> ----

sure, you can kick in handlers at various stages (assuming that you keep 
in mind where you kick them in as there is some order involved)

> For example, could I use one of the "*_linebreak_filter"s (or the
> "append_to_vlist_filter") to "return" an empty value and hence reduce
> further computation downstream in the pipeline?

you can but linebreak is not the most costly one, you probbaly want to 
intercept the list builder but when you do that you can as well do a 
\stoptext which prevents further reading of content (but i probably 
misunderstand)

> Could I use the "pre_output_filter" to "return" an empty value and
> hence "stop" PDF generation?

assuming a properky structured document forcing a \stoptext should work 
in most cases

> (I realize that these callbacks *are* a currently fast moving target. I
> am happy to follow their changes, equally I would be testing their
> usefulness and/or impact)

actually, the callbacks themselves hardly change, but the code plugged 
into them might occasionally (a lot of mkiv code is already quite old so 
i'm now looking at it and see if i can use some recent tricks)

> ALTERNATIVE QUESTION: Would it be possible to provide official
> ConTeXt-LMTX "modes" which suppress further computation at these points?

the question is: what do you want to suppress? best first identify the 
bottleneck and then figure out what can be skipped (as mentioned: 
multipass data can be made more independent I guess but it still demands 
some calculations and analyzing and it's that bit that takes the time)

> This alternative, while some more work for the writing of
> ConTeXt-LMTX, would ensure less direct external dependence on the
> LuaMetaTeX callbacks, but would almost certainly be welcomed by the
> ConTeXt community.

i need more info (also from others then) about what the reason, goal and 
possible gain is

- tex: the context code is quite efficient, and tex is quite fast, so 
there's little to gain there (but as said one can write slow macros that 
spoil that game)

- lua: on the average lua is fast but garbage collection can be of 
influence (i need to see code in order to be able to tell if there is a 
gain there); the lua code in context is quite ok but for instance 
messing with node lists will always come at a cost (crossing the c 
boundary and such)

- pdf: the backend code in luametatex is somewhat slower than in luatex 
but we're gaining there (because in related areas we can do things 
different, although there is new functionality that when used also comes 
  at a price); but as far as i can tell a luametatex run here is on the 
average some 20% faster than a luatex run so the pdf generation slowdown 
gets kind of obscured by it

> ----
> QUESTION: Are the "stages" I have identified major, computationally
> expensive, "steps" in the overall ConTeXt "computation"?
> ----

basic typesetting (hyphenation, font handling): takes a bit of time, 
extra features that you use add some too: some timings are reported 
after a run so you get an idea

par building: unles hz is used, quite fast

page building: fast but depending on what features are enables 
finalizing the page can take some time

expansion: pretty fast on the average

summary: try to identify where the bottlenecks are

you can run with

   \enabletrackers[pages.timing]

(put it on cont-loc.mkxl somewhere in in texmf-local) and get timings 
per page (i have that enabled on my machine)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Stephen Gaito]

Hans,

Many thanks for your swift and helpful comments.

After some *very crude* tests using the `luametatex` and `luametafun`
documents, I find that while I *can* stop effective processing at
various points in the LuaMetaTeX pipeline, the time difference overall
is not really significant enough to bother with this approach.

The principle problem is, as you suggested below, "stopping" the
pipeline at the PDF stage (using for example the `pre_output_filter`)
corrupted the `*.tuc` data which is for my purposes, critical.

Your comment was: 

> but keep in mind that multipass data is flushed as part of the
> shipout (because it is often location and order bound)

For the record, using the `append_to_vlist_filter` callback, I did
manage to drastically reduce the "pages" (which were all blank, not
surprisingly).

However, on my elderly desktop from 2008, both callbacks essentially cut
only 6-8 seconds out of 18 seconds, for the `luametatex` document, and
190 seconds, for the `luametafun` document.

In the case of the `luametafun` document, it is the MetaFun/MetaPost
processing which, of course, is taking a long time (as it should, the
graphics computations represent important but complex computations).

My ultimate goal is to parallelize the production of large, heavily
cross-referenced, ConTeXt documents... more on this in a future email...

Again, many thanks for your comments!

Regards,

Stephen Gaito

On Mon, 30 Nov 2020 19:59:07 +0100
Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 11/30/2020 10:51 AM, Stephen Gaito wrote:
> > Hello,
> > 
> > I am slowly working on a Mathematical problem requiring underlying
> > computation.
> > 
> > As Mathematicians (myself included) are rather "conservative", I
> > need to discuss each "chunk" of code with the full set of
> > Mathematical notation.
> > 
> > A couple of years ago I started using ConTeXt-MKIV as a
> > Mathematically-Literate-Programming tool by using its excellent Lua
> > interface to capture the code and dump it to disk for external
> > compilation.
> > 
> > I am now revisiting my original design and want to redo my tools
> > using ConTeXt-LMTX.
> > 
> > I would *like* to be able to "stop" the ConTeXt typesetting at
> > various points for differing purposes:
> > 
> > 1. After all macro expansions (and hence after *my* calls into Lua)
> >     but before line/paragraph/page layout begins.
> 
> maybe something
> 
> \startmystuff
> 
> \stopmystuff
> 
> and then you can hook something into startmystuff and \stopmystuff
> 
> > 2. After line/paragraph/page layout but before PDF generation.
> 
> pdf is generated per page, if needed one can kick in a shipout
> overload
> 
> but keep in mind that multipass data is flushed as part of the
> shipout (because it is often location and order bound)
> 
> > 3. After all PDF generated (ie. a "normal" "full" ConTeXt run).
> > 
> > Stopping after all macro expansions would allow my code generation
> > builds to proceed without the un-needed page setting or PDF
> > generation.
> 
> hm, the problem is always in the 'state' of all kind of variables
> 
> > Stopping after the line/paragraph/page layout would allow multiple
> > "faster(?)" ConTeXt runs while the "*.tuc" file converges to a
> > complete set of page numbers and cross references (etc). Then, once
> > the "*.tuc" file has converged, a full ConTeXt run with PDF output
> > could be done.
> 
> not sure what you mean here ... what is fast? or: how slow is it now? 
> what is the bottleneck? can you cache data that didn't change?
> 
> a large document is normally split up in sections that can be
> processed independent
> 
> \starttext
>      \dorecurse{10000}{\samplefile{ward}\par}
> \stoptext
> 
> runs on my 2013 laptop at over 65 pages per second
> 
> quite often performance is hit by inefficient styling and such ..
> it's no problem to bring a tex system a grinding halt
> 
> > I am very aware that *internally* ConTeXt is probably structured as
> > a tight pipeline with each of the "traditional" TeX stages "Mouth",
> > "Stomach", "page setting", PDF generation.... tightly "chained"...
> > This means that there is no "one" place in the code where all macro
> > expansions have completed but before the page setting "starts", or
> > similarly, after the page setting has finished but before the PDF
> > generation "starts".
> 
> yes and often something is left over for a next page so it's kind of
> fluid
> 
> > ----
> > QUESTION: Is it possible to use the new LuaMetaTeX callbacks (found
> > in chapter 10 of the "LuaMetaTEX Reference Manual") to "suppress"
> > any further computation at various points in the ConTeXt pipeline?
> > ----
> 
> sure, you can kick in handlers at various stages (assuming that you
> keep in mind where you kick them in as there is some order involved)
> 
> > For example, could I use one of the "*_linebreak_filter"s (or the
> > "append_to_vlist_filter") to "return" an empty value and hence
> > reduce further computation downstream in the pipeline?
> 
> you can but linebreak is not the most costly one, you probbaly want
> to intercept the list builder but when you do that you can as well do
> a \stoptext which prevents further reading of content (but i probably 
> misunderstand)
> 
> > Could I use the "pre_output_filter" to "return" an empty value and
> > hence "stop" PDF generation?
> 
> assuming a properky structured document forcing a \stoptext should
> work in most cases
> 
> > (I realize that these callbacks *are* a currently fast moving
> > target. I am happy to follow their changes, equally I would be
> > testing their usefulness and/or impact)
> 
> actually, the callbacks themselves hardly change, but the code
> plugged into them might occasionally (a lot of mkiv code is already
> quite old so i'm now looking at it and see if i can use some recent
> tricks)
> 
> > ALTERNATIVE QUESTION: Would it be possible to provide official
> > ConTeXt-LMTX "modes" which suppress further computation at these
> > points?
> 
> the question is: what do you want to suppress? best first identify
> the bottleneck and then figure out what can be skipped (as mentioned: 
> multipass data can be made more independent I guess but it still
> demands some calculations and analyzing and it's that bit that takes
> the time)
> 
> > This alternative, while some more work for the writing of
> > ConTeXt-LMTX, would ensure less direct external dependence on the
> > LuaMetaTeX callbacks, but would almost certainly be welcomed by the
> > ConTeXt community.
> 
> i need more info (also from others then) about what the reason, goal
> and possible gain is
> 
> - tex: the context code is quite efficient, and tex is quite fast, so 
> there's little to gain there (but as said one can write slow macros
> that spoil that game)
> 
> - lua: on the average lua is fast but garbage collection can be of 
> influence (i need to see code in order to be able to tell if there is
> a gain there); the lua code in context is quite ok but for instance 
> messing with node lists will always come at a cost (crossing the c 
> boundary and such)
> 
> - pdf: the backend code in luametatex is somewhat slower than in
> luatex but we're gaining there (because in related areas we can do
> things different, although there is new functionality that when used
> also comes at a price); but as far as i can tell a luametatex run
> here is on the average some 20% faster than a luatex run so the pdf
> generation slowdown gets kind of obscured by it
> 
> > ----
> > QUESTION: Are the "stages" I have identified major, computationally
> > expensive, "steps" in the overall ConTeXt "computation"?
> > ----
> 
> basic typesetting (hyphenation, font handling): takes a bit of time, 
> extra features that you use add some too: some timings are reported 
> after a run so you get an idea
> 
> par building: unles hz is used, quite fast
> 
> page building: fast but depending on what features are enables 
> finalizing the page can take some time
> 
> expansion: pretty fast on the average
> 
> summary: try to identify where the bottlenecks are
> 
> you can run with
> 
>    \enabletrackers[pages.timing]
> 
> (put it on cont-loc.mkxl somewhere in in texmf-local) and get timings 
> per page (i have that enabled on my machine)
> 
> Hans
> 
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Hans Hagen]

On 12/2/2020 10:40 AM, Stephen Gaito wrote:

> Many thanks for your swift and helpful comments.
> 
> After some *very crude* tests using the `luametatex` and `luametafun`
> documents, I find that while I *can* stop effective processing at
> various points in the LuaMetaTeX pipeline, the time difference overall
> is not really significant enough to bother with this approach.
> 
> The principle problem is, as you suggested below, "stopping" the
> pipeline at the PDF stage (using for example the `pre_output_filter`)
> corrupted the `*.tuc` data which is for my purposes, critical.
> 
> Your comment was:
> 
>> but keep in mind that multipass data is flushed as part of the
>> shipout (because it is often location and order bound)
> 
> For the record, using the `append_to_vlist_filter` callback, I did
> manage to drastically reduce the "pages" (which were all blank, not
> surprisingly).
> 
> However, on my elderly desktop from 2008, both callbacks essentially cut
> only 6-8 seconds out of 18 seconds, for the `luametatex` document, and
> 190 seconds, for the `luametafun` document.

hm, on my 2013 laptop the luametatex manual needs 10 sec (i have all the 
fonts, so that includes a bunch) and a metafun manual should do about 20

a test on am M1 mini needs half those times as reported yesterday

i bet that on a modern desktop the luatex manual will do < 5 sec

> In the case of the `luametafun` document, it is the MetaFun/MetaPost
> processing which, of course, is taking a long time (as it should, the
> graphics computations represent important but complex computations).

One run or many due to xref? Maybe your machine has no significant cpu 
cache? Do you run from disk or ssd? How much memory?

> My ultimate goal is to parallelize the production of large, heavily
> cross-referenced, ConTeXt documents... more on this in a future email...
Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Stephen Gaito]

Hans,

As I said my desktop is elderly... it has a 2.8GHz processor, 16Gb of
DDR3 memory, and a couple of old SATA1 hard disks, and only 3Mb of CPU
cache...

... all well past its use by date for single threaded ConTeXt. ;-(

So one way to get better performance for ConTeXt is to invest in a new
ultra fast processor. Which will cost a lot, and use a lot of power
which has to be cooled, which uses even more power....

Alternatively, for the same costs (or less), I can buy cheaper slower
processors but have lots of threads (a cluster of Raspberry Pi 4 8Gb
cards)...

Alas this requires finding some way to parallelize ConTeXt....

(Fools rush in where Angels fear to tread ;-(

Regards,

Stephen Gaito 

On Wed, 2 Dec 2020 14:04:18 +0100
Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 12/2/2020 10:40 AM, Stephen Gaito wrote:
> 
> > Many thanks for your swift and helpful comments.
> > 
> > After some *very crude* tests using the `luametatex` and
> > `luametafun` documents, I find that while I *can* stop effective
> > processing at various points in the LuaMetaTeX pipeline, the time
> > difference overall is not really significant enough to bother with
> > this approach.
> > 
> > The principle problem is, as you suggested below, "stopping" the
> > pipeline at the PDF stage (using for example the
> > `pre_output_filter`) corrupted the `*.tuc` data which is for my
> > purposes, critical.
> > 
> > Your comment was:
> >   
> >> but keep in mind that multipass data is flushed as part of the
> >> shipout (because it is often location and order bound)  
> > 
> > For the record, using the `append_to_vlist_filter` callback, I did
> > manage to drastically reduce the "pages" (which were all blank, not
> > surprisingly).
> > 
> > However, on my elderly desktop from 2008, both callbacks
> > essentially cut only 6-8 seconds out of 18 seconds, for the
> > `luametatex` document, and 190 seconds, for the `luametafun`
> > document.  
> 
> hm, on my 2013 laptop the luametatex manual needs 10 sec (i have all
> the fonts, so that includes a bunch) and a metafun manual should do
> about 20
> 
> a test on am M1 mini needs half those times as reported yesterday
> 
> i bet that on a modern desktop the luatex manual will do < 5 sec
> 
> > In the case of the `luametafun` document, it is the MetaFun/MetaPost
> > processing which, of course, is taking a long time (as it should,
> > the graphics computations represent important but complex
> > computations).  
> 
> One run or many due to xref? Maybe your machine has no significant
> cpu cache? Do you run from disk or ssd? How much memory?
> 
> > My ultimate goal is to parallelize the production of large, heavily
> > cross-referenced, ConTeXt documents... more on this in a future
> > email...  
> Hans
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Taco Hoekwater]

> On 3 Dec 2020, at 11:35, Stephen Gaito <stephen@perceptisys.co.uk> wrote:
> 
> Hans,
> 
> As I said my desktop is elderly... it has a 2.8GHz processor, 16Gb of
> DDR3 memory, and a couple of old SATA1 hard disks, and only 3Mb of CPU
> cache...
> 
> ... all well past its use by date for single threaded ConTeXt. ;-(
> 
> So one way to get better performance for ConTeXt is to invest in a new
> ultra fast processor. Which will cost a lot, and use a lot of power
> which has to be cooled, which uses even more power....

Startup time can be improved quite a bit with an SSD. Even a cheap SATA
SSD is already much faster than a traditional harddisk. Doesn’t help
with longer documents, but it could be a fairly cheap upgrade.

I can’t comment on how to speed up the rest of what you are doing,
but generally multi-threading TeX typesetting jobs is so hard as to
be impossible in practise. About the only step that can be split off
is the generation of the PDF, and even there the possible gain is 
quite small (as you noticed already).

Typesetting is a compilation job, so the two main ways to speed things
along are

1) split the source into independent tasks, like in a code compiler
   that splits code over separate .c / .cpp / .m / .p etc. files,
   and then combine the results (using e.g. mutool)

2) precompile recurring stuff (in TeX, that would mean embedding
   separately generated pdfs or images)

Best wishes,
Taco





___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Hans Hagen]

On 12/3/2020 12:15 PM, Taco Hoekwater wrote:
> 
> 
>> On 3 Dec 2020, at 11:35, Stephen Gaito <stephen@perceptisys.co.uk> wrote:
>>
>> Hans,
>>
>> As I said my desktop is elderly... it has a 2.8GHz processor, 16Gb of
>> DDR3 memory, and a couple of old SATA1 hard disks, and only 3Mb of CPU
>> cache...
>>
>> ... all well past its use by date for single threaded ConTeXt. ;-(
>>
>> So one way to get better performance for ConTeXt is to invest in a new
>> ultra fast processor. Which will cost a lot, and use a lot of power
>> which has to be cooled, which uses even more power....
> 
> Startup time can be improved quite a bit with an SSD. Even a cheap SATA
> SSD is already much faster than a traditional harddisk. Doesn’t help
> with longer documents, but it could be a fairly cheap upgrade.

also, an empty context run

\starttext
\stoptext

only takes 0.490 seconds on my machine, which means:

- starting mtxrun, which includes quite a bit of lua plus loading the
file database etc
- loading mtx-context that itself does some checking
- and then launches the engine (it could be intgerated but then we run 
into issues when we have fatal errors as well as initializations so in 
the end it doesn't pay off at all)
- the tex runs means: loading the format and initializing hundreds of 
lua scripts including all kind of unicode related stuff

so, the .5 sec is quite acceptable to me and i knwo that when i would 
have amore recent machine it would go down to half of that

now, making a tex run persistent is not really a solution: one has to 
reset all kinds of counters, dimensions etc wipe node and token space, 
etc an done would also have to reset the pdf output which includes all 
kind of housekeeping states ... adding all kind of resetters and hooks 
for that (plus all the garbage collection needed) will never pay back 
and a 'wipe all and reload' is way more efficient then

of course, when i ever run into a secenario where I have to creeate tens 
of thousands of one/few page docs very fast i might add some 'reset the 
pdf state' because that is kind of doable with some extra code but to be 
honest, no one ever came up with a project that had any real demands on 
the engine that could not be met (the fact that tex is a good solution 
for rendering doesn't mean that there is demand for it ... it is seldom 
on the radar of those who deal with that, who then often prefer some pdf 
library, also because quality doesn't really matter)

these kind of performance things are demand driven (read: i need a 
pretty good reason to spend time on it)

> I can’t comment on how to speed up the rest of what you are doing,
> but generally multi-threading TeX typesetting jobs is so hard as to
> be impossible in practise. About the only step that can be split off
> is the generation of the PDF, and even there the possible gain is
> quite small (as you noticed already).

indeed, see above

> Typesetting is a compilation job, so the two main ways to speed things
> along are
> 
> 1) split the source into independent tasks, like in a code compiler
>     that splits code over separate .c / .cpp / .m / .p etc. files,
>     and then combine the results (using e.g. mutool)
> 
> 2) precompile recurring stuff (in TeX, that would mean embedding
>     separately generated pdfs or images)
right

(and we are old enough and have been around long enough to have some gut 
feeling about that)

Hans

ps. When it comes to performance of tex, lua, context etc it is no 
problem, when googling a bit, to run into 'nonsense' arguments of why 
something is slow ... so don't take it for granted, just ask here on 
this list

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Stephen Gaito]

Taco,

Thanks for your comments... see below...

On Thu, 3 Dec 2020 12:15:46 +0100
Taco Hoekwater <taco@elvenkind.com> wrote:

> > On 3 Dec 2020, at 11:35, Stephen Gaito <stephen@perceptisys.co.uk>
> > wrote:
> > 
> > Hans,
> > 
> > As I said my desktop is elderly... it has a 2.8GHz processor, 16Gb
> > of DDR3 memory, and a couple of old SATA1 hard disks, and only 3Mb
> > of CPU cache...
> > 
> > ... all well past its use by date for single threaded ConTeXt. ;-(
> > 
> > So one way to get better performance for ConTeXt is to invest in a
> > new ultra fast processor. Which will cost a lot, and use a lot of
> > power which has to be cooled, which uses even more power....  
> 
> Startup time can be improved quite a bit with an SSD. Even a cheap
> SATA SSD is already much faster than a traditional harddisk. Doesn’t
> help with longer documents, but it could be a fairly cheap upgrade.
> 
> I can’t comment on how to speed up the rest of what you are doing,
> but generally multi-threading TeX typesetting jobs is so hard as to
> be impossible in practise. About the only step that can be split off
> is the generation of the PDF, and even there the possible gain is 
> quite small (as you noticed already).
> 
> Typesetting is a compilation job, so the two main ways to speed things
> along are
> 
> 1) split the source into independent tasks, like in a code compiler
>    that splits code over separate .c / .cpp / .m / .p etc. files,
>    and then combine the results (using e.g. mutool)
> 

This is basically my approach... *However*, while the dependency graph
for a standard compilation has been engineered to be an acyclic tree,
for a ConTeXt "compilation", the "*.tex" file has a cyclic dependency
on the (generated) "*.tuc" file.

Basically my parallelization "build manager" has to unroll or other
wise reimplement the mtx-context.lua 

```
  for currentrun=1,maxnofruns do
    ...
  end
```

loop until the `multipass_changed(oldhash,newhash)` returns `false`.

Followed by a "pdf-linking" stage (possibly implemented, as you suggest,
by `mutool`) ....

Not great.... but it might work... (and I am willing to give it a
try)...

> 2) precompile recurring stuff (in TeX, that would mean embedding
>    separately generated pdfs or images)
> 
> Best wishes,
> Taco
> 
> 
> 
> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an
> entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context webpage  :
> http://www.pragma-ade.nl / http://context.aanhet.net archive  :
> https://bitbucket.org/phg/context-mirror/commits/ wiki     :
> http://contextgarden.net
> ___________________________________________________________________________________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Using ConTeXt-LMTX for modern Mathematically-Literate-Programming 1/2

[Stephen Gaito]

Hans,

Again many thanks for your thoughts! (See below....)

On Thu, 3 Dec 2020 13:15:28 +0100
Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 12/3/2020 12:15 PM, Taco Hoekwater wrote:
> > 
> >   
> >> On 3 Dec 2020, at 11:35, Stephen Gaito <stephen@perceptisys.co.uk>
> >> wrote:
> >>
> >> Hans,
> >>
> >> As I said my desktop is elderly... it has a 2.8GHz processor, 16Gb
> >> of DDR3 memory, and a couple of old SATA1 hard disks, and only 3Mb
> >> of CPU cache...
> >>
> >> ... all well past its use by date for single threaded ConTeXt. ;-(
> >>
> >> So one way to get better performance for ConTeXt is to invest in a
> >> new ultra fast processor. Which will cost a lot, and use a lot of
> >> power which has to be cooled, which uses even more power....  
> > 
> > Startup time can be improved quite a bit with an SSD. Even a cheap
> > SATA SSD is already much faster than a traditional harddisk.
> > Doesn’t help with longer documents, but it could be a fairly cheap
> > upgrade.  
> 
> also, an empty context run
> 
> \starttext
> \stoptext
> 
> only takes 0.490 seconds on my machine, which means:
> 
> - starting mtxrun, which includes quite a bit of lua plus loading the
> file database etc
> - loading mtx-context that itself does some checking
> - and then launches the engine (it could be intgerated but then we
> run into issues when we have fatal errors as well as initializations
> so in the end it doesn't pay off at all)
> - the tex runs means: loading the format and initializing hundreds of 
> lua scripts including all kind of unicode related stuff
> 
> so, the .5 sec is quite acceptable to me and i knwo that when i would 
> have amore recent machine it would go down to half of that
> 

I will agree that this is acceptable for the complexity ConTeXt
represents... ConTeXt has a complex task... it *will* have to take some
time... that is OK.

> now, making a tex run persistent is not really a solution: one has to 
> reset all kinds of counters, dimensions etc wipe node and token
> space, etc an done would also have to reset the pdf output which
> includes all kind of housekeeping states ... adding all kind of
> resetters and hooks for that (plus all the garbage collection needed)
> will never pay back and a 'wipe all and reload' is way more efficient
> then

I also agree, keeping a pool of "warm" running ConTeXts, as you are
essentially describing, would be nice... but I suspect the complexity
does preclude this approach. Keep it Simple... Simply killing and
restarting ConTeXt as a new process is OK.

> 
> of course, when i ever run into a secenario where I have to creeate
> tens of thousands of one/few page docs very fast i might add some
> 'reset the pdf state' because that is kind of doable with some extra
> code but to be honest, no one ever came up with a project that had
> any real demands on the engine that could not be met (the fact that
> tex is a good solution for rendering doesn't mean that there is
> demand for it ... it is seldom on the radar of those who deal with
> that, who then often prefer some pdf library, also because quality
> doesn't really matter)
> 
> these kind of performance things are demand driven (read: i need a 
> pretty good reason to spend time on it)

Understood.

> 
> > I can’t comment on how to speed up the rest of what you are doing,
> > but generally multi-threading TeX typesetting jobs is so hard as to
> > be impossible in practise. About the only step that can be split off
> > is the generation of the PDF, and even there the possible gain is
> > quite small (as you noticed already).  
> 
> indeed, see above
> 
> > Typesetting is a compilation job, so the two main ways to speed
> > things along are
> > 
> > 1) split the source into independent tasks, like in a code compiler
> >     that splits code over separate .c / .cpp / .m / .p etc. files,
> >     and then combine the results (using e.g. mutool)
> > 
> > 2) precompile recurring stuff (in TeX, that would mean embedding
> >     separately generated pdfs or images)  
> right
> 
> (and we are old enough and have been around long enough to have some
> gut feeling about that)
> 

I have a deep respect for both your vision, and experience in this
matter.

However, way back when you started ConTeXt, very few people would
have said it was possible/worth embedding Lua in TeX....

Given that Lua *is* now embedded inside ConTeXt, I am simply making a
*crude* attempt to see if I can parallelize the overall ConTeXt
production cycle (with out changing ConTeXt-LMTX itself).

"Fools rush in..."

> Hans
> 
> ps. When it comes to performance of tex, lua, context etc it is no 
> problem, when googling a bit, to run into 'nonsense' arguments of why 
> something is slow ... so don't take it for granted, just ask here on 
> this list
> 

I am not distracted by this noise... Complex things take time... what I
am attempting is complex... and requires the best tool available... and
ConTeXt is the *best* tool available!  Many many thanks for your vision
and work!

> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an
> entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context webpage  :
> http://www.pragma-ade.nl / http://context.aanhet.net archive  :
> https://bitbucket.org/phg/context-mirror/commits/ wiki     :
> http://contextgarden.net
> ___________________________________________________________________________________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________