Snooping around in LMTX: Questions about Lua global variables

Snooping around in LMTX: Questions about Lua global variables

[Neven Sajko]

Hello again,

While learning about how to drive TeX through Lua, I decided to
recursively list all Lua global variables (actually this is traversing
the _G table) in the LMTX environment, half to learn more Lua, half
for getting to know ConTeXt better.

I was quite surprised by the huge size of the environment, a file that
contains the listing of all the globals is 42 MB long! I wonder if it
would be possible to reduce the exposed globals by replacing some of
them by getter and setter-like functions? That seems like it would be
much nicer and less error prone - because Lua is so dynamic, polluting
the global namespace seems even more dangerous than in C-like
languages. I guess it could also be more performant, because Lua would
conceivably spend less time managing huge tables.

There were also some global variables with suspicious random variation
in values between runs of ConTeXt: I ran my Lua script like so
multiple times (attached, in case someone is interested in it):

    context s.lua
    rm s.tuc s.pdf s.log

And I found that the values of some variables unpredictably and randomly vary:

The variable resolvers.suffixmap.lua sometimes has the value
"scripts", and sometimes "lua". I think this means that files with the
file name suffix ".lua" are sometimes classified as general scripts
and sometimes as Lua scripts. This seems like it could even be a bug?

The variables storage.tofmodules and storage.toftables are also
interesting: they vary from run to run like this:

tofmodules: 0.175483 0.149536 0.150493 0.150005

toftables: 0.008407 0.008118 0.008395 0.008116

I'd like to know what is their purpose, if it's not to involved to explain?

Thanks,
Neven
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Neven Sajko]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

Oops, I forgot to attach the scriptlet before.


> [...] I guess it could also be more performant, because Lua would
> conceivably spend less time managing huge tables.

Now that I think about this some more, it doesn't actually make sense.
However I'm still interested in whether it is really necessary to have
that many globals exposed.

Thanks,
Neven

[-- Attachment #2: s.lua.gz --]
[-- Type: application/gzip, Size: 622 bytes --]

[-- Attachment #3: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Hans Hagen]

On 12/22/2020 6:36 PM, Neven Sajko wrote:
> Hello again,
> 
> While learning about how to drive TeX through Lua, I decided to
> recursively list all Lua global variables (actually this is traversing
> the _G table) in the LMTX environment, half to learn more Lua, half
> for getting to know ConTeXt better.
> 
> I was quite surprised by the huge size of the environment, a file that
> contains the listing of all the globals is 42 MB long! I wonder if it
> would be possible to reduce the exposed globals by replacing some of
> them by getter and setter-like functions? That seems like it would be

not sure what you refer to but even then you need to 'get' and 'set them 
someplace which then involves tables ... it's just a large system and 
that won't change

also, when you run such tests, don't include the characters.* tables as 
most is data, and of you do it from a tex run you also see fonts and 
their data (which then means shared tables too)

> much nicer and less error prone - because Lua is so dynamic, polluting
> the global namespace seems even more dangerous than in C-like
> languages. I guess it could also be more performant, because Lua would
> conceivably spend less time managing huge tables.

Wait, you traverse global tablers so their entries are *not* global.

\starttext

\startluacode
     context.starttabulate { "|T|T|" }
     for k, v in table.sortedhash(_G) do
         context.NC() context(type(v)) context.NC() context(k) 
context.NC() context.NR()
     end
     context.stoptabulate()
\stopluacode

\stoptext

These are global. Many come from lua itself, then there are libraries 
than come with luametatex. The rest is context specific and again some 
are just helper modules. I notices some 6 stray locals that I fixed.

> There were also some global variables with suspicious random variation
> in values between runs of ConTeXt: I ran my Lua script like so
> multiple times (attached, in case someone is interested in it):
> 
>      context s.lua
>      rm s.tuc s.pdf s.log
> 
> And I found that the values of some variables unpredictably and randomly vary:

Maybe weak tables?

> The variable resolvers.suffixmap.lua sometimes has the value
> "scripts", and sometimes "lua". I think this means that files with the
> file name suffix ".lua" are sometimes classified as general scripts
> and sometimes as Lua scripts. This seems like it could even be a bug?

could be but probably more a side effect ... part of that resolver stuff 
is there for usage in tds and could be simplified in the meantime .. if 
there are hashes they can differ per document

> The variables storage.tofmodules and storage.toftables are also
> interesting: they vary from run to run like this:
> 
> tofmodules: 0.175483 0.149536 0.150493 0.150005
> 
> toftables: 0.008407 0.008118 0.008395 0.008116
> 
> I'd like to know what is their purpose, if it's not to involved to explain?
timers, so indeed they can differ per run

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Neven Sajko]

Thank you very much for your answers!

Neven
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Hans Hagen]

On 12/22/2020 6:57 PM, Neven Sajko wrote:
> Oops, I forgot to attach the scriptlet before.
> 
> 
>> [...] I guess it could also be more performant, because Lua would
>> conceivably spend less time managing huge tables.
> 
> Now that I think about this some more, it doesn't actually make sense.
> However I'm still interested in whether it is really necessary to have
> that many globals exposed.
most of what you see in that generated file is either unicode data or 
font resources ... all needed (and geared for performance)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Hans Hagen]

On 12/22/2020 7:13 PM, Neven Sajko wrote:
> Thank you very much for your answers!

you can run:

s-inf-01.mkiv
s-inf-03.mkiv
s-inf-05.mkiv

i need to update them but they show the picture

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Neven Sajko]

On Tue, 22 Dec 2020 at 18:15, Hans Hagen <j.hagen@xs4all.nl> wrote:
>
> On 12/22/2020 6:57 PM, Neven Sajko wrote:
> > Oops, I forgot to attach the scriptlet before.
> >
> >
> >> [...] I guess it could also be more performant, because Lua would
> >> conceivably spend less time managing huge tables.
> >
> > Now that I think about this some more, it doesn't actually make sense.
> > However I'm still interested in whether it is really necessary to have
> > that many globals exposed.
> most of what you see in that generated file is either unicode data or
> font resources ... all needed (and geared for performance)

OK, now that I think just about the real global variables (instead of
the "recursive" globals): would it make sense to transfer all the
non-Lua-default globals into two tables, one for Lua(Meta)Tex, and
another table for ConTeXt, so those would be the only two additional
global variables?

I'm not proposing you do it, since it seems like it could be a lot of
work, I'm just wondering what you think about that, because it seems
like things would be much tidier like that (less chance of
accidentally accessing a global in Lua code, etc.).

Thanks,
Neven
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Snooping around in LMTX: Questions about Lua global variables

[Hans Hagen]

On 12/22/2020 7:32 PM, Neven Sajko wrote:
> On Tue, 22 Dec 2020 at 18:15, Hans Hagen <j.hagen@xs4all.nl> wrote:
>>
>> On 12/22/2020 6:57 PM, Neven Sajko wrote:
>>> Oops, I forgot to attach the scriptlet before.
>>>
>>>
>>>> [...] I guess it could also be more performant, because Lua would
>>>> conceivably spend less time managing huge tables.
>>>
>>> Now that I think about this some more, it doesn't actually make sense.
>>> However I'm still interested in whether it is really necessary to have
>>> that many globals exposed.
>> most of what you see in that generated file is either unicode data or
>> font resources ... all needed (and geared for performance)
> 
> OK, now that I think just about the real global variables (instead of
> the "recursive" globals): would it make sense to transfer all the
> non-Lua-default globals into two tables, one for Lua(Meta)Tex, and
> another table for ConTeXt, so those would be the only two additional
> global variables?

well, and then if you load some library that would add some global again 
... so it's a chicken egg problem.

> I'm not proposing you do it, since it seems like it could be a lot of
> work, I'm just wondering what you think about that, because it seems
> like things would be much tidier like that (less chance of
> accidentally accessing a global in Lua code, etc.).
You can create your own instance:

\definenamedlua[mylua]

\startmyluacode
global.context("USER 1")
context.par()
context("USER 2")
context.par()
if characters then
     context("ACCESS directly")
elseif global.characters then
     context("ACCESS via global")
else
     context("NO ACCESS at all")
end
context.par()
if bogus then
     context("ACCESS directly")
elseif global.bogus then
     context("ACCESS via global")
else
     context("NO ACCESS at all")
end
context.par()
\stopmyluacode


I admit that I never run into conflicts. Ok, I never load libraries, 
mostly because we havne plenty of helpers on board. Changing the 
approach now would invalidate a lot of user code (and also mean a lot of 
work plus probably make us run into the > 200 locals issue due to 
aliasing).

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________