protected macro vs protected luacall

protected macro vs protected luacall

[Henri Menke via ntg-context]

Dear list,

I have noticed a somewhat strange difference between protected macros
and protected luacalls, namely that the latter cannot be expanded by
\the.  In the example below I define a Lua function that prints
\numexpr 17\relax to the token stream and I call it once from a
protected TeX macro wrapping \directlua and once from a protected
luacall registered with token.set_lua.


\directlua{
  userdata = userdata or {}
  function userdata.test()
    tex.print("\string\\numexpr 17\string\\relax")
  end
}

\tt
\protected\def\test{\directlua{userdata.test()}}
\meaning\test\par
\edef\x{\test}\meaning\x\par
\the\test

\directlua{
  local t = lua.get_functions_table()
  t[\string#t + 1] = userdata.test
  token.set_lua("test", userdata.test, \string#t, "protected")
}
\meaning\test\par
\edef\x{\test}\meaning\x\par
\the\test

\bye


However, the second \the\test fails with the following error:


! You can't use `luacall 0' after \the.
l.21 \the\test
            

Is this asymmetry between luacalls and regular macros intended? Is
there a workaround where I can have a luacall that doesn't expand
inside of \edef but still expands after \the?

Cheers, Henri
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Hans Hagen via ntg-context]

On 1/26/2023 9:33 PM, Henri Menke via ntg-context wrote:
> \directlua{
>    userdata = userdata or {}
>    function userdata.test()
>      tex.print("\string\\numexpr 17\string\\relax")
>    end
> }
> 
> \tt
> \protected\def\test{\directlua{userdata.test()}}
> \meaning\test\par
> \edef\x{\test}\meaning\x\par
> \the\test
> 
> \directlua{
>    local t = lua.get_functions_table()
>    t[\string#t + 1] = userdata.test
>    token.set_lua("test", userdata.test, \string#t, "protected")
> }
> \meaning\test\par
> \edef\x{\test}\meaning\x\par
> \the\test
from the code it looks like we're talking plain luatex ...

\the is very selective and doesn't work for all primitives or macros

compare it to \number which accepts any number representation following 
it while \the doesn't work ok 123

that said, your set_lua is wrong and passes a function as well as an index

\directlua{
    lua.get_functions_table()[999] = function()
      tex.print("\string\\numexpr 17\string\\relax")
    end
    token.set_lua("test", 999)
}

\the    \test % needs the \numexpr

\number \test % doesn't

\count0 \test % needs the \numexpr

\count0=\test % doesn't

etc

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Henri Menke via ntg-context]

On Thu, 2023-01-26 at 23:07 +0100, Hans Hagen via ntg-context wrote:
> On 1/26/2023 9:33 PM, Henri Menke via ntg-context wrote:
> > \directlua{
> >    userdata = userdata or {}
> >    function userdata.test()
> >      tex.print("\string\\numexpr 17\string\\relax")
> >    end
> > }
> > 
> > \tt
> > \protected\def\test{\directlua{userdata.test()}}
> > \meaning\test\par
> > \edef\x{\test}\meaning\x\par
> > \the\test
> > 
> > \directlua{
> >    local t = lua.get_functions_table()
> >    t[\string#t + 1] = userdata.test
> >    token.set_lua("test", userdata.test, \string#t, "protected")
> > }
> > \meaning\test\par
> > \edef\x{\test}\meaning\x\par
> > \the\test
> from the code it looks like we're talking plain luatex ...
> 
> \the is very selective and doesn't work for all primitives or macros
> 
> compare it to \number which accepts any number representation
> following 
> it while \the doesn't work ok 123
> 
> that said, your set_lua is wrong and passes a function as well as an
> index
> 
> \directlua{
>     lua.get_functions_table()[999] = function()
>       tex.print("\string\\numexpr 17\string\\relax")
>     end
>     token.set_lua("test", 999)
> }
> 
> \the    \test % needs the \numexpr
> 
> \number \test % doesn't
> 
> \count0 \test % needs the \numexpr
> 
> \count0=\test % doesn't
> 
> etc

Thanks for pointing out my mistake. Your example works fine but now the
luacall is no longer protected and will therefore fully expand in
\edef. So do I understand this correctly that there is no way to make
the second \the\test work?


\directlua{
    lua.get_functions_table()[999] = function()
      tex.pri
nt("\string\\numexpr 17\string\\relax")
    end
}

\tt
\directlua{token.set_lua("test", 999)}
\meaning\test\par % expandable luacall 999
\edef\x{\test}\meaning\x\par % macro:->\numexpr 17\relax
\the\test % 17

\directlua{token.set_lua("test", 999, "protected")}
\meaning\test\par % luacall 999
\edef\x{\test}\meaning\x\par % macro:->\test
\the\test % ! You can't use `luacall 999' after \the.

\bye


Cheers, Henri

> 
> Hans
> 
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
> 
> _____________________________________________________________________
> ______________
> If your question is of interest to others as well, please add an
> entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / 
> https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : https://contextgarden.net
> _____________________________________________________________________
> ______________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Hans Hagen via ntg-context]

On 1/27/2023 10:13 AM, Henri Menke via ntg-context wrote:
> On Thu, 2023-01-26 at 23:07 +0100, Hans Hagen via ntg-context wrote:
>> On 1/26/2023 9:33 PM, Henri Menke via ntg-context wrote:
>>> \directlua{
>>>     userdata = userdata or {}
>>>     function userdata.test()
>>>       tex.print("\string\\numexpr 17\string\\relax")
>>>     end
>>> }
>>>
>>> \tt
>>> \protected\def\test{\directlua{userdata.test()}}
>>> \meaning\test\par
>>> \edef\x{\test}\meaning\x\par
>>> \the\test
>>>
>>> \directlua{
>>>     local t = lua.get_functions_table()
>>>     t[\string#t + 1] = userdata.test
>>>     token.set_lua("test", userdata.test, \string#t, "protected")
>>> }
>>> \meaning\test\par
>>> \edef\x{\test}\meaning\x\par
>>> \the\test
>> from the code it looks like we're talking plain luatex ...
>>
>> \the is very selective and doesn't work for all primitives or macros
>>
>> compare it to \number which accepts any number representation
>> following
>> it while \the doesn't work ok 123
>>
>> that said, your set_lua is wrong and passes a function as well as an
>> index
>>
>> \directlua{
>>      lua.get_functions_table()[999] = function()
>>        tex.print("\string\\numexpr 17\string\\relax")
>>      end
>>      token.set_lua("test", 999)
>> }
>>
>> \the    \test % needs the \numexpr
>>
>> \number \test % doesn't
>>
>> \count0 \test % needs the \numexpr
>>
>> \count0=\test % doesn't
>>
>> etc
> 
> Thanks for pointing out my mistake. Your example works fine but now the
> luacall is no longer protected and will therefore fully expand in
> \edef. So do I understand this correctly that there is no way to make
> the second \the\test work?
>
> \directlua{
>      lua.get_functions_table()[999] = function()
>        tex.pri
> nt("\string\\numexpr 17\string\\relax")
>      end
> }
> 
> \tt
> \directlua{token.set_lua("test", 999)}
> \meaning\test\par % expandable luacall 999
> \edef\x{\test}\meaning\x\par % macro:->\numexpr 17\relax
> \the\test % 17
> 
> \directlua{token.set_lua("test", 999, "protected")}
> \meaning\test\par % luacall 999
> \edef\x{\test}\meaning\x\par % macro:->\test
> \the\test % ! You can't use `luacall 999' after \the.
> 
> \bye
you can do

\directlua{
    function TestCmd()
      tex.print("\string\\numexpr 17\string\\relax")
    end
}

\protected\def\test{\directlua{TestCmd()}}

or, more efficient:

\directlua{
    function TestCmd()
      tex.print("17")
    end
}

\protected\def\test{\numexpr\directlua{TestCmd()}\relax}

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Henri Menke via ntg-context]

On Fri, 2023-01-27 at 10:30 +0100, Hans Hagen via ntg-context wrote:
> On 1/27/2023 10:13 AM, Henri Menke via ntg-context wrote:
> > On Thu, 2023-01-26 at 23:07 +0100, Hans Hagen via ntg-context
> > wrote:
> > > On 1/26/2023 9:33 PM, Henri Menke via ntg-context wrote:
> > > > \directlua{
> > > >     userdata = userdata or {}
> > > >     function userdata.test()
> > > >       tex.print("\string\\numexpr 17\string\\relax")
> > > >     end
> > > > }
> > > > 
> > > > \tt
> > > > \protected\def\test{\directlua{userdata.test()}}
> > > > \meaning\test\par
> > > > \edef\x{\test}\meaning\x\par
> > > > \the\test
> > > > 
> > > > \directlua{
> > > >     local t = lua.get_functions_table()
> > > >     t[\string#t + 1] = userdata.test
> > > >     token.set_lua("test", userdata.test, \string#t,
> > > > "protected")
> > > > }
> > > > \meaning\test\par
> > > > \edef\x{\test}\meaning\x\par
> > > > \the\test
> > > from the code it looks like we're talking plain luatex ...
> > > 
> > > \the is very selective and doesn't work for all primitives or
> > > macros
> > > 
> > > compare it to \number which accepts any number representation
> > > following
> > > it while \the doesn't work ok 123
> > > 
> > > that said, your set_lua is wrong and passes a function as well as
> > > an
> > > index
> > > 
> > > \directlua{
> > >      lua.get_functions_table()[999] = function()
> > >        tex.print("\string\\numexpr 17\string\\relax")
> > >      end
> > >      token.set_lua("test", 999)
> > > }
> > > 
> > > \the    \test % needs the \numexpr
> > > 
> > > \number \test % doesn't
> > > 
> > > \count0 \test % needs the \numexpr
> > > 
> > > \count0=\test % doesn't
> > > 
> > > etc
> > 
> > Thanks for pointing out my mistake. Your example works fine but now
> > the
> > luacall is no longer protected and will therefore fully expand in
> > \edef. So do I understand this correctly that there is no way to
> > make
> > the second \the\test work?
> > 
> > \directlua{
> >      lua.get_functions_table()[999] = function()
> >        tex.pri
> > nt("\string\\numexpr 17\string\\relax")
> >      end
> > }
> > 
> > \tt
> > \directlua{token.set_lua("test", 999)}
> > \meaning\test\par % expandable luacall 999
> > \edef\x{\test}\meaning\x\par % macro:->\numexpr 17\relax
> > \the\test % 17
> > 
> > \directlua{token.set_lua("test", 999, "protected")}
> > \meaning\test\par % luacall 999
> > \edef\x{\test}\meaning\x\par % macro:->\test
> > \the\test % ! You can't use `luacall 999' after \the.
> > 
> > \bye
> you can do
> 
> \directlua{
>     function TestCmd()
>       tex.print("\string\\numexpr 17\string\\relax")
>     end
> }
> 
> \protected\def\test{\directlua{TestCmd()}}
> 
> or, more efficient:
> 
> \directlua{
>     function TestCmd()
>       tex.print("17")
>     end
> }
> 
> \protected\def\test{\numexpr\directlua{TestCmd()}\relax}

Thanks, this is indeed the workaround that I currently use. However,
the downside is that this has to tokenize the contents of \directlua
every time (and is therefore susceptible to surrounding \catcode
shenanigans) and it expands in at minimum two steps, whereas a luacall
always expands in a single step. But I guess that's what I will have to
live with.

Cheers, Henri

> 
> Hans
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
> 
> _____________________________________________________________________
> ______________
> If your question is of interest to others as well, please add an
> entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / 
> https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : https://contextgarden.net
> _____________________________________________________________________
> ______________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Hans Hagen via ntg-context]

On 1/27/2023 10:35 AM, Henri Menke via ntg-context wrote:

> Thanks, this is indeed the workaround that I currently use. However,
> the downside is that this has to tokenize the contents of \directlua
> every time (and is therefore susceptible to surrounding \catcode
> shenanigans) and it expands in at minimum two steps, whereas a luacall
> always expands in a single step. But I guess that's what I will have to
> live with.
on my 2018 laptop, with

   function FoO() end

this

   \directlua{FoO()}}

takes 0.0000014 seconds so probably the least of your worries compared 
to other macro and lua magic

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Henri Menke via ntg-context]

On Fri, 2023-01-27 at 11:01 +0100, Hans Hagen via ntg-context wrote:
> On 1/27/2023 10:35 AM, Henri Menke via ntg-context wrote:
> 
> > Thanks, this is indeed the workaround that I currently use.
> > However,
> > the downside is that this has to tokenize the contents of
> > \directlua
> > every time (and is therefore susceptible to surrounding \catcode
> > shenanigans) and it expands in at minimum two steps, whereas a
> > luacall
> > always expands in a single step. But I guess that's what I will
> > have to
> > live with.
> on my 2018 laptop, with
> 
>    function FoO() end
> 
> this
> 
>    \directlua{FoO()}}
> 
> takes 0.0000014 seconds so probably the least of your worries
> compared 
> to other macro and lua magic

True, but then there is still the problem with the number of
expansions. With a nested \directlua I always need at least two steps
to get the result from Lua.


\directlua{
    function TestCmd()
      tex.print("\string\\numexpr 17\string\\relax")
    end
    lua.get_functions_table()[999] = TestCmd
}
\let\:\expandafter % I'm lazy
\tt
\directlua{token.set_lua("test", 999)}
\:\def\:\x\:{\test}\meaning\x % macro:->\numexpr 17\relax

\protected\def\test{\directlua{TestCmd()}}
\:\def\:\x\:{\test}\meaning\x\par % macro:->\directlua {TestCmd()}
\:\:\:\def\:\:\:\x\:\:\:{\test}\meaning\x % macro:->\numexpr 17\relax

\bye


How much do you think would break if protected luacall were treated
like protected macros after \the in the engine?

That would probably need an extra branch in the switch(cur_cmd) in
scan_something_internal. Currently protected luacall just goes to the
default branch (You can't use ...).

Cheers, Henri

> 
> Hans
> 
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
> 
> _____________________________________________________________________
> ______________
> If your question is of interest to others as well, please add an
> entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / 
> https://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : https://contextgarden.net
> _____________________________________________________________________
> ______________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: protected macro vs protected luacall

[Hans Hagen via ntg-context]

On 1/27/2023 12:03 PM, Henri Menke via ntg-context wrote:

> How much do you think would break if protected luacall were treated
> like protected macros after \the in the engine?

That kind of incompatibility is no option (I'm not going to analyze it 
for mkiv let along other macro package usage which is out of my scope).

> That would probably need an extra branch in the switch(cur_cmd) in
> scan_something_internal. Currently protected luacall just goes to the
> default branch (You can't use ...).
For various reasons luatex got frozen around version 1 and demand for 
stability (which also excluded experiments) is one of them. So, one has 
to work with what one gets (not much different from the rest of tex 
functionality). With careful coding the performance hit is not that 
large and machines still get a bit get faster so that compensates it 
anyway. Although some low level approaches can bring benefits it's never 
homogenious and largely depends on use cases.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________