Lua related change in LuaMetaTeX?

Lua related change in LuaMetaTeX?

[Marcel Fabian Krüger]

Hi,

starting with LuaMetaTeX 2.07.01, a C library I load from Lua started to
segfault and valgrind indicates that it accesses invalid memory in
multiple places in the middle of Lua internal functions?!

The whole thing looks almost like the Lua versions in LuaMetaTeX and in
the library are incompatible, but both seem to use 5.4.0. So I asked
myself if there might be changes not reflected by the version number.

tl;dr: Has the embedded Lua interpreter changed in some non ABI-compatible
way in LuaMetaTeX 2.07.01? (Especially on Linux, x86-64. I didn't test
other platforms yet)

Best regards,
Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Lua related change in LuaMetaTeX?

[Hans Hagen]

On 7/29/2020 6:22 AM, Marcel Fabian Krüger wrote:
> Hi,
> 
> starting with LuaMetaTeX 2.07.01, a C library I load from Lua started to
> segfault and valgrind indicates that it accesses invalid memory in
> multiple places in the middle of Lua internal functions?!
> 
> The whole thing looks almost like the Lua versions in LuaMetaTeX and in
> the library are incompatible, but both seem to use 5.4.0. So I asked
> myself if there might be changes not reflected by the version number.
> 
> tl;dr: Has the embedded Lua interpreter changed in some non ABI-compatible
> way in LuaMetaTeX 2.07.01? (Especially on Linux, x86-64. I didn't test
> other platforms yet)
there have been a couple of bug fixes after the formal 5.4.0 release but 
afaik not in the virtual machine code (which normally is the most 
sensitive); i used to mark low level changes so that it got signaled (in 
context) but the byte code version info changed late in 5.4 dev so i no 
longer do that now

although i occasionally have run into buglets in 5.4 code - which to me 
is quite natural as it concerns dev / exp versions so one gets what one 
downloads, i.e. one can only blame oneself for using such versions and 
hardly complain - lua is quite stable and reliable (given the massive 
amount of lua code that we use in context) ... so, normally patience - 
read: wait for updates - is the answer

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Lua related change in LuaMetaTeX?

[Marcel Fabian Krüger]

> there have been a couple of bug fixes after the formal 5.4.0 release but afaik not in the virtual machine code (which normally is the most sensitive); i used to mark low level changes so that it got signaled (in context) but the byte code version info changed late in 5.4 dev so i no longer do that now
>
> although i occasionally have run into buglets in 5.4 code - which to me is quite natural as it concerns dev / exp versions so one gets what one downloads, i.e. one can only blame oneself for using such versions and hardly complain - lua is quite stable and reliable (given the massive amount of lua code that we use in context) ... so, normally patience - read: wait for updates - is the answer

Oh I see, I wasn't aware that the dev versions are used. After
recompiling with the latest development code the error is gone.

One other question about LuaMetaTeX 2.7.1: I noticed that the
terminal_input callback is gone. Does this mean that the current
behaviour of basically freezing when previously terminal input was
requested is hardcoded or is some other callback invoked which can then
provide some tokens?

Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Lua related change in LuaMetaTeX?

[Hans Hagen]

On 7/29/2020 6:11 PM, Marcel Fabian Krüger wrote:
>> there have been a couple of bug fixes after the formal 5.4.0 release but afaik not in the virtual machine code (which normally is the most sensitive); i used to mark low level changes so that it got signaled (in context) but the byte code version info changed late in 5.4 dev so i no longer do that now
>>
>> although i occasionally have run into buglets in 5.4 code - which to me is quite natural as it concerns dev / exp versions so one gets what one downloads, i.e. one can only blame oneself for using such versions and hardly complain - lua is quite stable and reliable (given the massive amount of lua code that we use in context) ... so, normally patience - read: wait for updates - is the answer
> 
> Oh I see, I wasn't aware that the dev versions are used. After
> recompiling with the latest development code the error is gone.
> 
> One other question about LuaMetaTeX 2.7.1: I noticed that the
> terminal_input callback is gone. Does this mean that the current
> behaviour of basically freezing when previously terminal input was
> requested is hardcoded or is some other callback invoked which can then
> provide some tokens?
first line on console was already gone (only the filename gets pushed 
into the input because tex forces an input

anyway, read/write channel zero are for the terminal

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Lua related change in LuaMetaTeX?

[Marcel Fabian Krüger]

On Wed, Jul 29, 2020 at 09:37:13PM -0700, Hans Hagen wrote:
> > One other question about LuaMetaTeX 2.7.1: I noticed that the
> > terminal_input callback is gone. Does this mean that the current
> > behaviour of basically freezing when previously terminal input was
> > requested is hardcoded or is some other callback invoked which can then
> > provide some tokens?
> > 
> > Marcel
> 
> first line on console was already gone (only the filename gets pushed into the input because tex forces an input

I think this is a misunderstanding. I wasn't referring to the first
line, but to lines after the main input is fully evaluated. E.g. as a
ConTeXt example

  \starttext
  \def\dosomethinguseful#1{Do something useful with #1}
  \dosomethinguseful{
    Ups, I "forgot" to close this brace
  \stoptext

This used to end up asking for terminal input. Especially since the
terminal_input callback was available, it was possible to detect such
situations and trigger an error. With the current ConTeXt code, I
couldn't find a nice solution for this and when running such documents,
the interpreter basically freezes, starts consuming memory and finally
gets killed by the out of memory handler. Experients with
token.scan_next suggest that instead of asking for input, TeX keeps
seeing a special relax-like token. I was hoping that there is some
callback which can be used to intercept this and either fail gracefully
with an error message or to provide more useful tokens.
> 
> anyway, read/write channel zero are for the terminal
> 

That's good to know, thank you.

Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________