Bug: Segmentation fault

Bug: Segmentation fault

[Henri Menke]

Dear devs,

The example below crashes LuaTeX with a segmentation fault.  It is reproducible
with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built 7226 from
source with --debugopt to get a backtrace, which can be found below as well.

Cheers, Henri

---

\directlua{token.set_macro(-1, "foo","\string\\,")}
\show\foo
\foo
\bye

---

(gdb) run
Starting program: /usr/local/bin/luatex test.tex
This is LuaTeX, Version 1.11.2 (TeX Live 2020/dev)
  restricted system commands enabled.
(./test.tex
Program received signal SIGSEGV, Segmentation fault.
get_sa_item (head=0x41002, n=n@entry=92) at ../../../source/texk/web2c/luatexdir/utils/managed-sa.c:65
65	    if (head->tree != NULL) {
(gdb) bt full
#0  get_sa_item (head=0x41002, n=n@entry=92) at ../../../source/texk/web2c/luatexdir/utils/managed-sa.c:65
No locals.
#1  0x00000000005607d0 in get_cat_code (h=h@entry=-1, n=n@entry=92) at ../../../source/texk/web2c/luatexdir/tex/textcodes.c:68
         sa_value = <optimized out>
         s = <optimized out>
#2  0x00000000004d8a8c in set_macro (L=<optimized out>) at ../../../source/texk/web2c/luatexdir/lua/lnewtokenlib.c:1204
         p = <optimized out>
         q = <optimized out>
         t = <optimized out>
         se = 0x295bf9a ""
         name = <optimized out>
         str = <optimized out>
         s = <optimized out>
         lname = 3
         lstr = 2
         cs = 743
         cc = <optimized out>
         ct = -1
         n = <optimized out>
         a = 0
         nncs = <optimized out>
#3  0x00000000006dc924 in luaD_precall (L=L@entry=0x1202678, func=func@entry=0x2214490, nresults=nresults@entry=0)
     at ../../../source/libs/lua53/lua53-src/src/ldo.c:434
         n = <optimized out>
         f = 0x4d8890 <set_macro>
         ci = 0x123c830
#4  0x00000000006f62c2 in luaV_execute (L=L@entry=0x1202678) at ../../../source/libs/lua53/lua53-src/src/lvm.c:1134
         b = <optimized out>
         nresults = 0
         i = <optimized out>
         ra = 0x2214490
         ci = 0x12221c0
         cl = 0x2952f00
         k = 0x294ea40
         base = <optimized out>
#5  0x00000000006dcef7 in luaD_call (nResults=<optimized out>, func=<optimized out>, L=0x1202678) at ../../../source/libs/lua53/lua53-src/src/ldo.c:499
No locals.
#6  luaD_callnoyield (L=0x1202678, func=<optimized out>, nResults=<optimized out>) at ../../../source/libs/lua53/lua53-src/src/ldo.c:509
No locals.
#7  0x00000000006db78c in luaD_rawrunprotected (L=0x1202678, f=0x6d0800 <f_call>, ud=0x7fffffffdf80) at ../../../source/libs/lua53/lua53-src/src/ldo.c:142
         oldnCcalls = 0
         lj = {previous = 0x0, b = {{__jmpbuf = {832, -3576387603206626035, 18884312, 1, 18884216, 0, 3576387604335174925, -3576387282536356595},
               __mask_was_saved = 0, __saved_mask = {__val = {0, 18884312, 18884216, 7198250, 35733824, 72057594037927937, 140737488346976, 0, 0, 0, 0, 0, 0,
                   0, 0, 0}}}}, status = 0}
#8  0x00000000006dd3af in luaD_pcall (L=L@entry=0x1202678, func=func@entry=0x6d0800 <f_call>, u=u@entry=0x7fffffffdf80, old_top=832, ef=<optimized out>)
     at ../../../source/libs/lua53/lua53-src/src/ldo.c:729
         status = <optimized out>
         old_ci = 0x12026d8
         old_allowhooks = 1 '\001'
         old_nny = 1
         old_errfunc = 0
         oldtop = <optimized out>
#9  0x00000000006d3907 in lua_pcallk (L=0x1202678, nargs=nargs@entry=0, nresults=nresults@entry=0, errfunc=errfunc@entry=51, ctx=ctx@entry=0, k=k@entry=0x0)
     at ../../../source/libs/lua53/lua53-src/src/lapi.c:969
         c = {func = 0x2214480, nresults = 0}
         status = <optimized out>
         func = <optimized out>
#10 0x0000000000463b5f in luatokencall (p=p@entry=9030, nameptr=nameptr@entry=0) at ../../../source/texk/web2c/luatexdir/lua/luastuff.c:653
         base = 51
         ls = {s = 0x295e890 "\240܂\367\377\177", size = 0}
         i = 0
         l = 32
         s = 0x0
         lua_id = <optimized out>
         stacktop = 50
#11 0x0000000000567a83 in conv_toks () at ../../../source/texk/web2c/luatexdir/tex/textoken.c:2859
         old_setting = <optimized out>
         p = <optimized out>
         q = <optimized out>
         save_scanner_status = 0
         save_def_ref = 0
         save_warning_index = 0
         bool = <optimized out>
         s = 9030
         sn = 0
         u = 0
         c = <optimized out>
         str = <optimized out>
         i = 0
#12 0x0000000000503a25 in expand () at ../../../source/texk/web2c/luatexdir/tex/expand.c:259
         t = <optimized out>
         p = <optimized out>
---Type <return> to continue, or q <return> to quit---
         cur_ptr = <optimized out>
         cv_backup = 0
         cvl_backup = 0
         radix_backup = 0
         co_backup = 0
         backup_backup = 7961
         save_scanner_status = <optimized out>
#13 0x0000000000503f85 in get_x_token () at ../../../source/texk/web2c/luatexdir/tex/expand.c:501
No locals.
#14 0x0000000000513ae5 in main_control () at ../../../source/texk/web2c/luatexdir/tex/maincontrol.c:1014
No locals.
#15 0x000000000050f4ac in main_body () at ../../../source/texk/web2c/luatexdir/tex/mainbody.c:567
         pdftex_map = "pdftex.map"
         bad = <optimized out>
#16 0x0000000000446d2e in main (ac=<optimized out>, av=<optimized out>) at ../../../source/texk/web2c/luatexdir/luatex.c:609
No locals.
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 478 bytes --]

On Tue, Dec 10, 2019 at 9:27 PM Henri Menke <henrimenke@gmail.com> wrote:

> Dear devs,
>
> The example below crashes LuaTeX with a segmentation fault.  It is
> reproducible
> with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built 7226
> from
> source with --debugopt to get a backtrace, which can be found below as
> well.
>
>
confirmed, we will see asap
(btw,  I use --debug ie CFLAGS="-g -O0" because it's hard to follow the
flow of optimized code)


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 923 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[Hans Hagen]

On 12/10/2019 9:26 PM, Henri Menke wrote:
> Dear devs,
> 
> The example below crashes LuaTeX with a segmentation fault.  It is 
> reproducible
> with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built 
> 7226 from
> source with --debugopt to get a backtrace, which can be found below as 
> well.
i see what's going on ... more later


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 530 bytes --]

On Tue, Dec 10, 2019 at 9:27 PM Henri Menke <henrimenke@gmail.com> wrote:

> Dear devs,
>
> The example below crashes LuaTeX with a segmentation fault.  It is
> reproducible
> with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built 7226
> from
> source with --debugopt to get a backtrace, which can be found below as
> well.
>
> Cheers, Henri
>
> ---
>
> \directlua{token.set_macro(-1, "foo","\string\\,")}
> \show\foo
> \foo
> \bye
>

I guess that we have to check if catcodetable -1 is already defined...

--
luigi

[-- Attachment #1.2: Type: text/html, Size: 1152 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[Hans Hagen]

On 12/11/2019 12:03 AM, luigi scarso wrote:
> 
> 
> On Tue, Dec 10, 2019 at 9:27 PM Henri Menke <henrimenke@gmail.com 
> <mailto:henrimenke@gmail.com>> wrote:
> 
>     Dear devs,
> 
>     The example below crashes LuaTeX with a segmentation fault.  It is
>     reproducible
>     with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built
>     7226 from
>     source with --debugopt to get a backtrace, which can be found below
>     as well.
> 
>     Cheers, Henri
> 
>     ---
> 
>     \directlua{token.set_macro(-1, "foo","\string\\,")}
>     \show\foo
>     \foo
>     \bye
> 
> 
> I guess that we have to check if catcodetable -1 is already defined...
i already sent you a patch, probbaly also ok for trunk

but one needs to be careful as fo rinstance \string\foo is seen as 
<formfeed>oo so often one also needs to escape properly

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 1359 bytes --]

On Wed, Dec 11, 2019 at 12:10 AM Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 12/11/2019 12:03 AM, luigi scarso wrote:
> >
> >
> > On Tue, Dec 10, 2019 at 9:27 PM Henri Menke <henrimenke@gmail.com
> > <mailto:henrimenke@gmail.com>> wrote:
> >
> >     Dear devs,
> >
> >     The example below crashes LuaTeX with a segmentation fault.  It is
> >     reproducible
> >     with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built
> >     7226 from
> >     source with --debugopt to get a backtrace, which can be found below
> >     as well.
> >
> >     Cheers, Henri
> >
> >     ---
> >
> >     \directlua{token.set_macro(-1, "foo","\string\\,")}
> >     \show\foo
> >     \foo
> >     \bye
> >
> >
> > I guess that we have to check if catcodetable -1 is already defined...
> i already sent you a patch, probbaly also ok for trunk
>


yes, seen



> but one needs to be careful as fo rinstance \string\foo is seen as
> <formfeed>oo so often one also needs to escape properly
>

 I always start with  [==[ ... ]==]  ie
\directlua{token.set_macro(-1, [==[foo]==],[==[\string\\,]==])}
 gdb reports this [==[\string\\,]==] string as
 "\\\\,"
which is an array of char of length 3 ie {'\\','\\',','}

With \directlua{token.set_macro(-1, [==[foo]==],"\string\\,")}
gdb reports "\\,"
ie {'\\','\,'}

I always check which one is the correct one...

-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 2511 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[Henri Menke]

On 12/11/19 12:23 PM, luigi scarso wrote:
> On Wed, Dec 11, 2019 at 12:10 AM Hans Hagen <j.hagen@xs4all.nl> wrote:
> 
>> On 12/11/2019 12:03 AM, luigi scarso wrote:
>>>
>>>
>>> On Tue, Dec 10, 2019 at 9:27 PM Henri Menke <henrimenke@gmail.com
>>> <mailto:henrimenke@gmail.com>> wrote:
>>>
>>>      Dear devs,
>>>
>>>      The example below crashes LuaTeX with a segmentation fault.  It is
>>>      reproducible
>>>      with LuaTeX 1.10.1 7143 and LuaTeX 1.11.2 7226.  To this end I built
>>>      7226 from
>>>      source with --debugopt to get a backtrace, which can be found below
>>>      as well.
>>>
>>>      Cheers, Henri
>>>
>>>      ---
>>>
>>>      \directlua{token.set_macro(-1, "foo","\string\\,")}
>>>      \show\foo
>>>      \foo
>>>      \bye
>>>
>>>
>>> I guess that we have to check if catcodetable -1 is already defined...
>> i already sent you a patch, probbaly also ok for trunk
>>
> 
> 
> yes, seen
> 
> 
> 
>> but one needs to be careful as fo rinstance \string\foo is seen as
>> <formfeed>oo so often one also needs to escape properly
>>
> 
>   I always start with  [==[ ... ]==]  ie
> \directlua{token.set_macro(-1, [==[foo]==],[==[\string\\,]==])}
>   gdb reports this [==[\string\\,]==] string as
>   "\\\\,"
> which is an array of char of length 3 ie {'\\','\\',','}
> 
> With \directlua{token.set_macro(-1, [==[foo]==],"\string\\,")}
> gdb reports "\\,"
> ie {'\\','\,'}
> 
> I always check which one is the correct one...

In this case I wanted to generate the thin space (\,).  My expectation
(which worked elsewhere) was that full expansion will turn "\string\\,"
into "\\," where the two backslashes have catcode 12 and will not be
expanded further.  Then the Lua string will turn the escape sequence \\
into a single backslash, therefore ending up with \, in the Lua string.
Demo below.

Cheers, Henri


$ luatex <(echo '\relax\directlua{print("\string\\,")}\bye')
This is LuaTeX, Version 1.11.2 (TeX Live 2020/dev)
  restricted system commands enabled.
(/dev/fd/63\,
)
warning  (pdf backend): no pages of output.
Transcript written on 63.log.

> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
> 
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 1616 bytes --]

On Wed, Dec 11, 2019 at 1:28 AM Henri Menke <henrimenke@gmail.com> wrote:

> In this case I wanted to generate the thin space (\,).  My expectation
> (which worked elsewhere) was that full expansion will turn "\string\\,"
> into "\\," where the two backslashes have catcode 12 and will not be
> expanded further.  Then the Lua string will turn the escape sequence \\
> into a single backslash, therefore ending up with \, in the Lua string.
>
>


\directlua{token.set_macro("foo","\relax")}
(gdb) x/7c str
0x555559376c18: 13 '\r' 101 'e' 108 'l' 97 'a'  120 'x' 32 ' '  0 '\000'
(gdb) p lstr
$24 = 6
where \foo is
 \foo=macro:
->#!.

vs
\directlua{token.set_macro("foo",[==[\relax]==])}
(gdb) x/8c str
0x555559376ea8: 92 '\\' 114 'r' 101 'e' 108 'l' 97 'a'  120 'x' 32 ' '  0
'\000'
(gdb) p lstr
$25 = 7
\foo=macro:
->\relax .

Another example is
\directlua{token.set_macro("foo","\medskip")}\show\foo
which gives the error:   invalid escape sequence near '"^^Kskip \m'.
vs
\directlua{token.set_macro("foo",[==[\medskip]==])}\show\foo
 which gives
\foo=macro:
->\vskip \medskipamount .

Your example:
\directlua{token.set_macro("foo","\string\\,")}\show\foo
gives
(gdb) x/3c str
0x5555593765a8: 92 '\\' 44 ','  0 '\000'
(gdb) p lstr
$29 = 2
 \foo=macro:
->BAD,.

vs
\directlua{token.set_macro("foo",[==[\,]==])}\show\foo
(gdb) x/20c str
0x555559376908: 92 '\\' 109 'm' 115 's' 107 'k' 105 'i' 112 'p' 32 ' '  92
'\\'
0x555559376910: 116 't' 104 'h' 105 'i' 110 'n' 109 'm' 117 'u' 115 's' 107
'k'
0x555559376918: 105 'i' 112 'p' 32 ' '  0 '\000'
(gdb) p lstr
$31 = 19
 \foo=macro:
->\mskip \thinmuskip .


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 2932 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[Hans Hagen]

On 12/11/2019 1:28 AM, Henri Menke wrote:

> $ luatex <(echo '\relax\directlua{print("\string\\,")}\bye')
> This is LuaTeX, Version 1.11.2 (TeX Live 2020/dev)
>   restricted system commands enabled.
> (/dev/fd/63\,
> )

I don't even want to think about what piping adds to this (some command 
line processing).

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Bug: Segmentation fault

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 288 bytes --]

On Wed, Dec 11, 2019 at 12:10 AM Hans Hagen <j.hagen@xs4all.nl> wrote:

> On 12/11/2019 12:03 AM, luigi scarso wrote:
> > I guess that we have to check if catcodetable -1 is already defined...
> i already sent you a patch, probbaly also ok for trunk
>

Committed revision 7235.
-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 677 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________