[BUG] Segfault if zcompile followed by source, on large file

[BUG] Segfault if zcompile followed by source, on large file

[Sebastian Gniazdowski]

Hello
I noticed that zcompiling a 4.29 MiB source and then sourcing it causes segfault. The file is: https://github.com/zdharma/hacking-private/blob/master/eval-room/srctest.zsh. It contains only assigns.

Restarting shell and sourcing the already compiled file – no segfault. Ran bt in lldb:

  * frame #0: 0x00000001090d6c3a zsh-5.4.2-dev-0`addvars(state=0x00007fff56b3d290, pc=<unavailable>, addflags=<unavailable>) + 1482 at exec.c:2435
    frame #1: 0x00000001090d2785 zsh-5.4.2-dev-0`execsimple(state=0x00007fff56b3d290) + 245 at exec.c:1209

The code:

(lldb) down
frame #0: 0x00000001090d6c3a zsh-5.4.2-dev-0`addvars(state=0x00007fff56b3d290, pc=<unavailable>, addflags=<unavailable>) + 1482 at exec.c:2435
   2432		doneps4 = 1;
   2433	    }
   2434	    state->pc = pc;
-> 2435	    while (wc_code(ac = *state->pc++) == WC_ASSIGN) {
   2436		int myflags = flags;
   2437		name = ecgetstr(state, EC_DUPTOK, &htok);
   2438		if (htok)

A dubious try to narrow a cause:

(lldb) p *state->pc
error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory

I exponentially enlarged the file starting from 1 line, and first segfault occurred at 292k lines, 146k was fine.

--  
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: [BUG] Segfault if zcompile followed by source, on large file

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Tue, 02 Jan 2018 15:49 +0100:
> Hello
> I noticed that zcompiling a 4.29 MiB source and then sourcing it causes 
> segfault. The file 
> is: https://github.com/zdharma/hacking-private/blob/master/eval-room/srctest.zsh. 
> It contains only assigns.

For the archives, that file contains the output of `repeat 300000 echo "a=1; b=2; c=3;"`.

Re: [BUG] Segfault if zcompile followed by source, on large file

[Sebastian Gniazdowski]

On 2 Jan 2018 at 15:49:22, Sebastian Gniazdowski (psprint@zdharma.org) wrote:
> I exponentially enlarged the file starting from 1 line, and first segfault occurred  
> at 292k lines, 146k was fine.

Ran valgrind, expecting that something will came up in zcompile, some heap corruption. However it's sourcing that yields something, and it's the thing known from backtrace:

==49954== Memcheck, a memory error detector
==49954== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==49954== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==49954== Command: zsh-5.4.2-dev-0 ./test2.src.script
==49954==
==49954== Invalid read of size 4
==49954==    at 0x100019C3A: addvars (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015784: execsimple (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001506D: execlist (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100014CF3: execode (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x1000351FE: source (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10000EB09: bin_dot (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100001D25: execbuiltin (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001F0E2: execcmd_exec (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001A9EE: execpline2 (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015B80: execpline (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015307: execlist (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100014CF3: execode (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==  Address 0x1016c4000 is not stack'd, malloc'd or (recently) free'd

==49954== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==49954==  Access not within mapped region at address 0x1016C4000
==49954==    at 0x100019C3A: addvars (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015784: execsimple (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001506D: execlist (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100014CF3: execode (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x1000351FE: source (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10000EB09: bin_dot (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100001D25: execbuiltin (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001F0E2: execcmd_exec (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x10001A9EE: execpline2 (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015B80: execpline (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==    by 0x100015307: execlist (in /usr/local/bin/zsh-5.4.2-dev-0)
#zcompile test2.src
==49954==    by 0x100014CF3: execode (in /usr/local/bin/zsh-5.4.2-dev-0)
==49954==  If you believe this happened as a result of a stack
==49954==  overflow in your program's main thread (unlikely but
==49954==  possible), you can try to increase the size of the
==49954==  main thread stack using the --main-stacksize= flag.
==49954==  The main thread stack size used in this run was 8388608.

Valgrind hints that this might be a stack overflow:

==52921==  If you believe this happened as a result of a stack
==52921==  overflow in your program's main thread (unlikely but
==52921==  possible), you can try to increase the size of the
==52921==  main thread stack using the --main-stacksize= flag.
==52921==  The main thread stack size used in this run was 8388608.

However I ran it again with --main-stacksize=9388608 and 19388608, and the issue repeated.

--  
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: [BUG] Segfault if zcompile followed by source, on large file

[Peter Stephenson]

On Tue, 2 Jan 2018 15:49:22 +0100
Sebastian Gniazdowski <psprint@zdharma.org> wrote:
> I noticed that zcompiling a 4.29 MiB source and then sourcing it
> causes segfault.
>
Daniel Shahaf <d.s@daniel.shahaf.name> wrote
> For the archives, that file contains the output of `repeat 300000 echo
> "a=1; b=2; c=3;"`.

I don't think anyone now understands quite how this works, but the
failure comes the first time prog->strs is referred to.  Short strings
are hidden within the word code itself, so prog->strs is only used
for long strings.  It's not clear that should actually be needed here
with the assignments in question.

In any case, tracking through check_dump_file suggests that prog->strs
is empty --- at least, it's being set to point to the end of the valid
mapped ragne, so accessing it at all is invalid.  That may mean
the compiler (which I haven't walked through) also thought it wasn't
needed.

I suspect it's simply that this file is too long for word code
compilation, as word code has intrinisic limits on valid says based
around 32-bit values, with some bits taken away for special encoding so
you don't necessarily get the full range.  However, there are no checks
internally.

pws

Re: [BUG] Segfault if zcompile followed by source, on large file

[Bart Schaefer]

On Fri, Jan 5, 2018 at 2:23 AM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> I suspect it's simply that this file is too long for word code
> compilation, as word code has intrinisic limits on valid says based
> around 32-bit values, with some bits taken away for special encoding so
> you don't necessarily get the full range.

Hmm.  There's no way to execute a construct that can't be converted
into wordcode, is there?  E.g. a sufficiently long function body would
trigger this?

I'm not very familiar with the wordcode internals.  Are there internal
pointers that refer forward or backward to previous locations in the
wordcode, other than for caching of strings?