Bug with long multiline strings?

Bug with long multiline strings?

[Frank Terbeck]

Hi workers!

I've tried to reproduce

  <https://bugs.launchpad.net/ubuntu/+source/zsh/+bug/1070820>


Turns out, I can:

[snip]
i-(6000)-~% echo "asdf
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asfd
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asdf
dquote> asdf
 mem.c:604: BUG: hrealloc() called for non-heap memory.
[1]    2554 segmentation fault (core dumped)  zsh
[snap]

Backtrace:

[snip]
Reading symbols from /usr/local/bin/zsh...done.
[New LWP 2554]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `zsh'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000460225 in hrealloc (p=0x7f1e52779050 "\232asdf\nasdf\nasdf\nasdf\nasdf\nasfd\nasdf\nasdf\nasdf\nasdf\nasdf\nasdf\nasd%_> ", old=64, new=128) at mem.c:605
605	    DPUTS(h->sp && arena(h) + h->sp->used > p,
(gdb) #0  0x0000000000460225 in hrealloc (p=0x7f1e52779050 "\232asdf\nasdf\nasdf\nasdf\nasdf\nasfd\nasdf\nasdf\nasdf\nasdf\nasdf\nasdf\nasd%_> ", old=64, new=128) at mem.c:605
        h = 0x0
        ph = 0x7f1e52772000
#1  0x0000000000456291 in add (c=100) at lex.c:575
        newbsiz = 128
#2  0x00000000004583ea in dquote_parse (endchar=34 '"', sub=0) at lex.c:1576
        pct = 0
        brct = 0
        bct = 0
        intick = 0
        err = 0
        c = 100
        math = 0
        zlemath = 0
#3  0x0000000000457b2a in gettokstr (c=34, sub=0) at lex.c:1371
        act = 15
        e = 34
        inbl = 0
        bct = 0
        pct = 0
        brct = 0
        fdpar = 0
        intpos = 1
        in_brace_param = 0
        peek = 34
        inquote = -1189248448
        unmatched = 0
        ocmdsp = 0
#4  0x0000000000457061 in gettok () at lex.c:993
        c = 34
        d = -1189248128
        peekfd = -1
        peek = 32767
#5  0x0000000000455d73 in zshlex () at lex.c:395
No locals.
#6  0x0000000000479a87 in par_simple (complex=0x7fffb91d8448, nr=0) at parse.c:1678
        redir_var = 0
        oecused = 3
        isnull = 1
        r = 3
        argc = 1
        p = 3
        isfunc = 0
        sr = 0
        c = 0
        nrediradd = 0
        assignments = 0
#7  0x0000000000477857 in par_cmd (complex=0x7fffb91d8448) at parse.c:879
        sr = 0
        r = 3
        nr = 0
#8  0x00000000004772f3 in par_pline (complex=0x7fffb91d8448) at parse.c:728
        p = 2
        line = 1
#9  0x00000000004772ac in par_sublist2 (complex=0x7fffb91d8448) at parse.c:709
        f = 0
#10 0x000000000047713d in par_sublist (complex=0x7fffb91d8470) at parse.c:664
        f = 4545929
        p = 1
        c = 1
#11 0x0000000000476b58 in par_event () at parse.c:477
        r = 0
        p = 0
        c = 0
#12 0x0000000000476ad2 in parse_event () at parse.c:454
No locals.
#13 0x0000000000448f34 in loop (toplevel=1, justonce=0) at init.c:132
        prog = 0x7fffb91d84e0
        err = 0
        non_empty = 0
#14 0x000000000044c6c5 in zsh_main (argc=1, argv=0x7fffb91d8648) at init.c:1567
        t = 0x7fffb91d8650
        runscript = 0x0
        t0 = 158
#15 0x000000000041024c in main (argc=1, argv=0x7fffb91d8648) at ./main.c:93
No locals.
[snap]

Information about this system:

% echo $ZSH_VERSION-$ZSH_PATCHLEVEL
5.0.0-dev-0-1.5699

% uname -srm
Linux 3.2.0-3-amd64 x86_64

% /lib32/libc.so.6
GNU C Library (Debian EGLIBC 2.13-35) stable release version 2.13, by
Roland McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.7.
Compiled on a Linux 3.2.21 system on 2012-07-22.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.



Regards, Frank

Re: Bug with long multiline strings?

[Peter Stephenson]

On Wed, 24 Oct 2012 17:21:05 +0200
Frank Terbeck <ft@bewatermyfriend.org> wrote:
> Hi workers!
> 
> I've tried to reproduce
> 
>   <https://bugs.launchpad.net/ubuntu/+source/zsh/+bug/1070820>
> 
> 
> Turns out, I can:
> 
> [snip]
> i-(6000)-~% echo "asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asfd
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
> dquote> asdf
>  mem.c:604: BUG: hrealloc() called for non-heap memory.
> [1]    2554 segmentation fault (core dumped)  zsh
> [snap]

Something very funny is going on here.  That's basic shell allocation
stuff that's failing, which gets used all the time.  Also, it shouldn't
need to reallocate at all until there are 256 characters in the input
token, which you haven't got close to.  There are no interrupts or
recursion involved.

Looks like there's a Mystery Ingredient.

pws

Re: Bug with long multiline strings?

[Peter Stephenson]

On Wed, 24 Oct 2012 17:10:23 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> Also, it
> shouldn't need to reallocate at all until there are 256 characters in
> the input token, which you haven't got close to.

Scratch that: it's initialised at the top of the file to 256, then
initialised when reading a token to 32, which is a bit wayward.  So I
think probably your buffer size has been doubled once and is failing on
the second doubling.

pws

Re: Bug with long multiline strings?

[Peter Stephenson]

On Wed, 24 Oct 2012 17:10:23 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> Looks like there's a Mystery Ingredient.

The only thing I've been able to think of is that some editor widget is
running, and it's doing something which isn't saving and restoring the
lexical state properly.  If it's something like that, it wouldn't be too
hard, if you can see the problem, to rustle up something that checks if
tokstr/bptr or the heap they're part of are being swept away
incorrectly.  It's not happening on any of my installations.

pws

Re: Bug with long multiline strings?

[Frank Terbeck]

Peter Stephenson wrote:
> On Wed, 24 Oct 2012 17:10:23 +0100
> Peter Stephenson <p.stephenson@samsung.com> wrote:
>> Looks like there's a Mystery Ingredient.
>
> The only thing I've been able to think of is that some editor widget is
> running, and it's doing something which isn't saving and restoring the
> lexical state properly.  If it's something like that, it wouldn't be too
> hard, if you can see the problem, to rustle up something that checks if
> tokstr/bptr or the heap they're part of are being swept away
> incorrectly.  It's not happening on any of my installations.


Yeah, I cannot reproduce the issue from "zsh -f".  So there's something
in my setup that helps triggering it. I'll bisect through my
configuration to see what is causing this. But I won't get to it before
Sunday in all likeliness.

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Bug with long multiline strings?

[Frank Terbeck]

Hey list,

here's a blast from the past.  Axel Beckert reminded me, that I wanted to
do this more than a year ago... ;)

Frank Terbeck wrote:
> Peter Stephenson wrote:
>> On Wed, 24 Oct 2012 17:10:23 +0100
>> Peter Stephenson <p.stephenson@samsung.com> wrote:
>>> Looks like there's a Mystery Ingredient.
>>
>> The only thing I've been able to think of is that some editor widget is
>> running, and it's doing something which isn't saving and restoring the
>> lexical state properly.  If it's something like that, it wouldn't be too
>> hard, if you can see the problem, to rustle up something that checks if
>> tokstr/bptr or the heap they're part of are being swept away
>> incorrectly.  It's not happening on any of my installations.
>
> Yeah, I cannot reproduce the issue from "zsh -f".  So there's something
> in my setup that helps triggering it. I'll bisect through my
> configuration to see what is causing this. But I won't get to it before
> Sunday in all likeliness.

Turns out, I didn't get to it before *that* particular sunday. :)

Again, here's what triggers the bug:

[snip]
zsh% foo='asdf
quote> asdf
quote> asdf
quote> asdf
quote> asdf
quote> asfd
quote> asdf
quote> asdf
quote> asdf
quote> asdf
quote> asdf
quote> asdf
quote> asdf
 mem.c:604: BUG: hrealloc() called for non-heap memory.
[1]    2554 segmentation fault (core dumped)  zsh
[snap]

>From "zsh -f", I can trigger the bug with one of the following
two snippets:

#+BEGIN_SRC shell-script
zle-line-init() {
    for w in a; do : $w; done
}

zle -N zle-line-init
#+END_SRC

...and also:

#+BEGIN_SRC shell-script
zle-line-finish() {
    for w in a; do : $w; done
}

zle -N zle-line-finish
#+END_SRC

I tried to minimise the code further, but I couldn't trigger it without
the loop within the function. But it's enough to define one of the two;
although defining both will trigger it as well.

I did take a look at the code, but I didn't get anywhere.


Regards, Frank

Re: Bug with long multiline strings?

[Bart Schaefer]

On Jan 17,  9:24pm, Frank Terbeck wrote:
}
} Again, here's what triggers the bug:
} 
} [snip]
} zsh% foo='asdf
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asfd
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asdf
} quote> asdf
}  mem.c:604: BUG: hrealloc() called for non-heap memory.
} [1]    2554 segmentation fault (core dumped)  zsh
} [snap]
} 
} >From "zsh -f", I can trigger the bug with one of the following
} two snippets:
} 
} #+BEGIN_SRC shell-script
} zle-line-init() {
}     for w in a; do : $w; done
} }
} 
} zle -N zle-line-init
} 
} I tried to minimise the code further, but I couldn't trigger it without
} the loop within the function. But it's enough to define one of the two;
} although defining both will trigger it as well.

Took me a few passes, but I traced the bug to this:

#0  freeheap () at ../../zsh-5.0/Src/mem.c:382
#1  0x0808bd23 in execfor (state=0xbfeb4a20, do_exec=0)
    at ../../zsh-5.0/Src/loop.c:188
#2  0x08067e53 in execcmd (state=0xbfeb4a20, input=0, output=0, how=18, 
    last1=2) at ../../zsh-5.0/Src/exec.c:3232
#3  0x08063b8a in execpline2 (state=0xbfeb4a20, pcode=131, how=18, input=0, 
    output=0, last1=0) at ../../zsh-5.0/Src/exec.c:1691
#4  0x08062f2f in execpline (state=0xbfeb4a20, slcode=12290, how=18, last1=0)
    at ../../zsh-5.0/Src/exec.c:1478
#5  0x08062807 in execlist (state=0xbfeb4a20, dont_change_job=1, exiting=0)
    at ../../zsh-5.0/Src/exec.c:1261
#6  0x0806225a in execode (p=0x91a75f8, dont_change_job=1, exiting=0, 
    context=0x8138102 "shfunc") at ../../zsh-5.0/Src/exec.c:1070
#7  0x0806b22a in runshfunc (prog=0x91a75f8, wrap=0x0, 
    name=0xb7d82f60 "zle-line-init") at ../../zsh-5.0/Src/exec.c:4865
#8  0x0806af66 in doshfunc (shfunc=0x919fa58, doshargs=0x0, noreturnval=1)
    at ../../zsh-5.0/Src/exec.c:4756
#9  0x0811a256 in execzlefunc (func=0x919fca8, args=0xbfeb4d60, set_bindk=1)
    at ../../../zsh-5.0/Src/Zle/zle_main.c:1390


That freeheap() is discarding a heap that is still in use.  I'm not sure
why the mem.c line number shows up as the end of the function, this is a
gdb watchpoint so there must be some kind of optimization involved; the
actual event is that heaps = NULL is happening at line 380 even though
the accumlated multi-line buffer is still live in the heap.

The freeheap() is there so that we don't grow memory continuously over
the course of a long loop, but apparently in some cases it has a bad
interaction with the surrounding pushheap/popheap.  If I simply skip
the heaps = NULL in freeheap, popheap crashes, so something subtle is
going on.

I'm out of time to chase this for now, maybe somebody else can pick it up
before I get back to it.

Re: Bug with long multiline strings?

[Bart Schaefer]

On Jan 18,  1:34pm, Bart Schaefer wrote:
}
} Took me a few passes, but I traced the bug to this:
} 
} #0  freeheap () at ../../zsh-5.0/Src/mem.c:382
} #1  0x0808bd23 in execfor (state=0xbfeb4a20, do_exec=0)
}     at ../../zsh-5.0/Src/loop.c:188
} 
} That freeheap() is discarding a heap that is still in use.

Patch below seems to do the trick.  The bug is a consequence of my change
in workers/29175, and it's only tickled if heap memory is allocated and
re-allocated in chunks of a certain size (because if I repeat Frank's
steps using longer lines at each PS2 prompt, it doesn't happen).

I imagine Peter is getting rather tired of crash bugs getting found and
fixed only days after he does a release.

diff --git a/Src/mem.c b/Src/mem.c
index 5275c6c..d15721c 100644
--- a/Src/mem.c
+++ b/Src/mem.c
@@ -367,6 +367,15 @@ freeheap(void)
 	    }
 #endif
 	} else {
+	    if (h == fheap && h != heaps) {
+		/*
+		 * When deallocating the last arena with free space,
+		 * loop back through the list to find another one.
+		 */
+		fheap = NULL;
+		hn = heaps;
+		continue;
+	    }
 #ifdef USE_MMAP
 	    munmap((void *) h, h->size);
 #else

Re: Bug with long multiline strings?

[Frank Terbeck]

Bart Schaefer wrote:
[...]
> Patch below seems to do the trick. [...]

Great! I tried for a while to trigger the issue but couldn't with this
patch in place.


Regards, Frank

Bug reports after releases (was: Re: Bug with long multiline strings?)

[Axel Beckert]

Hi,

On Sat, Jan 18, 2014 at 08:59:28PM -0800, Bart Schaefer wrote:
> I imagine Peter is getting rather tired of crash bugs getting found and
> fixed only days after he does a release.

Well, trigger was likely the fact that this bug had some activity on
Ubuntu's Launchpad after someone was able to reproduce it with 5.0.5
in Debian.

Since that bug had multiple reports which weren't merged, I started to
review _all_ segfault causing bugs reported in Debian and Ubuntu for
duplicates, and secondarily also for not being forwarded to
zsh-workers, but reproducible. Hence that bunch of forwarded bugs this
weekend.

We recently had that topic of bugs being found _after_ releases (or in
case of distributions after uploads of binary packages) also on
#pkg-zsh (Freenode IRC network, home of Debian's zsh packages) and we
came to the conclusion that it's quite common: Once a release is done,
it triggers a bunch of activity which again triggers a bunch of
bug-reports. This can be nicely seen in the zsh 5.0.x releases: 5.0.0
lasted long, 5.0.1 didn't. 5.0.2 lasted long, 5.0.3 and 5.0.4 didn't.

"-test" releases help, but only reach a small (but active) group.
Uploading just to http://www.zsh.org/pub/ but not SourceForge and
sending the announcement only to zsh-workers/-users and not
zsh-announce clearly helps further as seen with 5.0.3 and 5.0.4. But
some corner-case issues are only found after a release is out or even
only after it propagated into the stable releases of some popular
Linux or BSD distributions... (Yeah, somehow this feels like boiling
down to "release early, release often". :-)

But back to the current situation: All those recently fixed bug
reports about crashes (at least those reports where I was involved)
are bugs which exist for quite a long time (over 1 year at least), so
I don't think there's any kind of urgency to make a new release just
because of them. (Of course I wouldn't mind if they would trigger a
release. :-)

</braindump>

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://noone.org/abe/ (Web)