Segfault completing: for f in 1; do <[here] x

Segfault completing: for f in 1; do <[here] x

[solo-zsh]

Hi!

I'm getting a segfault when trying to complete after the "<" in:
for f in 1; do < x

This happens every time, tested both on 5.0.0-2 Ubuntu,
4.3.17-1 Debian and git HEAD on Debian, and a nice guy/gal
on IRC reproduced it on HEAD/Debian too.

Derived from a much more reasonable:
for f in pom.xml **/pom.xml; do <pom[here] bar -O '//baz'

...which goes into 100% cpu usage sometimes.
(Yes, this code is obviously wrong, there was some ongoing brainfail.)


faux@om:~/code/zsh% git desc
zsh-5.0.2-130-gc5d9abc

faux@om:~/code/zsh% zsh --version
zsh 5.0.2-dev-0 (x86_64-unknown-linux-gnu)

faux@om:~/code/zsh% gdb --args zsh -f
GNU gdb (GDB) 7.4.1-debian
...
(gdb) r
Starting program: /home/faux/bin/zsh -f
...

om% for f in 1; do < x
Program received signal SIGSEGV, Segmentation fault.
0x00000000004a36ce in itype_end (ptr=0x0, itype=128, once=1) at utils.c:3581
3581            while (*ptr) {
(gdb) bt
#0  0x00000000004a36ce in itype_end (ptr=0x0, itype=128, once=1) at utils.c:3581
#1  0x00007ffff645af64 in get_comp_string () at zle_tricky.c:1481
#2  0x00007ffff64584d1 in docomplete (lst=4) at zle_tricky.c:666
#3  0x00007ffff6457907 in expandorcomplete (args=0x7ffff64770a8) at zle_tricky.c:315
#4  0x00007ffff6445541 in execzlefunc (func=0x7ffff6473538, args=0x7ffff64770a8, set_bindk=0) at zle_main.c:1339
#5  0x00007ffff6444715 in zlecore () at zle_main.c:1063
#6  0x00007ffff644508b in zleread (lp=0x4de940, rp=0x0, flags=3, context=0, init=0x7ffff6468d28 "zle-line-init", finish=0x7ffff6468d18 "zle-line-finish")
    at zle_main.c:1240
#7  0x00007ffff6447b4d in zle_main_entry (cmd=1, ap=0x7fffffffe1a0) at zle_main.c:1901
#8  0x000000000044d3d5 in zleentry (cmd=1) at init.c:1453
#9  0x000000000044e065 in inputline () at input.c:283
#10 0x000000000044dedc in ingetc () at input.c:219
#11 0x0000000000441e4b in ihgetc () at hist.c:279
#12 0x0000000000457b8f in gettok () at lex.c:714
#13 0x00000000004572f7 in zshlex () at lex.c:395
#14 0x0000000000478368 in parse_event () at parse.c:451
#15 0x000000000044a050 in loop (toplevel=1, justonce=0) at init.c:132
#16 0x000000000044d8e6 in zsh_main (argc=2, argv=0x7fffffffe558) at init.c:1619
#17 0x000000000041075c in main (argc=2, argv=0x7fffffffe558) at ./main.c:93


And, for the "pom" case, ctrl+c'ing it after a second or so:

om% for f in pom.xml **/pom.xml; do <pom bar -O '//baz'
Program received signal SIGINT, Interrupt.
__memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:43
43      ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
(gdb) bt
#0  __memset_x86_64 () at ../sysdeps/x86_64/multiarch/../memset.S:43
#1  0x00000000004a4f15 in mb_metacharlenconv_r (s=0x4fefa1 "", wcp=0x0, mbsp=0x4d43c8) at utils.c:4532
#2  0x00000000004a4ff4 in mb_metacharlenconv (s=0x4fefa1 "", wcp=0x0) at utils.c:4576
#3  0x00007ffff645b013 in get_comp_string () at zle_tricky.c:1495
#4  0x00007ffff64584d1 in docomplete (lst=4) at zle_tricky.c:666
#5  0x00007ffff6457907 in expandorcomplete (args=0x7ffff64770a8) at zle_tricky.c:315
#6  0x00007ffff6445541 in execzlefunc (func=0x7ffff6473538, args=0x7ffff64770a8, set_bindk=0) at zle_main.c:1339
#7  0x00007ffff6444715 in zlecore () at zle_main.c:1063
#8  0x00007ffff644508b in zleread (lp=0x4de940, rp=0x0, flags=3, context=0, init=0x7ffff6468d28 "zle-line-init", finish=0x7ffff6468d18 "zle-line-finish")
    at zle_main.c:1240
#9  0x00007ffff6447b4d in zle_main_entry (cmd=1, ap=0x7fffffffe1b0) at zle_main.c:1901
#10 0x000000000044d3d5 in zleentry (cmd=1) at init.c:1453
#11 0x000000000044e065 in inputline () at input.c:283
#12 0x000000000044dedc in ingetc () at input.c:219
#13 0x0000000000441e4b in ihgetc () at hist.c:279
#14 0x0000000000457b8f in gettok () at lex.c:714
#15 0x00000000004572f7 in zshlex () at lex.c:395
#16 0x0000000000478368 in parse_event () at parse.c:451
#17 0x000000000044a050 in loop (toplevel=1, justonce=0) at init.c:132
#18 0x000000000044d8e6 in zsh_main (argc=2, argv=0x7fffffffe568) at init.c:1619
#19 0x000000000041075c in main (argc=2, argv=0x7fffffffe568) at ./main.c:93

I'm not going anywhere near that kind of code,
so I hope someone else is up for it?

Chris.

Re: Segfault completing: for f in 1; do <[here] x

[Bart Schaefer]

On Jul 26, 11:20am, solo-zsh@goeswhere.com wrote:
}
} I'm getting a segfault when trying to complete after the "<" in:
} for f in 1; do < x

Hmm.  Line numbers below refer to zle_tricky.c.

Both of the cases you backtraced indicate that get_comp_string() has
run off the end of [it's internal copy of] the input line.  Stepping
through in the debugger, it correctly determines that it's in a "for"
loop, which is recorded by the mysteriously-named variable "ins", but
then that information is discarded at line 1243 when the ";" is found.

This in turn means that "do" is not treated as a keyword on the next pass
around the loop at line 1224.  Thus it is believed the command word has
been found, and tt0 is assigned NULLTOK.

Next the "<" token is found, but because "do" appeared to be the command
word, redirpos is NOT incremented at line 1216.  It's actually a bit more
complicated than this because wordpos is incremented at line 1353 even
when the word is a separator, so I think the test on 1215 is valid only
for very simple command lines.

In any case the NULLTOK branch at 1257 is now taken, which is correctly
identifying that the word under the cursor has been reached, which
happens to be the dummy string "x" that was added at line 1136 (or in
the second example, the word "pom").  The dummy x is removed and the
result is recorded as the word to be completed (either the empty string
or "pom").

Here's where things get weird.  Because the end of the input line has
not yet been reached, the loop starts again even though the word to be
completed has been found.  (This isn't by itself wrong, we have to fill
in the whole $words array even though CURRENT is going to point into
the middle of it.)  The two examples diverge at this point because of
the number of words that follow the "<".

Because the relative positions of the command word and the redirection
have become confused ("do" not recognized), the code ends up arriving in
the "Check if we are in an array subscript" block at line 1476.  In the
first example no value has ever been assigned to the local variable "s",
so itype_end() derefs a null pointer and segfaults.

In the second example two more passes are done to reach the word "-O".
At this point, zlemetacs_qsub == 36 and wb == 33, so the "for" loop
at line 1485 spins forever trying to step forward three characters in a
two-character string.  The infinite loop could be fixed by breaking
when *tt == 0 in the loop condition or when nclen == 0 at line 1495,
and probably that should get fixed just for sanity, but it doesn't help
with the underlying problem.

That's as far as I've had time to get today.  If someone else wants to
pick up from here, have at it.

Re: Segfault completing: for f in 1; do <[here] x

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 2823 bytes --]

(This may come through twice, I'm having some DNS problems so one copy is
stuck in the sendmail queue.)

On Fri, Jul 26, 2013 at 3:20 AM, <solo-zsh@goeswhere.com> wrote:

>
> I'm getting a segfault when trying to complete after the "<" in:
> for f in 1; do < x
>

Hmm.  Line numbers below refer to zle_tricky.c.

Both of the cases you backtraced indicate that get_comp_string() has run
off the end of [it's internal copy of] the input line.  Stepping through in
the debugger, it correctly determines that it's in a "for" loop, which is
recorded by the mysteriously-named variable "ins", but then that
information is discarded at line 1243 when the ";" is found.

This in turn means that "do" is not treated as a keyword on the next pass
around the loop at line 1224.  Thus it is believed the command word has
been found, and tt0 is assigned NULLTOK.

Next the "<" token is found, but because "do" appeared to be the command
word, redirpos is NOT incremented at line 1216.  It's actually a bit more
complicated than this because wordpos is incremented at line 1353 even when
the word is a separator, so I think the test on 1215 is valid only for very
simple command lines.

In any case the NULLTOK branch at 1257 is now taken, which is correctly
identifying that the word under the cursor has been reached, which happens
to be the dummy string "x" that was added at line 1136 (or in the second
example, the word "pom").  The dummy x is removed and the result is
recorded as the word to be completed (either the empty string or "pom").

Here's where things get weird.  Because the end of the input line has not
yet been reached, the loop starts again even though the word to be
completed has been found.  (This isn't by itself wrong, we have to fill in
the whole $words array even though CURRENT is going to point into the
middle of it.)  The two examples diverge at this point because of the
number of words that follow the "<".

Because the relative positions of the command word and the redirection have
become confused ("do" not recognized), the code ends up arriving in the
"Check if we are in an array subscript" block at line 1476.  In the first
example no value has ever been assigned to the local variable "s", so
itype_end() derefs a null pointer and segfaults.

In the second example two more passes are done to reach the word "-O". At
this point, zlemetacs_qsub == 36 and wb == 33, so the "for" loop at line
1485 spins forever trying to step forward three characters in a
two-character string.  The infinite loop could be fixed by breaking when
*tt == 0 in the loop condition or when nclen == 0 at line 1495, and
probably that should get fixed just for sanity, but it doesn't help with
the underlying problem.

That's as far as I've had time to get today.  If someone else wants to pick
up from here, have at it.

Re: Segfault completing: for f in 1; do <[here] x

[Peter Stephenson]

[Sorry, meant to send this on Friday, which would have helped Bart's
investigation.  Comments welcome.]


On Fri, 26 Jul 2013 11:20:52 +0100
solo-zsh@goeswhere.com wrote:
> I'm getting a segfault when trying to complete after the "<" in:
> for f in 1; do < x

Yes, this is easy to reproduce.

Unfortunately we're in one of those functions that nobody dares rewrite
because it never really got written properly in the first place.

The following tweaks fix the problem above and appear to be moderately safe.
The incomprehensible "ins < 2" test safeguards against hitting the
tricks used with completions after e.g. "repeat 2" which is done a hacky
way.  I don't understand why we need to set "ins" just to be able to
detect the DOLOOP token on the next word, but I definitely don't dare
change that.

Could do with some playing with in similar cases with complicated
syntax.

Anyone with some free time is welcome to have a go at this function and
report back even if they don't actually achieve any improvements (nobody
else ever has).

It's being so cheerful as keeps me going.

diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
index 78a9fa4..610055c 100644
--- a/Src/Zle/zle_tricky.c
+++ b/Src/Zle/zle_tricky.c
@@ -1095,6 +1095,7 @@ get_comp_string(void)
      * the command word is not at index zero in the array.
      */
     int redirpos;
+    int noword;
     char *s = NULL, *tmp, *p, *tt = NULL, rdop[20];
     char *linptr, *u;
 
@@ -1165,7 +1166,7 @@ get_comp_string(void)
     * and whatnot. */
 
     do {
-        qsub = 0;
+        qsub = noword = 0;
 
 	lincmd = ((incmdpos && !ins && !incond) ||
 		  (oins == 2 && wordpos == 2) ||
@@ -1239,6 +1240,19 @@ get_comp_string(void)
 	     * leave the loop.                                           */
 	    if (tt)
 		break;
+	    if (ins < 2) {
+		/*
+		 * Don't add this as a word, because we're about to start
+		 * a new command line: pretend there's no string here.
+		 * We don't dare do this if we're using one of the
+		 * *really* gross hacks with ins to get later words
+		 * to look like command words, because we don't
+		 * understand how they work.  Quite possibly we
+		 * should be using a mechanism like the one here rather
+		 * than the ins thing.
+		 */
+		noword = 1;
+	    }
 	    /* Otherwise reset the variables we are collecting data in. */
 	    wordpos = cp = rd = ins = redirpos = 0;
 	    tt0 = NULLTOK;
@@ -1253,6 +1267,14 @@ get_comp_string(void)
 	    /* If everything before is a redirection, don't reset the index */
 	    if (wordpos != redirpos)
 		wordpos = redirpos = 0;
+	} else if (tok == SEPER) {
+	    /*
+	     * A following DOLOOP should cause us to reset to the start
+	     * of the command line.  For some reason we only recognise
+	     * DOLOOP for this purpose (above) if ins is set.  Why?
+	     * Don't ask pointless questions.
+	     */
+	    ins = 1;
 	}
 	if (!lexflags && tt0 == NULLTOK) {
 	    /* This is done when the lexer reached the word the cursor is on. */
@@ -1322,7 +1344,7 @@ get_comp_string(void)
 	    else if (tok == DAMPER)
 		tokstr = "&&";
 	}
-	if (!tokstr)
+	if (!tokstr || noword)
 	    continue;
 	/* Hack to allow completion after `repeat n do'. */
 	if (oins == 2 && !wordpos && !strcmp(tokstr, "do"))


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/