Crash when capturing command output in completion

Crash when capturing command output in completion

[Vin Shelton]

Using the latest sources, I'm getting a crash during completion with
capturing command output.

$ installed_packages

==>

apel build c-support calc cc-mode cedet-common debug dired easypg ecb
ecrypto edebug ediff edit-utils efs eieio elib emerge fsf-compat
general-docs gnus ibuffer igrep ilisp ispell mail-lib mmm-mode
net-utils os-utils pc pcl-cvs perl-modes pgg prog-modes ps-print psgml
psgml-dtds python-modes re-builder ruby-modes semantic sgml sh-script
speedbar supercite texinfo text-modes tramp vc xemacs-base
xemacs-devel

$ scm_update $(installed_<TAB>

==>

  scm_update $(installed_packages )<TAB>

==> Boom!  My terminal emulator disappeared.
==> Here is the backtrace:

gdb /opt/zsh-2015-01-15-1337/bin/zsh core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/zsh-2015-01-15-1337/bin/zsh...done.
[New LWP 4891]
Core was generated by `/opt/zsh-2015-01-15-1337/bin/zsh -l'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa93238cd40 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007fa93238cd40 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fa930d66d6a in callcompfunc (s=0x7fa933009ff0 "installed_packages",
    fn=0x1aaaa90 "_main_complete") at compcore.c:743
#2  0x00007fa930d67ced in makecomplist (s=0x7fa933009ff0
"installed_packages", incmd=0, lst=0)
    at compcore.c:989
#3  0x00007fa930d65915 in do_completion (dummy=0x7fa9313dcb48 <zlehooks+40>,
    dat=0x7fff21af6710) at compcore.c:349
#4  0x00000000004682ac in runhookdef (h=0x7fa9313dcb48 <zlehooks+40>,
d=0x7fff21af6710)
    at module.c:996
#5  0x00007fa9311c22ed in docompletion (s=0x1af0d08
"\205installed_packages", lst=0, incmd=0)
    at zle_tricky.c:2282
#6  0x00007fa9311bda42 in docomplete (lst=0) at zle_tricky.c:866
#7  0x00007fa9311bc345 in expandorcomplete (args=0x7fa9313dd0c8 <zlenoargs>)
    at zle_tricky.c:315
#8  0x00007fa9311bbf2a in completecall (args=0x7fa9313dd0c8
<zlenoargs>) at zle_tricky.c:208
#9  0x00007fa9311a9502 in execzlefunc (func=0x7fa9313d96a0 <thingies+1920>,
    args=0x7fa9313dd0c8 <zlenoargs>, set_bindk=0) at zle_main.c:1345
#10 0x00007fa9311a86d6 in zlecore () at zle_main.c:1066
#11 0x00007fa9311a9060 in zleread (lp=0x6dbcf0 <prompt>, rp=0x0,
flags=3, context=0,
    init=0x7fa9311cf14d "zle-line-init", finish=0x7fa9311cf13d
"zle-line-finish")
    at zle_main.c:1253
#12 0x00007fa9311abb65 in zle_main_entry (cmd=1, ap=0x7fff21af6cf8) at
zle_main.c:1914
#13 0x000000000044f2a0 in zleentry (cmd=1) at init.c:1516
#14 0x0000000000450197 in inputline () at input.c:287
#15 0x000000000045000b in ingetc () at input.c:221
#16 0x0000000000443699 in ihgetc () at hist.c:361
#17 0x0000000000459b4a in gettok () at lex.c:583
#18 0x0000000000459322 in zshlex () at lex.c:264
#19 0x000000000047c78f in parse_event (endtok=37) at parse.c:538
#20 0x000000000044bd75 in loop (toplevel=1, justonce=0) at init.c:145
#21 0x000000000044f794 in zsh_main (argc=2, argv=0x7fff21af7088) at init.c:1671
#22 0x000000000040e82d in main (argc=2, argv=0x7fff21af7088) at ./main.c:93
(gdb)

I cannot reproduce this in a zsh started with "-f", so this has
something to do with my customizations.  Can anyone else reproduce
this?

Thank you,
  Vin

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 15,  2:34pm, Vin Shelton wrote:
}
} #10 0x00007fa9311a86d6 in zlecore () at zle_main.c:1066
} #11 0x00007fa9311a9060 in zleread (lp=0x6dbcf0 <prompt>, rp=0x0,
} flags=3, context=0,
}     init=0x7fa9311cf14d "zle-line-init", finish=0x7fa9311cf13d
} "zle-line-finish")
}     at zle_main.c:1253

So I suspect you're using something like the auto-fu plugin or OhMyZsh,
because you have recursively entered ZLE from zle-line-init.

We have had other crashes with this configuration, but a stack trace is
not really helpful because typically there will have been a pointer or
counter error somewhere down the stack which causes a crash as the
stack is unwound, so the trace doesn't show the origin of the issue.
We need a run under valgrind or some similar memory debugger.

Here it looks like something has gone wrong with zlemetaline and/or the
calculation of where in the line the right-paren of $( ) appears.

Re: Crash when capturing command output in completion

[Vin Shelton]

Hey, Bart -

On Thu, Jan 15, 2015 at 4:53 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Jan 15,  2:34pm, Vin Shelton wrote:
> }
> } #10 0x00007fa9311a86d6 in zlecore () at zle_main.c:1066
> } #11 0x00007fa9311a9060 in zleread (lp=0x6dbcf0 <prompt>, rp=0x0,
> } flags=3, context=0,
> }     init=0x7fa9311cf14d "zle-line-init", finish=0x7fa9311cf13d
> } "zle-line-finish")
> }     at zle_main.c:1253
>
> So I suspect you're using something like the auto-fu plugin or OhMyZsh,
> because you have recursively entered ZLE from zle-line-init.

I don't know what the auto-fu plugin is.  :-)

I will try not to be offended by your accusing me of using oh-my-zsh,
but I will point out that I am not.  And if I ever visited the site, I
didn't inhale (to misquote former President Clinton).

> We have had other crashes with this configuration, but a stack trace is
> not really helpful because typically there will have been a pointer or
> counter error somewhere down the stack which causes a crash as the
> stack is unwound, so the trace doesn't show the origin of the issue.
> We need a run under valgrind or some similar memory debugger.
>
> Here it looks like something has gone wrong with zlemetaline and/or the
> calculation of where in the line the right-paren of $( ) appears.

That's a very useful observation.  I will see if I can make any
headway on debugging.

  - Vin

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 15,  6:53pm, Vin Shelton wrote:
}
} I don't know what the auto-fu plugin is.  :-)

In a nutshell, it's incremental-complete-word re-implemented using
recursive-edit so it doesn't have to do input key handling in script
code.

} I will try not to be offended by your accusing me of using oh-my-zsh,

Apologies :-)  "something like" ...

What's in your zle-line-init function?

-- 
Barton E. Schaefer

Re: Crash when capturing command output in completion

[Vin Shelton]

On Thu, Jan 15, 2015 at 7:27 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> What's in your zle-line-init function?

Here's a simpler formulation, requiring only compint:

(gdb) run
Starting program: /raid-3tb/opt/zsh-2015-01-15-1337/bin/zsh -f
legolas-i5% autoload -U compinit
legolas-i5% compinit -u
legolas-i5% installed_packages
apel build c-support calc cc-mode cedet-common debug dired easypg ecb
ecrypto edebug ediff edit-utils efs eieio elib emerge fsf-compat
general-docs gnus ibuffer igrep ilisp ispell mail-lib mmm-mode
net-utils os-utils pc pcl-cvs perl-modes pgg prog-modes ps-print psgml
psgml-dtds python-modes re-builder ruby-modes semantic sgml sh-script
speedbar supercite texinfo text-modes tramp vc xemacs-base
xemacs-devel
legolas-i5% echo $(installed_packages<TAB>

This prints " )" before

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7379d40 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff7379d40 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff6367d6a in callcompfunc (s=0x7ffff7ff47a8 "installed_packages",
    fn=0x754df0 "_main_complete") at compcore.c:743
#2  0x00007ffff6368ced in makecomplist (s=0x7ffff7ff47a8
"installed_packages", incmd=0, lst=0)
    at compcore.c:989
#3  0x00007ffff6366915 in do_completion (dummy=0x7ffff67dab48 <zlehooks+40>,
    dat=0x7fffffffd800) at compcore.c:349
#4  0x00000000004682ac in runhookdef (h=0x7ffff67dab48 <zlehooks+40>,
d=0x7fffffffd800)
    at module.c:996
#5  0x00007ffff65c02ed in docompletion (s=0x75f128 "", lst=0, incmd=0)
at zle_tricky.c:2282
#6  0x00007ffff65bba42 in docomplete (lst=0) at zle_tricky.c:866
#7  0x00007ffff65ba345 in expandorcomplete (args=0x7ffff67db0c8 <zlenoargs>)
    at zle_tricky.c:315
#8  0x00007ffff65b9f2a in completecall (args=0x7ffff67db0c8
<zlenoargs>) at zle_tricky.c:208
#9  0x00007ffff65a7502 in execzlefunc (func=0x7ffff67d76a0 <thingies+1920>,
    args=0x7ffff67db0c8 <zlenoargs>, set_bindk=0) at zle_main.c:1345
#10 0x00007ffff65a66d6 in zlecore () at zle_main.c:1066
#11 0x00007ffff65a7060 in zleread (lp=0x6dbcf0 <prompt>, rp=0x0,
flags=3, context=0,
    init=0x7ffff65cd14d "zle-line-init", finish=0x7ffff65cd13d
"zle-line-finish")
    at zle_main.c:1253
#12 0x00007ffff65a9b65 in zle_main_entry (cmd=1, ap=0x7fffffffdde8) at
zle_main.c:1914
#13 0x000000000044f2a0 in zleentry (cmd=1) at init.c:1516
#14 0x0000000000450197 in inputline () at input.c:287
#15 0x000000000045000b in ingetc () at input.c:221
#16 0x0000000000443699 in ihgetc () at hist.c:361
#17 0x0000000000459b4a in gettok () at lex.c:583
#18 0x0000000000459322 in zshlex () at lex.c:264
#19 0x000000000047c78f in parse_event (endtok=37) at parse.c:538
#20 0x000000000044bd75 in loop (toplevel=1, justonce=0) at init.c:145
#21 0x000000000044f794 in zsh_main (argc=2, argv=0x7fffffffe178) at init.c:1671
#22 0x000000000040e82d in main (argc=2, argv=0x7fffffffe178) at ./main.c:93
(gdb)

  - Vin

Re: Crash when capturing command output in completion

[Vin Shelton]

On Thu, Jan 15, 2015 at 10:11 PM, Vin Shelton <acs@alumni.princeton.edu> wrote:
> Here's a simpler formulation, requiring only compint:
>
> (gdb) run
> Starting program: /raid-3tb/opt/zsh-2015-01-15-1337/bin/zsh -f
> legolas-i5% autoload -U compinit
> legolas-i5% compinit -u
> legolas-i5% installed_packages
> apel build c-support calc cc-mode cedet-common debug dired easypg ecb
> ecrypto edebug ediff edit-utils efs eieio elib emerge fsf-compat
> general-docs gnus ibuffer igrep ilisp ispell mail-lib mmm-mode
> net-utils os-utils pc pcl-cvs perl-modes pgg prog-modes ps-print psgml
> psgml-dtds python-modes re-builder ruby-modes semantic sgml sh-script
> speedbar supercite texinfo text-modes tramp vc xemacs-base
> xemacs-devel
> legolas-i5% echo $(installed_packages<TAB>

I'm sorry, the recipe was wrong.

echo $(installed_packages)<TAB>

results in the crash.

  - Vin

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 15, 10:17pm, Vin Shelton wrote:
}
} echo $(installed_packages)<TAB>
} 
} results in the crash.

OK, I can pretty trivially reproduce this, but what I get is:

schaefer<501> echo $(ls)
zsh: fatal error: out of memory

742                 compisuffix = (char *) zalloc((l = parwe - we) + 1);
(gdb) p parwe
$1 = 10
(gdb) p we
$2 = 12

So this ends up calling zcalloc(-1), and kapow.

"we" becomes 12 at compcore.c line 1261 in this block:

	    /* And adjust wb, we, and offs again. */
	    offs -= b - s;
	    wb = zlemetacs - offs;
	    we = wb + e - b;
	    ispar = (br >= 2 ? 2 : 1);
	    b[we-wb] = '\0';
	    return b;

We come into that block at line 1219 like this:

(gdb) p offs
$11 = 1
(gdb) p e
$12 = 0x818263 "\211x"
(gdb) p s
$13 = 0x818260 "\205ls\211x"
(gdb) p b
$14 = 0x818261 "ls\211x"
(gdb) p s
$15 = 0x818260 "\205ls\211x"
(gdb) p zlemetacs
$16 = 10
(gdb) p wb
$17 = 9
(gdb) p we
$18 = 10
(gdb) p br
$19 = 1

Neither "test" nor "set" is true, so we enter "if (compfunc)" at line
1251 to /* Save the prefix. */  This does nothing but assign "$" to
parpre, so the initial state of all those variables is unchanged by
the time we start to "adjust" them.

I'm not sure what's supposed to happen at this point.  None of these
values seem to line up with where the "adjustments" expect them.  I
suspect we've accidentally attempted to treat "(ls)" as a parameter
name.

The only recent change in compcore.c is lexsave() -> zcontext_save();
the last thing to actually touch this function was parameter modifier
handling, which is a different [new] branch of the same if/else and
happens independent of the branch causing this error.

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 15,  8:43pm, Bart Schaefer wrote:
}
} schaefer<501> echo $(ls)
} zsh: fatal error: out of memory

As probably should have been obvious to me, if I back out to before
workers/34160 (commit c0d01a6fe0c67911650730cf13a2b9a0db16e59b) then
this error goes away.

However, it looks to have something to do with factoring lexsave/restore.
Probably zle_chline isn't getting initialized in all the places that it
needs to be.

Re: Crash when capturing command output in completion

[Ray Andrews]

$ echo $(ls zsh)
zsh-5.0.7-175-g7e74495
zsh

$ echo $(ls zsh)                    << up arrow to recall command.
zsh-5.0.7-175-g7e74495
zsh


$ echo $(ls zsh)
zsh-5.0.7-217-g39a6e8a
zsh

$ echo ls zsh)                        << up arrow to recall command.
zsh: parse error near `)'

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 15, 10:43pm, Ray Andrews wrote:
}
} $ echo $(ls zsh)
} zsh-5.0.7-217-g39a6e8a
} zsh
} 
} $ echo ls zsh)                        << up arrow to recall command.
} zsh: parse error near `)'

Is that "zsh -f"?

I can't reproduce anything similar.

Re: Crash when capturing command output in completion

[Peter Stephenson]

On Thu, 15 Jan 2015 21:20:36 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jan 15,  8:43pm, Bart Schaefer wrote:
> }
> } schaefer<501> echo $(ls)
> } zsh: fatal error: out of memory
> 
> As probably should have been obvious to me, if I back out to before
> workers/34160 (commit c0d01a6fe0c67911650730cf13a2b9a0db16e59b) then
> this error goes away.
> 
> However, it looks to have something to do with factoring lexsave/restore.
> Probably zle_chline isn't getting initialized in all the places that it
> needs to be.

It's not just the rearrangement of saving and restoring, anyway.

The old skipcomm() didn't interact directly with any of the stuff
kludged into the lexer to handle completion, as far as I can see.  It
might be to do with the differences in saving and restoring of state
over the internals of $(...).  If the integration between completion and
the lexer were neat enough, this would be too deep for it to care: it's
just parsing a string recursively.  However, it's never that simple.

Later, completion tries to execute the $(...) to replace the output.
In theory that's not really changed because all it needs is the
string that's been parsed a bit differently...  however, the entry point
to the lexer is also different here.

One thing that needs looking at is whether the history input function
pointers are special over completion.  However, a quick grep suggested
they weren't.

An approach to debugging might be to put back just the old skipcomm(),
which depends less on the rest of the system than the new one, and
nothing else (one replacement of bptr with lexbuf.ptr is needed) and
look for what the resulting differences are.  This appears to
work the way I'd expect (doesn't crash, expands output) but I don't have
time to look further.

(If the worst comes to the worst and this proves intractable, using
a fallback skipcomm() for completion only would be possible.  However, I
suspect this is revealing some interesting difference that should be
fixable.)

pws

Re: Crash when capturing command output in completion

[Peter Stephenson]

Only very lightly tested and so probably partial fix --- however, I'll
commit this because (i) I'm fairly sure it's going in the right
direction (ii) it does work in some cases and hence the shell is less
crash-prone (iii) I get to insert a whiny message about the horrible
interface which always improves my morale.

The first hunk is just a bit of clarity about what belongs to the
lex/zle non-interface for future reference.

I'll remove the ZSH_OLD_SKIPCOMM code when we have a final fix for
this (though it's not actually doing any harm).

pws

diff --git a/Src/lex.c b/Src/lex.c
index b0cd86c..96da1cb 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -90,6 +90,12 @@ int inalmore;
 int nocorrect;
 
 /*
+ * TBD: the following exported variables are part of the non-interface
+ * with ZLE for completion.  They are poorly named and the whole
+ * scheme is incredibly brittle.  One piece of robustness is applied:
+ * the variables are only set if LEXFLAGS_ZLE is set.  Improvements
+ * should therefore concentrate on areas with this flag set.
+ *
  * Cursor position and line length in zle when the line is
  * metafied for access from the main shell.
  */
@@ -113,6 +119,16 @@ mod_export int addedx;
 /**/
 mod_export int wb, we;
 
+/**/
+mod_export int wordbeg;
+
+/**/
+mod_export int parbegin;
+
+/**/
+mod_export int parend;
+
+
 /* 1 if aliases should not be expanded */
 
 /**/
@@ -134,15 +150,6 @@ mod_export int noaliases;
 /**/
 mod_export int lexflags;
 
-/**/
-mod_export int wordbeg;
-
-/**/
-mod_export int parbegin;
-
-/**/
-mod_export int parend;
-
 /* don't recognize comments */
 
 /**/
@@ -585,7 +592,8 @@ gettok(void)
     if (lexstop)
 	return (errflag) ? LEXERR : ENDINPUT;
     isfirstln = 0;
-    wordbeg = inbufct - (qbang && c == bangchar);
+    if ((lexflags & LEXFLAGS_ZLE))
+	wordbeg = inbufct - (qbang && c == bangchar);
     hwbegin(-1-(qbang && c == bangchar));
     /* word includes the last character read and possibly \ before ! */
     if (dbparens) {
@@ -1813,6 +1821,78 @@ zshlex_raw_back(void)
 static int
 skipcomm(void)
 {
+#ifdef ZSH_OLD_SKIPCOMM
+    int pct = 1, c, start = 1;
+
+    cmdpush(CS_CMDSUBST);
+    SETPARBEGIN
+    c = Inpar;
+    do {
+	int iswhite;
+	add(c);
+	c = hgetc();
+	if (itok(c) || lexstop)
+	    break;
+	iswhite = inblank(c);
+	switch (c) {
+	case '(':
+	    pct++;
+	    break;
+	case ')':
+	    pct--;
+	    break;
+	case '\\':
+	    add(c);
+	    c = hgetc();
+	    break;
+	case '\'': {
+	    int strquote = lexbuf.ptr[-1] == '$';
+	    add(c);
+	    STOPHIST
+	    while ((c = hgetc()) != '\'' && !lexstop) {
+		if (c == '\\' && strquote) {
+		    add(c);
+		    c = hgetc();
+		}
+		add(c);
+	    }
+	    ALLOWHIST
+	    break;
+	}
+	case '\"':
+	    add(c);
+	    while ((c = hgetc()) != '\"' && !lexstop)
+		if (c == '\\') {
+		    add(c);
+		    add(hgetc());
+		} else
+		    add(c);
+	    break;
+	case '`':
+	    add(c);
+	    while ((c = hgetc()) != '`' && !lexstop)
+		if (c == '\\')
+		    add(c), add(hgetc());
+		else
+		    add(c);
+	    break;
+	case '#':
+	    if (start) {
+		add(c);
+		while ((c = hgetc()) != '\n' && !lexstop)
+		    add(c);
+		iswhite = 1;
+	    }
+	    break;
+	}
+	start = iswhite;
+    }
+    while (pct);
+    if (!lexstop)
+	SETPAREND
+    cmdpop();
+    return lexstop;
+#else
     char *new_tokstr;
     int new_lexstop, new_lex_add_raw;
     struct lexbufstate new_lexbuf;
@@ -1860,6 +1940,18 @@ skipcomm(void)
     tokstr_raw = new_tokstr;
     lexbuf_raw = new_lexbuf;
     lex_add_raw = new_lex_add_raw;
+    /*
+     * Don't do any ZLE specials down here: they're only needed
+     * when we return the string from the recursive parse.
+     * (TBD: this probably means we should be initialising lexflags
+     * more consistently.)
+     *
+     * Note that in that case we're still using the ZLE line reading
+     * function at the history layer --- this is consistent with the
+     * intention of maintaining the history and input layers across
+     * the recursive parsing.
+     */
+    lexflags &= ~LEXFLAGS_ZLE;
 
     if (!parse_event(OUTPAR) || tok != OUTPAR)
 	lexstop = 1;
@@ -1907,4 +1999,5 @@ skipcomm(void)
     cmdpop();
 
     return lexstop;
+#endif
 }

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 16, 12:57pm, Peter Stephenson wrote:
}
} Only very lightly tested and so probably partial fix --- however, I'll
} commit this because (i) I'm fairly sure it's going in the right
} direction (ii) it does work in some cases and hence the shell is less
} crash-prone (iii) I get to insert a whiny message about the horrible
} interface which always improves my morale.

OK, this changed something -- I don't get a crash any more (though I
probably should as there is obviously a rogue pointer) but *without*
-DZSH_OLD_SKIPCOMM I get this:

schaefer<501> echo ÿÿÿÿÿÿÿÿ$(ls)
Completing all expansions
ÿÿÿÿÿÿÿÿ$(ls)
Completing original
$(ls)


*With* ZSH_OLD_SKIPCOMM everything works.  So we're looking in the
right place, anyway.

Re: Crash when capturing command output in completion

[Ray Andrews]

On 01/15/2015 11:49 PM, Bart Schaefer wrote:
> On Jan 15, 10:43pm, Ray Andrews wrote:
> }
> } $ echo $(ls zsh)
> } zsh-5.0.7-217-g39a6e8a
> } zsh
> }
> } $ echo ls zsh)                        << up arrow to recall command.
> } zsh: parse error near `)'
>
> Is that "zsh -f"?
>
> I can't reproduce anything similar.
>
No. But all those use the debugging config if that matters. When I saw 
the completion problem posting, I stepped back through my builds, and 
that's the point at which it appears.  I also could not complete the 
names of my builds using '-217-' or latter, I either got an error msg. 
or my xterm was killed, but the above seems similar to the test you used 
to duplicate the problem.

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 16,  8:18am, Bart Schaefer wrote:
}
} OK, this changed something -- I don't get a crash any more (though I
} probably should as there is obviously a rogue pointer) but *without*
} -DZSH_OLD_SKIPCOMM I get this:
} 
} ÿÿÿÿÿÿÿÿ$(ls)

This has to mean it's a difference between hgetc() and lex_add_raw()
or whatever; I don't really follow yet what's happening with those
new routines.

Re: Crash when capturing command output in completion

[Bart Schaefer]

On Jan 16,  8:21am, Ray Andrews wrote:
}
} > } zsh-5.0.7-217-g39a6e8a
} > } $ echo ls zsh)                        << up arrow to recall command.
} > } zsh: parse error near `)'
} >
} > Is that "zsh -f"?
} >
} No. But all those use the debugging config if that matters. When I saw 
} the completion problem posting, I stepped back through my builds, and 
} that's the point at which it appears.

That could be a clue that it's related to this:

34234: use structures for normal and raw lexical buffer state

Re: Crash when capturing command output in completion

[Peter Stephenson]

On Fri, 16 Jan 2015 08:18:43 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jan 16, 12:57pm, Peter Stephenson wrote:
> }
> } Only very lightly tested and so probably partial fix --- however, I'll
> } commit this because (i) I'm fairly sure it's going in the right
> } direction (ii) it does work in some cases and hence the shell is less
> } crash-prone (iii) I get to insert a whiny message about the horrible
> } interface which always improves my morale.
> 
> OK, this changed something -- I don't get a crash any more (though I
> probably should as there is obviously a rogue pointer) but *without*
> -DZSH_OLD_SKIPCOMM I get this:
> 
> schaefer<501> echo ÿÿÿÿÿÿÿÿ$(ls)
> Completing all expansions
> ÿÿÿÿÿÿÿÿ$(ls)
> Completing original
> $(ls)

That's not happening here.  I had no problem with inserting the output
of ls in Src word by word, though there was some evidence of display
funnies when it tried to insert the complete set of words as an
expansion.

If it still happens after rebuilding from scratch, could you see if it's
related to what ls is actually outputting or some completion option?

> This has to mean it's a difference between hgetc() and lex_add_raw()
> or whatever; I don't really follow yet what's happening with those
> new routines.

The sole purpose of lex_add_raw() is to track the input buffer to be
copied back raw as the contents of the $(...).  Meanwhile, on top of
that, it's being parsed, but that's no use to us later as we need the
string for the command line argument.

The complexity comes when the input tracked in this way (tokstr_raw)
gets reinstalled in the normal string (tokstr) when we get to the end of
the parentheses.  From then on it's just a string again, until the
$(...) is actually executed.

pws

Re: Crash when capturing command output in completion

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Jan 16, 2015 10:09 AM, "Peter Stephenson" <p.w.stephenson@ntlworld.com>
wrote:
>
> On Fri, 16 Jan 2015 08:18:43 -0800
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > OK, this changed something -- I don't get a crash any more (though I
> > probably should as there is obviously a rogue pointer) but *without*
> > -DZSH_OLD_SKIPCOMM I get this:
> >
> > schaefer<501> echo ÿÿÿÿÿÿÿÿ$(ls)
> > Completing all expansions
> > ÿÿÿÿÿÿÿÿ$(ls)
> > Completing original
> > $(ls)
>
> That's not happening here.

It also does not happen to me when compiling on MacOS, so I will try again
from "make clean" when I am back at my other build machine.

Re: Crash when capturing command output in completion

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

On Jan 16, 2015 10:58 AM, "Bart Schaefer" <schaefer@brasslantern.com> wrote:
>
> On Jan 16, 2015 10:09 AM, "Peter Stephenson" <p.w.stephenson@ntlworld.com>
wrote:
> >
> > On Fri, 16 Jan 2015 08:18:43 -0800
> > Bart Schaefer <schaefer@brasslantern.com> wrote:
> > > schaefer<501> echo ÿÿÿÿÿÿÿÿ$(ls)
> > > Completing all expansions
> > > ÿÿÿÿÿÿÿÿ$(ls)
> > > Completing original
> > > $(ls)
> >
> > That's not happening here.
>
> It also does not happen to me when compiling on MacOS

However, Ray's history recall problem DOES happen to me on MacOS ... start
from zsh -f and setopt histreduceblanks to see it.  Seems odd that would
have been affected.

Re: Crash when capturing command output in completion

[Ray Andrews]

Just now:

$ echo $(ls zsh)
zsh-5.0.7-231-gdb05cc5
zsh

$ echo ls zsh)                        << up arrow as before.
zsh: parse error near `)'

Re: Crash when capturing command output in completion

[Peter Stephenson]

On Fri, 16 Jan 2015 11:16:28 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> However, Ray's history recall problem DOES happen to me on MacOS ... start
> from zsh -f and setopt histreduceblanks to see it.

That explains that...  You'd also have seen it when trying to extract
words from the history.

I wonder how many oddities are left...

diff --git a/Src/hist.c b/Src/hist.c
index 1c5d045..c77b5dd 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -134,6 +134,8 @@ mod_export int hist_skip_flags;
 /* Bits of histactive variable */
 #define HA_ACTIVE	(1<<0)	/* History mechanism is active */
 #define HA_NOINC	(1<<1)	/* Don't store, curhist not incremented */
+#define HA_INWORD       (1<<2)  /* We're inside a word, don't add
+				   start and end markers */
 
 /* Array of word beginnings and endings in current history line. */
 
@@ -299,6 +301,22 @@ hist_context_restore(const struct hist_stack *hs, int toplevel)
     cmdsp = hs->csp;
 }
 
+/*
+ * Mark that the current level of history is or is not
+ * within a word, whatever turns up.  This is used for nested
+ * parsing of substitutions.
+ */
+
+/**/
+void
+hist_in_word(int yesno)
+{
+    if (yesno)
+	histactive |= HA_INWORD;
+    else
+	histactive &= ~HA_INWORD;
+}
+
 /* restore history context */
 
 /* add a character to the current history word */
@@ -1496,7 +1514,7 @@ int hwgetword = -1;
 void
 ihwbegin(int offset)
 {
-    if (stophist == 2)
+    if (stophist == 2 || (histactive & HA_INWORD))
 	return;
     if (chwordpos%2)
 	chwordpos--;	/* make sure we're on a word start, not end */
@@ -1516,7 +1534,7 @@ ihwbegin(int offset)
 void
 ihwend(void)
 {
-    if (stophist == 2)
+    if (stophist == 2 || (histactive & HA_INWORD))
 	return;
     if (chwordpos%2 && chline) {
 	/* end of word reached and we've already begun a word */
diff --git a/Src/lex.c b/Src/lex.c
index 96da1cb..6d45c70 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -1922,6 +1922,7 @@ skipcomm(void)
 	new_lexbuf = lexbuf;
 
 	zcontext_save_partial(ZCONTEXT_LEX|ZCONTEXT_PARSE);
+	hist_in_word(1);
     } else {
 	/*
 	 * Set up for nested command subsitution, however
@@ -1992,6 +1993,7 @@ skipcomm(void)
 	tokstr = new_tokstr;
 	lexbuf = new_lexbuf;
 	lexstop = new_lexstop;
+	hist_in_word(0);
     }
 
     if (!lexstop)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/