Segfault in hrealloc somewhere between rpromts and syntax highlighting

Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Sebastian Götte]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

Hi,

the problem occurs when entering/pasting the contents of the attached file named "codegolf" into a zsh.

I attached a minimal zshrc causing the problem. The zsh-syntax-highlighting folder contains a clone of git://github.com/zsh-users/zsh-syntax-highlighting.git

newton~ <3 uname -a
uname -a
Linux newton 3.13.5-1-ARCH #1 SMP PREEMPT Sun Feb 23 00:25:24 CET 2014 x86_64 GNU/Linux
newton~ <3 zsh --version
zsh --version
zsh 5.0.5 (x86_64-unknown-linux-gnu)
newton~ <3 gdb zsh
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from zsh...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/zsh 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
newton~ <3 echo '
quote> while 1:print(t>>15&(t>>(2<<t%2))%(3+(t>>(8+t%2*3))%4)+(t>>10)|42&t>>7&t<<9,end="");t+=1'

Program received signal SIGSEGV, Segmentation fault.
0x0000000000451de7 in hrealloc ()
(gdb) bt
#0  0x0000000000451de7 in hrealloc ()
#1  0x00000000004496d5 in add ()
#2  0x000000000044a617 in ?? ()
#3  0x000000000044bba1 in zshlex ()
#4  0x0000000000465e4c in ?? ()
#5  0x0000000000466b0a in ?? ()
#6  0x0000000000466d6a in ?? ()
#7  0x0000000000466dfb in ?? ()
#8  0x0000000000467991 in ?? ()
#9  0x0000000000468631 in parse_event ()
#10 0x000000000043dc49 in loop ()
#11 0x0000000000440f5e in zsh_main ()
#12 0x00007ffff70e7b05 in __libc_start_main () from /usr/lib/libc.so.6
#13 0x000000000040f7be in _start ()
(gdb) 

[-- Attachment #2: minimal-zshrc --]
[-- Type: text/plain, Size: 158 bytes --]

setopt promptsubst
RPROMPT='$("%s")'
ZSH_HIGHLIGHT_HIGHLIGHTERS=(main brackets pattern)
source ~/dotfiles/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

[-- Attachment #3: codegolf --]
[-- Type: text/plain, Size: 120 bytes --]

python3 -c 't=0
while 1:print(t>>15&(t>>(2<<t%2))%(3+(t>>(8+t%2*3))%4)+(t>>10)|42&t>>7&t<<9,end="");t+=1'|aplay -c2 -r4

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Bart Schaefer]

I'm only a little surprised that nobody else responded to this while I was
offline.  It's been known for quite some time that there are crash bugs in
region_highlight that are violently tickled by zsh-syntax-highlighting.
None of the regular developers uses zsh-syntax-highlighting as far as I
know (I certainly don't), so we're not encountering this directly.

Unfortunately, the actual error is somewhere far removed from the point
where the crash occurs -- something is leaving a corrupted heap as an
unintentional booby-trap for hrealloc to trip only after the evidence
has been destroyed -- so the stack traces we get are not helpful.

I do have one question for you:

On Mar 30,  3:13pm, Sebastian Götte said this was a minimal zshrc:
} 
} setopt promptsubst
} RPROMPT='$("%s")'
} ZSH_HIGHLIGHT_HIGHLIGHTERS=(main brackets pattern)
} source ~/dotfiles/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh


If I make that my .zshrc (with appropriate tweak to the path to the
zsh-syntax-highlighting clone), I get this:

zsh: command not found: %s                                                      
torch% 
zsh: command not found: %s                                                      
torch% 

Obviously something is missing here.  What's supposed to fill in that %s
in the RPROMPT?

I'm otherwise not able to reproduce the crash with the sample you provided,
though running under valgrind creates a continuous stream of leaked memory
warnings during highlighting.  Here's a patch for those leaks.


diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 8ce6787..80be27f 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -444,6 +444,7 @@ void
 set_region_highlight(UNUSED(Param pm), char **aval)
 {
     int len;
+    char **av = aval;
     struct region_highlight *rhp;
 
     len = aval ? arrlen(aval) : 0;
@@ -490,6 +491,8 @@ set_region_highlight(UNUSED(Param pm), char **aval)
 
 	match_highlight(strp, &rhp->atr);
     }
+
+    freearray(av);
 }
 
 
diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
index 3c7cff9..b916bd6 100644
--- a/Src/Zle/zle_tricky.c
+++ b/Src/Zle/zle_tricky.c
@@ -2795,6 +2795,7 @@ doexpandhist(void)
     if (!err) {
 	zlemetacs = excs;
 	if (strcmp(zlemetaline, ol)) {
+	    zle_restore_positions();
 	    unmetafy_line();
 	    /* For vi mode -- reset the beginning-of-insertion pointer   *
 	     * to the beginning of the line.  This seems a little silly, *
diff --git a/Src/hist.c b/Src/hist.c
index 1845bd8..1624912 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -1764,7 +1764,8 @@ chrealpath(char **junkptr)
 	str++;
     }
 
-    *junkptr = metafy(bicat(real, nonreal), -1, META_HEAPDUP);
+    *junkptr = metafy(str = bicat(real, nonreal), -1, META_HEAPDUP);
+    zsfree(str);
 #ifdef HAVE_CANONICALIZE_FILE_NAME
     free(real);
 #endif

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Peter Stephenson]

On Sun, 06 Apr 2014 00:05:41 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm otherwise not able to reproduce the crash with the sample you provided,

I wasn't either, where I've tried it (Fedora, which most of my instances
of zsh use; I have access to a RedHat Enterprise system which doesn't
vary the base much), as with previous similar reports.

Probably my only other local option is Cygwin, though even if
reproducible there it might not be a great debugging experience.

It would be interesting to hear experiences of other systems where the
memory corruption associated with syntax highlighting is or is not
easily reproducible as an effor to help narrow it down --- configuration
flags as well as OS might conceivably be relevant.

pws

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Bart Schaefer]

On Apr 6,  5:02pm, Peter Stephenson wrote:
} Subject: Re: Segfault in hrealloc somewhere between rpromts and syntax hig
}
} On Sun, 06 Apr 2014 00:05:41 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > I'm otherwise not able to reproduce the crash
} 
} I wasn't either, where I've tried it

I suppose it's possible that this hunk of my patch is related:

> @@ -2795,6 +2795,7 @@ doexpandhist(void)
>      if (!err) {
>  	zlemetacs = excs;
>  	if (strcmp(zlemetaline, ol)) {
> +	    zle_restore_positions();
>  	    unmetafy_line();
>  	    /* For vi mode -- reset the beginning-of-insertion pointer   *
>  	     * to the beginning of the line.  This seems a little silly, *

If the highlight positions were not properly restored, highlighting might
reference non-existent parts of the line?  But then I would have expected
that incorrect highlighting would be a visible symptom in some cases.

In any event, Peter, I'd appreciate it if you took a closer look at that
particular change (zle_tricky.c) because it's the only one where I'm not
certain I was able to repeat the steps that caused the complaint after I
made the edit.

-- 
Barton E. Schaefer

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Peter Stephenson]

On Sun, 06 Apr 2014 09:50:53 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I suppose it's possible that this hunk of my patch is related:
> 
> > @@ -2795,6 +2795,7 @@ doexpandhist(void)
> >      if (!err) {
> >  	zlemetacs = excs;
> >  	if (strcmp(zlemetaline, ol)) {
> > +	    zle_restore_positions();
> >  	    unmetafy_line();
> >  	    /* For vi mode -- reset the beginning-of-insertion pointer   *
> >  	     * to the beginning of the line.  This seems a little silly, *
> 
> If the highlight positions were not properly restored, highlighting might
> reference non-existent parts of the line?  But then I would have expected
> that incorrect highlighting would be a visible symptom in some cases.
> 
> In any event, Peter, I'd appreciate it if you took a closer look at that
> particular change (zle_tricky.c) because it's the only one where I'm not
> certain I was able to repeat the steps that caused the complaint after I
> made the edit.

Looks unimpeachable --- save and restore have to be paired.

But yes, you might think a failure there would be pretty catastrophic,
certainly not just show hard to reproduce problems in hrealloc().

Still, it would be interesting to get a fresh report after this fix.

pws

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Jun T.]

2014/04/06 16:05, Bart Schaefer <schaefer@brasslantern.com> wrote:
> diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
> index 3c7cff9..b916bd6 100644
> (snip)
> 	if (strcmp(zlemetaline, ol)) {
> +	    zle_restore_positions();

It seems this causes the following problem:

$ echo foobar<RET>
$ echo !$<TAB>
$ echo fo

i.e., <TAB> completes '!$' to 'fo', not 'foobar'.
'echo !$<RET>' correctly expands to 'echo foobar', of course.

Maybe zlemetacs/zlematall need not to be restored?
(but the memory allocated by zle_save_positions() must be freed.)

Re: Segfault in hrealloc somewhere between rpromts and syntax highlighting

[Bart Schaefer]

On Apr 13,  2:29am, Jun T. wrote:
} Subject: Re: Segfault in hrealloc somewhere between rpromts and syntax hig
}
} 2014/04/06 16:05, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
} > index 3c7cff9..b916bd6 100644
} > (snip)
} > 	if (strcmp(zlemetaline, ol)) {
} > +	    zle_restore_positions();
} 
} It seems this causes the following problem:
} 
} $ echo foobar<RET>
} $ echo !$<TAB>
} $ echo fo
} 
} Maybe zlemetacs/zlematall need not to be restored?

Hmm, I think it's more complicated than that.  Assuming that the various
positions have been correctly updated when expansion changed the buffer,
then we don't want to restore *anything*, we just want to discard the
most recent saved state.

I'm not sure how that affects the state of any previous call to
zle_save_positions(), but I have to assume that if it's possible for
the stack to be more than one element deep here, then whoever else is
calling zle_store_positions() is planning to do a complete unwind of
whatever doexpandhist() may have done.

On the other hand, compresult.c simply makes sure that restoring the
positions doesn't affect zlemetall, so perhaps this is too much.


diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
index b916bd6..499c4ae 100644
--- a/Src/Zle/zle_tricky.c
+++ b/Src/Zle/zle_tricky.c
@@ -2795,7 +2795,7 @@ doexpandhist(void)
     if (!err) {
 	zlemetacs = excs;
 	if (strcmp(zlemetaline, ol)) {
-	    zle_restore_positions();
+	    zle_free_positions();
 	    unmetafy_line();
 	    /* For vi mode -- reset the beginning-of-insertion pointer   *
 	     * to the beginning of the line.  This seems a little silly, *
diff --git a/Src/Zle/zle_utils.c b/Src/Zle/zle_utils.c
index 9cfa881..1089e27 100644
--- a/Src/Zle/zle_utils.c
+++ b/Src/Zle/zle_utils.c
@@ -710,6 +710,27 @@ zle_restore_positions(void)
 }
 
 /*
+ * Discard positions previously saved, the line has been updated.
+ */
+
+/**/
+mod_export void
+zle_free_positions(void)
+{
+    struct zle_position *oldpos = zle_positions;
+    struct zle_region *oldrhp;
+
+    zle_positions = oldpos->next;
+    oldrhp = oldpos->regions;
+    while (oldrhp) {
+	struct zle_region *nextrhp = oldrhp->next;
+	zfree(oldrhp, sizeof(*oldrhp));
+	oldrhp = nextrhp;
+    }
+    zfree(oldpos, sizeof(*oldpos));
+}
+
+/*
  * Basic utility functions for adding to line or removing from line.
  * At this level the counts supplied are raw character counts, so
  * the calling code must be aware of combining characters where