Very odd behaviour with zsh, maybe corruption bug

Very odd behaviour with zsh, maybe corruption bug

[martin.ebourne]

Hi,

Seeing Bart's suggestion to Oliver Kiddle about the use of
history-beginning-search-backward/up-line-or-history, I decided that was
nearly what I wanted, so decided to fiddle with it until it was exactly
what I wanted. Anyhow, it essentially works except that a condition which
should evaluate to true is sometimes returning false for no reason I can
figure.

I'm using zsh-4.1.0-dev-2, and I can reproduce this with zsh -f. It is
however not completely reproducible in that it appears to involve some kind
of random effect/corruption so it needs a few goes before it happens. I'm
using Zsh on:
SunOS gdd-odybin2 5.6 Generic_105181-28 sun4u sparc
SUNW,Ultra-Enterprise-10000

First extract this into a file:

=== START bizzarre.sh
_up_fn() {
  if [[ $LBUFFER == *$'\n'* ]]
  then
    _searching=''
    zle .up-line-or-history
  else
    _searching=$WIDGET
    zle .history-beginning-search-backward
  fi
}

_down_fn() {
  setopt local_options
  set -x
  if [[ ${LASTWIDGET} == ${_searching} ]]
  then
    _searching=$WIDGET
    zle .history-beginning-search-forward
  else
    if [[ $RBUFFER == *$'\n'* ]]
    then
      _searching=''
      zle .down-line-or-history
    else
      _searching=$WIDGET
      zle .history-beginning-search-forward
    fi
  fi
}

zle -N new-up _up_fn
zle -N new-down _down_fn

bindkey "^[[A" new-up
bindkey "^[[B" new-down
=== END

Then execute as here:

% zsh -f
gdd-odybin2% . ./bizzarre.sh
gdd-odybin2% "abc
dquote> def"
zsh: command not found: abc\ndef
gdd-odybin2%

Now if I type a single " and play around with using the four cursor keys
trying to use the history matching & down, after a couple of minutes I can
be pretty sure it will have gone wrong. (Sometimes I ctrl-c or press enter
and try again - don't know if that matters.) Once it has gone wrong it
appears to continually go wrong until the down function is re-defined
(that's the minimum it seems to need to 'reset' it). Due to the 'set -x'
you'll get a messed up screen display, but just try and ignore that!

The 'gone wrong' is where the following line returns false even though both
variables are the same (you can see because of the 'set -x' I added - this
was only added for debugging, and isn't needed to make it go wrong).
--->  if [[ ${LASTWIDGET} == ${_searching} ]]

Here is an example trace where it has gone wrong (once the bug had been
'activated' I just pressed up followed by down to get this):

gdd-odybin2% +_down_fn:3> [[ new-up == new-up ]]
+_down_fn:8> [[ "abc
def" == *
* ]]
+_down_fn:10> _searching=
+_down_fn:11> zle .down-line-or-history

gdd-odybin2%

As far as I can analyse these variables they are exactly the same, as it
appears in the listing, and this first condition should return true, but is
actually giving false. Please tell me if I'm doing something obviously
stupid, its by far the easist solution. ;)

Cheers,

Martin.




This e-mail message is CONFIDENTIAL and may contain legally privileged
information.  If you are not the intended recipient you should not  read,
copy, distribute, disclose or otherwise use the information in this e-mail.
Please also telephone or fax us immediately and delete the message from
your system.  E-mail may be susceptible to data corruption, interception
and unauthorised amendment, and we do not accept liability for any such
corruption, interception or amendment or the consequences thereof.

Re: Very odd behaviour with zsh, maybe corruption bug

[Bart Schaefer]

On Oct 11, 11:57am, martin.ebourne@arcordia.com wrote:
}
} SunOS gdd-odybin2 5.6 Generic_105181-28 sun4u sparc
} SUNW,Ultra-Enterprise-10000
} 
} gdd-odybin2% +_down_fn:3> [[ new-up == new-up ]]
} +_down_fn:8> [[ "abc
} def" == *
} * ]]
} +_down_fn:10> _searching=
} +_down_fn:11> zle .down-line-or-history

I can't reproduce this on my linux machine, even after ten minutes or so
of fooling with it.

For [[ ]], the trace output is generated before the right-hand-side is
copied and compiled into a pattern for comparison to the left-hand-side,
so it could be the case that memory errors result in a bad comparison;
but they'd be far more likely to produce a "bad pattern" error instead.

This is a long, long, long shot, but:  You don't happen to be using a
debug trap?  (I.e., a TRAPDEBUG function, or `trap ... DEBUG')

Have you tried configuring with --enable-zsh-debug ?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Very odd behaviour with zsh, maybe corruption bug

[Bart Schaefer]

On Oct 11, 11:57am, martin.ebourne@arcordia.com wrote:
}
} _down_fn() {
}   setopt local_options
}   set -x
}   if [[ ${LASTWIDGET} == ${_searching} ]]
}   then
}     _searching=$WIDGET
}     zle .history-beginning-search-forward
}   else
}     if [[ $RBUFFER == *$'\n'* ]]
}     then
}       _searching=''
}       zle .down-line-or-history
}     else
}       _searching=$WIDGET
}       zle .history-beginning-search-forward
}     fi
}   fi
} }
} 
} Here is an example trace where it has gone wrong (once the bug had been
} 'activated' I just pressed up followed by down to get this):
} 
} gdd-odybin2% +_down_fn:3> [[ new-up == new-up ]]
} +_down_fn:8> [[ "abc
} def" == *
} * ]]
} +_down_fn:10> _searching=
} +_down_fn:11> zle .down-line-or-history

I've gotten a bit closer to what's going on here, but I'm still not sure
exactly what's happening.

In evalcond(), there's this fragment:

	    int test, npat = state->pc[1];
	    Patprog pprog = state->prog->pats[npat];

	    if (pprog == dummy_patprog1 || pprog == dummy_patprog2) {
		char *opat;
		int save;

		right = opat = dupstring(ecrawstr(state->prog, state->pc,
						  &htok));
		if (htok)
		    singsub(&right);
		save = (!(state->prog->flags & EF_HEAP) &&
			!strcmp(opat, right) && pprog != dummy_patprog2);

		if (!(pprog = patcompile(right, (save ? PAT_ZDUP : PAT_STATIC),
					 NULL)))
		    zerr("bad pattern: %s", right, 0);
		else if (save)
		    state->prog->pats[npat] = pprog;
	    }

On the *next* call to _down_fn *after* the call which executes
}       _searching=''
}       zle .down-line-or-history

The C code above gets into this state:

(gdb) p opat
$21 = 0x4011b220 ""
(gdb) p right
$22 = 0x4011b230 ""

Thus `save' is true, so we execute `state->prog->pats[npat] = pprog;' and
thereafter the test at _down_fn:3 can never succeed again.  The problem
is that before calling `singsub(&right);', the string is not empty:

(gdb) p ecrawstr(state->prog, state->pc, &htok)
$26 = 0x8140d5f "\205\215_searching\216"

Hence `save' should not have become true, but it did.

This *must* be a bug in prefork() called via singsub(); it must be zeroing
`right' (and thus also zeroing `opat') before copying the string.  But I
just can't seem to catch it happening.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

PATCH: Re: Very odd behaviour with zsh, maybe corruption bug

[Bart Schaefer]

On Oct 12,  5:02am, Bart Schaefer wrote:
}
} This *must* be a bug in prefork() called via singsub(); it must be zeroing
} `right' (and thus also zeroing `opat') before copying the string.  But I
} just can't seem to catch it happening.

Having traced through this more closely, which was rather laborious, I
have come to the conclusion that the bug is right there in evalcond(),
and that the following is the fix.  At least, I can't see how this patch
could possibly cause -incorrect- behavior, though it might result in an
optimization being skipped unnecessarily in some cases.

Sven, please dispute this if you can.

Index: Src/cond.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/Src/cond.c,v
retrieving revision 1.1.1.1
diff -c -r1.1.1.1 cond.c
--- Src/cond.c	2001/04/09 20:14:12	1.1.1.1
+++ Src/cond.c	2001/10/14 21:48:55
@@ -202,7 +202,7 @@
 		char *opat;
 		int save;
 
-		right = opat = dupstring(ecrawstr(state->prog, state->pc,
+		right = dupstring(opat = ecrawstr(state->prog, state->pc,
 						  &htok));
 		if (htok)
 		    singsub(&right);


-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: PATCH: Re: Very odd behaviour with zsh, maybe corruption bug

[Sven Wischnowsky]

Bart Schaefer wrote:

> On Oct 12,  5:02am, Bart Schaefer wrote:
> }
> } This *must* be a bug in prefork() called via singsub(); it must be zeroing
> } `right' (and thus also zeroing `opat') before copying the string.  But I
> } just can't seem to catch it happening.
> 
> Having traced through this more closely, which was rather laborious, I
> have come to the conclusion that the bug is right there in evalcond(),
> and that the following is the fix.  At least, I can't see how this patch
> could possibly cause -incorrect- behavior, though it might result in an
> optimization being skipped unnecessarily in some cases.
> 
> Sven, please dispute this if you can.
> 
> ...
>  
> -		right = opat = dupstring(ecrawstr(state->prog, state->pc,
> +		right = dupstring(opat = ecrawstr(state->prog, state->pc,
>  						  &htok));

This is right. I hadn't realised that paramsubst() (called indirectly
from singsub()) modifies the string it gets even though it gets a
pointer (it's that `*s++ = '\0', I think).

We need a similar fix in execcase(), below is a patch for both of
these, which I will commit to both branches.


Bye
  Sven

Index: Src/cond.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/cond.c,v
retrieving revision 1.2
diff -u -r1.2 cond.c
--- Src/cond.c	2000/04/01 20:49:48	1.2
+++ Src/cond.c	2001/10/15 11:23:26
@@ -202,7 +202,7 @@
 		char *opat;
 		int save;
 
-		right = opat = dupstring(ecrawstr(state->prog, state->pc,
+		right = dupstring(opat = ecrawstr(state->prog, state->pc,
 						  &htok));
 		if (htok)
 		    singsub(&right);
Index: Src/loop.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/loop.c,v
retrieving revision 1.5
diff -u -r1.5 loop.c
--- Src/loop.c	2001/06/25 16:07:52	1.5
+++ Src/loop.c	2001/10/15 11:23:26
@@ -543,7 +543,7 @@
 	if (isset(XTRACE)) {
 	    char *pat2, *opat;
 
-	    opat = pat = ecgetstr(state, EC_DUP, NULL);
+	    pat = dupstring(opat = ecrawstr(state->prog, state->pc, NULL));
 	    singsub(&pat);
 	    save = (!(state->prog->flags & EF_HEAP) &&
 		    !strcmp(pat, opat) && *spprog != dummy_patprog2);
@@ -565,7 +565,7 @@
 		char *opat;
 		int htok = 0;
 
-		opat = pat = dupstring(ecrawstr(state->prog,
+		pat = dupstring(opat = ecrawstr(state->prog,
 						state->pc - 2, &htok));
 		if (htok)
 		    singsub(&pat);

-- 
Sven Wischnowsky                    wischnow@informatik.hu-berlin.de

Re: Very odd behaviour with zsh, maybe corruption bug

[martin.ebourne]

   On Oct 11,  6:08pm, martin.ebourne@arcordia.com wrote:
   } However, using this version I've got a guaranteed sequence which
   } triggers it (maybe also on the last version, but I don't know). I

   That does in fact reproduce it on my machine as well.  You didn't Cc
   your
   message to zsh-workers -- you might want to send the above instructions
   there so somebody else can have a crack at it too.

Actually I've found a much shorter sequence too. As before execute
bizzarre.sh, but afterwards all you need is to press down, up, down and
you'll get this trace:

% ./zsh -f
gdd-odybin2% . ~/bizzarre.sh
gdd-odybin2% +_down_fn:4> [[ accept-line ==  ]]
+_down_fn:9> [[  == *
* ]]
+_down_fn:14> _searching=new-down
+_down_fn:15> zle .history-beginning-search-forward
+_down_fn:4> [[ new-up == new-up ]]
+_down_fn:9> [[ . ~/bizzarre.sh        == *
* ]]
+_down_fn:14> _searching=new-down
+_down_fn:15> zle .history-beginning-search-forward

No need to enter any other lines. Taking a guess, it would seem that the
critical bit that breaks it is the first down, where it tries to
history-beginning-search-forward, but fails with a beep. No guesses as to
why though!

Cheers,

Martin.

Re: Very odd behaviour with zsh, maybe corruption bug

[martin.ebourne]

<forgot to reply to list also last time>



   I can't reproduce this on my linux machine, even after ten minutes or so
   of fooling with it.

   For [[ ]], the trace output is generated before the right-hand-side is
   copied and compiled into a pattern for comparison to the left-hand-side,
   so it could be the case that memory errors result in a bad comparison;
   but they'd be far more likely to produce a "bad pattern" error instead.

I was wondering if perhaps the internal 'compiled' function definition was
getting modified. After all, once the problem has occurred it appears to be
guaranteed to happen until the function is redefined.

   This is a long, long, long shot, but:  You don't happen to be using a
   debug trap?  (I.e., a TRAPDEBUG function, or `trap ... DEBUG')

No, not at all. I can reproduce it with zsh -f, with just the stuff listed
in the previous email.

   Have you tried configuring with --enable-zsh-debug ?

I have now. I used all the ones I could find: --enable-zsh-mem-warning
--enable-zsh-secure-free --enable-zsh-hash-debug --enable-zsh-debug.

Are they supposed to tell me anything directly or do I need to do anything
special - I didn't see anything new? However, using this version I've got a
guaranteed sequence which triggers it (maybe also on the last version, but
I don't know). I follow the sequence exactly as per my previous email, and
after entering the lone " character I perform the following key sequence:
up, right, down, down, enter, up, up, down. This has just worked 4/5 times
in a row on this solaris machine, but doesn't mean it will on Linux!

I don't know the code at all, but if you can suggest a sensible place or
two to break on I'll have a poke at it in dbx.

Cheers,

Martin.
PS. make check core dumps on the V01zmodload test (and did before I built
debug too). Do you know about this or should I send a backtrace/shell
trace? The module loading obviously works normally though...







This e-mail message is CONFIDENTIAL and may contain legally privileged
information.  If you are not the intended recipient you should not  read,
copy, distribute, disclose or otherwise use the information in this e-mail.
Please also telephone or fax us immediately and delete the message from
your system.  E-mail may be susceptible to data corruption, interception
and unauthorised amendment, and we do not accept liability for any such
corruption, interception or amendment or the consequences thereof.