history-incremental-search-backward weird behavior for multiline commands in 5.0.x

history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Augie Fackler]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

I upgraded my work machine from zsh 4.3.17 to zsh 5.0.5 (also tested 5.0.2
first, as that was what was in apt), and now
history-incremental-search-backward doesn't work correctly for multiline
history entries loaded from a history file on disk. As an example:

$ echo foo \
>  bar
foo bar
$ ^R bar
# finds the full command

but then in a new shell:
$ ^R bar
# finds only the word bar, not the full echo command.

This works fine in zsh 4.3.17 and zsh 4.3.11 (I get the same behavior even
if the history entry was loaded from a file).

Am I missing something obvious? setopt hist_lex_words didn't make a
difference, and I tried turning off hist_save_no_dups and
inc_append_history.

Thanks!

Re: history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Bart Schaefer]

On Jul 16,  8:48am, Augie Fackler wrote:
}
} $ echo foo \
} >  bar
} foo bar
} $ ^R bar
} # finds the full command
} 
} but then in a new shell:
} $ ^R bar
} # finds only the word bar, not the full echo command.

Hmm, it looks like history loading may be mis-handling the case of an
INTENTIONAL backslash-newline pair.  This gets stored in the history
file as e.g.

echo foo \\
 bar

This is then re-loaded as the two lines

echo foo \\

(note the backslash is still doubled, though it was not when it was
originally input) and

 bar

which obviously is wrong.  I'm not sure whether the fix is to store
it with three backslashes, or to parse it differently on load, or both.

By contrast, if you had used quotes or a multi-line loop construct, each
line would have ended with a single backslash which is reloaded correctly.

} Am I missing something obvious? setopt hist_lex_words didn't make a
} difference, and I tried turning off hist_save_no_dups and
} inc_append_history.

hist_lex_words would need to be set very early, i.e., before the history
is loaded, in order to make any difference, but in this case it doesn't
matter.

Re: history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Peter Stephenson]

On Wed, 16 Jul 2014 09:10:07 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Hmm, it looks like history loading may be mis-handling the case of an
> INTENTIONAL backslash-newline pair.  This gets stored in the history
> file as e.g.
> 
> echo foo \\
>  bar
> 
> This is then re-loaded as the two lines
> 
> echo foo \\
> 
> (note the backslash is still doubled, though it was not when it was
> originally input) and
> 
>  bar
> 
> which obviously is wrong.  I'm not sure whether the fix is to store
> it with three backslashes, or to parse it differently on load, or both.

I think the load parsing must be broken --- I'm pretty sure it always
used doubled backslashes to signal this and it obviously used to work.

pws

Re: history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Bart Schaefer]

On Jul 16,  5:30pm, Peter Stephenson wrote:
} Subject: Re: history-incremental-search-backward weird behavior for multil
}
} I think the load parsing must be broken --- I'm pretty sure it always
} used doubled backslashes to signal this and it obviously used to work.

This appears to have somethign to do with workers/28332 and 28339, but
I'm not yet sure exactly what.

Re: history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Peter Stephenson]

On Wed, 16 Jul 2014 10:54:17 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jul 16,  5:30pm, Peter Stephenson wrote:
> } Subject: Re: history-incremental-search-backward weird behavior for multil
> }
> } I think the load parsing must be broken --- I'm pretty sure it always
> } used doubled backslashes to signal this and it obviously used to work.
> 
> This appears to have somethign to do with workers/28332 and 28339, but
> I'm not yet sure exactly what.

Certainly if I comment out code in readhistline() as follows, it reads
back in again the same as 4.3.11 (which just happens to be the first old
version that worked).

	else {
	    buf[len - 1] = '\0';
	    if (len > 1 && buf[len - 2] == '\\' /*&&
		(len < 3 || buf[len - 3] != '\\')*/) {
		buf[--len - 1] = '\n';
		if (!feof(in))
		    return readhistline(len, bufp, bufsiz, in);
	    }
	}

However, that's not the whole story --- the extended history is written
out with each segment:

: 1405536461:0;echo foo \\
: 1405536465:0;bar \\
: 1405536465:0;rod

which can't be right because it reads back as

echo foo \          
: 1405536465:0;bar \
: 1405536465:0;rod

in both the latest code and 4.3.11.  The latter saves without the
extended history,

: 1405536927:0;echo foo1 \\
bar1 \\
rod1

pws

Re: history-incremental-search-backward weird behavior for multiline commands in 5.0.x

[Augie Fackler]

Peter Stephenson <p.w.stephenson <at> ntlworld.com> writes:

> 
> On Wed, 16 Jul 2014 10:54:17 -0700
> Bart Schaefer <schaefer <at> brasslantern.com> wrote:
> > On Jul 16,  5:30pm, Peter Stephenson wrote:
> > } Subject: Re: history-incremental-search-backward weird behavior for 
multil
> > }
> > } I think the load parsing must be broken --- I'm pretty sure it always
> > } used doubled backslashes to signal this and it obviously used to work.
> > 
> > This appears to have somethign to do with workers/28332 and 28339, but
> > I'm not yet sure exactly what.
> 
> Certainly if I comment out code in readhistline() as follows, it reads
> back in again the same as 4.3.11 (which just happens to be the first old
> version that worked).
> 
> 	else {
> 	    buf[len - 1] = '\0';
> 	    if (len > 1 && buf[len - 2] == '\\' /*&&
> 		(len < 3 || buf[len - 3] != '\\')*/) {
> 		buf[--len - 1] = '\n';
> 		if (!feof(in))
> 		    return readhistline(len, bufp, bufsiz, in);
> 	    }
> 	}
> 
> However, that's not the whole story --- the extended history is written
> out with each segment:
> 
> : 1405536461:0;echo foo \\
> : 1405536465:0;bar \\
> : 1405536465:0;rod
> 
> which can't be right because it reads back as
> 
> echo foo \          
> : 1405536465:0;bar \
> : 1405536465:0;rod
> 
> in both the latest code and 4.3.11.  The latter saves without the
> extended history,
> 
> : 1405536927:0;echo foo1 \\
> bar1 \\
> rod1

I just posted a patch for this and then saw the thread (not subscribed, so 
hopefully gmane does the right thing on this response).

I came up with the same fix yesterday and have been using it for a day - 
with the patch applied, zsh 5.0.x will write out history the same way 4.3.11 
did. It *won't* go back and fix up mangled history files that zsh 5 trashed, 
so you get the extended history droppings in the multiline entries. I think 
you could fix a corrupted history file by turning off extended history, 
loading it, closing zsh, and then turning extended history back on (once all 
your zsh 5 installations were fixed, naturally). I have not tested that 
theory since it's trivial enough to load the command up in emacs and use 
kill-rectangle on it.

> 
> pws
> 
>