Issue with histreduceblanks

Issue with histreduceblanks

[Raghavendra D Prabhu]

Hi

I have been noticing an issue with mangling of history (randomly) from a few 
weeks, today I decided to check which option may have been 
causing that. It seems to be histreduceblanks

=================
zsh -f
Archie% setopt histreduceblanks
Archie% Only two kinds of witnesses exist.  The first live in a neighborhood where
zsh: command not found: Only
Archie% Only two kinds of witnesses exist. he  fisst lvee in a neighorrhood werre
Archie% zsh --version
zsh 5.0.2 (x86_64-unknown-linux-gnu)
Archie% echo $TERM
xterm-256color
Archie%
===============================

As you can see when that line is recalled later it is mangled (I 
chose that line since it is long enough and was able to 
reproduce).

I got this thread - http://www.zsh.org/mla/workers/1997/msg00542.html
from search, but it looks like it turned out to be an issue with
terminal capabilities.

I have had this option (in ~/.zshrc) for a while though, but I started seeing 
this issue only recently.

Has anyone else encountered this lately?

I use rxvt-unicode normally and noticed the issue on that first. 
However, as you can see above, I can reproduce this with xterm as 
well.



Regards,
-- 
Raghavendra Prabhu

Re: Issue with histreduceblanks

[Peter Stephenson]

On Thu, 19 Sep 2013 00:20:58 +0530
Raghavendra D Prabhu <raghu.prabhu13@gmail.com> wrote:
> I have been noticing an issue with mangling of history (randomly) from a few 
> weeks, today I decided to check which option may have been 
> causing that. It seems to be histreduceblanks
>...
> I got this thread - http://www.zsh.org/mla/workers/1997/msg00542.html
> from search, but it looks like it turned out to be an issue with
> terminal capabilities.
>...
> I use rxvt-unicode normally and noticed the issue on that first. 
> However, as you can see above, I can reproduce this with xterm as 
> well.

It's probably worth making sure whether this is a shell issue or a
terminal issue.

Recall the line, then (regardless of it being mangled) go to the
start of it and enter "print ", then hit return.  If it's a terminal
issue the line you see will be correct.  It seems a bit unlikely that
that's the problem if it's associated with the option, though.

Are there any non-ASCII characters on the line that might be confusing
the algorithm (the option pre-dates handling of multibyte characters)?

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Issue with histreduceblanks

[Bart Schaefer]

On Sep 18,  9:12pm, Peter Stephenson wrote:
}
} Are there any non-ASCII characters on the line that might be confusing
} the algorithm (the option pre-dates handling of multibyte characters)?

If that were the case, I would expect history ops that reference words by
position would also be confused.  E.g., something like

zsh% print Only two kinds of witnesses exist.  The first live in a neighborhood
where !#:7 !#:8 !#:9

would likely substitute the wrong substrings for !#:7 etc., because the
same array of word positions is used to condense blanks.

Re: Issue with histreduceblanks

[Raghavendra Prabhu]

On Thu, Sep 19, 2013 at 1:42 AM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> On Thu, 19 Sep 2013 00:20:58 +0530
> Raghavendra D Prabhu <raghu.prabhu13@gmail.com> wrote:
>> I have been noticing an issue with mangling of history (randomly) from a few
>> weeks, today I decided to check which option may have been
>> causing that. It seems to be histreduceblanks
>>...
>> I got this thread - http://www.zsh.org/mla/workers/1997/msg00542.html
>> from search, but it looks like it turned out to be an issue with
>> terminal capabilities.
>>...
>> I use rxvt-unicode normally and noticed the issue on that first.
>> However, as you can see above, I can reproduce this with xterm as
>> well.
>
> It's probably worth making sure whether this is a shell issue or a
> terminal issue.
>
> Recall the line, then (regardless of it being mangled) go to the
> start of it and enter "print ", then hit return.  If it's a terminal
> issue the line you see will be correct.  It seems a bit unlikely that
> that's the problem if it's associated with the option, though.
>
> Are there any non-ASCII characters on the line that might be confusing
> the algorithm (the option pre-dates handling of multibyte characters)?
>
> --
> Peter Stephenson <p.w.stephenson@ntlworld.com>
> Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Hi


Sorry for the delay in replying (it had ended in my Junk due to
Precedence headers).

a) I had also noticed that it writes to the history file mangled.

==================================================
zsh -f
Archie% setopt appendhistory sharehistory histreduceblanks
Archie% HISTFILE=/tmp/.zsh_history
Archie% HISTSIZE=5000
Archie% SAVEHIST=5000
Archie% cat /tmp/.zsh_history
setopt appendhistory
HISTFILE=/tmp/.zsh_history
HISTSIZE=5000
SAVEHIST=5000
true
cat /tmp/.zsh_history
setopt rcs
setopt sharehistory
true
cat /tmp/.zsh_history
: 1380136272:0;setopt appendhistory sharehistory histreduceblanks
: 1380136276:0;HISTFILE=/tmp/.zsh_history
: 1380136279:0;HISTSIZE=5000
: 1380136282:0;SAVEHIST=5000
: 1380136285:0;cat /tmp/.zsh_history
Archie% Only two kinds of witnesses exist.  The first live in a
neighborhood where
zsh: command not found: Only
Archie% Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre
Archie% cat /tmp/.zsh_history
setopt appendhistory
HISTFILE=/tmp/.zsh_history
HISTSIZE=5000
SAVEHIST=5000
true
cat /tmp/.zsh_history
setopt rcs
setopt sharehistory
true
cat /tmp/.zsh_history
: 1380136272:0;setopt appendhistory sharehistory histreduceblanks
: 1380136276:0;HISTFILE=/tmp/.zsh_history
: 1380136279:0;HISTSIZE=5000
: 1380136282:0;SAVEHIST=5000
: 1380136285:0;cat /tmp/.zsh_history
: 1380136298:0;Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre
: 1380136309:0;cat /tmp/.zsh_history



b)  Also tested with print as requested

============================================
zsh -f
Archie% setopt histreduceblanks
Archie% Only two kinds of witnesses exist.  The first live in a
neighborhood where
zsh: command not found: Only
Archie% Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre
Archie% print Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre
Only two kinds of witnesses exist. he fisst lvee in a neighorrhood werre
Archie%
==================================================


So, is no one else facing this? In that case I wonder if this is an
issue with libraries - glibc etc.


pacman -Q zsh glibc
zsh 5.0.2-4
glibc 2.18-5

Re: Issue with histreduceblanks

[Bart Schaefer]

On Sep 26, 12:52am, Raghavendra Prabhu wrote:
}
} On Thu, Sep 19, 2013 at 1:42 AM, Peter Stephenson
} <p.w.stephenson@ntlworld.com> wrote:
} >
} > Are there any non-ASCII characters on the line that might be confusing
} > the algorithm (the option pre-dates handling of multibyte characters)?
} 
} Sorry for the delay in replying (it had ended in my Junk due to
} Precedence headers).

You didn't really answer Peter's question.  Also I think something is
being lost in the cut-and-paste-and-email process, because this ...

} Archie% Only two kinds of witnesses exist.  The first live in a
} neighborhood where
} zsh: command not found: Only

... obviously has a newline where there must not have been one in your
original string.  To have much hope of debugging this, we need a copy
of EXACTLY what you typed, including what might be characters that
"look like" spaces but really are not.

} b)  Also tested with print as requested

Did you see/try this?

> If that were the case, I would expect history ops that reference words by
> position would also be confused.  E.g., something like
> 
> zsh% print Only two kinds of witnesses exist.  The first live in a neighborhood where !#:7 !#:8 !#:9
> 
> would likely substitute the wrong substrings for !#:7 etc., because the
> same array of word positions is used to condense blanks.

Re: Issue with histreduceblanks

[Raghavendra Prabhu]

On Thu, Sep 26, 2013 at 10:42 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Sep 26, 12:52am, Raghavendra Prabhu wrote:
> }
> } On Thu, Sep 19, 2013 at 1:42 AM, Peter Stephenson
> } <p.w.stephenson@ntlworld.com> wrote:
> } >
> } > Are there any non-ASCII characters on the line that might be confusing
> } > the algorithm (the option pre-dates handling of multibyte characters)?
> }
> } Sorry for the delay in replying (it had ended in my Junk due to
> } Precedence headers).
>
> You didn't really answer Peter's question.  Also I think something is
> being lost in the cut-and-paste-and-email process, because this ..

Ah, sorry for not replying on that earlier, there are no non-ASCII
characters there, the line is just  "Only two kinds of witnesses
exist.  The first live in a neighborhood where".
.
>
> } Archie% Only two kinds of witnesses exist.  The first live in a
> } neighborhood where
> } zsh: command not found: Only
>
> ... obviously has a newline where there must not have been one in your
> original string.

There is no newline there, it was just that the line was wrapped in the mail.

>To have much hope of debugging this, we need a copy
> of EXACTLY what you typed, including what might be characters that
> "look like" spaces but really are not.
>
> } b)  Also tested with print as requested
>
> Did you see/try this?
Yes.

Tested with different combinations:

==================
zsh -f
Archie% setopt histreduceblanks
Archie% Only two kinds of witnesses exist.  The first live in a
neighborhood where
zsh: command not found: Only
Archie% Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre
Archie% print Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre !#:7 !#:8 !#:9
print Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood werre he fisst lvee
Only two kinds of witnesses exist. he fisst lvee in a neighorrhood
werre he fisst lvee
Archie% print Only two kinds of witnesses exist. he fisst leee in a
neighrrrhood werre he fisst veee
Archie% print Only two kinds of witnesses exist.  The first live in a
neighborhood where !#:7 !#:8 !#:9
print Only two kinds of witnesses exist.  The first live in a
neighborhood where The first live
Only two kinds of witnesses exist. The first live in a neighborhood
where The first live
Archie% print Only two kinds of witnesses exist. he  fisst lvee in a
neighorrhood whrre he  fisst ivee
Archie%
========================


>
>> If that were the case, I would expect history ops that reference words by
>> position would also be confused.  E.g., something like
>>
>> zsh% print Only two kinds of witnesses exist.  The first live in a neighborhood where !#:7 !#:8 !#:9
>>
>> would likely substitute the wrong substrings for !#:7 etc., because the
>> same array of word positions is used to condense blanks.
>

Re: Issue with histreduceblanks

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

On Thu, Sep 26, 2013 at 10:47 AM, Raghavendra Prabhu <
raghu.prabhu13@gmail.com> wrote:

> On Thu, Sep 26, 2013 at 10:42 PM, Bart Schaefer
> <schaefer@brasslantern.com> wrote:> Did you see/try this?
> Yes.
>
> Tested with different combinations:


> ==================
> zsh -f
> Archie% print Only two kinds of witnesses exist.  The first live in a
> neighborhood where !#:7 !#:8 !#:9
> print Only two kinds of witnesses exist.  The first live in a
> neighborhood where The first live
> Only two kinds of witnesses exist. The first live in a neighborhood
> where The first live
>

OK, this is a clue, of sorts.  When you reference !#:7, the parser first
builds an array of the starting and ending positions of each of the words
on the command line, and then pulls out the characters between the seventh
start and seventh end.  So the fact that !#:7 !#:8 !#:9 worked correctly
means it's not having trouble locating the beginning and end of words.


> Archie% print Only two kinds of witnesses exist. he  fisst lvee in a
> neighorrhood whrre he  fisst ivee
>

And here we see that e.g. "live" is sometimes crunched into "lvee" but
other times into "ivee".  Which points to a problem with memcpy() being
handed overlapping regions when doing the actual blank reduction.   This
might be a compiler optimization issue; in any case I think memcpy() is not
guaranteed to work with overlapping regions and histreduceblanks() ought to
be using memmove() instead.  I seem to recall us having to fix this in a
different part of the shell some while ago.