zsh history bug ?

zsh history bug ?

[Bernhard Tittelbach]

Hi,

for a long time now I've had a strange issue with zsh,
the source of which I haven't been able to track down.

It occurs on several of my computers and thus across several zsh
versions as well.
It probably has something to do with my zsh config, however very few
computers / users using it actually experience this problem.
It is very hard to reproduce the error as it happens only sporadically.
Were it not happing on several computers, I would put it down to memory
or hard disk errors.

This is what's happening:

# I type a command like the following:
% dtc -E  -I dts test.dts -O dtb -o test.dtb <enter>

# Then I get the last command from the history by using <arrow up>
# but strangley some chars seem to have moved and the CL looks like this
% dtc -E all -I ts  tst.dtss -O tb  -o est.dtbb<enter>

# again <arrow up>
% dtc -Wall -I ts tt.dtsss -O tb -o t.dtbbbb<enter>

# and again <arrow up> makes it even worse
% dtc -f -I dts tt.dtsss -O tb -o t.dtbbbb<enter>

# after entering another command, everything is fine again.

version: zsh 5.0.2 (x86_64-pc-linux-gnu)
config: https://www.tittelbach.at/zsh/.zshrc
	https://www.tittelbach.at/zsh/.zshrc.local
<arrow up> is bound to: up-line-or-history

any suggestions ?
cheers,
Bernhard

Re: zsh history bug ?

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

On Nov 4, 2014 3:27 PM, "Bernhard Tittelbach" <xro@realraum.at> wrote:

> # Then I get the last command from the history by using <arrow up>
> # but strangley some chars seem to have moved and the CL looks like this
> % dtc -E all -I ts  tst.dtss -O tb  -o est.dtbb<enter>

If possible, please try this in a newer version of the shell.  It sounds
very much like something that was fixed within the past several months.
(The current zsh version is 5.0.7.)

Re: zsh history bug ?

[Bernhard Tittelbach]

Am 2014-11-05 um 01:12 schrieb Bart Schaefer:
> On Nov 4, 2014 3:27 PM, "Bernhard Tittelbach" <xro@realraum.at> wrote:
> 
>> # Then I get the last command from the history by using <arrow up>
>> # but strangley some chars seem to have moved and the CL looks like this
>> % dtc -E all -I ts  tst.dtss -O tb  -o est.dtbb<enter>
> 
> If possible, please try this in a newer version of the shell.  It sounds
> very much like something that was fixed within the past several months.
> (The current zsh version is 5.0.7.)
> 

I've been working a few days with 5.0.7 now and can report the following:

1) I seem to encounter the bug less often
2) The bug, or a variant thereof is still present

i.e.:

% mkdir -p libs/jquery
% cd <Alt>.
resulted in
% cd lib/jquuery

When I previously encountered the bug, I believe only the lookup from
history was garbled, i.e. the contents of .zsh_history were fine.
I'll have to confirm this on a different system though.

Now the .zsh_history definitely holds a garbled version of the executed
command line:

% ls
libs/
% ls libs/
jquery/
% tail  ~/.zsh_history | grep mkdir
: 1415928181:0;mkdir -p lib/jquuery

Re: zsh history bug ?

[Mikael Magnusson]

On Fri, Nov 14, 2014 at 2:34 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
> Am 2014-11-05 um 01:12 schrieb Bart Schaefer:
>> On Nov 4, 2014 3:27 PM, "Bernhard Tittelbach" <xro@realraum.at> wrote:
>>
>>> # Then I get the last command from the history by using <arrow up>
>>> # but strangley some chars seem to have moved and the CL looks like this
>>> % dtc -E all -I ts  tst.dtss -O tb  -o est.dtbb<enter>
>>
>> If possible, please try this in a newer version of the shell.  It sounds
>> very much like something that was fixed within the past several months.
>> (The current zsh version is 5.0.7.)
>>
>
> I've been working a few days with 5.0.7 now and can report the following:
>
> 1) I seem to encounter the bug less often
> 2) The bug, or a variant thereof is still present
>
> i.e.:
>
> % mkdir -p libs/jquery
> % cd <Alt>.
> resulted in
> % cd lib/jquuery
>
> When I previously encountered the bug, I believe only the lookup from
> history was garbled, i.e. the contents of .zsh_history were fine.
> I'll have to confirm this on a different system though.
>
> Now the .zsh_history definitely holds a garbled version of the executed
> command line:
>
> % ls
> libs/
> % ls libs/
> jquery/
> % tail  ~/.zsh_history | grep mkdir
> : 1415928181:0;mkdir -p lib/jquuery

Check if setopt nohistreduceblanks stops the problem.

-- 
Mikael Magnusson

Re: zsh history bug ?

[Bernhard Tittelbach]

Am 2014-11-14 um 02:51 schrieb Mikael Magnusson:
> On Fri, Nov 14, 2014 at 2:34 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
>> Am 2014-11-05 um 01:12 schrieb Bart Schaefer:
>>> On Nov 4, 2014 3:27 PM, "Bernhard Tittelbach" <xro@realraum.at> wrote:
>>>
>>>> # Then I get the last command from the history by using <arrow up>
>>>> # but strangley some chars seem to have moved and the CL looks like this
>>>> % dtc -E all -I ts  tst.dtss -O tb  -o est.dtbb<enter>
>>> If possible, please try this in a newer version of the shell.  It sounds
>>> very much like something that was fixed within the past several months.
>>> (The current zsh version is 5.0.7.)
>>>
>> I've been working a few days with 5.0.7 now and can report the following:
>>
>> 1) I seem to encounter the bug less often
>> 2) The bug, or a variant thereof is still present
>>
>> i.e.:
>>
>> % mkdir -p libs/jquery
>> % cd <Alt>.
>> resulted in
>> % cd lib/jquuery
>>
>> When I previously encountered the bug, I believe only the lookup from
>> history was garbled, i.e. the contents of .zsh_history were fine.
>> I'll have to confirm this on a different system though.
>>
>> Now the .zsh_history definitely holds a garbled version of the executed
>> command line:
>>
>> % ls
>> libs/
>> % ls libs/
>> jquery/
>> % tail  ~/.zsh_history | grep mkdir
>> : 1415928181:0;mkdir -p lib/jquuery
> Check if setopt nohistreduceblanks stops the problem.
>
Alright, I've not encountered the bug for quite some time now,
your suggestion worked!

Thanks a lot !!

Re: zsh history bug ?

[Bart Schaefer]

On Dec 10,  6:23am, Bernhard Tittelbach wrote:
} Subject: Re: zsh history bug ?
}
} Am 2014-11-14 um 02:51 schrieb Mikael Magnusson:
} > On Fri, Nov 14, 2014 at 2:34 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
} >>
} >> I've been working a few days with 5.0.7 now and can report the following:
} >>
} >> 1) I seem to encounter the bug less often
} >> 2) The bug, or a variant thereof is still present
} >>
} >> i.e.:
} >>
} >> % mkdir -p libs/jquery
} >> % cd <Alt>.
} >> resulted in
} >> % cd lib/jquuery

I'm not able to reproduce this on MacOS (the platform where the bug
originally was reported, IIRC).

Here was the patch:

2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>

        * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
        regions.

I've re-checked Src/hist.c and there are no other cases where memcpy() is
being used except with a newly-allocated buffer as the destination (so, no
overlapping regions).

} >> When I previously encountered the bug, I believe only the lookup from
} >> history was garbled, i.e. the contents of .zsh_history were fine.
} >> I'll have to confirm this on a different system though.

Are you sharing history via a network-mounted home directory from 
multiple hosts?

} >> Now the .zsh_history definitely holds a garbled version of the executed
} >> command line:
} >>
} >> % ls
} >> libs/
} >> % ls libs/
} >> jquery/
} >> % tail  ~/.zsh_history | grep mkdir
} >> : 1415928181:0;mkdir -p lib/jquuery
} > Check if setopt nohistreduceblanks stops the problem.
} >
} Alright, I've not encountered the bug for quite some time now,
} your suggestion worked!

Can you give us some additional details of the OS and hardware where you
are seeing this error?  The remaining possibility is that your build of
zsh does not define HAVE_MEMMOVE *and* the implementation of its
replacement (which might be bcopy(), or might be a macro in the system
headers) does not properly handle overlapping pointers.

Re: zsh history bug ?

[Bernhard Tittelbach]

Am 2014-12-10 um 18:19 schrieb Bart Schaefer:
> On Dec 10,  6:23am, Bernhard Tittelbach wrote:
> } Subject: Re: zsh history bug ?
> }
> } Am 2014-11-14 um 02:51 schrieb Mikael Magnusson:
> } > On Fri, Nov 14, 2014 at 2:34 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
> } >>
> } >> I've been working a few days with 5.0.7 now and can report the following:
> } >>
> } >> 1) I seem to encounter the bug less often
> } >> 2) The bug, or a variant thereof is still present
> } >>
> } >> i.e.:
> } >>
> } >> % mkdir -p libs/jquery
> } >> % cd <Alt>.
> } >> resulted in
> } >> % cd lib/jquuery
>
> I'm not able to reproduce this on MacOS (the platform where the bug
> originally was reported, IIRC).
>
> Here was the patch:
>
> 2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>
>
>         * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
>         regions.
>
> I've re-checked Src/hist.c and there are no other cases where memcpy() is
> being used except with a newly-allocated buffer as the destination (so, no
> overlapping regions).
>
> } >> When I previously encountered the bug, I believe only the lookup from
> } >> history was garbled, i.e. the contents of .zsh_history were fine.
> } >> I'll have to confirm this on a different system though.
>
> Are you sharing history via a network-mounted home directory from 
> multiple hosts?
The home directories are on local drives only.
I don't share history between computers at all.

>
> } >> Now the .zsh_history definitely holds a garbled version of the executed
> } >> command line:
> } >>
> } >> % ls
> } >> libs/
> } >> % ls libs/
> } >> jquery/
> } >> % tail  ~/.zsh_history | grep mkdir
> } >> : 1415928181:0;mkdir -p lib/jquuery
> } > Check if setopt nohistreduceblanks stops the problem.
> } >
> } Alright, I've not encountered the bug for quite some time now,
> } your suggestion worked!
>
> Can you give us some additional details of the OS and hardware where you
> are seeing this error?  The remaining possibility is that your build of
> zsh does not define HAVE_MEMMOVE *and* the implementation of its
> replacement (which might be bcopy(), or might be a macro in the system
> headers) does not properly handle overlapping pointers.
sure..
There are two running Ubuntu x86_64. One CoreDuo Laptop, one Core i5
Desktop.

I also have an old Pentium4 machine running Gentoo i686 and an Eeebook
running Ubuntu i686 with the same zsh config.
I don't think I've ever encountered the bug on these i686 machines, but
then I don't use these very often.
Thus It's hard to tell if the bug is present there, since it only
appears sporadically and is hard to reproduce.

Originally, on the x86_64 machine, I was running zsh 5.0.3 from the
Ubuntu Package,
but per your suggestion I compiled 5.0.7 from source last month. The bug
persisted until
Mikael suggested setting "nohistreduceblanks" i.e. disabling
"histreduceblanks"

Since then I've not encountered the bug anymore or at least the bug has
become so seldom
I've not encountered it yet again within the last month. Previously It
appeared about once in a day of work on the shell.

Re: zsh history bug ?

[Mikael Magnusson]

On Fri, Dec 12, 2014 at 10:48 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
> Am 2014-12-10 um 18:19 schrieb Bart Schaefer:
>> On Dec 10,  6:23am, Bernhard Tittelbach wrote:
>>
>> I'm not able to reproduce this on MacOS (the platform where the bug
>> originally was reported, IIRC).
>>
>> Here was the patch:
>>
>> 2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>
>>
>>         * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
>>         regions.
>>
>> I've re-checked Src/hist.c and there are no other cases where memcpy() is
>> being used except with a newly-allocated buffer as the destination (so, no
>> overlapping regions).
>>
>> } >> When I previously encountered the bug, I believe only the lookup from
>> } >> history was garbled, i.e. the contents of .zsh_history were fine.
>> } >> I'll have to confirm this on a different system though.
>>
>> Are you sharing history via a network-mounted home directory from
>> multiple hosts?
> The home directories are on local drives only.
> I don't share history between computers at all.
>>[...]
> Originally, on the x86_64 machine, I was running zsh 5.0.3 from the
> Ubuntu Package,
> but per your suggestion I compiled 5.0.7 from source last month. The bug
> persisted until
> Mikael suggested setting "nohistreduceblanks" i.e. disabling
> "histreduceblanks"
>
> Since then I've not encountered the bug anymore or at least the bug has
> become so seldom
> I've not encountered it yet again within the last month. Previously It
> appeared about once in a day of work on the shell.

Are you totally sure there were no more 5.0.3 instances running at
that time? Or are you saying it works with histreduceblanks enabled
again?

-- 
Mikael Magnusson

Re: zsh history bug ?

[Bernhard Tittelbach]

Am 2014-12-12 um 11:01 schrieb Mikael Magnusson:
> On Fri, Dec 12, 2014 at 10:48 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
>> Am 2014-12-10 um 18:19 schrieb Bart Schaefer:
>>> On Dec 10,  6:23am, Bernhard Tittelbach wrote:
>>>
>>> I'm not able to reproduce this on MacOS (the platform where the bug
>>> originally was reported, IIRC).
>>>
>>> Here was the patch:
>>>
>>> 2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>
>>>
>>>         * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
>>>         regions.
>>>
>>> I've re-checked Src/hist.c and there are no other cases where memcpy() is
>>> being used except with a newly-allocated buffer as the destination (so, no
>>> overlapping regions).
>>>
>>> } >> When I previously encountered the bug, I believe only the lookup from
>>> } >> history was garbled, i.e. the contents of .zsh_history were fine.
>>> } >> I'll have to confirm this on a different system though.
>>>
>>> Are you sharing history via a network-mounted home directory from
>>> multiple hosts?
>> The home directories are on local drives only.
>> I don't share history between computers at all.
>>> [...]
>> Originally, on the x86_64 machine, I was running zsh 5.0.3 from the
>> Ubuntu Package,
>> but per your suggestion I compiled 5.0.7 from source last month. The bug
>> persisted until
>> Mikael suggested setting "nohistreduceblanks" i.e. disabling
>> "histreduceblanks"
>>
>> Since then I've not encountered the bug anymore or at least the bug has
>> become so seldom
>> I've not encountered it yet again within the last month. Previously It
>> appeared about once in a day of work on the shell.
> Are you totally sure there were no more 5.0.3 instances running at
> that time? Or are you saying it works with histreduceblanks enabled
> again?
>

I'm saying If I enable histreduceblanks, the bug is present.
Regardles of the zsh version. Both 5.0.3 and 5.0.7 are affected.

Re: zsh history bug ?

[Mikael Magnusson]

On Fri, Dec 12, 2014 at 11:11 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
> Am 2014-12-12 um 11:01 schrieb Mikael Magnusson:
>> On Fri, Dec 12, 2014 at 10:48 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
>>> Am 2014-12-10 um 18:19 schrieb Bart Schaefer:
>>>> On Dec 10,  6:23am, Bernhard Tittelbach wrote:
>>>>
>>>> I'm not able to reproduce this on MacOS (the platform where the bug
>>>> originally was reported, IIRC).
>>>>
>>>> Here was the patch:
>>>>
>>>> 2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>
>>>>
>>>>         * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
>>>>         regions.
>>>>
>>>> I've re-checked Src/hist.c and there are no other cases where memcpy() is
>>>> being used except with a newly-allocated buffer as the destination (so, no
>>>> overlapping regions).
>>>>
>>>> } >> When I previously encountered the bug, I believe only the lookup from
>>>> } >> history was garbled, i.e. the contents of .zsh_history were fine.
>>>> } >> I'll have to confirm this on a different system though.
>>>>
>>>> Are you sharing history via a network-mounted home directory from
>>>> multiple hosts?
>>> The home directories are on local drives only.
>>> I don't share history between computers at all.
>>>> [...]
>>> Originally, on the x86_64 machine, I was running zsh 5.0.3 from the
>>> Ubuntu Package,
>>> but per your suggestion I compiled 5.0.7 from source last month. The bug
>>> persisted until
>>> Mikael suggested setting "nohistreduceblanks" i.e. disabling
>>> "histreduceblanks"
>>>
>>> Since then I've not encountered the bug anymore or at least the bug has
>>> become so seldom
>>> I've not encountered it yet again within the last month. Previously It
>>> appeared about once in a day of work on the shell.
>> Are you totally sure there were no more 5.0.3 instances running at
>> that time? Or are you saying it works with histreduceblanks enabled
>> again?
>>
>
> I'm saying If I enable histreduceblanks, the bug is present.
> Regardles of the zsh version. Both 5.0.3 and 5.0.7 are affected.

And just to make sure, if you run
echo $ZSH_VERSION
in a shell, it displays 5.0.7? (We had an instance recently in #zsh
where someone upgraded, but their login shell was actually /bin/zsh5,
and they used zsh --version to check the version).

-- 
Mikael Magnusson

Re: zsh history bug ?

[Bernhard Tittelbach]

On 2014-12-12 13:24, Mikael Magnusson wrote:
> On Fri, Dec 12, 2014 at 11:11 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
>> Am 2014-12-12 um 11:01 schrieb Mikael Magnusson:
>>> On Fri, Dec 12, 2014 at 10:48 AM, Bernhard Tittelbach <xro@realraum.at> wrote:
>>>> Am 2014-12-10 um 18:19 schrieb Bart Schaefer:
>>>>> On Dec 10,  6:23am, Bernhard Tittelbach wrote:
>>>>>
>>>>> I'm not able to reproduce this on MacOS (the platform where the bug
>>>>> originally was reported, IIRC).
>>>>>
>>>>> Here was the patch:
>>>>>
>>>>> 2013-09-26  Barton E. Schaefer  <schaefer@zsh.org>
>>>>>
>>>>>         * 31770: Src/hist.c: memmove() instead of memcpy() for overlapping
>>>>>         regions.
>>>>>
>>>>> I've re-checked Src/hist.c and there are no other cases where memcpy() is
>>>>> being used except with a newly-allocated buffer as the destination (so, no
>>>>> overlapping regions).
>>>>>
>>>>> } >> When I previously encountered the bug, I believe only the lookup from
>>>>> } >> history was garbled, i.e. the contents of .zsh_history were fine.
>>>>> } >> I'll have to confirm this on a different system though.
>>>>>
>>>>> Are you sharing history via a network-mounted home directory from
>>>>> multiple hosts?
>>>> The home directories are on local drives only.
>>>> I don't share history between computers at all.
>>>>> [...]
>>>> Originally, on the x86_64 machine, I was running zsh 5.0.3 from the
>>>> Ubuntu Package,
>>>> but per your suggestion I compiled 5.0.7 from source last month. The bug
>>>> persisted until
>>>> Mikael suggested setting "nohistreduceblanks" i.e. disabling
>>>> "histreduceblanks"
>>>>
>>>> Since then I've not encountered the bug anymore or at least the bug has
>>>> become so seldom
>>>> I've not encountered it yet again within the last month. Previously It
>>>> appeared about once in a day of work on the shell.
>>> Are you totally sure there were no more 5.0.3 instances running at
>>> that time? Or are you saying it works with histreduceblanks enabled
>>> again?
>>>
>>
>> I'm saying If I enable histreduceblanks, the bug is present.
>> Regardles of the zsh version. Both 5.0.3 and 5.0.7 are affected.
> 
> And just to make sure, if you run
> echo $ZSH_VERSION
> in a shell, it displays 5.0.7? (We had an instance recently in #zsh
> where someone upgraded, but their login shell was actually /bin/zsh5,
> and they used zsh --version to check the version).
> 

Well caught !!

indeed, $ZSH_VERSION was different from zsh --version. My mistake!

I installed zsh 5.0.7 in /usr/bin which comes first in $path,
while zsh 5.0.2 was still in /bin/ where passwd pointed to.

I fixed this now, re-enabled histreduceblanks and since then I haven't
encountered the error again. (Though I haven't tested quite long enough
to really be sure.)

Sorry for the trouble. Thanks for your help!


-- 
Remarkable Claims Require Remarkable Proof - Carl Sagan