Re: [BUG] complist interactive mode overwrites command line

Re: [BUG] complist interactive mode overwrites command line

[Andrea Manenti]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

On Wed, 4 Aug 2021 19:30:10 +0300 Marlon Richert <marlon.richert@xxxxxxxxx> wrote:
> % zmodload zsh/complist
> % bindkey '^I' menu-select
> % MENUMODE=interactive
> % touch test{1,2}
> % : ; foobar
>     ^ 1. Type the above line in its entirety.
>       2. Place the cursor before the ;
>       3. Press Tab.
>       4. Press Enter.
> % : test1 bar
>     ^ Completion is written over existing buffer contents.

I am also experiencing this bug, I'm running zsh 5.9.

I hope that someone else is also interested in solving this because, unfortunately,
I lack the expertise to have a go at it.

If it can be helpful, this is my .zshrc configuration:

```
autoload -Uz compinit
compinit

setopt noautomenu
setopt globcomplete
bindkey '^i' complete-word
setopt nolisttypes

zstyle ':completion:*' list-colors ${(s.:.)LS_COLORS}
zstyle ':completion:*' menu yes select interactive
```

[-- Attachment #2: Type: text/html, Size: 3152 bytes --]

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Fri, Jul 15, 2022 at 6:45 AM Andrea Manenti <andrea.manenti@yahoo.com> wrote:
>
> On Wed, 4 Aug 2021 19:30:10 +0300 Marlon Richert <marlon.richert@xxxxxxxxx> wrote:
> > % zmodload zsh/complist
> > % bindkey '^I' menu-select

I don't know if this is also a new thing, but menu-selection also
makes the display pretty confused when stepping through with the arrow
keys if some of the entries are longer than will fit on the command
line without wrapping at the right margin.  Try it in a directory with
a mix of very long and very short file names.

> > % MENUMODE=interactive
> > % touch test{1,2}
> > % : ; foobar
>
> >     ^ 1. Type the above line in its entirety.
> >       2. Place the cursor before the ;
> >       3. Press Tab.
> >       4. Press Enter.
> > % : test1 bar
> >     ^ Completion is written over existing buffer contents.

> I am also experiencing this bug, I'm running zsh 5.9.

The issue seems to be that interactive mode assumes you're going to
use it interactively -- as in, type a single character at a time until
you've reduced the set to only one match -- so it only adjusts the
buffer spacing on single keystrokes.  When you accept with TAB (or
ENTER) domenuselect() is relying on do_single() to fix everything up,
but the state required by do_single() is not fully populated.  I don't
know precisely what else is needed, though, and am out of time to
study it today.

> autoload -Uz compinit
> compinit
>
> setopt noautomenu
> setopt globcomplete
> bindkey '^i' complete-word
> setopt nolisttypes
>
> zstyle ':completion:*' list-colors ${(s.:.)LS_COLORS}
> zstyle ':completion:*' menu yes select interactive

This configuration doesn't appear to matter to the underlying issue.

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

On Sat, Jul 16, 2022 at 11:11 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> The issue seems to be that interactive mode assumes you're going to
> use it interactively -- as in, type a single character at a time until
> you've reduced the set to only one match -- so it only adjusts the
> buffer spacing on single keystrokes.  When you accept with TAB (or
> ENTER) domenuselect() is relying on do_single() to fix everything up,
> but the state required by do_single() is not fully populated.  I don't
> know precisely what else is needed, though, and am out of time to
> study it today.

Had to take a break from other stuff (weather too hot) so came back to
look at this.

I believe the following fixes it.  The example from Marlon's report
works, even after interactively typing the first few letters before
accepting.

[-- Attachment #2: complist.txt --]
[-- Type: text/plain, Size: 516 bytes --]

diff --git a/Src/Zle/complist.c b/Src/Zle/complist.c
index 0dc64db6a..f12c14e8e 100644
--- a/Src/Zle/complist.c
+++ b/Src/Zle/complist.c
@@ -2447,6 +2447,8 @@ domenuselect(Hookdef dummy, Chdata dat)
             strncpy(zlemetaline, origline, l);
             zlemetacs = origcs;
             setmstatus(status, NULL, 0 , 0, NULL, NULL, NULL);
+	    minfo.end -= minfo.len;
+	    minfo.len = 0;
         } else if (strpfx("search", s)) {
             mode = (strstr(s, "back") ? MM_BSEARCH : MM_FSEARCH);
         }

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Mon, Jul 18, 2022 at 6:20 AM Andrea Manenti <andrea.manenti@yahoo.com> wrote:
>
> Great work, thanks! I patched it and it works, albeit with a little detail off:
>
> say I do ([] is the cursor)
>
> ```
> ls fo[] folder1
> <Tab>
> ls fo[] folder1
> folder2 folder3 ...
> <Tab>
> ls folder2/[] folder1
> (so far so good, but if <Tab> again)
> ls folder2/subfolder/[]older2/ folder1
> ```
>
> In other words, the thing that was just autocompleted ("older2/") gets repeated at the end.

I'm not able to reproduce the above with the minimal configuration
from your earlier email.  I do get a different odd behavior:

% ls fo[] folder1
% ls fo[] folder1
folder1 folder2 folder3

where "folder1" is selected and the "o" in "fo" at the prompt is
highlighted in boldface.  If accept one of the menu choices (e.g.,
press Enter):

% ls folder1/o[] folder1

I haven't figured out where the highlighting is coming from or why
that causes the insertion to occur one character to the left of the
cursor.

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Fri, Jul 22, 2022 at 9:32 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> % ls fo[] folder1
> folder1 folder2 folder3
>
> where "folder1" is selected and the "o" in "fo" at the prompt is
> highlighted in boldface.  If accept one of the menu choices (e.g.,
> press Enter):
>
> % ls folder1/o[] folder1

This seems to fix that, let's see if it handles your problem too.

diff --git a/Src/Zle/complist.c b/Src/Zle/complist.c
index 0dc64db6a..1b87103d7 100644
--- a/Src/Zle/complist.c
+++ b/Src/Zle/complist.c
@@ -2447,6 +2447,9 @@ domenuselect(Hookdef dummy, Chdata dat)
             strncpy(zlemetaline, origline, l);
             zlemetacs = origcs;
             setmstatus(status, NULL, 0 , 0, NULL, NULL, NULL);
+        minfo.insc = minfo.end;
+        minfo.end -= minfo.len;
+        minfo.len = 0;
         } else if (strpfx("search", s)) {
             mode = (strstr(s, "back") ? MM_BSEARCH : MM_FSEARCH);
         }

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Fri, Jul 22, 2022 at 10:55 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> This seems to fix that, let's see if it handles your problem too.

Ah, well, no, that just creates a different odd behavior.  Something
else still not right.

Re: [BUG] complist interactive mode overwrites command line

[Andrea Manenti]

>Subject: Re: [BUG] complist interactive mode overwrites command line
>Date: Friday, 22 July 2022, 20:23:46 CEST
>From: Bart Schaefer <schaefer@brasslantern.com>
>
> On Fri, Jul 22, 2022 at 10:55 AM Bart Schaefer
> <schaefer@brasslantern.com> wrote:
> >
> > This seems to fix that, let's see if it handles your problem too.
> 
> Ah, well, no, that just creates a different odd behavior.  Something
> else still not right.
>

This last patch does fix the weird behavior I pointed out previously but reintroduces the bug we started from the very beginning!

I wish I could be more helpful but I don't understand the code enough, I am always available for testing though. 

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

Another thread rising from the grave:

On Sat, Jul 16, 2022 at 11:11 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> > On Wed, 4 Aug 2021 19:30:10 +0300 Marlon Richert <marlon.richert@xxxxxxxxx> wrote:
> > > % zmodload zsh/complist
> > > % bindkey '^I' menu-select
> > > % MENUMODE=interactive
> > > % touch test{1,2}
> > > % : ; foobar
> >
> > >     ^ 1. Type the above line in its entirety.
> > >       2. Place the cursor before the ;
> > >       3. Press Tab.
> > >       4. Press Enter.
> > > % : test1 bar
> > >     ^ Completion is written over existing buffer contents.
>
> The issue seems to be that interactive mode assumes you're going to
> use it interactively -- as in, type a single character at a time until
> you've reduced the set to only one match -- so it only adjusts the
> buffer spacing on single keystrokes.  When you accept with TAB (or
> ENTER) domenuselect() is relying on do_single() to fix everything up,
> but the state required by do_single() is not fully populated.

This entirely seems to come down to minfo.len causing the wrong thing
to happen in compresult.c:do_single(), specifically here:

    /* If we are already in a menu-completion or if we have done a *
     * glob completion, we have to delete some of the stuff on the *
     * command line.                                               */
    if (minfo.cur)
        l = minfo.len + minfo.insc;
    else
        l = we - wb;

    minfo.insc = 0;
    zlemetacs = minfo.pos;
    foredel(l, CUT_RAW);

In normal menu completion, when entering the menu, the first match is
already inserted on the command line.  This code is used in common by
interactive completion, so when accept-line is invoked, minfo.len is
the length of that first completion.  However, what minfo.len is
supposed to represent is how many characters have already been
inserted on the line, which in this case is zero because interactively
nothing gets inserted until you type a matching character.  Thus the
foredel() call there deletes what it thinks is the completion already
on the command line, but instead deletes other stuff after the cursor.

This can trivially be fixed for Marlon's specific example by setting
minfo.len = 0 in the "if (first)" block in domenuselect(), but that's
only correct for this specific case.  Anything that resets the
appearance of the line -- such as hitting TAB again after accept-line,
because (as in the "user defined widget doesn't execute suffix
removal" thread), we are actually still in menu completion at that
point even though we've left interactive completion -- also loses
track of the fact that the original line is back the way it was, and
so minfo.len becomes wrong again and things like navigating the menu
with the arrow keys start foredel()-ing too much.

Interactive completion gets in its own way here a bit, because there
are several cases where it attempts to restore the original state and
gets it subtly wrong.

If anyone thinks they have a better grasp of all the special cases and
widget handling in complist.c, please chime in.  Of course 2/3 of the
lines of complist.c are Sven W's and a good chunk of the rest is
comments from other people trying to explain what Sven did, so I'm not
very hopeful.

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Mon, Mar 4, 2024 at 3:30 AM Samy Dulor <samy.dulor@orange.fr> wrote:
>
> However, is it not another bug that the one that concerned me in the
> first place?

It all comes back to approximately the same thing:  Interactive
completion has "forgotten" where its input position is and how much it
inserted to the right of that.

> So, trying your test case on the clean Arch repo's zsh package,
> I noticed that if you don't press <TAB> on step 3., pressing the arrow
> will give you this:
> % ls Src/
>
> Is it really the expected behavior?

No, it isn't.

> Would it not be better if it left
> you with:
> % ls autom4te.cache config Src/

That is what's wanted, yes.

> Also, back at my initial issue and my patch, I noticed I made an
> assumption that might be wrong about the expected behavior:
>
> Marlon's test case from the 2021 thread was:
>
> > % touch test{1,2}
> > % : ; foobar
> >      ^ 1. Type the above line in its entirety.
> >        2. Place the cursor before the ;
> >        3. Press Tab.
> >        4. Press Enter.
> > % : test1 bar
> >      ^ Completion is written over existing buffer contents.
>
> In that case, should the ';' still be present?
> ie. % : test1; foobar

I believe so, yes.

Re: [BUG] complist interactive mode overwrites command line

[Samy Dulor]

On 04/03/2024 07:33, Bart Schaefer wrote:
> % ls Src/utom4te.cache config S
> Scripts/       Src/           StartupFiles/
>
> This occurred after using right-arrow to select Src from the menu.
> % ls <TAB>
> Then use interactive completion to fill in
> % ls autom4te.cache config
> Then add S and press TAB:
> % ls autom4te.cache config S<TAB>
> Then right arrow from Script to Src and end up with the mess above.
>
> Settings:
>    bindkey ^I menu-select
>    MENUMODE=interactive

Dammit, I wasn't aware of the 2023 follow-up thread you linked in the 
previous mail...

I'll look into it and on the the issue you just mentionned.

However, is it not another bug that the one that concerned me in the 
first place?
(Not that I wouldn't want to spend time on this one too :p)


So, trying your test case on the clean Arch repo's zsh package,
I noticed that if you don't press <TAB> on step 3., pressing the arrow 
will give you this:
% ls Src/

Is it really the expected behavior? Would it not be better if it left 
you with:
% ls autom4te.cache config Src/
?

I'm asking since without thinking or looking much into it, I can think 
of two possible solutions.


Also, back at my initial issue and my patch, I noticed I made an 
assumption that might be wrong about the expected behavior:

Marlon's test case from the 2021 thread was:

> % touch test{1,2}
> % : ; foobar
>      ^ 1. Type the above line in its entirety.
>        2. Place the cursor before the ;
>        3. Press Tab.
>        4. Press Enter.
> % : test1 bar
>      ^ Completion is written over existing buffer contents.

with my patch, he would end up with
% : test1 foobar

In that case, should the ';' still be present?
ie. % : test1; foobar

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Sun, Mar 3, 2024 at 6:05 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> I think I have an acceptable solution to this issue

It gets closer than any previous attempts, I think.

> The real test is whether it handles this ...
> >> Anything that resets the
> >> appearance of the line -- such as hitting TAB again after accept-line

% ls Src/utom4te.cache config S
Scripts/       Src/           StartupFiles/

This occurred after using right-arrow to select Src from the menu.
% ls <TAB>
Then use interactive completion to fill in
% ls autom4te.cache config
Then add S and press TAB:
% ls autom4te.cache config S<TAB>
Then right arrow from Script to Src and end up with the mess above.

Settings:
  bindkey ^I menu-select
  MENUMODE=interactive

Re: [BUG] complist interactive mode overwrites command line

[Bart Schaefer]

On Sun, Mar 3, 2024 at 5:44 PM Samy Dulor <samy.dulor@orange.fr> wrote:
>
> So I just tried fixing it by correcting the values in minfo when domenuselect restores the command line content,

Yours may be the one combination I did not try, in the thread ending here:
https://www.zsh.org/mla/workers/2023/msg01041.html (workers/52312)

The real test is whether it handles this ...
>> Anything that resets the
>> appearance of the line -- such as hitting TAB again after accept-line,
>> because (as in the "user defined widget doesn't execute suffix
>> removal" thread), we are actually still in menu completion at that
>> point even though we've left interactive completion -- also loses
>> track of the fact that the original line is back the way it was, and
>> so minfo.len becomes wrong again and things like navigating the menu
>> with the arrow keys start foredel()-ing too much.

Re: [BUG] complist interactive mode overwrites command line

[Samy Dulor]

[-- Attachment #1: Type: text/plain, Size: 2371 bytes --]


On 04/08/2021 18:30, Marlon Richert wrote:
> % zmodload zsh/complist
> % bindkey '^I' menu-select
> % MENUMODE=interactive
> % touch test{1,2}
> % : ; foobar
>      ^ 1. Type the above line in its entirety.
>        2. Place the cursor before the ;
>        3. Press Tab.
>        4. Press Enter.
> % : test1 bar
>      ^ Completion is written over existing buffer contents.


Hello,

I think I have an acceptable solution to this issue, although I’m quite 
unaware of all the intricacies of the Zsh source code,
and had 0 knowledge of Zsh internals until 2 days ago...
All I can say is that it doesn’t cause the tests to fail,
And that after a day of usage it doesn’t seem to break anything else (in 
my limited use-cases).

Basically, when entering an interactive select completion menu, the 
first completion in the list gets “selected” (it’s highlighted) but not 
inserted in the command line.
In the above test case, that would be “test1”.
At that time, compresult.c:do_single was called with “test1” completion, 
then complist.c:domenuselect restored the command line to its original 
content.
That means the minfo struct was updated by do_single. In that case, we 
would get minfo { .len = 5, .end = 7 }.

When leaving the interactive mode (pressing Enter to accept the first 
completion, or selecting another completion like using arrow keys),
minfo.lencharacters are removed to “insert the next” completion (5 
characters, but in fact our command line is still “: ; foobar”, so only 
1 character should be erased).

So I just tried fixing it by correcting the values in minfo when 
domenuselect restores the command line content,
Which seems to me, is what should happen as far as I understand :

diff --git a/Src/Zle/complist.c b/Src/Zle/complist.c
index 9cb89a60d..f2eb3837d 100644
--- a/Src/Zle/complist.c
+++ b/Src/Zle/complist.c
@@ -2443,6 +2443,8 @@ domenuselect(Hookdef dummy, Chdata dat)
              strncpy(zlemetaline, origline, l);
              zlemetacs = origcs;
              setmstatus(status, NULL, 0 , 0, NULL, NULL, NULL);
+           minfo.len = origlpre;
+           minfo.end = minfo.pos + minfo.len;
          } else if (strpfx("search", s)) {
              mode = (strstr(s, "back") ? MM_BSEARCH : MM_FSEARCH);
          }

[-- Attachment #2: Type: text/html, Size: 4073 bytes --]

[BUG] complist interactive mode overwrites command line

[Marlon Richert]

% zmodload zsh/complist
% bindkey '^I' menu-select
% MENUMODE=interactive
% touch test{1,2}
% : ; foobar
    ^ 1. Type the above line in its entirety.
      2. Place the cursor before the ;
      3. Press Tab.
      4. Press Enter.
% : test1 bar
    ^ Completion is written over existing buffer contents.