Strange completion display with wrong matcher-list

Strange completion display with wrong matcher-list

[Martin Vaeth]

Is the following a bug in zsh?
I have in my .zshrc:

zstyle ':completion:*' matcher-list '' 'm:{a-zA-Z}={A-Za-z}'

I had copied it somewhere from the net, but it seems
to me now that the first entry (the empty string) is buggy.
However, instead of an error message or reliably wrong behaviour,
it triggers a very strange thing.

For some(!) completion functions in some(!) zsh versions,
when I press "tab" twice, the display of the options is broken.
Below is a minimal example of a completion file which breaks with
zsh-5.3.1 and the above zstyle.

Note that very subtle changes to the function
(e.g. replacing ->cmds directly by _files)
avoid the problem.
Or maybe the function is using -> in a broken manner?

The behavior is this: After pressing "my -<tab><tab>" I see
something unreadable like (in this case)

--long
-s
-- an option
--long
-s
-- an option

instead of the expeected

--long  -s  -- an option

Here is the (broken?) completion file:

#compdef my
local curcontext="$curcontext" state state_descr line
typeset -A opt_args
_arguments -C : \
{'(--long)-s','(-s)--long'}'[an option]' \
'1:an arg:->cmds'
local ret=$?
case $state in
(cmds)
	_files
	ret=$?;;
esac
return ret

Re: Strange completion display with wrong matcher-list

[Martin Vaeth]

Martin Vaeth <martin@mvath.de> wrote:
>
> zstyle ':completion:*' matcher-list '' 'm:{a-zA-Z}={A-Za-z}'
>
> I had copied it somewhere from the net

It is an example in the zsh manpage!

Re: Strange completion display with wrong matcher-list

[Oliver Kiddle]

Martin Vaeth wrote:
> Is the following a bug in zsh?

Not as such, there's a problem in your _my function, though it would
be nice if zsh could avoid the messed up display.

> zstyle ':completion:*' matcher-list '' 'm:{a-zA-Z}={A-Za-z}'
>
> I had copied it somewhere from the net, but it seems
> to me now that the first entry (the empty string) is buggy.
> However, instead of an error message or reliably wrong behaviour,
> it triggers a very strange thing.

What it says is that completion should first try no matcher. If that
fails, it will try again with case-insensitivity enabled.

The return status of completion functions is used to indicate success or
failure and usually corresponds to whether any matches were added. The
path through your function goes as follows:
  +_my:3> _arguments -C : '(--long)-s[an option]' '(-s)--long[an option]' '1:an arg:->cmds'
  +_my:6> local ret=0
  +_my:7> case cmds (cmds)
  +_my:9> _files
  +_my:10> ret=1 
  +_my:12> return ret

_arguments added matches, _files didn't but the return status from files
_is used meaning that _my indicates failure. The result is that this is
_run a second time with the different matcher.

To fix your function, start it with local ret=1 and put && ret=0
after both _arguments and _files.

> For some(!) completion functions in some(!) zsh versions,
> when I press "tab" twice, the display of the options is broken.

You can see the same effect with the following function:
  _my () {
    _arguments {-s,--long}'[an option]'                    
    _arguments {-s,--long}'[an option]'                               
  }

_describe will also show the effect. Laying out the descriptions
assumes (and requires) full control of the group of matches. It's
never a good idea to mix more than one call to _arguments, _describe
or _values. Even if the tags differ, someone might not have the
group-name style set.

zsh internals see 6 display strings: the two options plus the
description which is padded with spaces so is very long. _describe
forces a column first layout and fixed match order. The two
descriptions will not fit side-by-side due to the padding so it
lays them out in a single column. When I wrote _dates, which uses
a rows first layout, getting the padding right was very tricky. The
internals could be improved to make it easier, at the very least
allow auto-padding of a hidden match. Changing _describe to use row
first ordering would help with this issue but repeated tab presses
would then cycle matches in a less ideal order.

Oliver

Re: Strange completion display with wrong matcher-list

[Martin Vaeth]

Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> [...] there's a problem in your _my function

You are right, my guess about the cause was wrong.
There remains the change of behaviour between zsh versions:

> _arguments added matches, _files didn't

In my original example (which I had unfortunately shortened
for the posting), I had passed to the _arguments function
the option -A '-*'.

I had expected that _arguments (at least with -A '-*') would
return with an empty "state" when completing "my -<tab>".

In zsh-5.2, this was indeed the case (at least with -A '-*');
in zsh-5.3 it is no longer so (and so the _my function becomes
faulty for the reasons you explained).

Perhaps the cause is the following "bugfix":

 * 39611: Src/Zle/computil.c: with _arguments sets completion
          stopped if one of the rest arguments starts with a dash

Thanks for your explanation and sorry for the noise.

Re: Strange completion display with wrong matcher-list

[Oliver Kiddle]

Martin Vaeth wrote:
> Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> >
> > [...] there's a problem in your _my function
>
> You are right, my guess about the cause was wrong.
> There remains the change of behaviour between zsh versions:
>
> > _arguments added matches, _files didn't
>
> In my original example (which I had unfortunately shortened
> for the posting), I had passed to the _arguments function
> the option -A '-*'.

The purpose of the -A option is something different: it is to indicate
that no further options should be completed after a word that matches
that pattern. Consider something like:
  grep pattern -l file

Some commands allow the -l to be positioned after the normal argument,
others don't. Whether a command allows this is typically independent of
whether it allows for the normal argument to start with a dash. So while
grep allows -l in the later position, if your pattern starts with -, you
need grep -- -pattern.

Even for a command where -A '-*' is appropriate, I don't think it would
be right to assume that the rest arguments can't be completed if the
current word matches the pattern. For one thing the pattern can be
anything, '*:' perhaps and for the purposes of completion it is a fair
bet to assume that at the cursor position, things are incomplete so a
trailing colon might be meant as something in the middle.

Also, an initial - is typically actually valid in things like files and
patterns so it wouldn't be right to not try to complete them. Also,
with approximate completion and matching control, an initial typed dash
doesn't necessarily imply that the match has to have an initial dash.

Does it cause problems in the case of your function? If completion of
the normal arguments is slow it'd be reasonable to do an explicit
pattern match on the current word.

> In zsh-5.2, this was indeed the case (at least with -A '-*');
> in zsh-5.3 it is no longer so (and so the _my function becomes
> faulty for the reasons you explained).
>
> Perhaps the cause is the following "bugfix":

Actually, it's due to:
  * 39026: pattern specified with _arguments' -A option shouldn't be
  checked against words after the cursor

I'm fairly sure that the original behaviour was not intentional but due
to forgetting to consider the case of the cursor being somewhere other
than the end of the line (not an uncommon mistake).

Oliver

Re: Strange completion display with wrong matcher-list

[Martin Vaeth]

Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>
> The purpose of the -A option is something different

I know, but it is this option which "caused" the _argument
function to work as I had expected in <zsh-5.3 concerning ->.
Only now I learnt that this expectation is considered a bug
in >=zsh-5.3.

Today I fixed all 19 of my packages which had completion files
relying on this expectation...
https://github.com/vaeth?tab=overview&from=2017-01-07

> Also, an initial - is typically actually valid in things like files and
> patterns so it wouldn't be right to not try to complete them.

I am not so sure: Except for echo - which is very special - and
commands which do not accept any options at all, I do not know
_any_ real-world example of a command for which

command -some_argument

is valid without interpreting the argument as an option.
(The only exception is sometimes a bare "-", but for this
completion is rather pointless...)

Of course, as you mentioned, for some commands the situation
is different after a non-option, and for most commands the
situation is different after "--": But that's what -A
and -S are for.