completion oddity

completion oddity

[Danek Duvall]

If I have a completion function like this:

    _k () { _arguments --r1-word --r2-word }

and I press TAB at the end of the following commandline:

    % k --r

then I get

    % k --r-word

Another tab beeps and lists the choices, a third tab beeps and chooses
--r1-word, and thereon tabs simply cycle between the two.  That's more or
less expected.

If I define _k instead as

    _k () { _arguments --r1-word --really-r1-word }

then the second tab simply completes --r1-word, which seems wrong to me.

This is all with "zsh -f", with nothing but

    autoload -Uz compinit
    compinit -i
    compdef _k k

run before attempting the completion.

A co-worker says he noticed this behavior change (from better to worse)
sometime around the switch in Solaris from 4.3.10 to 4.3.12, but I can
reproduce it on our builds that had 4.3.9, so I'm guessing it's been around
for a while.

Thanks,
Danek

Re: completion oddity

[Bart Schaefer]

On May 21,  2:40pm, Danek Duvall wrote:
}
}     _k () { _arguments --r1-word --really-r1-word }
} 
} then the second tab simply completes --r1-word, which seems wrong to me.

It's the hyphens.

Completion works on words.  When you have a compound word joined with
hyphens, it tries to split it up and complete the individual components.
In cases where this is ambiguous, it fills in any word fragments that
are common to all the possible choices and leaves the cursor at the
first ambiguous location.  (Most of this can be changed with zstyle,
for the predefined completion set.)

With --r-word on the line, there is only one possible completion that
has exactly one embedded hyphen.  This is considered a unique match, so
when you press TAB, it fills in that word and is done.  If that isn't
what you want, it's up to you to type (in this example) one of "e" or
"-" at the point where the cursor was left, to disambiguate the choice
before attempting completion again.

Keep in mind that completion is designed to minimize keystrokes, not
to minimize the number of different keys pressed.  Thus TAB e TAB is
considered "just as good" as TAB TAB TAB in terms of saving effort.
This isn't directly expressed in the code, but it's the philosophy
behind disambiguating the match as far as possible before stopping,
because if --r1-word is what you wanted then you save one keystroke,
and if it was not there's no extra cost.

If you want to treat the hyphenated strings as single words rather
than as compound words, then you have to do something not entirely
obvious, which is to override the default "matcher" used when creating
the internal representation of the _arguments specifications.  You do
this like so:

    _k () { _arguments -M '' --r1-word --really-r1-word }

The empty string there turns off matching.  You might want something
else, see "Completion Matching Control" in the manual.  What I can't
find documented anywhere is the default of "r:|[_-]=* r:|=*" which is
part of the (entirely undocumented) internals of the comparguments
builtin.

-- 
Barton E. Schaefer

Re: completion oddity

[Danek Duvall]

On Mon, May 21, 2012 at 07:42:32PM -0700, Bart Schaefer wrote:

> If you want to treat the hyphenated strings as single words rather
> than as compound words, then you have to do something not entirely
> obvious, which is to override the default "matcher" used when creating
> the internal representation of the _arguments specifications.  You do
> this like so:
> 
>     _k () { _arguments -M '' --r1-word --really-r1-word }
> 
> The empty string there turns off matching.

Wonderful.  I'll have to spend some time tomorrow figuring out exactly how
I want to deal with this.  While your explanation makes sense to me, I'm
not sure that it's a natural behavior, or that I could convince anyone else
of it.  I'm tempted to just put -M "" in where needed, but I kinda like the
idea of the multiple partial-word matching, even if it'd never occurred to
me to try it before myself.

I did a quick test with the matcher-list zstyle, though, but couldn't get
it to work (without the -M "" in _k).  I tried both

    zstyle ":completion::complete:k::" matcher-list ""
    zstyle ":completion:*" matcher-list ""

(the first being the string I get when executing _complete_help, and the
second just being super-inclusive) but after both, I got the original
behavior.  I also tried "matcher", but no dice.  Any help here would be
appreciated.

Thanks,
Danek

Re: completion oddity

[Danek Duvall]

I also took another look through the GNU tar options for a similar
situation, and found

    --show-transformed-names
    --show-stored-names

And tried

    gtar --show--names<TAB>

and it beeps and moves me to between the second double-hyphen, then after
another beep, starts cycling between the two.  If I just have "--show"
there, then it ends up cycling among four alternatives.  Perhaps this isn't
sufficiently similar to my test case, but I'm not sure why not.  I'll take
a closer look tomorrow, but if it's easy to explain, I wouldn't mind.

Oh, and the default match spec is documented in the -M option to
_arguments. 

Thanks,
Danek

Re: completion oddity

[Bart Schaefer]

On May 21, 10:01pm, Danek Duvall wrote:
}
} I did a quick test with the matcher-list zstyle, though, but couldn't
} get it to work (without the -M "" in _k).

Two things ...

}     zstyle ":completion::complete:k::" matcher-list ""

(1) The matcher-list is looked up very early in completion.  The context
isn't yet this specific at that point.  (I think this is in the FAQ.)

}     zstyle ":completion:*" matcher-list ""

The matcher-list style is only used when it is not empty!

_description: [[ -n "$_matcher" ]] && opts=($opts -M "$_matcher")

So you can't use matcher-list to turn off matching, only to modify it.

} I also took another look through the GNU tar options for a similar
} situation, and found

Hmm, when I try gtar completion I get

_arguments:comparguments:312: invalid option definition: -[0-7]lmh[specify drive and density]

So I'm not really able to reproduce this.

}     gtar --show--names<TAB>
} 
} and it beeps and moves me to between the second double-hyphen, then after
} another beep, starts cycling between the two.  If I just have "--show"
} there, then it ends up cycling among four alternatives.

Do all four alternatives have the same number of hyphens?

Re: completion oddity

[Greg Klanderman]

>>>>> On May 22, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:

> The matcher-list style is only used when it is not empty!

> _description: [[ -n "$_matcher" ]] && opts=($opts -M "$_matcher")

> So you can't use matcher-list to turn off matching, only to modify it.

Hi Bart,

Is there something I can put in matcher-list to effectively turn off
this default and globally get the behavior as when you added "-M ''"
below:

    _k () { _arguments -M '' --r1-word --really-r1-word }

I tried these:

% zstyle ':completion:*' matcher-list ' ' # space
% zstyle ':completion:*' matcher-list 'm:{a-z}={a-z}'

and neither seems to do the trick..

Greg

Re: completion oddity

[Bart Schaefer]

On Tue, May 22, 2012 at 9:35 AM, Greg Klanderman <gak@klanderman.net> wrote:
>
> Is there something I can put in matcher-list to effectively turn off
> this default and globally get the behavior as when you added "-M ''"

No.  The sneaky thing here is that generically, the matcher-list
zstyle is for matching file names, not command options.

Consequently _arguments doesn't use the matcher-list style itself;
it's up to the individual context completion functions (like _k in the
example) to look up the style (presumably in an option-specific style
context) and pass it to _arguments -M.

Practically speaking, the assumption is that the completion function
must "know" exactly what the command options "look like" for the
command it is completing, so there's no reason to provide a way for
the user to override it.  File names, on the other hand, could look
like anything, so the user is given hooks to tell completion about
those file names.

I can't recall any discussion of leading to a conscious decision to
have it work this way; it just fell out so from the incremental
process of making the completion system handle common cases cleverly
(without requiring the user to tweak everything).

Re: completion oddity

[Greg Klanderman]

>>>>> On May 22, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Tue, May 22, 2012 at 9:35 AM, Greg Klanderman <gak@klanderman.net> wrote:
>> 
>> Is there something I can put in matcher-list to effectively turn off
>> this default and globally get the behavior as when you added "-M ''"

> No.  The sneaky thing here is that generically, the matcher-list
> zstyle is for matching file names, not command options.

> Consequently _arguments doesn't use the matcher-list style itself;
> it's up to the individual context completion functions (like _k in the
> example) to look up the style (presumably in an option-specific style
> context) and pass it to _arguments -M.

OK, thanks for looking into this Bart.  So _main_complete is still
looping over zstyle matcher-list values in completing parameters but
it's not used?  I do see that _arguments uses _description which uses
_matcher if non-empty (as you noted earlier).  I have to confess I
cannot follow all this completion code though.

Greg


> Practically speaking, the assumption is that the completion function
> must "know" exactly what the command options "look like" for the
> command it is completing, so there's no reason to provide a way for
> the user to override it.  File names, on the other hand, could look
> like anything, so the user is given hooks to tell completion about
> those file names.

> I can't recall any discussion of leading to a conscious decision to
> have it work this way; it just fell out so from the incremental
> process of making the completion system handle common cases cleverly
> (without requiring the user to tweak everything).

Re: completion oddity

[Danek Duvall]

On Mon, May 21, 2012 at 11:35:07PM -0700, Bart Schaefer wrote:

> On May 21, 10:01pm, Danek Duvall wrote:
> }
> } I did a quick test with the matcher-list zstyle, though, but couldn't
> } get it to work (without the -M "" in _k).
> 
> Two things ...
> 
> }     zstyle ":completion::complete:k::" matcher-list ""
> 
> (1) The matcher-list is looked up very early in completion.  The context
> isn't yet this specific at that point.  (I think this is in the FAQ.)
> 
> }     zstyle ":completion:*" matcher-list ""
> 
> The matcher-list style is only used when it is not empty!
> 
> _description: [[ -n "$_matcher" ]] && opts=($opts -M "$_matcher")
> 
> So you can't use matcher-list to turn off matching, only to modify it.

So if I set up my own style processing in order to override this behavior
(both the earliness and the use of the empty value) as you suggested to
Greg, would it be kosher to use the matcher-list style, or would I want to
use my own style name?

> }     gtar --show--names<TAB>
> } 
> } and it beeps and moves me to between the second double-hyphen, then after
> } another beep, starts cycling between the two.  If I just have "--show"
> } there, then it ends up cycling among four alternatives.
> 
> Do all four alternatives have the same number of hyphens?

No; three have two, one has one:

    --show-defaults
    --show-omitted-dirs
    --show-transformed-names
    --show-stored-names

I'm still trying to formulate a question about why the default behavior
here is user friendly, but I'm having trouble.  I can understand the logic
of each step, but it doesn't seem like it's the right thing overall.

Thanks,
Danek

#conpdef and compadd

[Eric Smith]

I use #compdef and compadd in a function to define completion
that follows a command (or commands).

How do I define a compadd that looks up in a list of files when there is no 
preceding command, so it is for the first token on the command
line?

-- 
- Eric Smith

Re: #conpdef and compadd

[Mikael Magnusson]

On 15/06/2012, Eric Smith <es@fruitcom.com> wrote:
> I use #compdef and compadd in a function to define completion
> that follows a command (or commands).
>
> How do I define a compadd that looks up in a list of files when there is no
>
> preceding command, so it is for the first token on the command
> line?

-command-

-- 
Mikael Magnusson

Re: #conpdef and compadd -> command regex

[Eric Smith]

Thank Mikael

That works but I realise that I only want completion if the
command commences with a single underscore. How would I define a
regex for the command that needs to be satisfied for the compadd
rule to apply?

-- 
- Eric Smith
Mikael Magnusson said:
> On 15/06/2012, Eric Smith <es@fruitcom.com> wrote:
> > I use #compdef and compadd in a function to define completion
> > that follows a command (or commands).
> >
> > How do I define a compadd that looks up in a list of files when there is no
> >
> > preceding command, so it is for the first token on the command
> > line?
> 
> -command-
> 
> -- 
> Mikael Magnusson

Re: #conpdef and compadd -> command regex

[Mikael Magnusson]

On 15/06/2012, Eric Smith <es@fruitcom.com> wrote:
> Thank Mikael
>
> That works but I realise that I only want completion if the
> command commences with a single underscore. How would I define a
> regex for the command that needs to be satisfied for the compadd
> rule to apply?

You can't afaik, but in your completer you can do whatever you want,
including checking what the word looks like and calling another
completer if it's not what you want. (I'll admit I don't know which
that would be in this case, _command_names is close, but _autocd could
also be called depending on options).

-- 
Mikael Magnusson

Re: completion oddity

[Greg Klanderman]

>>>>> On May 21, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:

> With --r-word on the line, there is only one possible completion that
> has exactly one embedded hyphen.  This is considered a unique match, so
> when you press TAB, it fills in that word and is done.

Then it seems like the previous press of TAB should not have completed
to '--r-word'.  Is it not also the case that completion is designed to
list the choices when there is ambiguity, at least when the autolist
and listambiguous options are set?

In this example, completing from 'k --' twice completely loses the
ambiguity, and completes to '--r1-word' as though that were the only
option starting with '--'.

The thing that seems even more broken is if I compare completion in
the two cases below..

setup:

zsh -f 
% autoload -Uz compinit
% compinit -i
% _k () { _arguments --r1-word --really-r1-word }
% compdef _k k
% setopt autolist listambiguous completeinword noautomenu

now compare this:

% k --
# hit <tab> at end of line
% k --r-word
# cursor now on the final hyphen; hit <tab> again
% k --r1-word 
# cursor now after appended space

vs:

% k --r-word
# with cursor on final hyphen hit <tab>:
# nothing inserted; the *two* options are listed
# no matter how many times you hit <tab>

So there must be some hidden state from the initial completion in the
first case, because otherwise I would expect hitting <tab> on the same
command line with the cursor in the same position to give the same
results.

Greg

Re: completion oddity

[Bart Schaefer]

On Tue, May 22, 2012 at 9:18 AM, Greg Klanderman <gak@klanderman.net> wrote:
>
>>>>>> On May 21, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:
>
>> With --r-word on the line, there is only one possible completion that
>> has exactly one embedded hyphen.  This is considered a unique match, so
>> when you press TAB, it fills in that word and is done.
>
> Then it seems like the previous press of TAB should not have completed
> to '--r-word'.

Sven W. was a clever but highly suggestible university student.
Someone said, "Hey, all these possible variations end with the same
substring, why isn't completion smart enough to fill that in for me?"
and Sven responded, "OK, now it is that smart," and all the possible
side-effects were either not considered or considered to be less
important than the smarts.

> % k --r-word
> # with cursor on final hyphen hit <tab>:
> # nothing inserted; the *two* options are listed
> # no matter how many times you hit <tab>
>
> So there must be some hidden state from the initial completion in the
> first case, because otherwise I would expect hitting <tab> on the same
> command line with the cursor in the same position to give the same
> results.

You're right; in the first case the completion system has recorded a
list of "ambiguous positions" that distinguish what it filled in vs.
what the user typed.  It decides whether the match is unique based on
whether there is another ambiguous position to which it can advance.
This is all part of the aforementioned smarts.

Re: completion oddity

[Greg Klanderman]

>>>>> On May 22, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:

> You're right; in the first case the completion system has recorded a
> list of "ambiguous positions" that distinguish what it filled in vs.
> what the user typed.  It decides whether the match is unique based on
> whether there is another ambiguous position to which it can advance.
> This is all part of the aforementioned smarts.

Oh the horrors never end.. could you be so kind as to point me to some
of the relevant bits of code that are doing this logic and how that
state is stored?  Please tell me it's not in the even more
impenetrable C code..

Greg

Re: completion oddity

[Bart Schaefer]

On May 22,  6:55pm, Greg Klanderman wrote:
}
} OK, thanks for looking into this Bart.  So _main_complete is still
} looping over zstyle matcher-list values in completing parameters but
} it's not used?  I do see that _arguments uses _description which uses
} _matcher if non-empty (as you noted earlier).

Hmm, hadn't thought about that ... _parameters calls _wanted so it does
go through _description, which means the matcher-list is also applied
to parameter names.  At this point I have no idea whether that is
intentional.

} I have to confess I cannot follow all this completion code though.

I'm not sure anyone can.
 
On May 22,  7:09pm, Greg Klanderman wrote:
}
} > You're right; in the first case the completion system has recorded a
} > list of "ambiguous positions" that distinguish what it filled

Sorry, thinko there -- it actually records UNambiguous positions, and
insert_positions which are the places that something needs to be added
or changed in order that the word on the line match one of the possible
completions.

} Oh the horrors never end.. could you be so kind as to point me to some
} of the relevant bits of code that are doing this logic and how that
} state is stored?  Please tell me it's not in the even more
} impenetrable C code..

The state is available in the compstate hash, which is actually pretty
thoroughly documented (though exactly what the internals do with the
values isn't, so much).  However, yes, the logic that figures out what
are insert_positions and what are unambiguous_positions is all deep
in the C code.  There is exactly one example of shell code making use
of one of those values:  Functions/Zle/cycle-completion-positions --
but it's not very helpful.

Re: completion oddity

[Danek Duvall]

On Tue, May 22, 2012 at 03:57:45PM -0700, Bart Schaefer wrote:

> Sven W. was a clever but highly suggestible university student.  Someone
> said, "Hey, all these possible variations end with the same substring,
> why isn't completion smart enough to fill that in for me?" and Sven
> responded, "OK, now it is that smart," and all the possible side-effects
> were either not considered or considered to be less important than the
> smarts.

Hm.  So maybe undoing that locally is a possibility ... is there some way,
either in a user's configuration or within a single completion function, to
turn off the "end with the same substring" completion, while maintaining
the dash-separates-words aspect of the completion, so that

    k --r<TAB>

is considered fully ambiguous, and either cycles through the options or
lists the ambiguous completions, while

    k --r-r-w<TAB>

would consider --really-r1-word unambiguous and complete only it?

I think that would be a decent compromise in my mind.

Thanks,
Danek

Re: completion oddity

[Bart Schaefer]

On May 22,  4:06pm, Danek Duvall wrote:
}
} So if I set up my own style processing in order to override this
} behavior (both the earliness and the use of the empty value) as you
} suggested to Greg, would it be kosher to use the matcher-list style,
} or would I want to use my own style name?

Doesn't really matter.  Using the matcher-list name means that any
definition of that for any less-specific context will end up applying 
to your more specific situation, but if you're planning always to
provide the most specific definition then that won't matter.

} > }     gtar --show--names<TAB>
} > } 
} > } and it beeps and moves me to between the second double-hyphen,
} > } then after another beep, starts cycling between the two. If I
} > } just have "--show" there, then it ends up cycling among four
} > } alternatives.
} 
}     --show-defaults
}     --show-omitted-dirs
}     --show-transformed-names
}     --show-stored-names

Oh, well, that all makes sense.  When you have all four possibilities
there are no common suffixes, and when you do have a common suffix
there are the same number of words; in neither case is there the
possibility of resolving the ambiguity.

On May 22,  6:03pm, Danek Duvall wrote:
} Subject: Re: completion oddity
}
} ... is there some way, either in a user's configuration or within
} a single completion function, to turn off the "end with the same
} substring" completion, while maintaining the dash-separates-words
} aspect of the completion [...?]

In fact, there is, and it's at once so obvious and so unintuitive
that I'm quite sure it wasn't done on purpose.

Define _k this way:

    _k() { _arguments --really-r1-word --r1-word }

The specifications passed to _arguments are tried in order, so if
the longer one (which I suspect means the one having the most words
rather than the most characters) is tried first, the ambiguity is
re-discovered and you get menu completion.

Re: completion oddity

[Danek Duvall]

On Tue, May 22, 2012 at 11:31:55PM -0700, Bart Schaefer wrote:

> On May 22,  6:03pm, Danek Duvall wrote:
> } Subject: Re: completion oddity
> }
> } ... is there some way, either in a user's configuration or within
> } a single completion function, to turn off the "end with the same
> } substring" completion, while maintaining the dash-separates-words
> } aspect of the completion [...?]
> 
> In fact, there is, and it's at once so obvious and so unintuitive
> that I'm quite sure it wasn't done on purpose.
> 
> Define _k this way:
> 
>     _k() { _arguments --really-r1-word --r1-word }

Perfect!  Now it'll work out of the box for everyone (well, me with my
particular completion configuration and my colleague, who has none).

Thanks so much,
Danek

Re: completion oddity

[Bart Schaefer]

On May 22, 12:18pm, Greg Klanderman wrote:
}
} % k --r-word
} # with cursor on final hyphen hit <tab>:
} # nothing inserted; the *two* options are listed
} # no matter how many times you hit <tab>

I took your word for this when I was replying before (and I didn't say
anything incorrect) but now that I try it myself I don't get this
behavior.  I get --r-word completed to --r1-word no matter which way
I put --r-word onto the line in the first place and no matter where
within the word I place the cursor.

However, see mail I'm about to send to Danek ...

Re: completion oddity

[Greg Klanderman]

>>>>> On May 23, 2012 Bart Schaefer <schaefer@brasslantern.com> wrote:

> On May 22, 12:18pm, Greg Klanderman wrote:
> }
> } % k --r-word
> } # with cursor on final hyphen hit <tab>:
> } # nothing inserted; the *two* options are listed
> } # no matter how many times you hit <tab>

> I took your word for this when I was replying before (and I didn't say
> anything incorrect) but now that I try it myself I don't get this
> behavior.  I get --r-word completed to --r1-word no matter which way
> I put --r-word onto the line in the first place and no matter where
> within the word I place the cursor.

Hmmm I'm not fully up to date with CVS:

phl% echo $ZSH_VERSION $ZSH_PATCHLEVEL 
4.3.17-dev-0 1.5607

but only by a couple months.  I'll update and see if anything changes.
But I just tried again with this version and see exactly as I posted.
Also tried 4.3.11-dev-1 1.5222 and 4.3.17-dev-0 1.5600 with same
result.

You did run the setopt line I had included in the setup section right?

Greg

> However, see mail I'm about to send to Danek ...

Re: completion oddity

[Bart Schaefer]

On May 23,  2:48pm, Greg Klanderman wrote:
}
} You did run the setopt line I had included in the setup section right?

Aha.  I thought so, but had not.

Setting completeinword is the key here.  With that not set, it's always
moving the cursor to the end of the word and then completing from there.
That's the underlying reason why it doesn't detect the ambiguity in the
original (way back in Derek's first post) example.