Bug: Completion of the 'zsh' command.

Bug: Completion of the 'zsh' command.

[Kunshan Wang]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

Hi,

Filename arguments of the 'zsh' command cannot be completed.

Distro: ArchLinux
Kernel: linux 3.11.4-1
Arch: x86-64
zsh: 5.0.2

Also reproducable on a Mac OSX 10.8 with zsh installed from Homebrew.

How to reproduce:

1. Create a new user account on my Linux to set up a clean environment.
Then use compinstall to initialize a default setting. (This is optional
since it is usually reproducable whenever completion is enabled)

2. Create a new text file:

    $ echo 'echo hello' > hello.zsh

3. Type 'zsh ./h' (no quote) and press TAB

What actually happens: The file hello.zsh is listed below, but continue
pressing TAB will not select the file 'hello.zsh'. Nothing happens.

What I think it should happen: zsh should cycle through all filenames in
the current directory beginning with 'h' (because I typed './h') and
complete it.

More information:

* The Mac OSX comes with an older 4.x version of zsh and it completes
the filename normally.

* The zsh FAQ 4.6 (http://zsh.sourceforge.net/FAQ/zshfaq04.html#l50)
works. It will start file completion normally, but I don't understand
why the completer of the 'zsh' command is not expecting any file in the
current directory. Also there are many scripts in
/usr/share/zsh/functions/Completion and many scripts for zsh's internal
commands, but there is no script for the 'zsh' command itself. Is it
placed elsewhere?

Regards,
Kunshan Wang


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: Bug: Completion of the 'zsh' command.

[Bart Schaefer]

On Oct 10,  1:15am, Kunshan Wang wrote:
}
} 3. Type 'zsh ./h' (no quote) and press TAB
} 
} What actually happens: The file hello.zsh is listed below, but continue
} pressing TAB will not select the file 'hello.zsh'. Nothing happens.

Hm.  To answer your other question first:

} Also there are many scripts in /usr/share/zsh/functions/Completion and
} many scripts for zsh's internal commands, but there is no script for
} the 'zsh' command itself. Is it placed elsewhere?

Zsh and other shells are all handled by the _sh function.  In general
you can't tell just by the file name what commands a completion function
is used for.  To find this out, you can do one of two things:

(1) Run "compinit" and then exampine the $_comps array, e.g. $_comps[zsh]

(2) Do something similar to:  egrep compdef.+zsh $^fpath/*(N.)

} What I think it should happen: zsh should cycle through all filenames in
} the current directory beginning with 'h' (because I typed './h') and
} complete it.

This does in fact work when completing for "sh" which uses the same
completion function.  The difference seems to be that, for zsh only,
there is first a call to _arguments which has the unintended (?) side-
effect of erasing compstate[insert].  The rest of completion is then
tried, but with more basic functions that don't expect to need to put
compstate[insert] back again, so their results are only listed and not
added to the command line.

I suspect _arguments really expects to be in charge of the whole process,
not to be used as an adjunct to simpler defaults.

There might be some different magic it's possible to apply here, but the
simplest thing I can think of is to simply move the _arguments part to
the end of _sh instead of doing it first.  I can't find anything obvious
this gets wrong, but there's always a chance that e.g. _default should
have been skipped if _arguments succeeded, which this patch would break.

I confess I don't understand the bit about always returning zero no
matter whether _options succeeds or fails, so I've tweaked it, but maybe
someone else can remember.  Furthermore, I don't remember what '*:' does
for _arguments; I would have expected '*:something:', and the only two
places in the whole Completion/ tree where '*:' is passed to _arguments
are here and in _ethtool.  Someone please jog my memory?


diff --git a/Completion/Unix/Command/_sh b/Completion/Unix/Command/_sh
index 7258e42..59aafc2 100644
--- a/Completion/Unix/Command/_sh
+++ b/Completion/Unix/Command/_sh
@@ -5,10 +5,7 @@ if [[ $service == zsh ]]; then
   if [[ ${words[CURRENT-1]} == -o ]]; then
     _options
     # no other possibilities
-    return 0
-  fi
-  if _arguments -S -s -- '*:'; then
-    return 0
+    return
   fi
 fi
 
@@ -25,3 +22,11 @@ else
   fi
   _default
 fi
+
+local ret=$?
+
+if [[ $service == zsh ]] && _arguments -S -s -- '*:'; then
+  ret=0
+fi
+
+return ret

Re: Bug: Completion of the 'zsh' command.

[Bart Schaefer]

On Oct 9,  8:01pm, Bart Schaefer wrote:
}
} Furthermore, I don't remember what '*:' does for _arguments

Figured out that it's shorthand for '*:::' which causes _arguments to
go though this branch (some whitespace removed for compactness):

    if [[ "$action" = \ # ]]; then
      # An empty action means that we should just display a message.
      _message -e "$subc" "$descr"

} there is first a call to _arguments which has the unintended (?) side-
} effect of erasing compstate[insert].

That's coming from "_message -e ..." which has:

  (( $compstate[nmatches] )) || compstate[insert]=

The problem of course is that we haven't finished generating all the
matches yet, so it's premature to test (( $compstate[nmatches] )).

Several possible approaches come to mind:

(1) The patch I already sent, which assures that this call to _message
happens late enough for $compstate[nmatches] to be valid.

(2) Instead of '*:' aka '*:::' as the last argument of _arguments in _sh,
actually give it an action, '*:::_default' (or something more clever).
This bypasses the call to _message -e, but makes it appear that every
zsh option (except -c) takes an argument, which (I think) changes the
zstyle context ... which probably hardly ever matters, but ...

(3) Fiddle with _message somehow, for example, shove the assignment to
compstate[insert] into a comppostfuncs callback -- except then I'm not
sure it happens soon enough, so we've exchanged one problem for another.

Anybody have an opinion?

Re: Bug: Completion of the 'zsh' command.

[Peter Stephenson]

On Thu, 10 Oct 2013 12:41:13 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> (1) The patch I already sent, which assures that this call to _message
> happens late enough for $compstate[nmatches] to be valid.

Perhaps combining this with a note in compsys.yo about this side effect
to _message and hence also of _arguments when displaying messages would
cover enough bases.  For _arguments the caveat is perhaps more what you
said early --- that it doesn't espect to be combined arbitrarily with
other forms of completion ("arbitrarily" because it does have ways of
returning control to the caller to take the next steps).

pws

Re: Bug: Completion of the 'zsh' command.

[Bart Schaefer]

On Oct 11,  9:56am, Peter Stephenson wrote:
} Subject: Re: Bug: Completion of the 'zsh' command.
}
} On Thu, 10 Oct 2013 12:41:13 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > (1) The patch I already sent, which assures that this call to _message
} > happens late enough for $compstate[nmatches] to be valid.
} 
} Perhaps combining this with a note in compsys.yo about this side effect
} to _message and hence also of _arguments when displaying messages would
} cover enough bases.  For _arguments the caveat is perhaps more what you
} said early --- that it doesn't espect to be combined arbitrarily with
} other forms of completion ("arbitrarily" because it does have ways of
} returning control to the caller to take the next steps).

OK, I got a bit carried away here:  I've rearranged and slightly altered
the _arguments section of the manual to try to pull at least a mention
of the options of _arguments itself up to the top, and otherwise keep
the option details in proximity to any related descriptions.

I also changed a couple of examples to use -xy instead of -ab, because
that seems less confusing in the context of the -A description.  There
are other examples that could also be made less confusing, such as the
one using the -C option of "tar" which has nothing to do with the -C
option of _arguments, but I got worn down before tackling that stuff.

Some of this is pretty baroque and I may have it wrong.  Please proof.

Re-patch of _sh included to keep it all in one article number.

diff --git a/Completion/Unix/Command/_sh b/Completion/Unix/Command/_sh
index 7258e42..104c757 100644
--- a/Completion/Unix/Command/_sh
+++ b/Completion/Unix/Command/_sh
@@ -5,10 +5,7 @@ if [[ $service == zsh ]]; then
   if [[ ${words[CURRENT-1]} == -o ]]; then
     _options
     # no other possibilities
-    return 0
-  fi
-  if _arguments -S -s -- '*:'; then
-    return 0
+    return
   fi
 fi
 
@@ -25,3 +22,9 @@ else
   fi
   _default
 fi
+
+local ret=$?
+
+[[ $service == zsh ]] && _arguments -S -s -- '*:' && ret=0
+
+return ret
diff --git a/Doc/Zsh/compsys.yo b/Doc/Zsh/compsys.yo
index 4a436da..d42ad97 100644
--- a/Doc/Zsh/compsys.yo
+++ b/Doc/Zsh/compsys.yo
@@ -3470,14 +3470,26 @@ Like tt(_tags) this function supports the tt(-C) option to give a
 different name for the argument context field.
 )
 findex(_arguments)
-item(tt(_arguments) [ tt(-nswWACRS) ] [ tt(-O) var(name) ] [ tt(-M) var(matchspec) ] [ tt(:) ] var(spec) ...)(
-This function can be used to give a complete specification for
-completion for a command whose arguments follow standard UNIX option and
-argument conventions.  The following forms specify individual sets of
-options and arguments; to avoid ambiguity, these may be separated from the
-options to tt(_arguments) itself by a single colon.  Options to
-tt(_arguments) itself must be in separate words, i.e. tt(-s -w), not
-tt(-sw).
+xitem(tt(_arguments) [ tt(-nswWCRS) ] [ tt(-A) var(pat) ] [ tt(-O) var(name) ] [ tt(-M) var(matchspec) ] [ tt(:) ] var(spec)...)
+item(tt(_arguments) [ var(opts)... ] tt(-)tt(-) [ -i var(pats) ] [ tt(-s) var(pair) ] [ var(helpspec)... ])(
+This function can be used to give a complete specification for completion
+for a command whose arguments follow standard UNIX option and argument
+conventions.  Options to tt(_arguments) itself must be in separate words,
+i.e. tt(-s -w), not tt(-sw).
+
+When calling tt(_arguments), all var(spec)s that describe options of the
+analyzed command line must precede all var(spec)s that describe non-option
+(aka "normal") arguments of the analyzed line.  To avoid ambiguity, all
+options to tt(_arguments) itself may be separated from the var(spec) forms
+by a single colon.
+
+The tt(-s -w -W -A) and tt(-S) options describe how parsing of the command
+line should proceed, and are discussed in context below.  The `tt(-)tt(-)'
+form is used to intuit var(spec) forms from the help output of the command
+being analyzed, and is described in detail below.  The var(opts) for the
+`tt(-)tt(-)' form are otherwise the same options as the first form.  Note
+that `tt(-s)' following `tt(-)tt(-)' has a distinct meaning from `tt(-s)'
+preceding `tt(-)tt(-)', and both may appear.
 
 With the option tt(-n), tt(_arguments) sets the parameter tt(NORMARG)
 to the position of the first normal argument in the tt($words) array,
@@ -3486,6 +3498,17 @@ has not been reached, tt(NORMARG) is set to tt(-1).  The caller
 should declare `tt(integer NORMARG)' if the tt(-n) option is passed;
 otherwise the parameter is not used.
 
+The option `tt(-M) var(matchspec)' sets a match specification to use to
+completion option names and values.  The default var(matchspec) is:
+
+example(tt(r:|[_-]=* r:|=*))
+
+This allows partial word completion after `tt(_)' and `tt(-)', for example
+`-f-b' can be completed to `tt(-foo-bar)'.
+
+Each of the following forms is a var(spec) describing individual sets of
+options or arguments on the command line being analyzed.
+
 startitem()
 xitem(var(n)tt(:)var(message)tt(:)var(action))
 item(var(n)tt(::)var(message)tt(:)var(action))(
@@ -3530,10 +3553,10 @@ suitable for standard GNU options.
 
 The combination of tt(-s) with tt(-w) allows single-letter options to be
 combined in a single word even if one or more of the options take
-arguments.  For example, if tt(-a) takes an argument, with no
-tt(-s) `tt(-ab)' is considered as a single (unhandled) option; with
-tt(-s) tt(-ab) is an option with the argument `tt(b)'; with both tt(-s)
-and tt(-w), tt(-ab) may be the option tt(-a) and the option tt(-b) with
+arguments.  For example, if tt(-x) takes an argument, with no
+tt(-s) `tt(-xy)' is considered as a single (unhandled) option; with
+tt(-s) tt(-xy) is an option with the argument `tt(y)'; with both tt(-s)
+and tt(-w), tt(-xy) may be the option tt(-x) and the option tt(-y) with
 arguments still to come.
 
 The option tt(-W) takes this a stage further:  it is possible to
@@ -3603,6 +3626,23 @@ enditem()
 It is possible for options with a literal `PLUS()' or `tt(=)' to
 appear, but that character must be quoted, for example `tt(-\+)'.
 
+The options tt(-S) and tt(-A) are available to simplify the specifications
+for commands with standard option parsing.  With tt(-S), no option will be
+completed after a `tt(-)tt(-)' appearing on its own on the line; this
+argument will otherwise be ignored; hence in the line
+
+example(foobar -x -- -y)
+
+the `tt(-x)' is considered an option but the `tt(-y)' is considered an
+argument, while the `tt(-)tt(-)' is considered to be neither.
+
+With tt(-A), no options will be completed after the first non-option
+argument on the line.  The tt(-A) must be followed by a pattern matching
+all strings which are not to be taken as arguments.  For example, to make
+tt(_arguments) stop completing options after the first normal argument, but
+ignoring all strings starting with a hyphen even if they are not described
+by one of the var(optspec)s, the form is `tt(-A "-*")'.
+
 Each var(optarg) following an var(optspec) must take one of the
 following forms:
 
@@ -3676,6 +3716,11 @@ form below, the var(action) will be executed by calling the
 tt(_all_labels) function to process all tag labels.  No special handling
 of tags is needed unless a function call introduces a new one.
 
+The option `tt(-O) var(name)' specifies the name of an array whose elements
+will be passed as arguments to functions called to execute var(action)s.
+For example, this can be used to pass the same set of options for the
+tt(compadd) builtin to all var(action)s.
+
 The forms for var(action) are as follows.
 
 startitem()
@@ -3804,6 +3849,18 @@ Furthermore, during the evaluation of the var(action) the context name in
 the tt(curcontext) parameter is altered to append the same string that is
 stored in the tt(context) parameter.
 
+The option tt(-C) tells tt(_arguments) to modify the tt(curcontext)
+parameter for an action of the form `tt(->)var(state)'.  This is the
+standard parameter used to keep track of the current context.  Here it
+(and not the tt(context) array) should be made local to the calling
+function to avoid passing back the modified value and should be
+initialised to the current value at the start of the function:
+
+example(local curcontext="$curcontext")
+
+This is useful where it is not possible for multiple states to be valid
+together.
+
 It is possible to specify multiple sets of options and
 arguments with the sets separated by single hyphens.  The specifications
 before the first hyphen (if any) are shared by all the remaining sets.
@@ -3850,47 +3907,6 @@ A useful alternative is often an option specification with rest-arguments
 (as in `tt(-foo:*:...)'); here the option tt(-foo) swallows up all
 remaining arguments as described by the var(optarg) definitions.
 
-The options tt(-S) and tt(-A) are available to simplify the specifications
-for commands with standard option parsing.  With tt(-S), no option will be
-completed after a `tt(-)tt(-)' appearing on its own on the line; this
-argument will otherwise be ignored; hence in the line
-
-example(foobar -a -- -b)
-
-the `tt(-a)' is considered an option but the `tt(-b)' is considered an
-argument, while the `tt(-)tt(-)' is considered to be neither.
-
-With tt(-A), no options will be completed after the first non-option
-argument on the line.  The tt(-A) must be followed by a pattern matching
-all strings which are not to be taken as arguments.  For example, to make
-tt(_arguments) stop completing options after the first normal argument, but
-ignoring all strings starting with a hyphen even if they are not described
-by one of the var(optspec)s, the form is `tt(-A "-*")'.
-
-The option `tt(-O) var(name)' specifies the name of an array whose elements
-will be passed as arguments to functions called to execute var(actions).
-For example, this can be used to pass the same set of options for the
-tt(compadd) builtin to all var(action)s.
-
-The option `tt(-M) var(spec)' sets a match specification to use to
-completion option names and values.  It must appear before the first
-argument specification.  The default is `tt(r:|[_-]=* r:|=*)': this allows
-partial word completion after `tt(_)' and `tt(-)', for example `-f-b'
-can be completed to `tt(-foo-bar)'.
-
-The option tt(-C) tells tt(_arguments) to modify
-the tt(curcontext) parameter for an action of the form
-`tt(->)var(state)'.  This is the standard parameter used to keep track of
-the current context.  Here it (and not the tt(context) array) should be
-made local to the calling function
-to avoid passing back the modified value and should be initialised to the
-current value at the start of the function:
-
-example(local curcontext="$curcontext")
-
-This is useful where it is not possible for multiple states to be valid
-together.
-
 The option `tt(-)tt(-)' allows tt(_arguments) to work out the names of long
 options that support the `tt(-)tt(-help)' option which is standard in many
 GNU commands.  The command word is called with the argument
@@ -3901,11 +3917,16 @@ the behaviour of the command is unspecified.
 In addition to options, `tt(_arguments -)tt(-)' will try to deduce the
 types of arguments available for options when the form
 `tt(-)tt(-)var(opt)=var(val)' is valid.  It is also possible to provide
-hints by examining the help text of the command and adding specifiers of
-the form `var(pattern)tt(:)var(message)tt(:)var(action)'; note that normal
-tt(_arguments) specifiers are not used.  The var(pattern) is matched
+hints by examining the help text of the command and adding var(helpspec) of
+the form `var(pattern)tt(:)var(message)tt(:)var(action)'; note that other
+tt(_arguments) var(spec) forms are not used.  The var(pattern) is matched
 against the help text for an option, and if it matches the var(message) and
-var(action) are used as for other argument specifiers.  For example:
+var(action) are used as for other argument specifiers.  The special case
+of `tt(*:)' means both var(message) and var(action) are empty, which has
+the effect of causing options having no description in the help output to
+be ordered in listings ahead of options that have a description.
+
+For example:
 
 example(_arguments -- '*\*:toggle:(yes no)' \ 
               '*=FILE*:file:_files' \ 
@@ -3947,12 +3968,23 @@ will cause completion to ignore the options
 useful with GNU tt(configure)).
 
 The `tt(_arguments -)tt(-)' form can also be followed by the option `tt(-s)
-var(pair)' to describe option aliases.  Each var(pair) consists of a
-pattern and a replacement.  For example, some tt(configure)-scripts
-describe options only as `tt(-)tt(-enable-foo)', but also accept
+var(pair)' to describe option aliases.  The var(pair) consists of a list
+of alternating patterns and corresponding replacements, enclosed in parens
+and quoted so that it forms a single argument word in the tt(_arguments)
+call.
+
+For example, some tt(configure)-script help output describes options only
+as `tt(-)tt(-enable-foo)', but the script also accepts the negated form
 `tt(-)tt(-disable-foo)'.  To allow completion of the second form:
 
-example(_arguments -- -s "LPAR()#-tt(-enable- -)tt(-disable-RPAR()"))
+example(tt(_arguments -- -s "LPAR()#-)tt(-enable- -)tt(-disable-RPAR()"))
+
+Finally, note that tt(_arguments) generally expects to be the primary
+function handling any completion for which it is used.  It may have side
+effects which change the treatment of any matches added by other functions
+called after it.  To combine tt(_arguments) with other functions, those
+functions should be called either before tt(_arguments), as an var(action)
+within a var(spec), or in handlers for `tt(->)var(state)' actions.
 
 Here is a more general example of the use of tt(_arguments):
 
@@ -4251,11 +4283,14 @@ which already contains an expanded description.
 The tt(-12VJ) options and the var(group) are passed to tt(compadd) and
 hence determine the group the message string is added to.
 
-The second form gives a description for completions with the tag
-var(tag) to be shown even if there are no matches for that tag.  The tag
-can be omitted and if so the tag is taken from the parameter
-tt($curtag); this is maintained by the completion system and so is
-usually correct.
+The second tt(-e) form gives a description for completions with the tag
+var(tag) to be shown even if there are no matches for that tag.  This form
+is called by tt(_arguments) in the event that there is no action for an
+option specification.  The tag can be omitted and if so the tag is taken
+from the parameter tt($curtag); this is maintained by the completion
+system and so is usually correct.  Note that if there are no matches at
+the time this function is called, tt(compstate[insert]) is cleared, so
+additional matches generated later are not inserted on the command line.
 )
 findex(_multi_parts)
 item(tt(_multi_parts) var(sep) var(array))(

Re: Bug: Completion of the 'zsh' command.

[Peter Stephenson]

On Fri, 11 Oct 2013 23:18:01 -0700
> Some of this is pretty baroque and I may have it wrong.  Please proof.

It looks quite sensible, but without sitting down and following through
every baroque feature I'm taking much of it on trust.

pws

Re: Bug: Completion of the 'zsh' command.

[Bart Schaefer]

On Oct 13,  7:06pm, Peter Stephenson wrote:
} Subject: Re: Bug: Completion of the 'zsh' command.
}
} On Fri, 11 Oct 2013 23:18:01 -0700
} > Some of this is pretty baroque and I may have it wrong.  Please proof.
} 
} It looks quite sensible, but without sitting down and following through
} every baroque feature I'm taking much of it on trust.

Most of it is just moving stuff around, so it can be assumed correct if
Sven W. described it accurately in the first place all those years ago.

The part that's most likely to be inaccurate is my attempt to explain
the "--" inuition magic better, particularly the stuff about the -i
and -s options and "helpspec" vs. "normal spec".

In any case I've gone ahead and committed it.