Re: Let's finish this new completion stuff

Re: Let's finish this new completion stuff

[Sven Wischnowsky]

Another reply to this, not directly connected to the implementation of 
the completion stuff.

Peter Stephenson wrote:

> By the way, is it
> really necessary to have the `cmd=$(whence -p $COMMAND)' on every call to
> command/argument completion?  I haven't thought about it, but maybe it can
> be optimised away in a lot of cases.

The problem is that this is immediatly used when testing pattern
completions. And I think this is important to have since `*/X11/*' is
a better test for X-programs than `x*', which is just plain wrong.
I first tried `eval cmd2\==$COMMAND' but that makes zsh stop if there
is no such command (`zsh: foo not found').

> I'm still vaguely of the opinion it would be quite nice to have long
> Posix-style options for the comp* commands, as I suggested some time ago,
> e.g. 
> 
> complist -mf
> 
> could also be
> 
> complist --external --files
> 
> or something for readability.

Yes, but maybe we should do this is some generic way in builtin.c, so
that we can add such long options to (almost) all builtins. I.e. a way 
to store the long names together with the short names they are aliases 
for and then make execbuiltin() put them in the ops array as usual.


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Sven Wischnowsky]

Bart Schaefer wrote:

> I don't want to get into anything that looks like a multidimensional
> array.

We already allow multiple subscripst in parameter expansion, not in
assignments.

> OK would be $matches[prefix(num)] or something ... though in
> thinking about it a bit further, I think I prefer requiring the (r) or
> (R) flag to activate any such magic, e.g. $matches[(r)prefix(num)].
> The connection to patterns is useful; it's just a special case of a
> pattern.

Yes, I'd prefer requiring the `(r)', too. (Hm, I already said that,
didn't I? I think so...)

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Sven Wischnowsky]

Bart Schaefer wrote:

> By the way, your doc for the new completion stuff doesn't mention any
> of the $argv-modifying etc. side-effects of the condition codes (except
> for -iprefix changing $IPREFIX), as far as I can tell.

I had it for `-string' and `-class', too, but forgot it for `-after'
and `-between'.

Bye
 Sven

--- od/Zsh/compwid.yo	Mon Feb 22 10:49:09 1999
+++ Doc/Zsh/compwid.yo	Mon Feb 22 11:10:07 1999
@@ -382,15 +382,19 @@
 var(min) and equal to or less than var(max)
 )
 item(tt(-after) var(string))(
-true if the cursor is after a word that is equal to var(string)
+true if the cursor is after a word that is equal to var(string); this
+removes all words up to and including the matched word from the
+positional parameters
 )
 item(tt(-mafter) var(pattern))(
 like tt(-after) but using pattern matching
 )
 item(tt(-between) var(string1) var(string2))(
 true if the cursor is after a word that is equal to var(string1), if
-there is also a word that is equal to va(string2), this is true only
-if the cursor is before it
+there is also a word that is equal to var(string2), this is true only
+if the cursor is before it; as a side effect, all words before
+var(string1) and after var(string2) (both inclusive) are removed from
+the positional parameters
 )
 item(tt(-mbetween) var(pattern1) var(pattern2))(
 like tt(-between) but using pattern matching

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Sven Wischnowsky]

I'm trying to reply simultaneously to Peter and Bart here...

Bart:
> Preliminary remark:  I think it's premature to be talking about "finishing"
> this stuff.  There are, what, three? people actually trying this out right
> now?  And one of them is the implementor of 90% of the underlying code.
> This is not the way to do a usability study.  I imagine we're going to be
> tweaking things for a long while.

Yes, I know that only too well. It was an attempt to draw more
attention to the mail... (sorry).

> } - Using the positional parameters as we do now has some drawbacks. We
> }   could make the completion widget store them in another array but
> }   then the modifying tests (see below) would still change `argv'
> 
> Let me see if I understand this.  Referring to the tests that modify the
> word list as a side-effect:
> 
> As of the current date, those tests (a) expect to find the previous word
> list in the positionals, and (b) always stuff the new word list into the
> positional parameters of whatever function calls them.  So if you forget
> to pass the previous word list to a sub-function as its arguments, none
> of the tests work as expected.
> 
> Have I got that right?

Yep.

> If we don't use the positionals, we should use another special variable
> with a hardwired name, as PWS suggested.

That was what I meant, fine.

> What do you do in cases where COMMAND presently doesn't contain a real
> command name?

The `context'-name would be in the `CONTEXT'-parameter (or the one we
use to replace `CONTEXT'), the words from the line would still be in
the `words' array (e.g the words when completing inside an array value 
or inside a condition).

> } - We could also put the `COMMAND' into the `CONTEXT' parameter which I 
> }   would like to replace with a associative array with keys:
> 
> So whichever of these was uninteresting would simply not be set?

Yes (or empty).

> Anyway, let's call this hypothetical associative array `compstate' for
> purposes of discussion.
> 
> }   - `command'   if this is used as the replacement for `COMMAND'.
> }   - `redirect'  for the string of the redirection operator
> }   - `parameter' for the `value' and `subscript' contexts where it will 
> }                 contain the name of the parameter
> }   - `type'      as the replacement for the old `CONTEXT'
> 
> I like `context' better.  `Type' means too many different things.

I was thinking about calling the array `context' and didn't like
`$context[context]'. If the parameter has another name, I'd prefer
using `context' as the key, too.

Peter now:
> This sounds OK, but if $COMMAND is really just going to be $words[1] or
> whatever it might be a bit confusing to put it here too --- for example, if
> you are completing after `nice' so that $words gets narrowed to whatever's
> after, does $context[command] refer to `nice' or to the next word, and when
> you change the context (shift words; (( CURRENT-- )) ) is the user or the
> shell responsible for changing it?  I think we should just stick to using
> $words or whatever name anyone else suggests, if that's agreed upon.

I meant to either store it in `words' exclusive-or in
`context'. The narrowing would be done by the shell when using the
modifying whatevers but could, of course be done in shell code (and
with everything in the `words' array this would be much easier).

Bart again:
> Why not put everything that's not another array in here?
> 
> 	CURRENT		-->	compstate[current]
> 	PREFIX		-->	compstate[prefix]
> 	IPREFIX		-->	compstate[ignored]
> 	SUFFIX		-->	compstate[suffix]
> 	NMATCHES	-->	compstate[nmatches]
> 	MATCHER		-->	compstate[matcher]

Actually, I thought about this, too. But `$PREFIX' looks so
nice. Still, I wouldn't mind stuffing everything into one associative
array.

Peter:
> > - The example code currently uses the return value of the shell
> >   functions or a parameter (in the new version I just sent) to decide
> >   whether more completion definitions should be used. This can not be
> >   combined with `compcall' which currently gives no information about
> >   things like that. I'm not too sure about this, but maybe we would
> >   want to add a special parameter that will be used by `compcall' and
> >   that can also (of course) be used in the shell code. (Someone else
> >   should decide this since I don't care that much about `compctl' any
> >   more.)
> 
> I'd have thought for anyone still using compctl it would be good enough
> just to be able to test $NMATCHES.  If anybody wants anything more
> sophisticated, they're going to be going over to new completion anyway.

*Good* (That's what I wanted to hear ;-)

Bart:
> } The testing problem:
> } 
> } Some of the condition codes we currently have could easily be replaced
> } by shell code so we should probably remove them. The question was/is
> } with what we would replace them.
> 
> I have a radical suggestion:  Replace them with magic elements of the
> compstate array.  A special associative predefined by zsh doesn't have
> to be implemented by a simple hash table; it can do its own special
> interpretation of the subscripts.
> 
> Then we can invent a whole lot of new syntax for the completion stuff,
> and hide it where the lexer already skipparens() it.

I like `magic' syntax, but I'm not sure if the users would like it...

> [ examples for implementing tests in shell code ]
> 
> These are so messy I'm not going to try work them out on the fly here:
> 
> [[ -string str ]]
> ...
> [[ -mbetween pat1 pat2 ]]

I showed shell code for them in message 5386, possible, but...

> An alternate idea is to require the (r) or (R) subscript flags to make
> the magic syntax active, so you get something like:
> 
>     $compstate[(r)position(x)]	 if compstate[current] == x, expands
>     				 to $words[x], else expands to nothing
> 
>     $compstate[(R)position(x,y)] if compstate[current] is between x
> 				 and y, expands to $words[x,y]

Hm, I like this, but I'll need some more comments on it before
trying to implement something like this.

> I suggest making the existing list available in a parameter that can
> be tested and modified directly; the sets.afn of that array can do the
> equivalent of `compadd`.  For booleans like "are we menu-completing,"
> see the AA idea above.

Making the list of completions generated so far available isn't that
easy without disallowing adding further completions. Also, we can't
easily deliver all the information stored with an array. But now that
we are speaking about sepcial subscript-syntax anyway, how about:

  $matches[num][prefix]  gives the `compadd -p'-prefix of the num'th
                         match
  $matches[num][string]  the string to list

And so on. I've no idea how difficult to implement this would be,
though.

> } - Options to say what should be done with the matches genereated,
> }   e.g.: `list them', `don't list them', `use menu-completion', `insert 
> }   it into the line', etc.
> 
> Don't those options belong on complist/compadd?  Or is it not possible
> to have some of the matches listed and others not?  (That is, is it a
> requirement that there be a single global switch for this stuff?)

With `compadd' one can add matches that are not to be shown. But what
I meant is the post-processing and that can not be set on a per-match
basis (we can't have one half of the matches menu-completed and the
other half with normal completion). I hadn't thought about using the
associative array for this, though (as Peter suggested, too). But,
yes, you are right, if we make this a more complete
`context-information' thing, this kind of information could (or
should) be stored there (and taken from there after the completion
widgets finish).

> By the way, again, one thing I haven't figured out how to do with the
> new completion stuff is the equivalent of `compctl -X`.  How does one
> add an explanation string, and when does it get shown?

You just use `complist -X ...'. Explanation strings are stored with
groups of matches. So a `complist -X ...' stores a string in the
sorted group named `default'. To store strings in other groups one
would use `complist -J ... -X ...' (or `-V').

Peter:
> In other words, internal control of what the widget is really doing: `I'm
> fed up with being an expand-or-complete, I want to be a list-choices'.
> Maybe this needs to integrate somehow with `zle ...' calls, but I can't
> offhand think how.

Hm, hadn't thought about `zle ...' and have no ideas now.

> I presume it's easy enough to tell whether you're completing after a ~
> anyway -- just look to see if there's a slash yet.  The advantage of
> something like a context would be you can finally solve the problem compctl
> -T was invented to solve in a modular way, i.e. making your own list of
> usernames.  With a context you can just plug in your own comps[-tilde-] by
> the autoload-scanning mechanism.  Thinks: this doesn't have to be a context
> generated by the C code, there's nothing to stop _normal looking up
> $comps[-tilde-] by itself if it thinks that's a good idea, [and maybe
> completing users if there isn't one (it can even check $COMPSKIP) --- but
> perhaps the best idea would be to provide a _tilde handler for -tilde-
> which the user then modifies directly].

Yes, that's what I was thniking about. Let the C-code do the test for
`$...' (which isn't that easy with all the flag-parsing and so on) and 
let the widget do the `[[ -iprefix '~' ]] ...' or whatever.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Bart Schaefer]

Just a few words ...

Sven Wischnowsky writes:
 > 
 > Making the list of completions generated so far available isn't that
 > easy without disallowing adding further completions. Also, we can't
 > easily deliver all the information stored with an array. But now that
 > we are speaking about sepcial subscript-syntax anyway, how about:
 > 
 >   $matches[num][prefix]  gives the `compadd -p'-prefix of the num'th
 >                          match
 >   $matches[num][string]  the string to list

I don't want to get into anything that looks like a multidimensional
array.  OK would be $matches[prefix(num)] or something ... though in
thinking about it a bit further, I think I prefer requiring the (r) or
(R) flag to activate any such magic, e.g. $matches[(r)prefix(num)].
The connection to patterns is useful; it's just a special case of a
pattern.

Re: Let's finish this new completion stuff

[Sven Wischnowsky]

Matt Armstrong wrote:

> > Before I get tired of asking the same questions again and
> > again, I'll write down all the things I'm thinking about
> > and which I'd like to discuss before we start implementing
> > them.
> >
> > *Please* take the time to send comments, suggestions, answers,
> > questions or whatever.
> 
> I think the completion stuff is one of the most important aspects of zsh 
> (it is what motivated me to switch to zsh from tcsh).  So, I'd love to 
> contribute to this discussion.
> 
> However, I must admit that I haven't understood most of the completion 
> related discussion to date.  I think this is in part because it is a 
> complex subject, but also because I came in in the middle of the 
> development.
> 
> The thing that'd help me understand most is an easy way to see a 
> function call trace when __complete_expand-or-complete and friends get 
> called.  Is something like that possible?

This is for normal completion (the one not bound to special keys):

- The C-Code calls `_main_complete', that is the completion widget.
- This function first looks up the entry for `-first-' in the
  associative array `comps' which is used as the table to connect
  command names (and a few special names like `-first-', `-default-',
  etc) with the names of the functions that generate the completions
  for that command.
  If there is an entry for `-first-', the function is called, after
  that `COMPSKIP' is tested (the replacement for `compctl -t'): if it
  is set, no other completions are generated.
  All functions generating matches get the words from the command line 
  as positional parameters.
- When other completions (vaguely speaking: for the command itself)
  are to be generated, `_main_complete' looks at the value of
  `CONTEXT' to find out which function to call. For contexts like
  mathematical expressions and so on, other special names surrounded
  by hyphens are used for looking at `comps'. Again, the function
  found, if any, is invoked.
- Completion in command position and in arguments is handled in
  another function: `_normal'. For completion in command position it
  just calls the entry for `-command-' in the `comps' array. For
  arguments this function first gets the command name to use. These
  may actually be two names: if a path is given on the line, the whole
  path and the last component are tried. If the name on the line starts
  with `=' the full pathname is used. Otherwise the name on the line
  and the pathname of that command is used.
  With these names, the (normal) array `patcomps' is searched. Entries 
  in this array look like `pat func', i.e.: a pattern and the name of
  a function separated by a space. Whenever an element is found whose
  pattern matches one of the command names produced, the function is
  invoked to generate matches. After that, `COMPSKIP' is tested again.
  After the patterns were tried, lookup in the associative array
  `comps' is done again, with the two command names (one after
  another), if no entry for them is found, the entry for `-default-'
  is used again. As soon as an entry is found, the function is
  invoked.

So, the call chain for normal argument-completion is:

  C-Code
    _main_complete
      _normal
        functions for patterns, if any
        function for the command or for `-default-

For non-argument-or-command completion the call chain is:

  C-Code
    _main_complete
      function for special context or for `-default-'

Completion bound to special keys just calls the function for that key
directly with no intervening helper functions.

The functions generating the matches may call `compalso <name> <args>'
to let the completion code generate matches for another command named
`<name>', where the `<args>' are the words from the line (i.e. the
positional parameters). That function looks at `comps' to find the
function to call. Completion generating functions that need to do
sub-command completion (e.g. `find' between `-(ok|exec)' and `\;') can 
simple call `_normal' to do the completion-lookup and match-generation 
on the word-range which is normally set by invoking one of the
modifying condition codes.

Hope this helps.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Matt Armstrong]

> Before I get tired of asking the same questions again and
> again, I'll write down all the things I'm thinking about
> and which I'd like to discuss before we start implementing
> them.
>
> *Please* take the time to send comments, suggestions, answers,
> questions or whatever.

I think the completion stuff is one of the most important aspects of zsh 
(it is what motivated me to switch to zsh from tcsh).  So, I'd love to 
contribute to this discussion.

However, I must admit that I haven't understood most of the completion 
related discussion to date.  I think this is in part because it is a 
complex subject, but also because I came in in the middle of the 
development.

The thing that'd help me understand most is an easy way to see a 
function call trace when __complete_expand-or-complete and friends get 
called.  Is something like that possible?


______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com

Re: Let's finish this new completion stuff

[Peter Stephenson]

"Matt Armstrong" wrote:
> The thing that'd help me understand most is an easy way to see a 
> function call trace when __complete_expand-or-complete and friends get 
> called.  Is something like that possible?

You can use set -x (aka setopt xtrace), that's the main reason I've
enhanced it to trace conditions, function calls and arithmetic evaluations.
I've been using it a lot myself.  There are two problems: it messes up the
display, so it's hard to see what's happening, though if you're just
listing a completion it's not so bad since all the trace information is
above and you can still read the list.  You may be able to `exec
2>xtrace.log' since the tty is maintained separately (if it doesn't work,
it's probably a bug).  Note you can change PROMPT4, which escaped me until
the weekend.

Secondly (if it's a problem) set -x is different from ksh in that ksh turns
it off inside a function where it's not explicitly set, which zsh doesn't,
so you maybe get more info than you want.  (Does anybody think this should
be altered?  I think it's trivial to do that.  `emulate zsh' doesn't change
xtrace, but `emulate -R zsh' does, but even so you get everything up to and
including the emulate statement.)  `setopt localoptions [no]xtrace' does
work, of course.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Let's finish this new completion stuff

[Sven Wischnowsky]

Before I get tired of asking the same questions again and again, I'll
write down all the things I'm thinking about and which I'd like to
discuss before we start implementing them.


*Please* take the time to send comments, suggestions, answers,
questions or whatever.


The special parameters:

- Using the positional parameters as we do now has some drawbacks. We
  could make the completion widget store them in another array but
  then the modifying tests (see below) would still change `argv'
  unless we make the test get the name of the array to work upon as an 
  argument. But then we could equally well let users use `read' to get 
  the words (which I wouldn't like but that may be only my problem).
  Of course we could also make the completion code store the words in
  an array and give them as positional parameters. The we would have
  the problem mentioned by Bart that the value of CURRENT may be off
  by one for those who use `ksharrays', unless we use the setting of
  that option when setting CURRENT in the completion code.
- I would like to remove the parameter `COMMAND' since this has
  already become yet another way to describe the context (`COMMAND' is 
  a bit of a misnomer currently). When using an array to report the
  words on the line we could store it in this array as the first word
  which should make non-automatic modification of this array
  easier. With that we could get rid of the `command' or `argument'
  CONTEXT, replacing both of them with one name.
- We could also put the `COMMAND' into the `CONTEXT' parameter which I 
  would like to replace with a associative array with keys:
  - `command'   if this is used as the replacement for `COMMAND'.
  - `redirect'  for the string of the redirection operator
  - `parameter' for the `value' and `subscript' contexts where it will 
                contain the name of the parameter
  - `type'      as the replacement for the old `CONTEXT'
  - `quote'     if we want to do this, this could be set to `single',
                `double', or the empty string giving information about 
		the quotes the completion code thinks we are in
  We may find other interesting things here and with an associative
  array it would be easy to add new types of context information.
- The example code currently uses the return value of the shell
  functions or a parameter (in the new version I just sent) to decide
  whether more completion definitions should be used. This can not be
  combined with `compcall' which currently gives no information about
  things like that. I'm not too sure about this, but maybe we would
  want to add a special parameter that will be used by `compcall' and
  that can also (of course) be used in the shell code. (Someone else
  should decide this since I don't care that much about `compctl' any
  more.)


The testing problem:

Some of the condition codes we currently have could easily be replaced
by shell code so we should probably remove them. The question was/is
with what we would replace them. There seems to be support for doing
this with a builtin and indeed some of the examples have shown that we 
sometimes only need the modification, not the result of the test. So
that is already almost decided, but I'd like to put it into a builtin
that does not only the testing. I already spoke about my wish to add a 
builtin that gives control over some of the surrounding completion
code and I'd like to put it there. That builtin would have several
options (again):

- Options to control the special parameters. For the modifications we
  could either add one option per modification (e.g. using the
  characters known from `compctl -x': `-s <str>' and the like), or use 
  (nearly) the syntax Bart sometimes used (although I don't know if
  this was suggested syntax): `-m ignored-prefix <str>'.
  With these/this option(s) the builtin would set it's return value in 
  a way that it can be used for testing, but that would only be a
  `side effect'.
  Here we could also add a built-in replacement for the `compsave' and 
  `compreset' aliases from the example code. Let's give it two options 
  to save and restore the state of all the special parameters. It
  would be the user's responsibility to make those calls symmetrical.
- Options to get information from the calling completion code,
  e.g. things like `is there already a valid list', `will menu-
  completion be used' (which would also test for automenu) and the
  like.
- Options to say what should be done with the matches genereated,
  e.g.: `list them', `don't list them', `use menu-completion', `insert 
  it into the line', etc.

I really think we should have a way to specify things like the last
ones, giving full control to the user. Later we may use this builtin
to give access to the matches already generated, allowing users to
look at them, change them, or delete them. Also we could then clean up 
the `-U' and `-m' options to `compadd' (the way Peter already suggested
it): making `compadd' do matching unless the `-U' option is used and
remove the `-m' option. What keeps me from doing this already is that
`-U' switches on menu-completion which is not what one wants if the
function using `compadd' did the matching.


Miscellaneous:

- As Bart already said, the name `complist' is a misnomer (I was
  thinking about `it adds a list of matches...'). Any suggestion for a 
  better name from someone who knows more English synonyms or has more 
  ideas than I would be welcome (we could use `compctl' for it, making 
  it behave differently when called from a completion widget...).
- Since `compctl' is already used I'd also like to hear suggestions
  for the control-builtin I proposed.
- I already spoke about the `problem' with completion after `~', `=',
  and `$' when `complist' is used. Currently the code automatically
  uses the builtin way to complete in such places. I'd like to change
  this by making `complist' behave as if it didn't know better and
  adding contexts reported to the user via the `context' parameter for 
  these circumstances. The question is: should I add contexts for `~'
  and `=' - almost doesn't look like it's worth it.
- Bart suggested making the leading underscore special in the
  `functions' builtin and `compctl -K ...'. I like this, but I'd like
  to hear what others think about it before...


Ok. I'm curious if I get answers this time.


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Let's finish this new completion stuff

[Peter Stephenson]

Sven Wischnowsky wrote:
> - Using the positional parameters as we do now has some drawbacks. 

I haven't experienced any myself yet --- though I realise I'm danger of
forgetting to tack "$@" on the end of a subfunction --- but I've no
objection to using $words or whatever. (Anything beginning with `arg' is a
bit suspect because it's too like $argv which is exactly what it wouldn't
be; $words isn't very specific, but it's pretty memorable and I get to
quote Hamlet:

Polonius:  What do you read, my Lord?
Hamlet: $words, $words, $words.)

I would agree it's more logical to keep them separate from the function
call scope.

>   We
>   could make the completion widget store them in another array but
>   then the modifying tests (see below) would still change `argv'
>   unless we make the test get the name of the array to work upon as an 
>   argument.

Why is this a problem if the array name for the command-line arguments is
hard-wired?  And why would you not want to hard-wire it?  Do we finally
need to tackle the thorny question of localising special parameters?
(There's already similar stuff to handle `special=tempvalue builtin', it
seems a shame not to show it off.)

>   But then we could equally well let users use `read' to get 
>   the words (which I wouldn't like but that may be only my problem).

I don't like this either.  I'd prefer to keep the interface simple and use
parameters wherever possible.

>   Of course we could also make the completion code store the words in
>   an array and give them as positional parameters. The we would have
>   the problem mentioned by Bart that the value of CURRENT may be off
>   by one for those who use `ksharrays', unless we use the setting of
>   that option when setting CURRENT in the completion code.

I'd be happier to do one or the other, rather than this compromise.

> - I would like to remove the parameter `COMMAND' since this has
>   already become yet another way to describe the context (`COMMAND' is 
>   a bit of a misnomer currently). When using an array to report the
>   words on the line we could store it in this array as the first word
>   which should make non-automatic modification of this array
>   easier. With that we could get rid of the `command' or `argument'
>   CONTEXT, replacing both of them with one name.

In other words, you test [[ $CURRENT -eq 1 ]] and use $words[1] ?
That seems pretty logical.  But I would still want _normal able to handle
this separately, i.e. have something special it does when [[ $CURRENT -eq 1
]] just as it does now for [[ $CONTEXT == command ]].  By the way, is it
really necessary to have the `cmd=$(whence -p $COMMAND)' on every call to
command/argument completion?  I haven't thought about it, but maybe it can
be optimised away in a lot of cases.

> - We could also put the `COMMAND' into the `CONTEXT' parameter which I 
>   would like to replace with a associative array with keys:
>   - `command'   if this is used as the replacement for `COMMAND'.
>   - `redirect'  for the string of the redirection operator
>   - `parameter' for the `value' and `subscript' contexts where it will 
>                 contain the name of the parameter
>   - `type'      as the replacement for the old `CONTEXT'
>   - `quote'     if we want to do this, this could be set to `single',
>                 `double', or the empty string giving information about 
> 		the quotes the completion code thinks we are in
>   We may find other interesting things here and with an associative
>   array it would be easy to add new types of context information.

This sounds OK, but if $COMMAND is really just going to be $words[1] or
whatever it might be a bit confusing to put it here too --- for example, if
you are completing after `nice' so that $words gets narrowed to whatever's
after, does $context[command] refer to `nice' or to the next word, and when
you change the context (shift words; (( CURRENT-- )) ) is the user or the
shell responsible for changing it?  I think we should just stick to using
$words or whatever name anyone else suggests, if that's agreed upon.

> - The example code currently uses the return value of the shell
>   functions or a parameter (in the new version I just sent) to decide
>   whether more completion definitions should be used. This can not be
>   combined with `compcall' which currently gives no information about
>   things like that. I'm not too sure about this, but maybe we would
>   want to add a special parameter that will be used by `compcall' and
>   that can also (of course) be used in the shell code. (Someone else
>   should decide this since I don't care that much about `compctl' any
>   more.)

I'd have thought for anyone still using compctl it would be good enough
just to be able to test $NMATCHES.  If anybody wants anything more
sophisticated, they're going to be going over to new completion anyway.

> Some of the condition codes we currently have could easily be replaced
> by shell code so we should probably remove them. The question was/is
> with what we would replace them. There seems to be support for doing
> this with a builtin and indeed some of the examples have shown that we 
> sometimes only need the modification, not the result of the test.

I think more and more that `comptest' (or options of a more general command
e.g.  `compset') would be OK and that if you want to use [[ ... ]] the
parameters will be enough.  So [[ $[I]PREFIX = ... ]] wouldn't do any
modification, of course, but compset -iprefix ... would.  It is quite nice
being able to have complete lines like
  elif [[ -iprefix '+' || -iprefix '@' || -current -1 -draftfolder ]]; then
but, as Bart said, having tests with side-effects is just too much.

>   Here we could also add a built-in replacement for the `compsave' and 
>   `compreset' aliases from the example code. Let's give it two options 
>   to save and restore the state of all the special parameters. It
>   would be the user's responsibility to make those calls symmetrical.

I'm still vaguely of the opinion it would be quite nice to have long
Posix-style options for the comp* commands, as I suggested some time ago,
e.g. 

complist -mf

could also be

complist --external --files

or something for readability.  I mention it here since `compset --push' and
`compset --pop' are the obvious options in this case.

> - Options to get information from the calling completion code,
>   e.g. things like `is there already a valid list', `will menu-
>   completion be used' (which would also test for automenu) and the
>   like.

This goes nicely with associative arrays --- it's hard to get information
from builtins without forking and maybe this is a natural part of the
$context.  What I was murmuring about the other day was something like `if
there is a valid list, use it instead of generating a new one'; the latter
bit would presumably require some compset (I just invented that name but it
seems as good anything?) trickery --- or should it automatically use an
existing list of no new completions are generated?  (The next point, if
implemented, would make this pretty much irrelevant.)

> - Options to say what should be done with the matches genereated,
>   e.g.: `list them', `don't list them', `use menu-completion', `insert 
>   it into the line', etc.

In other words, internal control of what the widget is really doing: `I'm
fed up with being an expand-or-complete, I want to be a list-choices'.
Maybe this needs to integrate somehow with `zle ...' calls, but I can't
offhand think how.  (My first thought to do what I mentioned above was,
test if we just had a _complete-* as $LASTWIDGET and if so run `zle
list-choices' and return, but that doesn't work.)  Or maybe the ability to
set context[widget]=list-choices is good enough (hmm, altering your own
context is getting a bit post-modern, maybe the assoc array needs to be
called something else).

> making `compadd' do matching unless the `-U' option is used and
> remove the `-m' option. What keeps me from doing this already is that
> `-U' switches on menu-completion which is not what one wants if the
> function using `compadd' did the matching.

That seems sensible.

> - As Bart already said, the name `complist' is a misnomer (I was
>   thinking about `it adds a list of matches...'). Any suggestion for a 
>   better name from someone who knows more English synonyms or has more 
>   ideas than I would be welcome (we could use `compctl' for it, making 
>   it behave differently when called from a completion widget...).

compmatch? compflags? compaddflags [i.e. it's like compadd, but instead of
the matches themselves you specify flags for them]? compgen? complete
[don't ask what lete means, there's a clash with tcsh there, of course]?
compmake? compsel[ect]?

> - Since `compctl' is already used I'd also like to hear suggestions
>   for the control-builtin I proposed.

I suggested `compset' above, since it sets things; other possiblities are
things like `compopt[s]', but it does more than just change options.

> - I already spoke about the `problem' with completion after `~', `=',
>   and `$' when `complist' is used. Currently the code automatically
>   uses the builtin way to complete in such places. I'd like to change
>   this by making `complist' behave as if it didn't know better and
>   adding contexts reported to the user via the `context' parameter for 
>   these circumstances.

OK

>   The question is: should I add contexts for `~'
>   and `=' - almost doesn't look like it's worth it.

I presume it's easy enough to tell whether you're completing after a ~
anyway -- just look to see if there's a slash yet.  The advantage of
something like a context would be you can finally solve the problem compctl
-T was invented to solve in a modular way, i.e. making your own list of
usernames.  With a context you can just plug in your own comps[-tilde-] by
the autoload-scanning mechanism.  Thinks: this doesn't have to be a context
generated by the C code, there's nothing to stop _normal looking up
$comps[-tilde-] by itself if it thinks that's a good idea, [and maybe
completing users if there isn't one (it can even check $COMPSKIP) --- but
perhaps the best idea would be to provide a _tilde handler for -tilde-
which the user then modifies directly].

> - Bart suggested making the leading underscore special in the
>   `functions' builtin and `compctl -K ...'. I like this, but I'd like
>   to hear what others think about it before...

Sounds OK to me, but are we morally obliged to have functions generate all
functions including _* in ksh compatibility mode?  Seems a bit too trivial
to have its own option.  [[ -o posixbuiltins ]] && ...?  There's no such
problem with compctl -K, unless people are using a _ convention for that.

Another issue: ordering of pattern completions.  I think this can already
be done quite naturally.  `init' (should we turn the files into compinit
and compdump btw?) stores the patterns in $patcomps in the order it finds
them in the directory, which is globbing order, which is alphabetic, and
`dump' preserves this order, and _normal goes through them in the same
order.  So the user can have e.g.
 _patcomp01_X_files              # the completion is out there...
 _patcomp02_zftp
if they need ordering (looks a bit like /etc/rc.d, in fact).  It's harder
if they live in more than one directory, but at least they come up in the
order they do in $fpath, which seems entirely natural; e.g. if
fpath=(~/myfunc /usr/local/lib/zsh/functions), then first you get all your
own, then all the standard ones, if any.  We could point this out in the
rubric for #defpatcomp and I think that's the end of the matter.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: Let's finish this new completion stuff

[Bart Schaefer]

Preliminary remark:  I think it's premature to be talking about "finishing"
this stuff.  There are, what, three? people actually trying this out right
now?  And one of them is the implementor of 90% of the underlying code.
This is not the way to do a usability study.  I imagine we're going to be
tweaking things for a long while.

On Feb 19,  4:21pm, Sven Wischnowsky wrote:
} Subject: Let's finish this new completion stuff
}
} - Using the positional parameters as we do now has some drawbacks. We
}   could make the completion widget store them in another array but
}   then the modifying tests (see below) would still change `argv'

Let me see if I understand this.  Referring to the tests that modify the
word list as a side-effect:

As of the current date, those tests (a) expect to find the previous word
list in the positionals, and (b) always stuff the new word list into the
positional parameters of whatever function calls them.  So if you forget
to pass the previous word list to a sub-function as its arguments, none
of the tests work as expected.

Have I got that right?

That was not what I envisioned when I suggested passing the word list in
the positionals, but then, I didn't envision tests with side-effects.

I was about to suggest that we put the words in an associative array with
elements named "0", "1", etc., but it occurred to me that you wouldn't be
able to do arithmetic in the subscripts, or slice assignment.

}   unless we make the test get the name of the array to work upon as an 
}   argument.

If we don't use the positionals, we should use another special variable
with a hardwired name, as PWS suggested.

Fortunately, because of some magic zsh does, $words[0] would always be
the command name, although with ksharrays NOT set, so would $words[1].

I just had a really wild idea for how we could handle -string, -iprefix,
and so on, with one or more special associative arrays.  Hmm, again.

} - I would like to remove the parameter `COMMAND' since this has
}   already become yet another way to describe the context (`COMMAND' is 
}   a bit of a misnomer currently). When using an array to report the
}   words on the line we could store it in this array as the first word

What do you do in cases where COMMAND presently doesn't contain a real
command name?

} - We could also put the `COMMAND' into the `CONTEXT' parameter which I 
}   would like to replace with a associative array with keys:

So whichever of these was uninteresting would simply not be set?

Thinking about that caused me to notice this bit of oddness:

zagzig% typeset -A foo
zagzig% echo ${+foo[x]}
1
zagzig% typeset -a bar
zagzig% echo ${+bar[x]}
0

That is, all fields of associative arrays currently appear to be set,
even when they aren't; this is not true of plain arrays.

Anyway, let's call this hypothetical associative array `compstate' for
purposes of discussion.

}   - `command'   if this is used as the replacement for `COMMAND'.
}   - `redirect'  for the string of the redirection operator
}   - `parameter' for the `value' and `subscript' contexts where it will 
}                 contain the name of the parameter
}   - `type'      as the replacement for the old `CONTEXT'

I like `context' better.  `Type' means too many different things.

}   - `quote'     if we want to do this, this could be set to `single',
}                 `double', or the empty string giving information about 
} 		the quotes the completion code thinks we are in

I suggest putting the actual quote character in there.  Then you don't
even have to test the value if all you want to do is close the quote;
you just append $compstate[quote] to whatever you're compadd-ing.

}   We may find other interesting things here and with an associative
}   array it would be easy to add new types of context information.

Yes.  Hmm, hmm, hmm.

Why not put everything that's not another array in here?

	CURRENT		-->	compstate[current]
	PREFIX		-->	compstate[prefix]
	IPREFIX		-->	compstate[ignored]
	SUFFIX		-->	compstate[suffix]
	NMATCHES	-->	compstate[nmatches]
	MATCHER		-->	compstate[matcher]

} - The example code currently uses the return value of the shell
}   functions or a parameter (in the new version I just sent) to decide
}   whether more completion definitions should be used. This can not be
}   combined with `compcall' which currently gives no information about
}   things like that.

I'm not sure exactly what you're asking about.

} The testing problem:
} 
} Some of the condition codes we currently have could easily be replaced
} by shell code so we should probably remove them. The question was/is
} with what we would replace them.

I have a radical suggestion:  Replace them with magic elements of the
compstate array.  A special associative predefined by zsh doesn't have
to be implemented by a simple hash table; it can do its own special
interpretation of the subscripts.

Then we can invent a whole lot of new syntax for the completion stuff,
and hide it where the lexer already skipparens() it.

} There seems to be support for doing this with a builtin [....]  That
} builtin would have several options (again):
} 
} - Options to control the special parameters. For the modifications we
}   could either add one option per modification (e.g. using the
}   characters known from `compctl -x': `-s <str>' and the like), or use 
}   (nearly) the syntax Bart sometimes used (although I don't know if
}   this was suggested syntax): `-m ignored-prefix <str>'.

Let's think about this a bit.  These are all easy:

[[ -nmatches x ]]	((compstate[nmatches] == x))
[[ -matcher x ]]	((compstate[matcher] == x))

[[ -word x str ]]	[[ $words[x] == 'str' ]]
[[ -mword x str ]]	[[ $words[x] == str ]]
[[ -current x str ]]	[[ $words[compstate[current]+offset] == 'str' ]]
[[ -mcurrent x str ]]	[[ $words[compstate[current]+offset] == str ]]

These aren't too bad:

[[ -prefix str ]]	[[ -z "${compstate[prefix]%%str*}" ]]
[[ -iprefix str ]]	[[ -z "${compstate[prefix]%%str*}" ]] &&
			    { compstate[ignored]="$compstate[ignored]str"
			      compstate[prefix]="${compstate[prefix]#str}" }

These are OK (but handling the negative numbers gets tricky, not shown):

[[ -words x ]]		(($#words == x))
[[ -words x y ]]	(($#words >= x && $#words <= y))

[[ -position x ]]	[[ $compstate[current] == x ]]
[[ -position x y ]]	((compstate[current] >= &&
			  compstate[current] <= y)) &&
			    words=($words[x,y])
			    # Can assignment to $words change $compstate?
			    # If not, this gets messy.

These are so messy I'm not going to try work them out on the fly here:

[[ -string str ]]
[[ -string x str ]]
[[ -class str ]]
[[ -class x pat ]]
[[ -after str ]]
[[ -mafter pat ]]
[[ -between str1 str2 ]]
[[ -mbetween pat1 pat2 ]]

So, let's invent a magic subscript for each of those last two groups.
Choosing `position' as an example, we can do:

    $compstate[position(x)]	if compstate[current] == x, expands
    				to x, else expands to nothing

    $compstate[position(x,y)]	if compstate[current] is between x
				and y, expands to x,y

Now, hey presto:

[[ -position x y ]]		words=($words[$compstate[position(x,y)]])

(That doesn't quite work because of the way tokenization happens in the
[] of $words[...], but you get the idea.)

An alternate idea is to require the (r) or (R) subscript flags to make
the magic syntax active, so you get something like:

    $compstate[(r)position(x)]	 if compstate[current] == x, expands
    				 to $words[x], else expands to nothing

    $compstate[(R)position(x,y)] if compstate[current] is between x
				 and y, expands to $words[x,y]

Similar things could be invented for the rest of the conditions, to make
it simple to have tests without the side-effects and then duplicate those
side-effects in shell code.

}   With these/this option(s) the builtin would set it's return value in 
}   a way that it can be used for testing, but that would only be a
}   `side effect'.

By the way, your doc for the new completion stuff doesn't mention any
of the $argv-modifying etc. side-effects of the condition codes (except
for -iprefix changing $IPREFIX), as far as I can tell.

}   Here we could also add a built-in replacement for the `compsave' and 
}   `compreset' aliases from the example code. Let's give it two options 
}   to save and restore the state of all the special parameters. It
}   would be the user's responsibility to make those calls symmetrical.

I'm still not thrilled with this.

} - Options to get information from the calling completion code,
}   e.g. things like `is there already a valid list', `will menu-
}   completion be used' (which would also test for automenu) and the
}   like.

I suggest making the existing list available in a parameter that can
be tested and modified directly; the sets.afn of that array can do the
equivalent of `compadd`.  For booleans like "are we menu-completing,"
see the AA idea above.

} - Options to say what should be done with the matches genereated,
}   e.g.: `list them', `don't list them', `use menu-completion', `insert 
}   it into the line', etc.

Don't those options belong on complist/compadd?  Or is it not possible
to have some of the matches listed and others not?  (That is, is it a
requirement that there be a single global switch for this stuff?)

By the way, again, one thing I haven't figured out how to do with the
new completion stuff is the equivalent of `compctl -X`.  How does one
add an explanation string, and when does it get shown?

On Feb 20,  4:01pm, Peter Stephenson wrote:
} Subject: Re: Let's finish this new completion stuff
}
} Another issue: ordering of pattern completions.  I think this can already
} be done quite naturally.

Your scheme sounds OK to me, though I reserve judgement until I see it
in action.

Now I have to run.  More long messages next weekend, maybe.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com