Re: new completion behaviour version 2

Re: new completion behaviour version 2

[Sven Wischnowsky]

Bart Schaefer wrote:

> ...
> 
> 
> If we're going to add associative arrays anyway (which I take it would
> just be some kind of parameter syntax for frobbing a hash table) then
> I'd just as soon have the set of scalars (CMDSTR, CURWORD, etc.) be in
> such an associative array.

Interesting idea, I'll have to think about this a bit more...

> 
> As for the words of the current command ... I'm thinking I'd like them
> to come in as the positional parameters.  We already have syntax for
> altering the $* array (shift, set, and 3=foo among other things); it
> should be possible to recapture the contents after the function exits;
> and it is an extremely intuitive way to refer to the command line.
> 
> (You can almost do this today by using
> 	read -Ac argv ; shift
> in a compctl -K function, except that stomps the $1/$2 prefix/suffix.)

Yes, sounds good.

> 
> } For producing the actual matches there is another builtin, say
> } `compadd'.  [...]
> } with a different way to handle control flags (-U, -q,
> } grouping,...). We could make these flags be used `from now on' and
> } probably have a way to save/restore the current set of control flags
> } (whose state would be stored together with the matches produced),
> } e.g. (in an easy-to-read syntax):
> } 
> }   compadd -push
> }   compadd -quote -suffix '/'
> }   compadd -files               # optional glob pattern here
> }   ...
> }   compadd -pop
> 
> The main problem I have with this is that it's too easy to forget the
> "pop".  If you want to do something of this sort, it really needs its
> own syntax.  One way to accomplish that without having to muck about
> with the zsh parser is to supply ONLY the "push" version of the command,
> but define it to extend exactly to the end of the current function, as
> for "local" variables and "setopt localoptions".  The user can control
> scope by defining functions that call one another.
>  

I wasn't too happy about this, just because this `pop'. On the other
side I think that the function scope may be too big for this. But
maybe we can just do what you suggest and if we find out that we would 
like to have a possibility to `pop' to a previous state we can add it
later.

> } I.e. use the normal shell tests and the special variables. But there
> } is a problem with this: with compctl the `-x' - tests not only test
> } for a certain condition, but also report some information back,
> } namely: the length of an prefix that should be ignored in the
> } completion string and (with `r[a,b]') a restriction on the command
> } words that should be used with `compctl -L'.
> 
> Pardon my density, but I don't follow the connection between r[a,b] and
> `compctl -L'.  Do you mean `compctl -l'?

Yep, of course, this was a typo.

> 
> }   complist -k
> }   complist -c
> 
> Incidentally, with respect to menu completion, I don't think the widget
> should get involved directly in cycling through the menu choices.  (I
> can easily be convinced otherwise, but right now I don't see a compelling
> argument.)  Once the the control flags have indicated that you have a
> menu of choices, don't invoke the widget again until the prefix changes.
> That is, the widget can say "here are the completions, and here is where
> you insert them" but then let hardwired code take over cycling through
> the menu if there are multiple matches.

Interesting point of view. I thought we wanted to give the user the
possibility to build widgets that have full control over the
completion mechanism, i.e. the possibility to re-implement all the
things the completion code offers now.
But again, yes, we probably should start with a minimal solution and
improve that later if we need/want to have it.

> ...
> 
> Recexact can be emulated entirely by the wiget by simply producing only
> a single match.  Listambiguous can be a flag attached to the generated
> list of matches, can it not?
> 

Yes. The question is: should the completion code do this automatically 
or should the user request it explicitly.

> ...
> 
> What I had in mind a while back was an interface to add one or more
> matches with options for the controls associated with that collection
> of matches; plus interfaces (possibly commands, possibly variables) to
> obtain the information that's currently only available to compctl and
> pass it to the add-matches interface.
> 
> So for example there would be a command or an array variable that gives
> the names of all the disabled commands, and you'd do
> 
> 	compadd $disabled
> 
> instead of
> 
> 	compadd -dFB
> 
> (That's probably not the best example, but just to give the idea.)
> 

I like this. But I wouldn't like to have to do

    compadd $(compinfo -disabled)

The $(...) is what I don't like. Using variables would be much more
effective, but these would be special variables and there seems to be
reluctance against more special variables (not from me, I always liked 
them).

> ...
>
> I agree, and I liked your comptest idea.  For example, something like
> 
> 	if comptest index -1 / ; then
> 
> to mean what n[-1,/] accomplishes now.  (There could even be a flag in
> [[ ]] to mean "test completions", e.g., [[ -C index -1 / ]] for the
> above.)

I didn't think about changing [[...]], interesting. (Although that
would require changes in completely differnt parts of the code, of
course.)

> ...
> 
> Recursively invoking completion functions is a good idea, and IMNSHO is
> the best argument that I can think of for passing the command words as
> the positional parameters.  (There is of course still the issue of $0,
> but that's not too terrible.  Just assign to CMDSTR or whatever takes
> its place in the associative array, before calling the other function.)

Of course. My suggestion was to have a top-level function that calls
other functions which might call other functions...


> 
> On Nov 3,  4:58pm, Sven Wischnowsky wrote:
> } Subject: new completion behaviour version 2
> }
> } The coding scheme for xor'ed completion is trivial. The code for `-x'
> } stuff will be done using `if'-tests with the information stored in
> } local variables. We then need a way to notify the completion code
> } about two things it will have to use (to be able to do the same things 
> } `-x' can): the length of an ignored prefix of the current word and
> } (for `-L') which part of the words-array should be taken as command
> } words.
> 
> I still think you must mean `-l' there.  Un-confuse me, please.

Yes, this was `copying a typo without further thinking'.


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: new completion behaviour version 2

[Bart Schaefer]

On Nov 9, 10:04am, Sven Wischnowsky wrote:
} Subject: Re: new completion behaviour version 2
}
} Bart Schaefer wrote:
} > 
} > That is, the widget can say "here are the completions, and here is where
} > you insert them" but then let hardwired code take over cycling through
} > the menu if there are multiple matches.
} 
} Interesting point of view. I thought we wanted to give the user the
} possibility to build widgets that have full control

Well, yes, that's why I said I could easily be convinced otherwise; but:

} > Listambiguous can be a flag attached to the generated
} > list of matches, can it not?
} 
} Yes. The question is: should the completion code do this automatically 
} or should the user request it explicitly.

It's quite a burden on the widget writer to check for LISTAMBIGUOUS and
spit out the listing if so.  On the other hand, the widget ought to be
able to emit a listing (and have zle deal with it properly) even when
the LISTAMBIGUOUS is not set.

It seems to me there are three ways to deal with this:

1. The widget function always produces the entire list of matches, with
   an annotation for how and where to insert them.
   + Returning a single value forces recexact.
   + The function always executes in the same way and always returns
     the same values, so its easier to write.
   + Efficient, compiled builtin code handles menus and listings, and
     tests for menucomplete, automenu, autolist, and listambiguous.
   + There's only one call to interpreted user code per set of results.
   - The annotations have to describe prefix/suffix removal, replacement,
     or addition for correct insertion on the command line.
   - It may be unclear exactly what to display in a listing, i.e., what
     if any prefix/suffix changes to perform on the returned strings.

2. The widget function always produces one match at a time, with an
   annotation for how and where to insert it.
   + Builtin code still tests for menucomplete, automenu, autolist, and
     listambiguous.
   + The function can change the insertion annotations on every match.
   - The function has to be told which match to produce; it's harder to
     write efficient functions because it's necessary e.g. to branch on
     whether the full list has previously been generated and on whether
     the prefix has changed.
   - Lots of calls to interpreted code when generating listings and when
     cycling through menus.
   - It requires either an annotation or two calls to the function to
     force recexact.
   - It still may be difficult to determine what to display for a listing.

3. The widget function not only produces the matches, but also either
   inserts them or produces a listing as appropriate.
   + The function can drive zle directly (e.g., through $BUFFER) to do
     complex insertions/changes.
   + No annotations are needed.
   + The function can produce exactly what should appear in the listing.
   - The function has to be told what match to produce or to produce a
     complete list of possible matches; it's even harder to write simple
     efficient functions.
   - Lots of calls to interpreted code when generating menus.
   - The widget writer has to figure out all the prefix/suffix changes
     to perform on every insertion, including dealing with menus (but an
     implicit "undo" of the last insertion before calling the function
     for the next item could make that easier).
   - The widget itself has to test automenu, autolist, and listambiguous.

There are of course points somewhere in between each of these choices, but
they all have something of those flavors.  I think the choice of which to
provide depends on what we can come up with in terms of the annotations
needed by the first two.

} > 	compadd $disabled
} > instead of
} > 	compadd -dFB
} 
} I like this. But I wouldn't like to have to do
} 
}     compadd $(compinfo -disabled)

I agree; if the choice is between that and something like

	compadd disabled

then I'd take the latter.

} > [[ ]] to mean "test completions", e.g., [[ -C index -1 / ]] for the
} > above.)
} 
} I didn't think about changing [[...]], interesting. (Although that
} would require changes in completely differnt parts of the code, of
} course.)

Yes, but I've been thinking that perhaps the tests supported by [[ ]]
should be extensible through the modules mechanism anyway.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: new completion behaviour version 2

[Bart Schaefer]

Finally catching up on this part, too ... there's a bit of skipping
around in the following because I don't want to rehash things that
were already discussed, such as using an associative array instead of
a "compfunc" command.

On Nov 2, 11:36am, Sven Wischnowsky wrote:
} Subject: completion behaviour (was: zsh-workers: zsh-3.1.5 released)
}
} We use zle widgets, probably a new kind of widgets that uses some more 
} special variables, like:
} 
}   CMDSTR   - the name of the current command, with some special values
}              it could be used for certain contexts (conditions, math
}              expressions, ...)
}   words    - the words of the current command (an array)
}   CURWORD  - the number of the current word (for words[CURWORD])
}   PREFIX   - the prefix of the current word
}   SUFFIX   - ...
}   NMATCHES - the number of matches produced so far

If we're going to add associative arrays anyway (which I take it would
just be some kind of parameter syntax for frobbing a hash table) then
I'd just as soon have the set of scalars (CMDSTR, CURWORD, etc.) be in
such an associative array.

As for the words of the current command ... I'm thinking I'd like them
to come in as the positional parameters.  We already have syntax for
altering the $* array (shift, set, and 3=foo among other things); it
should be possible to recapture the contents after the function exits;
and it is an extremely intuitive way to refer to the command line.

(You can almost do this today by using
	read -Ac argv ; shift
in a compctl -K function, except that stomps the $1/$2 prefix/suffix.)

} For producing the actual matches there is another builtin, say
} `compadd'.  [...]
} with a different way to handle control flags (-U, -q,
} grouping,...). We could make these flags be used `from now on' and
} probably have a way to save/restore the current set of control flags
} (whose state would be stored together with the matches produced),
} e.g. (in an easy-to-read syntax):
} 
}   compadd -push
}   compadd -quote -suffix '/'
}   compadd -files               # optional glob pattern here
}   ...
}   compadd -pop

The main problem I have with this is that it's too easy to forget the
"pop".  If you want to do something of this sort, it really needs its
own syntax.  One way to accomplish that without having to muck about
with the zsh parser is to supply ONLY the "push" version of the command,
but define it to extend exactly to the end of the current function, as
for "local" variables and "setopt localoptions".  The user can control
scope by defining functions that call one another.
 
} I.e. use the normal shell tests and the special variables. But there
} is a problem with this: with compctl the `-x' - tests not only test
} for a certain condition, but also report some information back,
} namely: the length of an prefix that should be ignored in the
} completion string and (with `r[a,b]') a restriction on the command
} words that should be used with `compctl -L'.

Pardon my density, but I don't follow the connection between r[a,b] and
`compctl -L'.  Do you mean `compctl -l'?

}   complist -k
}   complist -c

Incidentally, with respect to menu completion, I don't think the widget
should get involved directly in cycling through the menu choices.  (I
can easily be convinced otherwise, but right now I don't see a compelling
argument.)  Once the the control flags have indicated that you have a
menu of choices, don't invoke the widget again until the prefix changes.
That is, the widget can say "here are the completions, and here is where
you insert them" but then let hardwired code take over cycling through
the menu if there are multiple matches.

} `complist' should also offer ways to access some more information,
} especially we need a way to decide if the recexact and listambiguous
} behaviour should be emulated.

Recexact can be emulated entirely by the wiget by simply producing only
a single match.  Listambiguous can be a flag attached to the generated
list of matches, can it not?

} Finally we need a way to make the completion code list the matches
} found. This could be done with `complist' or by using the return code
} of the completion widget.

I do think the return code should be meaningful, but only in a success/
failure sense like any other shell function; e.g. if the widget returns
nonzero and there are no other functions left to try, then feep).

On Nov 2, 11:29am, Zefram wrote:
} Subject: Re: completion behaviour (was: zsh-workers: zsh-3.1.5 released)
}
} >For producing the actual matches there is another builtin, say
} >`compadd'.
} 
} OTOH, I think this perhaps should be split into two builtins.  One to
} add sets of matches in a manner based on the compctl syntax (this could
} even be part of the compctl module, and use the compctl code to handle
} *all* the compctl options).  Another, more basic builtin would merely
} provide an interface for adding a single match (or a set of matches),
} with options to separately control each part of the match structure.

What I had in mind a while back was an interface to add one or more
matches with options for the controls associated with that collection
of matches; plus interfaces (possibly commands, possibly variables) to
obtain the information that's currently only available to compctl and
pass it to the add-matches interface.

So for example there would be a command or an array variable that gives
the names of all the disabled commands, and you'd do

	compadd $disabled

instead of

	compadd -dFB

(That's probably not the best example, but just to give the idea.)

} >is a problem with this: with compctl the `-x' - tests not only test
} >for a certain condition, but also report some information back,
} >namely: the length of an prefix that should be ignored in the
} >completion string
} 
} This should be part of the match structure, and the user should be able
} to control it explicitly.

I agree that the user should be able to control it, but it may be easier
for the completion code to compute a default.  Expose power you can use
if you want it, but don't require that you use it.

On Nov 2,  1:00pm, Sven Wischnowsky wrote:
} Subject: Re: completion behaviour (was: zsh-workers: zsh-3.1.5 released)
}
} Partly agreed (see above), but I would still like to give the user
} some more support. Think of `r[-exec,;]'. The user would have to do the
} matching and before (s)he can produce matches (s)he would have to tell
} the completion code which command words should not be used. This can
} get a bit nasty.

I agree, and I liked your comptest idea.  For example, something like

	if comptest index -1 / ; then

to mean what n[-1,/] accomplishes now.  (There could even be a flag in
[[ ]] to mean "test completions", e.g., [[ -C index -1 / ]] for the
above.)

The issue I see is with keeping the results sane should the user change
CURWORD (or whatever represents it).  But that's no more serious than
answering the question "where do I zle-insert this match if the widget
has repositioned the cursor?"

On Nov 2, 12:31pm, Zefram wrote:
} Subject: Re: completion behaviour (was: zsh-workers: zsh-3.1.5 released)
}
} What you need there is for the user-defined completion function, after
} recognising the "-exec", to recursively invoke the completion mechanism
} on the restricted range of words.  As long as completion can be invoked
} recursively, there's no need for more specific support for this type
} of construct.  (Consider also how to properly handle completion of the
} word after "sh -c".)

Recursively invoking completion functions is a good idea, and IMNSHO is
the best argument that I can think of for passing the command words as
the positional parameters.  (There is of course still the issue of $0,
but that's not too terrible.  Just assign to CMDSTR or whatever takes
its place in the associative array, before calling the other function.)

On Nov 3,  4:58pm, Sven Wischnowsky wrote:
} Subject: new completion behaviour version 2
}
} The coding scheme for xor'ed completion is trivial. The code for `-x'
} stuff will be done using `if'-tests with the information stored in
} local variables. We then need a way to notify the completion code
} about two things it will have to use (to be able to do the same things 
} `-x' can): the length of an ignored prefix of the current word and
} (for `-L') which part of the words-array should be taken as command
} words.

I still think you must mean `-l' there.  Un-confuse me, please.

} To be
} able to correctly save/restore this type of information we will need a 
} way to get the current state in a way that can be used to restore it
} later

Once again I suggest using function scope; save and restore like local
variables and setopts.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: new completion behaviour version 2

[Bart Schaefer]

I have some comments I'd like to make on this, but I've been quite busy the
last 24 hours and will continue to be until sometime this afternoon.  I'll
try to respond as soon as I can.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

new completion behaviour version 2

[Sven Wischnowsky]

Ok, here is the update I promised...

We need some kind of C-wrapper function around the completion code, so 
we either have one new builtin widget that calls a shell function to
do the real work or we have a new type of user-defined-widgets which
do the work, probably calling other shell functions to produce the
matches.

Information about the context for producing the matches is normally
obtained in the top level shell function (or the new-type widget),
stored in local variables and thereby made accessible in called
functions, but of course users can implement this differently. Of the
information that is important to have the command words and the index
into this array are already available via `read'. The prefix and
suffix can be made accessible via `read', too, or as arguments when
calling the top level function (as with compctl-K-functions). Another
interesting information is the environment, i.e. `condition',
`redirection', and the like, which could be made accessible like the
prefix/suffix. To be able to implement things like xor'ed completion
in a shell function we need a way to access the number of matches
produced so far. This information differs from the other ones because
it changes whenever matches are added (of course). If we avoid special 
variables for all the other things we shouldn't add one for this, so
the users will have to call `read' repeatedly to get the current
number of matches.

Since most people will put their completion definitions in shell
functions, we need a way to at least map strings (command names) to
other strings (function names). This will be done either with
associative arrays or with the `$funcs[$cmds[(i)$CMDNAME]]'-trick. For 
command name patterns we can start using a simple array until we find
something more clever.

For producing matches we have a new builtin that uses the
compctl-flags (and part of it's code) to specify which matches should
be generated. Either this builtin or another new builtin will support
a way to directly set all the different things that are stored
together with the matches.

The coding scheme for xor'ed completion is trivial. The code for `-x'
stuff will be done using `if'-tests with the information stored in
local variables. We then need a way to notify the completion code
about two things it will have to use (to be able to do the same things 
`-x' can): the length of an ignored prefix of the current word and
(for `-L') which part of the words-array should be taken as command
words. Of course we should do this in a very general manner, giving
users the possibility to report restrictions `-x' doesn't use. To be
able to correctly save/restore this type of information we will need a 
way to get the current state in a way that can be used to restore it
later (so that independently developed completion functions can
restore the restrictions to the state they were in when the function
was called). All this could be done with the builtin that produces the 
matches (or the one that gives full control over the match structure
if we put this into a separate builtin).

Finally we need a way to put a match in the command line and to list
matches. Listing is simple (one way or the other). More interesting is 
putting matches on the line. I still find it tempting to think about
ways to use the special variables [LR]BUFFER for this. The only
problem with this are automagically removed suffixes (getting the
prefix/suffix from the completion code is simple nowadays). So if we
add a way to make removable suffixes completely user-controllable this 
may be the best solution (and giving the user full control over this
is interesting in itself, just think about the `compctl ... -S/ -q'
example from Bart). So we would end up with a builtin that allows one
to retrieve information about the matches (the prefix/suffix to
insert, and probably all the other things stored with the matches),
starting a new set of matches, listing them, and so on. With this
builtin we could also let users remove specific matches, change them,
etc. All this could later be added if we find uses that make them
desirable.

So, I hope this approach is simple enough. Comments?

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de