Re: change to __path_files and clean up of Functions/Completion needed

Re: change to __path_files and clean up of Functions/Completion needed

[Sven Wischnowsky]

Andrej Borsenkow wrote:

> # This function behaves as if you have a matcher definition like:
> #   compctl -M 'r:|[-.,_/]=* r:|=* m:{a-z}={A-Z} m:-=_ m:.=,' \
> #              'm:{a-z}={A-Z} l:|=* r:|=*'
> # so you may want to modify this.
> 
> I think, it is unacceptable. It is the same, as if users had to modify C
> sources to change compctl behaviour.

I don't think so. This is shell code after all. I agree, though, that
we draw more attention to this -- I added the comment for this but I
know that this may be not enough. This function was initially written
to show some of the nice things that can be done with the new
completion stuff.

> I'd suggest, that __path_files would
> take -M argument to do the same.

Functions like `_path_files' do all the the matching themselves. If we
make them get only a `-M ...' argument, the function would have to
parse it and derive a set of `:gs/.../.../' modifiers from it. This
can get so hard, I don't even want to think about implementing it. But 
I just had another idea: add two autoloaded functions: `_match_test'
and `_match_pattern' or something like that. Functions like
`_path_files' would call the first one at the beginning:

  _match_test || return

This function would test the value of `MATCHER' to see if the calling
function should try to build matches, the standard implementation
would be:

  (( MATCHER == 1 ))

The function `_match_pattern' would be called with the name of a
parameter. It's duty is to modify the value of this parameter so that
it contains the pattern to use for matching. The standard
implementation would be an empty function.

That way the user can supply for the match-specs he uses (more
precisely: for the match specs he wants to use in such functions) by
simply writing files into a directory that is searched before the
standard directory.

As a reply to your other mail: yes, maybe we should already start
splitting the contents of the directory. Currently this was intended
as a repository from which users might copy the files they want into
some other directory.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

RE: change to __path_files and clean up of Functions/Completion needed

[Andrej Borsenkow]

> >
> > I think, it is unacceptable. It is the same, as if users had to modify C
> > sources to change compctl behaviour.
>
> I don't think so. This is shell code after all.

O.K. I hope it is not too late.

The good old completion has some nice features:

1. It works out-of-the-box. Not setup is needed; immediately after
installing zsh users can start with compctl.

2. it is easy to modify completion on-the-fly. What is important, it is the
same command with the same syntax as used in startup files. It is
invariant - 'compctl $(compctl -L cd) cd' is noop as it should be.

3. it is easy to use. It is not a joke. Using compctl amounts simply to
listing what is considered a match - no shell programming is needed. It can
be used immediately after reading manual. (Please, I know about -K. But most
cases are actually quite simple). What's more important, no shell
porgramming is needed for extended/conditional completion. All is limited to
a single command.

4. (this is of more personal nature) It does not pollute namespace (why
don't we have anonymous functions :) It does not suddenly defines
variables/functions/aliases that a completion user is not interested in at
all (and probably, should not know of as well)

I think, it is time to decide, if we want new style completion be for
wizards only or intended for general user community. If it should be of
general use, it should be at least as easy to use, as compctl. Exactly for
these reasons my first reaction was to stay with compctl as a single entry
point and use new style completion to extend it's ability.

What I suggest, is some framework that IMHO makes new completion almost as
easy to use as compctl.

1. use separate array (cpath?) for a completion stuff. Mixing it with fpath
is probably a bad idea (at least, I suddenly get a bunch of autoloaded
functions that actually dont exist :)

2. automatically install at least run-time for new completion (and probably
the completion for zsh builtins) in standard system-wide location and
initialize cpath to point to this location

3. provide a single command to make life easier (compctl is taken, sigh)
This command would need at least

  init - initialize completion. Traverse cpath loading
         definitions. This would allow users to
         override system-wide completion by adding own
         directories to cpath after system location

  load - load a (single) definition. With options to read
         from stdio, single file or a directory. And may be
         directly as argument. Great for testing :)

  dump - printout of current definition(s) for selected command(s)
         (--default-- etc). This should be directly usable as input
         to load.

4. The run-time for completion stuff should _not_ require modification. Even
more so, because these functions are very close to winners of Obfuscated Zsh
Programming Contest :-)

This command should behave as function (or be implemented as such). This
allows to start with emulate -RL zsh, set all needed options (extendedglob
problem :), define any needed local variables without fear to stomp on
user's environment. Anything, that this function exports, should 'course be
documented.

In this case, all pluses of compctl are retained. Users can start with 'xxx
init' (with additional advantage, that they immediately get standard
completions for builtins), modify a simgle completion with 'xxx load' and
look at what's going on with 'xxx dump'.

I really admire the job, Sven, Peter and others have done to improve this
stuff. Let's give it a final touch ...

regards

/andrej

RE: change to __path_files and clean up of Functions/Completion needed

[Sven Wischnowsky]

Andrej Borsenkow wrote:

> > > I think, it is unacceptable. It is the same, as if users had to modify C
> > > sources to change compctl behaviour.
> >
> > I don't think so. This is shell code after all.
> 
> O.K. I hope it is not too late.

Certainly not, we are far from a final version and it is mails like
this one that tells the more development-oriented of us what other
would like to have.

> The good old completion has some nice features:
> 
> 1. It works out-of-the-box. Not setup is needed; immediately after
> installing zsh users can start with compctl.

It's a shift of means, yes. Instead of a builtin you use functions.

> 2. it is easy to modify completion on-the-fly. What is important, it is the
> same command with the same syntax as used in startup files. It is
> invariant - 'compctl $(compctl -L cd) cd' is noop as it should be.

There are the `def*comp' functions and defining completion functions
on-the-fly isn't that complicated (and unusual) either (at least I
though that...). We probably should merge the `def*comp' functions
into `compdef' with options for defining pattern/key completion
functions and give it an option to delete definitions, btw.

> 3. it is easy to use. It is not a joke. Using compctl amounts simply to
> listing what is considered a match - no shell programming is needed. It can
> be used immediately after reading manual. (Please, I know about -K. But most
> cases are actually quite simple). What's more important, no shell
> porgramming is needed for extended/conditional completion. All is limited to
> a single command.

I had the impression that some think `compctl -x' is more complicated
than shell programming. Do not look to much at the things currently
needed, things will improve over time.

> 4. (this is of more personal nature) It does not pollute namespace (why
> don't we have anonymous functions :) It does not suddenly defines
> variables/functions/aliases that a completion user is not interested in at
> all (and probably, should not know of as well)

Not fully to the mark, but: the aliases will almost certainly
disappear in one of the next versions (at least I'm planning to remove 
them when we have the new test-syntax). Also, the global parameters
should probably be changed to `_comps' and `_patcomps', reducing the
probability of name clashes.

> I think, it is time to decide, if we want new style completion be for
> wizards only or intended for general user community. If it should be of
> general use, it should be at least as easy to use, as compctl. Exactly for
> these reasons my first reaction was to stay with compctl as a single entry
> point and use new style completion to extend it's ability.

I'd like to think the other way round: use `compctl' if you like it
and it is enough for you. If yo don't like it or you have a problem
you can't (easily) solve there, use the new completion stuff. And the
Functions/Completion we currently have are *only examples*. You can
easily write a completion widget that does:

  complete() {
    case "$COMMAND" in
    foo) ...complicated stuff for comamnd foo...;;
      ... probably for other commands ...
    *) compcall;;  # use compctl elsewhere
    esac
  }

Something like this will eventually appear as an example in the
manual, I think. The examples we currently have began with my attempt
to show that everything `compctl' does can be done with the stuff I
implemented (actually, I wanted to show that already more is possible, 
`_path_files' for example).

This, btw, is also the reason why I still don't like calling
completion widgets from `compctl'. If once `compctl' isn't enough any
more, use completion widgets. Implementing a simple one is easy and
you can still use your old `compctl's with `compcall' -- no real need
to make this two-way.

> What I suggest, is some framework that IMHO makes new completion almost as
> easy to use as compctl.
> 
> 1. use separate array (cpath?) for a completion stuff. Mixing it with fpath
> is probably a bad idea (at least, I suddenly get a bunch of autoloaded
> functions that actually dont exist :)

If you mean: another builtin path parameter I'm against it. I want the 
user to be completely free in what kind of completion code he wants to 
implement and I don't even want to direct his thinking into a certain
direction by such a parameter. If you mean that `cpath' should be used 
only in shell code, I would be for it but I would want to have the
functions autoloaded -- and for this we need `fpath' (again: this
could be solved if `autoload foo=<path>' would work).

> 2. automatically install at least run-time for new completion (and probably
> the completion for zsh builtins) in standard system-wide location and
> initialize cpath to point to this location

Again, I want the users to decide what kind of completion function
he'd like to use, so we can't pre-install something, especially not
with cpath which the user probably doesn't need.

> 3. provide a single command to make life easier (compctl is taken, sigh)
> This command would need at least
> 
>   init - initialize completion. Traverse cpath loading
>          definitions. This would allow users to
>          override system-wide completion by adding own
>          directories to cpath after system location
> 
>   load - load a (single) definition. With options to read
>          from stdio, single file or a directory. And may be
>          directly as argument. Great for testing :)
> 
>   dump - printout of current definition(s) for selected command(s)
>          (--default-- etc). This should be directly usable as input
>          to load.

Once we renamed `init' and `dump' to `compinit' and `compdump' (what
Peter probably will do in pws-10 -- at least we already talked about
it) and if we have only one function for defining them (`compdef'), we 
have almost reached the state you described. Putting everything into
one function isn't really needed, is it? Especially since dumping
won't be done that often (but adding a function that calls/sources
`init', `dump', and `compdef' depending on the options it gets would
be easy). But then again: this would be an improvement affecting (and
intersting for) only the the current example.

> 4. The run-time for completion stuff should _not_ require modification. Even
> more so, because these functions are very close to winners of Obfuscated Zsh
> Programming Contest :-)

You mean the `-M' problem, right? I moved this into the `_match_*'
functions to make it usable for all functions doing matching. But as a 
side effect this also saves users from the shock they may get when
seeing functions like `_path_files'. We should probably add comments
in `init' pointing to the places where the user might wish to look at
(as Peter did with `dump', hm, so it was only me who forgot to do
this, oops).

> This command should behave as function (or be implemented as such). This
> allows to start with emulate -RL zsh, set all needed options (extendedglob
> problem :), define any needed local variables without fear to stomp on
> user's environment. Anything, that this function exports, should 'course be
> documented.

The examples need global parameters to keep the state that has to live 
longer than the execution of the function. If we had parameter
namespaces, we could cleanly solve this (ksh has them, should we...),
and we could solve it in a way that will not collide with any
parameters used by anyone.
As to the documentation, if been thinking about this -- the question
is: where should we document it? Certainly not in the manual, since
these are only example functions like `zls'.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

completion cleanup discussion + patch

[Peter Stephenson]

(I hope my comments have kept pace with the changes, I've been editing
furiously as new stuff arrives...)

> I would want to have the
> functions autoloaded -- and for this we need `fpath' (again: this
> could be solved if `autoload foo=<path>' would work).

Not entirely, since currently init searches the autoload directories for
completion definitions.  One possibility is simply for init to assume that
its own directory is where the functions are, and add that to the front of
fpath if it's not there.  If the namespace problem is really bad we can
implement Bart's special _* suggestion, perhaps as a shell option.  I also
quite like the flexibility of the function mechanism for completions.

> Once we renamed `init' and `dump' to `compinit' and `compdump' (what
> Peter probably will do in pws-10 -- at least we already talked about
> it) and if we have only one function for defining them (`compdef'), we 
> have almost reached the state you described.

I was going to say what I was thinking of doing.  The way the files are
sorted into directories may want changing.

init -> compinit
dump -> compdump

Make three directories under Functions/Completion, containing

Core:  stuff the user will need and probably won't want to touch:
compinit, compdump, _files, _path_files, _main_complete, _normal

Base:  stuff the user will probably want, but doesn't need or may want to
edit:  _default, _match_pattern, _match_test, _subscript, _precommand;
maybe some of the more basic functions like _setopt, maybe even _cd, any
offers?

User:  `lucky dip' for everything else.

Probably a README in the top level directory would be a nice touch when
things get settled down.

While I remember: init and dump miscount the number of files which are to
be loaded when there are ones with the same name in different directories,
e.g. if there is a personalised _default as well as a system wide one it
thinks there are two different files to load, whereas only one will be
found.  This isn't as bad as I first though since init and dump count the
same way (though I could have sworn I was getting too many dumps at one
point), but anyway the patch below uses typeset -U and the basename of the
file so it should get the count right (except that . isn't ignored in
fpath).

> Putting everything into
> one function isn't really needed, is it? Especially since dumping
> won't be done that often (but adding a function that calls/sources
> `init', `dump', and `compdef' depending on the options it gets would
> be easy). But then again: this would be an improvement affecting (and
> intersting for) only the the current example.

I'd have thought it would be OK just to have one init file that gets
sourced at the start, and then one command for management (e.g. the
extended compdef).  compdef should probably also let you see how things are
defined, so that the user doesn't need to look at the arrays directly.  I
don't think having compinit as a function is a good idea, even apart from
the typeset's having local effect (I could write a flag not to make the
variable local, now I've neatened bin_typeset() up marginally), since you
only need to run it once and it takes up memory for the whole session.

> > 4. The run-time for completion stuff should _not_ require
> > modification. Even more so, because these functions are very close to
> > winners of Obfuscated Zsh Programming Contest :-)
> 
> You mean the `-M' problem, right? 

I agree the run-time stuff shouldn't *need* modification, but that's
already the case.  It's a slight worry that defining matchers is different
for _path_files, so _match_pattern is probably destined to be slightly
strange, though there are comments on how it should work (I have to admit
I've only vaguely registered what's going on there).

> > This command should behave as function (or be implemented as such). This
> > allows to start with emulate -RL zsh, set all needed options (extendedglob
> > problem :), define any needed local variables without fear to stomp on
> > user's environment. Anything, that this function exports, should 'course be
> > documented.
> 
> The examples need global parameters to keep the state that has to live 
> longer than the execution of the function. If we had parameter
> namespaces, we could cleanly solve this (ksh has them, should we...),
> and we could solve it in a way that will not collide with any
> parameters used by anyone.

(Since then we have underlines in front of comps and patcomps).  I don't
think parameter namespaces would be too hard to implement; just keep
paramtab in another hash table, but I don't have a version of ksh that does
this.  There are actually only four or so variables used (apart from
specials, which are local to completion functions and don't clash with the
use of the same variable outside).

> As to the documentation, if been thinking about this -- the question
> is: where should we document it? Certainly not in the manual, since
> these are only example functions like `zls'.

This is part of a wider problem, which Andrej mentioned, about
installation.  There should at least be an install target for this, though
I suppose people won't want it done with a basic install.  In that case we
need to arrange so that simply sourcing one file will set up all the stuff
that has been installed; maybe a configure option could pick 'core',
'basic' or 'all' to choose how much to install in a single target
directory, then an $fpath check in compinit would do the trick.  Then we
could have a separate manual for the standard function suite which gets
installed with all this stuff

I was thinking about a similar mechanism for the zftp functions, so
similarly I would write one for the zftp function suite and install those
together if requested.  We would definitely need wider feedback from
potential installers to get this right.


--- Functions/Completion/dump.bk	Thu Feb 25 17:23:03 1999
+++ Functions/Completion/dump	Thu Feb 25 17:33:58 1999
@@ -16,7 +16,8 @@
 
 _d_file=${COMPDUMP-${0:h}/init.dump}
 
-_d_files=( ${^~fpath}/_*~*~(N) )
+typeset -U _d_files
+_d_files=( ${^~fpath}/_*~*~(N:t) )
 
 print "#files: $#_d_files" > $_d_file
 
--- Functions/Completion/init.bk	Thu Feb 25 17:23:03 1999
+++ Functions/Completion/init	Thu Feb 25 17:33:56 1999
@@ -211,7 +211,8 @@
   setopt extendedglob
 fi
 
-_i_files=( ${^~fpath}/_*~*~(N) )
+typeset -U _i_files
+_i_files=( ${^~fpath}/_*~*~(N:t) )
 _i_initname=$0
 _i_done=''
 
-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: completion cleanup discussion

[Bart Schaefer]

Andrej Borsenkow writes:
 > 
 > The good old completion has some nice features:
 > 
 > 1. It works out-of-the-box. Not setup is needed; immediately after
 > installing zsh users can start with compctl.

One of the things that STILL annoys me about 3.1.2+ is that all the
default compctl settings are gone.  Completion of anything but files
is entirely useless "out of the box."

 > 2. it is easy to modify completion on-the-fly. What is important, it
 > is the same command with the same syntax as used in startup files. It
 > is invariant - 'compctl $(compctl -L cd) cd' is noop as it should be.
 >
 > 3. it is easy to use. It is not a joke. Using compctl amounts simply to
 > listing what is considered a match - no shell programming is needed.

I don't think anyone is proposing removing the simple compctl options.
Even the complex `compctl -x ...` bits will hang around until the new
stuff has been through at least one "official" "production" release,
I expect.

 > What I suggest, is some framework that IMHO makes new completion almost
 > as easy to use as compctl.
 > 
 > 1. use separate array (cpath?) for a completion stuff. Mixing it with
 > fpath is probably a bad idea

I disagree.  However, perhaps the `autoload` commands for the completion
function names could be delayed until nearer the time they're needed.

 > 2. automatically install at least run-time for new completion (and
 > probably the completion for zsh builtins) in standard system-wide
 > location

I've long thought that copying the Functions/ subdir to somewhere in the
configure-selected $(libdir) would be a good idea, regardless of the new
completion code.  I always do so manually when I install zsh myself.

As for enabling default completions, no argument from me; see rant above.

 > 3. provide a single command to make life easier

Given (2), I think it's fine for this to be a shell function rather than a
builtin.  Perhaps the init function needs to go in the "regular" fpath and
all the auxiliary functions into subdirs that aren't added to fpath until
the init function runs.

 > 4. The run-time for completion stuff should _not_ require modification.

This would be ideal, but ... look at Misc/compctl-examples.  There are in
some cases two or three ways to get not-quite-identical completions in
different circumstances.  Further, there are lots of minor differences in
things like completion of command arguments (how many variants of tar,
find, dd and ls are there?) that require tweaking on different platforms
or even different installations of supposedly the same platform.

A better goal should be that the completion run-time should be easy to
modify when necessary -- which was the intent of moving it into shell
functions in the first place.

Sven Wischnowsky writes:
 > > I think, it is time to decide, if we want new style completion be for
 > > wizards only or intended for general user community.
 > 
 > I'd like to think the other way round: use `compctl' if you like it
 > and it is enough for you. If yo don't like it or you have a problem
 > you can't (easily) solve there, use the new completion stuff.
 > 
 > This, btw, is also the reason why I still don't like calling
 > completion widgets from `compctl'.

If I want to use exactly one completion for which there already happens
to be a widget implementation, why should I have to redo all my key
bindings and load other extra functions and dispatch-table arrays just
to get at it?

If we're working out all these complex widget functions, they at least
should be maximally re-usable.

Seriously, although I instigated this whole effort, it remains the case
that 98% of my own completion needs are covered by simple compctls.  If I
want to complete something fancier, say cvs, I use ESC-q to push the
current command line, then run an autoloaded shell function that installs
my set of `compctl -x ...` commands, and then I finish completing when
zle pops the buffer stack at the next prompt.

 > The examples need global parameters to keep the state that has to live
 > longer than the execution of the function. If we had parameter
 > namespaces, we could cleanly solve this (ksh has them, should we...),

What do ksh parameter namespaces look like, syntactically?

 > As to the documentation, if been thinking about this -- the question
 > is: where should we document it?  Certainly not in the manual, since
 > these are only example functions like `zls'.

Why not in the manual?  The Perl manual has sections that are entirely
dedicated to "template" implementations of various Perl idioms.  If we do
arrange for installation in $(libdir) of the Functions/ directory, there's
no reason not to treat it as a real part of zsh and have a manual section
devoted to it.

Re: completion cleanup discussion

[Sven Wischnowsky]

Bart Schaefer wrote:

> If I want to use exactly one completion for which there already happens
> to be a widget implementation, why should I have to redo all my key
> bindings and load other extra functions and dispatch-table arrays just
> to get at it?

Touche.

> If we're working out all these complex widget functions, they at least
> should be maximally re-usable.

Certainly, yes.

> What do ksh parameter namespaces look like, syntactically?

I had this from the bash-FAQ, but they aren't really supported, it
seems. But still: in ksh one can use dots in parameter names (that's
what they are referring to in the bash-FAQ). I can't find any other
fancy stuff you can do with prefixes (i.e. namespaces) in the ksh
manual now that I had a look at it. But that shouldn't stop us from
adding real namespace with the same syntax (`.complete.comps',
`.complete.patcomps') if we feel that this is interesting to have. I'm 
not proposing to implement this, though.

> Why not in the manual?  The Perl manual has sections that are entirely
> dedicated to "template" implementations of various Perl idioms.  If we do
> arrange for installation in $(libdir) of the Functions/ directory, there's
> no reason not to treat it as a real part of zsh and have a manual section
> devoted to it.

What I really meant was: not in the completion manual. As an extra
manual section (included in zshall, info, ...): ok, yes.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de