Completion quirks

Completion quirks

[Wayne Davison]

My normal completion options have been to have "auto_list" set and nothing
else.  With this setup there has been a long-standing quirk that if you
hit the completion key with a string that exactly matches a possible
completion, zsh would choose that match even if other (longer) matches
exist.  Just today I installed hzoli10.1+ (including patch2) and its
behavior is different but also wrong -- it chooses one of longer matches.

For example, in one directory of mine I have some picture processing
binaries:

ppmquant	pp	ppr	pp.sh	ppmmerge	ppmhist

If I type "ls pp" and hit tab using zsh 2.6 beta8, it generates a space.
Hitting tab again would start a new completion.

If I use zsh 2.6 beta10hzoli10.1+, it generates "ls ppmhist" -- the last
item in the (unsorted) directory (I don't know if that's significant).
If I hit tab a second it seems to try to treat it as a menu complete (even
though it's not turned on), but fails to update correctly -- I get this:
"ls ppmhist mhist".

Turning on "menu_complete" in hzoli10.1+ behaves exactly the same way.

Turning on "auto_menu" without menu_complete also fails, but in a
different way:  it displays "ls ppmhist " (no beep).  One more tab beeps
and backs up a space.  Another tab displays "ls ppmhist .sh".  Another
tab displays "ls ppmhist mhist".

Turning on "auto_menu" in combination with menu_complete actually works,
even though the documentation says that auto_menu is overridden by
menu_complete.  The behavior in this case is to beep and list the choices
on the first tab, and then start menu-completion with proper screen
updating on subsequent tabs.  I like this behavior a lot, so I've made
it my new default, so for the moment I'm happy.

I'm running zsh this under SCO 3.2v4.2 using gcc 2.7.0 -- is anyone
else having a problem with this?

..wayne..

Re: Completion quirks

[Zefram]

>My normal completion options have been to have "auto_list" set and nothing
>else.  With this setup there has been a long-standing quirk that if you
>hit the completion key with a string that exactly matches a possible
>completion, zsh would choose that match even if other (longer) matches
>exist.

The REC_EXACT option does this.  It would be helpful, when posting this
kind of query, for you to include a *complete* list of option
settings.  Completion is affected by a great many options.

>        Just today I installed hzoli10.1+ (including patch2) and its
>behavior is different but also wrong -- it chooses one of longer matches.

Curious.  Setting REC_EXACT, I got the same behaviour.  My non-hzoli beta10 plus lots of patches picks the right match, as does plain beta10.

[...]
>If I use zsh 2.6 beta10hzoli10.1+, it generates "ls ppmhist" -- the last
>item in the (unsorted) directory (I don't know if that's significant).
>If I hit tab a second it seems to try to treat it as a menu complete (even
>though it's not turned on), but fails to update correctly -- I get this:
>"ls ppmhist mhist".
>
>Turning on "menu_complete" in hzoli10.1+ behaves exactly the same way.
>
>Turning on "auto_menu" without menu_complete also fails, but in a
>different way:  it displays "ls ppmhist " (no beep).  One more tab beeps
>and backs up a space.  Another tab displays "ls ppmhist .sh".  Another
>tab displays "ls ppmhist mhist".

With hzoli10 I got this behaviour, and also with my own patched beta10
(though starting with the correct match).  This problem is almost
certainly a result of my completion list patches (which I can't test
with every option, though I did give it a bit of a work out).  Plain
beta10 for me misbehaved on the second completion, much in the way you
describe, so I think it is safe to say that REC_EXACT is, for the
moment, broken.  I'll see what I can do...

-zefram

P.S.  Can we expect an beta11-test release any time soon, Richard?

Re: about beta11

[Richard Coleman]

> -zefram
> 
> P.S.  Can we expect an beta11-test release any time soon, Richard?
> 

I've been working on some ambitious changes for beta11.  As usual, it
is taking longer than I expected.  I was hoping to have a test
release available this week, but now I'm not sure.  I'll be leaving
for vacation (just one week) at the end of next week (Jun 22).

One of the things I'm doing for beta11 is splitting the hash table
cmdnamtab into three separate tables for shell functions, builtins,
and external commands.  This fixes some of the weird command name
space problems that would have been difficult to fix any other way.
This required lots of changes to builtin.c and exec.c (actually just
execcmd).  I've already fixed everything in builtin.c (which lead to
much cleaner code in many of the builtins).  What is taking so long
is the changes to exec.c since it is quite complicated and has very
little in the way of comments.

One of the benefits of this change is that is allows much more
precise control of the various hash tables.  There are a few
changes to various builtins to deal with this.

1) enable/disable by default will only work on builtin functions
2) enable -f, disable -f will enable/disable shell functions
3) enable -r, disable -r will enable/disable reserved words
4) unhash by default will only unhash external commands.
5) unhash -f will remove shell functions.  unfunction is now
   just an alias for unhash -f.
6) unhash -a will remove aliases.  unalias is now just an
   alias for unhash -a.

Zefram, were you the one that wrote the code for named directories?
Is there anything preventing us from using a hash table for this
rather than a linked list (which is what it currently uses)?  This
way we could reuse all the new hashtable code.  For instance, then
unhash -d could be used to remove entries from the named directories
table.

rc

Re: about beta11

[Zefram]

-----BEGIN PGP SIGNED MESSAGE-----

>One of the things I'm doing for beta11 is splitting the hash table
>cmdnamtab into three separate tables for shell functions, builtins,
>and external commands.  This fixes some of the weird command name
>space problems that would have been difficult to fix any other way.

Does this mean that command completion, equals substitution and the
hash builtin will all agree on whether a command exists?

>1) enable/disable by default will only work on builtin functions
>2) enable -f, disable -f will enable/disable shell functions
>3) enable -r, disable -r will enable/disable reserved words

Great.  Atm, one can disable functions, but not reserved words or
external commands (though they will be added to the disable list), but
can't enable anything other than a builtin.  It's all rather silly.

>4) unhash by default will only unhash external commands.
>5) unhash -f will remove shell functions.  unfunction is now
>   just an alias for unhash -f.
>6) unhash -a will remove aliases.  unalias is now just an
>   alias for unhash -a.

Neat.

>Zefram, were you the one that wrote the code for named directories?

Some of it, yes.  hash/rehash -d is my code.

>Is there anything preventing us from using a hash table for this
>rather than a linked list (which is what it currently uses)?

'fraid so.  The named directory table is sorted.  (Actually, I'm pretty
sure it's a big array, not a linked list, hence the occasional comment
that calls it a hash table even though there's no real hashing
involved.)  It has to be sorted so that %~ in prompts gives the
shortest possible name for the directory.  That capability isn't used
anywhere else, but unless you have a brilliant way to do that with a
hash table then hash -d will have to stay as an exception.

I think it might make the builtins a bit neater if hash/rehash -d, and
unhash -d if it gets added (it should be), were handled by separate
functions from the normal bin_{,re,un}hash.  There would be a statement
"if(ops['d']) return dir_hash(whatever)" at the beginning of each
bin_*hash function, and the rest of the function would be the normal
hash table stuff.

It *might* also be good to merge bin_hash and bin_rehash, and have
rehash merely as an alias to hash -r.  It should be documented that way
anyway.

Speaking of builtin aliases, is "where" (whence -ca) going to appear at
some point?  I sent a one-line patch for it, and Zoltan's release has
that patch plus documentation.  It is genuinely useful to have a
builtin that does a whence -a (I alias it).  And has anyone considered
adding the csh "glob" command, which would be something like "print
 -rN"?  (Less useful, but it occurred to me as being appropriate.  Does
anyone use glob these days?)

At this point I was going to request that the remaining disgusting bits
of inconsistent ! quoting be fixed, as Zoltan's release contains a
patch to make it just a normal character.  That was certainly the case
a few betas ago.  But then I tested, just to make sure:

% echo $ZSH_VERSION
2.6-beta10
% echo \\!
!
% # Double escaping.  Ugh.
% zshz
% echo $ZSH_VERSION
2.6-beta10-hzoli10.1
% echo \\!
zsh: Oops. You discovered a bug in hungetc

I'd love to see this quoting mess sorted out -- once Zoltan has the
above problem sorted out.  In any case, the man page that says that SQs
will not quote !s is now wrong, thankfully.

Finally, I remember there being a very simple (two line) patch that
made AUTO_REMOVE_SLASH remove an automatically added slash if the next
character typed is a slash, as well as if it's a newline.  I think the
consensus was in favour of it being used.  (Its purpose is to stop a
menu completion on a directory name and continue completion on
pathnames under that directory, without having to add a character and
then delete it again.)  Could this be put into the next release?

 -zefram

-----BEGIN PGP SIGNATURE-----
Version: 2.6.i

iQBVAgUBMAdJmWWJ8JfKi+e9AQE/YQH/c5iqTbPjMSspjJpEbNs5k2ISMQsPrqhC
/xwa4454BkftZFxL4HBoBY7K2GrJ/l5J8zSX0SK024ja14NZJei/Sw==
=d3Hs
-----END PGP SIGNATURE-----

Re: about beta11

[Richard Coleman]

> >One of the things I'm doing for beta11 is splitting the hash table
> >cmdnamtab into three separate tables for shell functions, builtins,
> >and external commands.  This fixes some of the weird command name
> >space problems that would have been difficult to fix any other way.
> 
> Does this mean that command completion, equals substitution and the
> hash builtin will all agree on whether a command exists?

I've just started looking at the completion code changes necessary.
I hope to keep all this consistent.  Most likely, these new changes
will actually make things much more consistent than they currently are.
I'll probably have to change some of the options to compctl to do
this.  Currently -d tells compctl to expect names of disabled commands.
This won't be good enough anymore.  What would be better would be
to make -d a modifier that says require the DISABLED flag on whatever
you are completing.  Something like:

compctl -dF      # complete the name of disable shell functions
compctl -eF      # complete the name of enabled shell functions
compctl -F       # complete the name of shell functions

> >1) enable/disable by default will only work on builtin functions
> >2) enable -f, disable -f will enable/disable shell functions
> >3) enable -r, disable -r will enable/disable reserved words
> 
> Great.  Atm, one can disable functions, but not reserved words or
> external commands (though they will be added to the disable list), but
> can't enable anything other than a builtin.  It's all rather silly.

Yes, the way things were before Peter split the reserved words table
was that to remove a reserved word, you used unalias (reserved words
and aliases shared a hash table).  Unfortunately you couldn't re-enable
it after that.  Also, using disable on a shell function actually caused
it to be removed so that it couldn't be re-enable it either.  With the
changes I've made, disable/enable only adds/removes the DISABLED flag.
Nothing gets removed from the hash table.  Therefore you can easily
re-enable it.  To remove something you use unhash (or one of its aliases).

Also, I can (trivially) add it so that disable -a, enable -a will
disable/enable aliases.  I didn't do this yet, since I wasn't sure
if this would be useful.  It would only be a two line change to the
code I currently have.  I might add it just for completeness sake.

> >Is there anything preventing us from using a hash table for this
> >rather than a linked list (which is what it currently uses)?
> 
> 'fraid so.  The named directory table is sorted.  (Actually, I'm pretty
> sure it's a big array, not a linked list, hence the occasional comment
> that calls it a hash table even though there's no real hashing
> involved.)  It has to be sorted so that %~ in prompts gives the
> shortest possible name for the directory.  That capability isn't used
> anywhere else, but unless you have a brilliant way to do that with a
> hash table then hash -d will have to stay as an exception.

I was afraid something like that would be the case.  I guess we'll just
have to leave this that way.

> I think it might make the builtins a bit neater if hash/rehash -d, and
> unhash -d if it gets added (it should be), were handled by separate
> functions from the normal bin_{,re,un}hash.  There would be a statement
> "if(ops['d']) return dir_hash(whatever)" at the beginning of each
> bin_*hash function, and the rest of the function would be the normal
> hash table stuff.

I was planning on adding unhash -d.  I don't know if I'll get to it
this version.  I'll probably move all the name directory code into
a separate function like you suggest.

> It *might* also be good to merge bin_hash and bin_rehash, and have
> rehash merely as an alias to hash -r.  It should be documented that way
> anyway.

I've been planning on this for awhile now.  I started working on this
yesterday but didn't finish it.  Unfortunately this doesn't save as
much code as the previous changes.  One change I've been looking at
is changing the syntax of hash to more closely match that of the
builtin alias.  So to add a named directory, you would use

hash -d foo=bar

rather than

hash -d foo bar

This way I will probably be able to combine the code for alias and
hash like I've already done for unalias and unhash.  It also allows
you to add multiple named directories by

hash -d foo1=bar1 foo2=bar2

It also makes it more consistent with the syntax of other zsh builtins.

> Speaking of builtin aliases, is "where" (whence -ca) going to appear at
> some point?  I sent a one-line patch for it, and Zoltan's release has
> that patch plus documentation.  It is genuinely useful to have a
> builtin that does a whence -a (I alias it).  And has anyone considered
> adding the csh "glob" command, which would be something like "print
>  -rN"?  (Less useful, but it occurred to me as being appropriate.  Does
> anyone use glob these days?)

Hmm...  My current feelings are that since we already have type/which/
whence, that another alias for this seems excessive.  But if enough
people thinks it is useful, I could add it.  I always just use
"type -a" myself.

> At this point I was going to request that the remaining disgusting bits
> of inconsistent ! quoting be fixed, as Zoltan's release contains a
> patch to make it just a normal character.  That was certainly the case
> a few betas ago.  But then I tested, just to make sure:

Yes, there are many patches for little things like this that I've
been ignoring lately.  I've just been so preoccupied which fixing
some of the underlying code, that I've sorta ignored some of the
more user-visible bug fixes.  I haven't forgotten about them.  Once
I finish all these changes to the various hash table and add some
of the other big patches (such as Peter's changes to history.c and
some of the patches of Zoltan's I've been procrastinating about),
I'll start adding many of these small patches.  Hopefully we soon
can begin the final shakedown necessary before we go to production
release.

Also, since this letter is already too long I might as well mention
that the next production release will be zsh-3.00 rather than zsh-2.7.
There have been so many changes that I think a new major version
number is justified.  I realize that this breaks the practice of using
even version numbers for development and odd version numbers for
production releases.  I would like to either reverse this (use odd
for development, even for production) which would make us match the
way Linux does version numbers, or just drop this practice altogether.

I've said before that I want the next production release to be the
best zsh ever.  Since it will have a new major version number, I'm
even more determined for this to be the case.

rc

Re: about beta11

[Zefram]

-----BEGIN PGP SIGNED MESSAGE-----

>I'll probably have to change some of the options to compctl to do
>this.  Currently -d tells compctl to expect names of disabled commands.
>This won't be good enough anymore.  What would be better would be
>to make -d a modifier that says require the DISABLED flag on whatever
>you are completing.  Something like:
>
>compctl -dF      # complete the name of disable shell functions
>compctl -eF      # complete the name of enabled shell functions
>compctl -F       # complete the name of shell functions

Hmm, more modifiers.  Time to roll out one of my nascent compctl
ideas...  compctl options that control what is completed to fall into
two completely distinct categories, with no overlap.  The options -P,
- -S, -q and -Q; and -d and -e if you do as you suggest; and possibly
others if the compctl options are ever reviewed in the interests of
orthogonality; all modify a basic choice of what to complete to (which
is decided by most of the other options).

At present, any single completion (one command, or one XOR'ed
completion, or one branch of an extended completion) can only have one
set of basic options and one set of modifiers, and all the modifiers
apply to all the basic options.  It would increase flexibility
enormously if one could do an *inclusive* OR of completions, which
could if necessary be nested inside an exclusive OR, and so on.  For
example, if = separates OR'ed sets of flags:

compctl -dF = -oeB gronk

(assuming -d and -e act as you describe above) makes completion of
arguments to gronk(1) include options, enabled builtins and disabled
functions.  Note that there is already an implicit OR'ing of options.
IMO, the lack of an explicit OR syntax is what has lead to the current
situation of compctl having separate options for, for example, (a)
variables, (b) arrays, (c) scalars, (d) integer variables, (e)
non-exported parameters, (f) exported parameters, (g) readonly
variables, and (h) special parameters.  If there were an explicit OR
syntax, most of these options could become modifiers in nature, and
have an AND-type effect individually.

I see this type of syntax being even more useful if different
completions can have different -S suffixes, particularly if completion
functions can return a list of suffixes in addition to the list of
completions.  Is this a reasonable feature to add at some point?

>Also, I can (trivially) add it so that disable -a, enable -a will
>disable/enable aliases.  I didn't do this yet, since I wasn't sure
>if this would be useful.  It would only be a two line change to the
>code I currently have.  I might add it just for completeness sake.

Do it.  It's a stupendously crufty bit of orthogonality.

[re the named directory table]
>I was afraid something like that would be the case.  I guess we'll just
>have to leave this that way.

Actually, it might be worth temporarily rewriting finddir() to do a
linear search, in order to do some performance tests.  (It would have
to go through the entire table, storing the index of the most
preferable name found so far.)  The comments in it indicate that the
current binary search, which is followed by a linear search of usually
a great deal of the table anyway, doesn't make much difference.  The
caching of the last response (code already there) may actually make the
performance loss acceptable, even on a system with several thousand
users.

If you're too busy to try this, I'll copy the code across to the other
computer system here (Solaris 2.4, NIS, about eleven thousand users)
and make a comparison.

>                                    One change I've been looking at
>is changing the syntax of hash to more closely match that of the
>builtin alias.  So to add a named directory, you would use
>
>hash -d foo=bar
>
>rather than
>
>hash -d foo bar
>
>This way I will probably be able to combine the code for alias and
>hash like I've already done for unalias and unhash.  It also allows
>you to add multiple named directories by
>
>hash -d foo1=bar1 foo2=bar2
>
>It also makes it more consistent with the syntax of other zsh builtins.

Looks good.  In the light of this new syntax, I'd like to draw everyone's attention to the output format of hash -d.  It currently give output like

~cs126=/tmp
~dain=/home/dain
~=/home/zefram

Should it have the ~ there?  When I wrote the code I rationalised that
hash alone gives output of the form <command>=<path>, which I
interpreted as meaning <command> is translated into <path>.  Hence the
hash -d output ~<name>=<path>, meaning that ~<name> is translated into
<path>.  Should it instead output <name>=<path>, to match the setting
syntax?

Another oddity I just noticed: if I do

% hash -d '' /foo/bar

then finddir() says that ~ can only be used if I'm under /foo/bar, but
"echo ~" still gives my real home directory.  This doesn't happen with
~user (i.e. "echo ~user" gives my bogus path).  What *should* the
semantics be?

>Hmm...  My current feelings are that since we already have type/which/
>whence, that another alias for this seems excessive.

Not really.  We have type from sh/bash, whence from ksh, which from
csh, so it makes perfect sense to provide the tcsh command as well.
And it's only one line in the builtin table, and doesn't clash with any
real commands.

 -zefram

-----BEGIN PGP SIGNATURE-----
Version: 2.6.i

iQBVAgUBMAdzRWWJ8JfKi+e9AQH9eAIAqj3YPDFHDo7AfC5EaLDg0ls2YioTnPgh
iQOzDTgiQpRXOV1ONsIA5LvNvy2e0CTcjrx+tDDxELQE0Nh/2kv3jg==
=srNg
-----END PGP SIGNATURE-----

Re: Completion quirks

[Zoltan Hidvegi]

> Much appreciated!  I still observe the wrong "exact" match being appended
> when REC_EXACT is set without any menu completion options, but all the
> other problems have been solved.  This remaining problem turns out to be
> some missing braces on one of the last ifs in domatch().  This problem is
> not in the stock beta10 since that if used to have just one statement (it
> used commas to do multiple assignments).  I've appended a patch.

Thanks for that patch. Actually it fixes a bug in an old patch from Sven to
fix some fignore bug. It was sometime in march or april, and for some reason
it did not get into the baseline (there were a lot of patches for fignore in
that time, and somehow that was forgotten).

Zoltan

Re: Completion quirks

[Wayne Davison]

> The patch below fixes both of the observed problems with REC_EXACT (one
> of which was present in beta10 and probably earlier betas too).

Much appreciated!  I still observe the wrong "exact" match being appended
when REC_EXACT is set without any menu completion options, but all the
other problems have been solved.  This remaining problem turns out to be
some missing braces on one of the last ifs in domatch().  This problem is
not in the stock beta10 since that if used to have just one statement (it
used commas to do multiple assignments).  I've appended a patch.

Also, thanks for pointing out what rec_exact does -- I've turned it
off.  It was an option I blindly snagged from the example zshrc that
comes with zsh -- perhaps it should be removed to make it less
confusing for new zsh users?

..wayne..
---8<------8<------8<------8<---cut here--->8------>8------>8------>8---
Index: zle_tricky.c
@@ -1462,12 +1462,13 @@
 	*fm = t;
 	*sp = 100000;
     }
-    if (!ispattern && (sl = strlen(t)) < *sp)
+    if (!ispattern && (sl = strlen(t)) < *sp) {
 	*sp = sl;
 	if (l == fmatches)
 	    fshortest = t;
 	else
 	    shortest = t;
+    }
     if (sav)
 	*e = sav;
 }
---8<------8<------8<------8<---cut here--->8------>8------>8------>8---

Re: Completion quirks

[Zefram]

-----BEGIN PGP SIGNED MESSAGE-----

I wrote:
>With hzoli10 I got this behaviour, and also with my own patched beta10
>(though starting with the correct match).  This problem is almost
>certainly a result of my completion list patches (which I can't test
>with every option, though I did give it a bit of a work out).  Plain
>beta10 for me misbehaved on the second completion, much in the way you
>describe, so I think it is safe to say that REC_EXACT is, for the
>moment, broken.  I'll see what I can do...

The patch below fixes both of the observed problems with REC_EXACT (one
of which was present in beta10 and probably earlier betas too).  The
problem of the completion prefix being inserted a second time was due
to a missing invalidatelist(), just like the previous case of this
problem (that Zoltan pointed out).  A quick grep for do_single
indicates that all calls to do_single() that should be followed by an
invalidatelist() now are.  The older problem is due to listambig being
set, and remaining set, when REC_EXACT is invoked, letting AUTO_MENU
start menu completion on the next tab when it shouldn't.  This is fixed
by moving the line that sets listambig.

 -zefram

      *** Src/zle_tricky.c.old	Sat Jul 15 03:40:02 1995
      --- Src/zle_tricky.c	Sat Jul 15 03:47:19 1995
      ***************
      *** 3187,3193 ****
        {
            int p = (usemenu || ispattern), atend = (cs == we);
        
      -     lastambig = 1;
            menucmp = 0;
        
            /* If we have to insert the first match, call do_single(). */
      --- 3187,3192 ----
      ***************
      *** 3194,3200 ****
      --- 3193,3201 ----
            if (shortest && shortl == 0 && isset(RECEXACT) &&
        	(usemenu == 0 || unset(AUTOMENU))) {
        	do_single(shortest);
      + 	invalidatelist();
            } else {
      + 	lastambig = 1;
        	if (p)
        	    /* Otherwise we have to do menu-completion... */
        	    do_ambig_menu();

-----BEGIN PGP SIGNATURE-----
Version: 2.6.i

iQBVAgUBMAcusmWJ8JfKi+e9AQGUpgH/d7kksV5LZ9LB3hCleFHVSPeDxzYQ8v08
Jx+xAYQs4r5QlhQeqxWL26bJyaeofm6+8PeS+M0/JkuUw8HM5ra/Ww==
=njj8
-----END PGP SIGNATURE-----