How completions work, do they require fpath?

How completions work, do they require fpath?

[Sebastian Gniazdowski]

Hello,
my current understanding of the completions is that if a file-function
with name starting with "_", and having "#compdef ..." in first line
of content, is found in $fpath, then, if compinit is run, the function
will be autoloaded and used by _main_complete. Am I missing something
here?

Does this mean that compinit should be best run once after all needed
fpath updates?

A trick to avoid fpath for regular autoload functions has emerged on
this list. Is similar trick possible for a completion? How to load one
without running compinit and most of all, without updating fpath? That
is needed for Zplugin, which is now in nice state (except maybe for
lengthy code and zparseopts issues), e.g. can unload modules (deleting
bindkeys, restoring options, removing aliases, etc.), but doesn't
support adding completions via plugins.

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Sebastian Gniazdowski]

I've found following line in compaudit command:

  if [[ $#_i_files -lt 20 || $_compdir = */Base || -d $_compdir/Base ]]; then

So this suggests that if there is Base sub directory in $_compdir, it
will be used by completion. Tested this and that's true. Quite
contradict to manual which says:

"For  incomplete  installations,  if compinit does not find enough
files eginning with an underscore (fewer than twenty) in the search
path, it will try to find more by adding the directory _compdir  to
the  search path."

Anyway, it seems that I will be able to add completions as follows: a)
create completions/Base/Core directory following the following lines
from compaudit:

    if [[ -d $_compdir/Base/Core ]]; then
      # Add all the Completion subdirectories (CVS-layout)
      _i_addfiles=(${_compdir}/*/*(/^M))

b) mkdir plugin's directory in Base
c) simlink completion files (starting with "_") into that subdirectory
c) set _compdir and run compinit

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Bart Schaefer]

On Jan 26,  8:51pm, Sebastian Gniazdowski wrote:
} Subject: Re: How completions work, do they require fpath?
}
} So this suggests that if there is Base sub directory in $_compdir, it
} will be used by completion. Tested this and that's true. Quite
} contradict to manual which says:

No, it doesn't contradict the manual.  The check for a directory named
"Base" is to distinguish "configure --enable-function-subdirs" from the
default configuration which is to avoid creating subdirectories.  If
the "Base" directory is present it's expected to contain all the
files from Completion/Base/ in the source tree, and although the
explicit check is not done the directories AIX/, BSD/, Cygwin/, etc.
are also expected to be found within $_compdir in that case.

This is NOT an indication that you can co-opt these directories for
plugins, nor is it "safe" to run compinit more than once with a new
value of $_compdir -- I'm fairly sure you will mangle, or misuse, or
both, the ~/.zcompdump file if you do so.

Re: How completions work, do they require fpath?

[Sebastian Gniazdowski]

On 27 January 2016 at 08:20, Bart Schaefer <schaefer@brasslantern.com> wrote:
> No, it doesn't contradict the manual.  The check for a directory named
> "Base" is to distinguish "configure --enable-function-subdirs" from the
> default configuration which is to avoid creating subdirectories.  If
> the "Base" directory is present it's expected to contain all the
> files from Completion/Base/ in the source tree, and although the
> explicit check is not done the directories AIX/, BSD/, Cygwin/, etc.
> are also expected to be found within $_compdir in that case.

So using _compdir would be a hack, and also, someone can utilize
_compdir for what's it's intended, and overwriting _compdir would
broke their setup.

> This is NOT an indication that you can co-opt these directories for
> plugins, nor is it "safe" to run compinit more than once with a new
> value of $_compdir -- I'm fairly sure you will mangle, or misuse, or
> both, the ~/.zcompdump file if you do so.

So it seems that fpath is needed. I saw Peter's email (workers/12379)
suggesting that work has been done to make fpath the choice for
providing new completions. I wanted to have fpath "clean", and I guess
I can still have it like this: create ~/.zplugin/completions, add it
to fpath, simlink any plugin's "_*" files there. But this will broke
if a completion uses autoload of some functions from plugin's
directory. A very rare case however, I guess.

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Bart Schaefer]

On Jan 26,  6:18pm, Sebastian Gniazdowski wrote:
}
} my current understanding of the completions is that if a file-function
} with name starting with "_", and having "#compdef ..." in first line
} of content, is found in $fpath, then, if compinit is run, the function
} will be autoloaded and used by _main_complete. Am I missing something
} here?

No.

} Does this mean that compinit should be best run once after all needed
} fpath updates?

Ideally yes, but really #compdef is just a way to have the compdef
function invoked automatically for some files.  If the plugin is
already calling autoload + compdef directly, compinit isn't needed.

Note that nothing checks for conflicting #compdef, so if two files
both specify the same key bindings the one found later in fpath
will create the binding, but the one found earlier in fpath will
be autoloaded if the two have the same basename.  (So far it has
never come up for two files with the same name, though I believe
there are some conflicting key bindings.)

} A trick to avoid fpath for regular autoload functions has emerged on
} this list. Is similar trick possible for a completion? How to load one
} without running compinit and most of all, without updating fpath?

There's a chunk of code near the end of compinit that handles the
parsing of the #compdef lines.  That would need to be extracted into
its own function so it could be run repeatedly for new files.

Other than that, the shadow-autoload trick already used in zplugin
should work for completion autoloads, although they would not get
recorded in ~/.zcompdump (and you wouldn't want them to be, in that
circumstance).

On Jan 27,  8:59am, Sebastian Gniazdowski wrote:
} Subject: Re: How completions work, do they require fpath?
}
} On 27 January 2016 at 08:20, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > No, it doesn't contradict the manual.  The check for a directory named
} > "Base" is to distinguish "configure --enable-function-subdirs"

Actually I'm only partly right about that.  The real purpose of those
checks in compaudit are so that for example you can start in the zsh
source tree and run:

    Src/zsh -f
    fpath+=($PWD/Completion)
    autoload -Uz compinit
    _compdir=$PWD/Completion compinit -D

and have everything set up to do compsys testing, without having to
first "make install" etc.

} So using _compdir would be a hack, and also, someone can utilize
} _compdir for what's it's intended, and overwriting _compdir would
} broke their setup.

Correct.

} So it seems that fpath is needed.

Perhaps, but see other responses from Peter and above about compdef.

} I wanted to have fpath "clean", and I guess I can still have it

I've been a little puzzled all along about why you object to having
entries in fpath.

} like this: create ~/.zplugin/completions, add it to fpath, simlink
} any plugin's "_*" files there.

There's another possibility:  use zcompile.  Gather up the names of
all the autoloadable files defined by the plugins, and then run for
example

    zcompile ZPLUGINS $all_the_paths_of_autoloadable_files

This will create a (large) file ZPLUGINS.zwc containing all potential
autoloads.  Then just add that file name (not a directory name in
this special case) to fpath:

    fpath+=($PWD/ZPLUGINS.zwc)

and you're done.  Some checks that you don't clobber the file when
a new shell starts up are advised, because zsh typically mmap's it and
shares it.

Re: How completions work, do they require fpath?

[Sebastian Gniazdowski]

On 28 January 2016 at 08:11, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jan 26,  6:18pm, Sebastian Gniazdowski wrote:
> }
> } my current understanding of the completions is that if a file-function
> } with name starting with "_", and having "#compdef ..." in first line
> } of content, is found in $fpath, then, if compinit is run, the function
> } will be autoloaded and used by _main_complete. Am I missing something
> } here?
>
> No.

One thing emerged – I was expecting that empty say "_cp" will just be
ignored, because of lack of #compdef, however it isn't – it blocks
other "_cp" files from loading.

> } Does this mean that compinit should be best run once after all needed
> } fpath updates?
>
> Ideally yes, but really #compdef is just a way to have the compdef
> function invoked automatically for some files.  If the plugin is
> already calling autoload + compdef directly, compinit isn't needed.

Interesting, have to test this

> Other than that, the shadow-autoload trick already used in zplugin
> should work for completion autoloads, although they would not get
> recorded in ~/.zcompdump (and you wouldn't want them to be, in that
> circumstance).

That worries me, I hope my shadowing doesn't interfere with completion
autoloads. That said, it seems that plugin would have to invoke
"compinit" to trigger such autoloads?

> } I wanted to have fpath "clean", and I guess I can still have it
>
> I've been a little puzzled all along about why you object to having
> entries in fpath.

I wasn't 100% sure myself if my expectations of fpath being "clean"
are reasonable. However now after implementing completion support to
Zplugin I see it was a good thing. Really, really nice thing to just
have "~/.zplugin/completions" instead of 10 entries:

https://asciinema.org/a/dmacq18klhw9rfxfffrhweh6a

> } like this: create ~/.zplugin/completions, add it to fpath, simlink
> } any plugin's "_*" files there.
>
> There's another possibility:  use zcompile.  Gather up the names of
> all the autoloadable files defined by the plugins, and then run for
> example
>
>     zcompile ZPLUGINS $all_the_paths_of_autoloadable_files
>
> This will create a (large) file ZPLUGINS.zwc containing all potential
> autoloads.  Then just add that file name (not a directory name in
> this special case) to fpath:
>
>     fpath+=($PWD/ZPLUGINS.zwc)
>
> and you're done.  Some checks that you don't clobber the file when
> a new shell starts up are advised, because zsh typically mmap's it and
> shares it.

Will think about that, currently I've implemented completion handling
via simlinks, and this gives a nice feature – I can check from which
plugin a completion comes from (as seen in the asciinema video)

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Sebastian Gniazdowski]

On 28 January 2016 at 08:37, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> On 28 January 2016 at 08:11, Bart Schaefer <schaefer@brasslantern.com> wrote:
>> Ideally yes, but really #compdef is just a way to have the compdef
>> function invoked automatically for some files.  If the plugin is
>> already calling autoload + compdef directly, compinit isn't needed.
>
> Interesting, have to test this

PS. Without compinit, compdef isn't available?

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Bart Schaefer]

On Jan 28,  8:37am, Sebastian Gniazdowski wrote:
}
} One thing emerged - I was expecting that empty say "_cp" will just be
} ignored, because of lack of #compdef, however it isn't - it blocks
} other "_cp" files from loading.

That doesn't have anything to do with compinit, that's just the way
autoloading always works.  Compinit marks the names to be autoloaded,
but it doesn't control where they come from when autoloading finally
happens.  This was bashed out in agonizing detail with Ray Andrews
back in September.

The reload-and-run trick could circumvent this, that is, if your
autoload shadow were defined before compinit runs, then all of the
functions it touches would load from the place they were found at
compinit time rather than at execution time.

But it would make compinit a bit slower and use a bit more memory.

Re: How completions work, do they require fpath?

[Bart Schaefer]

On Jan 28,  8:39am, Sebastian Gniazdowski wrote:
}
} PS. Without compinit, compdef isn't available?

The compdef function is defined when compinit runs, yes.

The upshot is that you can either:

1. Put your function in a file whose name begins with "_", add a #compdef
   line, put the file in a directory in $fpath, and finally run "compinit"

2. Name your function file any way that you like, run compinit whenever
   it is convenient, put your file in a directory in $fpath or load it
   some other way e.g. "source", but finally add your function to the
   completion system by calling "compdef".

The intention is that #1 is simple and can be rubber-stamped for every
new completion function, but you always have #2 if #1 doesn't work for
some reason.  Either way "compinit" has to happen to set up the rest of
compsys, so there's no point in defining "compdef" independently.

Re: How completions work, do they require fpath?

[Sebastian Gniazdowski]

On 29 January 2016 at 00:23, Bart Schaefer <schaefer@brasslantern.com> wrote:
> The intention is that #1 is simple and can be rubber-stamped for every
> new completion function, but you always have #2 if #1 doesn't work for
> some reason.  Either way "compinit" has to happen to set up the rest of
> compsys, so there's no point in defining "compdef" independently.

There is a plugin (voronkovich/get-jquery) that calls compdef. The
problem is that this requires compinit being run before loading
plugins. But after loading them, compinit is also to be run, to take
account of possible new completions added by the plugins. I guess I
will shadow compdef and load compinit, and warn user about that in
report, that he will have compinit run twice. Unless there is a way to
provide compdef without whole compinit?

Best regards,
Sebastian Gniazdowski

Re: How completions work, do they require fpath?

[Bart Schaefer]

On Jan 29,  9:02am, Sebastian Gniazdowski wrote:
}
} There is a plugin (voronkovich/get-jquery) that calls compdef. The
} problem is that this requires compinit being run before loading
} plugins.

Well, either that or you have to divide the plugins into before/after
compinit subsets, somehow.

There's probably always going to be a way for an uncooperative author
to muck up a plugin manager, just as you could write a C module that
abuses the zmodload linkage conventions.

} Unless there is a way to provide compdef without whole compinit? 

compdef is going to manipulate global variables used/needed by compinit
and later by the rest of compsys.  It's not really possible to run it
standalone beforehand.

You could try this:  Define your own "compdef" whose only action is
to cache its arguments somewhere.  Load all the plugins, then after
compinit is run, play back the cached compdef arguments with the real
compdef.