Are completions in some way heavy?

Are completions in some way heavy?

[Sebastian Gniazdowski]

Hello,
I've started a project that basically re-delivers all Zsh completion functions:

https://github.com/psprint/allcompletions

The point is: with Zplugin it's possible to enable only selected
completions (http://psprint.github.io/Zplugins-completion-management/).
This should provide speed up, to start with 15 enabled completions,
instead of 800, and enable completions at request with simple: zplg
cenable "xauth", etc. I missed the fact that compdump is a final
optimization. My gains are: compinit runs for 36 ms instead of 112 ms,
for almost half of 774 completions disabled.

That's a failure but I thought that maybe shell will be much lighter
with <100 completions pre-loaded? Maybe situation with compsys isn't
that bright because of the number of _* files? How can I reveal
effects of much lighter $functions, etc., is it possible?

Also, overall, it would be nice that I would collect all mainstream _*
patches, and provide the completions for all Zsh versions. Also, maybe
other way around would be possible – rapid iterations on the _* files
in allcompletions tree, then rare but high in volume transfer to the
mainstream. Maybe Github way of development would result in e.g. more
files being compatible with emulate ksh or sh, but I'm not sure if
there would be any interest in such quality-pull-up coding.

Best regards,
Sebastian Gniazdowski

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 6,  2:43pm, Sebastian Gniazdowski wrote:
}
} This should provide speed up, to start with 15 enabled completions,
} instead of 800 [...]
} 
} That's a failure but I thought that maybe shell will be much lighter
} with <100 completions pre-loaded? Maybe situation with compsys isn't
} that bright because of the number of _* files? How can I reveal
} effects of much lighter $functions, etc., is it possible?

Functions that are marked for autoloading but haven't yet loaded are
already very lightweight.  That was one reason for using a suite of
files, so the autoload mechanism could take care of ignoring those
that are never needed in a particular session.  Then compdump avoids
the overhead of scanning directories to find those files.

Making it the user's responsibility to explicitly choose what to enable
is exactly what we wanted to *avoid*.  If that's a requirement in order
for the system to be "light enough" then we have failed.
 
} Also, overall, it would be nice that I would collect all mainstream _*
} patches, and provide the completions for all Zsh versions.

Having a location from which to grab .../share/$ZSH_VERSION/functions/
trees for past zsh releases would be useful, particularly if fixes from
more recent versions were backported when possible.

I would not suggest attempting to create forward-portable versions of
the functions.

Re: Are completions in some way heavy?

[Sebastian Gniazdowski]

On 6 June 2016 at 18:01, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jun 6,  2:43pm, Sebastian Gniazdowski wrote:
> }
> } This should provide speed up, to start with 15 enabled completions,
> } instead of 800 [...]
> }
> } That's a failure but I thought that maybe shell will be much lighter
> } with <100 completions pre-loaded? Maybe situation with compsys isn't
> } that bright because of the number of _* files? How can I reveal
> } effects of much lighter $functions, etc., is it possible?
>
> Functions that are marked for autoloading but haven't yet loaded are
> already very lightweight.  That was one reason for using a suite of
> files, so the autoload mechanism could take care of ignoring those
> that are never needed in a particular session.  Then compdump avoids
> the overhead of scanning directories to find those files.

The autoloading mechanism is indeed nice. The same for the compdump.
What's also nice is how easy it was to detach all completions from
Zsh. Simple copy of all "_*" files. And instant working when placed in
other directory that's in $fpath.

> Making it the user's responsibility to explicitly choose what to enable
> is exactly what we wanted to *avoid*.  If that's a requirement in order
> for the system to be "light enough" then we have failed.

I in general don't like completions, must admit. They often interfere
with what I want to do. Giving examples is much hard, such unpleasant
situations don't stick in memory. Maybe su. It completes fancy user
names on OS X . But ok, that's an acceptable must maybe (couldn't it
choose usernames with uid > 500 or something..?). But it doesn't
complete user names when invoked as "su - <TAB>". I guess patience is
needed for completions. They're like small man pages that disturb
view. One has to keep calm and try to read. Then it's fine.

There are orders in magnitude of quality in such tasks like
completion. This recalls me the various bootable CDs like Trinity,
Hiren's, Falcon, etc. How much effort can man put in making such
things better and better, more automated and robust? But only a few
people love doing such perfect things. The amount of work that I think
is needed for completions to cross next order of the magnitude would
require rapid iterations I guess, that's why I mentioned it. And I
wouldn't mind if I could disable and enable before that boundary is
crossed. But it's fine, completions are in general good. And there
undoubtedly is progress. Sometimes weird, like on Ubuntu where
kill<TAB> invokes fzf. Don't like the software because they did
history Ctrl-R via a hack: they don't parse multi-line entries,
instead they pass history event numbers together with one-line-packed
history entries, and then invoke vi-fetch-history:

https://github.com/junegunn/fzf/blob/master/shell/key-bindings.zsh#L42-L48

I ended up with 240 "crucial" completions when selecting what to keep
by default. But it was in 30% a hard decision process. 70% of what has
been rejected was easy. But uncertainty stays. Also, zplg cenable
takes completion's file name without _. This can of course not match
command name. Guess I could enable all multiple-argument compdefs by
default. grep "compdef [a-z]\+ [a-z]\+" * | wc -l says there are 172
such completions. What's the point currently? Having plugin manager
part-free. The 50-80ms is quite significant when compared to 250ms
when Zplugins loads 9 plugins with full shadowing, reporting,
including heavy plugins like z-sy-h and autosuggestions (70 ms in
"light" mode without reporting). But on other hand, what's really fun
is having completions detached from Zsh.

> } Also, overall, it would be nice that I would collect all mainstream _*
> } patches, and provide the completions for all Zsh versions.
>
> Having a location from which to grab .../share/$ZSH_VERSION/functions/
> trees for past zsh releases would be useful, particularly if fixes from
> more recent versions were backported when possible.
>
> I would not suggest attempting to create forward-portable versions of
> the functions.

Forward-portable?

Best regards,
Sebastian Gniazdowski

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 6,  9:28pm, Sebastian Gniazdowski wrote:
}
} On 6 June 2016 at 18:01, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Making it the user's responsibility to explicitly choose what to enable
} > is exactly what we wanted to *avoid*.

To your later point, *dis*abling a completion isn't something that I
recall ever being closely considered.  The assumption was that if you
didn't need a completion, it was either because you never used the
corresponding command (maybe it isn't even available to you), or you
always type out the arguments without pressing tab.

Nevertheless:

} I in general don't like completions, must admit. They often interfere
} with what I want to do.

This statement is a bit baffling.  If this were really the case "in
general," why haven't you long ago re-bound TAB to something that
does not interfere, or simply stopped pressing it?

} Maybe su. It completes fancy user names on OS X . But ok, that's an
} acceptable must maybe (couldn't it choose usernames with uid > 500 or
} something..?).

I believe the problem is that some people expect completions to be more
comprehensive/intuitive than they were ever intended to be.  Completion
is supposed to save you keystrokes and occasionally to jog your memory;
it's not supposed to be an AI that eliminates the need to think.  When
building compsys we made a pretty clear decision that completions would
only need to cover the majority of common cases and that offering too
many results which the user could narrow down was better than offering
too few.  Iterative refinement to perfection was never in scope, and in
fact is probably impossible because for every person who wants only UID
over 500 there's somebody else who only cares about the weird pseudo-
users he can't easily remember.

Which is why zstyle got added, so that you can do things like

    zstyle -e ':completion:*:su:*' users \
	'reply=( $(awk -F : "\$3 >= 500 { print \$1 }" < /etc/passwd) )'

} But it doesn't complete user names when invoked as "su - <TAB>".

Obviously some things are going to be in a grey area of whether they
are a "common case" and usually those things are worth fixing when
they turn up.  Still, it's telling that "su" completion has been around
for a long time and nobody bothered to complain about this before.

} I guess patience is needed for completions. They're like small man
} pages that disturb view. One has to keep calm and try to read. Then
} it's fine.

Again if they are actually disturbing you, why are you invoking them
in the first place?  They're not meant to slow you down.  If you have
to patiently read, then seemingly you did need your memory jogged and
the information is worthwhile?

} There are orders in magnitude of quality in such tasks like
} completion. [...]  Sometimes weird, like on Ubuntu where
} kill<TAB> invokes fzf.

Rather unfair to hold zsh compsys responsible for something built on
top of it by an OS distributor.

} I ended up with 240 "crucial" completions when selecting what to keep
} by default. But it was in 30% a hard decision process. 70% of what has
} been rejected was easy.

It might be worth having a command (other than "unset _comps[thing]")
to easily remove an unwanted completion.

There is the under-utilized -first- context and corresponding _first
function which can be overridden to do things like suppress certain
completions.  The default _first is empty.

} Forward-portable?

Meaning, a function deliberately written using syntax from an older
version with the intention that it will run unchanged on all future
versions.  That's fine for plugins that are meant to be compatible
with many versions, but avoid replacing an improved version of
something shipped with a newer zsh for no other reason than this.

Re: Are completions in some way heavy?

[Daniel Shahaf]

Bart Schaefer wrote on Mon, Jun 06, 2016 at 09:01:04 -0700:
> On Jun 6,  2:43pm, Sebastian Gniazdowski wrote:
> } Also, overall, it would be nice that I would collect all mainstream _*
> } patches, and provide the completions for all Zsh versions.
> 
> Having a location from which to grab .../share/$ZSH_VERSION/functions/
> trees for past zsh releases would be useful, particularly if fixes from
> more recent versions were backported when possible.

Sebastian, your project currently only covers completion functions
(Completion/**/*), not runtime functions (Functions/**/*); do you plan
to cover the latter too?

The "most fixes made in master are applicable to older versions too"
point applies to non-completion functions too, after all.

Cheers,

Daniel

Re: Are completions in some way heavy?

[Sebastian Gniazdowski]

On 7 June 2016 at 22:44, Bart Schaefer <schaefer@brasslantern.com> wrote:
> To your later point, *dis*abling a completion isn't something that I
> recall ever being closely considered.  The assumption was that if you
> didn't need a completion, it was either because you never used the
> corresponding command (maybe it isn't even available to you), or you
> always type out the arguments without pressing tab.

This is fine, all logical. However from practice about "having a clean
FPATH" I can maybe signal a topic about need of order of users.
Ability to do "disable debian", "disable cygwin", "enable osx", to
turn on and off needed/unneeded completions would bring this sense of
order maybe. But logically this is futile. Compdump optimizes greatly,
autoload keeps functions fit, user uses completions that he needs to
use, what's the point of disable/enable.

> I believe the problem is that some people expect completions to be more
> comprehensive/intuitive than they were ever intended to be.  Completion
> is supposed to save you keystrokes and occasionally to jog your memory;
> it's not supposed to be an AI that eliminates the need to think.
...
> } I guess patience is needed for completions. They're like small man
> } pages that disturb view. One has to keep calm and try to read. Then
> } it's fine.
>
> Again if they are actually disturbing you, why are you invoking them
> in the first place?  They're not meant to slow you down.  If you have
> to patiently read, then seemingly you did need your memory jogged and
> the information is worthwhile?

Luckily an example can be provided. I was greeping for a text. It was
returning git objects. Wanted to disable lookup in binary files. But
can never remember is it "with-binary-files" or something similar
(turns out it's --binary-files, but my memory didn't play nice on
this). So I entered grep --with<TAB>. Nothing was returned. I might
have tried grep --binary and nothing would be returned because turns
out grep completion is very short on long options on OS X (no one
complained?). Reverted to manual. Entered "binary", found what was
needed, used it.

So why did I try completion. To take a guess maybe. This leads to an
idea of how this could be done:

- tab invokes "small man page" as it usually does after "command -<TAB>"
- the text can be searched – entering something greps lines in the "man page"
- bits of text can be navigated like in file completion when properly
configured – I mean cursor keys move the reverse highlight over what's
available

I have a proof of concept tool for this – h-list, horizontal list
(allows to browse elements horizontally). Might not look perfect as I
do (z) split on current line and don't restore spaces when displaying
(easy to fix):

https://asciinema.org/a/ewb3uj9095ixakd45yiztnqey
https://github.com/psprint/zsh-tools/

Also a bug revealed, searching for binary doesn't return
"--binary-files". Nevertheless, Zsh has apparently proper tools for
this task. Zcurses, desired performance (n-list, a vertical-only
version, processes 120k elements with no problem), approximate
matching.

> } Forward-portable?
>
> Meaning, a function deliberately written using syntax from an older
> version with the intention that it will run unchanged on all future
> versions.  That's fine for plugins that are meant to be compatible
> with many versions, but avoid replacing an improved version of
> something shipped with a newer zsh for no other reason than this.

Yes that's something not wanted. Maybe generations of the completions
could be done. Of course if anything will be actually done in the
topic as performance gain is a fail due to compdump optimizations.
Anyway, allocompletions-4, allcompletions-5.0.2, allcompletions-5.3.

Best regards,
Sebastian Gniazdowski

Re: Are completions in some way heavy?

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> So why did I try completion. To take a guess maybe. This leads to an
> idea of how this could be done:
>
> - tab invokes "small man page" as it usually does after "command -<TAB>"
> - the text can be searched – entering something greps lines in the "man page"
> - bits of text can be navigated like in file completion when properly
> configured – I mean cursor keys move the reverse highlight over what's
> available

Menu selection (the zsh/complist module) has an incremental search, so
what you describe above can already be done.

Oliver

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 8,  6:34am, Sebastian Gniazdowski wrote:
} Subject: Re: Are completions in some way heavy?
}
} On 7 June 2016 at 22:44, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Again if they are actually disturbing you, why are you invoking them
} > in the first place?
} 
} Luckily an example can be provided. I was greeping for a text. It was
} returning git objects. Wanted to disable lookup in binary files. But
} can never remember is it "with-binary-files" or something similar
} (turns out it's --binary-files, but my memory didn't play nice on
} this). So I entered grep --with<TAB>. Nothing was returned.

I see.  I don't really have any good suggestions here -- there is the
_complete_fallback idea but all it does is provide alternate guesses
for the possible argument positions, which doesn't help in this case.

It might be do-able to revert to executing run-help when there are no
completions found, but that would be wrong in a lot of cases (e.g.,
on file name not matched, displaying a manual page isn't useful).

} I might have tried grep --binary and nothing would be returned because
} turns out grep completion is very short on long options on OS X (no
} one complained?).

No complaints, no.  It appears from cursory examination that the person
who initally wrote _grep only had access to GNU grep, and _grep has had
only a few minor updates since 2002.


On Jun 8, 11:03am, Oliver Kiddle wrote:
} Subject: Re: Are completions in some way heavy?
}
} Sebastian Gniazdowski wrote:
} > So why did I try completion. To take a guess maybe. This leads to an
} > idea of how this could be done:
} >
} > - tab invokes "small man page" as it usually does after "command -<TAB>"
} > - the text can be searched - entering something greps lines in the "man page"
} > - bits of text can be navigated like in file completion when properly
} > configured - I mean cursor keys move the reverse highlight over what's
} > available
} 
} Menu selection (the zsh/complist module) has an incremental search, so
} what you describe above can already be done.

I think Sebastian wants to search through the descriptions, whereas the
complist search only examines the completions themselves.

None of this helps with the situation Sebastian described above wherein
the completion results/descriptions do not include the desired content.

Re: Are completions in some way heavy?

[Sebastian Gniazdowski]

On 8 June 2016 at 20:40, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jun 8, 11:03am, Oliver Kiddle wrote:
> } Menu selection (the zsh/complist module) has an incremental search, so
> } what you describe above can already be done.
>
> I think Sebastian wants to search through the descriptions, whereas the
> complist search only examines the completions themselves.

It searches through descriptions of options. Quite nice thing. This
single zstyle:

zstyle ':completion:*:*:*:default' menu yes select search

configures the completion to a high degree I think. Often "force-list"
is used and I think other combinations of zstyles to actually see menu
completion. Works quite well, more enters are required but it's done
to be quite natural. Don't know yet how to cycle through matches,
would like to leave completion on space, don't get what "interactive"
mode is. But a very nice thing, I had success of finding mpv
--no-video option without ever giving options to the program before
and considering it's -vo null like in mplayer. Nevertheless I need to
recognize this more.

And nevertheless I decided to continue with the proof of concept :)

https://asciinema.org/a/8k8v5ni5lnnpoc3z5jzl6jgav
https://github.com/psprint/zzcomplete

What I think is needed is context search, to return text surrounding
matches. Might be difficult to implement and unreadable (like grep -C,
it's difficult to read when untrained).

Best regards,
Sebastian Gniazdowski

Re: Are completions in some way heavy?

[Daniel Shahaf]

Bart Schaefer wrote on Wed, Jun 08, 2016 at 11:40:09 -0700:
> On Jun 8,  6:34am, Sebastian Gniazdowski wrote:
> } Subject: Re: Are completions in some way heavy?
> }
> } On 7 June 2016 at 22:44, Bart Schaefer <schaefer@brasslantern.com> wrote:
> } > Again if they are actually disturbing you, why are you invoking them
> } > in the first place?
> } 
> } Luckily an example can be provided. I was greeping for a text. It was
> } returning git objects. Wanted to disable lookup in binary files. But
> } can never remember is it "with-binary-files" or something similar
> } (turns out it's --binary-files, but my memory didn't play nice on
> } this). So I entered grep --with<TAB>. Nothing was returned.
> 
> I see.  I don't really have any good suggestions here -- there is the
> _complete_fallback idea but all it does is provide alternate guesses
> for the possible argument positions, which doesn't help in this case.
> 

I use the _match completer, so «grep --*bin<TAB>» completes to
«--binary-files=».  It doesn't match against descriptions though.

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 10,  2:22pm, Sebastian Gniazdowski wrote:
}
} > I think Sebastian wants to search through the descriptions, whereas
} > the complist search only examines the completions themselves.
} 
} It searches through descriptions of options. Quite nice thing.

Hmm, indeed it does.  It only highlights the option, not the matched
search string, which is probably OK.  I didn't realize this worked
any differently than "interactive" in this respect (see below).

} zstyle ':completion:*:*:*:default' menu yes select search

Hmm again.  Seems to work inconsistently, with e.g.

    zstyle ':completion:*' menu 'yes=long' 'select=9' search

Sometimes I have to press TAB twice to enter incremental search, other
times it starts the search immediately so a second TAB exits from the
search again (though not all the way out of the menu).

} Don't know yet how to cycle through matches,

You have to bind a key in the menuselect keymap, e.g.,

    bindkey -M menuselect ^S history-incremental-search-forward
    bindkey -M menuselect ^R history-incremental-search-backward

If I ever knew why that isn't bound by default, I've forgotten.  It
probably should copy the bindings from the main keymap.

} would like to leave completion on space

    bindkey -M menuselect ' ' accept-line

(But then you can't search the descriptions for multiple word phrases.)

} don't get what "interactive" mode is.

It's like search, except instead of displaying the whole menu and
searching through it, the list of possible matches is narrowed down
as you type.  THIS mode definitely does NOT search descriptions, and
if both interactive and search are specified, interactive wins.

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 10,  5:35pm, Daniel Shahaf wrote:
}
} Bart Schaefer wrote on Wed, Jun 08, 2016 at 11:40:09 -0700:
} > I see.  I don't really have any good suggestions here -- there is the
} > _complete_fallback idea but all it does is provide alternate guesses
} > for the possible argument positions, which doesn't help in this case.
} > 
} 
} I use the _match completer, so "grep --*bin<TAB>" completes to
} "--binary-files=".  It doesn't match against descriptions though.

Unfortunately that doesn't help either, because the problem is that the
completion function didn't provide --binary-files as a possible match
in the first place.

The question is whether we can provide any useful help when a completion
can't find a match, sort of like the way Google suggests an alternate
query when your search turns up no pages.

Re: Are completions in some way heavy?

[Sebastian Gniazdowski]

On 10 June 2016 at 19:42, Bart Schaefer <schaefer@brasslantern.com> wrote:
> You have to bind a key in the menuselect keymap, e.g.,
>
>     bindkey -M menuselect ^S history-incremental-search-forward
>     bindkey -M menuselect ^R history-incremental-search-backward

Cannot bind Ctrl-S here, but Ctrl-F works fine

> } would like to leave completion on space
>
>     bindkey -M menuselect ' ' accept-line
>
> (But then you can't search the descriptions for multiple word phrases.)

I can even do:

bindkey -M menuselect ' ' 'accept-line ; accept-line'

which ends search instantly (normally that are two spaces). Didn't recognize
this well yet, but it seems to work good

> } don't get what "interactive" mode is.
>
> It's like search, except instead of displaying the whole menu and
> searching through it, the list of possible matches is narrowed down
> as you type.  THIS mode definitely does NOT search descriptions, and
> if both interactive and search are specified, interactive wins.

Too bad for the descriptions, it would be an interesting
functionality. This is rather important, e.g. doing ps -<TAB> and
searching for "format", nice thing

-- 
Best regards,
Sebastian Gniazdowski

Re: Are completions in some way heavy?

[Bart Schaefer]

On Jun 11,  9:15am, Sebastian Gniazdowski wrote:
}
} I can even do:
} 
} bindkey -M menuselect ' ' 'accept-line ; accept-line'
} 
} which ends search instantly

This is equivalent to

    bindkey -M menuselect ' ' undefined-key

because "accept-line ; accept-line" is not a valid widget name.

Re: Are completions in some way heavy?

[Sebastian Gniazdowski]

On 11 June 2016 at 19:18, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jun 11,  9:15am, Sebastian Gniazdowski wrote:
> }
> } I can even do:
> }
> } bindkey -M menuselect ' ' 'accept-line ; accept-line'
> }
> } which ends search instantly
>
> This is equivalent to
>
>     bindkey -M menuselect ' ' undefined-key
>
> because "accept-line ; accept-line" is not a valid widget name.

Tried first with double-accept-line widget, which called accept-line
twice, reverted to inline code after it worked the same. Thanks for
pointing this out it's better to be correct

Best regards,
Sebastian Gniazdowski