Re: Bug#468386: zsh-beta: Slow command completion

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Thu, Feb 28, 2008 at 06:03:41PM +0100, Romain Francoise wrote:
> zsh-beta's command cache is apparently broken: completing a command
> twice takes the same amount of time both times, whereas in zsh4
> (from the zsh package) the first completion takes a few seconds and
> subsequent completions are much faster.

This looks to be a side effect of 24570. Any ideas?

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Feb 28, 12:27pm, Clint Adams wrote:
} Subject: Re: Bug#468386: zsh-beta: Slow command completion
}
} On Thu, Feb 28, 2008 at 06:03:41PM +0100, Romain Francoise wrote:
} > zsh-beta's command cache is apparently broken: completing a command
} > twice takes the same amount of time both times, whereas in zsh4
} > (from the zsh package) the first completion takes a few seconds and
} > subsequent completions are much faster.
} 
} This looks to be a side effect of 24570. Any ideas?

Scrap the part of _path_commands that reads "whatis" output?  As I said
in 24570, I think that was a truly horrible idea in the first place.

Or at least protect it with a style that is not generally set for any
other purpose.

The workaround for Romain is probably to enable caching of the output
by creating a cache-policy style for :completion::complete:-command-::
context.

Re: Bug#468386: zsh-beta: Slow command completion

[Peter Stephenson]

Clint Adams wrote:
> On Thu, Feb 28, 2008 at 06:03:41PM +0100, Romain Francoise wrote:
> > zsh-beta's command cache is apparently broken: completing a command
> > twice takes the same amount of time both times, whereas in zsh4
> > (from the zsh package) the first completion takes a few seconds and
> > subsequent completions are much faster.
> 
> This looks to be a side effect of 24570. Any ideas?

Limit the extra information for commands to people who *really* want it,
by using another style ("extra_verbose") with an appropriate warning
in the doc that this level of verbosity has speed implications.  The
style is then reusable if this crops up elsewhere.

Bart may have more drastic suggestions...

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Thu, Feb 28, 2008 at 05:54:28PM +0000, Peter Stephenson wrote:
> Limit the extra information for commands to people who *really* want it,
> by using another style ("extra_verbose") with an appropriate warning
> in the doc that this level of verbosity has speed implications.  The
> style is then reusable if this crops up elsewhere.

On Thu, Feb 28, 2008 at 09:54:14AM -0800, Bart Schaefer wrote:
> Or at least protect it with a style that is not generally set for any
> other purpose.

I went with a hyphen for consistency's sake.

Index: Completion/Unix/Type/_path_commands
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Type/_path_commands,v
retrieving revision 1.2
diff -u -r1.2 _path_commands
--- Completion/Unix/Type/_path_commands	19 Feb 2008 22:52:22 -0000	1.2
+++ Completion/Unix/Type/_path_commands	28 Feb 2008 18:22:14 -0000
@@ -26,7 +26,7 @@
 _path_commands() {
 local need_desc expl ret=1
 
-if zstyle -T ":completion:${curcontext}:" verbose; then
+if zstyle -T ":completion:${curcontext}:" extra-verbose; then
   local update_policy first
   if [[ $+_command_descriptions -eq 0 ]]; then
     first=yes
Index: Doc/Zsh/compsys.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/compsys.yo,v
retrieving revision 1.204
diff -u -r1.204 compsys.yo
--- Doc/Zsh/compsys.yo	23 Feb 2008 00:10:25 -0000	1.204
+++ Doc/Zsh/compsys.yo	28 Feb 2008 18:22:17 -0000
@@ -1672,6 +1672,12 @@
 tt(ignored-patterns) style, so they can be restored to consideration by
 the tt(_ignored) completer.
 )
+kindex(extra-verbose, completion style)
+item(tt(extra-verbose))(
+If set, the completion listing is more verbose at the cost of
+a probable decrease in completion speed.  Completion performance
+will suffer if this style is set to `true'.
+)
 kindex(ignored-patterns, completion style)
 item(tt(ignored-patterns))(
 A list of patterns; any trial completion matching one of the patterns

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Feb 28,  1:24pm, Clint Adams wrote:
} Subject: Re: Bug#468386: zsh-beta: Slow command completion
}
} On Thu, Feb 28, 2008 at 05:54:28PM +0000, Peter Stephenson wrote:
} > Limit the extra information for commands to people who *really* want it,
} > by using another style ("extra_verbose") with an appropriate warning
} 
} I went with a hyphen for consistency's sake.

Actually, the "verbose" style is just a boolean now, isn't it?  It
could instead become a string style that describes various levels
of verbosity.  Instead of "extra-verbose" with a value of "yes", the
"verbose" style would have a value of "extra".

Or something like that.  Certainly "extra-verbose" is fine, too.

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Thu, Feb 28, 2008 at 05:11:20PM -0800, Bart Schaefer wrote:
> Actually, the "verbose" style is just a boolean now, isn't it?  It
> could instead become a string style that describes various levels
> of verbosity.  Instead of "extra-verbose" with a value of "yes", the
> "verbose" style would have a value of "extra".

No objection from me.

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Feb 28,  9:07pm, Clint Adams wrote:
} Subject: Re: Bug#468386: zsh-beta: Slow command completion
}
} On Thu, Feb 28, 2008 at 05:11:20PM -0800, Bart Schaefer wrote:
} > Actually, the "verbose" style is just a boolean now, isn't it?  It
} > could instead become a string style that describes various levels
} > of verbosity.  Instead of "extra-verbose" with a value of "yes", the
} > "verbose" style would have a value of "extra".
} 
} No objection from me.

Unfortunately that would mess with every "zstyle -T ... verbose" in the
whole completion tree, so it's probably best just to leave it at your
patch for now, unless someone feels strongly about it.

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Fri, Feb 29, 2008 at 09:12:01AM -0800, Bart Schaefer wrote:
> Unfortunately that would mess with every "zstyle -T ... verbose" in the
> whole completion tree, so it's probably best just to leave it at your
> patch for now, unless someone feels strongly about it.

Speaking of which, -T is probably not at all what we want.

Index: Completion/Unix/Type/_path_commands
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Type/_path_commands,v
retrieving revision 1.3
diff -u -r1.3 _path_commands
--- Completion/Unix/Type/_path_commands	28 Feb 2008 18:29:05 -0000	1.3
+++ Completion/Unix/Type/_path_commands	1 Mar 2008 23:04:17 -0000
@@ -26,7 +26,7 @@
 _path_commands() {
 local need_desc expl ret=1
 
-if zstyle -T ":completion:${curcontext}:" extra-verbose; then
+if zstyle -t ":completion:${curcontext}:" extra-verbose; then
   local update_policy first
   if [[ $+_command_descriptions -eq 0 ]]; then
     first=yes

Re: Bug#468386: zsh-beta: Slow command completion

[Romain Francoise]

Hi Bart,

Bart Schaefer <schaefer@brasslantern.com> writes:

> The workaround for Romain is probably to enable caching of the output
> by creating a cache-policy style for :completion::complete:-command-::
> context.

Even after enabling the cache it's still unbearably slow on my
machine (_path_commands already defines a cache-policy style).
Each completion takes about two seconds.

oprofile reports the following:

| CPU: Core 2, speed 1600 MHz (estimated)
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with
| a unit mask of 0x00 (Unhalted core cycles) count 100000
| samples  %        symbol name
| 11356    31.2528  ztrdup
| 5635     15.5080  zalloc
| 3244      8.9278  .plt
| 2766      7.6123  setarrvalue
| 1942      5.3446  zhalloc
| 1460      4.0181  arrlen
| 1457      4.0098  freearray

Which suggests that most of the time is spent building the arrays
fed to compadd and/or formatting $descs w/ zformat.  AIUI, before
your change for 24570 most of the contents of $commands were being
thrown away, and it's no longer the case.

It matters since the array is quite large, $commands has 3901
elements here, 2805 of which have descriptions:

$ echo $#_command_descriptions
2805
$ stat -c %s ~/.zsh/command-descriptions
145962
$

-- 
Romain Francoise <rfrancoise@debian.org>
http://people.debian.org/~rfrancoise/

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Mar 2,  4:07pm, Romain Francoise wrote:
}
} Which suggests that most of the time is spent building the arrays
} fed to compadd and/or formatting $descs w/ zformat.  AIUI, before
} your change for 24570 most of the contents of $commands were being
} thrown away, and it's no longer the case.

That's correct.  Before 24570 *all* of the contents of $commands were
being thrown away in some circumstances.  You get a choice between
slow and correct or fast and worthless.

After Clint's 24650 you can also have fast and correct but less helpful
(if the command descriptions really were helpful to begin with).

It belatedly occurs to me that the following might work as a replacement
for ${(k)commands[(I)$PREFIX]} and speed things up, except on the very
first call when the cache is populating (which may take a very long time).

--- ../zsh-forge/current/Completion/Unix/Type/_path_commands	2008-03-02 09:57:33.000000000 -0800
+++ Completion/Unix/Type/_path_commands	2008-03-02 10:10:54.000000000 -0800
@@ -50,9 +50,10 @@
 fi
 
 if [[ -n $need_desc ]]; then
-  typeset -a dcmds descs cmds
+  typeset -a dcmds descs cmds matches
   local desc cmd sep
-  for cmd in ${(k)commands}; do
+  compadd "$@" -O matches -k commands
+  for cmd in $matches; do
     desc=$_command_descriptions[$cmd]
     if [[ -z $desc ]]; then
       cmds+=$cmd

Re: Bug#468386: zsh-beta: Slow command completion

[Romain Francoise]

Bart Schaefer <schaefer@brasslantern.com> writes:

> It belatedly occurs to me that the following might work as a
> replacement for ${(k)commands[(I)$PREFIX]} and speed things up,
> except on the very first call when the cache is populating (which
> may take a very long time).

That does speed things up considerably, it's almost as fast as
before and definitely usable again.  Thanks!

-- 
Romain Francoise <rfrancoise@debian.org>
http://people.debian.org/~rfrancoise/

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Mar 2,  8:29pm, Romain Francoise wrote:
} Subject: Re: Bug#468386: zsh-beta: Slow command completion
}
} > It belatedly occurs to me that the following might work as a
} > replacement for ${(k)commands[(I)$PREFIX]} and speed things up
} 
} That does speed things up considerably, it's almost as fast as
} before and definitely usable again.  Thanks!

OK, so one remaining problem with _path_commands is that "whatis -s"
is not portable.

What does "whatis --version" say on Debian?

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Sun, Mar 02, 2008 at 12:52:08PM -0800, Bart Schaefer wrote:
> OK, so one remaining problem with _path_commands is that "whatis -s"
> is not portable.
> 
> What does "whatis --version" say on Debian?

whatis 2.5.1

2.5.1 is the version of man-db whence whatis comes.

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Mar 2,  4:46pm, Clint Adams wrote:
} Subject: Re: Bug#468386: zsh-beta: Slow command completion
}
} On Sun, Mar 02, 2008 at 12:52:08PM -0800, Bart Schaefer wrote:
} > What does "whatis --version" say on Debian?
} 
} whatis 2.5.1
} 
} 2.5.1 is the version of man-db whence whatis comes.

Right.  On my RHEL system it says:

whatis from man-1.5o1

I should have asked before, but does Debian have "apropos"?

I'm trying to figure out the best incarnation of something like this:

  _call_whatis() {
    case "$(whatis --version)" in
    (whatis from *)
      local -A args
      zparseopts -D -A args s: r:
      apropos "${args[-r]:-"$@"}" | fgrep "($args[-s]"
      ;;
    (*) whatis "$@";;
    esac
  }

It might be better to test for existence of apropos than to try to
parse the output of whatis --version.  Or it might not.

Re: Bug#468386: zsh-beta: Slow command completion

[Clint Adams]

On Sun, Mar 02, 2008 at 02:45:12PM -0800, Bart Schaefer wrote:
> I should have asked before, but does Debian have "apropos"?

lrwxrwxrwx 1 root root 6 2008-02-08 18:27 /usr/bin/apropos -> whatis

% apropos --version
apropos 2.5.1

% diff -u <(whatis --help) <(apropos --help)
--- /proc/self/fd/11    2008-03-02 18:02:03.840316938 -0500
+++ /proc/self/fd/12    2008-03-02 18:02:03.840316938 -0500
@@ -1,9 +1,11 @@
-Usage: whatis [OPTION...] KEYWORD...
+Usage: apropos [OPTION...] KEYWORD...
 
   -d, --debug                emit debugging messages
   -v, --verbose              print verbose warning messages
+  -e, --exact                search each keyword for exact match
   -r, --regex                interpret each keyword as a regex
   -w, --wildcard             the keyword(s) contain wildcards
+  -a, --and                  require all keywords to match
   -l, --long                 do not trim output to terminal width
   -C, --config-file=FILE     use this user configuration file
   -L, --locale=LOCALE        define the locale for this search
@@ -17,4 +19,6 @@
 Mandatory or optional arguments to long options are also mandatory or optional
 for any corresponding short options.
 
+The --regex option is enabled by default.
+
 Report bugs to cjwatson@debian.org.

> It might be better to test for existence of apropos than to try to
> parse the output of whatis --version.  Or it might not.

No idea.

Re: Bug#468386: zsh-beta: Slow command completion

[Bart Schaefer]

On Mar 2,  6:02pm, Clint Adams wrote:
}
} > It might be better to test for existence of apropos than to try to
} > parse the output of whatis --version.  Or it might not.
} 
} No idea.

Since both have "apropos" as well, it seems testing --version is the
best way to go.

Re: Bug#468386: zsh-beta: Slow command completion

[Oliver Kiddle]

Bart wrote:
> On Mar 2,  6:02pm, Clint Adams wrote:
> }
> } > It might be better to test for existence of apropos than to try to
> } > parse the output of whatis --version.  Or it might not.
> 
> Since both have "apropos" as well, it seems testing --version is the
> best way to go.

I'd suspect that using man -k is the most portable.
I'm sure I've seen systems that lack apropos. -s to whatis on Solaris
is for specifying a section number. I had never heard of whatis -s. Note
that I haven't really followed the thread if this doesn't make sense.

Oliver