startup delay from compinit

startup delay from compinit

[Anthony Heading]

Hi,

I'd like to start using the "new" completion system, but
running compinit from .zshrc seems to take about two seconds,
which is frustrating since otherwise a new xterm takes only
a few tens of milliseconds to be ready to use. (For ref,
I'm using a 750Mhz Solaris sparcv9 machine). 

I believe I've followed the instructions in zshcompsys, and
have .zcompdump being created, and so on.  So am I doing something
wrong, or is this time overhead to be expected?  If so, is
there any alternative way to load all the completion functions
lazily?

Regards

Anthony

This communication is for informational purposes only.  It is not intended as
an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. All market prices, data
and other information are not warranted as to completeness or accuracy and
are subject to change without notice. Any comments or statements made herein
do not necessarily reflect those of J.P. Morgan Chase & Co., its
subsidiaries and affiliates.

Re: startup delay from compinit

[Dan Nelson]

In the last episode (Oct 07), Anthony Heading said:
> I'd like to start using the "new" completion system, but running
> compinit from .zshrc seems to take about two seconds, which is
> frustrating since otherwise a new xterm takes only a few tens of
> milliseconds to be ready to use. (For ref, I'm using a 750Mhz Solaris
> sparcv9 machine).
> 
> I believe I've followed the instructions in zshcompsys, and have
> .zcompdump being created, and so on.  So am I doing something wrong,
> or is this time overhead to be expected?  If so, is there any
> alternative way to load all the completion functions lazily?

Is your home direcory used by different machines running zsh?  compinit
will regenerate the .zcompdump file if it thinks it doesn't match the
current system.

I run this in my startup script instead of plain compinit:

compinit -d $(print -P "${ZDOTDIR:-$HOME}/.zcompdump_%m")

which creates a separate ~/.zcompdump_machinename for each machine I
log into.
 
-- 
	Dan Nelson
	dnelson@allantgroup.com

Re: startup delay from compinit

[Bart Schaefer]

On Oct 7,  7:58pm, Anthony Heading wrote:
}
} I'd like to start using the "new" completion system, but
} running compinit from .zshrc seems to take about two seconds
[...]
} I'm using a 750Mhz Solaris sparcv9 machine). 

That's rather odd.  On my paltry 233MHz i586-linux PC, it takes on the
order of 1 second to run "compinit -D" (which explicitly skips and does
not recreate .zcompdump), about 2 seconds to run "compinit" when it is
creating .zcompdump, and less than 1 second when loading from .zcompdump
after it exists.  Starting from a fresh "zsh -f" every time and running
compinit by hand, that is.

Are you perhaps dealing with automounted NFS filesystems that frequently
expire and have to be remounted?

} I believe I've followed the instructions in zshcompsys, and
} have .zcompdump being created, and so on.  So am I doing something
} wrong, or is this time overhead to be expected?

If you want us to determine whether you're doing something wrong,
you'll have to show us exactly what you added to your .zshrc or other
startup files for this, not just tell us vaguely what you believe.

} If so, is there any alternative way to load all the completion
} functions lazily?

The functions themselves are all loaded lazily no matter what -- all
compinit does is create autoloads for them.  If anything is taking time,
it's scanning the directories to discover what functions are available.
That always happens unless you use "compinit -C", regardless of whether
the .zcompdump file exists.

Did you build and install zsh yourself?  Did you configure it with
--enable-function-subdirs?  It may be taking a lot longer to scan a
directory hierarchy than it would a flat directory -- you should try
--disable-function-subdirs, which is what I use, and see if that
speeds things up.

Re: startup delay from compinit

[Anthony Heading]

On Tue, Oct 07, 2003 at 03:40:01PM +0000, Bart Schaefer wrote:
> That's rather odd.

Thanks. That was really the hint I needed. Turned out that I
broke everything last time I tried to get it working: I'd
tried to nudge compaudit to accept that files which we have
carefully installed with a special "application" group are
_not_ in fact insecure.  (Looks rather like there is a similar
existing hack for the 'staff' group triggered
by -f /etc/debian_version)

And I was trying to force that externally since I didn't
want to maintain a variant of compaudit in perpetuity,
since compaudit contains the directory scanning code.

But I evidently got it wrong - I'd returned OK from compaudit
without correctly having set _i_files, which meant the
function count was zero, which meant the compdump file
was rejected.

Thanks for the help - I'll have to mull how best to fix
this - turning off security measures (i.e. using -u here)
isn't normally appealing on principle, but when the checking
rules admit to being so arbitrary and system-conditional
they're clearly not always going to work.

Thanks

Anthony

PS. Small patch for conceptual anachronism.  (Young people
today blaa blaa, think they invented everything...)  It's
actually misleading, because it doesn't appear to apply
only to RedHat, yet there is code which is Debian specific
in the file.


--- Completion/compaudit.Orig   Wed Oct  8 11:12:15 2003
+++ Completion/compaudit        Wed Oct  8 11:14:07 2003
@@ -76,7 +76,7 @@
 
 [[ $_i_fail == use ]] && return 0
 
-# RedHat Linux "per-user groups" check.  This is tricky, because it's very
+# "Per-user groups" check.  This is tricky, because it's very
 # difficult to tell whether the sysadmin has put someone else into your
 # "private" group (e.g., via the default group field in /etc/passwd, or
 # by NFS group sharing with an untrustworthy machine).  So we must assume

This communication is for informational purposes only.  It is not intended as
an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. All market prices, data
and other information are not warranted as to completeness or accuracy and
are subject to change without notice. Any comments or statements made herein
do not necessarily reflect those of J.P. Morgan Chase & Co., its
subsidiaries and affiliates.

Re: startup delay from compinit

[Bart Schaefer]

On Oct 8, 11:29am, Anthony Heading wrote:
} Subject: Re: startup delay from compinit
}
} tried to nudge compaudit to accept that files which we have
} carefully installed with a special "application" group are
} _not_ in fact insecure.

This sounds like a job for a zstyle.  "trusted-groups" or some such?
I'll think about it and perhaps send a patch to zsh-workers later.

} Thanks for the help - I'll have to mull how best to fix
} this - turning off security measures (i.e. using -u here)
} isn't normally appealing on principle, but when the checking
} rules admit to being so arbitrary and system-conditional
} they're clearly not always going to work.

The major problem with compaudit is of course that if you compromise
the directory where compaudit itself lives, then everything else it
does is suspect.

Also, I recently noticed that compaudit is not handling symbolic links
properly.  Patch below.
 
} PS. Small patch for conceptual anachronism.  (Young people
} today blaa blaa, think they invented everything...)  It's
} actually misleading, because it doesn't appear to apply
} only to RedHat, yet there is code which is Debian specific
} in the file.

There's really a vendor other than RedHat who regularly creates a group
for each user with the same name as the username?  Who?  In the early
1990s I was porting software to something like 25 unix variants, and I'd
never encountered it before RedHat 5 came out.

In any case that comment is there because I ran into issues with the
per-user groups on my RedHat system and wrote that code to compensate.
The comment as it stands is historically accurate; I deliberately made
the code not care if the system actually was RedHat, but RedHat is the
reason it's there.  The Debian-specific stuff was added much later, by
someone else.

Here's the symlinked fpath directories patch.  Essentially, we have to
follow symlinks when testing the files themselves, and appending /.. to
a symlink will find the parent of the directory to which the symlink
refers, rather than the directory containing the symlink -- the latter
is really the one we care about.

Index: Completion/compaudit
===================================================================
--- compaudit	15 Sep 2001 19:16:20 -0000	1.3
+++ compaudit	8 Oct 2003 03:19:16 -0000
@@ -102,11 +102,11 @@
 #   (including zwc files)
 
 if [[ $GROUP == $LOGNAME && ( -z $GROUPMEM || $GROUPMEM == $LOGNAME ) ]]; then
-  _i_wdirs=( ${^fpath}(Nf:g+w:^g:${GROUP}:,f:o+w:,^u0u${EUID})
-             ${^fpath}/..(Nf:g+w:^g:${GROUP}:,f:o+w:,^u0u${EUID}) )
+  _i_wdirs=( ${^fpath}(N-f:g+w:^g:${GROUP}:,-f:o+w:,-^u0u${EUID})
+             ${^fpath:h}(N-f:g+w:^g:${GROUP}:,-f:o+w:,-^u0u${EUID}) )
 else
-  _i_wdirs=( ${^fpath}(Nf:g+w:,f:o+w:,^u0u${EUID})
-             ${^fpath}/..(Nf:g+w:,f:o+w:,^u0u${EUID}) )
+  _i_wdirs=( ${^fpath}(N-f:g+w:,-f:o+w:,-^u0u${EUID})
+             ${^fpath:h}(N-f:g+w:,-f:o+w:,-^u0u${EUID}) )
 fi
 
 if [[ -f /etc/debian_version ]]
@@ -115,8 +115,8 @@
 _i_wdirs=( ${_i_wdirs:#/usr/local/*} ${^_i_ulwdir}(Nf:g+ws:^g:staff:,f:o+w:,^u0) )
 fi
 
-_i_wdirs=( $_i_wdirs ${^fpath}.zwc^([^_]*|*~)(N^u0u${EUID}) )
-_i_wfiles=( ${^fpath}/^([^_]*|*~)(N^u0u${EUID}) )
+_i_wdirs=( $_i_wdirs ${^fpath}.zwc^([^_]*|*~)(N-^u0u${EUID}) )
+_i_wfiles=( ${^fpath}/^([^_]*|*~)(N-^u0u${EUID}) )
 
 case "${#_i_wdirs}:${#_i_wfiles}" in
 (0:0) _i_q= ;;