completing automounts with fake-files setting hangs zsh

completing automounts with fake-files setting hangs zsh

[Greg Klanderman]

Hi,

Is the proper way to configure completion of automounts by using the
fake-files style setting?  It seems to work for small numbers of
automounts, but at work we have hundreds, and hitting tab to complete
when the fake-files style setting is fully populated hangs zsh pretty
badly - you cannot use control-C to abort and have to kill it from
another shell.

thanks,
Greg

Re: completing automounts with fake-files setting hangs zsh

[Greg Klanderman]

Further investigation seems to indicate that the problem lies in it
stat'ing all those "fake" files, and some of those stats are taking a
very long time.  Is there some way to have it not do that?  I guess I
really need a "fake-directories" style so it can not stat them, but
assume the automounts are directories.

thanks,
Greg

Re: completing automounts with fake-files setting hangs zsh

[Bart Schaefer]

Back in August, 2007, I wrote:
: Really this whole idea of attempting to complete things that don't
: exist yet puts a huge amount of overhead on the cases where nothing
: ever will.

Now, I get to say, "Here we go again."

On Jan 24,  5:47pm, Greg Klanderman wrote:
} 
} Is the proper way to configure completion of automounts by using the
} fake-files style setting?

On Jan 24, 10:45pm, Greg Klanderman wrote:
} 
} Further investigation seems to indicate that the problem lies in it
} stat'ing all those "fake" files, and some of those stats are taking a
} very long time.  Is there some way to have it not do that?

I don't know the specific method you're using to populate fake-files,
but my suggested approach is to use "zstyle -e" to filter the value
before returning it to the completion system.  Remember completion is
designed so that the shell functions typically collect all potential
matches for a given situation and then leave it up to the internals
to filter them against the command line.

So if you are able to push some of that filtering into the functions
by use of hooks like "zstyle -e", you can optimize much better than
the generic internal filters are able.

Re: completing automounts with fake-files setting hangs zsh

[Greg Klanderman]

>>>>> Bart Schaefer <schaefer@brasslantern.com> writes:

> So if you are able to push some of that filtering into the functions
> by use of hooks like "zstyle -e", you can optimize much better than
> the generic internal filters are able.

I don't think that's going to be a workable solution; if I or one of
my users ever have the misfortune to hit <tab> after an automount
point with not enough text to narrow the match list, then their shell
is just done.  I've waited up to half an hour - it doesn't come back.

Here's what I have done for over 10 years in compctl, based on the
compctl-examples file:

function _comp_autonet  () { ... } # parse etc/auto.net and return all completions
function _comp_autohome () { ... } # parse etc/auto.home and return all completions

compctl -T -x 's[/net/] C[0,^/net/[^/]#/*]'   -K _comp_autonet  -S/ -tn \
            - 's[/home/] C[0,^/home/[^/]#/*]' -K _comp_autohome -S/ -tn

So compctl could deal with the full set of completions without
pre-filtering against the user text.

Under the new completion system, I have something like:

zstyle ':completion:*' fake-files \
       '/net:foo bar baz ..' \
       '/home:user1 user2 user3 ..'

The completion system is smart enough that this doesn't cause any
problem with several hundred fake files defined per automount point,
until I actually try to complete on those fake files.  In fact it's
even fine if the set of completions is reasonably small, so it appears
to be filtering against the user input before stat'ing all the
completions.  It's only when there are a large number of matches that
it gets completely wedged stat'ing them all.

I traced the code from the _path_files function into the compfiles
builtin in the zsh/computil module but then I'm getting pretty lost.
It seems that the compctl mechanism worked because it treated the
completions simply as strings rather than files, and set the -S/
suffix argument so they would behave as directory completions.

If I append a '/' to each file in the fake-files style setting, then
everything is fast, but it appends a space after the completion is
inserted (after the '/').

So maybe the right thing is to modify the compfiles builtin to treat
fake files ending in '/' as directories without stat'ing them?  I'll
poke at that some more if you agree it's reasonable..

thanks,
Greg

Re: completing automounts with fake-files setting hangs zsh

[Greg Klanderman]

>>>>> Greg Klanderman <gak@klanderman.net> writes:

> So maybe the right thing is to modify the compfiles builtin to treat
> fake files ending in '/' as directories without stat'ing them?  I'll
> poke at that some more if you agree it's reasonable..

Well, I now see that the problem is not in compfiles.. it handles the
fake files, and returns them in the result array back to _path_files.
I'm gonna move on to something else for a bit..

greg

Re: completing automounts with fake-files setting hangs zsh

[Phil Pennock]

On 2009-01-25 at 13:56 -0500, Greg Klanderman wrote:
> 
> >>>>> Greg Klanderman <gak@klanderman.net> writes:
> 
> > So maybe the right thing is to modify the compfiles builtin to treat
> > fake files ending in '/' as directories without stat'ing them?  I'll
> > poke at that some more if you agree it's reasonable..
> 
> Well, I now see that the problem is not in compfiles.. it handles the
> fake files, and returns them in the result array back to _path_files.
> I'm gonna move on to something else for a bit..

Bear in mind that tab-completion often looks for spelling mistakes in
previous components of the path.  Some knobs I have tuned at work:

----------------------------8< cut here >8------------------------------
# Prevent reading in parent directories for tab-expansion:
zstyle ':completion:*' preserve-prefix '(/home/*|/auto/*)/'
# Or: zstyle ':completion:*' preserve-prefix '*/'
# Post-4.3.6 flag which does things 'properly'; any prefix component
# which matches an existing dir should be directly accepted, instead of
# being subject to expansion:
zstyle ':completion:*' accept-exact-dirs true
----------------------------8< cut here >8------------------------------

We also have /home/ as an automount for some number of thousands of
users (>10); for me, I only see those entries for which I've already
populated the directory, so readdir() only returns items which have been
pulled into the local host.  Mind, /etc/auto.home is a flat-file being
distributed, rather than LDAP or NIS or whatever.

The other part is that the userdirs variable, for ~<user> expansion,
will by default walk getpwent(), which can get rather tiresome when the
network is congested; userdirs is exposed by zsh/parameter but as a
read-only variable.

Here's how I get around it; zfilter_comments is another function, which
is just a "cat without comment lines, internal to shell" which I'll
include at the botton, since it's not relevant to the core
functionality.
----------------------------8< cut here >8------------------------------
function reset_userdirs {
# There is a zsh internal map, userdirs, exposed by zsh/parameter;
# it's read-only though.
        [[ -f ~/.userdirs ]] || return
        local _u
        local -a _ud
        _ud=( $(zfilter_comments ~/.userdirs) )
        for _u in $_ud; do hash -d $_u="/home/$_u"; done
        hash -d desktop="$HOME/Desktop"
#...
}
 Hack to turn off userdirs completion by overriding userdirs.
# This overrides userdirs as a local variable inside the completion
# system, not touching the global variable.  We don't need to load
# zsh/parameter to achieve this.
#zmodload -i zsh/parameter
_comp_setup+=$'\ntypeset -a userdirs\nreset_userdirs'
reset_userdirs
----------------------------8< cut here >8------------------------------

Other completion system tags you can look at defining include:
  accounts my-accounts other-accounts users


There is in zsh room for improvement in a few areas to deal with large
maps loaded in by the OS slowly in large user environments or slow
filesystems; a way to explore might be adding a couple of zstyle tags
which activate more cautious lookup and adjust the relevant completion
functions to use those, as needed.  After all, that's what 'users',
'accounts', etc do already.  Perhaps with setting those and
preserve-prefix or accept-exact-dirs, you'll be okay; for me, the above
has made things work cleanly and smoothly enough that I haven't been
motivated to come up with decent improvements to feed back into zsh
(since it's only a problem at work and I have higher priorities when
there).

Regards,
-Phil

----------------------------8< cut here >8------------------------------
function zfilter_comments {
        local f infile="$1"
        while read f; do
                [[ -n ${f%%[$' \t']*\#*} && ${f#[#;]} == $f ]] || continue
                print -r -- ${f%%[$' \t']*\#*}
        done < "$infile"
}
----------------------------8< cut here >8------------------------------