[Feature suggestion] (user configurable) timeout for generating completion lists

[Feature suggestion] (user configurable) timeout for generating completion lists

[Arseny Tolmachev]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

Good time of day to everyone.

Sometimes you press [Tab] by instinct when in directory that is
network-mounted and has some millions files in it.
In this situation zsh basically ignores the [Ctrl+C] and don't respond to
anything for a bit of time.
It would be very great if there was a way to specify a timeout for
completion generation for zsh. Error message can be something like
"completion timeouted".
What do you think about this?

There are some questions on stackexchange from users wanting to get the
feature
http://unix.stackexchange.com/questions/32363/how-to-specify-timeout-for-tab-completion-in-zsh
http://unix.stackexchange.com/questions/39404/cancel-zsh-tab-completion
with nothing as answer except the SIGINT, however in my case zsh didn't
respond for SIGINT as well.


Thank you!

Arseny.

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Bart Schaefer]

On Jan 20,  4:13pm, Arseny Tolmachev wrote:
} 
} Sometimes you press [Tab] by instinct when in directory that is
} network-mounted and has some millions files in it.
} In this situation zsh basically ignores the [Ctrl+C] and don't respond to
} anything for a bit of time.

There are two possible situations here.  (Probably more than two, but
only two likely ones.)

(1) The shell process is actually in "disk wait" condition, blocked to
wait for the NFS volume to respond.  If this is the case, there's not
a lot that can be done; any timeout that might be set up will also be
held until the disk condition clears.

(2) The shell is chewing CPU cycles for a large amount of file data,
but ignoring or deferring some signals while it does so.  In this case
there's some hope of a timeout working.

It could also be switching back and forth between these two states ...
but that can be treated as case (2) except that it might take longer
for the timeout to work.

} It would be very great if there was a way to specify a timeout for
} completion generation for zsh. Error message can be something like
} "completion timeouted".
} What do you think about this?

I've been poking at it for a couple of hours trying to repeatably induce
case (2), and I can't get reliably force it to happen.  It would be
helpful if somebody who experiences this problem could attach to the
stuck zsh with gdb and grab a stack trace.  Note that in case (1) above,
gdb is likely to end up wedged as well.

Anyway, anything we could do for a timeout would likely require handling
a signal, which pretty much means we're going to bail out of the shell
functions (e.g., _main_complete and everything below it) that perform
completion.  I don't immediately see how to produce an error message in
that case without messing up the ZLE display unless we also have a shell
level trap handler (instead of a purely internal one).  But in order to
even get that far, we have to know what's holding things up.

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Nikolai Weibull]

On Wed, Jan 22, 2014 at 9:04 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Jan 20,  4:13pm, Arseny Tolmachev wrote:
> }
> } Sometimes you press [Tab] by instinct when in directory that is
> } network-mounted and has some millions files in it.
> } In this situation zsh basically ignores the [Ctrl+C] and don't respond to
> } anything for a bit of time.

> There are two possible situations here.  (Probably more than two, but
> only two likely ones.)

> (1) The shell process is actually in "disk wait" condition, blocked to
> wait for the NFS volume to respond.  If this is the case, there's not
> a lot that can be done; any timeout that might be set up will also be
> held until the disk condition clears.


Time to delegate completion to a subprocess? That’d make it possible
to use Zsh completions from outside the shell as well.

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Bart Schaefer]

On Jan 23,  1:04pm, Nikolai Weibull wrote:
}
} Time to delegate completion to a subprocess? That'd make it possible
} to use Zsh completions from outside the shell as well.

I just knew someone was going to say this.

Completion is very closely tied into ZLE, and ZLE is very closely tied
into the shell.  It's not going to turn into a readline library without
a complete rewrite.

On the other hand, the completion functions themselves are just shell
code, so it's possible to almost arbitrarily replace stuff they do inside
the shell with stuff that's done outside.  The current set of functions
in fact go to an inordinate amount of effort to do just the opposite, on
the assumption that forking and reading from pipes is more expensive /
time-consuming than doing internal string manipulation and globbing.

Back on the first hand, you might not gain much, because the compsys
model is to have the shell functions generate all possible strings in
the correct category and then let the internals filter them against the
contents of the command line.  This is what allows some of the really
powerful stuff like expanding c.u.q into comp.unix.questions.  If you
rely on an external process, you either lose that or duplicate the
effort of calculating it.

We've already seen the "do it externally" plan break down with the _git
issue discussed in the thread leading up to zsh-users/18321 (about which
we're sort of awaiting your input, Nikolai).

However, there are schemes like Thiago Padilha's zpty-based approach that
would allow a separate zsh to compute the completions in the background,
which would in turn allow the foreground zsh to ignore the results if they
didn't come back soon enough.  This is potentially duplication of effort,
and what you'd still lose there (unless you duplicate still more effort)
is file-type-based coloring or tagging and the like, which depend, e.g.,
upon ZLE having examined the filesystem.

But it's jumping the gun a little to discuss all of this before we even
know what's really causing the issue.

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Nikolai Weibull]

On Fri, Jan 24, 2014 at 2:16 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Jan 23,  1:04pm, Nikolai Weibull wrote:
> }
> } Time to delegate completion to a subprocess? That'd make it possible
> } to use Zsh completions from outside the shell as well.

> I just knew someone was going to say this.

:-)

> Completion is very closely tied into ZLE, and ZLE is very closely tied
> into the shell.  It's not going to turn into a readline library without
> a complete rewrite.

True.

> We've already seen the "do it externally" plan break down with the _git
> issue discussed in the thread leading up to zsh-users/18321 (about which
> we're sort of awaiting your input, Nikolai).

Uh, I’ve totally missed this.  I guess a zstyle would work, but
couldn’t one check if completion is being done by partial-path?

I’m sorry, but that’s about as much input as I’ll likely be able to
provide.  I have a negative amount of time available to allocate to
_git.  It seemed like Torstein and others had picked up the pieces?

_git and partial completion, again

[Bart Schaefer]

On Jan 24, 10:00am, Nikolai Weibull wrote:
}
} > We've already seen the "do it externally" plan break down with the _git
} > issue discussed in the thread leading up to zsh-users/18321 (about which
} > we're sort of awaiting your input, Nikolai).
} 
} Uh, I've totally missed this.  I guess a zstyle would work, but
} couldn't one check if completion is being done by partial-path?

I can't think of a good way to distinguish a partial completion from a full
completion of a path that exists only on the remote repo.

Someone suggested that we simply avoid calling __git_files in some cases,
but I don't know git details well enough to make that determination.

So, my best guess is below, which just calls ls-files more than once.
There are probably cases where it could do better; e.g., for modified
files it cbuld first glob $pref* locally and, if anything matches, use
ls-files to filter for only the modified subset.  However, that wouldn't
be appropriate for deleted files, so this may be the best we can do
without some kind of a switch on $opts.


diff --git a/Completion/Unix/Command/_git b/Completion/Unix/Command/_git
index 43a01d9..c09f255 100644
--- a/Completion/Unix/Command/_git
+++ b/Completion/Unix/Command/_git
@@ -5683,9 +5683,17 @@ __git_files () {
   # TODO: --directory should probably be added to $opts when --others is given.
 
   local pref=$gitcdup$gitprefix$PREFIX
+
+  # First allow ls-files to pattern-match in case of remote repository
   files=(${(0)"$(_call_program files git ls-files -z --exclude-standard $opts -- ${pref:+$pref\\\*} 2>/dev/null)"})
   __git_command_successful $pipestatus || return
 
+  # If ls-files succeeded but returned nothing, try again with no pattern
+  if [[ -z "$files" && -n "$pref" ]]; then
+    files=(${(0)"$(_call_program files git ls-files -z --exclude-standard $opts -- 2>/dev/null)"})
+    __git_command_successful $pipestatus || return
+  fi
+
 #  _wanted $tag expl $description _files -g '{'${(j:,:)files}'}' $compadd_opts -
   _wanted $tag expl $description _multi_parts -f $compadd_opts - / files
 }

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Greg Klanderman]

>>>>> On January 23, 2014 Bart Schaefer <schaefer@brasslantern.com> wrote:

> I just knew someone was going to say this.

Ha ha.. I'm sure many were thinking it at least.

> But it's jumping the gun a little to discuss all of this before we even
> know what's really causing the issue.

It will be interesting to see what's really causing the OP's issue.

It does seem that if the OP's issue is slow NFS mounts, it might be
possible to move just the necessary filesystem access calls during
completion to a separate process.

For slow completion due to calling out to a separate process for
completions (_git comes to mind) if that is not currently
interruptible it seems like there might be some hope of adding a
timeout or ensuring C-c will interrupt.

In general I'd much prefer C-c working to interrupt than adding an
arbitrary timeout, as I might want to decide on a case by case basis
how long I'm willing to wait.

The biggest problem I've had with completion being unresponsive was
due to automounts at work (currently have ~53k under a handful of
mount points).  Using the fake-files zstyle mostly works until
completion tries to stat all those automounts to determine the file
type in order to append a '/' or ' '.  So locally I run zsh with a
small patch (to ztat()) that uses a shell variable to configure
automount roots under which all immediate entries are assumed to be
directories without stat'ing.  Works great, though I don't think
several years ago you were willing to incorporate that into zsh.

Greg

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Bart Schaefer]

On Jan 24,  9:46am, Greg Klanderman wrote:
}
} For slow completion due to calling out to a separate process for
} completions (_git comes to mind) if that is not currently
} interruptible it seems like there might be some hope of adding a
} timeout or ensuring C-c will interrupt.

This is the thing I had a hard time reproducing.  Even if I force
the completion function to be very busy with a loop, I can interrupt
it.  (Oddly the value of $? is always 0 in _main_complete after I
hit ^C, so I can't detect the signal and issue a message.)

} The biggest problem I've had with completion being unresponsive was
} due to automounts at work (currently have ~53k under a handful of
} mount points).  Using the fake-files zstyle mostly works until
} completion tries to stat all those automounts

I think this is analogous to the situation that started this thread.

} to determine the file type in order to append a '/' or ' '.

... which is difficult to do if there's an external program doing the
stat.  Not impossible, but it means passing a lot of structured data
through a pipe, which might slow down more common cases.

(Maybe this is why Meino wants "find -print0 -ls". :-> )

} So locally I run zsh with a
} small patch (to ztat()) that uses a shell variable to configure
} automount roots under which all immediate entries are assumed to be
} directories without stat'ing.  Works great, though I don't think
} several years ago you were willing to incorporate that into zsh.

Doesn't sound terrible, but I don't think it would solve the original
problem, which if I'm right is that NFS listing is big/slow after the
mount is already completed.

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Greg Klanderman]

>>>>> On January 25, 2014 Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Jan 24,  9:46am, Greg Klanderman wrote:
> }
> } For slow completion due to calling out to a separate process for
> } completions (_git comes to mind) if that is not currently
> } interruptible it seems like there might be some hope of adding a
> } timeout or ensuring C-c will interrupt.

> This is the thing I had a hard time reproducing.  Even if I force
> the completion function to be very busy with a loop, I can interrupt
> it.  (Oddly the value of $? is always 0 in _main_complete after I
> hit ^C, so I can't detect the signal and issue a message.)

Did you try a busy completion function, or external process?

Great if this case works well.. I didn't test it and haven't noticed a
problem, but I don't think I commonly use any completion that calls
out to an external program and is slow.  I know some have complained
about the git completion with big repos, but I'm fairly new to git and
haven't found a case that's slow yet.  If someone has an example, let
me know..

> } The biggest problem I've had with completion being unresponsive was
> } due to automounts at work (currently have ~53k under a handful of
> } mount points).  Using the fake-files zstyle mostly works until
> } completion tries to stat all those automounts

> I think this is analogous to the situation that started this thread.

Similar issue, though my case will only come up if you go to the
trouble of setting up a fake-files zstyle with the automount points.
The slowness is from stat'ing 53k automounts, and it does not seem to
be interruptible with C-c.  Also if you hit tab after "/home/" it ends
up actually mounting as many of the automounts under /home as it can
before you manage to kill the shell from another window, which is
non-ideal.

In the original post it seemed that the slow call was probably reading
a large directory on a slow NFS mount.

> } to determine the file type in order to append a '/' or ' '.

> ... which is difficult to do if there's an external program doing the
> stat.  Not impossible, but it means passing a lot of structured data
> through a pipe, which might slow down more common cases.

The protobuf library works well for that sort of thing though I'm not
sure if it has straight C bindings.

> (Maybe this is why Meino wants "find -print0 -ls". :-> )

> } So locally I run zsh with a
> } small patch (to ztat()) that uses a shell variable to configure
> } automount roots under which all immediate entries are assumed to be
> } directories without stat'ing.  Works great, though I don't think
> } several years ago you were willing to incorporate that into zsh.

> Doesn't sound terrible,

I'll attach the patch below, if you'd be willing to incorporate
something like it let me know, I'd certainly prefer to have it part of
zsh.  I would add documentation of course.

It gets used like this:

completion_nostat_dirs=(/home /net)
zstyle ':completion:*' fake-files '/home:foo bar baz' '/net:aaa bbb ccc' ...

> but I don't think it would solve the original
> problem, which if I'm right is that NFS listing is big/slow after the
> mount is already completed.

Correct.

Greg

Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.122
diff -u -r1.122 init.c
--- Src/init.c	5 Jan 2012 20:01:25 -0000	1.122
+++ Src/init.c	9 Jan 2012 20:09:36 -0000
@@ -827,6 +827,8 @@
     modulestab = newmoduletable(17, "modules");
     linkedmodules = znewlinklist();
 
+    completion_nostat_dirs = mkarray(NULL);
+
     /* Set default prompts */
     if(unset(INTERACTIVE)) {
 	prompt = ztrdup("");
Index: Src/params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/params.c,v
retrieving revision 1.177
diff -u -r1.177 params.c
--- Src/params.c	7 Jan 2012 23:21:00 -0000	1.177
+++ Src/params.c	9 Jan 2012 20:09:36 -0000
@@ -104,6 +104,10 @@
      zoptind,		/* $OPTIND      */
      shlvl;		/* $SHLVL       */
 
+/**/
+mod_export
+char **completion_nostat_dirs;
+
 /* $histchars */
  
 /**/
@@ -356,6 +360,8 @@
 IPDEF9F("*", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY),
 IPDEF9F("@", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY),
 
+IPDEF9("completion_nostat_dirs", &completion_nostat_dirs, NULL),
+
 /*
  * This empty row indicates the end of parameters available in
  * all emulations.
Index: Src/Zle/compresult.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Zle/compresult.c,v
retrieving revision 1.82
diff -u -r1.82 compresult.c
--- Src/Zle/compresult.c	14 May 2011 00:07:42 -0000	1.82
+++ Src/Zle/compresult.c	9 Jan 2012 20:09:37 -0000
@@ -868,11 +868,25 @@
 ztat(char *nam, struct stat *buf, int ls)
 {
     int ret;
+    char **pp;
 
     nam = unmeta(nam);
     if (!nam)
 	return -1;
 
+    for (pp = completion_nostat_dirs; *pp; pp++) {
+        int len = strlen(*pp);
+
+        if ((*pp)[len-1] == '/')
+            len--;
+
+        if (strncmp(nam, *pp, len) == 0 && nam[len] == '/' && strchr(nam+len+1, '/') == NULL) {
+            memset(buf, 0, sizeof(*buf));
+            buf->st_mode = S_IFDIR; /* pretend it's a directory */
+            return 0;
+        }
+    }
+
     if ((ret = ls ? lstat(nam, buf) : stat(nam, buf))) {
 	char *p, *q;
 

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Greg Klanderman]

Hi Bart, I just found one issue with interrupting completion - I hit
tab after "ssh" and it asked:

| zsh: do you wish to see all 209401 possibilities (104701 lines)? 

I accidentally hit tab again, and it looks like there is no way to
interrupt the *output* of all those completions.  I had to kill the
shell, with -9, as after several minutes it was only up to the 'B's.

This is with a freshly updated-from-git zsh.

thanks,
Greg

Re: [Feature suggestion] (user configurable) timeout for generating completion lists

[Shawn Wilson]

Arseny Tolmachev <eiennohito@gmail.com> wrote:
>Good time of day to everyone.
>
>Sometimes you press [Tab] by instinct when in directory that is
>network-mounted and has some millions files in it.
>In this situation zsh basically ignores the [Ctrl+C] and don't respond
>to
>anything for a bit of time.
>It would be very great if there was a way to specify a timeout for
>completion generation for zsh. Error message can be something like
>"completion timeouted".
>What do you think about this?
>

IDK what the problem with NFS might be however IIRC this works with rsync (haven't tried with sshfs though). So maybe try that?

>There are some questions on stackexchange from users wanting to get the
>feature
>http://unix.stackexchange.com/questions/32363/how-to-specify-timeout-for-tab-completion-in-zsh
>http://unix.stackexchange.com/questions/39404/cancel-zsh-tab-completion
>with nothing as answer except the SIGINT, however in my case zsh didn't
>respond for SIGINT as well.
>
>
>Thank you!
>
>Arseny.