* [Feature suggestion] (user configurable) timeout for generating completion lists @ 2014-01-20 7:13 Arseny Tolmachev 2014-01-22 8:04 ` Bart Schaefer 2014-01-22 13:51 ` Shawn Wilson 0 siblings, 2 replies; 11+ messages in thread From: Arseny Tolmachev @ 2014-01-20 7:13 UTC (permalink / raw) To: zsh-users [-- Attachment #1: Type: text/plain, Size: 806 bytes --] Good time of day to everyone. Sometimes you press [Tab] by instinct when in directory that is network-mounted and has some millions files in it. In this situation zsh basically ignores the [Ctrl+C] and don't respond to anything for a bit of time. It would be very great if there was a way to specify a timeout for completion generation for zsh. Error message can be something like "completion timeouted". What do you think about this? There are some questions on stackexchange from users wanting to get the feature http://unix.stackexchange.com/questions/32363/how-to-specify-timeout-for-tab-completion-in-zsh http://unix.stackexchange.com/questions/39404/cancel-zsh-tab-completion with nothing as answer except the SIGINT, however in my case zsh didn't respond for SIGINT as well. Thank you! Arseny. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-20 7:13 [Feature suggestion] (user configurable) timeout for generating completion lists Arseny Tolmachev @ 2014-01-22 8:04 ` Bart Schaefer 2014-01-23 12:04 ` Nikolai Weibull 2014-01-22 13:51 ` Shawn Wilson 1 sibling, 1 reply; 11+ messages in thread From: Bart Schaefer @ 2014-01-22 8:04 UTC (permalink / raw) To: zsh-users On Jan 20, 4:13pm, Arseny Tolmachev wrote: } } Sometimes you press [Tab] by instinct when in directory that is } network-mounted and has some millions files in it. } In this situation zsh basically ignores the [Ctrl+C] and don't respond to } anything for a bit of time. There are two possible situations here. (Probably more than two, but only two likely ones.) (1) The shell process is actually in "disk wait" condition, blocked to wait for the NFS volume to respond. If this is the case, there's not a lot that can be done; any timeout that might be set up will also be held until the disk condition clears. (2) The shell is chewing CPU cycles for a large amount of file data, but ignoring or deferring some signals while it does so. In this case there's some hope of a timeout working. It could also be switching back and forth between these two states ... but that can be treated as case (2) except that it might take longer for the timeout to work. } It would be very great if there was a way to specify a timeout for } completion generation for zsh. Error message can be something like } "completion timeouted". } What do you think about this? I've been poking at it for a couple of hours trying to repeatably induce case (2), and I can't get reliably force it to happen. It would be helpful if somebody who experiences this problem could attach to the stuck zsh with gdb and grab a stack trace. Note that in case (1) above, gdb is likely to end up wedged as well. Anyway, anything we could do for a timeout would likely require handling a signal, which pretty much means we're going to bail out of the shell functions (e.g., _main_complete and everything below it) that perform completion. I don't immediately see how to produce an error message in that case without messing up the ZLE display unless we also have a shell level trap handler (instead of a purely internal one). But in order to even get that far, we have to know what's holding things up. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-22 8:04 ` Bart Schaefer @ 2014-01-23 12:04 ` Nikolai Weibull 2014-01-24 1:16 ` Bart Schaefer 0 siblings, 1 reply; 11+ messages in thread From: Nikolai Weibull @ 2014-01-23 12:04 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh Users On Wed, Jan 22, 2014 at 9:04 AM, Bart Schaefer <schaefer@brasslantern.com> wrote: > On Jan 20, 4:13pm, Arseny Tolmachev wrote: > } > } Sometimes you press [Tab] by instinct when in directory that is > } network-mounted and has some millions files in it. > } In this situation zsh basically ignores the [Ctrl+C] and don't respond to > } anything for a bit of time. > There are two possible situations here. (Probably more than two, but > only two likely ones.) > (1) The shell process is actually in "disk wait" condition, blocked to > wait for the NFS volume to respond. If this is the case, there's not > a lot that can be done; any timeout that might be set up will also be > held until the disk condition clears. Time to delegate completion to a subprocess? That’d make it possible to use Zsh completions from outside the shell as well. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-23 12:04 ` Nikolai Weibull @ 2014-01-24 1:16 ` Bart Schaefer 2014-01-24 9:00 ` Nikolai Weibull 2014-01-24 14:46 ` [Feature suggestion] (user configurable) timeout for generating completion lists Greg Klanderman 0 siblings, 2 replies; 11+ messages in thread From: Bart Schaefer @ 2014-01-24 1:16 UTC (permalink / raw) To: Zsh Users On Jan 23, 1:04pm, Nikolai Weibull wrote: } } Time to delegate completion to a subprocess? That'd make it possible } to use Zsh completions from outside the shell as well. I just knew someone was going to say this. Completion is very closely tied into ZLE, and ZLE is very closely tied into the shell. It's not going to turn into a readline library without a complete rewrite. On the other hand, the completion functions themselves are just shell code, so it's possible to almost arbitrarily replace stuff they do inside the shell with stuff that's done outside. The current set of functions in fact go to an inordinate amount of effort to do just the opposite, on the assumption that forking and reading from pipes is more expensive / time-consuming than doing internal string manipulation and globbing. Back on the first hand, you might not gain much, because the compsys model is to have the shell functions generate all possible strings in the correct category and then let the internals filter them against the contents of the command line. This is what allows some of the really powerful stuff like expanding c.u.q into comp.unix.questions. If you rely on an external process, you either lose that or duplicate the effort of calculating it. We've already seen the "do it externally" plan break down with the _git issue discussed in the thread leading up to zsh-users/18321 (about which we're sort of awaiting your input, Nikolai). However, there are schemes like Thiago Padilha's zpty-based approach that would allow a separate zsh to compute the completions in the background, which would in turn allow the foreground zsh to ignore the results if they didn't come back soon enough. This is potentially duplication of effort, and what you'd still lose there (unless you duplicate still more effort) is file-type-based coloring or tagging and the like, which depend, e.g., upon ZLE having examined the filesystem. But it's jumping the gun a little to discuss all of this before we even know what's really causing the issue. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-24 1:16 ` Bart Schaefer @ 2014-01-24 9:00 ` Nikolai Weibull 2014-01-25 20:09 ` _git and partial completion, again Bart Schaefer 2014-01-24 14:46 ` [Feature suggestion] (user configurable) timeout for generating completion lists Greg Klanderman 1 sibling, 1 reply; 11+ messages in thread From: Nikolai Weibull @ 2014-01-24 9:00 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh Users On Fri, Jan 24, 2014 at 2:16 AM, Bart Schaefer <schaefer@brasslantern.com> wrote: > On Jan 23, 1:04pm, Nikolai Weibull wrote: > } > } Time to delegate completion to a subprocess? That'd make it possible > } to use Zsh completions from outside the shell as well. > I just knew someone was going to say this. :-) > Completion is very closely tied into ZLE, and ZLE is very closely tied > into the shell. It's not going to turn into a readline library without > a complete rewrite. True. > We've already seen the "do it externally" plan break down with the _git > issue discussed in the thread leading up to zsh-users/18321 (about which > we're sort of awaiting your input, Nikolai). Uh, I’ve totally missed this. I guess a zstyle would work, but couldn’t one check if completion is being done by partial-path? I’m sorry, but that’s about as much input as I’ll likely be able to provide. I have a negative amount of time available to allocate to _git. It seemed like Torstein and others had picked up the pieces? ^ permalink raw reply [flat|nested] 11+ messages in thread
* _git and partial completion, again 2014-01-24 9:00 ` Nikolai Weibull @ 2014-01-25 20:09 ` Bart Schaefer 0 siblings, 0 replies; 11+ messages in thread From: Bart Schaefer @ 2014-01-25 20:09 UTC (permalink / raw) To: Zsh Users On Jan 24, 10:00am, Nikolai Weibull wrote: } } > We've already seen the "do it externally" plan break down with the _git } > issue discussed in the thread leading up to zsh-users/18321 (about which } > we're sort of awaiting your input, Nikolai). } } Uh, I've totally missed this. I guess a zstyle would work, but } couldn't one check if completion is being done by partial-path? I can't think of a good way to distinguish a partial completion from a full completion of a path that exists only on the remote repo. Someone suggested that we simply avoid calling __git_files in some cases, but I don't know git details well enough to make that determination. So, my best guess is below, which just calls ls-files more than once. There are probably cases where it could do better; e.g., for modified files it cbuld first glob $pref* locally and, if anything matches, use ls-files to filter for only the modified subset. However, that wouldn't be appropriate for deleted files, so this may be the best we can do without some kind of a switch on $opts. diff --git a/Completion/Unix/Command/_git b/Completion/Unix/Command/_git index 43a01d9..c09f255 100644 --- a/Completion/Unix/Command/_git +++ b/Completion/Unix/Command/_git @@ -5683,9 +5683,17 @@ __git_files () { # TODO: --directory should probably be added to $opts when --others is given. local pref=$gitcdup$gitprefix$PREFIX + + # First allow ls-files to pattern-match in case of remote repository files=(${(0)"$(_call_program files git ls-files -z --exclude-standard $opts -- ${pref:+$pref\\\*} 2>/dev/null)"}) __git_command_successful $pipestatus || return + # If ls-files succeeded but returned nothing, try again with no pattern + if [[ -z "$files" && -n "$pref" ]]; then + files=(${(0)"$(_call_program files git ls-files -z --exclude-standard $opts -- 2>/dev/null)"}) + __git_command_successful $pipestatus || return + fi + # _wanted $tag expl $description _files -g '{'${(j:,:)files}'}' $compadd_opts - _wanted $tag expl $description _multi_parts -f $compadd_opts - / files } ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-24 1:16 ` Bart Schaefer 2014-01-24 9:00 ` Nikolai Weibull @ 2014-01-24 14:46 ` Greg Klanderman 2014-01-25 20:49 ` Bart Schaefer 1 sibling, 1 reply; 11+ messages in thread From: Greg Klanderman @ 2014-01-24 14:46 UTC (permalink / raw) To: zsh-users >>>>> On January 23, 2014 Bart Schaefer <schaefer@brasslantern.com> wrote: > I just knew someone was going to say this. Ha ha.. I'm sure many were thinking it at least. > But it's jumping the gun a little to discuss all of this before we even > know what's really causing the issue. It will be interesting to see what's really causing the OP's issue. It does seem that if the OP's issue is slow NFS mounts, it might be possible to move just the necessary filesystem access calls during completion to a separate process. For slow completion due to calling out to a separate process for completions (_git comes to mind) if that is not currently interruptible it seems like there might be some hope of adding a timeout or ensuring C-c will interrupt. In general I'd much prefer C-c working to interrupt than adding an arbitrary timeout, as I might want to decide on a case by case basis how long I'm willing to wait. The biggest problem I've had with completion being unresponsive was due to automounts at work (currently have ~53k under a handful of mount points). Using the fake-files zstyle mostly works until completion tries to stat all those automounts to determine the file type in order to append a '/' or ' '. So locally I run zsh with a small patch (to ztat()) that uses a shell variable to configure automount roots under which all immediate entries are assumed to be directories without stat'ing. Works great, though I don't think several years ago you were willing to incorporate that into zsh. Greg ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-24 14:46 ` [Feature suggestion] (user configurable) timeout for generating completion lists Greg Klanderman @ 2014-01-25 20:49 ` Bart Schaefer 2014-01-29 16:29 ` Greg Klanderman 2014-02-25 15:35 ` Greg Klanderman 0 siblings, 2 replies; 11+ messages in thread From: Bart Schaefer @ 2014-01-25 20:49 UTC (permalink / raw) To: zsh-users On Jan 24, 9:46am, Greg Klanderman wrote: } } For slow completion due to calling out to a separate process for } completions (_git comes to mind) if that is not currently } interruptible it seems like there might be some hope of adding a } timeout or ensuring C-c will interrupt. This is the thing I had a hard time reproducing. Even if I force the completion function to be very busy with a loop, I can interrupt it. (Oddly the value of $? is always 0 in _main_complete after I hit ^C, so I can't detect the signal and issue a message.) } The biggest problem I've had with completion being unresponsive was } due to automounts at work (currently have ~53k under a handful of } mount points). Using the fake-files zstyle mostly works until } completion tries to stat all those automounts I think this is analogous to the situation that started this thread. } to determine the file type in order to append a '/' or ' '. ... which is difficult to do if there's an external program doing the stat. Not impossible, but it means passing a lot of structured data through a pipe, which might slow down more common cases. (Maybe this is why Meino wants "find -print0 -ls". :-> ) } So locally I run zsh with a } small patch (to ztat()) that uses a shell variable to configure } automount roots under which all immediate entries are assumed to be } directories without stat'ing. Works great, though I don't think } several years ago you were willing to incorporate that into zsh. Doesn't sound terrible, but I don't think it would solve the original problem, which if I'm right is that NFS listing is big/slow after the mount is already completed. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-25 20:49 ` Bart Schaefer @ 2014-01-29 16:29 ` Greg Klanderman 2014-02-25 15:35 ` Greg Klanderman 1 sibling, 0 replies; 11+ messages in thread From: Greg Klanderman @ 2014-01-29 16:29 UTC (permalink / raw) To: zsh-users >>>>> On January 25, 2014 Bart Schaefer <schaefer@brasslantern.com> wrote: > On Jan 24, 9:46am, Greg Klanderman wrote: > } > } For slow completion due to calling out to a separate process for > } completions (_git comes to mind) if that is not currently > } interruptible it seems like there might be some hope of adding a > } timeout or ensuring C-c will interrupt. > This is the thing I had a hard time reproducing. Even if I force > the completion function to be very busy with a loop, I can interrupt > it. (Oddly the value of $? is always 0 in _main_complete after I > hit ^C, so I can't detect the signal and issue a message.) Did you try a busy completion function, or external process? Great if this case works well.. I didn't test it and haven't noticed a problem, but I don't think I commonly use any completion that calls out to an external program and is slow. I know some have complained about the git completion with big repos, but I'm fairly new to git and haven't found a case that's slow yet. If someone has an example, let me know.. > } The biggest problem I've had with completion being unresponsive was > } due to automounts at work (currently have ~53k under a handful of > } mount points). Using the fake-files zstyle mostly works until > } completion tries to stat all those automounts > I think this is analogous to the situation that started this thread. Similar issue, though my case will only come up if you go to the trouble of setting up a fake-files zstyle with the automount points. The slowness is from stat'ing 53k automounts, and it does not seem to be interruptible with C-c. Also if you hit tab after "/home/" it ends up actually mounting as many of the automounts under /home as it can before you manage to kill the shell from another window, which is non-ideal. In the original post it seemed that the slow call was probably reading a large directory on a slow NFS mount. > } to determine the file type in order to append a '/' or ' '. > ... which is difficult to do if there's an external program doing the > stat. Not impossible, but it means passing a lot of structured data > through a pipe, which might slow down more common cases. The protobuf library works well for that sort of thing though I'm not sure if it has straight C bindings. > (Maybe this is why Meino wants "find -print0 -ls". :-> ) > } So locally I run zsh with a > } small patch (to ztat()) that uses a shell variable to configure > } automount roots under which all immediate entries are assumed to be > } directories without stat'ing. Works great, though I don't think > } several years ago you were willing to incorporate that into zsh. > Doesn't sound terrible, I'll attach the patch below, if you'd be willing to incorporate something like it let me know, I'd certainly prefer to have it part of zsh. I would add documentation of course. It gets used like this: completion_nostat_dirs=(/home /net) zstyle ':completion:*' fake-files '/home:foo bar baz' '/net:aaa bbb ccc' ... > but I don't think it would solve the original > problem, which if I'm right is that NFS listing is big/slow after the > mount is already completed. Correct. Greg Index: Src/init.c =================================================================== RCS file: /cvsroot/zsh/zsh/Src/init.c,v retrieving revision 1.122 diff -u -r1.122 init.c --- Src/init.c 5 Jan 2012 20:01:25 -0000 1.122 +++ Src/init.c 9 Jan 2012 20:09:36 -0000 @@ -827,6 +827,8 @@ modulestab = newmoduletable(17, "modules"); linkedmodules = znewlinklist(); + completion_nostat_dirs = mkarray(NULL); + /* Set default prompts */ if(unset(INTERACTIVE)) { prompt = ztrdup(""); Index: Src/params.c =================================================================== RCS file: /cvsroot/zsh/zsh/Src/params.c,v retrieving revision 1.177 diff -u -r1.177 params.c --- Src/params.c 7 Jan 2012 23:21:00 -0000 1.177 +++ Src/params.c 9 Jan 2012 20:09:36 -0000 @@ -104,6 +104,10 @@ zoptind, /* $OPTIND */ shlvl; /* $SHLVL */ +/**/ +mod_export +char **completion_nostat_dirs; + /* $histchars */ /**/ @@ -356,6 +360,8 @@ IPDEF9F("*", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY), IPDEF9F("@", &pparams, NULL, PM_ARRAY|PM_SPECIAL|PM_DONTIMPORT|PM_READONLY), +IPDEF9("completion_nostat_dirs", &completion_nostat_dirs, NULL), + /* * This empty row indicates the end of parameters available in * all emulations. Index: Src/Zle/compresult.c =================================================================== RCS file: /cvsroot/zsh/zsh/Src/Zle/compresult.c,v retrieving revision 1.82 diff -u -r1.82 compresult.c --- Src/Zle/compresult.c 14 May 2011 00:07:42 -0000 1.82 +++ Src/Zle/compresult.c 9 Jan 2012 20:09:37 -0000 @@ -868,11 +868,25 @@ ztat(char *nam, struct stat *buf, int ls) { int ret; + char **pp; nam = unmeta(nam); if (!nam) return -1; + for (pp = completion_nostat_dirs; *pp; pp++) { + int len = strlen(*pp); + + if ((*pp)[len-1] == '/') + len--; + + if (strncmp(nam, *pp, len) == 0 && nam[len] == '/' && strchr(nam+len+1, '/') == NULL) { + memset(buf, 0, sizeof(*buf)); + buf->st_mode = S_IFDIR; /* pretend it's a directory */ + return 0; + } + } + if ((ret = ls ? lstat(nam, buf) : stat(nam, buf))) { char *p, *q; ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-25 20:49 ` Bart Schaefer 2014-01-29 16:29 ` Greg Klanderman @ 2014-02-25 15:35 ` Greg Klanderman 1 sibling, 0 replies; 11+ messages in thread From: Greg Klanderman @ 2014-02-25 15:35 UTC (permalink / raw) To: zsh-users Hi Bart, I just found one issue with interrupting completion - I hit tab after "ssh" and it asked: | zsh: do you wish to see all 209401 possibilities (104701 lines)? I accidentally hit tab again, and it looks like there is no way to interrupt the *output* of all those completions. I had to kill the shell, with -9, as after several minutes it was only up to the 'B's. This is with a freshly updated-from-git zsh. thanks, Greg ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Feature suggestion] (user configurable) timeout for generating completion lists 2014-01-20 7:13 [Feature suggestion] (user configurable) timeout for generating completion lists Arseny Tolmachev 2014-01-22 8:04 ` Bart Schaefer @ 2014-01-22 13:51 ` Shawn Wilson 1 sibling, 0 replies; 11+ messages in thread From: Shawn Wilson @ 2014-01-22 13:51 UTC (permalink / raw) To: Arseny Tolmachev, zsh-users Arseny Tolmachev <eiennohito@gmail.com> wrote: >Good time of day to everyone. > >Sometimes you press [Tab] by instinct when in directory that is >network-mounted and has some millions files in it. >In this situation zsh basically ignores the [Ctrl+C] and don't respond >to >anything for a bit of time. >It would be very great if there was a way to specify a timeout for >completion generation for zsh. Error message can be something like >"completion timeouted". >What do you think about this? > IDK what the problem with NFS might be however IIRC this works with rsync (haven't tried with sshfs though). So maybe try that? >There are some questions on stackexchange from users wanting to get the >feature >http://unix.stackexchange.com/questions/32363/how-to-specify-timeout-for-tab-completion-in-zsh >http://unix.stackexchange.com/questions/39404/cancel-zsh-tab-completion >with nothing as answer except the SIGINT, however in my case zsh didn't >respond for SIGINT as well. > > >Thank you! > >Arseny. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-02-25 15:36 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-01-20 7:13 [Feature suggestion] (user configurable) timeout for generating completion lists Arseny Tolmachev 2014-01-22 8:04 ` Bart Schaefer 2014-01-23 12:04 ` Nikolai Weibull 2014-01-24 1:16 ` Bart Schaefer 2014-01-24 9:00 ` Nikolai Weibull 2014-01-25 20:09 ` _git and partial completion, again Bart Schaefer 2014-01-24 14:46 ` [Feature suggestion] (user configurable) timeout for generating completion lists Greg Klanderman 2014-01-25 20:49 ` Bart Schaefer 2014-01-29 16:29 ` Greg Klanderman 2014-02-25 15:35 ` Greg Klanderman 2014-01-22 13:51 ` Shawn Wilson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).