* zsh 5.6 regression: a pipe sometimes yields a TTOU signal @ 2018-09-05 9:03 ` Vincent Lefevre 2018-09-05 10:03 ` Peter Stephenson 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-05 9:03 UTC (permalink / raw) To: zsh-workers Hi, After the upgrade to zsh 5.6 under Debian/unstable, I sometimes get: ypig:~> dpkg -l|gr mercurial ii mercurial 4.7-1 amd64 easy-to-use, scalable distributed version control system ii mercurial-common 4.7-1 all easy-to-use, scalable distributed version control system (common files) zsh: exit 150 zsh: suspended (tty output) pager-wrapper grep --color=always --line-buffered -E mercurial ypig:~[TTOU]> where gr: aliased to pager-wrapper grep --color=always --line-buffered -E and pager-wrapper is: pager-wrapper () { local -a opt while [[ $1 == -* ]] do opt+=$1 shift done if [[ -t 1 ]] then $@ $opt | less -+c -FRX else $@ fi } There was no such issue with previous zsh versions. -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 9:03 ` zsh 5.6 regression: a pipe sometimes yields a TTOU signal Vincent Lefevre @ 2018-09-05 10:03 ` Peter Stephenson 2018-09-05 10:55 ` Vincent Lefevre 0 siblings, 1 reply; 13+ messages in thread From: Peter Stephenson @ 2018-09-05 10:03 UTC (permalink / raw) To: zsh-workers On Wed, 5 Sep 2018 11:03:39 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > After the upgrade to zsh 5.6 under Debian/unstable, I sometimes get: > > ypig:~> dpkg -l|gr mercurial > ii mercurial 4.7-1 amd64 easy-to-use, scalable distributed version control system > ii mercurial-common 4.7-1 all easy-to-use, scalable distributed version control system (common files) > zsh: exit 150 > zsh: suspended (tty output) pager-wrapper grep --color=always --line-buffered -E mercurial > ypig:~[TTOU]> > > where > > gr: aliased to pager-wrapper grep --color=always --line-buffered -E > > and pager-wrapper is: > > pager-wrapper () { > local -a opt > while [[ $1 == -* ]] > do > opt+=$1 > shift > done > if [[ -t 1 ]] > then > $@ $opt | less -+c -FRX > else > $@ > fi > } I managed to get this to happen once in a set of tests. With (echo mercurial) | gr mercurial I always get zsh: suspended (tty output) pager-wrapper grep --color=always --line-buffered -E mercurial This isn't going to be fixed quickly (or, quite likely, at all without creating different knock-on effects, as changes here always do). Anyone wanting to volunteer to look after job control would be very welcome --- currently like so much of the shell it's down to when Bart and I get a chance to look, as it has been for several decades now, which isn't a sustainable position. It's presumably down to the first job exiting and the other part of the pipeline not getting control quickly enough. pws ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 10:03 ` Peter Stephenson @ 2018-09-05 10:55 ` Vincent Lefevre 2018-09-05 12:22 ` Vincent Lefevre ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: Vincent Lefevre @ 2018-09-05 10:55 UTC (permalink / raw) To: zsh-workers On 2018-09-05 11:03:54 +0100, Peter Stephenson wrote: > With > > (echo mercurial) | gr mercurial > > I always get > > zsh: suspended (tty output) pager-wrapper grep --color=always --line-buffered -E mercurial Or just with: (echo foo) | { grep foo | less -+c -FX } > This isn't going to be fixed quickly (or, quite likely, at all without > creating different knock-on effects, as changes here always do). Is there any reason why this changed since zsh 5.5.1? If a new feature is the cause, couldn't it be reverted until this is fixed? -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 10:55 ` Vincent Lefevre @ 2018-09-05 12:22 ` Vincent Lefevre 2018-09-05 13:17 ` Vincent Lefevre 2018-09-05 12:53 ` Peter Stephenson 2018-09-05 13:21 ` Peter Stephenson 2 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-05 12:22 UTC (permalink / raw) To: zsh-workers On 2018-09-05 12:55:10 +0200, Vincent Lefevre wrote: > Or just with: > > (echo foo) | { grep foo | less -+c -FX } Even simpler: echo foo | { : | tput init } -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 12:22 ` Vincent Lefevre @ 2018-09-05 13:17 ` Vincent Lefevre 2018-09-05 13:37 ` Vincent Lefevre 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-05 13:17 UTC (permalink / raw) To: zsh-workers On 2018-09-05 14:22:46 +0200, Vincent Lefevre wrote: > Even simpler: > > echo foo | { : | tput init } And this one is even worse, as "fg" does clean the status: echo foo | { sleep 1 ; : | tput init } For instance: cventin% echo foo | { sleep 1 ; : | tput init } zsh: done echo foo | zsh: suspended (tty output) { sleep 1; : | tput init; } cventin% jobs [1] + done echo foo | suspended (tty output) { sleep 1; : | tput init; } cventin% fg [1] + done echo foo | continued { sleep 1; : | tput init; } zsh: done echo foo | { sleep 1; : | tput init; } cventin% jobs [1] + done echo foo | { sleep 1; : | tput init; } cventin% fg [1] + done echo foo | { sleep 1; : | tput init; } zsh: done echo foo | { sleep 1; : | tput init; } cventin% jobs [1] + done echo foo | { sleep 1; : | tput init; } cventin% -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 13:17 ` Vincent Lefevre @ 2018-09-05 13:37 ` Vincent Lefevre 2018-09-05 14:40 ` Peter Stephenson 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-05 13:37 UTC (permalink / raw) To: zsh-workers As a workaround, putting the middle command in { } can help a bit. For instance, instead of echo foo | { cat | less -+c -FX } writing echo foo | { { cat } | less -+c -FX } seems to avoid the issue most of the time, and echo foo | { { sleep 0.001; cat } | less -+c -FX } is even better. -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 13:37 ` Vincent Lefevre @ 2018-09-05 14:40 ` Peter Stephenson 2018-09-06 11:40 ` Vincent Lefevre 0 siblings, 1 reply; 13+ messages in thread From: Peter Stephenson @ 2018-09-05 14:40 UTC (permalink / raw) To: zsh-workers On Wed, 5 Sep 2018 15:37:04 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > As a workaround, putting the middle command in { } can help a bit. > For instance, instead of > > echo foo | { cat | less -+c -FX } > > writing > > echo foo | { { cat } | less -+c -FX } > > seems to avoid the issue most of the time, and > > echo foo | { { sleep 0.001; cat } | less -+c -FX } If you don't actually care about where the command is run, I'd suggest using ( ) instead of { } which should avoid the issue entirely --- no "Rimmerworld" nested pipeline management as each pipe is in a different shell. pws ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 14:40 ` Peter Stephenson @ 2018-09-06 11:40 ` Vincent Lefevre 2018-09-06 12:26 ` Vincent Lefevre 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-06 11:40 UTC (permalink / raw) To: zsh-workers On 2018-09-05 15:40:47 +0100, Peter Stephenson wrote: > If you don't actually care about where the command is run, I'd suggest > using ( ) instead of { } which should avoid the issue entirely --- > no "Rimmerworld" nested pipeline management as each pipe is in a > different shell. No, this doesn't solve the problem for all commands, e.g. with: sww () { svnwrapper diff "$@" | wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m' -d } where svnwrapper and wdiff are external command. -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-06 11:40 ` Vincent Lefevre @ 2018-09-06 12:26 ` Vincent Lefevre 2018-09-06 12:36 ` Vincent Lefevre 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-06 12:26 UTC (permalink / raw) To: zsh-workers On 2018-09-06 13:40:11 +0200, Vincent Lefevre wrote: > On 2018-09-05 15:40:47 +0100, Peter Stephenson wrote: > > If you don't actually care about where the command is run, I'd suggest > > using ( ) instead of { } which should avoid the issue entirely --- > > no "Rimmerworld" nested pipeline management as each pipe is in a > > different shell. > > No, this doesn't solve the problem for all commands, e.g. with: > > sww () { > svnwrapper diff "$@" | wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m' -d > } > > where svnwrapper and wdiff are external command. Sorry, this may be a different problem, this time specific to "less", but also introduced in zsh 5.6 (if not a bit before). echo | { (sleep 1; sww) | tput init; } echo | { (sleep 1; sww) | more; } echo | { (sleep 1; sww) | less; } are all fine with zsh 5.3.1, but the 3rd one (with "less") yields zsh: suspended (tty output) { ( sleep 1; sww; ) | less; } after quitting "less". -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-06 12:26 ` Vincent Lefevre @ 2018-09-06 12:36 ` Vincent Lefevre 2018-09-06 12:55 ` Peter Stephenson 0 siblings, 1 reply; 13+ messages in thread From: Vincent Lefevre @ 2018-09-06 12:36 UTC (permalink / raw) To: zsh-workers Perhaps I misunderstood. The solution is not something like echo | { ( sww ) | less } but echo | ( sww | less ) -- Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-06 12:36 ` Vincent Lefevre @ 2018-09-06 12:55 ` Peter Stephenson 0 siblings, 0 replies; 13+ messages in thread From: Peter Stephenson @ 2018-09-06 12:55 UTC (permalink / raw) To: zsh-workers On Thu, 6 Sep 2018 14:36:57 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > Perhaps I misunderstood. The solution is not something like > > echo | { ( sww ) | less } > > but > > echo | ( sww | less ) Yes, that's what I meant. In the first example there are still Rimmerworld(*) pipelines: "echo | ... " and "(sww) | less" are two different pipelines being managed in the same shell. This is the root of the problem, with various ramifications. In the second case, "echo | ..." is in one shell, and then "sww | less" is managed entirely by a forked copy of the shell, so the basic issue doesn't occur. In principle (and depending a bit on options, open FDs, etc.) the second one should be just as efficient --- the fork occurs earlier but the "less" is run by an exec of the forked shell. pws (*) https://www.youtube.com/watch?v=py3u3P9OpBE OK, the actual relevance *is* a bit obscure. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 10:55 ` Vincent Lefevre 2018-09-05 12:22 ` Vincent Lefevre @ 2018-09-05 12:53 ` Peter Stephenson 2018-09-05 13:21 ` Peter Stephenson 2 siblings, 0 replies; 13+ messages in thread From: Peter Stephenson @ 2018-09-05 12:53 UTC (permalink / raw) To: zsh-workers On Wed, 5 Sep 2018 12:55:10 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > On 2018-09-05 11:03:54 +0100, Peter Stephenson wrote: > > With > > > > (echo mercurial) | gr mercurial > > > > I always get > > > > zsh: suspended (tty output) pager-wrapper grep --color=always --line-buffered -E mercurial > > Or just with: > > (echo foo) | { grep foo | less -+c -FX } > > > This isn't going to be fixed quickly (or, quite likely, at all without > > creating different knock-on effects, as changes here always do). > > Is there any reason why this changed since zsh 5.5.1? If a new feature > is the cause, couldn't it be reverted until this is fixed? Bizarrely, yes there is actually a reason the code changed(!) There are no new features in this area and the last obvious related change I can see was in May, although I don't know if that's causing this or not --- it's not obvious why it would cause this, but working with multiple processes is subtle. (Hope the following doesn't appear grumpy, I'm just trying to be realistic about this.) Any changes were there to fix long-standing problems in the combination of pipes and builtin structures. This has always been buggy and the code is a total nightmare. Arguably we are doing too much work --- this dates from the late 90s, when Sven put in some extremely clever but a changes to allow job control with shell constructs to work a bit more naturally: they're extremely clever but a bit brittle and a lot hard to understand. I don't think there have been any new *features* since then, just gradual improvements in the things we know how to test. The code has evolved in such away that I don't think going back is sensible. It's never been perfect and --- given that people aren't doing enough testing between releases, as the emergence of the present issue shows --- arbitrarily backing off changes until it sort-of vaguely works is just asking for trouble. We *know* this will expose bugs. The choices I can see are: 1. Wait for me to do something about it on my own time and no, you don't get to hassle me about it. I'm not promising a date, I'm just saying it's quite likely if no one else looks at it I prbably will at some point but the rest of my life isn't going to stop. As it's reproducible it's probably hours work rather than days if I get the chance to look. 2. Someone else gets a go. Whoopee! I'm very glad to give as many pointers as you need. 3. Just lump it for the time being and test the pre-release better next time. With races like this we need everybody trying out all their awkward cases. (Ideally we need lots of people using the top of tree, in fact.) Cheers pws ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal 2018-09-05 10:55 ` Vincent Lefevre 2018-09-05 12:22 ` Vincent Lefevre 2018-09-05 12:53 ` Peter Stephenson @ 2018-09-05 13:21 ` Peter Stephenson 2 siblings, 0 replies; 13+ messages in thread From: Peter Stephenson @ 2018-09-05 13:21 UTC (permalink / raw) To: zsh-workers I've now been testing with echo foo | { : | more } though the exact form isn't going to be important. My theory is the following is happening: - Process "echo foo" gets control of the terminal. - The shell forks ":". This is group leader for the new pipeline but doesn't get control of the terminal. Note the { ... } is directly under the control of the main shell, as as is the "echo foo | ..." pipeline. We are in the world of nested pipelines. (Remember "Rimmerworld"? Yes, exactly.) - "echo" exits - The shell looks for a process to attach the tty to and finds ":" which is currently group leader. - ":" exits leaving the process group without a leader. - "more" is forked but doesn't get the TTY because nobody knows no one ain't got it. This is different from the case where there's no nested pipeline. As evidence, replacing ":" by "sleep 0.01" eliminates the problem (or "zselect -t 1" which is equivalent --- but it's not just builtins, you can put a "cat" there, but please don't swing it as you haven't got room, and the bug still happens). So I think there's some problem with the way transferring group leadership (the job gleader struct field) interacts with attaching the TTY (the call to attachtty()) when we've got a nested pipeline. Please tell me someone finds this an interesting problem... pws ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2018-09-06 12:56 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CGME20180905091407epcas1p1ff78da39bfd4f3e4201f467683288a7b@epcas1p1.samsung.com> 2018-09-05 9:03 ` zsh 5.6 regression: a pipe sometimes yields a TTOU signal Vincent Lefevre 2018-09-05 10:03 ` Peter Stephenson 2018-09-05 10:55 ` Vincent Lefevre 2018-09-05 12:22 ` Vincent Lefevre 2018-09-05 13:17 ` Vincent Lefevre 2018-09-05 13:37 ` Vincent Lefevre 2018-09-05 14:40 ` Peter Stephenson 2018-09-06 11:40 ` Vincent Lefevre 2018-09-06 12:26 ` Vincent Lefevre 2018-09-06 12:36 ` Vincent Lefevre 2018-09-06 12:55 ` Peter Stephenson 2018-09-05 12:53 ` Peter Stephenson 2018-09-05 13:21 ` Peter Stephenson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).