zsh 5.6 regression: a pipe sometimes yields a TTOU signal

zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

Hi,

After the upgrade to zsh 5.6 under Debian/unstable, I sometimes get:

ypig:~> dpkg -l|gr mercurial
ii  mercurial                                                   4.7-1                                    amd64        easy-to-use, scalable distributed version control system
ii  mercurial-common                                            4.7-1                                    all          easy-to-use, scalable distributed version control system (common files)
zsh: exit 150
zsh: suspended (tty output)  pager-wrapper grep --color=always --line-buffered -E mercurial
ypig:~[TTOU]>

where

gr: aliased to pager-wrapper grep --color=always --line-buffered -E

and pager-wrapper is:

pager-wrapper () {
        local -a opt
        while [[ $1 == -* ]]
        do
                opt+=$1 
                shift
        done
        if [[ -t 1 ]]
        then
                $@ $opt | less -+c -FRX
        else
                $@
        fi
}

There was no such issue with previous zsh versions.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Peter Stephenson]

On Wed, 5 Sep 2018 11:03:39 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> After the upgrade to zsh 5.6 under Debian/unstable, I sometimes get:
> 
> ypig:~> dpkg -l|gr mercurial  
> ii  mercurial                                                   4.7-1                                    amd64        easy-to-use, scalable distributed version control system
> ii  mercurial-common                                            4.7-1                                    all          easy-to-use, scalable distributed version control system (common files)
> zsh: exit 150
> zsh: suspended (tty output)  pager-wrapper grep --color=always --line-buffered -E mercurial
> ypig:~[TTOU]>  
> 
> where
> 
> gr: aliased to pager-wrapper grep --color=always --line-buffered -E
> 
> and pager-wrapper is:
> 
> pager-wrapper () {
>         local -a opt
>         while [[ $1 == -* ]]
>         do
>                 opt+=$1 
>                 shift
>         done
>         if [[ -t 1 ]]
>         then
>                 $@ $opt | less -+c -FRX
>         else
>                 $@
>         fi
> }

I managed to get this to happen once in a set of tests.

With

(echo mercurial) | gr mercurial

I always get

zsh: suspended (tty output)  pager-wrapper grep --color=always --line-buffered -E mercurial

This isn't going to be fixed quickly (or, quite likely, at all without
creating different knock-on effects, as changes here always do). Anyone
wanting to volunteer to look after job control would be very welcome ---
currently like so much of the shell it's down to when Bart and I
get a chance to look, as it has been for several decades now, which
isn't a sustainable position.

It's presumably down to the first job exiting and the other part
of the pipeline not getting control quickly enough.

pws

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

On 2018-09-05 11:03:54 +0100, Peter Stephenson wrote:
> With
> 
> (echo mercurial) | gr mercurial
> 
> I always get
> 
> zsh: suspended (tty output)  pager-wrapper grep --color=always --line-buffered -E mercurial

Or just with:

  (echo foo) | { grep foo | less -+c -FX }

> This isn't going to be fixed quickly (or, quite likely, at all without
> creating different knock-on effects, as changes here always do).

Is there any reason why this changed since zsh 5.5.1? If a new feature
is the cause, couldn't it be reverted until this is fixed?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

On 2018-09-05 12:55:10 +0200, Vincent Lefevre wrote:
> Or just with:
> 
>   (echo foo) | { grep foo | less -+c -FX }

Even simpler:

  echo foo | { : | tput init }

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

On 2018-09-05 14:22:46 +0200, Vincent Lefevre wrote:
> Even simpler:
> 
>   echo foo | { : | tput init }

And this one is even worse, as "fg" does clean the status:

  echo foo | { sleep 1 ; : | tput init }

For instance:

cventin% echo foo | { sleep 1 ; : | tput init }
zsh: done                    echo foo | 
zsh: suspended (tty output)  { sleep 1; : | tput init; }
cventin% jobs
[1]  + done                    echo foo | 
       suspended (tty output)  { sleep 1; : | tput init; }
cventin% fg
[1]  + done       echo foo | 
       continued  { sleep 1; : | tput init; }
zsh: done       echo foo | { sleep 1; : | tput init; }
cventin% jobs
[1]  + done       echo foo | { sleep 1; : | tput init; }
cventin% fg
[1]  + done       echo foo | { sleep 1; : | tput init; }
zsh: done       echo foo | { sleep 1; : | tput init; }
cventin% jobs
[1]  + done       echo foo | { sleep 1; : | tput init; }
cventin% 

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

As a workaround, putting the middle command in { } can help a bit.
For instance, instead of

  echo foo | { cat | less -+c -FX }

writing

  echo foo | { { cat } | less -+c -FX }

seems to avoid the issue most of the time, and

  echo foo | { { sleep 0.001; cat } | less -+c -FX }

is even better.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Peter Stephenson]

On Wed, 5 Sep 2018 15:37:04 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:

> As a workaround, putting the middle command in { } can help a bit.
> For instance, instead of
> 
>   echo foo | { cat | less -+c -FX }
> 
> writing
> 
>   echo foo | { { cat } | less -+c -FX }
> 
> seems to avoid the issue most of the time, and
> 
>   echo foo | { { sleep 0.001; cat } | less -+c -FX }

If you don't actually care about where the command is run, I'd suggest
using ( ) instead of { } which should avoid the issue entirely ---
no "Rimmerworld" nested pipeline management as each pipe is in a
different shell.

pws

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

On 2018-09-05 15:40:47 +0100, Peter Stephenson wrote:
> If you don't actually care about where the command is run, I'd suggest
> using ( ) instead of { } which should avoid the issue entirely ---
> no "Rimmerworld" nested pipeline management as each pipe is in a
> different shell.

No, this doesn't solve the problem for all commands, e.g. with:

sww () {
        svnwrapper diff "$@" | wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m' -d
}

where svnwrapper and wdiff are external command.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

On 2018-09-06 13:40:11 +0200, Vincent Lefevre wrote:
> On 2018-09-05 15:40:47 +0100, Peter Stephenson wrote:
> > If you don't actually care about where the command is run, I'd suggest
> > using ( ) instead of { } which should avoid the issue entirely ---
> > no "Rimmerworld" nested pipeline management as each pipe is in a
> > different shell.
> 
> No, this doesn't solve the problem for all commands, e.g. with:
> 
> sww () {
>         svnwrapper diff "$@" | wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m' -d
> }
> 
> where svnwrapper and wdiff are external command.

Sorry, this may be a different problem, this time specific to "less",
but also introduced in zsh 5.6 (if not a bit before).

  echo | { (sleep 1; sww) | tput init; }
  echo | { (sleep 1; sww) | more; }
  echo | { (sleep 1; sww) | less; }

are all fine with zsh 5.3.1, but the 3rd one (with "less") yields

zsh: suspended (tty output)  { ( sleep 1; sww; ) | less; }

after quitting "less".

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Vincent Lefevre]

Perhaps I misunderstood. The solution is not something like

  echo | { ( sww ) | less }

but

  echo | ( sww | less )

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Peter Stephenson]

On Thu, 6 Sep 2018 14:36:57 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> Perhaps I misunderstood. The solution is not something like
> 
>   echo | { ( sww ) | less }
> 
> but
> 
>   echo | ( sww | less )

Yes, that's what I meant.

In the first example there are still Rimmerworld(*) pipelines:
"echo | ... " and "(sww) | less" are two different pipelines
being managed in the same shell.  This is the root of the
problem, with various ramifications.

In the second case, "echo | ..." is in one shell, and then
"sww | less" is managed entirely by a forked copy of the shell,
so the basic issue doesn't occur.

In principle (and depending a bit on options, open FDs, etc.)
the second one should be just as efficient --- the fork occurs
earlier but the "less" is run by an exec of the forked shell.

pws

(*) https://www.youtube.com/watch?v=py3u3P9OpBE
OK, the actual relevance *is* a bit obscure.

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Peter Stephenson]

On Wed, 5 Sep 2018 12:55:10 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> On 2018-09-05 11:03:54 +0100, Peter Stephenson wrote:
> > With
> > 
> > (echo mercurial) | gr mercurial
> > 
> > I always get
> > 
> > zsh: suspended (tty output)  pager-wrapper grep --color=always --line-buffered -E mercurial  
> 
> Or just with:
> 
>   (echo foo) | { grep foo | less -+c -FX }
> 
> > This isn't going to be fixed quickly (or, quite likely, at all without
> > creating different knock-on effects, as changes here always do).  
> 
> Is there any reason why this changed since zsh 5.5.1? If a new feature
> is the cause, couldn't it be reverted until this is fixed?

Bizarrely, yes there is actually a reason the code changed(!)  There are
no new features in this area and the last obvious related change I can
see was in May, although I don't know if that's causing this or not ---
it's not obvious why it would cause this, but working with multiple
processes is subtle.

(Hope the following doesn't appear grumpy, I'm just trying to be
realistic about this.)

Any changes were there to fix long-standing problems in the combination
of pipes and builtin structures.  This has always been buggy and the
code is a total nightmare.  Arguably we are doing too much work --- this
dates from the late 90s, when Sven put in some extremely clever but a
changes to allow job control with shell constructs to work a
bit more naturally: they're extremely clever but a bit brittle and a lot
hard to understand.  I don't think there have been any new *features*
since then, just gradual improvements in the things we know how to
test.

The code has evolved in such away that I don't think going back is
sensible.  It's never been perfect and --- given that people aren't
doing enough testing between releases, as the emergence of the present
issue shows --- arbitrarily backing off changes until it sort-of vaguely
works is just asking for trouble.  We *know* this will expose bugs.

The choices I can see are:

1. Wait for me to do something about it on my own time and no, you don't
get to hassle me about it.  I'm not promising a date, I'm just saying
it's quite likely if no one else looks at it I prbably will at some
point but the rest of my life isn't going to stop.  As it's reproducible
it's probably hours work rather than days if I get the chance to look.

2. Someone else gets a go.  Whoopee!  I'm very glad to give as many
pointers as you need.

3. Just lump it for the time being and test the pre-release better next
time.  With races like this we need everybody trying out all their
awkward cases.  (Ideally we need lots of people using the top of tree,
in fact.)

Cheers
pws

Re: zsh 5.6 regression: a pipe sometimes yields a TTOU signal

[Peter Stephenson]

I've now been testing with

echo foo | { : | more }

though the exact form isn't going to be important.

My theory is the following is happening:

- Process "echo foo" gets control of the terminal.

- The shell forks ":".  This is group leader for the new pipeline but
doesn't get control of the terminal. Note the { ... } is directly under the control of the main shell, as as is the "echo foo | ..." pipeline.  We
are in the world of nested pipelines.  (Remember "Rimmerworld"? Yes,
exactly.)

- "echo" exits

- The shell looks for a process to attach the tty to and finds ":"
which is currently group leader.

- ":" exits leaving the process group without a leader.

- "more" is forked but doesn't get the TTY because nobody knows no one
ain't got it.  This is different from the case where there's no nested
pipeline.

As evidence, replacing ":" by "sleep 0.01" eliminates the problem (or
"zselect -t 1" which is equivalent --- but it's not just builtins, you
can put a "cat" there, but please don't swing it as you haven't got room,
and the bug still happens).  So I think there's some problem with the way
transferring group leadership (the job gleader struct field) interacts with
attaching the TTY (the call to attachtty()) when we've got a nested
pipeline.

Please tell me someone finds this an interesting problem...

pws