Weird issue with pipeviewer and multiple pipes

Weird issue with pipeviewer and multiple pipes

[Jason L Tibbitts III]

Someone brought this up on IRC; I thought I'd bring it to a larger
audience.  

Pipeviewer (pv) is a little program that sits in a pipeline, passes data
through, and outputs some progress information.
  http://www.ivarch.com/programs/pv.shtml
I didn't know about it until today, but it's kind of neat.  It's in most
distros/repositories/package managers/whatever.

Run this:
  echo 1 | pv -l -s $(echo 1)

Should give '1' and a progress bar.  Now tack on "|cat".  For me that
hangs.  ctrl-Z does nothing.  ctrl-C seems to do what I'd expect ctrl-Z
to do:

[2]  + 11459 done                    echo 1 |
       11461 suspended (tty output)  pv -l -s $(echo 1) |
       11462 interrupt               cat

If I type fg, the shell is pretty much hung.  You have to kill -9 the pv
process either way to get rid of the job.  (Lesser kills don't seem to
work.)

Could certainly be a bug in pv, but it's definitely weird and I'd just
dismiss it except that other shells seem to have no problems.

 - J<

Re: Weird issue with pipeviewer and multiple pipes

[Bart Schaefer]

On Tue, May 3, 2016 at 6:15 PM, Jason L Tibbitts III <tibbs@math.uh.edu> wrote:
>
> [2]  + 11459 done                    echo 1 |
>        11461 suspended (tty output)  pv -l -s $(echo 1) |
>        11462 interrupt               cat

Something weird is definitely going on with pv, because it's receiving
a TTOU signal even when the terminal stty setting is supposed to
suppress that.

It doesn't happen with "-s 1" in place of the $(echo 1) substitution,
but it also doesn't happen if pv is replaced by e.g. tee $(echo
/dev/fd/2), so it's some combination of using command substitution and
something pv is doing.  By examination of strace output, it has to be
ioctl(2, TCSETS, ...).

So what is pv doing with that ioctl()?  (There doesn't seem to be a
way to get strace to both print the entire structure and decode it
from hexadecimal.)  I suspect that it's putting itself in its own
process group, which means that kills of the parent job don't reach it
any longer, which is why zsh ends up waiting forever when it tries to
resume the whole pipeline.  But I don't know why the $(echo) makes any
difference to it receiving the TTOU in the first place.

All of these also avoid the problem:

    echo 1 | { trap '' TTOU ; pv -p -s $(echo 1) } | cat
    echo 1 | pv -p -s $(echo 1) 2>/dev/tty | cat
    echo 1 | { x=$(echo 1); pv -p -s $x } | cat

> If I type fg, the shell is pretty much hung.  You have to kill -9 the pv
> process either way to get rid of the job.  (Lesser kills don't seem to
> work.)

Lesser kills do work, you just have to follow them with -CONT to
resume the stopped process; e.g. with GNU killall,
  killall pv
  killall -CONT pv

Re: Weird issue with pipeviewer and multiple pipes

[Peter Stephenson]

On Tue, 03 May 2016 20:56:14 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Tue, May 3, 2016 at 6:15 PM, Jason L Tibbitts III <tibbs@math.uh.edu> wrote:
> >
> > [2]  + 11459 done                    echo 1 |
> >        11461 suspended (tty output)  pv -l -s $(echo 1) |
> >        11462 interrupt               cat
> 
> Something weird is definitely going on with pv, because it's receiving
> a TTOU signal even when the terminal stty setting is supposed to
> suppress that.

It's working with the latest zsh [which I just *happened* to have to
hand] and the pv supplied with Fedora 12, too.

pv 1.1.4 - Copyright(C) 2008 Andrew Wood <andrew.wood@ivarch.com>

Maybe zsh isn't the only utility to succumb to too-clever-by-half
syndrome.

pws

Re: Weird issue with pipeviewer and multiple pipes

[Bart Schaefer]

On May 5,  4:14pm, Peter Stephenson wrote:
}
} It's working with the latest zsh [which I just *happened* to have to
} hand] and the pv supplied with Fedora 12, too.

That's good, but there's still the oddity that it only happens when a
$(cmdsubst) is in play, and that zsh shouldn't hang uninterruptibly
forever just because one job in the pipeline has been TTOU'd.

Re: Weird issue with pipeviewer and multiple pipes

[Peter Stephenson]

On Tue, 3 May 2016 20:56:14 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
>On Tue, May 3, 2016 at 6:15 PM, Jaron L Tibbitts III <tibbs@math.uh.edu> wrote:
> >
> > [2]  + 11459 done                    echo 1 |
> >        11461 suspended (tty output)  pv -l -s $(echo 1) |
> >        11462 interrupt               cat
> 
> I don't know why the $(echo) makes any
> difference to it receiving the TTOU in the first place.

Actually, it looks like This Can't Possibly Happen, since $(echo 1) is
over and done with by the time the pv is started; we've got the output,
closed the fd, and waited for the process to finish (by PID).  So there
is some weird lingering side effect.
 
> All of these also avoid the problem:
> 
>     echo 1 | { trap '' TTOU ; pv -p -s $(echo 1) } | cat
>     echo 1 | pv -p -s $(echo 1) 2>/dev/tty | cat
>     echo 1 | { x=$(echo 1); pv -p -s $x } | cat

Also, a substitution of the form $(<file) doesn't do it, either.  That's
probably not surprising as there's no extra process there, but it does
leave not that much code in getoutput() to be causing it.  I tried
commenting out entersubsh() to see if the terminal handling in there
maede a difference, but it didn't.  In fact, even pruning down
the behaviour in the child to simply printing something didn't
have any effect.

pws

Re: Weird issue with pipeviewer and multiple pipes

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On May 6, 2016 2:41 PM, "Peter Stephenson" <p.w.stephenson@ntlworld.com>
wrote:
>
> On Tue, 3 May 2016 20:56:14 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> >On Tue, May 3, 2016 at 6:15 PM, Jaron L Tibbitts III <tibbs@math.uh.edu>
wrote:
> > >
> > > [2]  + 11459 done                    echo 1 |
> > >        11461 suspended (tty output)  pv -l -s $(echo 1) |
> > >        11462 interrupt               cat
> >
> > I don't know why the $(echo) makes any
> > difference to it receiving the TTOU in the first place.
>
> Actually, it looks like This Can't Possibly Happen, since $(echo 1) is
> over and done with by the time the pv is started; we've got the output,
> closed the fd, and waited for the process to finish (by PID).  So there
> is some weird lingering side effect.

Something to do with list_pipe_* handling?  Is the parent confused about
the state of the child that will eventually exec pv ?