process substitution and Ctrl-C

process substitution and Ctrl-C

[Vincent Lefevre]

Hi,

In the following example:

  { repeat 10 { date >&2; /bin/sleep 1 } } 2>>(cat -n; loop)

where "loop" is a program that consumes CPU time, is it normal that
when one interrupts the command with Ctrl-C, the substituted process
isn't killed? (I can see "loop" taking CPU time.)

The zsh man page says that the command is run asynchronously, but
this notion is never clearly defined.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / Arénaire project (LIP, ENS-Lyon)

Re: process substitution and Ctrl-C

[Peter Stephenson]

On Thu, 19 Aug 2010 14:41:42 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> In the following example:
> 
>   { repeat 10 { date >&2; /bin/sleep 1 } } 2>>(cat -n; loop)
> 
> where "loop" is a program that consumes CPU time, is it normal that
> when one interrupts the command with Ctrl-C, the substituted process
> isn't killed? (I can see "loop" taking CPU time.)

Without looking at the code, I wouldn't be at all surprised: unless we did
something special, SIGINT would go only to foreground processes, which
wouldn't include the process substitution.  Logically, you might have
thought that passing the SIGINT as received by the shell on to associated
processes (which are recorded in a part of the job record) should be
possible, but this sort of thing is fairly well down my personal list of
priorities.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: process substitution and Ctrl-C

[Peter Stephenson]

On Thu, 19 Aug 2010 14:07:30 +0100
Peter Stephenson <Peter.Stephenson@csr.com> wrote:
> On Thu, 19 Aug 2010 14:41:42 +0200
> Vincent Lefevre <vincent@vinc17.net> wrote:
> > In the following example:
> > 
> >   { repeat 10 { date >&2; /bin/sleep 1 } } 2>>(cat -n; loop)
> > 
> > where "loop" is a program that consumes CPU time, is it normal that
> > when one interrupts the command with Ctrl-C, the substituted process
> > isn't killed? (I can see "loop" taking CPU time.)
> 
> Without looking at the code, I wouldn't be at all surprised: unless
> we did something special, SIGINT would go only to foreground
> processes, which wouldn't include the process substitution.
> Logically, you might have thought that passing the SIGINT as received
> by the shell on to associated processes (which are recorded in a part
> of the job record) should be possible, but this sort of thing is
> fairly well down my personal list of priorities.

I managed to get to the point where I had zero time to spare.  That's up
significantly from the usual amount, so I looked at this.

In fact we're jumping through a fairly large and stable hoop not all that
far above the ground to behave as if "we" got SIGINT.  "We" here is just
the main shell.  You might think, logically, that any process attached to
any job marked as running in the current shell should also get the signal
passed, if we're going to maintain the fiction.  This is a slightly smaller
hoop held a little further over the ground, but I don't see any fundamental
reason why we shouldn't jump through it.  Anyway, the following seems to do
the trick here.

Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.78
diff -p -u -r1.78 jobs.c
--- Src/jobs.c	18 Aug 2010 21:21:17 -0000	1.78
+++ Src/jobs.c	19 Aug 2010 17:08:33 -0000
@@ -496,6 +496,27 @@ update_job(Job jn)
 		breaks = loops;
 		errflag = 1;
 	    }
+	    if (errflag) {
+		/*
+		 * As we're pretending we got the signal, we need to
+		 * pretend anything attached to a CURSH process
+		 * got it, too.
+		 */
+		int i, j;
+		for (i = 1; i <= maxjob; i++) {
+		    if ((jobtab[i].stat & (STAT_CURSH|STAT_DONE)) ==
+			STAT_CURSH) {
+			for (j = 0; j < 2; j++) {
+			    pn = j ? jobtab[i].auxprocs : jobtab[i].procs;
+			    for (; pn; pn = pn->next) {
+				if (pn->status == SP_RUNNING) {
+				    kill(pn->pid, sig);
+				}
+			    }
+			}
+		    }
+		}
+	    }
 	}
     }
 }


-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: process substitution and Ctrl-C

[Peter Stephenson]

I'm now wondering if we should do the same in the case where the shell
actually does get the SIGINT and abort processing as a result?  Theories
welcome.

Index: Src/jobs.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/jobs.c,v
retrieving revision 1.78
diff -p -u -r1.78 jobs.c
--- Src/jobs.c	18 Aug 2010 21:21:17 -0000	1.78
+++ Src/jobs.c	19 Aug 2010 20:16:53 -0000
@@ -320,6 +320,36 @@ update_process(Process pn, int status)
 }
 #endif
 
+/*
+ * Called when the current shell is behaving as if it received
+ * a interactively generated signal (sig).
+ * 
+ * As we got the signal or are pretending we did, we need to pretend
+ * anything attached to a CURSH process got it, too.
+ */
+/**/
+void
+check_cursh_sig(int sig)
+{
+    int i, j;
+
+    if (!errflag)
+	return;
+    for (i = 1; i <= maxjob; i++) {
+	if ((jobtab[i].stat & (STAT_CURSH|STAT_DONE)) ==
+	    STAT_CURSH) {
+	    for (j = 0; j < 2; j++) {
+		Process pn = j ? jobtab[i].auxprocs : jobtab[i].procs;
+		for (; pn; pn = pn->next) {
+		    if (pn->status == SP_RUNNING) {
+			kill(pn->pid, sig);
+		    }
+		}
+	    }
+	}
+    }
+}
+
 /* Update status of job, possibly printing it */
 
 /**/
@@ -496,6 +526,7 @@ update_job(Job jn)
 		breaks = loops;
 		errflag = 1;
 	    }
+	    check_cursh_sig(sig);
 	}
     }
 }
Index: Src/signals.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/signals.c,v
retrieving revision 1.58
diff -p -u -r1.58 signals.c
--- Src/signals.c	12 May 2010 10:07:01 -0000	1.58
+++ Src/signals.c	19 Aug 2010 20:16:53 -0000
@@ -580,6 +580,7 @@ zhandler(int sig)
                 breaks = loops;
                 errflag = 1;
 		inerrflush();
+		check_cursh_sig(SIGINT);
             }
         }
         break;

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: process substitution and Ctrl-C

[Bart Schaefer]

On Aug 19,  2:41pm, Vincent Lefevre wrote:
} 
} In the following example:
} 
}   { repeat 10 { date >&2; /bin/sleep 1 } } 2>>(cat -n; loop)
} 
} where "loop" is a program that consumes CPU time, is it normal that
} when one interrupts the command with Ctrl-C, the substituted process
} isn't killed? (I can see "loop" taking CPU time.)

The assumption is that process substitution consumes stdin and exits
after its stdin is closed.  Otherwise, why would you need to redirect
into it?

However, there may be a good reason for changing this.  If I try

    echo >>(cat; sleep 100)

then the shell is effectively frozen until "sleep 100" finishes, because
"echo" is already gone and the process substitution is not interruptible,
so nothing is paying attention to ^C or ^Z etc.  This appears to happen
with any builtin as the input to the process substitution, including a
brace construct like the one you used above; e.g. I tried:

    { tty } >>(cat; sleep 100)

and it hangs there for 100 seconds.  However, it doesn't happen with an
external command; e.g.

    tty >>(cat; sleep 100)

exits immediately, even before the sleep finishes, though the sleep is
left running.

} The zsh man page says that the command is run asynchronously, but
} this notion is never clearly defined.

For better or worse, the zsh man page has never been very good about
fully defining concepts that are "well known" from other shells.  It
was written from the standpoint of "you've already used (c)sh for a
while and know what all of this means, I'm just going to tell you what
is different about doing it in zsh."

So if you refer to (say) the bash manual:

   A shell allows execution of GNU commands, both synchronously and
   asynchronously. The shell waits for synchronous commands to complete
   before accepting more input; asynchronous commands continue to
   execute in parallel with the shell while it reads and executes
   additional commands.

   Command substitution, commands grouped with parentheses, and
   asynchronous commands are invoked in a subshell environment that is
   a duplicate of the shell environment, except that traps caught by
   the shell are reset to the values that the shell inherited from its
   parent at invocation.

   When job control is not in effect, asynchronous commands ignore
   `SIGINT' and `SIGQUIT' in addition to these inherited handlers.
   Commands run as a result of command substitution ignore the
   keyboard-generated job control signals `SIGTTIN', `SIGTTOU', and
   `SIGTSTP'.

What gets fuzzy, even in the bash manual, is the distinction between
an "asynchronous command" and a "background job":

   If a command is terminated by the control operator `&', the shell
   executes the command asynchronously in a subshell. This is known as
   executing the command in the BACKGROUND. The shell does not wait for
   the command to finish, and the return status is 0 (true). When job
   control is not active, the standard input for asynchronous commands,
   in the absence of any explicit redirections, is redirected from
   `/dev/null'.

   Background processes are those whose process group ID differs from
   the terminal's; such processes are immune to keyboard-generated
   signals. Only foreground processes are allowed to read from or write
   to the terminal. Background processes which attempt to read from
   (write to) the terminal are sent a `SIGTTIN' (`SIGTTOU') signal by
   the terminal driver, which, unless caught, suspends the process.

Zsh process substitution (and, I think, just about everything that zsh
runs "asynchronously") behave in almost every way like background jobs,
and therefore don't receive keyboard-generated signals.

Re: process substitution and Ctrl-C

[Peter Stephenson]

On Thu, 19 Aug 2010 08:32:37 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> If I try
> 
>     echo >>(cat; sleep 100)
> 
> then the shell is effectively frozen until "sleep 100" finishes, because
> "echo" is already gone and the process substitution is not interruptible,
> so nothing is paying attention to ^C or ^Z etc.

(This should have gone to zsh-workers ages ago but still hasn't.)

This is a separate problem.  It's not so much that the echo has gone as
that there are no processes in the tty process group to get the signal,
because the substitution is treated as asynchronous for the purposes of
job handling.  (In fact, I've decided I don't understand what
"asynchronous" means in this case, either.  We seem to have spent
chunks of the last twenty years making process substitution less and
less like an asynchronous process; first we started waiting for it to
finish, now we're passing signals to it.)

Any reactions to the following patch?  This makes the substitution
process always get the same signal as the rest of the command line.  If
that's an external process that would be interrupted, so you wouldn't
see the freeze, but even there without the patch the "sleep" carries
merrily on --- similar to Vincent's case.

The alternative is to make it synchronous only if the current job is
running within the shell.  I no longer really have any idea what the
intention is.

I'm assuming the other use of >(...), when not part of a redirection,
want identical treatment to whatever happens here.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.181
diff -p -u -r1.181 exec.c
--- Src/exec.c	15 Jul 2010 18:44:13 -0000	1.181
+++ Src/exec.c	19 Aug 2010 20:26:24 -0000
@@ -3828,7 +3828,7 @@ getproc(char *cmd, char **eptr)
 	zerr("can't open %s: %e", pnam, errno);
 	_exit(1);
     }
-    entersubsh(ESUB_ASYNC|ESUB_PGRP);
+    entersubsh(ESUB_PGRP);
     redup(fd, out);
 #else /* PATH_DEV_FD */
     int pipes[2];
@@ -3855,7 +3855,7 @@ getproc(char *cmd, char **eptr)
 	}
 	return pnam;
     }
-    entersubsh(ESUB_ASYNC|ESUB_PGRP);
+    entersubsh(ESUB_PGRP);
     redup(pipes[out], out);
     closem(FDT_UNUSED);   /* this closes pipes[!out] as well */
 #endif /* PATH_DEV_FD */
@@ -3905,7 +3905,7 @@ getpipe(char *cmd, int nullexec)
 	    addproc(pid, NULL, 1, &bgtime);
 	return pipes[!out];
     }
-    entersubsh(ESUB_ASYNC|ESUB_PGRP);
+    entersubsh(ESUB_PGRP);
     redup(pipes[out], out);
     closem(FDT_UNUSED);	/* this closes pipes[!out] as well */
     cmdpush(CS_CMDSUBST);

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: process substitution and Ctrl-C

[Vincent Lefevre]

On 2010-08-19 08:32:37 -0700, Bart Schaefer wrote:
> On Aug 19,  2:41pm, Vincent Lefevre wrote:
> } 
> } In the following example:
> } 
> }   { repeat 10 { date >&2; /bin/sleep 1 } } 2>>(cat -n; loop)
> } 
> } where "loop" is a program that consumes CPU time, is it normal that
> } when one interrupts the command with Ctrl-C, the substituted process
> } isn't killed? (I can see "loop" taking CPU time.)
> 
> The assumption is that process substitution consumes stdin and exits
> after its stdin is closed.  Otherwise, why would you need to redirect
> into it?

However there's a race condition, IMHO. In fact, even without a signal
to the main process. For instance, if you consider:

  ls >>(cat -n)

then the zsh prompt for the next command is displayed before "cat -n"
finishes. Does this mean that process substitution should not be used
for filtering, except when an end marker is used (as in the example
at the end of my message)? Now,

  { ls } >>(cat -n)

seems to work (as you said), and this can be seen with:

  { ls } >>(sleep 2; cat -n)

but no longer if the main process is interrupted with Ctrl-C.
For instance:

  { ls } >>(while read line; do sleep 1; echo $line; done)

outputs one line each second, and the main process is blocked as
wanted; but if one does a Ctrl-C, the main process is terminated
and one gets the prompt while the substituted process still outputs
the remaining lines each second. I don't think this is the behavior
that one expects.

In fact, I noticed the problem with the Ctrl-C due to a bug in one
of my scripts:

filter()
{
  unset brpipe
  while true
  do
    unset line
    while read -r -t 0.1 -k -u 0 ch
    do
      line="$line$ch"
      [[ $ch = $'\012' ]] && break
    done
    case $line in
      svnwrapper:term$'\012')
        break ;;
      *Broken\ pipe$'\012') brpipe=1 ;;
      ?*) printf "%s" "$line" >&2 ;;
    esac
  done
  [[ -z $brpipe ]] || kill -PIPE $$
}

{ svn "$@"; st=$?; echo "svnwrapper:term" >&2 } 2>>(filter)
exit $st

The problem here is that I didn't do the difference between a timeout
(due to -t 0.1) and an end of file: in the case of a Ctrl-C while svn
was running, the end marker svnwrapper:term would no longer be output
and filter() would start to take CPU time in an infinite loop. I've
fixed the script by using -t 0.1 as of the second iteration only:

[...]
    unset line timeout
    while read -r $timeout -k -u 0 ch
    do
      line="$line$ch"
      [[ $ch = $'\012' ]] && break
      timeout=(-t 0.1)
    done
[...]

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / Arénaire project (LIP, ENS-Lyon)

Re: process substitution and Ctrl-C

[Bart Schaefer]

Answering a whole bunch of messages at once ...

On Aug 19,  6:15pm, Peter Stephenson wrote:
}
} In fact we're jumping through a fairly large and stable hoop not all that
} the main shell.  You might think, logically, that any process attached to
} any job marked as running in the current shell should also get the signal
} passed, if we're going to maintain the fiction.

On Aug 19,  9:18pm, Peter Stephenson wrote:
}
} I'm now wondering if we should do the same in the case where the shell
} actually does get the SIGINT and abort processing as a result?

What I'm now wondering is whether there's a reason it was the other way,
more significant than just Paul never having gotten around to figuring
out how to fit it into the job code.  I begin to believe not, see below.

On Aug 19,  9:37pm, Peter Stephenson wrote:
}
} >     echo >>(cat; sleep 100)
} > 
} > then the shell is effectively frozen until "sleep 100" finishes
} 
} This is a separate problem.  It's not so much that the echo has gone as
} that there are no processes in the tty process group to get the signal,
} because the substitution is treated as asynchronous for the purposes of
} job handling.

Well, OK, but the main shell could still get the signal ...  That's
what would happen if "echo" were replaced by a loop construct, as is
demonstrated by Vincent's original example.

} (In fact, I've decided I don't understand what "asynchronous" means
} in this case, either. We seem to have spent chunks of the last twenty
} years making process substitution less and less like an asynchronous
} process; first we started waiting for it to finish, now we're passing
} signals to it.)

I just had a peek back at some really old code.  (Next paragraph may be
indecipherable to anyone not on zsh-workers, and to some of them.)

Mid-90s, job handling in entersubsh() was a lot simpler, and there was
no distinction corresponding to the current meanings of the ESUB_PGRP
and ESUB_ASYNC bitflags passed to entersubsh().  There was no provision
for tracking the process substitution in the job table, and the only
way to run process substitution in parallel with the foreground job
that was writing to it, was to background it.  This is essential for
doing things like "foo >>(bar) 2>>(baz)"; if the >>(bar) had not been
backgrounded, 2>>(baz) could not have run until it finished.

What has happened over time is that the shell has become capable of
running those jobs in parallel without losing track of them, which I
think means it's now OK to wait for them and make them interruptible,
given that [as best I can tell] >>({...}&) provides the old behavior.

However ...

} Any reactions to the following patch?  This makes the substitution
} process always get the same signal as the rest of the command line.
[...]
} -    entersubsh(ESUB_ASYNC|ESUB_PGRP);
} +    entersubsh(ESUB_PGRP);

I'm a little worried about this, because leaving off ESUB_ASYNC may
cause entersubsh() to attempt to attach a new process group leader to
the terminal, and I can't tell what consequences that may have for
the foreground job.  Maybe there no circumstance in which this would
happen, e.g. thisjob is always -1 when in this context.

} The alternative is to make it synchronous only if the current job is
} running within the shell.  I no longer really have any idea what the
} intention is.

Again we're confusing (or overloading?) the meaning of "synchronous".
The process substitution can't really be synchronous in the sense of
sequential with respect to the rest of the job.  The intention is to
run in parallel so that multios can send data to many processes at
once.  I think the rest of the behavior was ill-defined side-effect.

} I'm assuming the other use of >(...), when not part of a redirection,
} want identical treatment to whatever happens here.

That makes sense to me at the moment ...

On Aug 19, 11:17pm, Vincent Lefevre wrote:
}
} > The assumption is that process substitution consumes stdin and exits
} > after its stdin is closed.  Otherwise, why would you need to redirect
} > into it?
} 
} However there's a race condition, IMHO. In fact, even without a signal
} to the main process. For instance, if you consider:
} 
}   ls >>(cat -n)
} 
} then the zsh prompt for the next command is displayed before "cat -n"
} finishes. Does this mean that process substitution should not be used
} for filtering, except when an end marker is used (as in the example
} at the end of my message)?

I'm not sure what question you're asking, but process substitutions
share the stdout file descriptor of the command of which they are a
part, so if you use them in one of the commands in a pipeline they
become part of the stream feeding the right side of the pipe; and the
fact that the command writing to *them* has exited does not affect
what happens downstream.

On the other hand foo >>(bar) | baz is a multio and baz will get as
stdin the stdout of both foo and bar, in no particular order, so you
have to be careful to use { foo >>(bar) } | baz or unsetopt multios.

}   { ls } >>(cat -n)
} 
} seems to work (as you said)

If I understand what you mean to exemplify, that isn't exactly the
part of the behavior to which I was referring.

} For instance:
} 
}   { ls } >>(while read line; do sleep 1; echo $line; done)
} 
} outputs one line each second, and the main process is blocked as
} wanted; but if one does a Ctrl-C, the main process is terminated
} and one gets the prompt while the substituted process still outputs
} the remaining lines each second. I don't think this is the behavior
} that one expects.

As noted above, that's the only behavior that could be accomodated in
the original code base ... so it _is_ what someone who has been using
zsh since 1993-ish would expect. :-)

Re: process substitution and Ctrl-C

[Vincent Lefevre]

On 2010-08-20 01:19:05 -0700, Bart Schaefer wrote:
> The process substitution can't really be synchronous in the sense of
> sequential with respect to the rest of the job.  The intention is to
> run in parallel so that multios can send data to many processes at
> once.  I think the rest of the behavior was ill-defined side-effect.

I don't think this is specific to process substitution. This should
also be the case for pipelines, such as:

  prg1 | prg2 | prg3 | prg4

All of the should be run in parallel to avoid a possible deadlock
due to buffering (and this should also be faster on today's desktop
machines, which have several cores). But as far as I can see, a
Ctrl-C interrupts all of them.

> On Aug 19, 11:17pm, Vincent Lefevre wrote:
> } However there's a race condition, IMHO. In fact, even without a signal
> } to the main process. For instance, if you consider:
> } 
> }   ls >>(cat -n)
> } 
> } then the zsh prompt for the next command is displayed before "cat -n"
> } finishes. Does this mean that process substitution should not be used
> } for filtering, except when an end marker is used (as in the example
> } at the end of my message)?
> 
> I'm not sure what question you're asking, [...]

What I mean here is: the problem is that the main shell doesn't wait
for "cat -n" to finish. This means that on a typical machine, the zsh
prompt will here be displayed before the "cat -n" output. This should
be more clear with:

  ls >>(sleep 2; cat -n)

I would like the same behavior as:

  { ls } >>(sleep 2; cat -n)

> } For instance:
> } 
> }   { ls } >>(while read line; do sleep 1; echo $line; done)
> } 
> } outputs one line each second, and the main process is blocked as
> } wanted; but if one does a Ctrl-C, the main process is terminated
> } and one gets the prompt while the substituted process still outputs
> } the remaining lines each second. I don't think this is the behavior
> } that one expects.
> 
> As noted above, that's the only behavior that could be accomodated in
> the original code base ... so it _is_ what someone who has been using
> zsh since 1993-ish would expect. :-)

I think the average user would expect process substitution to behave
a bit like pipes w.r.t. signals, e.g. like:

  ls | while read line; do sleep 1; echo $line; done

The fact that the right part is in the foreground doesn't matter.
One can see that everything is killed with a more complex one:

  { echo foo; sleep 5; echo bar } | \
  { while read line; do sleep 1; echo $line; done; echo blah >&2 } | \
  { sleep 10; read a; echo $a; sleep 2; cat }

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / Arénaire project (LIP, ENS-Lyon)

Re: process substitution and Ctrl-C

[Bart Schaefer]

On Aug 20,  1:52pm, Vincent Lefevre wrote:
} Subject: Re: process substitution and Ctrl-C
}
} On 2010-08-20 01:19:05 -0700, Bart Schaefer wrote:
} > The process substitution can't really be synchronous in the sense of
} > sequential with respect to the rest of the job.
} 
} I don't think this is specific to process substitution. This should
} also be the case for pipelines [...]  But as far as I can see, a
} Ctrl-C interrupts all of them.

Yes, obviously.  The difference is that a pipeline is organized as a
process group whose leader's PID is recorded in the job table, so even
historically, it was all managed together as one job.  Conversely, a
process substitution (again historically) went into the background;
the closest pipeline analogy would be

    prg1 | ( exec prg2 & )

Subsequent changes mean this behavior is no longer necessary, it was
merely preserved so that the new code behaved like the old code.  I'm
not arguing that it should have been that way or should stay that way,
I'm just providing context.

} > On Aug 19, 11:17pm, Vincent Lefevre wrote:
} > } then the zsh prompt for the next command is displayed before "cat -n"
} > } finishes. Does this mean that process substitution should not be used
} > } for filtering, except when an end marker is used (as in the example
} > } at the end of my message)?
} > 
} > I'm not sure what question you're asking, [...]
} 
} What I mean here is: the problem is that the main shell doesn't wait
} for "cat -n" to finish.

Yes, I get that; what I don't follow is what that has to do with using
process substitution for filtering and whether or not there's an "end
marker".  Filtering implies that there's a downstream consumer of the
output of the filter, and using an end marker or not won't have any
effect on what's seen downstream.  If you botch handling EOF from the
upstream in the middle of a pipeline, you'll get the same problem you
had with your process substitution (except that you'll be able to
interrupt the pipeline, even in older zsh).

The reason the manual explicitly calls out that process substitutions
are run asynchronously is to warn you about the ( prg2 & ) behavior.
We're discussing whether to change that and I think PWS's last patch
does so, but in the meantime if you don't want the backgrounding effect,
you're correct, you can't use a process substitution as if it were the
tail of a pipeline.

Re: process substitution and Ctrl-C

[Vincent Lefevre]

On 2010-08-20 08:24:18 -0700, Bart Schaefer wrote:
> Yes, I get that; what I don't follow is what that has to do with using
> process substitution for filtering and whether or not there's an "end
> marker".

True, in fact I was just seeing the side effect of using { ... } for
the main process.

The end marker can still be useful so that the filter knows when
to stop reading. This is useful if the main process can execute
processes and some of them can remain running in background while
still connected to the pipe (this is the case wih svn + ssh).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / Arénaire project (LIP, ENS-Lyon)