exit value of intermediate program in pipe

exit value of intermediate program in pipe

[Steve Talley]

I have a function foo:

foo () {
	/bin/blah | grep -v "foo"
}

I would like this function to exit with the exit value from the
/bin/blah process, but it exits with the exit value from grep instead.

Is there any way to do this?

Steve

Re: exit value of intermediate program in pipe

[Sweth Chandramouli]

On Sat, May 02, 1998 at 04:24:40PM -0600, Steve Talley wrote:
> I have a function foo:
> 
> foo () {
> 	/bin/blah | grep -v "foo"
> }
> 
> I would like this function to exit with the exit value from the
> /bin/blah process, but it exits with the exit value from grep instead.

	if running /bin/blah isn't very compute-intensive, and doesn't
change its own subsequent input in any way (so that it can be run twice
consecutively with the same results), why not try

foo () {
   /bin/blah > /dev/null ; exitstatus=$?
   /bin/blah | grep -v "bar"
   return $exitstatus
}

	in theory, there should be some cleaner way to do this using
a two-way pipe, but i've never been able to get them to work correctly.
i think there are ways to export the value of a variable out of a code
block, which would also do the trick, but 

{ /bin/blah ; export exitstatus=$? } | grep -v "bar" ; return $exitstatus

, which is how i thought that was done, didn't work.

	-- sweth.

-- 
"Countin' on a remedy I've counted on before
Goin' with a cure that's never failed me
What you call the disease
I call the remedy"  -- The Mighty Mighty Bosstones

Re: exit value of intermediate program in pipe

[Timothy J Luoma]

	Author:        talley@boulder.Central.Sun.COM (Steve Talley)
	Original-Date: Sat, 2 May 1998 16:24:40 -0600
	Message-ID:    <199805022224.QAA03113@ipecac.Central.Sun.COM>

> foo () {
> /bin/blah | grep -v "foo"
> }
>
> I would like this function to exit with the exit value from the
> /bin/blah process, but it exits with the exit value from grep instead.
>
> Is there any way to do this?

Do you ming a tempfile ?

foo () {
	tmp=/tmp/$0.$USER.$$

	/bin/blah > $tmp 2>&1
	exit="$?"
	
	grep -v foo $tmp && /bin/rm -f $tmp
	
	exit $exit

}

You don't have to use /tmp as the tempdir if you have security concerns..  
$HOME would work just as well.

TjL

ps -- if you use this function in your shell, doesn't it kill the shell ?

Re: exit value of intermediate program in pipe

[Bart Schaefer]

} foo () {
} 	/bin/blah | grep -v "foo"
} }
} 
} I would like this function to exit with the exit value from the
} /bin/blah process, but it exits with the exit value from grep instead.

You can do this:

    foo() {
    	/bin/blah >>(grep -v "foo")
    }

That effectively runs grep in the background and blah in the foreground,
while still connecting them with a pipe.  Then you get the exit status of
blah, but with the side effect that the function returns as soon as blah
finishes, without waiting for the grep -- which may not be what you want.

I don't think there's any other way to do this without using a temp file.

    foo() {
	local outfile=${TMPPREFIX}blah.out
	/bin/blah > $outfile
	local exitval=$?
	grep -v "foo" $outfile
	command rm -f $outfile
	return $exitval
    }

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Re: exit value of intermediate program in pipe

[Sweth Chandramouli]

On Sat, May 02, 1998 at 07:08:31PM -0700, Bart Schaefer wrote:
> You can do this:
> 
>     foo() {
>     	/bin/blah >>(grep -v "foo")
>     }
> 
> That effectively runs grep in the background and blah in the foreground,
> while still connecting them with a pipe.  Then you get the exit status of
> blah, but with the side effect that the function returns as soon as blah
> finishes, without waiting for the grep -- which may not be what you want.

	isn't this what wait was designed for?

foo () {
   /bin/blah >>(grep -v "foo")
   wait
}

	what exactly is the syntax that >> uses to become a pipe
rather than a redirection to a file?  the way you describe it looks
a lot like a ksh two-way pipe, only cleaner; as i mentioned in my other
reply to this thread, i've never been able to get |& to work, so
any alternative would be welcome.

	-- sweth.

-- 
"Countin' on a remedy I've counted on before
Goin' with a cure that's never failed me
What you call the disease
I call the remedy"  -- The Mighty Mighty Bosstones

Re: exit value of intermediate program in pipe

[Bart Schaefer]

On May 3,  2:17am, Sweth Chandramouli wrote:
} Subject: Re: Re: exit value of intermediate program in pipe
}
} 	isn't this what wait was designed for?
} 
} foo () {
}    /bin/blah >>(grep -v "foo")
}    wait
} }

That doesn't work, because process substitutions [which is what >>(...)
is] and command substitutions [`...` and $(...)] are not placed in the
job table.  The wait command only waits for jobs that were backgrounded
with &.  (This could be considered a bug, I suppose.)

} 	what exactly is the syntax that >> uses to become a pipe

When the >> is immediately followed by a parenthesized command list, it
means pipe to that list.  There's also >(...) which uses a FIFO.

This is in the zsh tekinfo documentation, under "Process Substitution".
For input, <<(...), <(...), and =(...) use a pipe, a FIFO, and a temp
file, respectively.

} the way you describe it looks a lot like a ksh two-way pipe

Not having used ksh much and never having seen any ksh doc, I'm not sure
what that means; but the pipe isn't two-way.

} as i mentioned in my other
} reply to this thread, i've never been able to get |& to work

That just means redirect both stdout and stderr as the pipe input.  It is
a csh-ism, and not equivalent to the ksh syntax.  See the FAQ.  (I think
that means that you want the "coproc" builtin and <&p and >&p redirection
for setting up a "two-way pipe.")

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Re: exit value of intermediate program in pipe

[Sweth Chandramouli]

On Sun, May 03, 1998 at 02:30:14AM -0700, Bart Schaefer wrote:
> That just means redirect both stdout and stderr as the pipe input.  It is
> a csh-ism, and not equivalent to the ksh syntax.  See the FAQ.  (I think
> that means that you want the "coproc" builtin and <&p and >&p redirection
> for setting up a "two-way pipe.")
	man, if i could just learn to rtfm... i saw that section a while
ago in the man page, but since it was under "simple commands", i just
skimmed it and moved on.
	even reading it carefully, though, i'm still not sure what the
coproc syntax is.  to recreate the function we're discussing with a two-way
pipe (which would background one process so that the exit statuses could
be separated), in ksh i would do

{
grep -v bar |&
print -p `/bin/blah ; exitstatus=$?`
read -p output
echo $output
return $exitstatus
}

	or something like that.  for zsh, would i just do
{
grep -v bar coproc |
>&p `/bin/blah ; exitstatus=$?`
<&p output ; echo $output
return $exitstatus
}

	?

	(i'm not checking mail on a machine that i could test this
on, or i would just go ahead and do so.)

	what zsh really needs is something like the hawksbill book from
oreilly for ksh, that gives a lot of examples and compares it to other
shells; i'm sure that half of my failures at zsh scripting come from
trying to use ksh-isms that i assume are implemented.

	-- sweth.

-- 
"Countin' on a remedy I've counted on before
Goin' with a cure that's never failed me
What you call the disease
I call the remedy"  -- The Mighty Mighty Bosstones

Re: exit value of intermediate program in pipe

[Bart Schaefer]

On May 3,  6:15pm, Sweth Chandramouli wrote:
: Subject: Re: Re: exit value of intermediate program in pipe
:
: in ksh i would do
: 
: {
: grep -v bar |&
: print -p `/bin/blah ; exitstatus=$?`
: read -p output
: echo $output
: return $exitstatus
: }

You can do exactly that same thing, except insted of

	grep -v bar |&

You'd say

	coproc grep -v bar

Both print -p and read -p are the same in zsh as in ksh.

Of course, you probably want

	while read -p output
	do print -r $output
	done

as each read consumes only one line, not the entire stream.

: 	for zsh, would i just do
: 
: {
: grep -v bar coproc |
: >&p `/bin/blah ; exitstatus=$?`

That line doesn't do what you think.  What you'd want is just

	/bin/blah >&p
	exitstatus=$?

: <&p output ; echo $output

That's not right either.  You'd probably need

	cat <&p

and you'd probably have to start it before you started /bin/blah, if
blah could potentially produce more than a few kbytes of output.

: return $exitstatus
: }

The last problem is that grep won't exit until it sees EOF on its stdin,
but >&p dups the coproc input without actually closing it.  So the grep
won't get EOF when blah exits.  You have to shut it down some other way;
the only thing I've found is to start another coprocess.  I don't know
if this is a bug, or what.

So you get

    {
	coproc grep -v bar
	cat <&p &
	/bin/blah >&p
	exitstatus=$?
	coproc exit	# Shuts down grep by closing its input, as
			#   a side-effect of starting a new coproc
			#   which then immediately exits.  Ewww.
	wait		# Allows cat to finish, just in case.
	return $exitstatus
    }

I think { /bin/blah >>(grep -v bar) } is a lot nicer, don't you?

: 	what zsh really needs is something like the hawksbill book from
: oreilly for ksh, that gives a lot of examples and compares it to other
: shells

There's a lot of that in the FAQ, found in Etc/FAQ in the zsh dist.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Re: exit value of intermediate program in pipe

[Sweth Chandramouli]

On Sun, May 03, 1998 at 06:35:49PM -0700, Bart Schaefer wrote:
[snip]
> That line doesn't do what you think.  What you'd want is just
[snip]
> That's not right either.  You'd probably need
[snip]
	that's what i was talking about with the need for more examples;
the FAQ did have the fact that |& uses the csh syntax, which i had missed,
but has no reference that i could find to >&p or how to use it, and i 
think that my (wrong) interpretation was reasonable based on the description
in the manpage.

> The last problem is that grep won't exit until it sees EOF on its stdin,
> but >&p dups the coproc input without actually closing it.  So the grep
> won't get EOF when blah exits.  You have to shut it down some other way;
> the only thing I've found is to start another coprocess.  I don't know
> if this is a bug, or what.
	at first, i wasn't sure if this would be a bug or not; it would
make some conceptual sense to have a "coprocess exit" command that closed
out a coprocess, and when i tried to find some docs on coproc, all i could
find was the fmli coproc, which uses a similar cocreate/codestroy metaphor.
as far as i can tell, though, zsh coproc just swallows the eof, since doing
an 
echo "^D" >&p 
should otherwise have the same effect as "coproc exit", but doesn't.  and
_that_ is something that i _would_ consider a bug.

> So you get
> 
>     {
> 	coproc grep -v bar
> 	cat <&p &
> 	/bin/blah >&p
> 	exitstatus=$?
> 	coproc exit	# Shuts down grep by closing its input, as
> 			#   a side-effect of starting a new coproc
> 			#   which then immediately exits.  Ewww.
	does this mean that you can only have one coproc open at a
time?  i could see situations where it might be useful to have more
than one coproc open at once, rather than opening and closing the
same two executables over and over in rapid succession.

> 	wait		# Allows cat to finish, just in case.
> 	return $exitstatus
>     }
> 
> I think { /bin/blah >>(grep -v bar) } is a lot nicer, don't you?

	true, but it still ends when the first process ends, which
could cause problems if the other process were something other than
a grep, right?  making grep a bg-ed process (or coproc) was
the point of all this, after all, so that wait would recognize it.
	the next question would be, how efficient is the coproc? 
is it still worth it to go through all of these hoops rather than
just write the output to a temp file?  i would imagine so.   

	-- sweth.

-- 
"Countin' on a remedy I've counted on before
Goin' with a cure that's never failed me
What you call the disease
I call the remedy"  -- The Mighty Mighty Bosstones

Re: exit value of intermediate program in pipe

[Bernd Eggink]

Sweth Chandramouli wrote:
> > The last problem is that grep won't exit until it sees EOF on its stdin,
> > but >&p dups the coproc input without actually closing it.  So the grep
> > won't get EOF when blah exits.  You have to shut it down some other way;
> > the only thing I've found is to start another coprocess.  I don't know
> > if this is a bug, or what.
>         at first, i wasn't sure if this would be a bug or not; it would
> make some conceptual sense to have a "coprocess exit" command that closed
> out a coprocess, and when i tried to find some docs on coproc, all i could
> find was the fmli coproc, which uses a similar cocreate/codestroy metaphor.
> as far as i can tell, though, zsh coproc just swallows the eof, since doing
> an
> echo "^D" >&p
> should otherwise have the same effect as "coproc exit", but doesn't.  and
> _that_ is something that i _would_ consider a bug.

In ksh, the normal way to kill a coproc is

  exec 3<&p 3<-

because just killing the job doesn't close the file descriptor. In
zsh-3.1.3, this doesn't work (a bug, IMHO), but you can kill the job
without getting problems.


>         does this mean that you can only have one coproc open at a
> time?  i could see situations where it might be useful to have more
> than one coproc open at once, rather than opening and closing the
> same two executables over and over in rapid succession.

You can have more than one coprocs at a time. Just copy the fd's:

  coproc f
  exec 3>&p 4<&p   # or whatever numbers you like
  coproc g

Now you can communicate with f and g separately:

  print -u3 "to f"
  read -u4 x     # from f
  print -p "to g"
  read -p y      # from g

Regards,
	Bernd
--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

Re: exit value of intermediate program in pipe

[Bart Schaefer]

On May 4, 12:54am, Sweth Chandramouli wrote:
} Subject: Re: Re: exit value of intermediate program in pipe
}
} the FAQ did have the fact that |& uses the csh syntax, which i had missed,
} but has no reference that i could find to >&p or how to use it, and i 
} think that my (wrong) interpretation was reasonable based on the description
} in the manpage.

I've already bugged zsh-workers about the manpage deficiency.

} as far as i can tell, though, zsh coproc just swallows the eof, since doing
} an 
} echo "^D" >&p 
} should otherwise have the same effect as "coproc exit", but doesn't.  and
} _that_ is something that i _would_ consider a bug.

No, that's not a bug; ctrl-D only means EOF on tty devices; sending one
down any other kind of stream is just sending a byte with the value \004.
This isn't DOS where a magic character in any text stream is read as EOF.

} > I think { /bin/blah >>(grep -v bar) } is a lot nicer, don't you?
} 
} 	true, but it still ends when the first process ends

No, not quite; it gets EOF when the first process ends, but if it doesn't
happen to be a program that exits when it gets EOF on stdin, it'll just
keep running.

	echo goodbye >>(yes hello)

produces an unstoppable and rather difficult to kill stream of hellos on
your terminal.  On some OSs even exiting the tty session many not get rid
of it (linux 2.0.x being one of them).

} making grep a bg-ed process (or coproc) was
} the point of all this, after all, so that wait would recognize it.

That ought to be fixed by putting process substitutions in the job table.

} 	the next question would be, how efficient is the coproc? 
} is it still worth it to go through all of these hoops rather than
} just write the output to a temp file?  i would imagine so.   

Probably depends on just how much output we're talking about.  The coproc
is a pipe, so it doesn't have any filesize limit or quota on its input and
output.

On May 4, 11:43am, Bernd Eggink wrote:
} Subject: Re: exit value of intermediate program in pipe
}
} Sweth Chandramouli wrote:
} > > The last problem is that grep won't exit until it sees EOF on its stdin,
} > > but >&p dups the coproc input without actually closing it.
} 
} In ksh, the normal way to kill a coproc is
} 
}   exec 3<&p 3<-

Do you mean 3<&- or is <- magic in ksh?  (In zsh, <&- is magic but it only
works on stdin, it doesn't take a digit to the left.)

} because just killing the job doesn't close the file descriptor.

It may not in zsh either.  Anyway, I'm curious about that ksh-ism, because
it closes the coproc's *output*, not it's input -- so it's assuming that
the coproc will die on a "broken pipe" signal, which isn't necessarily
true.  (As my "yes" example demonstrates, closing the input won't always
work either, but presumably you don't normally coproc something that is
going to ignore its input.)

My "coproc exit" hack works because it closes the old coproc input so as
to not leak the descriptor.  But it wouldn't have been incorrect for it
to leave it open, as far as documented behavior goes (which isn't far).

} In
} zsh-3.1.3, this doesn't work (a bug, IMHO), but you can kill the job
} without getting problems.

If you kill the job, you may lose some of its output.  The only correct
way is to close the input and let it die in its own good time.

Can somebody out there who has ksh tell me whether

	cat |&
	echo >&p

causes cat to exit?  That is, does redirection to the coproc make the
coproc input no longer available to any other process?  In zsh, I can do

	coproc cat
	while true; do echo foo >&p; done &
	while true; do echo bar >&p; done &
	cat <&p

and get back

	foo
	foo
	foo
	bar
	foo
	bar
	foo
	bar
	bar
	foo

	(etc. until killed)

That is, zsh will re-open the same coproc input as often as you like,
and never closes it until you start a new coproc.  Does ksh do that?

} You can have more than one coprocs at a time. Just copy the fd's:
} 
}   coproc f
}   exec 3>&p 4<&p   # or whatever numbers you like
}   coproc g

Right; if you do that, then my "coproc exit" trick won't stop the first
coproc, because its input has already been dup'd once and the dup is
kept open.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: exit value of intermediate program in pipe

[Bernd Eggink]

Bart Schaefer wrote:

> On May 4, 11:43am, Bernd Eggink wrote:
> } Subject: Re: exit value of intermediate program in pipe
> }
> } Sweth Chandramouli wrote:
> } > > The last problem is that grep won't exit until it sees EOF on its stdin,
> } > > but >&p dups the coproc input without actually closing it.
> }
> } In ksh, the normal way to kill a coproc is
> }
> }   exec 3<&p 3<-
> 
> Do you mean 3<&- or is <- magic in ksh?  (In zsh, <&- is magic but it only
> works on stdin, it doesn't take a digit to the left.)
> 
> } because just killing the job doesn't close the file descriptor.
> 
> It may not in zsh either.  Anyway, I'm curious about that ksh-ism, because
> it closes the coproc's *output*, not it's input 

Huh? Of course it closes it's input; that's what < means! 

> -- so it's assuming that
> the coproc will die on a "broken pipe" signal, which isn't necessarily
> true.  (As my "yes" example demonstrates, closing the input won't always
> work either, but presumably you don't normally coproc something that is
> going to ignore its input.)
> 
> My "coproc exit" hack works because it closes the old coproc input so as
> to not leak the descriptor.  But it wouldn't have been incorrect for it
> to leave it open, as far as documented behavior goes (which isn't far).
> 
> } In
> } zsh-3.1.3, this doesn't work (a bug, IMHO), but you can kill the job
> } without getting problems.
> 
> If you kill the job, you may lose some of its output.  The only correct
> way is to close the input and let it die in its own good time.

That's what I said - but it doesn't work.

> Can somebody out there who has ksh tell me whether
> 
>         cat |&
>         echo >&p
> 
> causes cat to exit?  That is, does redirection to the coproc make the
> coproc input no longer available to any other process?  

Yes, it does.

[...]

> 
> That is, zsh will re-open the same coproc input as often as you like,
> and never closes it until you start a new coproc.  Does ksh do that?

No.

> } You can have more than one coprocs at a time. Just copy the fd's:
> }
> }   coproc f
> }   exec 3>&p 4<&p   # or whatever numbers you like
> }   coproc g
> 
> Right; if you do that, then my "coproc exit" trick won't stop the first
> coproc, because its input has already been dup'd once and the dup is
> kept open.

So in this case 'kill' seems to be the only way to get rid of the first
n-1 coprocs...

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

Re: exit value of intermediate program in pipe

[Bart Schaefer]

On May 4,  2:03pm, Bernd Eggink wrote:
} Subject: Re: exit value of intermediate program in pipe
}
} Bart Schaefer wrote:
} 
} > } In ksh, the normal way to kill a coproc is
} > }
} > }   exec 3<&p 3<-
} > 
} > Anyway, I'm curious about that ksh-ism, because
} > it closes the coproc's *output*, not it's input 
} 
} Huh? Of course it closes it's input; that's what < means! 

Think about it a moment.  If you do

	cat <&p

what happens?  The input _of cat_ is connected to the _output_ of the
coproc, right?  So if you do

	exec 3<&p

then what is descriptor 3?  That better be the _output_ of the coproc,
too, or ksh is doing some pretty funky special casing.

So perhaps you meant to say

	exec 3>&p 3>-

??

} > Can somebody out there who has ksh tell me whether
} > 
} >         cat |&
} >         echo >&p
} > 
} > causes cat to exit?  That is, does redirection to the coproc make the
} > coproc input no longer available to any other process?  
} 
} Yes, it does.

OK, then zsh either has a bug or an undocumented feature.  If zsh did as
ksh does, then zsh's coproc would get EOF as soon as the first thing you
redirected to it exited.

} > } You can have more than one coprocs at a time. Just copy the fd's:
} > }
} > }   coproc f
} > }   exec 3>&p 4<&p   # or whatever numbers you like
} > }   coproc g
} > 
} > Right; if you do that, then my "coproc exit" trick won't stop the first
} > coproc, because its input has already been dup'd once and the dup is
} > kept open.
} 
} So in this case 'kill' seems to be the only way to get rid of the first
} n-1 coprocs...

No, that's not true.  Once you've run "coproc g" then you can get "f" to
see EOF by simply closing fd 3.  It's only the coproc descriptor that gets
copied rather than "moved" when you redirect it.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: exit value of intermediate program in pipe

[Bernd Eggink]

On Mon, 4 May 1998, Bart Schaefer wrote:

> On May 4,  2:03pm, Bernd Eggink wrote:
> } Subject: Re: exit value of intermediate program in pipe
> }
> } Bart Schaefer wrote:
> } 
> } > } In ksh, the normal way to kill a coproc is
> } > }
> } > }   exec 3<&p 3<-
> } > 
> } > Anyway, I'm curious about that ksh-ism, because
> } > it closes the coproc's *output*, not it's input 
> } 
> } Huh? Of course it closes it's input; that's what < means! 
> 
> Think about it a moment.  If you do
> 
> 	cat <&p
> 
> what happens?  The input _of cat_ is connected to the _output_ of the
> coproc, right?  So if you do
> 
> 	exec 3<&p
> 
> then what is descriptor 3?  That better be the _output_ of the coproc,
> too, or ksh is doing some pretty funky special casing.
> 
> So perhaps you meant to say
> 
> 	exec 3>&p 3>-

Hm, I admit that I never tried this in zsh... In fact, it works
like this in zsh, but exactly the OTHER way round in ksh. In ksh
3<&p means "duplicate the coproc input to unit 3". IMHO this is
more consistent and intuitive than what zsh does.

After looking into the doc, I even suspect that this may be a bug...
Quoting from chapter 6, "redirection":

  <&p
  >&p   The input/output from/to the coprocess is moved to the 
        standard input/output.

If 'input' corresponds to <&p, this is fact what ksh does and what 
zsh DOESN'T!

Regards,
    Bernd

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

zsh vs. ksh coproc redirection semantics

[Bart Schaefer]

On May 5,  1:39pm, Bernd Eggink wrote:
} Subject: Re: exit value of intermediate program in pipe
}
} On Mon, 4 May 1998, Bart Schaefer wrote:
} 
} > 	cat <&p
} > 
} > what happens?  The input _of cat_ is connected to the _output_ of the
} > coproc, right?
} 
} Hm, I admit that I never tried this in zsh... In fact, it works
} like this in zsh, but exactly the OTHER way round in ksh.

So you're saying that in ksh, cat <&p means that the coproc input is
closed and cat takes its input from wherever the coproc was getting its
input?  That makes no sense at all.  (Which tells me that `exec 3<&p`
must be doing something special in ksh, if indeed it acts as you say.)

} In ksh 3<&p means "duplicate the coproc input to unit 3".

I think maybe we're having a cognitive dissonance here.  The coproc is a
two-way pipe:

    out +--- zsh ----+ in
        |            |
       >&p          <&p
        |            |
     in +-- coproc --+ out

According to what you're saying, ksh does it like this:

     in +--- ksh ----+ out
        |            |
       >&p          <&p
        |            |
     in +-- coproc --+ out

Is that correct?

} IMHO this is more consistent and intuitive than what zsh does.

Really?  I think having inputs connected to outputs is more intuitive
than having inputs duplicated.

} After looking into the doc, I even suspect that this may be a bug...
} Quoting from chapter 6, "redirection":
} 
}   <&p
}   >&p   The input/output from/to the coprocess is moved to the 
}         standard input/output.
} 
} If 'input' corresponds to <&p, this is fact what ksh does and what 
} zsh DOESN'T!

Re-parse that doc:

  <&p   The input from the coprocess is moved to the standard input.
  >&p   The output to the coprocess is moved to the standard output.

Note "input _from_" not "input of", and "output _to_" not "output of".

  out to +--- zsh ----+ in from
         |            |
        >&p          <&p
         |            |
   in of +-- coproc --+ out of

This may be the reverse of what ksh does, but I think zsh's doc tells
accurately (if not all that clearly) how zsh behaves.

However, I'm really surprised to hear that this is backwards in ksh.
That seems completely unintuitive to me.  If I say

	cat <&3

That means I want cat to read from fd 3, does it not?  So if I say

	cat <&p

That should mean I want cat to read from the coproc.  Which should mean
that <&p refers to "the input _from_ the coproc" which means the output
_of_ the coproc.  No?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: zsh vs. ksh coproc redirection semantics

[Bernd Eggink]

Bart Schaefer wrote:
> 
> On May 5,  1:39pm, Bernd Eggink wrote:
> } Subject: Re: exit value of intermediate program in pipe
> }
> } On Mon, 4 May 1998, Bart Schaefer wrote:
> }
> } >     cat <&p
> } >
> } > what happens?  The input _of cat_ is connected to the _output_ of the
> } > coproc, right?
> }
> } Hm, I admit that I never tried this in zsh... In fact, it works
> } like this in zsh, but exactly the OTHER way round in ksh.
> 
> So you're saying that in ksh, cat <&p means that the coproc input is
> closed and cat takes its input from wherever the coproc was getting its
> input?  That makes no sense at all.  (Which tells me that `exec 3<&p`
> must be doing something special in ksh, if indeed it acts as you say.)
> 
> } In ksh 3<&p means "duplicate the coproc input to unit 3".
> 
> I think maybe we're having a cognitive dissonance here. 
[ ... ]

Possibly... I'll try to be more specific:

In ksh, 3<&p means "duplicate the input _the coproc is reading from_ to
3". I consider that consistent with the fact that (in ksh as well as in
zsh) 3<&0 means "duplicate the input that is currently read from to 3".
Of course it doesn't make much sense to copy an input, except for the
purpose of closing it. And you must duplicate the coproc input for
closing, unless a special syntax is provided for that purpose; you can't
say "exec p<&-", because that, of course, would mean something
completely different.

> } IMHO this is more consistent and intuitive than what zsh does.
> 
> Really?  I think having inputs connected to outputs is more intuitive
> than having inputs duplicated.

In ksh, the connection is done by 'read -p' and 'print -p'. You can't
say "print foo >&p" or "read bar <&p", whereas in zsh you can say either
"print -p foo" or "print foo >&p". What I meant by "inconsistency" is
that in zsh >& and <& mean different things, depending on whether a 'p'
or a digit follows. This may be considered a matter of convention, but
in effect it prevents you from duplicating the coproc input, and such
from closing it (which, if I remember right, was your original problem).

As I just noticed, ksh has it's inconsistencies, too. While you can't
say "print foo >&p", you can say "exec 4>&p; print foo >&4", and it
works. Hm...

Reconsidering all that, I think the idea of connecting the coproc's
output and input to arbitrary file descriptors would in fact be the most
elegant solution.

Regards,
	Bernd

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

Re: zsh vs. ksh coproc redirection semantics

[Bart Schaefer]

On May 6, 12:47pm, Bernd Eggink wrote:
} Subject: Re: zsh vs. ksh coproc redirection semantics
}
} In ksh, 3<&p means "duplicate the input _the coproc is reading from_ to
} 3". I consider that consistent with the fact that (in ksh as well as in
} zsh) 3<&0 means "duplicate the input that is currently read from to 3".
} [...]
} In ksh, the connection is done by 'read -p' and 'print -p'. You can't
} say "print foo >&p" or "read bar <&p", whereas in zsh you can say either
} "print -p foo" or "print foo >&p". What I meant by "inconsistency" is
} that in zsh >& and <& mean different things, depending on whether a 'p'
} or a digit follows.

No, that's not true.  3>&1 means to copy the shell's standard output to
descriptor 3.  3>&p means copy the shell's coproc output to descriptor
3.  3<&0 means copy the shell's standard input to 3.  3<&p means copy the
shell's coproc input to 3 (not move the coproc's input to 3).

Ksh, on the other hand, appears to treat <&p as "the coproc end of the
two-way pipe" and >&p as "the ksh end of the two-way pipe".  This isn't
the same as other descriptors, but then other descriptors aren't pipes.

I think zsh's treatment ends up working a bit more consistently, because
you can always think of descriptor behavior in terms of where the current
shell will get or put data.

} This may be considered a matter of convention, but
} in effect it prevents you from duplicating the coproc input, and such
} from closing it (which, if I remember right, was your original problem).

I still think (based on what you've said in previous messages) that this
particular detail is not very consistent in ksh.  In zsh, if I do
	exec 4<&0
	exec 4<&-
then that closes only descriptor 4, not descriptor 0.  Why should
	exec 3<&p
	exec 3<&-
close both 3 and p?  In ksh, if you do
	exec 4>&p
can you later do
	exec 5>&p
and end up having both 4 and 5 connected to the coproc pipe at the same
time?  My guess is that you can't -- that once you've done 4>&p, then p
is gone and you can't do 5>&p.

Now, the question is, whether this particular inconsistency with p and
other fds is useful enough to emulate it.

(Is anybody on zsh-workers reading this?  Zefram, Zoltan, Peter?)

} Reconsidering all that, I think the idea of connecting the coproc's
} output and input to arbitrary file descriptors would in fact be the most
} elegant solution.

I still think that's completely independent.  It doesn't matter what
descriptors you can connect it to without some kind of special handling
to close the original descriptor at the same time that you close the one
you connected it to.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: zsh vs. ksh coproc redirection semantics

[Zoltan Hidvegi]

Let me quote the ksh manual:

       <&digit       The standard input is duplicated  from  file
                     descriptor  digit  (see  dup(2)).  Similarly
                     for the standard output using >&digit.

       <&digit-      The file descriptor given by digit is  moved
                     to  standard input.  Similarly for the stan­
                     dard output using >&digit-.

       <&-           The standard input is closed.  Similarly for
                     the standard output using >&-.

       <&p           The  input  from  the co-process is moved to
                     standard input.

       >&p           The output to the  co-process  is  moved  to
                     standard output.

Zsh lack the <&digit- feature.  Note the difference between &digit and
&p: &digit duplicates the file descriptor while &p moves it.  All of
these manipulate the descriptos of the shell, since once the coprocess is
started, it is a separate process, the shell does not have much control
over it.  The shell keeps two descriptors, one write-only descriptor
which is connectred to the input of the coprocess, and one read-only
descriptor which is connected to the output of the coprocess.

>&p moves the write descriptor, which writes to the co-process to
standard output.  So in ksh (at least in pdksh and in ksh93) you can do

echo foo >&p

>&p itself does not close the write descriptor, but when the command
whose output was redirected terminates, it brings down its file
descriptors, which closes the last output to the coprocess.  echo is a
bad example here, since it is a builtin, so it is handled specially.

Similarily

read <&p

moves the descriptor which reads from the coprocessor output to the
standard input, and when read terminates, its input is closes, which
closes the coprocess output.

Now that's theory, the practice is not that good.  It seems that both
ksh93 and pdksh has a buggy coprocess implementation (the most recent
pdksh may have a fix, I used one dated April 1996).  Before the coprocess
starts, two pipes are created, which means four file descriptors.  The
shell forks then:

ipipe[0] - the read descriptor for the parent handled by <&p and read -p
           closed in the child
ipipe[1] - the output of the coprocess in the child
           closed by the parent. ksh does not, zsh does close it
opipe[0] - the input of the coprocess in the child
           closed by the parent. ksh does, zsh does not close it
opipe[1] - the write descriptor for the parent handled by >&p and print -p
           closed by the child

Ksh above stands for both ksh93 and pdksh.

Zsh duplicates the descriptor on >&p and <&p unlike ksh which moves it.

echo foo >&p

On ksh this should move the write descriptor.  pdksh does the move.
ksh93 duplicates the descriptor instead of closing it, and then forgets
about it, so on ksh93 it is impossible to close the output to the
coprocess (opipe[1] above) after this.

read <&p

Both ksh93 and pdksh moves ipipe[1] (which is wrong).

Example ksh session (how it is supposed to work):

% pdksh
$ tr 'a-z' 'A-Z' |&
[1] 2330
$ echo foo >&p
$ 
[1] + Done                 tr a-z A-Z 
$ read <&p
$ echo $REPLY
FOO

This actually works on Linux with pdksh, but ls -l /proc/pid/fd when pid
is the PID of pdksh reveals the unclosed descriptors.

To fix the descriptor leak in zsh it is probably enough to add a
zclose(opipe[0]) right after the zclose(ipipe[1]) in exec.c near line
753.  I think it whould be preferable to modify zsh to match the
documented ksh behavior, i.e. moving the descriptor instead of
duplicating it.

Bart wrote:
> On May 6, 12:47pm, Bernd Eggink wrote:
> } In ksh, the connection is done by 'read -p' and 'print -p'. You can't
> } say "print foo >&p" or "read bar <&p", whereas in zsh you can say either

You can do it in ksh too.

> } "print -p foo" or "print foo >&p". What I meant by "inconsistency" is
> } that in zsh >& and <& mean different things, depending on whether a 'p'
> } or a digit follows.

In fact, zsh is consistent, ksh isn't, since zsh always duplicates
decriptors which ksh moves them then p is used.

> descriptor 3.  3>&p means copy the shell's coproc output to descriptor
> 3.  3<&0 means copy the shell's standard input to 3.  3<&p means copy the
> shell's coproc input to 3 (not move the coproc's input to 3).

That's correct provided that the term `shell's coproc output' means `the
shells output to the input of the coprocess' and the term `shell's coproc
input' menad `the shell's input from the output of the coprocess'.

> Ksh, on the other hand, appears to treat <&p as "the coproc end of the
> two-way pipe" and >&p as "the ksh end of the two-way pipe".  This isn't
> the same as other descriptors, but then other descriptors aren't pipes.

I'm not sure I understand what you mean here.  Looks like you are
confused about terms.  The pipe is a virtual special file (virtual in a
sense that it does not exists on any filesystem).  The pipe system call
creates this virtual fifo device, and opens two file descriptors, one
reading it and one writing it.  You can imagine that you are using a real
named fifo in /tmp, and instead of pipe, you do an mkfifo and two opens.
There is no real two-way pipe here, only two one-way pipe.  The coproc is
a separate process, and the shell has no control over its descriptors.
But when the shell closes its output to the coprocess, the coprocess
may notice an eof on its stdin and may chose to exit.

> close both 3 and p?  In ksh, if you do
> 	exec 4>&p
> can you later do
> 	exec 5>&p
> and end up having both 4 and 5 connected to the coproc pipe at the same
> time?  My guess is that you can't -- that once you've done 4>&p, then p
> is gone and you can't do 5>&p.

That's correct.

> Now, the question is, whether this particular inconsistency with p and
> other fds is useful enough to emulate it.

Probably is.  The other solution would be to inplement the >&digit-
syntax and have >&p duplicate, >&p- move the coprocess descriptor.  This
is consistent, but incompatible with ksh.  Since zsh is already not
compatible with ksh when creating coprocesses, this might be acceptable,
but this can greatly confuse people coming from ksh.

About the job table: ksh places the coprocess to the job table similarily
to zsh.  You can bring the coprocess to the foregroung with fg, which
does not makes much sense, but works.  You can save $! after coproc and
use kill to get rid of the coproc later.  In most cases kill %% works
too.

Zoltan

Re: zsh vs. ksh coproc redirection semantics

[Andrew Main]

Zoltan Hidvegi wrote:
>       <&digit-      The file descriptor given by digit is  moved
>                     to  standard input.  Similarly for the stan-
>                     dard output using >&digit-.
>
>       <&p           The  input  from  the co-process is moved to
>                     standard input.

Ah, now it makes sense.

>Probably is.  The other solution would be to inplement the >&digit-
>syntax and have >&p duplicate, >&p- move the coprocess descriptor.  This
>is consistent, but incompatible with ksh.  Since zsh is already not
>compatible with ksh when creating coprocesses, this might be acceptable,
>but this can greatly confuse people coming from ksh.

I think we should probably just implement the documented ksh behaviour.
It we makes fds above 9 visible, then the usual numerical syntax can be
used to copy, rather than move, the coprocess fds.

-zefram

Re: zsh vs. ksh coproc redirection semantics

[Bart Schaefer]

On May 7,  2:17am, Zoltan Hidvegi wrote:
} Subject: Re: zsh vs. ksh coproc redirection semantics
}
} Let me quote the ksh manual:
} 
}        <&digit-      The file descriptor given by digit is  moved
}                      to  standard input.  Similarly for the stan­
}                      dard output using >&digit-.
} 
}        <&-           The standard input is closed.  Similarly for
}                      the standard output using >&-.

I'm curious.  In zsh and bash (which also lacks the <&digit- form), <&-
closes the standard input of whatever command it suffixes, not of the
shell itself.  Which must mean that (semantically, if not in actual
implementation) stdin is first dup'd (as it always is when running a
new command) and then the dup is closed.

Is that what happens in ksh?

Is that what happens to fd `digit' in the <&digit- form in ksh?  I.e.,
does `digit' actually go away for good only upon `exec <&digit-` ?

If not, does
	cat <&0-
close the shell's standard input (leaving it connected to cat)?

If so, all I can say is, ick.  Yet, if you take `p' to be shorthand
for "the digit that is connected to the coproc, followed by `-'"
then that's the behavior of

}        <&p           The  input  from  the co-process is moved to
}                      standard input.
} 
}        >&p           The output to the  co-process  is  moved  to
}                      standard output.

which seems like yet another potential inconsistency that zsh might be
better off not having.

Incidentally, that documentation matches what zsh does much closer than
it matches what Bernd Eggink described ksh as doing.  According to that
excerpt, `exec 3<&p 3<&-` should have no effect on the input _of_ the
coprocess.

} Bart wrote:
} > Ksh, on the other hand, appears to treat <&p as "the coproc end of the
} > two-way pipe" and >&p as "the ksh end of the two-way pipe".  This isn't
} > the same as other descriptors, but then other descriptors aren't pipes.
} 
} I'm not sure I understand what you mean here.  Looks like you are
} confused about terms.

No, I know all that stuff about pipes.  I was trying to express concepts
that it seemed ksh wanted those redirections to embody, rather than the
way it has to be implemented underneath.  But now that I see the doc, I
think Bernd's description was of a ksh that got its own implementation
wrong, which is why my attempt to explain it doesn't work either.

} About the job table: ksh places the coprocess to the job table similarily
} to zsh.  You can bring the coprocess to the foregroung with fg, which
} does not makes much sense, but works.

Probably fg followed by ctrl-C is a common kill-the-coproc technique.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: zsh vs. ksh coproc redirection semantics

[Bart Schaefer]

On May 7,  9:34am, Andrew Main wrote:
} Subject: Re: zsh vs. ksh coproc redirection semantics
}
} I think we should probably just implement the documented ksh behaviour.

I'd take a poll first (and see if you can get PF to answer).  One aspect
of the current zsh behavior is that you can leave the same coproc running
all day and feed different stuff through it from time to time, which IIRC
was actually discussed in a long-ago introduction-to-zsh document that
Paul once distributed.

} It we makes fds above 9 visible, then the usual numerical syntax can be
} used to copy, rather than move, the coprocess fds.

With respect to that, I suggest simply making <&999 >&999 legal syntax
when a particular option is set.  It's unlikely to clash with existing
scripts and is simpler than trying to add some other syntactic marker.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: zsh vs. ksh coproc redirection semantics

[Andrew Main]

Bart Schaefer wrote:
>I'd take a poll first (and see if you can get PF to answer).  One aspect
>of the current zsh behavior is that you can leave the same coproc running
>all day and feed different stuff through it from time to time, which IIRC
>was actually discussed in a long-ago introduction-to-zsh document that
>Paul once distributed.

That is an issue.  Maybe there's reason enough for an option to flip
the behaviour of >&p.  (We could add >&p- and >&p+ which aren't affected
by options.)

>With respect to that, I suggest simply making <&999 >&999 legal syntax
>when a particular option is set.

That's already legal.  It's 999>&1 that's the problem.  I think an option
is a good solution.

-zefram

Re: zsh vs. ksh coproc redirection semantics

[Zoltan Hidvegi]

Bart wrote:
> On May 7,  9:34am, Andrew Main wrote:
> } Subject: Re: zsh vs. ksh coproc redirection semantics
> }
> } I think we should probably just implement the documented ksh behaviour.
> 
> I'd take a poll first (and see if you can get PF to answer).  One aspect
> of the current zsh behavior is that you can leave the same coproc running
> all day and feed different stuff through it from time to time, which IIRC
> was actually discussed in a long-ago introduction-to-zsh document that
> Paul once distributed.

You can do it with the ksh syntax by moving it first with exec ..&p
and then you can use the moved descriptor as many times as you wish.

p>&- can calso be done by 3>&p 3>&-.

Zoli

Re: zsh vs. ksh coproc redirection semantics

[Zoltan Hidvegi]

Bart wrote:
> I'm curious.  In zsh and bash (which also lacks the <&digit- form), <&-
> closes the standard input of whatever command it suffixes, not of the
> shell itself.  Which must mean that (semantically, if not in actual
> implementation) stdin is first dup'd (as it always is when running a
> new command) and then the dup is closed.

When running an external command the stdin is closed after fork thus
not affecting the shell.  Non special builtins should behave like
external commands.

Now the >&p case seems to be special in pdksh, as this does affect the
shell.

It looks like these coproc redirection really works well only with
exec, the rest is implementation dependent.

Zoli