zsh-users
 help / color / Atom feed
* subprocess
@ 2020-06-05  2:03 Pier Paolo Grassi
  2020-06-05 10:25 ` subprocess Mikael Magnusson
  0 siblings, 1 reply; 8+ messages in thread
From: Pier Paolo Grassi @ 2020-06-05  2:03 UTC (permalink / raw)
  To: Zsh-Users List


[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

hello, can someone explain why (with zsh 5.1.1):

efe() cat $1
efe <(seq 1 10) | wc -l
cat: /proc/self/fd/13: No such file or directory
0

gives error, instead these all works fine:
efe =(seq 1 10)| wc -l
cat <(seq 1 10)| wc -l

thanks
-- 
Pier Paolo Grassi

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-05  2:03 subprocess Pier Paolo Grassi
@ 2020-06-05 10:25 ` Mikael Magnusson
  2020-06-06  1:00   ` subprocess Daniel Shahaf
  0 siblings, 1 reply; 8+ messages in thread
From: Mikael Magnusson @ 2020-06-05 10:25 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Zsh-Users List

On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> hello, can someone explain why (with zsh 5.1.1):

Same in 5.8.0.2

> efe() cat $1
> efe <(seq 1 10) | wc -l
> cat: /proc/self/fd/13: No such file or directory
> 0
>
> gives error, instead these all works fine:
> efe =(seq 1 10)| wc -l
> cat <(seq 1 10)| wc -l

This also works,
% efe <(seq 1 10) > >(wc -l)
as does this,
% wc -l < <(efe <(seq 1 10))

-- 
Mikael Magnusson

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-05 10:25 ` subprocess Mikael Magnusson
@ 2020-06-06  1:00   ` Daniel Shahaf
  2020-06-06 18:19     ` subprocess Bart Schaefer
  0 siblings, 1 reply; 8+ messages in thread
From: Daniel Shahaf @ 2020-06-06  1:00 UTC (permalink / raw)
  To: Mikael Magnusson; +Cc: Pier Paolo Grassi, Zsh-Users List

Mikael Magnusson wrote on Fri, 05 Jun 2020 12:25 +0200:
> On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> > hello, can someone explain why (with zsh 5.1.1):  
> 
> Same in 5.8.0.2
> 
> > efe() cat $1
> > efe <(seq 1 10) | wc -l
> > cat: /proc/self/fd/13: No such file or directory
> > 0
> >
> > gives error, instead these all works fine:
> > efe =(seq 1 10)| wc -l
> > cat <(seq 1 10)| wc -l  
> 
> This also works,
> % efe <(seq 1 10) > >(wc -l)
> as does this,
> % wc -l < <(efe <(seq 1 10))

And this,

% () { cat $1 } <(seq 1 10) | wc -l

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-06  1:00   ` subprocess Daniel Shahaf
@ 2020-06-06 18:19     ` Bart Schaefer
  2020-06-06 19:53       ` subprocess Peter Stephenson
  2020-06-08 14:22       ` subprocess Pier Paolo Grassi
  0 siblings, 2 replies; 8+ messages in thread
From: Bart Schaefer @ 2020-06-06 18:19 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: Mikael Magnusson, Pier Paolo Grassi, Zsh-Users List

On Fri, Jun 5, 2020 at 6:01 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> > On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> > > efe() cat $1
> > > efe <(seq 1 10) | wc -l
> > > cat: /proc/self/fd/13: No such file or directory
> > > 0
> > >
> > > gives error, instead these all works fine:
> > > efe =(seq 1 10)| wc -l
> > > cat <(seq 1 10)| wc -l
>
> And this,
>
> % () { cat $1 } <(seq 1 10) | wc -l

You can see what's happening if you do it this way:

% efe() ls -l ${${1:+$1:h}:-/proc/self/fd}
% efe <(seq 1 10) | cat
total 0
lrwx------ 1 schaefer schaefer 64 Jun  6 10:59 0 -> /dev/pts/0
l-wx------ 1 schaefer schaefer 64 Jun  6 10:59 1 -> pipe:[1945962]
lrwx------ 1 schaefer schaefer 64 Jun  6 10:59 2 -> /dev/pts/0
lr-x------ 1 schaefer schaefer 64 Jun  6 10:59 3 -> /proc/5533/fd
% () { ls -l $1:h } <(seq 1 10) | cat
total 0
lrwx------ 1 schaefer schaefer 64 Jun  6 11:00 0 -> /dev/pts/0
l-wx------ 1 schaefer schaefer 64 Jun  6 11:00 1 -> pipe:[1945988]
lr-x------ 1 schaefer schaefer 64 Jun  6 11:00 11 -> pipe:[1944505]
lrwx------ 1 schaefer schaefer 64 Jun  6 11:00 2 -> /dev/pts/0
lr-x------ 1 schaefer schaefer 64 Jun  6 11:00 3 -> /proc/5537/fd

A named function on the left side of a pipeline becomes its own "job",
where all descriptors above 10 are closed.  An anonymous function
stays in the parent shell (which is still a forked subshell, because
of the pipeline) so those descriptors are not closed.

% efe =(seq 1 10) | cat
total 4
-rw------- 1 schaefer schaefer 21 Jun  6 11:09 zshkDpnRj

In that case a real file has been created and is referenced by name,
so the descriptors closing does not matter.

% efe <<(seq 1 10) | cat
total 0
lr-x------ 1 schaefer schaefer 64 Jun  6 11:13 0 -> pipe:[1946804]
l-wx------ 1 schaefer schaefer 64 Jun  6 11:13 1 -> pipe:[1946579]
lrwx------ 1 schaefer schaefer 64 Jun  6 11:13 2 -> /dev/pts/0
lr-x------ 1 schaefer schaefer 64 Jun  6 11:13 3 -> /proc/5599/fd

For <<(...) no file is created at all, instead the standard input of
the function is changed.

> Mikael Magnusson wrote on Fri, 05 Jun 2020 12:25 +0200:
> >
> > This also works,
> > % efe <(seq 1 10) > >(wc -l)
> > as does this,
> > % wc -l < <(efe <(seq 1 10))

In both of these cases, there is no pipeline, so efe stays in the
parent shell and all the descriptors remain open.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-06 18:19     ` subprocess Bart Schaefer
@ 2020-06-06 19:53       ` Peter Stephenson
  2020-06-08 19:55         ` subprocess Peter Stephenson
  2020-06-08 14:22       ` subprocess Pier Paolo Grassi
  1 sibling, 1 reply; 8+ messages in thread
From: Peter Stephenson @ 2020-06-06 19:53 UTC (permalink / raw)
  To: zsh-users

On Sat, 2020-06-06 at 11:19 -0700, Bart Schaefer wrote:
> On Fri, Jun 5, 2020 at 6:01 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> > > On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> > > > efe() cat $1
> > > > efe <(seq 1 10) | wc -l
> > > > cat: /proc/self/fd/13: No such file or directory
>
> A named function on the left side of a pipeline becomes its own "job",
> where all descriptors above 10 are closed.

Itʼs this call here.

How ahout simply reordering so we tidy up after the function is called?
That fixes the problem, hard to see how that can cause a leak...?

pws

diff --git a/Src/exec.c b/Src/exec.c
index 2b8e2167f..53d3aa048 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -3969,8 +3969,8 @@ execcmd_exec(Estate state, Execcmd_params eparams,
 
 	    if (is_shfunc) {
 		/* It's a shell function */
-		pipecleanfilelist(filelist, 0);
 		execshfunc((Shfunc) hn, args);
+		pipecleanfilelist(filelist, 0);
 	    } else {
 		/* It's a builtin */
 		LinkList assigns = (LinkList)0;


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-06 18:19     ` subprocess Bart Schaefer
  2020-06-06 19:53       ` subprocess Peter Stephenson
@ 2020-06-08 14:22       ` Pier Paolo Grassi
  2020-06-08 21:32         ` subprocess Bart Schaefer
  1 sibling, 1 reply; 8+ messages in thread
From: Pier Paolo Grassi @ 2020-06-08 14:22 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Daniel Shahaf, Mikael Magnusson, Zsh-Users List


[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

thanks for the debugging technique and the explanation, very interesting.
What would be the best way to learn more about these mechanisms without
digging in the source code of the shell or the kernel? I am not very
comfortable around c
regards

Il giorno sab 6 giu 2020 alle ore 20:19 Bart Schaefer <
schaefer@brasslantern.com> ha scritto:

> On Fri, Jun 5, 2020 at 6:01 PM Daniel Shahaf <d.s@daniel.shahaf.name>
> wrote:
> > > On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> > > > efe() cat $1
> > > > efe <(seq 1 10) | wc -l
> > > > cat: /proc/self/fd/13: No such file or directory
> > > > 0
> > > >
> > > > gives error, instead these all works fine:
> > > > efe =(seq 1 10)| wc -l
> > > > cat <(seq 1 10)| wc -l
> >
> > And this,
> >
> > % () { cat $1 } <(seq 1 10) | wc -l
>
> You can see what's happening if you do it this way:
>
> % efe() ls -l ${${1:+$1:h}:-/proc/self/fd}
> % efe <(seq 1 10) | cat
> total 0
> lrwx------ 1 schaefer schaefer 64 Jun  6 10:59 0 -> /dev/pts/0
> l-wx------ 1 schaefer schaefer 64 Jun  6 10:59 1 -> pipe:[1945962]
> lrwx------ 1 schaefer schaefer 64 Jun  6 10:59 2 -> /dev/pts/0
> lr-x------ 1 schaefer schaefer 64 Jun  6 10:59 3 -> /proc/5533/fd
> % () { ls -l $1:h } <(seq 1 10) | cat
> total 0
> lrwx------ 1 schaefer schaefer 64 Jun  6 11:00 0 -> /dev/pts/0
> l-wx------ 1 schaefer schaefer 64 Jun  6 11:00 1 -> pipe:[1945988]
> lr-x------ 1 schaefer schaefer 64 Jun  6 11:00 11 -> pipe:[1944505]
> lrwx------ 1 schaefer schaefer 64 Jun  6 11:00 2 -> /dev/pts/0
> lr-x------ 1 schaefer schaefer 64 Jun  6 11:00 3 -> /proc/5537/fd
>
> A named function on the left side of a pipeline becomes its own "job",
> where all descriptors above 10 are closed.  An anonymous function
> stays in the parent shell (which is still a forked subshell, because
> of the pipeline) so those descriptors are not closed.
>
> % efe =(seq 1 10) | cat
> total 4
> -rw------- 1 schaefer schaefer 21 Jun  6 11:09 zshkDpnRj
>
> In that case a real file has been created and is referenced by name,
> so the descriptors closing does not matter.
>
> % efe <<(seq 1 10) | cat
> total 0
> lr-x------ 1 schaefer schaefer 64 Jun  6 11:13 0 -> pipe:[1946804]
> l-wx------ 1 schaefer schaefer 64 Jun  6 11:13 1 -> pipe:[1946579]
> lrwx------ 1 schaefer schaefer 64 Jun  6 11:13 2 -> /dev/pts/0
> lr-x------ 1 schaefer schaefer 64 Jun  6 11:13 3 -> /proc/5599/fd
>
> For <<(...) no file is created at all, instead the standard input of
> the function is changed.
>
> > Mikael Magnusson wrote on Fri, 05 Jun 2020 12:25 +0200:
> > >
> > > This also works,
> > > % efe <(seq 1 10) > >(wc -l)
> > > as does this,
> > > % wc -l < <(efe <(seq 1 10))
>
> In both of these cases, there is no pipeline, so efe stays in the
> parent shell and all the descriptors remain open.
>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-06 19:53       ` subprocess Peter Stephenson
@ 2020-06-08 19:55         ` Peter Stephenson
  0 siblings, 0 replies; 8+ messages in thread
From: Peter Stephenson @ 2020-06-08 19:55 UTC (permalink / raw)
  To: zsh-users

On Sat, 2020-06-06 at 20:53 +0100, Peter Stephenson wrote:
> On Sat, 2020-06-06 at 11:19 -0700, Bart Schaefer wrote:
> > On Fri, Jun 5, 2020 at 6:01 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> > > > On 6/5/20, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> > > > > efe() cat $1
> > > > > efe <(seq 1 10) | wc -l
> > > > > cat: /proc/self/fd/13: No such file or directory
> > 
> > A named function on the left side of a pipeline becomes its own "job",
> > where all descriptors above 10 are closed.
> 
> Itʼs this call here.
> 
> How ahout simply reordering so we tidy up after the function is called?
> That fixes the problem, hard to see how that can cause a leak...?

I've taken the silence for lack of controversy rather than stupefaction
and submitted this.

(For workers: I will assume the same for the CLOBBER_EMPTY change
tomorrow.  I already have a reply from Daniel.)

pws


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: subprocess
  2020-06-08 14:22       ` subprocess Pier Paolo Grassi
@ 2020-06-08 21:32         ` Bart Schaefer
  0 siblings, 0 replies; 8+ messages in thread
From: Bart Schaefer @ 2020-06-08 21:32 UTC (permalink / raw)
  To: Pier Paolo Grassi; +Cc: Daniel Shahaf, Mikael Magnusson, Zsh-Users List

On Mon, Jun 8, 2020 at 7:22 AM Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
>
> What would be the best way to learn more about these mechanisms without
> digging in the source code of the shell or the kernel?

Hmm.  Digging in the source code probably isn't the best way even if
you were interested.  It's a tough question.

This is all documented, but not in any single coherent place, mostly
because there are just too many possible ways to combine things to
explain all the variations.  For example the closing of all
descriptors >= 10 when starting a new job is long-standing,
standardized shell behavior, so the zsh manual doesn't mention it,
you're just expected to have learned (elsewhere) that is how shells
work.  I'm not sure the "named functions get their own job" part is
explained very clearly anywhere, but it is explained that anonymous
functions do not behave that way.  (E.g., "Redirections may be applied
to the anonymous function in the same manner as to a current-shell
structure enclosed in braces.")  It's explained that <(...) uses a
special file representing a file descriptor ("If the system supports
the /dev/fd mechanism, the command argument is the name of the device
file corresponding to a file descriptor" -- that should probably
mention /proc/self/fd as well) but does not warn that it might
conflict with the "close all descriptors above 10" behavior.

The upshot is that the best way to learn is probably to ask somebody.
I don't think all these sorts of nuances are even covered by most
books on shell programming.

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, back to index

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-05  2:03 subprocess Pier Paolo Grassi
2020-06-05 10:25 ` subprocess Mikael Magnusson
2020-06-06  1:00   ` subprocess Daniel Shahaf
2020-06-06 18:19     ` subprocess Bart Schaefer
2020-06-06 19:53       ` subprocess Peter Stephenson
2020-06-08 19:55         ` subprocess Peter Stephenson
2020-06-08 14:22       ` subprocess Pier Paolo Grassi
2020-06-08 21:32         ` subprocess Bart Schaefer

zsh-users

Archives are clonable: git clone --mirror http://inbox.vuxu.org/zsh-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.zsh.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git