Process substitution and shell functions

Process substitution and shell functions

[Bart Schaefer]

I can't find anything about this in zsh-3.0.0/etc/FAQ or zsh-3.0.0/etc/BUGS
so I'm assuming it's fairly new.

zsh% function foo { echo $1 ; cat $1 }
zsh% foo =(echo hello world)
/tmp/zsha24295
cat: /tmp/zsha24295: No such file or directory

The temp file for the process substitution seems to be getting removed a bit
too aggressively.  It should stay around at least until the shell function
finishes, should it not?

-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"

Re: Process substitution and shell functions

[Louis Granboulan]

> I can't find anything about this in zsh-3.0.0/etc/FAQ or zsh-3.0.0/etc/BUGS
> so I'm assuming it's fairly new.
It is an old bug. zsh-2.5.0 works well, but zsh-2.5.03 has it.
This is due to some patch in exec.c, probably the big patch from 2.5.01
to 2.5.02 by Richard Coleman (Mon Jul 25 10:30:32 1994) to "allow
suspend/resume pipes ending in a list/loop/function and jobs with
a $(...) while that is running".

function foo { echo $1; head -1 $1 ; head -1 $1 }
foo =(echo hello world)

The variable last_file_list is recovered before the shell function
finishes.

-- Louis Granboulan <Louis.Granboulan@ens.fr>

Re: Process substitution and shell functions

[Louis Granboulan]

The removal of temporary files has many strange bugs.
At the end of this message, there is a patch that should fix all this.



The procedure deletepipejob (in exec.c) deletes a job without
deleting the filelist.  The variable last_file_list keeps track
of the filelist to delete. The value of last_file_list is
recovered in execpline.  It is saved and restored by execsave
and execrestore.

deletepipejob is called in execshfunc and execcursh.
The temporary files are removed at the end of the first command
called by zsh.

execshfunc
    This is the bug we've seen...
    First example:
	zsh% function foo { echo $1 ; cat $1 }
	zsh% foo =(echo hello world)
	/tmp/zsha24295
	cat: /tmp/zsha24295: No such file or directory
    Second example:
	zsh% autoload foobar
	zsh% foobar =(echo hello world)
	zsh: function not found: foobar
	zsh% ls -l /tmp/zsha21572
	-rw-------  1 granboul       12 Sep 24 18:01 /tmp/zsha21572
	zsh% ls -l /tmp/zsha21572
	/tmp/zsha21572 not found

execcursh
    I don't know why the removing of temporary files
    is delayed to the end of the first command.

My patch moves the recovering of last_file_list from execpline
to execshfunc.
I don't understand why this was in execpline...


*** Src/zsh.h.orig	Mon Aug 12 19:57:32 1996
--- Src/zsh.h	Tue Sep 24 18:08:07 1996
***************
*** 598,604 ****
      int list_pipe_child;
      int list_pipe_job;
      char list_pipe_text[JOBTEXTSIZE];
-     LinkList last_file_list;
      int lastval;
      int noeval;
      int badcshglob;
--- 598,603 ----
*** Src/exec.c.orig	Thu Aug 15 12:38:56 1996
--- Src/exec.c	Tue Sep 24 18:44:27 1996
***************
*** 119,136 ****
  static int nowait, pline_level = 0;
  static int list_pipe_child = 0, list_pipe_job;
  static char list_pipe_text[JOBTEXTSIZE];
- static LinkList last_file_list;
- 
- /**/
- void
- deletepipejob(void)
- {
-     if (!list_pipe) {
- 	last_file_list = jobtab[thisjob].filelist;
- 	jobtab[thisjob].filelist = NULL;
- 	deletejob(jobtab + thisjob);
-     }
- }
  
  /* execute a current shell command */
  
--- 119,124 ----
***************
*** 138,144 ****
  int
  execcursh(Cmd cmd)
  {
!     deletepipejob();
      execlist(cmd->u.list, 1, cmd->flags & CFLAG_EXEC);
      cmd->u.list = NULL;
      return lastval;
--- 126,132 ----
  int
  execcursh(Cmd cmd)
  {
!     if (!list_pipe) deletejob(jobtab + thisjob);
      execlist(cmd->u.list, 1, cmd->flags & CFLAG_EXEC);
      cmd->u.list = NULL;
      return lastval;
***************
*** 632,640 ****
      if (how & Z_TIMED)
  	jobtab[thisjob].stat |= STAT_TIMED;
  
-     jobtab[newjob].filelist = last_file_list;
-     last_file_list = NULL;
- 
      if (l->flags & PFLAG_COPROC) {
  	how = Z_ASYNC;
  	if (coprocin >= 0) {
--- 620,625 ----
***************
*** 2407,2416 ****
  {
      List funcdef;
      char *nam;
  
      if (errflag)
  	return;
!     deletepipejob();
  
      /* Are we dealing with an autoloaded shell function? */
      if (shf->flags & PM_UNDEFINED) {
--- 2392,2407 ----
  {
      List funcdef;
      char *nam;
+     LinkList last_file_list = NULL;
  
      if (errflag)
  	return;
! 
!     if (!list_pipe) {
! 	last_file_list = jobtab[thisjob].filelist;
! 	jobtab[thisjob].filelist = NULL;
! 	deletejob(jobtab + thisjob);
!     }
  
      /* Are we dealing with an autoloaded shell function? */
      if (shf->flags & PM_UNDEFINED) {
***************
*** 2418,2423 ****
--- 2409,2415 ----
  	if (!(funcdef = getfpfunc(nam))) {
  	    zerr("function not found: %s", nam, 0);
  	    lastval = 1;
+ 	    if (!list_pipe) deletefilelist(last_file_list);
  	    return;
  	}
  	PERMALLOC {
***************
*** 2433,2443 ****
--- 2425,2437 ----
  	if ((shf = (Shfunc) shfunctab->getnode(shfunctab, nam))
  	    && shf->funcdef && shf->funcdef != funcdef)
  	    doshfunc(shf->funcdef, cmd->args, shf->flags, 0);
+ 	if (!list_pipe) deletefilelist(last_file_list);
  	return;
      }
  
      /* Normal shell function execution */
      doshfunc(shf->funcdef, cmd->args, shf->flags, 0);
+     if (!list_pipe) deletefilelist(last_file_list);
  }
  
  /* execute a shell function */
***************
*** 2670,2676 ****
      es->list_pipe_child = list_pipe_child;
      es->list_pipe_job = list_pipe_job;
      strcpy(es->list_pipe_text, list_pipe_text);
-     es->last_file_list = last_file_list;
      es->lastval = lastval;
      es->noeval = noeval;
      es->badcshglob = badcshglob;
--- 2664,2669 ----
***************
*** 2698,2704 ****
      list_pipe_child = exstack->list_pipe_child;
      list_pipe_job = exstack->list_pipe_job;
      strcpy(list_pipe_text, exstack->list_pipe_text);
-     last_file_list = exstack->last_file_list;
      lastval = exstack->lastval;
      noeval = exstack->noeval;
      badcshglob = exstack->badcshglob;
--- 2691,2696 ----
*** Src/jobs.c.orig	Wed Jul 31 20:13:17 1996
--- Src/jobs.c	Tue Sep 24 18:38:04 1996
***************
*** 368,390 ****
  
  /**/
  void
  deletejob(Job jn)
  {
      struct process *pn, *nx;
-     char *s;
  
      for (pn = jn->procs; pn; pn = nx) {
  	nx = pn->next;
  	zfree(pn, sizeof(struct process));
      }
      zsfree(jn->pwd);
!     if (jn->filelist) {
! 	while ((s = (char *)getlinknode(jn->filelist))) {
! 	    unlink(s);
! 	    zsfree(s);
! 	}
! 	zfree(jn->filelist, sizeof(struct linklist));
!     }
      if (jn->ty)
  	zfree(jn->ty, sizeof(struct ttyinfo));
  
--- 368,399 ----
  
  /**/
  void
+ deletefilelist(LinkList file_list)
+ {
+     char *s;
+     if (file_list) {
+ 	while ((s = (char *)getlinknode(file_list))) {
+ 	    unlink(s);
+ 	    zsfree(s);
+ 	}
+ 	zfree(file_list, sizeof(struct linklist));
+     }
+ }
+ 
+ /**/
+ void
  deletejob(Job jn)
  {
      struct process *pn, *nx;
  
      for (pn = jn->procs; pn; pn = nx) {
  	nx = pn->next;
  	zfree(pn, sizeof(struct process));
      }
      zsfree(jn->pwd);
! 
!     deletefilelist(jn->filelist);
! 
      if (jn->ty)
  	zfree(jn->ty, sizeof(struct ttyinfo));
  

Re: Process substitution and shell functions

[Bart Schaefer]

On Sep 24,  6:47pm, Louis Granboulan wrote:
} Subject: Re: Process substitution and shell functions
}
} My patch moves the recovering of last_file_list from execpline
} to execshfunc.
} I don't understand why this was in execpline...

It can't go in execshfunc because execshfunc isn't executed when using
a simple builtin or an external command.

I think you have the right idea, it's just that the wrong job entry
was given the file list in the first place.  Or something.

-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"

Re: Process substitution and shell functions

[Peter Stephenson]

"Bart Schaefer" wrote:
> On Sep 24,  6:47pm, Louis Granboulan wrote:
> } Subject: Re: Process substitution and shell functions
> }
> } My patch moves the recovering of last_file_list from execpline
> } to execshfunc.
> } I don't understand why this was in execpline...
> 
> It can't go in execshfunc because execshfunc isn't executed when using
> a simple builtin or an external command.

Does that matter?  last_file_list was only ever set before from
execshfunc() and execcursh() via deletepipejob().  It was then set to
NULL at the beginning of every pipeline... goodness knows what effect
that was supposed to have, but I think LG is saying that the (real)
desired effect can be obtained locally in execshfunc() and
execcursh().

As far as I can see simple builtins/builtouts didn't need this
mechanism, the job.filelist element worked directly.  They would only
be affected when a new pipeline was started to run one, at which point
they would take over the file list from any shell function or {...}
construct and delete the appropriate files when they finished, which
seems a bit strange (and must have been the cause of the bug being
fixed).

I agree with LG that there's no obvious reason why shell functions and
{...}'s shouldn't be directly responsible for their own file lists.  I
think the `if (!list_pipe) ...' tests are OK since list_pipe is either
set and restored in the same function or unconditionally set to zero,
which would presumably be harmless here.

Anyway, the patch seems to work O.K. --- unless, of course, you know
better.  It's quite likely I'm missing something, or a lot.  The only
worry I have is that temporary files might get left over on an exec.

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77330
Deutches Electronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: Process substitution and shell functions

[Bart Schaefer]

On Sep 25, 10:51am, Peter Stephenson wrote:
} Subject: Re: Process substitution and shell functions
}
} "Bart Schaefer" wrote:
} > On Sep 24,  6:47pm, Louis Granboulan wrote:
} > } My patch moves the recovering of last_file_list from execpline
} > } to execshfunc.
} > It can't go in execshfunc because execshfunc isn't executed when using
} > a simple builtin or an external command.
} 
} Does that matter?  last_file_list was only ever set before from
} execshfunc() and execcursh() via deletepipejob().

Ah, I missed that.  I thought last_file_list was being used *every* time
a job table entry was deleted.

-- 
Bart Schaefer                             Brass Lantern Enterprises
http://www.well.com/user/barts            http://www.nbn.com/people/lantern

New male in /home/schaefer:
>N  2 Justin William Schaefer  Sat May 11 03:43  53/4040  "Happy Birthday"