Re: FPATH/autoload still strange in -dev-21

Re: FPATH/autoload still strange in -dev-21

[Sven Wischnowsky]

Bart Schaefer wrote:

> I just got -dev-21 + patches to 10477 compiled and installed, and I'm still
> having the strange problems with exported FPATH and (autoload -U) failing
> that I described in 10316.  I examined strace output and discovered that
> in the expression:
> 
> 	if (autoload -U 2> /dev/null); then
> 
> First zsh forks the subshell, and then the subshell forks again, apparently
> in order to redirect the output of the builtin; but then the subshell exits
> for some reason I can't follow, leaving the autoload running as an orphan.
> So zsh never sees the exit status of autoload.  (The process forked from
> the subshell is plugging away close()ing each of 256 file descriptors, 
> except for 0, 1, and 2, most of which are returning EBADF, while its parent
> subshell is exiting.)

The subshell shouldn't fork, of course. That needs only be done when
we are in a pipe, not for ordinary redirection...

> ...
> 
> It occurs to me that this could be related to Geoff's report of the "jobs"
> command claiming there is a nonexistent job 3. 

Interestingly, I had the same idea before coming to this paragraph.

> Why having FPATH in the
> environment makes ths happen, I have no idea, but I can reproduce it quite
> nicely, so if anyone (Sven?) wants the sample strace output I'd be happy
> to provide it.

Yes, please.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: FPATH/autoload still strange in -dev-21

[Zefram]

Bart Schaefer wrote:
>Hmm.  "2>& foo" does indeed do so, but "2&> foo" only makes one copy.  Is
>that also expected?

Hmm.  It appears that "2&>" is treated as "2 &>" -- the "&>" isn't
recognised as a redirection operator for the purposes of parsing the
I/O number.

% echo x 2&> foo
% cat foo
x 2
% echo x 2>& foo
x
% cat foo
% 

Curiously enough, bash (2.03.0(1)) does the same thing.

-zefram

Index: ChangeLog
===================================================================
RCS file: /cvsroot/zsh/zsh/ChangeLog,v
retrieving revision 1.27
diff -c -r1.27 ChangeLog
*** ChangeLog	2000/04/06 16:44:02	1.27
--- ChangeLog	2000/04/06 17:16:32
***************
*** 1,5 ****
--- 1,9 ----
+ 2000-04-06  Andrew Main  <zefram@zsh.org>
+ 
+ 	* zefram2: Src/lex.c: Support "3&> foo" etc.
+ 
  2000-04-06  Andrew Main  <zefram@zsh.org>
  
  	* zefram1: configure.in, Etc/zsh-development-guide: List of tools
  	required for development work, and a little more conspicuous
  	explanation of the config.status hack.
Index: Src/lex.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/lex.c,v
retrieving revision 1.2
diff -c -r1.2 lex.c
*** Src/lex.c	2000/04/04 01:16:25	1.2
--- Src/lex.c	2000/04/06 17:16:33
***************
*** 642,648 ****
  	return DOUTPAR;
      } else if (idigit(c)) {	/* handle 1< foo */
  	d = hgetc();
! 	if (d == '>' || d == '<') {
  	    peekfd = c - '0';
  	    c = d;
  	} else {
--- 642,659 ----
  	return DOUTPAR;
      } else if (idigit(c)) {	/* handle 1< foo */
  	d = hgetc();
! 	if(d == '&') {
! 	    d = hgetc();
! 	    if(d == '>') {
! 		peekfd = c - '0';
! 		hungetc('>');
! 		c = '&';
! 	    } else {
! 		hungetc(d);
! 		lexstop = 0;
! 		hungetc('&');
! 	    }
! 	} else if (d == '>' || d == '<') {
  	    peekfd = c - '0';
  	    c = d;
  	} else {
***************
*** 702,707 ****
--- 713,719 ----
  	else if (d == '!' || d == '|')
  	    return AMPERBANG;
  	else if (d == '>') {
+ 	    tokfd = peekfd;
  	    d = hgetc();
  	    if (d == '!' || d == '|')
  		return OUTANGAMPBANG;
***************
*** 715,721 ****
  	    }
  	    hungetc(d);
  	    lexstop = 0;
- 	    tokfd = -1;
  	    return AMPOUTANG;
  	}
  	hungetc(d);
--- 727,732 ----

Re: FPATH/autoload still strange in -dev-21

[Bart Schaefer]

On Apr 6,  4:56pm, Zefram wrote:
} Subject: Re: FPATH/autoload still strange in -dev-21
}
} Bart Schaefer wrote:
} >} Make `2>& <non-number>' be silently treated like `2> <non-number>'?
} >
} >I'd vote for that anyway.
} 
} I wouldn't.  At the moment it's perfectly clear: &> does two redirections.
} [...]  The expected behaviour of "2>& foo" is to put two copies of
} stderr into the file [...]

Hmm.  "2>& foo" does indeed do so, but "2&> foo" only makes one copy.  Is
that also expected?

} You're talking about adding an exception, which really *will* confuse
} people.

I don't feel especially strongly about it either way, but why would
anyone reasonably expect to get two dups of the same descriptor from a
single redirection operator?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: FPATH/autoload still strange in -dev-21

[Zefram]

Bart Schaefer wrote:
>} Make `2>& <non-number>' be silently treated like `2> <non-number>'?
>
>I'd vote for that anyway.

I wouldn't.  At the moment it's perfectly clear: &> does two redirections.
You're talking about adding an exception, which really *will* confuse
people.  The expected behaviour of "2>& foo" is to put two copies of
stderr into the file; this is a pretty innocuous failure mode if anyone
does do it by accident, and the cause will be pretty obvious.

-zefram

Re: FPATH/autoload still strange in -dev-21

[Sven Wischnowsky]

Bart Schaefer wrote:

> ...
>
> } I'm not sure any more if this is really a race condition. I suspect a
> } memory corruption of some kind, triggered by importing $FPATH from the 
> } environment (which would also explain why I can't reproduce, with
> } different allocation behaviours and all that).
> 
> And the memory corruption ... steps on the exit status of the subshell as
> returned to the parent, somehow?  I can't see how that could be ... the
> subshell is calling _exit(0) but the parent still takes the false branch.
> Hrm.

I was thinking about something that messes up some memory used
somewhere when that `if' is executed. Wild guessing, of course, but
memory problems are the only ones I can think of when thinking about
the difference between FPATH in the environment and not in the
environment.

I may be completely wrong, of course. Debugging session in execif()
could tell us something.

> } And the initial setup of fpath also depends on the site-function dirs
> } that are automatically added and so on...
> 
> I do NOT compile with --enable-function-subdirs, if that matters.

Hm. I have some default function dirs...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: FPATH/autoload still strange in -dev-21

[Bart Schaefer]

On Apr 6, 10:18am, Sven Wischnowsky wrote:
} Subject: Re: FPATH/autoload still strange in -dev-21
}
} Bart Schaefer wrote:
} 
} > 	if (autoload -U 2> /dev/null); then
} > 
} > First zsh forks the subshell, and then the subshell forks again
} 
} Bart sent me the strace output and there he also repeated the line
} above, but slightly different:
} 
}     if (autoload -U 2>& /dev/null); then
} 
} So, Bart, if that is the form you have in .zshenv, this explains the
} processes.

The second is indeed the form I had at the time I created that strace.
Sorry about the confusion; the first time I typed it by hand and the
second I cut'n'pasted ... (and it's yet a third way in 10316, because
I cut'n'pasted _that_ while in the midst of editing it different ways
to try to get zsh to tell me why it was failing ... gaah, sorry).

} `2>& /dev/null' makes the shell look at the word after the
} redirection. This isn't a number, so the whole thing is treated like
} `&>' on file descriptor 2.

I noticed that just after I sent the strace to you and deleted the `&',
but I didn't resend the strace because I forgot about ...

} This makes it treat it as a multio and the
} funny chap with the many close()s is the zsh-tee process.

... and because removing the `&' didn't change the visible failure.  Nor
does removing the entire redirection, for that matter.

} This is ugly. `&>' on file descriptor 2 makes no sense at all and if
} even Bart is confused, I think we should do something about it.

I wasn't confused ... it was an editing error.  I originally tried
`(autoload -U 2>&-)' which produced an error about an invalid file
descriptor, and then I inadvertently changed only the `-'.  However:

} Make `2>& <non-number>' be silently treated like `2> <non-number>'?

I'd vote for that anyway.

} > Now, the odd thing is, the subshell calls _exit(0), so you'd think that zsh
} > would take the "then" branch of the "if" -- but it doesn't, it takes the
} > "else" branch.  So there must be two race conditions (?) here -- one in the
} > subshell where it loses track of its child, and another where the topmost
} > zsh mishandles the exit of that subshell.
} 
} I'm not sure any more if this is really a race condition. I suspect a
} memory corruption of some kind, triggered by importing $FPATH from the 
} environment (which would also explain why I can't reproduce, with
} different allocation behaviours and all that).

And the memory corruption ... steps on the exit status of the subshell as
returned to the parent, somehow?  I can't see how that could be ... the
subshell is calling _exit(0) but the parent still takes the false branch.
Hrm.
 
} And the initial setup of fpath also depends on the site-function dirs
} that are automatically added and so on...

I do NOT compile with --enable-function-subdirs, if that matters.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: FPATH/autoload still strange in -dev-21

[Sven Wischnowsky]

Bart Schaefer wrote:

> I just got -dev-21 + patches to 10477 compiled and installed, and I'm still
> having the strange problems with exported FPATH and (autoload -U) failing
> that I described in 10316.  I examined strace output and discovered that
> in the expression:
> 
> 	if (autoload -U 2> /dev/null); then
> 
> First zsh forks the subshell, and then the subshell forks again, apparently
> in order to redirect the output of the builtin; but then the subshell exits
> for some reason I can't follow, leaving the autoload running as an orphan.
> So zsh never sees the exit status of autoload.  (The process forked from
> the subshell is plugging away close()ing each of 256 file descriptors, 
> except for 0, 1, and 2, most of which are returning EBADF, while its parent
> subshell is exiting.)

Bart sent me the strace output and there he also repeated the line
above, but slightly different:

    if (autoload -U 2>& /dev/null); then

So, Bart, if that is the form you have in .zshenv, this explains the
processes. `2>& /dev/null' makes the shell look at the word after the
redirection. This isn't a number, so the whole thing is treated like
`&>' on file descriptor 2. This makes it treat it as a multio and the
funny chap with the many close()s is the zsh-tee process.

This is ugly. `&>' on file descriptor 2 makes no sense at all and if
even Bart is confused, I think we should do something about it. But
what? Make `2>& <non-number>' be silently treated like `2> <non-number>'?
Make it emit a warning, or even an error?

With the line above, my strace output is exactly the same as for
Bart. But I still can't reproduce the bug Bart gets when FPATH is
exported.

> Now, the odd thing is, the subshell calls _exit(0), so you'd think that zsh
> would take the "then" branch of the "if" -- but it doesn't, it takes the
> "else" branch.  So there must be two race conditions (?) here -- one in the
> subshell where it loses track of its child, and another where the topmost
> zsh mishandles the exit of that subshell.

I'm not sure any more if this is really a race condition. I suspect a
memory corruption of some kind, triggered by importing $FPATH from the 
environment (which would also explain why I can't reproduce, with
different allocation behaviours and all that).

And the initial setup of fpath also depends on the site-function dirs
that are automatically added and so on... Hm, I would try to debug the 
fpath initialisation in setupvals() and createparamtable().


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

FPATH/autoload still strange in -dev-21

[Bart Schaefer]

I just got -dev-21 + patches to 10477 compiled and installed, and I'm still
having the strange problems with exported FPATH and (autoload -U) failing
that I described in 10316.  I examined strace output and discovered that
in the expression:

	if (autoload -U 2> /dev/null); then

First zsh forks the subshell, and then the subshell forks again, apparently
in order to redirect the output of the builtin; but then the subshell exits
for some reason I can't follow, leaving the autoload running as an orphan.
So zsh never sees the exit status of autoload.  (The process forked from
the subshell is plugging away close()ing each of 256 file descriptors, 
except for 0, 1, and 2, most of which are returning EBADF, while its parent
subshell is exiting.)

Now, the odd thing is, the subshell calls _exit(0), so you'd think that zsh
would take the "then" branch of the "if" -- but it doesn't, it takes the
"else" branch.  So there must be two race conditions (?) here -- one in the
subshell where it loses track of its child, and another where the topmost
zsh mishandles the exit of that subshell.

It occurs to me that this could be related to Geoff's report of the "jobs"
command claiming there is a nonexistent job 3.  Why having FPATH in the
environment makes ths happen, I have no idea, but I can reproduce it quite
nicely, so if anyone (Sven?) wants the sample strace output I'd be happy
to provide it.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com