funny subshell effect

funny subshell effect

[Mikael Magnusson]

This confused me for a few minutes, I was trying to play a random midi file with
pmidi *(oe:REPLY=\$RANDOM:[1])
but it plays the same one each time, but when i tried
echo *(oe:REPLY=\$RANDOM:[1])
it printed a different one each time

% repeat 3; do echo .(e:REPLY=\$RANDOM:); done
17
25549
6369
% repeat 3; do command echo .(e:REPLY=\$RANDOM:); done
5801
5801
5801

Is this something that must be so? (My guess is yes, but it can't hurt to ask).
(I know I can work around it easily by assigning to a var first).

-- 
Mikael Magnusson

Re: funny subshell effect

[Phil Pennock]

On 2011-03-10 at 00:18 +0100, Mikael Magnusson wrote:
> This confused me for a few minutes, I was trying to play a random midi file with
> pmidi *(oe:REPLY=\$RANDOM:[1])
> but it plays the same one each time, but when i tried
> echo *(oe:REPLY=\$RANDOM:[1])
> it printed a different one each time
> 
> % repeat 3; do echo .(e:REPLY=\$RANDOM:); done
> 17
> 25549
> 6369
> % repeat 3; do command echo .(e:REPLY=\$RANDOM:); done
> 5801
> 5801
> 5801
> 
> Is this something that must be so? (My guess is yes, but it can't hurt to ask).
> (I know I can work around it easily by assigning to a var first).

If you assign to RANDOM in the subshell, that's a call to srand() which
will re-seed the RNG.

At present, the retrieval is just:
zlong
randomgetfn(UNUSED(Param pm))
{
    return rand() & 0x7fff;
}

Given the amount of forking going on normally, reseeding on every fork
would be bad, but perhaps something similar to the code I added to Exim
would work, so:

zlong
randomgetfn(UNUSED(Param pm))
{
  static pid_t last_rand_pid = 0;
  pid_t p;

  p = getpid()
  if (p != last_rand_pid)
    {
    last_rand_pid = p;
#ifdef HAVE_SRANDOMDEV_OR_WHATEVER_ZSH_HAS_AS_GUARD
    srandomdev();
#else
    gettimeofday(&tv, NULL);
    srandom(tv.tv_sec | tv.tv_usec | getpid());
#endif
    }
  return rand() & 0x7fff;
}

Except that I actually made Exim prefer arc4random()/arc4random_stir()
if available, and actually uses much stronger randomness from OpenSSL if
linked against that, but that's because people cook up homebrew access
token stuff and I wanted to provide something "not guaranteed
cryptographically strong, but at least not a complete nightmare".

Anyway, does this look like a reasonable approach?

-Phil

Re: funny subshell effect

[Bart Schaefer]

On Mar 10, 12:18am, Mikael Magnusson wrote:
} Subject: funny subshell effect
}
} % repeat 3; do command echo .(e:REPLY=\$RANDOM:); done
} 5801
} 5801
} 5801
} 
} Is this something that must be so?

Yep, it's intentional.  Read the manual:

     The values of RANDOM form an intentionally-repeatable pseudo-random
     sequence; subshells that reference RANDOM will result in identical
     pseudo-random values unless the value of RANDOM is referenced or
     seeded in the parent shell in between subshell invocations.

Above that, "The random number generator can be seeded by assigning a
numeric value to RANDOM."

% for i in {1..3}; do command echo .(e:RANDOM=$i\;REPLY=\$RANDOM:); done
17767
6138
20026

If you're not generating more than one number per second, a good way
to see the generator is with $SECONDS.  Or if your system supports a
higher granularity clock, something like

    typeset -F SECONDS
    repeat 3; do
      command echo .(e:'RANDOM=$((SECONDS*1000));REPLY=$RANDOM':);
    done

Consequently, Phil ... your approach may be reasonable, but we don't
want to apply it. :-)

Re: funny subshell effect

[Mikael Magnusson]

On 10 March 2011 15:56, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Mar 10, 12:18am, Mikael Magnusson wrote:
> } Subject: funny subshell effect
> }
> } % repeat 3; do command echo .(e:REPLY=\$RANDOM:); done
> } 5801
> } 5801
> } 5801
> }
> } Is this something that must be so?
>
> Yep, it's intentional.  Read the manual:
>
>     The values of RANDOM form an intentionally-repeatable pseudo-random
>     sequence; subshells that reference RANDOM will result in identical
>     pseudo-random values unless the value of RANDOM is referenced or
>     seeded in the parent shell in between subshell invocations.

Well, I know that part, what confused me was that I wasn't starting
any subshells. But it seems that globbing is performed after forking
to run an external command, and of course zsh doesn't bother to fork
if the command is builtin. Is there some reason I'm not thinking of
why it's like that?

-- 
Mikael Magnusson

Re: funny subshell effect

[Bart Schaefer]

On Mar 10,  4:11pm, Mikael Magnusson wrote:
} Subject: Re: funny subshell effect
}
} On 10 March 2011 15:56, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > On Mar 10, 12:18am, Mikael Magnusson wrote:
} > } Subject: funny subshell effect
} > }
} > } % repeat 3; do command echo .(e:REPLY=\$RANDOM:); done
} > } 5801
} > } 5801
} > } 5801
} >
} ... what confused me was that I wasn't starting
} any subshells. But it seems that globbing is performed after forking
} to run an external command, and of course zsh doesn't bother to fork
} if the command is builtin. Is there some reason I'm not thinking of
} why it's like that?

It bothered me that I couldn't come up with a reason for this.  The one
that comes to mind is that if a glob goes runaway (you use ***/ with
a path that contains a cyclic symlink, for example) with a builtin you
may not be able to interrupt it; whereas with an external command where
the glob is running post-fork, you are able to interrupt.  Or that may
be a former bug and now it would be OK ... I wasn't willing to try to
lock up my shell to find out.

In fact zsh globs the word in command position separately before the
fork, and then globs the rest of the list afterward.  This leads to
strange bits like this:

% touch noglob
% n*b echo x*
x*
%

It also leads to this:

% repeat 3 do command echo x*  
repeat> done
zsh: no match
zsh: no match
zsh: no match
% repeat 3 do builtin echo x*
done
zsh: no match
% 

With the fork, the globbing error applies only to the forked process,
before the exec has occurred.  I tried wholesale moving the globlist()
up to before any decision about forking is made (sample patch below),
and all the "make check" tests pass; the only obvious difference is
that the two examples shown above become identical -- both run the
loop exactly once.

Compare this to discussion "exec -a and parameter expansion" from the
zsh-workers archive back in January.  That sort of died out without a
resolution.  I suspect that the patch below has some similar obscure
problems.  For example certain files in /proc will point to different
places, or possibly not exist at all, if globbing precedes forking;
that could in turn affect redirection, etc.


--- 8< --- EXAMPLE ONLY - NOT FOR ACTUAL USE --- 8< ---
Index: Src/exec.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/Src/exec.c,v
retrieving revision 1.40
diff -c -r1.40 exec.c
--- exec.c	21 Dec 2010 16:41:16 -0000	1.40
+++ exec.c	17 Mar 2011 03:10:42 -0000
@@ -2735,6 +2744,16 @@
 	}
     }
 
+    if ((esglob = !(cflags & BINF_NOGLOB)) && args && htok) {
+	LinkList oargs = args;
+	globlist(args, 0);
+	args = oargs;
+    }
+    if (errflag) {
+	lastval = 1;
+	goto err;
+    }
+
     /* This is nonzero if the command is a current shell procedure? */
     is_cursh = (is_builtin || is_shfunc || nullexec || type >= WC_CURSH);
 
@@ -2844,16 +2863,6 @@
 	is_exec = 1;
     }
 
-    if ((esglob = !(cflags & BINF_NOGLOB)) && args && htok) {
-	LinkList oargs = args;
-	globlist(args, 0);
-	args = oargs;
-    }
-    if (errflag) {
-	lastval = 1;
-	goto err;
-    }
-
     /* Make a copy of stderr for xtrace output before redirecting */
     fflush(xtrerr);
     if (isset(XTRACE) && xtrerr == stderr &&


--