[BUG] Directory glob picks up running or already-run scripts on OS X

[BUG] Directory glob picks up running or already-run scripts on OS X

[Zhiming Wang]

With zsh 5.2 or master running on OS X 10.11.5 or macOS 10.12 PB1, a glob for
directories could pick up scripts that are either running or have been
run. Note that this seems to only apply to scripts, not random executables
(e.g., an executable created by a C compiler). Here's a session that reproduces
the issue:

    $ cd $(mktemp -d)
    $ print '#!/usr/bin/env zsh' >script1
    $ print $'#!/usr/bin/env zsh\nprint -l */' >script2
    $ chmod +x script1 script2
    $ print -l */
    zsh: no matches found: */
    $ ./script1
    $ print -l */
    script1/
    $ ./script2
    script1/
    script2/
    $ print -l */
    script1/
    script2/
    $ print $'#include <stdio.h>\nint main() { printf("hello, world"); }' >hello.c
    $ cc -o hello hello.c
    $ ./hello
    hello, world%
    $ print -l */
    script1/
    script2/

I cannot reproduce this issue with zsh 5.0.8 on OS X, so something went wrong
in between. Unfortunately, due to some limitations I cannot test other versions
right now. I also cannot repoduce this issue on Linux with any zsh version
(including 5.2 or master), so this seems to be an OS X specific issue.

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Bart Schaefer]

On Fri, Jul 8, 2016 at 8:55 PM, Zhiming Wang <zmwangx@gmail.com> wrote:
> With zsh 5.2 or master running on OS X 10.11.5 or macOS 10.12 PB1, a glob for
> directories could pick up scripts that are either running or have been
> run.

I'm not able to reproduce this.  I happen to have an iMac that was
upgraded to El Capitan (10.11.5) only yesterday.  This morning I did
the following:

1. Install xcode and agree to the license.
2. Download automake and autoconf from ftp.gnu.org, build, install.
3. Pull the latest zsh from sourceforge git.
4. Recompile zsh with static module linkage so I can run from the build tree
5. Run: Src/zsh -f
6. Attempt to reproduce using your steps, modified a little because
"env zsh" won't find the binary I want.

macadamia% cd $(mktemp -d)
macadamia% print $'#!/usr/bin/env zsh' >script1
macadamia% print $'#!/usr/bin/env zsh\nprint -l */' >script2
macadamia% chmod +x script1 script2
macadamia% print -l */
zsh: no matches found: */
macadamia% ~-/Src/zsh -f script1
macadamia% print -l */
zsh: no matches found: */
macadamia% ~-/Src/zsh -f script2
script2:2: no matches found: */
macadamia% uname -a
Darwin macadamia.local 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19
18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
macadamia% print $ZSH_PATCHLEVEL
zsh-5.2-310-g304aa25


> I cannot reproduce this issue with zsh 5.0.8 on OS X, so something went wrong
> in between. Unfortunately, due to some limitations I cannot test other versions
> right now. I also cannot repoduce this issue on Linux with any zsh version
> (including 5.2 or master), so this seems to be an OS X specific issue.

This may even be a compiler-specific issue.  Note that I'm
deliberately using the stock compiler with neither homebrew nor
macports and minimal GNU tool install.

What did you use to build zsh?

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Bart Schaefer]

On Sun, Jul 10, 2016 at 5:31 PM, Zhiming Wang <zmwangx@gmail.com> wrote:
>
> However, the issue does not only show up on custom builds. It also affects
> Apple's stock build on macOS 10.12 Public Beta 1 (build 16A238m)

This is suddenly sounding vaguely familiar.  Does the problem
disappear if you build with compiler optimizations disabled?

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Zhiming Wang]

> On Jul 11, 2016, at 8:38 AM, Bart Schaefer <schaefer@brasslantern.com> wrote:
> 
> On Sun, Jul 10, 2016 at 5:31 PM, Zhiming Wang <zmwangx@gmail.com> wrote:
> 
> This is suddenly sounding vaguely familiar.  Does the problem
> disappear if you build with compiler optimizations disabled?

It doesn't. This time I compiled zsh master (304aa25) manually:

    $ Util/preconfig
    $ ./configure CFLAGS='-g -O0'
    ...
    zsh configuration
    -----------------
    zsh version               : 5.2-dev-1
    host operating system     : x86_64-apple-darwin15.5.0
    source code location      : .
    compiler                  : gcc
    preprocessor flags        :
    executable compiler flags : -g -O0
    module compiler flags     : -g -O0 -fno-common
    executable linker flags   : -Wl,-x  -rdynamic
    module linker flags       : -Wl,-x  -bundle -flat_namespace -undefined suppress
    library flags             : -lgdbm -liconv -ldl -ltermcap -lm  -lc
    installation basename     : zsh
    binary install path       : /usr/local/bin
    man page install path     : /usr/local/share/man
    info install path         : /usr/local/share/info
    functions install path    : /usr/local/share/zsh/5.2-dev-1/functions
    ...
    $ make

Of course here gcc is actually clang.

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Bart Schaefer]

I'm not sure whether this was introduced when adding the (Y) qualifier,
or if it was already lurking.  With the following DPUTS() added and zsh
compiled with debugging, Test/D02glob.ztst will fail (on any platform).

diff --git a/Src/glob.c b/Src/glob.c
index 2051016..5260b63 100644
--- a/Src/glob.c
+++ b/Src/glob.c
@@ -318,6 +318,8 @@ insert(char *s, int checked)
     char *news = s;
     int statted = 0;
 
+    DPUTS(!s || !*s, "BUG: adding empty string as glob match");
+
     queue_signals();
     inserts = NULL;
 

The problem seems to be that scanner() calls recursively one extra
time when looking at "notadirectory/".

In that case statfullpath("") is called and, for reasons I haven't
figured out, that returns success on MacOS when called in exactly
the circumstances that Zhiming describes.

This will eventually correct itself, which leads me to suspect it has
something to do with re-using the global static "pathbuf".

The simple fix would be for insert() to be a no-op when called on the
empty string, but it might be useful to examine scanner() in more
detail to try to avoid the extra recursion.

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Peter Stephenson]

On Sun, 10 Jul 2016 21:58:21 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm not sure whether this was introduced when adding the (Y) qualifier,
> or if it was already lurking.  With the following DPUTS() added and zsh
> compiled with debugging, Test/D02glob.ztst will fail (on any platform).
> 
> diff --git a/Src/glob.c b/Src/glob.c
> index 2051016..5260b63 100644
> --- a/Src/glob.c
> +++ b/Src/glob.c
> @@ -318,6 +318,8 @@ insert(char *s, int checked)
>      char *news = s;
>      int statted = 0;
>  
> +    DPUTS(!s || !*s, "BUG: adding empty string as glob match");
> +
>      queue_signals();
>      inserts = NULL;
>  
> 
> The problem seems to be that scanner() calls recursively one extra
> time when looking at "notadirectory/".

(You can easily see this in with the above change with "print */" in a
directory with some non-directory files.)

I don't see any evidence that's either new or a bug --- it's just the
rather icky way "*/" works.  The scanner doesn't know in advance the
"/" is the end, so it attempts to go down to the next level (hence the
recursion), where it adds an empty string as it has nothing else to
add.  Later it (actually, I think, a library test) sees the "/" is
at the end and prunes non-directory files so you get the right answer.

This isn't actually specific to this case.  Just to show this up (it's
not useful for any other purpose), try the patch at the bottom and then
e.g. (if you're in the Src directory)

print */zle_tricky.c

You'll see lots of messages like

Recursing to add zle_tricky.c to zsh.ico/
Recursing to add zle_tricky.c to options.c/
Recursing to add zle_tricky.c to subst.c/
Recursing to add zle_tricky.c to signals.pro/

plus the right answer at the end, Zle/zle_tricky.c.

If you missed out zle_tricky.c you'd get your case --- it's
got to the "/" and discovered there's no glob after it, so it simply
adds the string and sees if the file exists.  If it's not a directory,
the file doesn't exist.  The only difference in the case of a trailing
"/" is that because you've only added a null string, then the file
"foo/" exists if and only if foo is a directory --- as long as the
system test works like that, which is done in statfullpath(), which I
didn't get around to walking through so if you want to be quite sure you
can go do that.

I expect you could truncate it earlier with the right test --- I
certainly hadn't realised/remembered this is how a trailing "/" worked,
I'd assumed it was internally treated as (/).

pws

diff --git a/Src/glob.c b/Src/glob.c
index 2051016..add1406 100644
--- a/Src/glob.c
+++ b/Src/glob.c
@@ -463,6 +463,8 @@ insert(char *s, int checked)
  * with successive bits of the path until we've    *
  * tried all of it.                                */
 
+static int recursing = -1;
+
 /**/
 static void
 scanner(Complist q, int shortcircuit)
@@ -477,14 +479,18 @@ scanner(Complist q, int shortcircuit)
 	return;
     init_dirsav(&ds);
 
+    recursing++;
+
     if ((closure = q->closure)) {
 	/* (foo/)# - match zero or more dirs */
 	if (q->closure == 2)	/* (foo/)## - match one or more dirs */
 	    q->closure = 1;
 	else {
 	    scanner(q->next, shortcircuit);
-	    if (shortcircuit && shortcircuit == matchct)
+	    if (shortcircuit && shortcircuit == matchct) {
+		recursing--;
 		return;
+	    }
 	}
     }
     p = q->pat;
@@ -499,13 +505,18 @@ scanner(Complist q, int shortcircuit)
 	if (l + !l + pathpos - pathbufcwd >= PATH_MAX) {
 	    int err;
 
-	    if (l >= PATH_MAX)
+	    if (l >= PATH_MAX) {
+		recursing--;
 		return;
+	    }
 	    err = lchdir(unmeta(pathbuf + pathbufcwd), &ds, 0);
-	    if (err == -1)
+	    if (err == -1) {
+		recursing--;
 		return;
+	    }
 	    if (err) {
 		zerr("current directory lost during glob");
+		recursing--;
 		return;
 	    }
 	    pathbufcwd = pathpos;
@@ -532,8 +543,10 @@ scanner(Complist q, int shortcircuit)
 		    addpath(str, l);
 		    if (!closure || !statfullpath("", NULL, 1)) {
 			scanner((q->closure) ? q : q->next, shortcircuit);
-			if (shortcircuit && shortcircuit == matchct)
+			if (shortcircuit && shortcircuit == matchct) {
+			    recursing--;
 			    return;
+			}
 		    }
 		    pathbuf[pathpos = oppos] = '\0';
 		}
@@ -541,9 +554,13 @@ scanner(Complist q, int shortcircuit)
 	} else {
 	    if (str[l])
 		str = dupstrpfx(str, l);
+	    if (recursing)
+		fprintf(stderr, "Recursing to add %s to %s\n", str, pathbuf);
 	    insert(str, 0);
-	    if (shortcircuit && shortcircuit == matchct)
+	    if (shortcircuit && shortcircuit == matchct) {
+		recursing--;
 		return;
+	    }
 	}
     } else {
 	/* Do pattern matching on current path section. */
@@ -553,8 +570,10 @@ scanner(Complist q, int shortcircuit)
 	char *subdirs = NULL;
 	int subdirlen = 0;
 
-	if (lock == NULL)
+	if (lock == NULL) {
+	    recursing--;
 	    return;
+	}
 	while ((fn = zreaddir(lock, 1)) && !errflag) {
 	    /* prefix and suffix are zle trickery */
 	    if (!dirs && !colonmod &&
@@ -636,6 +655,7 @@ scanner(Complist q, int shortcircuit)
 		    insert(fn, 1);
 		    if (shortcircuit && shortcircuit == matchct) {
 			closedir(lock);
+			recursing--;
 			return;
 		    }
 		}
@@ -653,8 +673,10 @@ scanner(Complist q, int shortcircuit)
 		fn += sizeof(int);
 		/* scan next level */
 		scanner((q->closure) ? q : q->next, shortcircuit); 
-		if (shortcircuit && shortcircuit == matchct)
+		if (shortcircuit && shortcircuit == matchct) {
+		    recursing--;
 		    return;
+		}
 		pathbuf[pathpos = oppos] = '\0';
 	    }
 	    hrealloc(subdirs, subdirlen, 0);
@@ -668,6 +690,7 @@ scanner(Complist q, int shortcircuit)
 	    close(ds.dirfd);
 	pathbufcwd = pbcwdsav;
     }
+    recursing--;
     return;
 }
 

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Jun T.]

On 2016/07/11, at 5:12, Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm not able to reproduce this. 
(snip)
> macadamia% ~-/Src/zsh -f script1
> macadamia% print -l */
> zsh: no matches found: */

Please try running 'script1' and 'print -l */' in the same shell.
For example:
% zsh -f -c './script1; print -l */'

On 2016/07/11, at 13:58, Bart Schaefer <schaefer@brasslantern.com> wrote:
> The problem seems to be that scanner() calls recursively one extra
> time when looking at "notadirectory/".

> In that case statfullpath("") is called and, for reasons I haven't
> figured out, that returns success on MacOS when called in exactly
> the circumstances that Zhiming describes.

Near the end of statfullpath() (line 301 of glob.c), the access(2) system
call is used to check whether the path is actually a directory:
	access(buf, F_OK)
where buf is set to 'script1/.'; but it seems access(2) behaves quite
strangely on OS X.
On OS X 10.8.5 (Mountain Lion, a rather old version) I get the following
(note that no zsh is involved here):

bash$ cat atest.c
#include <unistd.h>
#include <stdio.h>
int main(int argc, char* argv[]) {
    if(argc==2) printf("%d\n",access(argv[1], F_OK));
    return 0;
}
bash$ cc -o atest atest.c
bash$ echo '#!/bogus/command' > foo	# /bogus/command does not exist
bash$ ./atest foo/.
-1                              # as expected
bash$ ./foo
bash: ./foo: Permission denied
bash$ ./atest foo/.
-1                              # still fails
bash$ chmod u+x foo
bash$ ./foo
bash: ./foo: /bogus/command: bad interpreter: No such file or directory
bash$ ./atest foo/.
0                               # now it succeeds
bash$ 

I will test on El Capitan later, but I guess I'll get the same result
since Zhiming is having the problem on El Capitan (or even 10.12 beta).

Jun

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Bart Schaefer]

On Jul 11,  5:51pm, Jun T. wrote:
}
} Near the end of statfullpath() (line 301 of glob.c), the access(2) system
} call is used to check whether the path is actually a directory:
} 	access(buf, F_OK)
} where buf is set to 'script1/.'; but it seems access(2) behaves quite
} strangely on OS X.
} On OS X 10.8.5 (Mountain Lion, a rather old version) I get the following

This is once again sounding vaguely familiar, as in, I think we ran
into problems with access() before.

What's curious is that this doesn't show up in zsh-5.0.8 as shipped
stock with El Capitan even though you find it in older versions of
OSX.  I grepped around and scanned changelogs looking for something
that may have changed in the configure-discovered compiler settings
but did not see anything that appeared related.

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Jun T.]

I tested on Mavericks (10.9.5) and El Capitan (10.11.5).

On OS X 10.9.5, the behavior is the same as 10.8.

But on 10.11.5, I can not reproduce the strange behavior of
access(2) if I use bash or /bin/zsh (5.0.8, built by Apple).
The problem exists only if I run the script and atest from
/usr/local/bin/zsh (git HEAD, local build).

The script itself (foo in my case) need not be a zsh script.
It can be
#!/bogus/command
to reproduce the problem.

I have no idea where the information that the script foo
has been run is "cached", why it can affect the result
of a system call.

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Jun T.]

Sorry, it is not necessary to run the script and atest (access(2)) from
within the same shell. Running the script by /usr/local/bin/zsh
is enough to 'cache' the info.

On El Captain:
bash$ ./atest foo/.
-1
bash$ /usr/local/bin/zsh -c ./foo
zsh:1: ./foo: bad interpreter: /bogus/command: no such file or directory
bash$ ./atest foo/.
0
bash$

Re: [BUG] Directory glob picks up running or already-run scripts on OS X

[Bart Schaefer]

On Jul 12,  4:18am, Jun T. wrote:
}
} I have no idea where the information that the script foo
} has been run is "cached", why it can affect the result
} of a system call.

Yes, this seems like something that should be escalated to Apple.  Is
there anyone on the list who can do that?