* [BUG] Directory glob picks up running or already-run scripts on OS X @ 2016-07-09 3:55 Zhiming Wang 2016-07-10 20:12 ` Bart Schaefer 0 siblings, 1 reply; 11+ messages in thread From: Zhiming Wang @ 2016-07-09 3:55 UTC (permalink / raw) To: zsh-workers With zsh 5.2 or master running on OS X 10.11.5 or macOS 10.12 PB1, a glob for directories could pick up scripts that are either running or have been run. Note that this seems to only apply to scripts, not random executables (e.g., an executable created by a C compiler). Here's a session that reproduces the issue: $ cd $(mktemp -d) $ print '#!/usr/bin/env zsh' >script1 $ print $'#!/usr/bin/env zsh\nprint -l */' >script2 $ chmod +x script1 script2 $ print -l */ zsh: no matches found: */ $ ./script1 $ print -l */ script1/ $ ./script2 script1/ script2/ $ print -l */ script1/ script2/ $ print $'#include <stdio.h>\nint main() { printf("hello, world"); }' >hello.c $ cc -o hello hello.c $ ./hello hello, world% $ print -l */ script1/ script2/ I cannot reproduce this issue with zsh 5.0.8 on OS X, so something went wrong in between. Unfortunately, due to some limitations I cannot test other versions right now. I also cannot repoduce this issue on Linux with any zsh version (including 5.2 or master), so this seems to be an OS X specific issue. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-09 3:55 [BUG] Directory glob picks up running or already-run scripts on OS X Zhiming Wang @ 2016-07-10 20:12 ` Bart Schaefer [not found] ` <B17666ED-6D14-477C-B45D-3B7F064B5F24@gmail.com> 2016-07-11 8:51 ` Jun T. 0 siblings, 2 replies; 11+ messages in thread From: Bart Schaefer @ 2016-07-10 20:12 UTC (permalink / raw) To: Zhiming Wang; +Cc: Zsh hackers list On Fri, Jul 8, 2016 at 8:55 PM, Zhiming Wang <zmwangx@gmail.com> wrote: > With zsh 5.2 or master running on OS X 10.11.5 or macOS 10.12 PB1, a glob for > directories could pick up scripts that are either running or have been > run. I'm not able to reproduce this. I happen to have an iMac that was upgraded to El Capitan (10.11.5) only yesterday. This morning I did the following: 1. Install xcode and agree to the license. 2. Download automake and autoconf from ftp.gnu.org, build, install. 3. Pull the latest zsh from sourceforge git. 4. Recompile zsh with static module linkage so I can run from the build tree 5. Run: Src/zsh -f 6. Attempt to reproduce using your steps, modified a little because "env zsh" won't find the binary I want. macadamia% cd $(mktemp -d) macadamia% print $'#!/usr/bin/env zsh' >script1 macadamia% print $'#!/usr/bin/env zsh\nprint -l */' >script2 macadamia% chmod +x script1 script2 macadamia% print -l */ zsh: no matches found: */ macadamia% ~-/Src/zsh -f script1 macadamia% print -l */ zsh: no matches found: */ macadamia% ~-/Src/zsh -f script2 script2:2: no matches found: */ macadamia% uname -a Darwin macadamia.local 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64 macadamia% print $ZSH_PATCHLEVEL zsh-5.2-310-g304aa25 > I cannot reproduce this issue with zsh 5.0.8 on OS X, so something went wrong > in between. Unfortunately, due to some limitations I cannot test other versions > right now. I also cannot repoduce this issue on Linux with any zsh version > (including 5.2 or master), so this seems to be an OS X specific issue. This may even be a compiler-specific issue. Note that I'm deliberately using the stock compiler with neither homebrew nor macports and minimal GNU tool install. What did you use to build zsh? ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <B17666ED-6D14-477C-B45D-3B7F064B5F24@gmail.com>]
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X [not found] ` <B17666ED-6D14-477C-B45D-3B7F064B5F24@gmail.com> @ 2016-07-11 0:38 ` Bart Schaefer 2016-07-11 0:48 ` Zhiming Wang 0 siblings, 1 reply; 11+ messages in thread From: Bart Schaefer @ 2016-07-11 0:38 UTC (permalink / raw) To: Zhiming Wang; +Cc: Zsh hackers list On Sun, Jul 10, 2016 at 5:31 PM, Zhiming Wang <zmwangx@gmail.com> wrote: > > However, the issue does not only show up on custom builds. It also affects > Apple's stock build on macOS 10.12 Public Beta 1 (build 16A238m) This is suddenly sounding vaguely familiar. Does the problem disappear if you build with compiler optimizations disabled? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 0:38 ` Bart Schaefer @ 2016-07-11 0:48 ` Zhiming Wang 2016-07-11 4:58 ` Bart Schaefer 0 siblings, 1 reply; 11+ messages in thread From: Zhiming Wang @ 2016-07-11 0:48 UTC (permalink / raw) To: Bart Schaefer; +Cc: zsh-workers > On Jul 11, 2016, at 8:38 AM, Bart Schaefer <schaefer@brasslantern.com> wrote: > > On Sun, Jul 10, 2016 at 5:31 PM, Zhiming Wang <zmwangx@gmail.com> wrote: > > This is suddenly sounding vaguely familiar. Does the problem > disappear if you build with compiler optimizations disabled? It doesn't. This time I compiled zsh master (304aa25) manually: $ Util/preconfig $ ./configure CFLAGS='-g -O0' ... zsh configuration ----------------- zsh version : 5.2-dev-1 host operating system : x86_64-apple-darwin15.5.0 source code location : . compiler : gcc preprocessor flags : executable compiler flags : -g -O0 module compiler flags : -g -O0 -fno-common executable linker flags : -Wl,-x -rdynamic module linker flags : -Wl,-x -bundle -flat_namespace -undefined suppress library flags : -lgdbm -liconv -ldl -ltermcap -lm -lc installation basename : zsh binary install path : /usr/local/bin man page install path : /usr/local/share/man info install path : /usr/local/share/info functions install path : /usr/local/share/zsh/5.2-dev-1/functions ... $ make Of course here gcc is actually clang. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 0:48 ` Zhiming Wang @ 2016-07-11 4:58 ` Bart Schaefer 2016-07-11 10:08 ` Peter Stephenson 0 siblings, 1 reply; 11+ messages in thread From: Bart Schaefer @ 2016-07-11 4:58 UTC (permalink / raw) To: zsh-workers I'm not sure whether this was introduced when adding the (Y) qualifier, or if it was already lurking. With the following DPUTS() added and zsh compiled with debugging, Test/D02glob.ztst will fail (on any platform). diff --git a/Src/glob.c b/Src/glob.c index 2051016..5260b63 100644 --- a/Src/glob.c +++ b/Src/glob.c @@ -318,6 +318,8 @@ insert(char *s, int checked) char *news = s; int statted = 0; + DPUTS(!s || !*s, "BUG: adding empty string as glob match"); + queue_signals(); inserts = NULL; The problem seems to be that scanner() calls recursively one extra time when looking at "notadirectory/". In that case statfullpath("") is called and, for reasons I haven't figured out, that returns success on MacOS when called in exactly the circumstances that Zhiming describes. This will eventually correct itself, which leads me to suspect it has something to do with re-using the global static "pathbuf". The simple fix would be for insert() to be a no-op when called on the empty string, but it might be useful to examine scanner() in more detail to try to avoid the extra recursion. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 4:58 ` Bart Schaefer @ 2016-07-11 10:08 ` Peter Stephenson 0 siblings, 0 replies; 11+ messages in thread From: Peter Stephenson @ 2016-07-11 10:08 UTC (permalink / raw) To: zsh-workers On Sun, 10 Jul 2016 21:58:21 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > I'm not sure whether this was introduced when adding the (Y) qualifier, > or if it was already lurking. With the following DPUTS() added and zsh > compiled with debugging, Test/D02glob.ztst will fail (on any platform). > > diff --git a/Src/glob.c b/Src/glob.c > index 2051016..5260b63 100644 > --- a/Src/glob.c > +++ b/Src/glob.c > @@ -318,6 +318,8 @@ insert(char *s, int checked) > char *news = s; > int statted = 0; > > + DPUTS(!s || !*s, "BUG: adding empty string as glob match"); > + > queue_signals(); > inserts = NULL; > > > The problem seems to be that scanner() calls recursively one extra > time when looking at "notadirectory/". (You can easily see this in with the above change with "print */" in a directory with some non-directory files.) I don't see any evidence that's either new or a bug --- it's just the rather icky way "*/" works. The scanner doesn't know in advance the "/" is the end, so it attempts to go down to the next level (hence the recursion), where it adds an empty string as it has nothing else to add. Later it (actually, I think, a library test) sees the "/" is at the end and prunes non-directory files so you get the right answer. This isn't actually specific to this case. Just to show this up (it's not useful for any other purpose), try the patch at the bottom and then e.g. (if you're in the Src directory) print */zle_tricky.c You'll see lots of messages like Recursing to add zle_tricky.c to zsh.ico/ Recursing to add zle_tricky.c to options.c/ Recursing to add zle_tricky.c to subst.c/ Recursing to add zle_tricky.c to signals.pro/ plus the right answer at the end, Zle/zle_tricky.c. If you missed out zle_tricky.c you'd get your case --- it's got to the "/" and discovered there's no glob after it, so it simply adds the string and sees if the file exists. If it's not a directory, the file doesn't exist. The only difference in the case of a trailing "/" is that because you've only added a null string, then the file "foo/" exists if and only if foo is a directory --- as long as the system test works like that, which is done in statfullpath(), which I didn't get around to walking through so if you want to be quite sure you can go do that. I expect you could truncate it earlier with the right test --- I certainly hadn't realised/remembered this is how a trailing "/" worked, I'd assumed it was internally treated as (/). pws diff --git a/Src/glob.c b/Src/glob.c index 2051016..add1406 100644 --- a/Src/glob.c +++ b/Src/glob.c @@ -463,6 +463,8 @@ insert(char *s, int checked) * with successive bits of the path until we've * * tried all of it. */ +static int recursing = -1; + /**/ static void scanner(Complist q, int shortcircuit) @@ -477,14 +479,18 @@ scanner(Complist q, int shortcircuit) return; init_dirsav(&ds); + recursing++; + if ((closure = q->closure)) { /* (foo/)# - match zero or more dirs */ if (q->closure == 2) /* (foo/)## - match one or more dirs */ q->closure = 1; else { scanner(q->next, shortcircuit); - if (shortcircuit && shortcircuit == matchct) + if (shortcircuit && shortcircuit == matchct) { + recursing--; return; + } } } p = q->pat; @@ -499,13 +505,18 @@ scanner(Complist q, int shortcircuit) if (l + !l + pathpos - pathbufcwd >= PATH_MAX) { int err; - if (l >= PATH_MAX) + if (l >= PATH_MAX) { + recursing--; return; + } err = lchdir(unmeta(pathbuf + pathbufcwd), &ds, 0); - if (err == -1) + if (err == -1) { + recursing--; return; + } if (err) { zerr("current directory lost during glob"); + recursing--; return; } pathbufcwd = pathpos; @@ -532,8 +543,10 @@ scanner(Complist q, int shortcircuit) addpath(str, l); if (!closure || !statfullpath("", NULL, 1)) { scanner((q->closure) ? q : q->next, shortcircuit); - if (shortcircuit && shortcircuit == matchct) + if (shortcircuit && shortcircuit == matchct) { + recursing--; return; + } } pathbuf[pathpos = oppos] = '\0'; } @@ -541,9 +554,13 @@ scanner(Complist q, int shortcircuit) } else { if (str[l]) str = dupstrpfx(str, l); + if (recursing) + fprintf(stderr, "Recursing to add %s to %s\n", str, pathbuf); insert(str, 0); - if (shortcircuit && shortcircuit == matchct) + if (shortcircuit && shortcircuit == matchct) { + recursing--; return; + } } } else { /* Do pattern matching on current path section. */ @@ -553,8 +570,10 @@ scanner(Complist q, int shortcircuit) char *subdirs = NULL; int subdirlen = 0; - if (lock == NULL) + if (lock == NULL) { + recursing--; return; + } while ((fn = zreaddir(lock, 1)) && !errflag) { /* prefix and suffix are zle trickery */ if (!dirs && !colonmod && @@ -636,6 +655,7 @@ scanner(Complist q, int shortcircuit) insert(fn, 1); if (shortcircuit && shortcircuit == matchct) { closedir(lock); + recursing--; return; } } @@ -653,8 +673,10 @@ scanner(Complist q, int shortcircuit) fn += sizeof(int); /* scan next level */ scanner((q->closure) ? q : q->next, shortcircuit); - if (shortcircuit && shortcircuit == matchct) + if (shortcircuit && shortcircuit == matchct) { + recursing--; return; + } pathbuf[pathpos = oppos] = '\0'; } hrealloc(subdirs, subdirlen, 0); @@ -668,6 +690,7 @@ scanner(Complist q, int shortcircuit) close(ds.dirfd); pathbufcwd = pbcwdsav; } + recursing--; return; } ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-10 20:12 ` Bart Schaefer [not found] ` <B17666ED-6D14-477C-B45D-3B7F064B5F24@gmail.com> @ 2016-07-11 8:51 ` Jun T. 2016-07-11 16:59 ` Bart Schaefer 1 sibling, 1 reply; 11+ messages in thread From: Jun T. @ 2016-07-11 8:51 UTC (permalink / raw) To: zsh-workers On 2016/07/11, at 5:12, Bart Schaefer <schaefer@brasslantern.com> wrote: > I'm not able to reproduce this. (snip) > macadamia% ~-/Src/zsh -f script1 > macadamia% print -l */ > zsh: no matches found: */ Please try running 'script1' and 'print -l */' in the same shell. For example: % zsh -f -c './script1; print -l */' On 2016/07/11, at 13:58, Bart Schaefer <schaefer@brasslantern.com> wrote: > The problem seems to be that scanner() calls recursively one extra > time when looking at "notadirectory/". > In that case statfullpath("") is called and, for reasons I haven't > figured out, that returns success on MacOS when called in exactly > the circumstances that Zhiming describes. Near the end of statfullpath() (line 301 of glob.c), the access(2) system call is used to check whether the path is actually a directory: access(buf, F_OK) where buf is set to 'script1/.'; but it seems access(2) behaves quite strangely on OS X. On OS X 10.8.5 (Mountain Lion, a rather old version) I get the following (note that no zsh is involved here): bash$ cat atest.c #include <unistd.h> #include <stdio.h> int main(int argc, char* argv[]) { if(argc==2) printf("%d\n",access(argv[1], F_OK)); return 0; } bash$ cc -o atest atest.c bash$ echo '#!/bogus/command' > foo # /bogus/command does not exist bash$ ./atest foo/. -1 # as expected bash$ ./foo bash: ./foo: Permission denied bash$ ./atest foo/. -1 # still fails bash$ chmod u+x foo bash$ ./foo bash: ./foo: /bogus/command: bad interpreter: No such file or directory bash$ ./atest foo/. 0 # now it succeeds bash$ I will test on El Capitan later, but I guess I'll get the same result since Zhiming is having the problem on El Capitan (or even 10.12 beta). Jun ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 8:51 ` Jun T. @ 2016-07-11 16:59 ` Bart Schaefer 2016-07-11 19:18 ` Jun T. 0 siblings, 1 reply; 11+ messages in thread From: Bart Schaefer @ 2016-07-11 16:59 UTC (permalink / raw) To: zsh-workers On Jul 11, 5:51pm, Jun T. wrote: } } Near the end of statfullpath() (line 301 of glob.c), the access(2) system } call is used to check whether the path is actually a directory: } access(buf, F_OK) } where buf is set to 'script1/.'; but it seems access(2) behaves quite } strangely on OS X. } On OS X 10.8.5 (Mountain Lion, a rather old version) I get the following This is once again sounding vaguely familiar, as in, I think we ran into problems with access() before. What's curious is that this doesn't show up in zsh-5.0.8 as shipped stock with El Capitan even though you find it in older versions of OSX. I grepped around and scanned changelogs looking for something that may have changed in the configure-discovered compiler settings but did not see anything that appeared related. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 16:59 ` Bart Schaefer @ 2016-07-11 19:18 ` Jun T. 2016-07-11 20:08 ` Jun T. 2016-07-14 5:27 ` Bart Schaefer 0 siblings, 2 replies; 11+ messages in thread From: Jun T. @ 2016-07-11 19:18 UTC (permalink / raw) To: zsh-workers I tested on Mavericks (10.9.5) and El Capitan (10.11.5). On OS X 10.9.5, the behavior is the same as 10.8. But on 10.11.5, I can not reproduce the strange behavior of access(2) if I use bash or /bin/zsh (5.0.8, built by Apple). The problem exists only if I run the script and atest from /usr/local/bin/zsh (git HEAD, local build). The script itself (foo in my case) need not be a zsh script. It can be #!/bogus/command to reproduce the problem. I have no idea where the information that the script foo has been run is "cached", why it can affect the result of a system call. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 19:18 ` Jun T. @ 2016-07-11 20:08 ` Jun T. 2016-07-14 5:27 ` Bart Schaefer 1 sibling, 0 replies; 11+ messages in thread From: Jun T. @ 2016-07-11 20:08 UTC (permalink / raw) To: zsh-workers Sorry, it is not necessary to run the script and atest (access(2)) from within the same shell. Running the script by /usr/local/bin/zsh is enough to 'cache' the info. On El Captain: bash$ ./atest foo/. -1 bash$ /usr/local/bin/zsh -c ./foo zsh:1: ./foo: bad interpreter: /bogus/command: no such file or directory bash$ ./atest foo/. 0 bash$ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] Directory glob picks up running or already-run scripts on OS X 2016-07-11 19:18 ` Jun T. 2016-07-11 20:08 ` Jun T. @ 2016-07-14 5:27 ` Bart Schaefer 1 sibling, 0 replies; 11+ messages in thread From: Bart Schaefer @ 2016-07-14 5:27 UTC (permalink / raw) To: zsh-workers On Jul 12, 4:18am, Jun T. wrote: } } I have no idea where the information that the script foo } has been run is "cached", why it can affect the result } of a system call. Yes, this seems like something that should be escalated to Apple. Is there anyone on the list who can do that? ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2016-07-14 5:27 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-07-09 3:55 [BUG] Directory glob picks up running or already-run scripts on OS X Zhiming Wang 2016-07-10 20:12 ` Bart Schaefer [not found] ` <B17666ED-6D14-477C-B45D-3B7F064B5F24@gmail.com> 2016-07-11 0:38 ` Bart Schaefer 2016-07-11 0:48 ` Zhiming Wang 2016-07-11 4:58 ` Bart Schaefer 2016-07-11 10:08 ` Peter Stephenson 2016-07-11 8:51 ` Jun T. 2016-07-11 16:59 ` Bart Schaefer 2016-07-11 19:18 ` Jun T. 2016-07-11 20:08 ` Jun T. 2016-07-14 5:27 ` Bart Schaefer
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).