/proc slows % echo /**/*/etc/r*(/N)

/proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

I used to do this kind of thing without any problems:

     % echo /**/*/etc/r*(/N)

... but just trying it, zsh labors for close to an hour, then spits out 
countless pages of:

/proc/9/root/etc/rc0.d /proc/9/root/etc/rc1.d /proc/9/root/etc/rc2.d 
/proc/9/root/etc/rc3.d /proc/9/root/etc/rc4.d /proc/9/root/etc/rc5.d 
/proc/9/root/etc/rc6.d /proc/9/root/etc/rcS.d /proc/9/root/etc/rsyslog.d 
/proc/9/task/9/cwd/etc/rc0.d /proc/9/task/9/cwd/etc/rc1.d 
/proc/9/task/9/cwd/etc/rc2.d /proc/9/task/9/cwd/etc/rc3.d 
/proc/9/task/9/cwd/etc/rc4.d /proc/9/task/9/cwd/etc/rc5.d 
/proc/9/task/9/cwd/etc/rc6.d /proc/9/task/9/cwd/etc/rcS.d 
/proc/9/task/9/cwd/etc/rsyslog.d /proc/9/task/9/root/etc/rc0.d 
/proc/9/task/9/root/etc/rc1.d /proc/9/task/9/root/etc/rc2.d 
/proc/9/task/9/root/etc/rc3.d /proc/9/task/9/root/etc/rc4.d 
/proc/9/task/9/root/etc/rc5.d /proc/9/task/9/root/etc/rc6.d 
/proc/9/task/9/root/etc/rcS.d /proc/9/task/9/root/etc/rsyslog.d

... I'm wondering why /proc wasn't a problem previously and if there's 
anything I can do to filter it out of the search -- probably all the 
Linux 'system' directories too.  I recall that there's some test, 
something in permissions, some flag -- that identifies these 
not-quite-real directories -- zsh knows who they are.  They're really in 
RAM, yes?  I know the tilde is used to make exceptions:

     for bb in *.eml~save*; do

... all files matching '*.eml' excepting those that match 'save*' ... 
but I can't get the syntax correct with the above.

     % echo /**/*/etc/r*~/proc*(/N)

     % echo /**/*/etc/r*(/N)~/proc*

... were busts.  But the right answer is to filter out all those virtual 
directories.  Don't know why this didn't used to be a problem.  But I 
think I did deal with this issue a few years ago, too.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Mark J. Reed]

[-- Attachment #1: Type: text/plain, Size: 2322 bytes --]

/^proc. will match everything except /proc, so you can easily look for
files like /^proc/**/whatever.

But on my system which has no /proc, `**` is just as much a culprit, since
that does a recursive search throughout the entire file system. For
instance, /**/etc matches /`foo/bar/baz/zoo/wicky/etc` and so on...

On Sun, Jan 21, 2024 at 12:20 PM Ray Andrews <rayandrews@eastlink.ca> wrote:

> I used to do this kind of thing without any problems:
>
>      % echo /**/*/etc/r*(/N)
>
> ... but just trying it, zsh labors for close to an hour, then spits out
> countless pages of:
>
> /proc/9/root/etc/rc0.d /proc/9/root/etc/rc1.d /proc/9/root/etc/rc2.d
> /proc/9/root/etc/rc3.d /proc/9/root/etc/rc4.d /proc/9/root/etc/rc5.d
> /proc/9/root/etc/rc6.d /proc/9/root/etc/rcS.d /proc/9/root/etc/rsyslog.d
> /proc/9/task/9/cwd/etc/rc0.d /proc/9/task/9/cwd/etc/rc1.d
> /proc/9/task/9/cwd/etc/rc2.d /proc/9/task/9/cwd/etc/rc3.d
> /proc/9/task/9/cwd/etc/rc4.d /proc/9/task/9/cwd/etc/rc5.d
> /proc/9/task/9/cwd/etc/rc6.d /proc/9/task/9/cwd/etc/rcS.d
> /proc/9/task/9/cwd/etc/rsyslog.d /proc/9/task/9/root/etc/rc0.d
> /proc/9/task/9/root/etc/rc1.d /proc/9/task/9/root/etc/rc2.d
> /proc/9/task/9/root/etc/rc3.d /proc/9/task/9/root/etc/rc4.d
> /proc/9/task/9/root/etc/rc5.d /proc/9/task/9/root/etc/rc6.d
> /proc/9/task/9/root/etc/rcS.d /proc/9/task/9/root/etc/rsyslog.d
>
> ... I'm wondering why /proc wasn't a problem previously and if there's
> anything I can do to filter it out of the search -- probably all the
> Linux 'system' directories too.  I recall that there's some test,
> something in permissions, some flag -- that identifies these
> not-quite-real directories -- zsh knows who they are.  They're really in
> RAM, yes?  I know the tilde is used to make exceptions:
>
>      for bb in *.eml~save*; do
>
> ... all files matching '*.eml' excepting those that match 'save*' ...
> but I can't get the syntax correct with the above.
>
>      % echo /**/*/etc/r*~/proc*(/N)
>
>      % echo /**/*/etc/r*(/N)~/proc*
>
> ... were busts.  But the right answer is to filter out all those virtual
> directories.  Don't know why this didn't used to be a problem.  But I
> think I did deal with this issue a few years ago, too.
>
>
>
>
>

-- 
Mark J. Reed <markjreed@gmail.com>

[-- Attachment #2: Type: text/html, Size: 3004 bytes --]

Re: /proc slows % echo /**/*/etc/r*(/N)

[Mark J. Reed]

[-- Attachment #1: Type: text/plain, Size: 2902 bytes --]

Gah, extraneous punctuation marks.  To reiterate, this pattern:

    /^proc

will match everything in the root directory except /proc. And this one:

    /**/etc

will match a directory named "etc" no matter how deep in the file system it
is, such as /foo/bar/baz/zoo/wicky/etc.  Zsh basically has to enumerate
every file on your disk drive to figure out what matches that.

On Sun, Jan 21, 2024 at 1:00 PM Mark J. Reed <markjreed@gmail.com> wrote:

> /^proc. will match everything except /proc, so you can easily look for
> files like /^proc/**/whatever.
>
> But on my system which has no /proc, `**` is just as much a culprit, since
> that does a recursive search throughout the entire file system. For
> instance, /**/etc matches /`foo/bar/baz/zoo/wicky/etc` and so on...
>
> On Sun, Jan 21, 2024 at 12:20 PM Ray Andrews <rayandrews@eastlink.ca>
> wrote:
>
>> I used to do this kind of thing without any problems:
>>
>>      % echo /**/*/etc/r*(/N)
>>
>> ... but just trying it, zsh labors for close to an hour, then spits out
>> countless pages of:
>>
>> /proc/9/root/etc/rc0.d /proc/9/root/etc/rc1.d /proc/9/root/etc/rc2.d
>> /proc/9/root/etc/rc3.d /proc/9/root/etc/rc4.d /proc/9/root/etc/rc5.d
>> /proc/9/root/etc/rc6.d /proc/9/root/etc/rcS.d /proc/9/root/etc/rsyslog.d
>> /proc/9/task/9/cwd/etc/rc0.d /proc/9/task/9/cwd/etc/rc1.d
>> /proc/9/task/9/cwd/etc/rc2.d /proc/9/task/9/cwd/etc/rc3.d
>> /proc/9/task/9/cwd/etc/rc4.d /proc/9/task/9/cwd/etc/rc5.d
>> /proc/9/task/9/cwd/etc/rc6.d /proc/9/task/9/cwd/etc/rcS.d
>> /proc/9/task/9/cwd/etc/rsyslog.d /proc/9/task/9/root/etc/rc0.d
>> /proc/9/task/9/root/etc/rc1.d /proc/9/task/9/root/etc/rc2.d
>> /proc/9/task/9/root/etc/rc3.d /proc/9/task/9/root/etc/rc4.d
>> /proc/9/task/9/root/etc/rc5.d /proc/9/task/9/root/etc/rc6.d
>> /proc/9/task/9/root/etc/rcS.d /proc/9/task/9/root/etc/rsyslog.d
>>
>> ... I'm wondering why /proc wasn't a problem previously and if there's
>> anything I can do to filter it out of the search -- probably all the
>> Linux 'system' directories too.  I recall that there's some test,
>> something in permissions, some flag -- that identifies these
>> not-quite-real directories -- zsh knows who they are.  They're really in
>> RAM, yes?  I know the tilde is used to make exceptions:
>>
>>      for bb in *.eml~save*; do
>>
>> ... all files matching '*.eml' excepting those that match 'save*' ...
>> but I can't get the syntax correct with the above.
>>
>>      % echo /**/*/etc/r*~/proc*(/N)
>>
>>      % echo /**/*/etc/r*(/N)~/proc*
>>
>> ... were busts.  But the right answer is to filter out all those virtual
>> directories.  Don't know why this didn't used to be a problem.  But I
>> think I did deal with this issue a few years ago, too.
>>
>>
>>
>>
>>
>
> --
> Mark J. Reed <markjreed@gmail.com>
>


-- 
Mark J. Reed <markjreed@gmail.com>

[-- Attachment #2: Type: text/html, Size: 4092 bytes --]

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

On 2024-01-21 10:02, Mark J. Reed wrote:
> Gah, extraneous punctuation marks.  To reiterate, this pattern:
>
>     /^proc

But I'm holding out for a way of filtering out all those virtual 
directories.  There's a few of them.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Bart Schaefer]

On Sun, Jan 21, 2024 at 9:19 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I used to do this kind of thing without any problems:
>
>      % echo /**/*/etc/r*(/N)
>
> ... but just trying it, zsh labors for close to an hour, then spits out
> countless pages of:
>
> /proc/9/root/etc/rc0.d /proc/9/root/etc/rc1.d /proc/9/root/etc/rc2.d

Why /**/*/ rather than just /**/ ?  The extra /*/ there will expand
symbolic links, which are normally skipped by /**/.  It's not
otherwise adding anything to the traversal.

/proc/9 is for PID 9 and /proc/9/root is a symbolic link to the root
directory for that process, so by expanding that you're traversing the
whole filesystem again, which takes you back to /proc/ again, and so
on for every PID until zsh gives up (probably because it can't
allocate any more memory).

> ... I'm wondering why /proc wasn't a problem previously

I suspect previously you weren't adding that extra /*/.  What do you
mean for that to accomplish?

> and if there's
> anything I can do to filter it out of the search -- probably all the
> Linux 'system' directories too.  I recall that there's some test,
> something in permissions, some flag -- that identifies these
> not-quite-real directories -- zsh knows who they are.

There's no glob qualifier for that, if that's what you mean.  As it
happens, those directories always appear to be empty, so you can find
them with /*(/L0) but you can't use glob qualifiers in the middle of a
longer pattern (only at the end) so that won't help with your
recursive search.

> I know the tilde is used to make exceptions:
> but I can't get the syntax correct with the above.

That syntax is applied as a string match after the glob has already
expanded, so putting it at the end like this ...

>      % echo /**/*/etc/r*~/proc*(/N)

... is just going to waste even more time.

As Mark suggested, there are exclusion patterns you can apply, but
/**/ is still going to search the whole filesystem at least once,
possibly crossing mount points into other filesystems if you have
anything NFS or similarly remote mounted.

Without trying it, which I'm not going to do, I think the answer to
your specific question might be:

echo /**/^(proc|sys)/etc/r*(N)

But that might actually miss /etc/r* because it would be looking for
/etc/etc/.  There's no syntax for excluding things specifically from
the ** pattern.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]


On 2024-01-21 11:09, Bart Schaefer wrote:
>> Why /**/*/ rather than just /**/ ?  The extra /*/ there will expand
>> symbolic links, which are normally skipped by /**/.

I just now realized that.

>> ... so by expanding that you're traversing the
>> whole filesystem again, which takes you back to /proc/ again, and so
>> on for every PID until zsh gives up (probably because it can't
>> allocate any more memory).

It had that looping feeling.  Didn't realize that it was crashing tho, I 
thought it was actually finishing.

> I suspect previously you weren't adding that extra /*/.  What do you
> mean for that to accomplish?

      % echo /**/*/etc/r*(/N) #BAD
      % echo /**/*etc/r*(/N)  #GOOD

... at some point the '/' got added, don't know. I just missed it.
All good.  Man, what a strange adventure tho.

> There's no glob qualifier for that, if that's what you mean.  As it
happens, those directories always appear to be empty, so you can find
them with /*(/L0)

Ok I'll look at that.  Mind, the above has solved the problem.  Still one might save a millisecond avoiding searches of those virtual directories -- or not.

> Without trying it, which I'm not going to do, I think the answer to
your specific question might be:

echo /**/^(proc|sys)/etc/r*(N)

Hangs. Never mind!



[-- Attachment #2: Type: text/html, Size: 2292 bytes --]

Re: /proc slows % echo /**/*/etc/r*(/N)

[Juergen Christoffel]

On Sun, Jan 21, 2024 at 10:08:27AM -0800, Ray Andrews wrote:
>
>But I'm holding out for a way of filtering out all those virtual 
>directories.  There's a few of them.

The heretic's solution: locate might be your friend, as it already built a
database of pathnames:

locate /etc/ | egrep -v '^/(proc|foo|bar|baz)'

       --jc

P.S. Or mdfind instead of locate on Macs. And maybe something else on
non-unixoid systems ...

-- 
  Replacing an offensive term to something different just converts the "something
  different" into an offensive term.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

On 2024-01-21 12:09, Juergen Christoffel wrote:
>
> The heretic's solution: locate might be your friend, as it already 
> built a
> database of pathnames:
>
> locate /etc/ | egrep -v '^/(proc|foo|bar|baz)'
>
Being by nature a masochistic purist, I'm trying to use zsh native code, 
and thanks to Mark and Bart I think this is figured out.  Strange 
lurking bug tho, it never showed up until I did a global search for 
'*/etc/r*'.

     eval "wholedisk=( $zsh_case/**/*$~1*(/N) )"

... served, tho of course it's filthy.  I'm still not quite sure why it 
doesn't like '/etc/r' but nevermind.  I now realize that my true friend 
the tilde (thanks Roman) is what was needed and if one is good, two are 
better:

     wholedisk=( $~zsh_case/**/*$~1*(/N) )

... so that gets me away from '/proc' and is just plain better code.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Bart Schaefer]

On Sun, Jan 21, 2024 at 11:59 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> On 2024-01-21 11:09, Bart Schaefer wrote:
>>
>> ... so by expanding that you're traversing the
>> whole filesystem again, which takes you back to /proc/ again, and so
>> on for every PID until zsh gives up (probably because it can't
>> allocate any more memory).
>
> It had that looping feeling.  Didn't realize that it was crashing tho, I thought it was actually finishing.

This is a situation wherein zsh can tell that it's not able to
allocate memory, so it just stops trying without crashing.  If it had
crashed you'd have got nothing back and probably lost your shell
session.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]


On 2024-01-21 13:48, Bart Schaefer wrote:
> This is a situation wherein zsh can tell that it's not able to
> allocate memory, so it just stops trying without crashing.  If it had
> crashed you'd have got nothing back and probably lost your shell
> session.

Yeah, I think I'm using the word too loosely.  Shouldn't there be an 
error msg in such cases? I don't think my abuse happens very often, 
still I had thought she had finished what she was doing. Some message 
would have alerted me to the reality of the loop.



>

[-- Attachment #2: Type: text/html, Size: 1191 bytes --]

Re: /proc slows % echo /**/*/etc/r*(/N)

[Bart Schaefer]

On Sun, Jan 21, 2024 at 4:07 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Shouldn't there be an error msg in such cases?

Are you sure there wasn't, somewhere above all that output?  As far as
I can tell there's no way it actually could have finished.  If it
didn't run out of memory, the lengths of some of the generated paths
must have exceeded the system PATH_MAX and stopped the descent that
way.

> Some message would have alerted me to the reality of the loop.

A symlink loop is not detectable without extraordinary measures such
as recording and testing the inode number of every directory visited.

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

On 2024-01-21 17:02, Bart Schaefer wrote:
> On Sun, Jan 21, 2024 at 4:07 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>> Shouldn't there be an error msg in such cases?
> Are you sure there wasn't, somewhere above all that output?

I'm not TTTT.  I presume it must come at the end, where the output 
stops, but ...

Anyway it overflowed my terminal buffer so I'd not be able to scroll 
back to see it.  I must say zsh can do a mountain of work before she 
runs out of resources.  She was grinding away for almost an hour.

> A symlink loop is not detectable without extraordinary measures such
> as recording and testing the inode number of every directory visited.

Sure.  Nevermind, I don't think even I have ever broken things to that 
extent before, it's hardly a concern.  I hope never to visit /proc ever 
again.  Thanks Bart.



>

Re: /proc slows % echo /**/*/etc/r*(/N)

[Mikael Magnusson]

On 1/21/24, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sun, Jan 21, 2024 at 9:19 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>>
>> I used to do this kind of thing without any problems:
>>
>>      % echo /**/*/etc/r*(/N)
>>
>> ... but just trying it, zsh labors for close to an hour, then spits out
>> countless pages of:
>
> Without trying it, which I'm not going to do, I think the answer to
> your specific question might be:
>
> echo /**/^(proc|sys)/etc/r*(N)
>
> But that might actually miss /etc/r* because it would be looking for
> /etc/etc/.  There's no syntax for excluding things specifically from
> the ** pattern.

You actually can, **/ is shorthand for (*/)# although the latter only
works with extended_glob enabled, the shorthand always works. You can
use any pattern in place of the * if you want (for example,
(^(a|b|c)/)# which would exclude a, b or c from any segment being
recursed), although in the case above I think you'd have actually
wanted /^(proc|sys)/**/etc/r* if the problem was not another typo.

-- 
Mikael Magnusson

Re: /proc slows % echo /**/*/etc/r*(/N)

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]


On 2024-01-21 22:00, Mikael Magnusson wrote:
> You actually can, **/ is shorthand for (*/)# although the latter only
> works with extended_glob enabled, the shorthand always works. You can
> use any pattern in place of the * if you want (for example,
> (^(a|b|c)/)# which would exclude a, b or c from any segment being
> recursed), although in the case above I think you'd have actually
> wanted /^(proc|sys)/**/etc/r* if the problem was not another typo.
>
% print -l /^(proc|sys)/**/etc/rc*(/N)

... nothing

% print -l /^(proc|sys)/**/etc/rc*
/aWorking/Changed/etc/rc.local

... but that's a file not a directory.

Now this is interesting.  Doing your verbatim shorthand swap with '(/N)' 
added:

0 /etc 0 % print -l /(*/)#etc/rc*(/N)
/etc/rc0.d
/etc/rc1.d
/etc/rc2.d
/etc/rc3.d
/etc/rc4.d
/etc/rc5.d
/etc/rc6.d
/etc/rcS.d

... but that took 37 seconds!  Trying the 'normal' way:

0 /etc 0 % print -l /**/etc/rc*(/N)
/etc/rc0.d
/etc/rc1.d
/etc/rc2.d
/etc/rc3.d
/etc/rc4.d
/etc/rc5.d
/etc/rc6.d
/etc/rcS.d

.... that took 3 seconds!  Not exactly six of one half a dozen of the 
other, tho the output is correct.

Going back to 'your way' but swapping out the asterisk as you say:


0 /etc 0 % print -l /(^(proc|sys)/)#etc/rc*(/N)
/etc/rc0.d
/etc/rc1.d
/etc/rc2.d
/etc/rc3.d
/etc/rc4.d
/etc/rc5.d
/etc/rc6.d
/etc/rcS.d

... 7 seconds!  Don't know what to make of it.  It seems very happy not 
to have to search in proc and sys  OTOH it's still twice as long as the 
'regular' syntax.  Leaves me scratchin' my head. Since I no longer have 
to worry about the /proc disaster, as Bart rescued me, the only issue 
now would be performance, and if skipping proc and sys would help with 
that, fine, but as we see it seems to backfire.  Matter of curiosity 
only!  Nothing broken, still one might wonder at these changes in 
performance.  I tend to think of syntax variations as all ending up as 
the same thing under the hood, but maybe not.  As to proc and sys, my 
guess would have been that skipping them would make very little 
difference since they aren't real directories anyway, and yet it makes 
5X difference.  Another loop of some sort?  Sorta like my /proc disaster 
but perhaps resolving itself somehow?





[-- Attachment #2: Type: text/html, Size: 3291 bytes --]