Re: path caching

Re: path caching

[DaviD W. Sanderson]

Since your scheme relies on symbolic links, it won't work on many
systems.

If you want a cache, then I think a purely memory-based cache would be
better.

Simply arrange to clear the cache when $path is assigned to.
No new variables, no new builtins required.

DaviD W. Sanderson (dws@cs.wisc.edu)

Re: path caching

[Scott Merrilees]

> DaviD W. Sanderson (dws@cs.wisc.edu)
> Since your scheme relies on symbolic links, it won't work on many
> systems.
Something I am prepared to wear, as the machines I mostly want to use
do have symlinks.

> If you want a cache, then I think a purely memory-based cache would be
> better.
This is neither persistant, sharable nor externally user manipulable.

> Simply arrange to clear the cache when $path is assigned to.
> No new variables, no new builtins required.
Then I have to modify both the path lookup code & the variable lookup
code.  I didn't want the rest of the shell to know about it.

Anyway, it will be more interesting when I have some data from the
trial I am running.

Sm
-- 
Scott Merrilees, BHP Information Technology, Newcastle, Australia
Internet: Sm@bhpese.oz.au   Phone: +61 49 40 2132   Fax: ... 2165

Re: path caching

[Matthew Farwell]

In article <199202100234.AA26625@cerberus.bhpese.oz.au> you write:
>> Why can't you do something like:
>> This is untested, since I don't have symbolic links.
>> cache = $home/.cache
>> fn rehash {
>> 	for (i in $path) {
>> 		@{ cd $i
>> 		for (j in *) {
>> 			if (! test -x $cache/$j) ln -s $i/$j $cache/$j
>> 		}
>> 		}
>> 	}
>> }
>
>Possible, but I wanted to keep the size of the cache down, so
>directory lookups are quicker.  If I did the above, I would have one
>directory with approx 1800 entries.  I would also have name clashes
>with system v vs bsd vs gnu stuff.  Still it may be worth testing.

You still have the name clashes anyway.

Dylan.

Re: path caching

[Scott Merrilees]

> Why can't you do something like:
> This is untested, since I don't have symbolic links.
> cache = $home/.cache
> fn rehash {
> 	for (i in $path) {
> 		@{ cd $i
> 		for (j in *) {
> 			if (! test -x $cache/$j) ln -s $i/$j $cache/$j
> 		}
> 		}
> 	}
> }

Possible, but I wanted to keep the size of the cache down, so
directory lookups are quicker.  If I did the above, I would have one
directory with approx 1800 entries.  I would also have name clashes
with system v vs bsd vs gnu stuff.  Still it may be worth testing.

Sm
--
Scott Merrilees, BHP Information Technology, Newcastle, Australia
Internet: Sm@bhpese.oz.au   Phone: +61 49 40 2132   Fax: ... 2165

Re: path caching

[Matthew Farwell]

In article <199202081101.AA05781@cerberus.bhpese.oz.au> you write:
>I've been playing with the idea of a file system based path cache, as
>my path is rather long.  Enclosed is some doco I wrote on the caching,
>and accompanying instrumentation.  This is currently for rc 1.2, I am
>just about to post it to 1.3b.  Comments ?
>
>[ deleted ]

Why can't you do something like:

This is untested, since I don't have symbolic links.

cache = $home/.cache
fn rehash {
	for (i in $path) {
		@{ cd $i
		for (j in *) {
			if (! test -x $cache/$j) ln -s $i/$j $cache/$j
		}
		}
	}
}

or something like that.  Then you only require one directory in your
path, ie $cache.

Dylan.

Re: path caching

[Scott Merrilees]

>  How many times the path cache 'hit' is uninteresting; the
> performance number wanted is how much time it saved. If
> you can show a definite useful time saving in a typical
> shell-script/user's session/whatever it might be interesting;
> otherwise, it's just gratuitous complexity.
> 	- cks

Defining a typical shell-script/user's session/whatever is rather difficult,
I'm in the just about to start a trial period of two weeks of cache use,
followed by two weeks of non-cache use.  I'll let you know the results,
but I would think that the results would vary greatly with the
composition of the path, eg number of components, size of directories,
whether components are network accessed, order of path, ram disc based
cache etc.

Sm
-- 
Scott Merrilees, BHP Information Technology, Newcastle, Australia
Internet: Sm@bhpese.oz.au   Phone: +61 49 40 2132   Fax: ... 2165

path caching

[Scott Merrilees]

I've been playing with the idea of a file system based path cache, as
my path is rather long.  Enclosed is some doco I wrote on the caching,
and accompanying instrumentation.  This is currently for rc 1.2, I am
just about to post it to 1.3b.  Comments ?

Sm
--
Scott Merrilees, BHP Information Technology, Newcastle, Australia
Internet: Sm@bhpese.oz.au   Phone: +61 49 40 2132   Fax: ... 2165
--
Cachepath modifications.

This is an attemp to add a file system based command lookup cacheing to rc.

Cachepath is a normal rc variable that is the pathname of a directory
to use for storing the cache entries.  Cache entries are symbolic links
to the real program.  Cachepath should be inserted where you want it
to be searched in the path variable.  Programs that are found after the
position of cachepath in path are symlinked into the cachepath, hence,
subsequent lookup should find them in the cachepath first, rather than
their real directory.  This adds about 30 lines code to which().

eg:
	; cachepath = $home/.cache
	; mkdir $cachepath
	; path = ( /bin /usr/bin $cachepath ... )

The advantages I see in using a filesystem based cache:

1.	the contents of the cache are avaiable to all programs that
	use path lookup

2.	the cache is easily user modifable by the user via rm & ln -s.
	the cache can also be primed/trimmed as per user whim.

3.	user can specify where in the path cachepath works.

4.	I didn't have to modify the shell syntax to include a new
	command to manipulate the cache, ala rehash in csh (vomit)

5.	Should be nice & fast if cachepath is on a memory based file
	system.

disadvantages:

1.	requires symlinks.

2.	can chew up a few inodes

3.	I keep spelling cache as cahce.

4.	requires a special variable name, although it is only known in
	one place, which().

5.	requires a extra varlookup() per call to which(), and some
	extra streq()'s, but this is probably overshadowed by kernel
	overhead in the stat() in rc_access().


Pathstats modificiations.

After doing cahcepath, I wanted a way to evaluate the usefulness of
it.  I decided on a way to measure the number of hits to each path
component.  This is implimented as a list variable, pathstats, which
is a list of numbers, which correspond to path components.  Pathstats
is a kind of readonly variable, as it is overwritten each time which()
succeeds on a path lookup.   I intend to use this to test the
effectiveness of cachepath & to generally test the efficiency of the
current path order.  Overheads involved are a varlookup() and a
varassign() per call to which().  Unfortunately, when rc forks
internally, pathstats from the child aren't propagated back to the
parent, & I don't think it worthwhile to make it do so.  This also
adds about 30 lines to which().


	; pathstats = 0 # enable
	; ...		# use rc normally for a while
	; echo $pathstats
	137 6 5 1 1 0 0 0 0 0 0 0 0 0 1 1
	; echo $path
	/bin /usr/bin /p/Sm/.cache /usr/ucb /usr/local/bin
	/usr/bsd43/bin /etc /usr/etc /usr/local/etc /usr/new /usr/hosts
	/usr/bin/X11 /usr/spool/MHSnet/_bin /usr/local/gnu/bin
	/p/Sm/bin /p/Sm/bin/mipseb .

Re: path caching

[Chris Siebenmann]

 How many times the path cache 'hit' is uninteresting; the
performance number wanted is how much time it saved. If
you can show a definite useful time saving in a typical
shell-script/user's session/whatever it might be interesting;
otherwise, it's just gratuitous complexity.

	- cks

Re: path caching

[John Mackin]

    Comments ?

Yes.  If you want this sort of tripe please be so kind as to use
some other shell.

OK,
John.