RE: PATH: autoload with explicit path

RE: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On 11 December 2016 at 23:18, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> Then I found it was pretty much as easy (easier, actually, since fewer
> special cases) to implement this generally:
>
> autoload -Uz /path/to/myfunc
>
> defines myfunc to be found in the directory /path/to by reusing the
> filename element of the shfunc structure that's currently unused at this
> stage.

I do the non-fpath autoloading by local FPATH parameter. There's
drawback: if autoloaded function does further autoloading of other
function, then the earlier supplied local FPATH parameter isn't active
anymore.

So it's not possible to have /home/user/functions/{fun1,fun2}, autoload
fun1, and have fun2 available for fun1. Now, with the new feature,
directory path can be attached to function and if it calls autoload
itself, then the attached directory can be examined before FPATH. Could
this be added?

There are variations possible in this, whether it should be
attached-path that overloads FPATH, or FPATH to overload attached-path.
I think that a straightforward way would be to promote attached-path
(examine it first), to allow basic /home/user/functions/{fun1,fun2}
organization of functions, which doesn't need much head scratching, it's
selecting functions "in-package", easy to understand. Having FPATH
examined first would allow loading of patched fun2 from regular fpath
dir, but this is inconsistent – if one would want to have patched fun1
in FPATH, then he would need to drop the path argument from autoload,
and copy fun2.

 -- 
  Sebastian Gniazdowski
  psprint2@fastmail.com

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Fri, 27 Jan 2017 07:12:59 -0800
Sebastian Gniazdowski <psprint2@fastmail.com> wrote:
> I do the non-fpath autoloading by local FPATH parameter. There's
> drawback: if autoloaded function does further autoloading of other
> function, then the earlier supplied local FPATH parameter isn't active
> anymore.
> 
> So it's not possible to have /home/user/functions/{fun1,fun2}, autoload
> fun1, and have fun2 available for fun1. Now, with the new feature,
> directory path can be attached to function and if it calls autoload
> itself, then the attached directory can be examined before FPATH.

You're worried about something like this?


# set up function
local path_that_only_appears_here=/foo/bar
autoload -Uz $path_that_only_appears_here/fun1


# definition of fun1
autoload -Uz <what-goes-here?>/fun2
fun2


You can already do this using funcsourcetrace from zsh/parameter:
${funcsourcetrace[1]%%:<->} is the source file for the immediately
enclosing scope, so take the directory part of that and you've
got what you want (being careful if you happen to have evals
or anonymous functions which generate their own scopes).

It's a bit limiting you can't get that information for any function,
since it's alrady available internally in the shfunc structure.  I'd be
happy to add an associative array in zsh/parameter parallel to
$functions, say $function_sources, that gives the information for all
functions.

pws

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Fri, Jan 27, 2017, at 08:24 AM, Peter Stephenson wrote:
> On Fri, 27 Jan 2017 07:12:59 -0800
> Sebastian Gniazdowski <psprint2@fastmail.com> wrote:
> > I do the non-fpath autoloading by local FPATH parameter. There's
> > drawback: if autoloaded function does further autoloading of other
> > function, then the earlier supplied local FPATH parameter isn't active
> > anymore.
> > 
> > So it's not possible to have /home/user/functions/{fun1,fun2}, autoload
> > fun1, and have fun2 available for fun1. Now, with the new feature,
> > directory path can be attached to function and if it calls autoload
> > itself, then the attached directory can be examined before FPATH.
> 
> You're worried about something like this?
> 
> 
> # set up function
> local path_that_only_appears_here=/foo/bar
> autoload -Uz $path_that_only_appears_here/fun1
> 
> 
> # definition of fun1
> autoload -Uz <what-goes-here?>/fun2
> fun2
> 
> 
> You can already do this using funcsourcetrace from zsh/parameter:
> ${funcsourcetrace[1]%%:<->} is the source file for the immediately
> enclosing scope, so take the directory part of that and you've
> got what you want (being careful if you happen to have evals
> or anonymous functions which generate their own scopes).

Internally via autoload it will be easy to create custom versions of
multiple functions. For example calendar does:

autoload -Uz calendar_{add,parse,read,scandate,show,lockfiles}

With the requested feature, I can copy any subset of above functions to
/home/user/functions, and "autoload -Uz /home/user/functions/calendar"
will pull them. This is also possible by prepending FPATH with
/home/user/functions, true. But deciding whether I want to pull in
custom calendar or not via FPATH management or by removing calendar*
files is hard. The new feature will be very convenient if user will have
multiple directories, for example a second one /home/user/new_functions.
To choose from where to pull in the calendar will be to simply change
path to it in autoload call.

So in .zshrc there will be:

# My custom calendar
autoload -Uz /home/user/functions/calendar

and that's it, no FPATH management or removing files from
/home/user/functions (to revert to official calendar, if the directory
is in FPATH).

> It's a bit limiting you can't get that information for any function,
> since it's alrady available internally in the shfunc structure.  I'd be
> happy to add an associative array in zsh/parameter parallel to
> $functions, say $function_sources, that gives the information for all
> functions.

Sounds nice, though it's nice that current implementation of the
absolute-path feature uses pointers to paths to limit memory usage, a
new parameter would probably spend that.

-- 
Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Fri, 27 Jan 2017 10:40:15 -0800
Sebastian Gniazdowski <psprint2@fastmail.com> wrote:
> Internally via autoload it will be easy to create custom versions of
> multiple functions. For example calendar does:
> 
> autoload -Uz calendar_{add,parse,read,scandate,show,lockfiles}
> 
> With the requested feature

I'm still not clear if your "requested feature" is what I'm talking
about or not, i.e. what you can do already with $funcsourcetrace.

> > It's a bit limiting you can't get that information for any function,
> > since it's alrady available internally in the shfunc structure.  I'd be
> > happy to add an associative array in zsh/parameter parallel to
> > $functions, say $function_sources, that gives the information for all
> > functions.
> 
> Sounds nice, though it's nice that current implementation of the
> absolute-path feature uses pointers to paths to limit memory usage, a
> new parameter would probably spend that.

The new parameter would only look up the internal information when you
refer to it, there's nothing permanent additional to be stored.

pws

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Fri, Jan 27, 2017, at 10:44 AM, Peter Stephenson wrote:
> On Fri, 27 Jan 2017 10:40:15 -0800
> Sebastian Gniazdowski <psprint2@fastmail.com> wrote:
> > Internally via autoload it will be easy to create custom versions of
> > multiple functions. For example calendar does:
> > 
> > autoload -Uz calendar_{add,parse,read,scandate,show,lockfiles}
> > 
> > With the requested feature
> 
> I'm still not clear if your "requested feature" is what I'm talking
> about or not, i.e. what you can do already with $funcsourcetrace.

We talk about the same thing, though I want to remove requirement of
using absolute paths in any function files. You wrote:

> # definition of fun1
> autoload -Uz <what-goes-here?>/fun2
> fun2

The point is to make <what-goes-here> not needed at all. The path is to
be taken from "dircache" (that's the name used in previous messages).
Autoload should use dircache of fun1 to find fun2, then follow to
process fpath. There can be an option to enable/disable this.

I've described how flexible and clean would be management of custom
version of functions this way. It's like having packages of software
instead of functions. Requiring the <what-goes-here> part will make this
difficult.

> > Sounds nice, though it's nice that current implementation of the
> > absolute-path feature uses pointers to paths to limit memory usage, a
> > new parameter would probably spend that.
> 
> The new parameter would only look up the internal information when you
> refer to it, there's nothing permanent additional to be stored.

Thought about it, it's a cool approach.

-- 
Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Fri, 27 Jan 2017, Peter Stephenson wrote:

> I'm still not clear if your "requested feature" is what I'm talking
> about or not, i.e. what you can do already with $funcsourcetrace.

It's not so much about what Sebastian can do as it is about what he has
to rely upon other function writers to do.

That is, if I'm in the habit of writing "autoload thing" in the bootstrap
function for a library of things, but Sebastian wants to put my entire
thing-library in a place outside of $fpath, he can't do so without first
editing all my "autoload"s to add a full path.

If every autoload were relative to the absolute-path information of the
calling function context, Sebastian's plan would work without needing
to edit anything, provided that he autoloads the bootstrap function with
an absolute path.

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Sat, 28 Jan 2017 10:05:46 -0800 (PST)
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Fri, 27 Jan 2017, Peter Stephenson wrote:
> 
> > I'm still not clear if your "requested feature" is what I'm talking
> > about or not, i.e. what you can do already with $funcsourcetrace.
> 
> It's not so much about what Sebastian can do as it is about what he has
> to rely upon other function writers to do.
> 
> That is, if I'm in the habit of writing "autoload thing" in the bootstrap
> function for a library of things, but Sebastian wants to put my entire
> thing-library in a place outside of $fpath, he can't do so without first
> editing all my "autoload"s to add a full path.
> 
> If every autoload were relative to the absolute-path information of the
> calling function context, Sebastian's plan would work without needing
> to edit anything, provided that he autoloads the bootstrap function with
> an absolute path.

I'm afraid this is just confusing matters.  Sebastian was asking for a
new option.  There's no question of changing the default behaviour to
prevent any files needing editing, that just screws up the current fpath
way of looking things up.  The question is how to get the function to
use the right directory, not by default, but using information that's
available from the current context.

(It's not entirely clear to me why you can't just "autoload
/wherever/it/is/*" to pick up all the files in the first place, avoiding
any additional autoload, but there are enough possible ways of arranging
a function suite I don't necessarily need to understand that bit.)

Instead of the option, I'm pointing out that using the absolute path
mechanism and the ability to find the path of the function does exactly
the same thing, so yet more potentially confusing (certainly to me)
options aren't actually getting you anything further.

The following patch means that

zmodload zsh/parameter
autoload -Uz ${functions_source[func]:h}/otherfunc

works, further giving you the power to decide where relative to the
source of "func" the code you want is to arbitrary complexity.
($functions_source is effectively a trick to access existing information;
there is no additional storage demand beyond the addition to the code
itself.  It's just a generalisation of the option Sebastian wanted that
presents path information in parameters rather than hiding it.)

I've also made it turn the directory into an absolute one when it's
loaded, so that if the function was found by e.g. fpath=(.) its
directory is recorded based on $PWD.  That's an optional extra, but I
presume it's more useful that being told the directory of a function's
source was ".". The resolution already happens for autoload -r, and you might
imagine that loading the file should resolve the path in the same way,
so I think it's natural anyway.

As this does everything necessary, I will not be adding any new options,
and I've done quite enough on this feature, but if you can argue among
yourselves for something else that doesn't yet work, feel free to go
ahead without me.

diff --git a/Doc/Zsh/mod_parameter.yo b/Doc/Zsh/mod_parameter.yo
index 3d260f8..942e4c5 100644
--- a/Doc/Zsh/mod_parameter.yo
+++ b/Doc/Zsh/mod_parameter.yo
@@ -37,6 +37,30 @@ vindex(dis_functions)
 item(tt(dis_functions))(
 Like tt(functions) but for disabled functions.
 )
+vindex(functions_source)
+item(tt(functions_source))(
+This readonly associative array maps names of enabled functions to the
+name of the file containing the source of the function.
+
+For an autoloaded function that has already been loaded, or marked for
+autoload with an absolute path, or that has had its path resolved with
+`tt(functions -r)', this is the file found for autoloading, resolved
+to an absolute path.
+
+For a function defined within the body of a script or sourced file,
+this is the name of that file.  In this case, this is the exact path
+originally used to that file, which may be a relative path.
+
+For any other function, including any defined at an interactive prompt or
+an autoload function whose path has not yet been resolved, this is
+the empty string.  However, the hash element is reported as defined
+just so long as the function is present:  the keys to this hash are
+the same as those to tt($funcions).
+)
+vindex(dis_functions_source)
+item(tt(dis_functions_source))(
+Like tt(functions_source) but for disabled functions.
+)
 vindex(builtins)
 item(tt(builtins))(
 This associative array gives information about the builtin commands
@@ -202,10 +226,13 @@ defined.  The line number is the line where the `tt(function) var(name)'
 or `var(name) tt(LPAR()RPAR())' started.  In the case of an autoloaded
 function  the line number is reported as zero.
 The format of each element is var(filename)tt(:)var(lineno).
+
 For functions autoloaded from a file in native zsh format, where only the
 body of the function occurs in the file, or for files that have been
 executed by the tt(source) or `tt(.)' builtins, the trace information is
-shown as var(filename)tt(:)var(0), since the entire file is the definition.
+shown as var(filename)tt(:)var(0), since the entire file is the
+definition.  The source file name is resolved to an absolute path when
+the function is loaded or the path to it otherwise resolved.
 
 Most users will be interested in the information in the
 tt(funcfiletrace) array instead.
diff --git a/Src/Modules/parameter.c b/Src/Modules/parameter.c
index 98bcaba..6e62287 100644
--- a/Src/Modules/parameter.c
+++ b/Src/Modules/parameter.c
@@ -515,6 +515,98 @@ scanpmdisfunctions(HashTable ht, ScanFunc func, int flags)
     scanfunctions(ht, func, flags, DISABLED);
 }
 
+/* Functions for the functions_source special parameter. */
+
+/* Retrieve the source file for a function by explicit name */
+
+/**/
+static HashNode
+getfunction_source(UNUSED(HashTable ht), const char *name, int dis)
+{
+    Shfunc shf;
+    Param pm = NULL;
+
+    pm = (Param) hcalloc(sizeof(struct param));
+    pm->node.nam = dupstring(name);
+    pm->node.flags = PM_SCALAR|PM_READONLY;
+    pm->gsu.s = dis ? &pmdisfunction_gsu :  &pmfunction_gsu;
+
+    if ((shf = (Shfunc) shfunctab->getnode2(shfunctab, name)) &&
+	(dis ? (shf->node.flags & DISABLED) : !(shf->node.flags & DISABLED))) {
+	pm->u.str = getshfuncfile(shf);
+	if (!pm->u.str)
+	    pm->u.str = dupstring("");
+    }
+    return &pm->node;
+}
+
+/* Retrieve the source file for functions by scanning the table */
+
+/**/
+static void
+scanfunctions_source(UNUSED(HashTable ht), ScanFunc func, int flags, int dis)
+{
+    struct param pm;
+    int i;
+    HashNode hn;
+
+    memset((void *)&pm, 0, sizeof(struct param));
+    pm.node.flags = PM_SCALAR|PM_READONLY;
+    pm.gsu.s = dis ? &pmdisfunction_gsu : &pmfunction_gsu;
+
+    for (i = 0; i < shfunctab->hsize; i++) {
+	for (hn = shfunctab->nodes[i]; hn; hn = hn->next) {
+	    if (dis ? (hn->flags & DISABLED) : !(hn->flags & DISABLED)) {
+		pm.node.nam = hn->nam;
+		if (func != scancountparams &&
+		    ((flags & (SCANPM_WANTVALS|SCANPM_MATCHVAL)) ||
+		     !(flags & SCANPM_WANTKEYS))) {
+		    pm.u.str = getshfuncfile((Shfunc)hn);
+		    if (!pm.u.str)
+			pm.u.str = dupstring("");
+		}
+		func(&pm.node, flags);
+	    }
+	}
+    }
+}
+
+/* Param table entry for retrieving functions_source element */
+
+/**/
+static HashNode
+getpmfunction_source(HashTable ht, const char *name)
+{
+    return getfunction_source(ht, name, 0);
+}
+
+/* Param table entry for retrieving ds_functions_source element */
+
+/**/
+static HashNode
+getpmdisfunction_source(HashTable ht, const char *name)
+{
+    return getfunction_source(ht, name, 1);
+}
+
+/* Param table entry for scanning functions_source table */
+
+/**/
+static void
+scanpmfunction_source(HashTable ht, ScanFunc func, int flags)
+{
+    scanfunctions_source(ht, func, flags, 0);
+}
+
+/* Param table entry for scanning dis_functions_source table */
+
+/**/
+static void
+scanpmdisfunction_source(HashTable ht, ScanFunc func, int flags)
+{
+    scanfunctions_source(ht, func, flags, 1);
+}
+
 /* Functions for the funcstack special parameter. */
 
 /**/
@@ -2095,6 +2187,8 @@ static struct paramdef partab[] = {
 	    NULL, getpmdisbuiltin, scanpmdisbuiltins),
     SPECIALPMDEF("dis_functions", 0, 
 	    &pmdisfunctions_gsu, getpmdisfunction, scanpmdisfunctions),
+    SPECIALPMDEF("dis_functions_source", PM_READONLY, NULL,
+		 getpmdisfunction_source, scanpmdisfunction_source),
     SPECIALPMDEF("dis_galiases", 0,
 	    &pmdisgaliases_gsu, getpmdisgalias, scanpmdisgaliases),
     SPECIALPMDEF("dis_patchars", PM_ARRAY|PM_READONLY,
@@ -2111,6 +2205,8 @@ static struct paramdef partab[] = {
 	    &funcstack_gsu, NULL, NULL),
     SPECIALPMDEF("functions", 0, &pmfunctions_gsu, getpmfunction,
 		 scanpmfunctions),
+    SPECIALPMDEF("functions_source", PM_READONLY, NULL,
+		 getpmfunction_source, scanpmfunction_source),
     SPECIALPMDEF("functrace", PM_ARRAY|PM_READONLY,
 	    &functrace_gsu, NULL, NULL),
     SPECIALPMDEF("galiases", 0,
diff --git a/Src/hashtable.c b/Src/hashtable.c
index 1f2789d..8987c85 100644
--- a/Src/hashtable.c
+++ b/Src/hashtable.c
@@ -1566,6 +1566,15 @@ dircache_set(char **name, char *value)
 	*name = NULL;
     } else {
 	/*
+	 * As the function path has been resolved to a particular
+	 * location, we'll store it as an absolute path.
+	 */
+	if (*value != '/') {
+	    value = zhtricat(metafy(zgetcwd(), -1, META_HEAPDUP),
+			     "/", value);
+	    value = xsymlink(value, 1);
+	}
+	/*
 	 * We'll maintain the cache at exactly the right size rather
 	 * than overallocating.  The rationale here is that typically
 	 * we'll get a lot of functions in a small number of directories
diff --git a/Test/C04funcdef.ztst b/Test/C04funcdef.ztst
index 5edbe26..176841d 100644
--- a/Test/C04funcdef.ztst
+++ b/Test/C04funcdef.ztst
@@ -325,10 +325,10 @@
     printf '%s\n' 'oops(){}' 'ninjas-earring(){}' 'oops "$@"' >oops
     autoload oops
     oops
-    whence -v oops
+    whence -v oops | sed -e "s%$PWD%CURDIR%"
   )
 0:whence -v of zsh-style autoload
->oops is a shell function from ./oops
+>oops is a shell function from CURDIR/oops
 
   (
     fpath=(.)
diff --git a/Test/V06parameter.ztst b/Test/V06parameter.ztst
index c7df35d..c2a2a4d 100644
--- a/Test/V06parameter.ztst
+++ b/Test/V06parameter.ztst
@@ -1,15 +1,15 @@
 %test
 
   print 'print In sourced file
-  print $LINENO + $functrace + $funcsourcetrace
+  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
   ' >sourcedfile
   print -r -- 'print Started functrace.zsh
   module_path=(./Modules)
-  print $LINENO + $functrace + $funcsourcetrace
+  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
   :
   fn() {
     print Inside function $0
-    print $LINENO + $functrace + $funcsourcetrace
+    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
   }
   :
   fn
@@ -17,7 +17,7 @@
   fpath=(. $fpath)
   :
   echo '\''print Inside $0
-    print $LINENO + $functrace + $funcsourcetrace
+    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
   '\'' >autofn
   :
   autoload autofn
@@ -32,9 +32,9 @@
 >Inside function fn
 >2 + ./functrace.zsh:10 + ./functrace.zsh:5
 >Inside autofn
->2 + ./functrace.zsh:20 + ./autofn:0
+>2 + ./functrace.zsh:20 + CURDIR/autofn:0
 >Inside autofn
->2 + ./functrace.zsh:21 + ./autofn:0
+>2 + ./functrace.zsh:21 + CURDIR/autofn:0
 >In sourced file
 >2 + ./functrace.zsh:22 + ./sourcedfile:0
 
@@ -66,6 +66,19 @@
 >./rocky3.zsh:13 (eval):2
 >./rocky3.zsh:14 ./rocky3.zsh:14
 
+  (
+    fpath=($PWD)
+    print "print I have been autoloaded" >myfunc
+    autoload $PWD/myfunc
+    print ${functions_source[myfunc]/#$PWD/CURDIR}
+    myfunc
+    print ${functions_source[myfunc]/#$PWD/CURDIR}
+  )
+0: $functions_source
+>CURDIR/myfunc
+>I have been autoloaded
+>CURDIR/myfunc
+
 %clean
 
- rm -f autofn functrace.zsh rocky3.zsh sourcedfile
+ rm -f autofn functrace.zsh rocky3.zsh sourcedfile myfunc

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Sat, 28 Jan 2017, Peter Stephenson wrote:

> On Sat, 28 Jan 2017 10:05:46 -0800 (PST)
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > If every autoload were relative to the absolute-path information of the
> > calling function context, Sebastian's plan would work without needing
> > to edit anything, provided that he autoloads the bootstrap function with
> > an absolute path.
>
> I'm afraid this is just confusing matters.  Sebastian was asking for a
> new option.

Well yes, the option would be on the autoload of the bootstrap function,
i.e.,

	autoload -P /some/hard/path/foo

where "foo" in turn calls "autoload bar" would cause the autoload of bar
to inherit /some/hard/path from foo.

> There's no question of changing the default behaviour

Indeed not.


("-P" chosen for example only and entirely without reference to whether it
conflicts with some other use of -P in typeset/functions/etc.)

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Sat, 28 Jan 2017 11:45:36 -0800 (PST)
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Well yes, the option would be on the autoload of the bootstrap function,
> i.e.,
> 
> 	autoload -P /some/hard/path/foo
> 
> where "foo" in turn calls "autoload bar" would cause the autoload of bar
> to inherit /some/hard/path from foo.

Oh.  That just doubles my suspicion that autoloading bar at the point
where you autoload foo is the right thing to do and anything else is
just getting you into the most horrible tangle.

Well, if you think you can implement that without getting into a tangle,
and people will actually use it, go ahead.  I'm 99% certain I'd just get
confused / I will implement it and people will just immediately demand
something completely different without even pausing to think / it will
instantly change into something even more bizarre and difficult to
understand.

pws

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Sat, Jan 28, 2017, at 11:56 AM, Peter Stephenson wrote:
> On Sat, 28 Jan 2017 11:45:36 -0800 (PST)
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > Well yes, the option would be on the autoload of the bootstrap function,
> > i.e.,
> > 
> > 	autoload -P /some/hard/path/foo
> > 
> > where "foo" in turn calls "autoload bar" would cause the autoload of bar
> > to inherit /some/hard/path from foo.
> 
> Oh.  That just doubles my suspicion that autoloading bar at the point
> where you autoload foo is the right thing to do and anything else is
> just getting you into the most horrible tangle.

I could copy calendar to /some/hard/path, modify and load it without
daunting task of altering FPATH. It's like a software package. This
simple approach shows that this isn't something really confusing.

For example, I could have in fpath:

/home/user/functions
/usr/local/share/zsh/site-functions
/usr/local/share/zsh/5.2-dev-2/functions
/home/user/functions2

Now, if I invoke "autoload calendar", from where it will be loaded?
First containing directory.

With the new functionality, I could load from anywhere I would want –
this is simpler than maintaining proper FPATH:

autoload -P /home/user/functions2/calendar
autoload -P /usr/local/share/zsh/site-functions/calendar
etc.

Such operation of using absolute paths is zshrc-like, a final stage
where instead of generalizing, we are defining. The rest is still
generalized – all paths in function files are still general. Only the
top (zshrc) is specific.

I think that this will make Zsh so much flexible that setups involving
not typical 2-3 function directories
(FPATH=my-own-functions:Zsh-default-functions), but say 5 paths will get
common – /home/.zsh/calendar1:/home/.zsh/calendar2 – doing this with
FPATH is rather impossible. With FPATH it's natural to add just 1 or 2
additional layers on top of /usr/local/share/zsh/5.2/functions, adding
more is just too demanding.

-- 
Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Sat, Jan 28, 2017, at 11:12 AM, Peter Stephenson wrote:
> As this does everything necessary, I will not be adding any new options,
> and I've done quite enough on this feature, but if you can argue among
> yourselves for something else that doesn't yet work, feel free to go
> ahead without me.

The new feature (absolute paths) is a great discovery. It's a way to
avoid FPATH. Fpath is abstract mechanism. System of layers. User is to
imagine the layers and establish what will be loaded. It's a full-blown
abstract system – actually nothing straightforward.

To make the new feature complete, absolute paths must descent to
autoloaded functions. Keeping the absolute path only to directly
autoloaded function is ersatz. It's a half-step. I autoload single
function, and then FPATH again takes control and does the "imagine the
layers" thing. This is a tragedy.

-- 
Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Sun, 29 Jan 2017, Sebastian Gniazdowski wrote:

> To make the new feature complete, absolute paths must descent to
> autoloaded functions. Keeping the absolute path only to directly
> autoloaded function is ersatz. It's a half-step. I autoload single
> function, and then FPATH again takes control and does the "imagine the
> layers" thing. This is a tragedy.

Respectfully, Sebastian, I disagree.

Consider for example that several of the autoloadable functions in the
zsh distribution rely on "autoload colors".

If autoload behaved the way you suggest, this would break, because the
colors function would not be found at the absolute path from which the
calling function was loaded -- unless the writer of the calling function
has used "autoload -d", of course.  But if we're now relying on the
author to use the correct options for autoload, we're right back where
we started -- might as well rely on the author to make his function
suite "relocatable" by way of $function_source as well, and that is
now possible.

There's no magic solution that automatically covers all the variations
without cooperation from both the writer and the installer of the suite.

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Sun, Jan 29, 2017, at 08:11 AM, Bart Schaefer wrote:
> On Sun, 29 Jan 2017, Sebastian Gniazdowski wrote:
> 
> > To make the new feature complete, absolute paths must descent to
> > autoloaded functions. Keeping the absolute path only to directly
> > autoloaded function is ersatz. It's a half-step. I autoload single
> > function, and then FPATH again takes control and does the "imagine the
> > layers" thing. This is a tragedy.
> 
> Respectfully, Sebastian, I disagree.
> 
> Consider for example that several of the autoloadable functions in the
> zsh distribution rely on "autoload colors".
> 
> If autoload behaved the way you suggest, this would break, because the
> colors function would not be found at the absolute path from which the
> calling function was loaded -- unless the writer of the calling function
> has used "autoload -d", of course. 

I wrote earlier that -P (let's use your proposed option name) should
prepend dirchache path to FPATH. In other words, -P causes any descent
autoload to try dircache part first, then follow to process fpath.

> But if we're now relying on the
> author to use the correct options for autoload, we're right back where
> we started -- might as well rely on the author to make his function
> suite "relocatable" by way of $function_source as well, and that is
> now possible.
> 
> There's no magic solution that automatically covers all the variations
> without cooperation from both the writer and the installer of the suite.

I think -P is a complete idea, as you know it comes from problems with
Zplugin, -P would solve those problems. So the idea has some real world
experience to back it up.

-- 
  Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Sun, 29 Jan 2017 08:11:04 -0800 (PST)
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Consider for example that several of the autoloadable functions in the
> zsh distribution rely on "autoload colors".
> 
> If autoload behaved the way you suggest, this would break, because the
> colors function would not be found at the absolute path from which the
> calling function was loaded -- unless the writer of the calling function
> has used "autoload -d", of course.  But if we're now relying on the
> author to use the correct options for autoload, we're right back where
> we started -- might as well rely on the author to make his function
> suite "relocatable" by way of $function_source as well, and that is
> now possible.
> 
> There's no magic solution that automatically covers all the variations
> without cooperation from both the writer and the installer of the suite.

That's roughly my thinking --- any attempt to magic it gets very messy
very quickly --- and you don't need to.  Here's how it can work.  I
think it puts the onus to do the clever stuff on the function code
writer, which is the right place.

1. User gets a heap of functions and is instructed to put them all in the
same directory (or under the same directory, we have the flexiblity to
make it work with a set of subdirectories).  That's not controversial
anyway, I don't think.

2. User puts:

autoload -Uz /wherever/they/put/it/mainfunc

in their start up.  Note it'll still work if they rely on $fpath ---
mainfunc is resolved when loaded, so the code below still works.  So you
are protected against the user doing anything as long as it's some valid
autoload that works in the version of the shell in us.  No special
instructions are needed.

3. mainfunc contains:

zmodload zsh/parameter
if (( ${+functions_source} )); then
  # If that's available, load by absolute path is available.
  # The -R barfs immediately if the function can't be found
  # (optional extra to simplify error handling).
  # You can add -d if you want to search $fpath, too.
  autoload -RUz ${functions_source[mainfunc]:h}/{other1,other2,other3}
  # ... or whatever.
else
  # We're stuck with old-fashioned fpath.  Add warning, check, whatever.
  autoload other1 other2 other3
fi

This can be tailored to suit.  Any half-baked magic in the shell C
implementation can't.  To repeat, I'm not in any case interested in
doing half-baked magic in the shell C implementation. so let's not
go down that route again.

pws

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Sun, 29 Jan 2017, Sebastian Gniazdowski wrote:

> I wrote earlier that -P (let's use your proposed option name) should
> prepend dirchache path to FPATH. In other words, -P causes any descent
> autoload to try dircache part first, then follow to process fpath.

That's still not sufficient for the general case, I think.  What happens
if two function suites have an autoload-dependency on one another, and
neither is in the default $fpath?

Re: PATH: autoload with explicit path

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 10030 bytes --]

(Peter I copied you directly because email from my gmail account has been
bouncing.)

I am seeing two failures with the latest updates, see below.  Please let me
know if you need more info.

Regards,
  Vin


cd Test ; make check
make[1]: Entering directory '/SSD-2TB/opt/build/zsh-2017-01-29/Test'
if test -n "gcc"; then \
  cd .. && DESTDIR= \
  make MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
fi
if ZTST_testlist="`for f in /opt/src/zsh-2017-01-29/Test/*.ztst; \
           do echo $f; done`" \
 ZTST_srcdir="/opt/src/zsh-2017-01-29/Test" \
 ZTST_exe=../Src/zsh \
 ../Src/zsh +Z -f /opt/src/zsh-2017-01-29/Test/runtests.zsh; then \
 stat=0; \
else \
 stat=1; \
fi; \
sleep 1; \
rm -rf Modules .zcompdump; \
exit $stat
/opt/src/zsh-2017-01-29/Test/A01grammar.ztst: starting.
This test hangs the shell when it fails...
/opt/src/zsh-2017-01-29/Test/A01grammar.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/A02alias.ztst: starting.
This test hangs the shell when it fails...
/opt/src/zsh-2017-01-29/Test/A02alias.ztst: all tests successful.

/opt/src/zsh-2017-01-29/Test/A03quoting.ztst: starting.
/opt/src/zsh-2017-01-29/Test/A03quoting.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/A04redirect.ztst: starting.
/opt/src/zsh-2017-01-29/Test/A04redirect.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/A05execution.ztst: starting.
This test takes 5 seconds to fail...
This test takes 3 seconds and hangs the shell when it fails...
/opt/src/zsh-2017-01-29/Test/A05execution.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/A06assign.ztst: starting.
/opt/src/zsh-2017-01-29/Test/A06assign.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/A07control.ztst: starting.
/opt/src/zsh-2017-01-29/Test/A07control.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B01cd.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B01cd.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B02typeset.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B02typeset.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B03print.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B03print.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B04read.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B04read.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B05eval.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B05eval.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B06fc.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B06fc.ztst: all tests successful.

/opt/src/zsh-2017-01-29/Test/B07emulate.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B07emulate.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B08shift.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B08shift.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/B09hash.ztst: starting.
/opt/src/zsh-2017-01-29/Test/B09hash.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/C01arith.ztst: starting.
/opt/src/zsh-2017-01-29/Test/C01arith.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/C02cond.ztst: starting.
This test takes two seconds...
/opt/src/zsh-2017-01-29/Test/C02cond.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/C03traps.ztst: starting.
This test takes at least three seconds...
This test, too, takes at least three seconds...
Another test that takes three seconds
/opt/src/zsh-2017-01-29/Test/C03traps.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/C04funcdef.ztst: starting.
--- /tmp/zsh.ztst.11675/ztst.out 2017-01-29 16:48:50.438882533 -0500
+++ /tmp/zsh.ztst.11675/ztst.tout 2017-01-29 16:48:50.442882549 -0500
@@ -1 +1 @@
-oops is a shell function from CURDIR/oops
+oops is a shell function from /SSD-2TBCURDIR/oops
Test /opt/src/zsh-2017-01-29/Test/C04funcdef.ztst failed: output differs
from expected as shown above for:
  (
    fpath=(.)
    printf '%s\n' 'oops(){}' 'ninjas-earring(){}' 'oops "$@"' >oops
    autoload oops
    oops
    whence -v oops | sed -e "s%$PWD%CURDIR%"
  )
Was testing: whence -v of zsh-style autoload
/opt/src/zsh-2017-01-29/Test/C04funcdef.ztst: test failed.
/opt/src/zsh-2017-01-29/Test/C05debug.ztst: starting.
/opt/src/zsh-2017-01-29/Test/C05debug.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D01prompt.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D01prompt.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D02glob.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D02glob.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D03procsubst.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D03procsubst.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D04parameter.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D04parameter.ztst: all tests successful.

/opt/src/zsh-2017-01-29/Test/D05array.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D05array.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D06subscript.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D06subscript.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D07multibyte.ztst: starting.
Testing multibyte with locale en_US.UTF-8
Test case skipped: No Polish UTF-8 locale found, skipping sort test
/opt/src/zsh-2017-01-29/Test/D07multibyte.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D08cmdsubst.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D08cmdsubst.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/D09brace.ztst: starting.
/opt/src/zsh-2017-01-29/Test/D09brace.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/E01options.ztst: starting.
This test hangs the shell when it fails...
/opt/src/zsh-2017-01-29/Test/E01options.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/E02xtrace.ztst: starting.
/opt/src/zsh-2017-01-29/Test/E02xtrace.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V01zmodload.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V01zmodload.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V02zregexparse.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V02zregexparse.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V03mathfunc.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V03mathfunc.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V04features.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V04features.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V05styles.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V05styles.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V06parameter.ztst: starting.
--- /tmp/zsh.ztst.14908/ztst.out 2017-01-29 16:48:55.798904574 -0500
+++ /tmp/zsh.ztst.14908/ztst.tout 2017-01-29 16:48:55.802904591 -0500
@@ -3,8 +3,8 @@
 Inside function fn
 2 + ./functrace.zsh:10 + ./functrace.zsh:5
 Inside autofn
-2 + ./functrace.zsh:20 + CURDIR/autofn:0
+2 + ./functrace.zsh:20 + /SSD-2TB/opt/build/zsh-2017-01-29/Test/autofn:0
 Inside autofn
-2 + ./functrace.zsh:21 + CURDIR/autofn:0
+2 + ./functrace.zsh:21 + /SSD-2TB/opt/build/zsh-2017-01-29/Test/autofn:0
 In sourced file
 2 + ./functrace.zsh:22 + ./sourcedfile:0
Test /opt/src/zsh-2017-01-29/Test/V06parameter.ztst failed: output differs
from expected as shown above for:
  print 'print In sourced file
  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
  ' >sourcedfile
  print -r -- 'print Started functrace.zsh
  module_path=(./Modules)
  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
  :
  fn() {
    print Inside function $0
    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
  }
  :
  fn
  :
  fpath=(. $fpath)
  :
  echo '\''print Inside $0
    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
  '\'' >autofn
  :
  autoload autofn
  :
  autofn
  autofn
  . ./sourcedfile' >functrace.zsh
  $ZTST_testdir/../Src/zsh +Z -f ./functrace.zsh
Was testing: Function tracing
/opt/src/zsh-2017-01-29/Test/V06parameter.ztst: test failed.
/opt/src/zsh-2017-01-29/Test/V07pcre.ztst: starting.
Testing PCRE multibyte with locale en_US.UTF-8
/opt/src/zsh-2017-01-29/Test/V07pcre.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V08zpty.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V08zpty.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V09datetime.ztst: starting.
Test case skipped: Japanese UTF-8 locale not supported
/opt/src/zsh-2017-01-29/Test/V09datetime.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/V10private.ztst: starting.
/opt/src/zsh-2017-01-29/Test/V10private.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/W01history.ztst: starting.
History tests write to /dev/tty
  print ten ten nine one print

  print   print one two three four five six seven eight nine ten one two

  print mystery sequence

  print one

  print two

  print mystery sequence

  print metaphor\? shmetaphor!

  print metaphor!

  print -l metophor, Molochi,

  echo $(  echo foo bar) again

  echo more $(   echo $(  echo foo bar) again )

/opt/src/zsh-2017-01-29/Test/W01history.ztst: all tests successful.

/opt/src/zsh-2017-01-29/Test/X02zlevi.ztst: starting.
This test may hang the shell when it fails...
/opt/src/zsh-2017-01-29/Test/X02zlevi.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/X03zlebindkey.ztst: starting.
/opt/src/zsh-2017-01-29/Test/X03zlebindkey.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/Y01completion.ztst: starting.
/opt/src/zsh-2017-01-29/Test/Y01completion.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/Y02compmatch.ztst: starting.
/opt/src/zsh-2017-01-29/Test/Y02compmatch.ztst: all tests successful.
/opt/src/zsh-2017-01-29/Test/Y03arguments.ztst: starting.
/opt/src/zsh-2017-01-29/Test/Y03arguments.ztst: all tests successful.
**************************************
46 successful test scripts, 2 failures, 0 skipped
**************************************
Makefile:187: recipe for target 'check' failed
make[1]: *** [check] Error 1
make[1]: Leaving directory '/SSD-2TB/opt/build/zsh-2017-01-29/Test'
Makefile:263: recipe for target 'check' failed
make: *** [check] Error 2
check_zsh: make check failed with error code 2.  See /tmp/zsh_check2065.out.


-- 
And my clothes don't fit me no more
I walked a thousand miles
Just to slip this skin

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Sun, 29 Jan 2017 16:53:01 -0500
Vin Shelton <acs@alumni.princeton.edu> wrote:
> (Peter I copied you directly because email from my gmail account has been
> bouncing.)
> 
> I am seeing two failures with the latest updates, see below.  Please let me
> know if you need more info.
> /opt/src/zsh-2017-01-29/Test/C04funcdef.ztst: starting.
> --- /tmp/zsh.ztst.11675/ztst.out 2017-01-29 16:48:50.438882533 -0500
> +++ /tmp/zsh.ztst.11675/ztst.tout 2017-01-29 16:48:50.442882549 -0500
> @@ -1 +1 @@
> -oops is a shell function from CURDIR/oops
> +oops is a shell function from /SSD-2TBCURDIR/oops
> /opt/src/zsh-2017-01-29/Test/V06parameter.ztst: starting.
> --- /tmp/zsh.ztst.14908/ztst.out 2017-01-29 16:48:55.798904574 -0500
> +++ /tmp/zsh.ztst.14908/ztst.tout 2017-01-29 16:48:55.802904591 -0500
> @@ -3,8 +3,8 @@
>  Inside function fn
>  2 + ./functrace.zsh:10 + ./functrace.zsh:5
>  Inside autofn
> -2 + ./functrace.zsh:20 + CURDIR/autofn:0
> +2 + ./functrace.zsh:20 + /SSD-2TB/opt/build/zsh-2017-01-29/Test/autofn:0
>  Inside autofn
> -2 + ./functrace.zsh:21 + CURDIR/autofn:0
> +2 + ./functrace.zsh:21 + /SSD-2TB/opt/build/zsh-2017-01-29/Test/autofn:0
>  In sourced file
>  2 + ./functrace.zsh:22 + ./sourcedfile:0

Let's try the following trick, which has been in use in B01cd.ztst for
some time so presumably works.

pws

diff --git a/Test/C04funcdef.ztst b/Test/C04funcdef.ztst
index 176841d..0cf2b58 100644
--- a/Test/C04funcdef.ztst
+++ b/Test/C04funcdef.ztst
@@ -2,6 +2,10 @@
 
   mkdir funcdef.tmp
   cd funcdef.tmp
+  setopt chaselinks
+  cd .
+  unsetopt chaselinks
+  mydir=$PWD
 
 %test
 
@@ -325,10 +329,10 @@
     printf '%s\n' 'oops(){}' 'ninjas-earring(){}' 'oops "$@"' >oops
     autoload oops
     oops
-    whence -v oops | sed -e "s%$PWD%CURDIR%"
+    whence -v oops
   )
-0:whence -v of zsh-style autoload
->oops is a shell function from CURDIR/oops
+0q:whence -v of zsh-style autoload
+>oops is a shell function from $mydir/oops
 
   (
     fpath=(.)
diff --git a/Test/V06parameter.ztst b/Test/V06parameter.ztst
index c2a2a4d..10e0a27 100644
--- a/Test/V06parameter.ztst
+++ b/Test/V06parameter.ztst
@@ -1,15 +1,22 @@
+%prep
+
+  setopt chaselinks
+  cd .
+  unsetopt chaselinks
+  mydir=$PWD
+
 %test
 
   print 'print In sourced file
-  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
+  print $LINENO + $functrace + ${funcsourcetrace}
   ' >sourcedfile
   print -r -- 'print Started functrace.zsh
   module_path=(./Modules)
-  print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
+  print $LINENO + $functrace + ${funcsourcetrace}
   :
   fn() {
     print Inside function $0
-    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
+    print $LINENO + $functrace + ${funcsourcetrace}
   }
   :
   fn
@@ -17,7 +24,7 @@
   fpath=(. $fpath)
   :
   echo '\''print Inside $0
-    print $LINENO + $functrace + ${funcsourcetrace/#$PWD/CURDIR}
+    print $LINENO + $functrace + ${funcsourcetrace}
   '\'' >autofn
   :
   autoload autofn
@@ -26,15 +33,15 @@
   autofn
   . ./sourcedfile' >functrace.zsh
   $ZTST_testdir/../Src/zsh +Z -f ./functrace.zsh
-0:Function tracing
+0q:Function tracing
 >Started functrace.zsh
 >3 + +
 >Inside function fn
 >2 + ./functrace.zsh:10 + ./functrace.zsh:5
 >Inside autofn
->2 + ./functrace.zsh:20 + CURDIR/autofn:0
+>2 + ./functrace.zsh:20 + $mydir/autofn:0
 >Inside autofn
->2 + ./functrace.zsh:21 + CURDIR/autofn:0
+>2 + ./functrace.zsh:21 + $mydir/autofn:0
 >In sourced file
 >2 + ./functrace.zsh:22 + ./sourcedfile:0
 
@@ -70,14 +77,14 @@
     fpath=($PWD)
     print "print I have been autoloaded" >myfunc
     autoload $PWD/myfunc
-    print ${functions_source[myfunc]/#$PWD/CURDIR}
+    print ${functions_source[myfunc]}
     myfunc
-    print ${functions_source[myfunc]/#$PWD/CURDIR}
+    print ${functions_source[myfunc]}
   )
-0: $functions_source
->CURDIR/myfunc
+0q: $functions_source
+>$mydir/myfunc
 >I have been autoloaded
->CURDIR/myfunc
+>$mydir/myfunc
 
 %clean
 

Re: PATH: autoload with explicit path

[Sebastian Gniazdowski]

On Sun, Jan 29, 2017, at 09:58 AM, Peter Stephenson wrote:
> That's roughly my thinking --- any attempt to magic it gets very messy
> very quickly --- and you don't need to.  Here's how it can work.  I
> think it puts the onus to do the clever stuff on the function code
> writer, which is the right place.


Look, your new functionality is a disaster currently. You can autoload
only one function via absolute-path, any descent autoload will fail, and
you non-FPATH function directory is useless.

-- 
  Sebastian Gniazdowski

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Mon, 30 Jan 2017 03:37:02 -0800
Sebastian Gniazdowski <psprint2@fastmail.com> wrote:
> Look, your new functionality is a disaster currently. You can autoload
> only one function via absolute-path, any descent autoload will fail, and
> you non-FPATH function directory is useless.

OK, I've now worked on this for some time, explained in some detail how
it's used, all of which does work as it's been tested, and simply got
abuse with no technical content.

That's the end of the discussion.  Do not bring this up again.  Do not
reply to this message.

pws

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Tue, 24 Jan 2017 11:17:34 +0100
Mikael Magnusson <mikachu@gmail.com> wrote:
> This patch is probably a bad idea, it's extremely common to do this:
> PROGRAM="$(which program)"
> and with the above change that no longer works.

Yes, that was supposed to be limited to the chattier -v output.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index 2fb1a70..219fbc9 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -3774,9 +3774,11 @@ bin_whence(char *nam, char **argv, Options ops, int func)
 		    if (wd) {
 			printf("%s: command\n", *argv);
 		    } else {
-			if (v && !csh)
+			if (v && !csh) {
 			    zputs(*argv, stdout), fputs(" is ", stdout);
-			quotedzputs(buf, stdout);
+			    quotedzputs(buf, stdout);
+			} else
+			    zputs(buf, stdout);
 			if (OPT_ISSET(ops,'s') || OPT_ISSET(ops, 'S'))
 			    print_if_link(buf, OPT_ISSET(ops, 'S'));
 			fputc('\n', stdout);
@@ -3806,9 +3808,11 @@ bin_whence(char *nam, char **argv, Options ops, int func)
 	    if (wd) {
 		printf("%s: command\n", *argv);
 	    } else {
-		if (v && !csh)
+		if (v && !csh) {
 		    zputs(*argv, stdout), fputs(" is ", stdout);
-		quotedzputs(cnam, stdout);
+		    quotedzputs(cnam, stdout);
+		} else
+		    zputs(cnam, stdout);
 		if (OPT_ISSET(ops,'s') || OPT_ISSET(ops,'S'))
 		    print_if_link(cnam, OPT_ISSET(ops,'S'));
 		fputc('\n', stdout);

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Mon, 16 Jan 2017 07:22:23 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Mon, Jan 16, 2017 at 2:37 AM, Peter Stephenson
> <p.stephenson@samsung.com> wrote:
> >
> > % whence -v my\ stupid\ function\ name
> > my stupid function name is a shell function from /export/home/pws/src/zsh-git/code/'my stupid function name'
> >
> > You might even consider that a feature.
> 
> When was this pushed?  I don't get any quoting.
> 
> % path+=(/tmp/one\ space)
> % whence -v two\ space
> two space is /tmp/one space/two space

That's not a shell function, that's a command.  Looks like that doesn't
do quoting.; I presume that's just a historical oversight.

This would handle the other verbose "is" cases.  However, in this case
the entire path is passed back by findcmd, so it's all quoted.  Updating
findcmd() would make it more efficient (don't need to copy the
directory) and more consistent with the function case, but it's too
minor for me to feel like doing.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index a683032..b1b6e2e 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -3747,7 +3747,7 @@ bin_whence(char *nam, char **argv, Options ops, int func)
 		    } else {
 			if (v && !csh)
 			    zputs(*argv, stdout), fputs(" is ", stdout);
-			zputs(buf, stdout);
+			quotedzputs(buf, stdout);
 			if (OPT_ISSET(ops,'s') || OPT_ISSET(ops, 'S'))
 			    print_if_link(buf, OPT_ISSET(ops, 'S'));
 			fputc('\n', stdout);
@@ -3779,7 +3779,7 @@ bin_whence(char *nam, char **argv, Options ops, int func)
 	    } else {
 		if (v && !csh)
 		    zputs(*argv, stdout), fputs(" is ", stdout);
-		zputs(cnam, stdout);
+		quotedzputs(cnam, stdout);
 		if (OPT_ISSET(ops,'s') || OPT_ISSET(ops,'S'))
 		    print_if_link(cnam, OPT_ISSET(ops,'S'));
 		fputc('\n', stdout);

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Mon, 16 Jan 2017 15:04:52 +0000
Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Peter Stephenson wrote on Mon, Jan 16, 2017 at 10:37:08 +0000:
> > One very minor change is that to
> > avoid the intermediate string quotedzputs() now runs separately on the
> > directory and the filename, e.g.
> > 
> > % whence -v my\ stupid\ function\ name
> > my stupid function name is a shell function from /export/home/pws/src/zsh-git/code/'my stupid function name'
> > 
> > You might even consider that a feature.
> 
> Does it do the right thing when RC_QUOTES is set?

You'll notice there's a "/" between the directory and the file.

pws

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Mon, Jan 16, 2017 at 2:37 AM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> % whence -v my\ stupid\ function\ name
> my stupid function name is a shell function from /export/home/pws/src/zsh-git/code/'my stupid function name'
>
> You might even consider that a feature.

When was this pushed?  I don't get any quoting.

% path+=(/tmp/one\ space)
% whence -v two\ space
two space is /tmp/one space/two space
%

% git whatchanged
commit f90a0447aada3ad2fa114210c62b855a3e60cb1d
Author: Peter Stephenson <pws@zsh.org>
Date:   Mon Jan 16 10:31:56 2017 +0000

    40353 with tweaks to whence -v: extend directory cache use.

Re: PATH: autoload with explicit path

[Daniel Shahaf]

Peter Stephenson wrote on Mon, Jan 16, 2017 at 10:37:08 +0000:
> One very minor change is that to
> avoid the intermediate string quotedzputs() now runs separately on the
> directory and the filename, e.g.
> 
> % whence -v my\ stupid\ function\ name
> my stupid function name is a shell function from /export/home/pws/src/zsh-git/code/'my stupid function name'
> 
> You might even consider that a feature.

Does it do the right thing when RC_QUOTES is set?

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Fri, 13 Jan 2017 18:04:23 +0000
Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Wed, 11 Jan 2017 20:51:22 +0000
> Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> > With a bit of work this can be expanded to paths for already loaded
> > functions, and possibly elsewhere though I think the pickings are
> > a lot smaller.
> 
> This does the former of that --- it was less work than I expected.
>...
> The only significant extra requirement is use of the heap to get the
> filename in "whence -v" --- that's a bit of laziness, we could just print
> out the components using the same logic as getshfuncfile().  Might fix
> that later.

I fixed this up when I pushed it.  One very minor change is that to
avoid the intermediate string quotedzputs() now runs separately on the
directory and the filename, e.g.

% whence -v my\ stupid\ function\ name
my stupid function name is a shell function from /export/home/pws/src/zsh-git/code/'my stupid function name'

You might even consider that a feature.

pws

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Wed, 11 Jan 2017 20:51:22 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> With a bit of work this can be expanded to paths for already loaded
> functions, and possibly elsewhere though I think the pickings are
> a lot smaller.

This does the former of that --- it was less work than I expected.

The upshot of this is that with the new absolute path scheme, storing
the filename at the point of loading the function is a no-op since we've
already got the components and keep them.  With a traditional
fpath-style load, we use the new directory cache at the point where we
load the file from the path, so this now also avoids an extra directory
allocation (and, unlike the similar mechanism used for the command hash,
avoids the brittle association with the original path variable).

There is a heap allocation when we enter the function to put the name on
the function stack (see getshfuncfile()), but there always was --- the
only difference is a little extra calculation for the three components.
So I think the penalty is trivial to non-existent.

The only significant extra requirement is use of the heap to get the
filename in "whence -v" --- that's a bit of laziness, we could just print
out the components using the same logic as getshfuncfile().  Might fix
that later.

Note that the PM_LOADDIR flag still represents exactly what it did,
it's just that the period of possible validity now lasts throughout the
function [nearly said product] life cycle.

pws

diff --git a/Src/exec.c b/Src/exec.c
index 7a5d2bd..6c82643 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -5124,12 +5124,12 @@ execautofn_basic(Estate state, UNUSED(int do_exec))
      * defined yet.
      */
     if (funcstack && !funcstack->filename)
-	funcstack->filename = dupstring(shf->filename);
+	funcstack->filename = getshfuncfile(shf);
 
     oldscriptname = scriptname;
     oldscriptfilename = scriptfilename;
     scriptname = dupstring(shf->node.nam);
-    scriptfilename = dupstring(shf->filename);
+    scriptfilename = getshfuncfile(shf);
     execode(shf->funcdef, 1, 0, "loadautofunc");
     scriptname = oldscriptname;
     scriptfilename = oldscriptfilename;
@@ -5150,13 +5150,47 @@ execautofn(Estate state, UNUSED(int do_exec))
     return execautofn_basic(state, 0);
 }
 
+/*
+ * Helper function to install the source file name of a shell function
+ * just autoloaded.
+ *
+ * We attempt to do this efficiently as the typical case is the
+ * directory part is a well-known directory, which is cached, and
+ * the non-directory part is the same as the node name.
+ */
+
+/**/
+static void
+loadautofnsetfile(Shfunc shf, char *fdir)
+{
+    /*
+     * If shf->filename is already the load directory ---
+     * keep it as we can still use it to get the load file.
+     * This makes autoload with an absolute path particularly efficient.
+     */
+    if (!(shf->node.flags & PM_LOADDIR) ||
+	strcmp(shf->filename, fdir) != 0) {
+	/* Old directory name not useful... */
+	dircache_set(&shf->filename, NULL);
+	if (fdir) {
+	    /* ...can still cache directory */
+	    shf->node.flags |= PM_LOADDIR;
+	    dircache_set(&shf->filename, fdir);
+	} else {
+	    /* ...no separate directory part to cache, for some reason. */
+	    shf->node.flags &= ~PM_LOADDIR;
+	    shf->filename = ztrdup(shf->node.nam);
+	}
+    }
+}
+
 /**/
 Shfunc
 loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 {
     int noalias = noaliases, ksh = 1;
     Eprog prog;
-    char *fname;
+    char *fdir;			/* Directory path where func found */
 
     pushheap();
 
@@ -5167,13 +5201,13 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	char *spec_path[2];
 	spec_path[0] = dupstring(shf->filename);
 	spec_path[1] = NULL;
-	prog = getfpfunc(shf->node.nam, &ksh, &fname, spec_path, 0);
+	prog = getfpfunc(shf->node.nam, &ksh, &fdir, spec_path, 0);
 	if (prog == &dummy_eprog &&
 	    (current_fpath || (shf->node.flags & PM_CUR_FPATH)))
-	    prog = getfpfunc(shf->node.nam, &ksh, &fname, NULL, 0);
+	    prog = getfpfunc(shf->node.nam, &ksh, &fdir, NULL, 0);
     }
     else
-	prog = getfpfunc(shf->node.nam, &ksh, &fname, NULL, 0);
+	prog = getfpfunc(shf->node.nam, &ksh, &fdir, NULL, 0);
     noaliases = noalias;
 
     if (ksh == 1) {
@@ -5192,7 +5226,6 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	return NULL;
     }
     if (!prog) {
-	zsfree(fname);
 	popheap();
 	return NULL;
     }
@@ -5205,10 +5238,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 		shf->funcdef = prog;
 	    else
 		shf->funcdef = dupeprog(prog, 0);
-	    shf->node.flags &= ~(PM_UNDEFINED|PM_LOADDIR);
-	    dircache_set(&shf->filename, NULL);
-	    /* Full filename, don't use dircache */
-	    shf->filename = fname;
+	    shf->node.flags &= ~PM_UNDEFINED;
+	    loadautofnsetfile(shf, fdir);
 	} else {
 	    VARARR(char, n, strlen(shf->node.nam) + 1);
 	    strcpy(n, shf->node.nam);
@@ -5220,7 +5251,6 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 		zwarn("%s: function not defined by file", n);
 		locallevel++;
 		popheap();
-		zsfree(fname);
 		return NULL;
 	    }
 	}
@@ -5230,10 +5260,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	    shf->funcdef = stripkshdef(prog, shf->node.nam);
 	else
 	    shf->funcdef = dupeprog(stripkshdef(prog, shf->node.nam), 0);
-	shf->node.flags &= ~(PM_UNDEFINED|PM_LOADDIR);
-	dircache_set(&shf->filename, NULL);
-	/* Full filename, don't use dircache */
-	shf->filename = fname;
+	shf->node.flags &= ~PM_UNDEFINED;
+	loadautofnsetfile(shf, fdir);
     }
     popheap();
 
@@ -5453,7 +5481,7 @@ doshfunc(Shfunc shfunc, LinkList doshargs, int noreturnval)
 	funcstack = &fstack;
 
 	fstack.flineno = shfunc->lineno;
-	fstack.filename = dupstring(shfunc->filename);
+	fstack.filename = getshfuncfile(shfunc);
 
 	prog = shfunc->funcdef;
 	if (prog->flags & EF_RUN) {
@@ -5623,16 +5651,17 @@ runshfunc(Eprog prog, FuncWrap wrap, char *name)
  * Search fpath for an undefined function.  Finds the file, and returns the
  * list of its contents.
  *
- * If test is 0, load the function; *fname is set to zalloc'ed location.
+ * If test is 0, load the function.
  *
  * If test_only is 1, don't load function, just test for it:
- * - Non-null return means function was found
- * - *fname points to path at which found (not duplicated)
+ * Non-null return means function was found
+ *
+ * *fdir points to path at which found (as passed in, not duplicated)
  */
 
 /**/
 Eprog
-getfpfunc(char *s, int *ksh, char **fname, char **alt_path, int test_only)
+getfpfunc(char *s, int *ksh, char **fdir, char **alt_path, int test_only)
 {
     char **pp, buf[PATH_MAX+1];
     off_t len;
@@ -5650,8 +5679,8 @@ getfpfunc(char *s, int *ksh, char **fname, char **alt_path, int test_only)
 	else
 	    strcpy(buf, s);
 	if ((r = try_dump_file(*pp, s, buf, ksh, test_only))) {
-	    if (fname)
-		*fname = test_only ? *pp : ztrdup(buf);
+	    if (fdir)
+		*fdir = *pp;
 	    return r;
 	}
 	unmetafy(buf, NULL);
@@ -5661,7 +5690,8 @@ getfpfunc(char *s, int *ksh, char **fname, char **alt_path, int test_only)
 		(len = lseek(fd, 0, 2)) != -1) {
 		if (test_only) {
 		    close(fd);
-		    *fname = *pp;
+		    if (fdir)
+			*fdir = *pp;
 		    return &dummy_eprog;
 		}
 		d = (char *) zalloc(len + 1);
@@ -5677,8 +5707,8 @@ getfpfunc(char *s, int *ksh, char **fname, char **alt_path, int test_only)
 		    r = parse_string(d, 1);
 		    scriptname = oldscriptname;
 
-		    if (fname)
-			*fname = ztrdup(buf);
+		    if (fdir)
+			*fdir = *pp;
 
 		    zfree(d, len + 1);
 
diff --git a/Src/hashtable.c b/Src/hashtable.c
index a3a38f7..0fc87f3 100644
--- a/Src/hashtable.c
+++ b/Src/hashtable.c
@@ -926,10 +926,12 @@ printshfuncnode(HashNode hn, int printflags)
 	       (f->node.flags & PM_UNDEFINED) ?
 	       " is an autoload shell function" :
 	       " is a shell function");
-	if (f->filename && (printflags & PRINT_WHENCE_VERBOSE) &&
-	    strcmp(f->filename, f->node.nam) != 0) {
-	    printf(" from ");
-	    quotedzputs(f->filename, stdout);
+	if (printflags & PRINT_WHENCE_VERBOSE) {
+	    char *fname = getshfuncfile(f);
+	    if (fname && strcmp(fname, f->node.nam) != 0) {
+		printf(" from ");
+		quotedzputs(fname, stdout);
+	    }
 	}
 	putchar('\n');
 	return;
@@ -959,7 +961,7 @@ printshfuncnode(HashNode hn, int printflags)
 	    zputs("builtin autoload -X", stdout);
 	    for (fl=0;fopt[fl];fl++)
 		if (f->node.flags & flgs[fl]) putchar(fopt[fl]);
-	    if (f->filename) {
+	    if (f->filename && (f->node.flags & PM_LOADDIR)) {
 		putchar(' ');
 		zputs(f->filename, stdout);
 	    }
@@ -1041,6 +1043,24 @@ printshfuncexpand(HashNode hn, int printflags, int expand)
     text_expand_tabs = save_expand;
 }
 
+/*
+ * Get a heap-duplicated name of the shell function, for
+ * use in tracing.
+ */
+
+/**/
+mod_export char *
+getshfuncfile(Shfunc shf)
+{
+    if (shf->node.flags & PM_LOADDIR) {
+	return zhtricat(shf->filename, "/", shf->node.nam);
+    } else if (shf->filename) {
+	return dupstring(shf->filename);
+    } else {
+	return NULL;
+    }
+}
+
 /**************************************/
 /* Reserved Word Hash Table Functions */
 /**************************************/
diff --git a/Src/signals.c b/Src/signals.c
index 2de5743..a717677 100644
--- a/Src/signals.c
+++ b/Src/signals.c
@@ -811,8 +811,11 @@ dosavetrap(int sig, int level)
 	    newshf->node.nam = ztrdup(shf->node.nam);
 	    newshf->node.flags = shf->node.flags;
 	    newshf->funcdef = dupeprog(shf->funcdef, 0);
-	    /* Assume this is a real filename, don't use dircache */
-	    newshf->filename = ztrdup(shf->filename);
+	    if (shf->node.flags & PM_LOADDIR) {
+		dircache_set(&newshf->filename, shf->filename);
+	    } else {
+		newshf->filename = ztrdup(shf->filename);
+	    }
 	    if (shf->sticky) {
 		newshf->sticky = sticky_emulation_dup(shf->sticky, 0);
 	    } else

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Wed, 11 Jan 2017 20:51:22 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Wed, 11 Jan 2017 11:42:32 +0000
> Peter Stephenson <p.stephenson@samsung.com> wrote:
> > The main disadvantage is each function record now contains the name of
> > the directory, so this takes extra memory --- it looks like a couple of
> > dozen k in the case of my function directory which is quite full.  This
> > could be optimised with indirection but I'm not sure it's worth the work.
> 
> This seemed like too good an idea not to follow up.  Now "autoload
> /blah/blah/*" is as efficient as the old way of doing it, too.

Update based on the previous fix, also fix a memory leak in the cache
and add a test to check what happens when bits of the cache are
deleted.  I think this is sane now.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index 716ddd4..a683032 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2958,13 +2958,13 @@ check_autoload(Shfunc shf, char *name, Options ops, int func)
 	    }
 	}
 	if (getfpfunc(shf->node.nam, NULL, &dir_path, NULL, 1)) {
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
 	    if (*dir_path != '/') {
 		dir_path = zhtricat(metafy(zgetcwd(), -1, META_HEAPDUP),
 				    "/", dir_path);
 		dir_path = xsymlink(dir_path, 1);
 	    }
-	    shf->filename = ztrdup(dir_path);
+	    dircache_set(&shf->filename, dir_path);
 	    shf->node.flags |= PM_LOADDIR;
 	    return 0;
 	}
@@ -3029,8 +3029,8 @@ add_autoload_function(Shfunc shf, char *funcname)
 	    *nam++ = '\0';
 	    dir = funcname;
 	}
-	zsfree(shf->filename);
-	shf->filename = ztrdup(dir);
+	dircache_set(&shf->filename, NULL);
+	dircache_set(&shf->filename, dir);
 	shf->node.flags |= PM_LOADDIR;
 	shfunctab->addnode(shfunctab, ztrdup(nam), shf);
     } else {
@@ -3280,8 +3280,8 @@ bin_functions(char *name, char **argv, Options ops, int func)
 		shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
 	    }
 	    if (*argv) {
-		zsfree(shf->filename);
-		shf->filename = ztrdup(*argv);
+		dircache_set(&shf->filename, NULL);
+		dircache_set(&shf->filename, *argv);
 		on |= PM_LOADDIR;
 	    }
 	    shf->node.flags = on;
diff --git a/Src/exec.c b/Src/exec.c
index 68c455b..7a5d2bd 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -4902,6 +4902,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 	shf = (Shfunc) zalloc(sizeof(*shf));
 	shf->funcdef = prog;
 	shf->node.flags = 0;
+	/* No dircache here, not a directory */
 	shf->filename = ztrdup(scriptfilename);
 	shf->lineno = lineno;
 	/*
@@ -4934,7 +4935,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 		    freeeprog(shf->funcdef);
 		    if (shf->redir) /* shouldn't be */
 			freeeprog(shf->redir);
-		    zsfree(shf->filename);
+		    dircache_set(&shf->filename, NULL);
 		    zfree(shf, sizeof(*shf));
 		    state->pc = end;
 		    return 1;
@@ -4965,7 +4966,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 	    freeeprog(shf->funcdef);
 	    if (shf->redir) /* shouldn't be */
 		freeeprog(shf->redir);
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
 	    zfree(shf, sizeof(*shf));
 	    break;
 	} else {
@@ -4974,7 +4975,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 		(signum = getsignum(s + 4)) != -1) {
 		if (settrap(signum, NULL, ZSIG_FUNC)) {
 		    freeeprog(shf->funcdef);
-		    zsfree(shf->filename);
+		    dircache_set(&shf->filename, NULL);
 		    zfree(shf, sizeof(*shf));
 		    state->pc = end;
 		    return 1;
@@ -5205,7 +5206,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	    else
 		shf->funcdef = dupeprog(prog, 0);
 	    shf->node.flags &= ~(PM_UNDEFINED|PM_LOADDIR);
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
+	    /* Full filename, don't use dircache */
 	    shf->filename = fname;
 	} else {
 	    VARARR(char, n, strlen(shf->node.nam) + 1);
@@ -5229,7 +5231,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	else
 	    shf->funcdef = dupeprog(stripkshdef(prog, shf->node.nam), 0);
 	shf->node.flags &= ~(PM_UNDEFINED|PM_LOADDIR);
-	zsfree(shf->filename);
+	dircache_set(&shf->filename, NULL);
+	/* Full filename, don't use dircache */
 	shf->filename = fname;
     }
     popheap();
@@ -5620,6 +5623,8 @@ runshfunc(Eprog prog, FuncWrap wrap, char *name)
  * Search fpath for an undefined function.  Finds the file, and returns the
  * list of its contents.
  *
+ * If test is 0, load the function; *fname is set to zalloc'ed location.
+ *
  * If test_only is 1, don't load function, just test for it:
  * - Non-null return means function was found
  * - *fname points to path at which found (not duplicated)
diff --git a/Src/hashtable.c b/Src/hashtable.c
index 2a8b585..22ebe34 100644
--- a/Src/hashtable.c
+++ b/Src/hashtable.c
@@ -889,7 +889,7 @@ freeshfuncnode(HashNode hn)
 	freeeprog(shf->funcdef);
     if (shf->redir)
 	freeeprog(shf->redir);
-    zsfree(shf->filename);
+    dircache_set(&shf->filename, NULL);
     if (shf->sticky) {
 	if (shf->sticky->n_on_opts)
 	    zfree(shf->sticky->on_opts,
@@ -1442,3 +1442,140 @@ freehistdata(Histent he, int unlink)
 	}
     }
 }
+
+
+/***********************************************************************
+ * Directory name cache mechanism
+ *
+ * The idea of this is that there are various shell structures,
+ * notably functions, that record the directories with which they
+ * are associated.  Rather than store the full string each time,
+ * we store a pointer to the same location and count the references.
+ * This is optimised so that retrieval is quick at the expense of
+ * searching the list when setting up the structure, which is a much
+ * rarer operation.
+ *
+ * There is nothing special about the fact that the strings are
+ * directories, except for the assumptions for efficiency that many
+ * structures will point to the same one, and that there are not too
+ * many different directories associated with the shell.
+ **********************************************************************/
+
+struct dircache_entry
+{
+    /* Name of directory in cache */
+    char *name;
+    /* Number of references to it */
+    int refs;
+};
+
+/*
+ * dircache is the cache, of length dircache_size.
+ * dircache_lastentry is the last entry used, an optimisation
+ * for multiple references to the same directory, e.g
+ * "autoload /blah/blah/\*".
+ */
+struct dircache_entry *dircache, *dircache_lastentry;
+int dircache_size;
+
+/*
+ * Set *name to point to a cached version of value.
+ * value is copied so may come from any source.
+ *
+ * If value is NULL, look for the existing value of *name (safe if this
+ * too is NULL) and remove a reference to it from the cache. If it's
+ * not found in the cache, it's assumed to be an allocated string and
+ * freed --- this currently occurs for a shell function that's been
+ * loaded as the filename is now a full path, not just a directory,
+ * though we may one day optimise this to a cached directory plus a
+ * name, too.  Note --- the function does *not* otherwise check
+ * if *name points to something already cached, so this is
+ * necessary any time *name may already be in the cache.
+ */
+
+/**/
+mod_export void
+dircache_set(char **name, char *value)
+{
+    struct dircache_entry *dcptr, *dcnew;
+
+    if (!value) {
+	if (!*name)
+	    return;
+	if (!dircache_size) {
+	    zsfree(*name);
+	    *name = NULL;
+	    return;
+	}
+
+	for (dcptr = dircache; dcptr < dircache + dircache_size; dcptr++)
+	{
+	    /* Must be a pointer much, not a string match */
+	    if (*name == dcptr->name)
+	    {
+		--dcptr->refs;
+		if (!dcptr->refs) {
+		    ptrdiff_t ind = dcptr - dircache;
+		    zsfree(dcptr->name);
+		    --dircache_size;
+
+		    if (!dircache_size) {
+			zfree(dircache, sizeof(*dircache));
+			dircache = NULL;
+			dircache_lastentry = NULL;
+			return;
+		    }
+		    dcnew = (struct dircache_entry *)
+			zalloc(dircache_size * sizeof(*dcnew));
+		    if (ind)
+			memcpy(dcnew, dircache, ind * sizeof(*dcnew));
+		    if (ind < dircache_size)
+			memcpy(dcnew + ind, dcptr + 1,
+			       (dircache_size - ind) * sizeof(*dcnew));
+		    zfree(dircache, (dircache_size+1)*sizeof(*dcnew));
+		    dircache = dcnew;
+		    dircache_lastentry = NULL;
+		}
+		*name = NULL;
+		return;
+	    }
+	}
+	zsfree(*name);
+	*name = NULL;
+    } else {
+	/*
+	 * We'll maintain the cache at exactly the right size rather
+	 * than overallocating.  The rationale here is that typically
+	 * we'll get a lot of functions in a small number of directories
+	 * so the complexity overhead of maintaining a separate count
+	 * isn't really matched by the efficiency gain.
+ 	 */
+	if (dircache_lastentry &&
+	    !strcmp(value, dircache_lastentry->name)) {
+	    *name = dircache_lastentry->name;
+	    ++dircache_lastentry->refs;
+	    return;
+	} else if (!dircache_size) {
+	    dircache_size = 1;
+	    dcptr = dircache =
+		(struct dircache_entry *)zalloc(sizeof(*dircache));
+	} else {
+	    for (dcptr = dircache; dcptr < dircache + dircache_size; dcptr++)
+	    {
+		if (!strcmp(value, dcptr->name)) {
+		    *name = dcptr->name;
+		    ++dcptr->refs;
+		    return;
+		}
+	    }
+	    ++dircache_size;
+	    dircache = (struct dircache_entry *)
+		zrealloc(dircache, sizeof(*dircache) * dircache_size);
+	    dcptr = dircache + dircache_size - 1;
+	}
+	dcptr->name = ztrdup(value);
+	*name = dcptr->name;
+	dcptr->refs = 1;
+	dircache_lastentry = dcptr;
+    }
+}
diff --git a/Src/signals.c b/Src/signals.c
index 9e05add..2de5743 100644
--- a/Src/signals.c
+++ b/Src/signals.c
@@ -811,6 +811,7 @@ dosavetrap(int sig, int level)
 	    newshf->node.nam = ztrdup(shf->node.nam);
 	    newshf->node.flags = shf->node.flags;
 	    newshf->funcdef = dupeprog(shf->funcdef, 0);
+	    /* Assume this is a real filename, don't use dircache */
 	    newshf->filename = ztrdup(shf->filename);
 	    if (shf->sticky) {
 		newshf->sticky = sticky_emulation_dup(shf->sticky, 0);
diff --git a/Test/C04funcdef.ztst b/Test/C04funcdef.ztst
index 7100280..370394b 100644
--- a/Test/C04funcdef.ztst
+++ b/Test/C04funcdef.ztst
@@ -429,6 +429,45 @@
 0: autoload -X interaction with absolute filename used for source location
 >Function was loaded correctly.
 
+  (
+    fpath=()
+    mkdir extra2
+    for f in fun2a fun2b; do
+      print "print $f" >extra2/$f
+    done
+    repeat 3; do
+      autoload $PWD/extra2/fun2{a,b} $PWD/extra/spec
+      fun2a
+      fun2b
+      spec
+      unfunction fun2a fun2b spec
+      autoload $PWD/extra2/fun2{a,b} $PWD/extra/spec
+      spec
+      fun2b
+      fun2a
+      unfunction fun2a fun2b spec
+    done
+  )
+0: Exercise the directory name cache for autoloads
+>fun2a
+>fun2b
+>I have been loaded by explicit path.
+>I have been loaded by explicit path.
+>fun2b
+>fun2a
+>fun2a
+>fun2b
+>I have been loaded by explicit path.
+>I have been loaded by explicit path.
+>fun2b
+>fun2a
+>fun2a
+>fun2b
+>I have been loaded by explicit path.
+>I have been loaded by explicit path.
+>fun2b
+>fun2a
+
 %clean
 
  rm -f file.in file.out

Re: PATH: autoload with explicit path

[Peter Stephenson]

Completion for this.

Didn't really understand what I was doing here, but it seems to work...

pws

diff --git a/Completion/Zsh/Command/_typeset b/Completion/Zsh/Command/_typeset
index 94f63cf..0961e98 100644
--- a/Completion/Zsh/Command/_typeset
+++ b/Completion/Zsh/Command/_typeset
@@ -21,6 +21,7 @@ allargs=(
   X '+X[immediately autoload function]'
   Z "($fopts -A -E -F -i)-Z+[right justify and fill with leading zeros]:width"
   a "($fopts -A -E -F -T -i)-a[specify that arguments refer to arrays]"
+  df "-d[default absolute path autoload to fpath]"
   f "($popts)-f[specify that arguments refer to functions]"
   g "($fopts -T)-+g[do not restrict parameter to local scope]"
   h "($fopts -T)-+h[hide specialness of parameter]"
@@ -31,6 +32,8 @@ allargs=(
   m '(-A -E -F -T -i)-m[treat arguments as patterns]'
   p '-p[output parameters in form of calls to typeset]'
   r '(-f)-+r[mark parameters as readonly]'
+  rf '-r[remember autoload path]'
+  Rf '-R[remember autoload path, error if not found]'
   t '(-T)-+t[tag parameters and turn on execution tracing for functions]'
   tf '(-T)-+t[turn on execution tracing for functions]'
   tp '(-T)-+t[tag parameters]'
@@ -46,7 +49,7 @@ use="AEFHLRTUZafghiklmprtuxz"
 
 case ${service} in
   autoload)
-    use="UTXktwz"
+    use="URTXdkrtwz"
     func=f
   ;;
   float) use="EFHghlprtux";;
@@ -74,7 +77,7 @@ onopts=${(j..)${${words[1,CURRENT-1]:#^-*}##-}}
 offopts=${(j..)${${words[1,CURRENT-1]:#^+*}##+}}
 
 for ((i=1;i<=$#use;++i)); do
-  args+=( ${allargs[${use[$i]}${${(s::)use[$i]}[(r)[UutT]]:+$func}]} )
+  args+=( ${allargs[${use[$i]}${${(s::)use[$i]}[(r)[dUurRtT]]:+$func}]} )
 done
 
 _arguments -C -s -A "-*" -S "${args[@]}" '*::vars:= ->vars_eq'
@@ -90,12 +93,17 @@ if [[ "$state" = vars_eq ]]; then
     elif (( $+opt_args[-w] )); then
       _wanted files expl 'zwc file' _files -g '*.zwc(-.)'
     elif [[ $service = autoload || -n $opt_args[(i)-[uU]] ]]; then
-      args=(${^fpath}/*(-.:t))
-      # Filter out functions already loaded or marked for autoload.
-      local -a funckeys
-      funckeys=(${(k)functions})
-      args=(${args:|funckeys})
-      _wanted functions expl 'shell function' compadd -a args
+      if [[ $PREFIX[1] = / ]]; then
+	# Autoload by absolute path
+	_files
+      else
+	  args=(${^fpath}/*(-.:t))
+	  # Filter out functions already loaded or marked for autoload.
+	  local -a funckeys
+	  funckeys=(${(k)functions})
+	  args=(${args:|funckeys})
+	  _wanted functions expl 'shell function' compadd -a args
+      fi
     elif [[ -n $onopts$offopts ]]; then
       if [[ -n $offopts ]]; then
 	args=(${(f)"$(functions +${offopts//[^UXkmtTuz]/})"})

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Wed, 11 Jan 2017 11:42:32 +0000
Peter Stephenson <p.stephenson@samsung.com> wrote:
> The main disadvantage is each function record now contains the name of
> the directory, so this takes extra memory --- it looks like a couple of
> dozen k in the case of my function directory which is quite full.  This
> could be optimised with indirection but I'm not sure it's worth the work.

This seemed like too good an idea not to follow up.  Now "autoload
/blah/blah/*" is as efficient as the old way of doing it, too.

With a bit of work this can be expanded to paths for already loaded
functions, and possibly elsewhere though I think the pickings are
a lot smaller.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index b986dd8..0ed081a 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2957,13 +2957,13 @@ check_autoload(Shfunc shf, char *name, Options ops, int func)
 	    }
 	}
 	if (getfpfunc(shf->node.nam, NULL, &dir_path, NULL, 1)) {
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
 	    if (*dir_path != '/') {
 		dir_path = zhtricat(metafy(zgetcwd(), -1, META_HEAPDUP),
 				    "/", dir_path);
 		dir_path = xsymlink(dir_path, 1);
 	    }
-	    shf->filename = ztrdup(dir_path);
+	    dircache_set(&shf->filename, dir_path);
 	    return 0;
 	}
 	if (OPT_ISSET(ops,'R')) {
@@ -3026,8 +3026,8 @@ add_autoload_function(Shfunc shf, char *funcname)
 	    *nam++ = '\0';
 	    dir = funcname;
 	}
-	zsfree(shf->filename);
-	shf->filename = ztrdup(dir);
+	dircache_set(&shf->filename, NULL);
+	dircache_set(&shf->filename, dir);
 	shfunctab->addnode(shfunctab, ztrdup(nam), shf);
     } else {
 	shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
@@ -3276,8 +3276,8 @@ bin_functions(char *name, char **argv, Options ops, int func)
 		shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
 	    }
 	    if (*argv) {
-		zsfree(shf->filename);
-		shf->filename = ztrdup(*argv);
+		dircache_set(&shf->filename, NULL);
+		dircache_set(&shf->filename, *argv);
 	    }
 	    shf->node.flags = on;
 	    ret = eval_autoload(shf, funcname, ops, func);
diff --git a/Src/exec.c b/Src/exec.c
index 7bec7ce..10a5249 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -4902,6 +4902,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 	shf = (Shfunc) zalloc(sizeof(*shf));
 	shf->funcdef = prog;
 	shf->node.flags = 0;
+	/* No dircache here, not a directory */
 	shf->filename = ztrdup(scriptfilename);
 	shf->lineno = lineno;
 	/*
@@ -4934,7 +4935,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 		    freeeprog(shf->funcdef);
 		    if (shf->redir) /* shouldn't be */
 			freeeprog(shf->redir);
-		    zsfree(shf->filename);
+		    dircache_set(&shf->filename, NULL);
 		    zfree(shf, sizeof(*shf));
 		    state->pc = end;
 		    return 1;
@@ -4965,7 +4966,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 	    freeeprog(shf->funcdef);
 	    if (shf->redir) /* shouldn't be */
 		freeeprog(shf->redir);
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
 	    zfree(shf, sizeof(*shf));
 	    break;
 	} else {
@@ -4974,7 +4975,7 @@ execfuncdef(Estate state, Eprog redir_prog)
 		(signum = getsignum(s + 4)) != -1) {
 		if (settrap(signum, NULL, ZSIG_FUNC)) {
 		    freeeprog(shf->funcdef);
-		    zsfree(shf->filename);
+		    dircache_set(&shf->filename, NULL);
 		    zfree(shf, sizeof(*shf));
 		    state->pc = end;
 		    return 1;
@@ -5204,7 +5205,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	    else
 		shf->funcdef = dupeprog(prog, 0);
 	    shf->node.flags &= ~PM_UNDEFINED;
-	    zsfree(shf->filename);
+	    dircache_set(&shf->filename, NULL);
+	    /* Full filename, don't use dircache */
 	    shf->filename = fname;
 	} else {
 	    VARARR(char, n, strlen(shf->node.nam) + 1);
@@ -5228,7 +5230,8 @@ loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 	else
 	    shf->funcdef = dupeprog(stripkshdef(prog, shf->node.nam), 0);
 	shf->node.flags &= ~PM_UNDEFINED;
-	zsfree(shf->filename);
+	dircache_set(&shf->filename, NULL);
+	/* Full filename, don't use dircache */
 	shf->filename = fname;
     }
     popheap();
@@ -5619,6 +5622,8 @@ runshfunc(Eprog prog, FuncWrap wrap, char *name)
  * Search fpath for an undefined function.  Finds the file, and returns the
  * list of its contents.
  *
+ * If test is 0, load the function; *fname is set to zalloc'ed location.
+ *
  * If test_only is 1, don't load function, just test for it:
  * - Non-null return means function was found
  * - *fname points to path at which found (not duplicated)
diff --git a/Src/hashtable.c b/Src/hashtable.c
index 2a8b585..110cea6 100644
--- a/Src/hashtable.c
+++ b/Src/hashtable.c
@@ -889,7 +889,7 @@ freeshfuncnode(HashNode hn)
 	freeeprog(shf->funcdef);
     if (shf->redir)
 	freeeprog(shf->redir);
-    zsfree(shf->filename);
+    dircache_set(&shf->filename, NULL);
     if (shf->sticky) {
 	if (shf->sticky->n_on_opts)
 	    zfree(shf->sticky->on_opts,
@@ -1442,3 +1442,134 @@ freehistdata(Histent he, int unlink)
 	}
     }
 }
+
+
+/***********************************************************************
+ * Directory name cache mechanism
+ *
+ * The idea of this is that there are various shell structures,
+ * notably functions, that record the directories with which they
+ * are associated.  Rather than store the full string each time,
+ * we store a pointer to the same location and count the references.
+ * This is optimised so that retrieval is quick at the expense of
+ * searching the list when setting up the structure, which is a much
+ * rarer operation.
+ *
+ * There is nothing special about the fact that the strings are
+ * directories, except for the assumptions for efficiency that many
+ * structures will point to the same one, and that there are not too
+ * many different directories associated with the shell.
+ **********************************************************************/
+
+struct dircache_entry
+{
+    /* Name of directory in cache */
+    char *name;
+    /* Number of references to it */
+    int refs;
+};
+
+/*
+ * dircache is the cache, of length dircache_size.
+ * dircache_lastentry is the last entry used, an optimisation
+ * for multiple references to the same directory, e.g
+ * "autoload /blah/blah/\*".
+ */
+struct dircache_entry *dircache, *dircache_lastentry;
+int dircache_size;
+
+/*
+ * Set *name to point to a cached version of value.
+ * value is copied so may come from any source.
+ *
+ * If value is NULL, look for the existing value of *name (safe if this
+ * too is NULL) and remove a reference to it from the cache. If it's
+ * not found in the cache, it's assumed to be an allocated string and
+ * freed --- this currently occurs for a shell function that's been
+ * loaded as the filename is now a full path, not just a directory,
+ * though we may one day optimise this to a cached directory plus a
+ * name, too.  Note --- the function does *not* otherwise check
+ * if *name points to something already cached, so this is
+ * necessary any time *name may already be in the cache.
+ */
+
+/**/
+mod_export void
+dircache_set(char **name, char *value)
+{
+    struct dircache_entry *dcptr, *dcnew;
+
+    if (!value) {
+	if (!dircache_size || !*name)
+	    return;
+
+	for (dcptr = dircache; dcptr < dircache + dircache_size; dcptr++)
+	{
+	    /* Must be a pointer much, not a string match */
+	    if (*name == dcptr->name)
+	    {
+		--dcptr->refs;
+		if (!dcptr->refs) {
+		    ptrdiff_t ind = dcptr - dircache;
+		    zsfree(dcptr->name);
+		    --dircache_size;
+
+		    if (!dircache_size) {
+			zfree(dircache, sizeof(*dircache));
+			dircache = NULL;
+			return;
+		    }
+		    dcnew = (struct dircache_entry *)
+			zalloc(dircache_size * sizeof(*dcnew));
+		    if (ind)
+			memcpy(dcnew, dircache, ind * sizeof(*dcnew));
+		    if (ind < dircache_size)
+			memcpy(dcnew + ind, dcptr + 1,
+			       (dircache_size - ind) * sizeof(*dcnew));
+		    zfree(dircache, (dircache_size+1)*sizeof(*dcnew));
+		    dircache = dcnew;
+		    dircache_lastentry = NULL;
+		}
+		*name = NULL;
+		return;
+	    }
+	}
+	zsfree(*name);
+	*name = NULL;
+    } else {
+	/*
+	 * We'll maintain the cache at exactly the right size rather
+	 * than overallocating.  The rationale here is that typically
+	 * we'll get a lot of functions in a small number of directories
+	 * so the complexity overhead of maintaining a separate count
+	 * isn't really matched by the efficiency gain.
+ 	 */
+	if (!dircache_size) {
+	    dircache_size = 1;
+	    dcptr = dircache =
+		(struct dircache_entry *)zalloc(sizeof(*dircache));
+	} else if (dircache_lastentry &&
+		   !strcmp(value, dircache_lastentry->name)) {
+	    *name = dircache_lastentry->name;
+	    ++dircache_lastentry->refs;
+	    return;
+	} else {
+	    for (dcptr = dircache; dcptr < dircache + dircache_size; dcptr++)
+	    {
+		if (!strcmp(value, dcptr->name)) {
+		    *name = dcptr->name;
+		    ++dcptr->refs;
+		    return;
+		}
+	    }
+	    ++dircache_size;
+	    dircache = (struct dircache_entry *)
+		zrealloc(dircache, sizeof(*dircache) * dircache_size);
+	    dcptr = dircache + dircache_size - 1;
+	}
+	dcptr->name = ztrdup(value);
+	*name = dcptr->name;
+	dcptr->refs = 1;
+	dircache_lastentry = dcptr;
+    }
+}
diff --git a/Src/signals.c b/Src/signals.c
index 9e05add..2de5743 100644
--- a/Src/signals.c
+++ b/Src/signals.c
@@ -811,6 +811,7 @@ dosavetrap(int sig, int level)
 	    newshf->node.nam = ztrdup(shf->node.nam);
 	    newshf->node.flags = shf->node.flags;
 	    newshf->funcdef = dupeprog(shf->funcdef, 0);
+	    /* Assume this is a real filename, don't use dircache */
 	    newshf->filename = ztrdup(shf->filename);
 	    if (shf->sticky) {
 		newshf->sticky = sticky_emulation_dup(shf->sticky, 0);

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Tue, 10 Jan 2017 19:31:02 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
>     Add features associated with autoloading a function using an absolute
>     path.

I pushed this, with the merge fixed.


The main feature most people might be interested in is that you can
now replace

fpath=(/path/to/my/functions)
autoload -Uz /path/to/my/functions/*(.:t)

with the more direct

autoload -Uz /path/to/my/functions/*(.)

The advantages are

- Your are guaranteed the right file will be autoloaded as $fpath is
not used.  If you previously autoloaded the file generically the record
is updated.

- "functions", "which" etc. tell you the directory that will be used.

The main disadvantage is each function record now contains the name of
the directory, so this takes extra memory --- it looks like a couple of
dozen k in the case of my function directory which is quite full.  This
could be optimised with indirection but I'm not sure it's worth the work.


The other possibility I can see being of general interest is that if you
have code that needs functions as a prerequisite, you can make the
calling code fail if the function isn't found immediately at the point
of the autoload by using the -R option, which also ensures that later
changes to fpath don't change the function that will be loaded.

pws

Re: PATH: autoload with explicit path

[Peter Stephenson]

Here is an improved patch for this with the features noted in the commit
message from the branch.

commit 6f74a54d0cc48e9d9928f18eaead0460963f7d95
Author: Peter Stephenson <p.w.stephenson@ntlworld.com>
Date:   Thu Dec 15 18:57:57 2016 +0000

    Add features associated with autoloading a function using an absolute
    path.
    
    -d defaults to normal fpath
    
    -r remembers the path without actually loading.  May be combined with -d.
    
    -R does the same but it's an error if not found
    
    -X can now take a directory path: this is used to output not yet loaded
    functions that have an associated path.

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index af336ef..23ccf76 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -147,20 +147,39 @@ ifnzman(noderef(Aliasing)).
 findex(autoload)
 cindex(functions, autoloading)
 cindex(autoloading functions)
-item(tt(autoload) [ {tt(PLUS())|tt(-)}tt(TUXkmtz) ] [ tt(-w) ] [ var(name) ... ])(
+item(tt(autoload) [ {tt(PLUS())|tt(-)}tt(RTUXdkmrtz) ] [ tt(-w) ] [ var(name) ... ])(
 vindex(fpath, searching)
-Equivalent to tt(functions -u), with the exception of tt(-X)/tt(+X) and
-tt(-w).  See the section `Autoloading Functions' in ifzman(zmanref(zshmisc))\
+See the section `Autoloading Functions' in ifzman(zmanref(zshmisc))\
 ifnzman(noderef(Functions)) for full details.  The tt(fpath) parameter
 will be searched to find the function definition when the function is
 first referenced.
 
-The flag tt(-X) may be used only inside a shell function, and may not be
-followed by a var(name).  It causes the calling function to be marked for
-autoloading and then immediately loaded and executed, with the current
-array of positional parameters as arguments.  This replaces the previous
-definition of the function.  If no function definition is found, an error
-is printed and the function remains undefined and marked for autoloading.
+If var(name) consists of an absolute path, the function is defined to
+load from the file given (searching as usual for dump files in the given
+location).  The name of the function is the base name (non-directory
+part) of the file.  It is normally an error if the function is not found
+in the given location; however, if the option tt(-d) is given, searching
+for the function defaults to tt($fpath).
+
+If the option tt(-r) or tt(-R) is given, the function is searched for
+immediately and the location is recorded internally for use when the
+function is executed; a relative path is expanded using the value of
+tt($PWD).  This protects against a change to tt($fpath) after the call
+to tt(autoload).  With tt(-r), if the function is not found, it is
+silently left unresolved until execution; with tt(-R), an error message
+is printed and command processing aborted immediately the search fails,
+i.e. at the tt(autoload) command rather than at function execution..
+
+The flag tt(-X) may be used only inside a shell function.  It causes the
+calling function to be marked for autoloading and then immediately
+loaded and executed, with the current array of positional parameters as
+arguments.  This replaces the previous definition of the function.  If
+no function definition is found, an error is printed and the function
+remains undefined and marked for autoloading.  If an argument is given,
+it is used as a directory (i.e. it does not include the name of the
+function) in which the function is to be found; this may be combined
+with the tt(-d) option to allow the function search to default to tt($fpath)
+if it is not in the given location.
 
 The flag tt(+X) attempts to load each var(name) as an autoloaded function,
 but does em(not) execute it.  The exit status is zero (success) if the
@@ -176,6 +195,13 @@ If the tt(-m) flag is also given each var(name) is treated as a
 pattern and all functions already marked for autoload that match the
 pattern are loaded.
 
+With the tt(-t) flag, turn on execution tracing; with tt(-T), turn on
+execution tracing only for the current function, turning it off on entry
+to any called functions that do not also have tracing enabled.
+
+With the tt(-U) flag, alias expansion is suppressed when the function is
+loaded.
+
 With the tt(-w) flag, the var(name)s are taken as names of files compiled
 with the tt(zcompile) builtin, and all functions defined in them are
 marked for autoloading.
@@ -193,6 +219,10 @@ example(emulate zsh -c 'autoload -Uz var(func)')
 
 arranges that when var(func) is loaded the shell is in native tt(zsh)
 emulation, and this emulation is also applied when var(func) is run.
+
+Some of the functions of tt(autoload) are also provided by tt(functions
+-u) or tt(functions -U), but tt(autoload) is a more comprehensive
+interface.
 )
 findex(bg)
 cindex(jobs, backgrounding)
@@ -811,7 +841,8 @@ xitem(tt(functions -M) var(mathfn) [ var(min) [ var(max) [ var(shellfn) ] ] ])
 xitem(tt(functions -M) [ tt(-m) var(pattern) ... ])
 item(tt(functions +M) [ tt(-m) ] var(mathfn) ... )(
 Equivalent to tt(typeset -f), with the exception of the tt(-x) and
-tt(-M) options.
+tt(-M) options.  For tt(functions -u) and tt(functions -U), see
+tt(autoload), which provides additional options.
 
 The tt(-x) option indicates that any functions output will have
 each leading tab for indentation, added by the shell to show syntactic
@@ -2034,7 +2065,9 @@ expansion to be suppressed when the function is loaded.  See the
 description of the `tt(autoload)' builtin for details.
 
 Note that the builtin tt(functions) provides the same basic capabilities
-as tt(typeset -f) but gives access to a few extra options.
+as tt(typeset -f) but gives access to a few extra options; tt(autoload)
+gives further additional options for the case tt(typeset -fu) and
+tt(typeset -fU).
 )
 item(tt(-h))(
 Hide: only useful for special parameters (those marked `<S>' in the table in
diff --git a/README b/README
index 414ee0b..1cce246 100644
--- a/README
+++ b/README
@@ -32,7 +32,7 @@ details, see the documentation.
 Incompatibilities since 5.3.1
 -----------------------------
 
-The default behaviour of code like the following has changed:
+1) The default behaviour of code like the following has changed:
 
   alias foo='noglob foo'
   foo() { print function body; }
@@ -52,8 +52,18 @@ ALIAS_FUNC_DEF, has been added, which can be set to make the shell
 behave as in previous versions.  It is in any case recommended to use
 the "function" keyword, as aliases are not expanded afterwards.
 
-Incompatibilities between 5.0.8 and 5.3.1
------------------------------------------
+2) It was an undocumented, and largely useless, feature that a function
+autoloaded with an absolute path was searched for along the normal fpath
+(as if the leading / was missing) and, if found, loaded under the full
+name including the leading slash.  This has been replaced with the more
+useful feature that the function is searched for only at the given
+absolute path; the name of the function is the base name of the file.
+Note that functions including a non-leading / behave as before,
+e.g. if `dir/name' is found anywhere under a directory in $fpath it is
+loaded as a function named `dir/name'.
+
+Incompatibilities between 5.0.8 and 5.3
+----------------------------------------
 
 1) In character classes delimited by "[" and "]" within patterns, whether
 used for filename generation (globbing) or other forms of pattern
@@ -182,6 +192,10 @@ following example illustrates how this differs from past versions.
      4 4 => 1  |  4 4 => 0   **
      4 5 => 1  |  4 5 => 1
 
+<<<<<<< HEAD
+=======
+
+>>>>>>> Add features associated with autoloading a function using an absolute
 The behaviour of the parameter flag (P) has changed when it appears
 in a nested parameter group, in order to make it more useful in
 such cases.  A (P) in the outermost parameter group behaves as
diff --git a/Src/builtin.c b/Src/builtin.c
index 0f04d14..78d67ca 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -46,7 +46,7 @@ static struct builtin builtins[] =
     BUILTIN(".", BINF_PSPECIAL, bin_dot, 1, -1, 0, NULL, NULL),
     BUILTIN(":", BINF_PSPECIAL, bin_true, 0, -1, 0, NULL, NULL),
     BUILTIN("alias", BINF_MAGICEQUALS | BINF_PLUSOPTS, bin_alias, 0, -1, 0, "Lgmrs", NULL),
-    BUILTIN("autoload", BINF_PLUSOPTS, bin_functions, 0, -1, 0, "mktTUwXz", "u"),
+    BUILTIN("autoload", BINF_PLUSOPTS, bin_functions, 0, -1, 0, "dmktrRTUwXz", "u"),
     BUILTIN("bg", 0, bin_fg, 0, -1, BIN_BG, NULL, NULL),
     BUILTIN("break", BINF_PSPECIAL, bin_break, 0, 1, BIN_BREAK, NULL, NULL),
     BUILTIN("bye", 0, bin_break, 0, 1, BIN_EXIT, NULL, NULL),
@@ -2922,9 +2922,59 @@ eval_autoload(Shfunc shf, char *name, Options ops, int func)
     }
 
     return !loadautofn(shf, (OPT_ISSET(ops,'k') ? 2 :
-			     (OPT_ISSET(ops,'z') ? 0 : 1)), 1);
+			     (OPT_ISSET(ops,'z') ? 0 : 1)), 1,
+		       OPT_ISSET(ops,'d'));
 }
 
+/* Helper for bin_functions() for -X and -r options */
+
+/**/
+static int
+check_autoload(Shfunc shf, char *name, Options ops, int func)
+{
+    if (OPT_ISSET(ops,'X'))
+    {
+	return eval_autoload(shf, name, ops, func);
+    }
+    if (OPT_ISSET(ops,'r') || OPT_ISSET(ops,'R'))
+    {
+	char *dir_path;
+	if (shf->filename) {
+	    char *spec_path[2];
+	    spec_path[0] = shf->filename;
+	    spec_path[1] = NULL;
+	    if (getfpfunc(shf->node.nam, NULL, &dir_path, spec_path, 1)) {
+		/* shf->filename is already correct. */
+		return 0;
+	    }
+	    if (!OPT_ISSET(ops,'d')) {
+		if (OPT_ISSET(ops,'R')) {
+		    zerr("%s: function definition file not found",
+			 shf->node.nam);
+		    return 1;
+		}
+		return 0;
+	    }
+	}
+	if (getfpfunc(shf->node.nam, NULL, &dir_path, NULL, 1)) {
+	    zsfree(shf->filename);
+	    if (*dir_path != '/') {
+		dir_path = zhtricat(metafy(zgetcwd(), -1, META_HEAPDUP),
+				    "/", dir_path);
+		dir_path = xsymlink(dir_path, 1);
+	    }
+	    shf->filename = ztrdup(dir_path);
+	    return 0;
+	}
+	if (OPT_ISSET(ops,'R')) {
+	    zerr("%s: function definition file not found",
+		 shf->node.nam);
+	    return 1;
+	}
+	/* with -r, we don't flag an error, just let it be found later. */
+    }
+    return 0;
+}
 
 /* List a user-defined math function. */
 static void
@@ -2962,6 +3012,28 @@ listusermathfunc(MathFunc p)
 }
 
 
+static void
+add_autoload_function(Shfunc shf, char *funcname)
+{
+    char *nam;
+    if (*funcname == '/' && funcname[1] &&
+	(nam = strrchr(funcname, '/')) && nam[1]) {
+	char *dir;
+	nam = strrchr(funcname, '/');
+	if (nam == funcname) {
+	    dir = "/";
+	} else {
+	    *nam++ = '\0';
+	    dir = funcname;
+	}
+	zsfree(shf->filename);
+	shf->filename = ztrdup(dir);
+	shfunctab->addnode(shfunctab, ztrdup(nam), shf);
+    } else {
+	shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
+    }
+}
+
 /* Display or change the attributes of shell functions.   *
  * If called as autoload, it will define a new autoloaded *
  * (undefined) shell function.                            */
@@ -3007,10 +3079,17 @@ bin_functions(char *name, char **argv, Options ops, int func)
 	off |= PM_KSHSTORED;
 	roff |= PM_KSHSTORED;
     }
+    if (OPT_MINUS(ops,'d')) {
+	on |= PM_CUR_FPATH;
+	off |= PM_CUR_FPATH;
+    } else if (OPT_PLUS(ops,'d')) {
+	off |= PM_CUR_FPATH;
+	roff |= PM_CUR_FPATH;
+    }
 
     if ((off & PM_UNDEFINED) || (OPT_ISSET(ops,'k') && OPT_ISSET(ops,'z')) ||
 	(OPT_ISSET(ops,'x') && !OPT_HASARG(ops,'x')) ||
-	(OPT_MINUS(ops,'X') && (OPT_ISSET(ops,'m') || *argv || !scriptname))) {
+	(OPT_MINUS(ops,'X') && (OPT_ISSET(ops,'m') || !scriptname))) {
 	zwarnnam(name, "invalid option(s)");
 	return 1;
     }
@@ -3165,47 +3244,55 @@ bin_functions(char *name, char **argv, Options ops, int func)
 	return returnval;
     }
 
-    /* If no arguments given, we will print functions.  If flags *
-     * are given, we will print only functions containing these  *
-     * flags, else we'll print them all.                         */
-    if (!*argv) {
-	int ret = 0;
-
+    if (OPT_MINUS(ops,'X')) {
+	Funcstack fs;
+	char *funcname = NULL;
+	int ret;
+	if (*argv && argv[1]) {
+	    zwarnnam(name, "-X: too many arguments");
+	    return 1;
+	}
 	queue_signals();
-	if (OPT_MINUS(ops,'X')) {
-	    Funcstack fs;
-	    char *funcname = NULL;
-	    for (fs = funcstack; fs; fs = fs->prev) {
-		if (fs->tp == FS_FUNC) {
-		    /*
-		     * dupstring here is paranoia but unlikely to be
-		     * problematic
-		     */
-		    funcname = dupstring(fs->name);
-		    break;
-		}
+	for (fs = funcstack; fs; fs = fs->prev) {
+	    if (fs->tp == FS_FUNC) {
+		/*
+		 * dupstring here is paranoia but unlikely to be
+		 * problematic
+		 */
+		funcname = dupstring(fs->name);
+		break;
 	    }
-	    if (!funcname)
-	    {
-		zerrnam(name, "bad autoload");
-		ret = 1;
+	}
+	if (!funcname)
+	{
+	    zerrnam(name, "bad autoload");
+	    ret = 1;
+	} else {
+	    if ((shf = (Shfunc) shfunctab->getnode(shfunctab, funcname))) {
+		DPUTS(!shf->funcdef,
+		      "BUG: Calling autoload from empty function");
 	    } else {
-		if ((shf = (Shfunc) shfunctab->getnode(shfunctab, funcname))) {
-		    DPUTS(!shf->funcdef,
-			  "BUG: Calling autoload from empty function");
-		} else {
-		    shf = (Shfunc) zshcalloc(sizeof *shf);
-		    shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
-		}
-		shf->node.flags = on;
-		ret = eval_autoload(shf, funcname, ops, func);
+		shf = (Shfunc) zshcalloc(sizeof *shf);
+		shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
 	    }
-	} else {
-	    if (OPT_ISSET(ops,'U') && !OPT_ISSET(ops,'u'))
+	    if (*argv)
+		shf->filename = ztrdup(*argv);
+	    shf->node.flags = on;
+	    ret = eval_autoload(shf, funcname, ops, func);
+	}
+	unqueue_signals();
+	return ret;
+    } else if (!*argv) {
+	/* If no arguments given, we will print functions.  If flags *
+	 * are given, we will print only functions containing these  *
+	 * flags, else we'll print them all.                         */
+	int ret = 0;
+
+	queue_signals();
+	if (OPT_ISSET(ops,'U') && !OPT_ISSET(ops,'u'))
 		on &= ~PM_UNDEFINED;
 	    scanshfunc(1, on|off, DISABLED, shfunctab->printnode,
 		       pflags, expand);
-	}
 	unqueue_signals();
 	return ret;
     }
@@ -3231,8 +3318,8 @@ bin_functions(char *name, char **argv, Options ops, int func)
 				!(shf->node.flags & DISABLED)) {
 				shf->node.flags = (shf->node.flags |
 					      (on & ~PM_UNDEFINED)) & ~off;
-				if (OPT_ISSET(ops,'X') &&
-				    eval_autoload(shf, shf->node.nam, ops, func)) {
+				if (check_autoload(shf, shf->node.nam,
+						   ops, func)) {
 				    returnval = 1;
 				}
 			    }
@@ -3258,8 +3345,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 	    if (on|off) {
 		/* turn on/off the given flags */
 		shf->node.flags = (shf->node.flags | (on & ~PM_UNDEFINED)) & ~off;
-		if (OPT_ISSET(ops,'X') &&
-		    eval_autoload(shf, shf->node.nam, ops, func))
+		if (check_autoload(shf, shf->node.nam, ops, func))
 		    returnval = 1;
 	    } else
 		/* no flags, so just print */
@@ -3282,7 +3368,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 	    shf->node.flags = on;
 	    shf->funcdef = mkautofn(shf);
 	    shfunc_set_sticky(shf);
-	    shfunctab->addnode(shfunctab, ztrdup(*argv), shf);
+	    add_autoload_function(shf, *argv);
 
 	    if (signum != -1) {
 		if (settrap(signum, NULL, ZSIG_FUNC)) {
@@ -3293,8 +3379,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 		}
 	    }
 
-	    if (ok && OPT_ISSET(ops,'X') &&
-		eval_autoload(shf, shf->node.nam, ops, func))
+	    if (ok && check_autoload(shf, shf->node.nam, ops, func))
 		returnval = 1;
 	} else
 	    returnval = 1;
diff --git a/Src/exec.c b/Src/exec.c
index a439aec..a41d05b 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -3124,7 +3124,7 @@ execcmd_exec(Estate state, Execcmd_params eparams,
 	if (is_shfunc)
 	    shf = (Shfunc)hn;
 	else {
-	    shf = loadautofn(state->prog->shf, 1, 0);
+	    shf = loadautofn(state->prog->shf, 1, 0, 0);
 	    if (shf)
 		state->prog->shf = shf;
 	    else {
@@ -5142,7 +5142,7 @@ execautofn(Estate state, UNUSED(int do_exec))
 {
     Shfunc shf;
 
-    if (!(shf = loadautofn(state->prog->shf, 1, 0)))
+    if (!(shf = loadautofn(state->prog->shf, 1, 0, 0)))
 	return 1;
 
     state->prog->shf = shf;
@@ -5151,7 +5151,7 @@ execautofn(Estate state, UNUSED(int do_exec))
 
 /**/
 Shfunc
-loadautofn(Shfunc shf, int fksh, int autol)
+loadautofn(Shfunc shf, int fksh, int autol, int current_fpath)
 {
     int noalias = noaliases, ksh = 1;
     Eprog prog;
@@ -5160,7 +5160,18 @@ loadautofn(Shfunc shf, int fksh, int autol)
     pushheap();
 
     noaliases = (shf->node.flags & PM_UNALIASED);
-    prog = getfpfunc(shf->node.nam, &ksh, &fname);
+    if (shf->filename && shf->filename[0] == '/')
+    {
+	char *spec_path[2];
+	spec_path[0] = dupstring(shf->filename);
+	spec_path[1] = NULL;
+	prog = getfpfunc(shf->node.nam, &ksh, &fname, spec_path, 0);
+	if (prog == &dummy_eprog &&
+	    (current_fpath || (shf->node.flags & PM_CUR_FPATH)))
+	    prog = getfpfunc(shf->node.nam, &ksh, &fname, NULL, 0);
+    }
+    else
+	prog = getfpfunc(shf->node.nam, &ksh, &fname, NULL, 0);
     noaliases = noalias;
 
     if (ksh == 1) {
@@ -5602,12 +5613,18 @@ runshfunc(Eprog prog, FuncWrap wrap, char *name)
     unqueue_signals();
 }
 
-/* Search fpath for an undefined function.  Finds the file, and returns the *
- * list of its contents.                                                    */
+/*
+ * Search fpath for an undefined function.  Finds the file, and returns the
+ * list of its contents.
+ *
+ * If test_only is 1, don't load function, just test for it:
+ * - Non-null return means function was found
+ * - *fname points to path at which found (not duplicated)
+ */
 
 /**/
 Eprog
-getfpfunc(char *s, int *ksh, char **fname)
+getfpfunc(char *s, int *ksh, char **fname, char **alt_path, int test_only)
 {
     char **pp, buf[PATH_MAX+1];
     off_t len;
@@ -5616,7 +5633,7 @@ getfpfunc(char *s, int *ksh, char **fname)
     Eprog r;
     int fd;
 
-    pp = fpath;
+    pp = alt_path ? alt_path : fpath;
     for (; *pp; pp++) {
 	if (strlen(*pp) + strlen(s) + 1 >= PATH_MAX)
 	    continue;
@@ -5624,9 +5641,9 @@ getfpfunc(char *s, int *ksh, char **fname)
 	    sprintf(buf, "%s/%s", *pp, s);
 	else
 	    strcpy(buf, s);
-	if ((r = try_dump_file(*pp, s, buf, ksh))) {
+	if ((r = try_dump_file(*pp, s, buf, ksh, test_only))) {
 	    if (fname)
-		*fname = ztrdup(buf);
+		*fname = test_only ? *pp : ztrdup(buf);
 	    return r;
 	}
 	unmetafy(buf, NULL);
@@ -5634,6 +5651,11 @@ getfpfunc(char *s, int *ksh, char **fname)
 	    struct stat st;
 	    if (!fstat(fd, &st) && S_ISREG(st.st_mode) &&
 		(len = lseek(fd, 0, 2)) != -1) {
+		if (test_only) {
+		    close(fd);
+		    *fname = *pp;
+		    return &dummy_eprog;
+		}
 		d = (char *) zalloc(len + 1);
 		lseek(fd, 0, 0);
 		if ((rlen = read(fd, d, len)) >= 0) {
@@ -5661,7 +5683,7 @@ getfpfunc(char *s, int *ksh, char **fname)
 		close(fd);
 	}
     }
-    return &dummy_eprog;
+    return test_only ? NULL : &dummy_eprog;
 }
 
 /* Handle the most common type of ksh-style autoloading, when doing a      *
diff --git a/Src/hashtable.c b/Src/hashtable.c
index 7c33675..2a8b585 100644
--- a/Src/hashtable.c
+++ b/Src/hashtable.c
@@ -949,16 +949,20 @@ printshfuncnode(HashNode hn, int printflags)
 	    zoutputtab(stdout);
 	}
 	if (!t) {
-	    char *fopt = "UtTkz";
+	    char *fopt = "UtTkzc";
 	    int flgs[] = {
 		PM_UNALIASED, PM_TAGGED, PM_TAGGED_LOCAL,
-		PM_KSHSTORED, PM_ZSHSTORED, 0
+		PM_KSHSTORED, PM_ZSHSTORED, PM_CUR_FPATH, 0
 	    };
 	    int fl;;
 
 	    zputs("builtin autoload -X", stdout);
 	    for (fl=0;fopt[fl];fl++)
 		if (f->node.flags & flgs[fl]) putchar(fopt[fl]);
+	    if (f->filename) {
+		putchar(' ');
+		zputs(f->filename, stdout);
+	    }
 	} else {
 	    zputs(t, stdout);
 	    zsfree(t);
diff --git a/Src/parse.c b/Src/parse.c
index ed6c4a8..314cc09 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -3338,7 +3338,7 @@ cur_add_func(char *nam, Shfunc shf, LinkList names, LinkList progs,
 	    return 1;
 	}
 	noaliases = (shf->node.flags & PM_UNALIASED);
-	if (!(prog = getfpfunc(shf->node.nam, NULL, NULL)) ||
+	if (!(prog = getfpfunc(shf->node.nam, NULL, NULL, NULL, 0)) ||
 	    prog == &dummy_eprog) {
 	    noaliases = ona;
 	    zwarnnam(nam, "can't load function: %s", shf->node.nam);
@@ -3580,7 +3580,7 @@ load_dump_file(char *dump, struct stat *sbuf, int other, int len)
 
 /**/
 Eprog
-try_dump_file(char *path, char *name, char *file, int *ksh)
+try_dump_file(char *path, char *name, char *file, int *ksh, int test_only)
 {
     Eprog prog;
     struct stat std, stc, stn;
@@ -3589,7 +3589,7 @@ try_dump_file(char *path, char *name, char *file, int *ksh)
 
     if (strsfx(FD_EXT, path)) {
 	queue_signals();
-	prog = check_dump_file(path, NULL, name, ksh);
+	prog = check_dump_file(path, NULL, name, ksh, test_only);
 	unqueue_signals();
 	return prog;
     }
@@ -3608,14 +3608,14 @@ try_dump_file(char *path, char *name, char *file, int *ksh)
     if (!rd &&
 	(rc || std.st_mtime > stc.st_mtime) &&
 	(rn || std.st_mtime > stn.st_mtime) &&
-	(prog = check_dump_file(dig, &std, name, ksh))) {
+	(prog = check_dump_file(dig, &std, name, ksh, test_only))) {
 	unqueue_signals();
 	return prog;
     }
     /* No digest file. Now look for the per-function compiled file. */
     if (!rc &&
 	(rn || stc.st_mtime > stn.st_mtime) &&
-	(prog = check_dump_file(wc, &stc, name, ksh))) {
+	(prog = check_dump_file(wc, &stc, name, ksh, test_only))) {
 	unqueue_signals();
 	return prog;
     }
@@ -3643,7 +3643,7 @@ try_source_file(char *file)
 
     if (strsfx(FD_EXT, file)) {
 	queue_signals();
-	prog = check_dump_file(file, NULL, tail, NULL);
+	prog = check_dump_file(file, NULL, tail, NULL, 0);
 	unqueue_signals();
 	return prog;
     }
@@ -3654,7 +3654,7 @@ try_source_file(char *file)
 
     queue_signals();
     if (!rc && (rn || stc.st_mtime > stn.st_mtime) &&
-	(prog = check_dump_file(wc, &stc, tail, NULL))) {
+	(prog = check_dump_file(wc, &stc, tail, NULL, 0))) {
 	unqueue_signals();
 	return prog;
     }
@@ -3667,7 +3667,8 @@ try_source_file(char *file)
 
 /**/
 static Eprog
-check_dump_file(char *file, struct stat *sbuf, char *name, int *ksh)
+check_dump_file(char *file, struct stat *sbuf, char *name, int *ksh,
+		int test_only)
 {
     int isrec = 0;
     Wordcode d;
@@ -3709,6 +3710,11 @@ check_dump_file(char *file, struct stat *sbuf, char *name, int *ksh)
     if ((h = dump_find_func(d, name))) {
 	/* Found the name. If the file is already mapped, return the eprog,
 	 * otherwise map it and just go up. */
+	if (test_only)
+	{
+	    /* This is all we need.  Just return dummy. */
+	    return &dummy_eprog;
+	}
 
 #ifdef USE_MMAP
 
@@ -3745,7 +3751,7 @@ check_dump_file(char *file, struct stat *sbuf, char *name, int *ksh)
 
 #endif
 
-	    {
+	{
 	    Eprog prog;
 	    Patprog *pp;
 	    int np, fd, po = h->npats * sizeof(Patprog);
diff --git a/Src/zsh.h b/Src/zsh.h
index 2a41638..67c5a35 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -1233,7 +1233,9 @@ struct cmdnam {
 
 struct shfunc {
     struct hashnode node;
-    char *filename;             /* Name of file located in */
+    char *filename;             /* Name of file located in.
+				   For not yet autoloaded file, name
+				   of explicit directory, if not NULL. */
     zlong lineno;		/* line number in above file */
     Eprog funcdef;		/* function definition    */
     Eprog redir;                /* redirections to apply */
@@ -1811,6 +1813,7 @@ struct tieddata {
 #define PM_UNALIASED	(1<<13)	/* do not expand aliases when autoloading   */
 
 #define PM_HIDE		(1<<14)	/* Special behaviour hidden by local        */
+#define PM_CUR_FPATH    (1<<14) /* (function): can use $fpath with filename */
 #define PM_HIDEVAL	(1<<15)	/* Value not shown in `typeset' commands    */
 #define PM_TIED 	(1<<16)	/* array tied to colon-path or v.v.         */
 #define PM_TAGGED_LOCAL (1<<16) /* (function): non-recursive PM_TAGGED      */
diff --git a/Test/C04funcdef.ztst b/Test/C04funcdef.ztst
index ab7b429..1821b78 100644
--- a/Test/C04funcdef.ztst
+++ b/Test/C04funcdef.ztst
@@ -330,6 +330,95 @@
 0:whence -v of zsh-style autoload
 >oops is a shell function from ./oops
 
+  (
+    fpath=(.)
+    mkdir extra
+    print 'print "I have been loaded by explicit path."' >extra/spec
+    autoload -Uz $PWD/extra/spec
+    spec
+  )
+0:autoload with explicit path
+>I have been loaded by explicit path.
+
+  (
+    fpath=(.)
+    print 'print "I have been loaded by default path."' >def
+    autoload -Uz $PWD/extra/def
+    def
+  )
+1:autoload with explicit path with function in normal path, no -d
+?(eval):5: def: function definition file not found
+
+  (
+    fpath=(.)
+    autoload -dUz $PWD/extra/def
+    def
+  )
+0:autoload with explicit path with function in normal path, with -d
+>I have been loaded by default path.
+
+  (
+    cd extra
+    fpath=(.)
+    autoload -r spec
+    cd ..
+    spec
+  )
+0:autoload -r
+>I have been loaded by explicit path.
+
+  (
+    cd extra
+    fpath=(.)
+    autoload -r def
+    cd ..
+    def
+  )
+0:autoload -r is permissive
+>I have been loaded by default path.
+
+  (
+    cd extra
+    fpath=(.)
+    autoload -R def
+  )
+1:autoload -R is not permissive
+?(eval):4: def: function definition file not found
+
+  (
+    spec() { autoload -XUz $PWD/extra; }
+    spec
+  )
+0:autoload -X with path
+>I have been loaded by explicit path.
+
+# The line number 1 here and in the next test seems suspect,
+# but this example proves it's not down to the new features
+# being tested here.
+  (
+    fpath=(.)
+    cod() { autoload -XUz; }
+    cod
+  )
+1:autoload -X with no path, failure
+?(eval):1: cod: function definition file not found
+
+  (
+    fpath=(.)
+    def() { autoload -XUz $PWD/extra; }
+    def
+  )
+1:autoload -X with wrong path and no -d
+?(eval):1: def: function definition file not found
+
+  (
+    fpath=(.)
+    def() { autoload -dXUz $PWD/extra; }
+    def
+  )
+0:autoload -dX with path
+>I have been loaded by default path.
+
 %clean
 
  rm -f file.in file.out

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Dec 12,  4:31pm, Peter Stephenson wrote:
}
} We're at the point where autoload isn't really a synonym of functions -u
} any more.  I'd be tempted to document "functions -u" as "implements a
} limited subset of..." rather than "equivalent to...".  (Does anyone use
} that interface except for compatibility?)

There's
  autoload
  functions -u
  typeset -f -u

but there are aready some variations of each that are not supported by
the others, so which ones say they are "equivalent to" which others is
probably overdue for an overhaul.

For example I don't think there's any way to autoload a math function in
a single command -- you have to autoload it and then mark it as math, or
something like that.

Re: PATH: autoload with explicit path

[Peter Stephenson]

On Mon, 12 Dec 2016 08:05:50 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> If I understand what you're saying, that means
> 
>     autoload this/here/file
> 
> creates a function named "this/here/file" by loading "file" from
> $^fpath/this/here  ... yes?
> 
> And then (before this patch)
> 
>     autoload /this/here/file
> 
> creates a function named "/this/here/file" from the same location?
> 
> So after this patch there is no way to autoload a function whose name
> begins with a slash.  Not that it was especially useful to be able to
> do so, just confirming.

Yes, that all looks like it's correct.

> If I may make a suggestion -- an equally useful variation would be for
> autoload to search the $fpath at the time of the autoload command and
> store the path where the file is found *iff* it is found.  This would
> be sort of like doing
> 
>     autoload -Uz $^fpath/myfunc(N)
> 
> except that it handles the situation where $fpath does not yet contain
> the "myfunc" file.
>
> Another useful variation might be to fall back to the current fpath if
> the function isn't found at the specified location.

These need some form of new syntax, but it looks like options are
probably good enough.  -r for remember the path where found (or -s for
save/store?) and -c for use current fpath as default (or -d for default?
anything with -f or -p being a bit vague in this context).

-R for remember and also ensure it's there with a hard error if it isn't
there would be possible if it seems useful --- it gives you a way you
don't currently have of expressing the fact the function is required and
you might as well bomb out immediately rather than once you're committed
to doing whatever.

Presumably -c should remember the path, even if it found it on $fpath
rather than the specified location, to avoid surprises.

We're at the point where autoload isn't really a synonym of functions -u
any more.  I'd be tempted to document "functions -u" as "implements a
limited subset of..." rather than "equivalent to...".  (Does anyone use
that interface except for compatibility?)

pws

Re: PATH: autoload with explicit path

[Bart Schaefer]

On Dec 11, 10:18pm, Peter Stephenson wrote:
}
} I wanted to autoload a function from an explicit path.
} [...] it strikes me it's a bit mean of the shell not simply to do
} something sensible with
} 
} autoload +X /path/to/myfunc

For +X only, this can be accomplished by

    FPATH=/path/to autoload +X myfunc

Sebastian's plugin manager uses this.

} That doesn't do anything sensible at the moment --- relative paths to
} functions do, with the full relative path as the name, which is entirely
} consistent, but absolute paths don't and in fact it tries the name as a
} relative path anyway which seems Just Plain Wrong.

If I understand what you're saying, that means

    autoload this/here/file

creates a function named "this/here/file" by loading "file" from
$^fpath/this/here  ... yes?

And then (before this patch)

    autoload /this/here/file

creates a function named "/this/here/file" from the same location?

So after this patch there is no way to autoload a function whose name
begins with a slash.  Not that it was especially useful to be able to
do so, just confirming.

} autoload -Uz /path/to/myfunc
} 
} defines myfunc to be found in the directory /path/to by reusing the
} filename element of the shfunc structure that's currently unused at this
} stage.
} 
} This seemed quite a useful feature.

Yes, this is specifically what Sebastian and (I think) Ray were asking
for several months ago.

If I may make a suggestion -- an equally useful variation would be for
autoload to search the $fpath at the time of the autoload command and
store the path where the file is found *iff* it is found.  This would
be sort of like doing

    autoload -Uz $^fpath/myfunc(N)

except that it handles the situation where $fpath does not yet contain
the "myfunc" file.

Another useful variation might be to fall back to the current fpath if
the function isn't found at the specified location.

PATH: autoload with explicit path

[Peter Stephenson]

Not to be committed until 5.3 is safely done and dusted, but you might
as well see it now.

I wanted to autoload a function from an explicit path.  (Not so auto, in
fact, but there isn't a trivial way of manually loading an autoload -z
format file without using autoload.)  I could rig something up in a
function, saving and restoring fpath or wrapping the contents of the
file, but it strikes me it's a bit mean of the shell not simply to do
something sensible with

autoload +X /path/to/myfunc

That doesn't do anything sensible at the moment --- relative paths to
functions do, with the full relative path as the name, which is entirely
consistent, but absolute paths don't and in fact it tries the name as a
relative path anyway which seems Just Plain Wrong.

Then I found it was pretty much as easy (easier, actually, since fewer
special cases) to implement this generally:

autoload -Uz /path/to/myfunc

defines myfunc to be found in the directory /path/to by reusing the
filename element of the shfunc structure that's currently unused at this
stage.

This seemed quite a useful feature.  No doc or tests at this stage.

Also TBD: output from "autoload" with no name.  Just needs to update the
output of the autoload -X line appropriately, so this is a minor detail,
I think.

Also TBD: I should clearly document that autoload has its normal
semantics i.e. if the function already exists that definition is kept.
Explicit path != forced reload.

Also TBD: should output a special error message if not found so
user knows where it was looking.

pws

diff --git a/Src/builtin.c b/Src/builtin.c
index 65e0cb1..96571df 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -2962,6 +2962,27 @@ listusermathfunc(MathFunc p)
 }
 
 
+static void
+add_autoload_function(Shfunc shf, char *funcname)
+{
+    char *nam;
+    if (*funcname == '/' && funcname[1] &&
+	(nam = strrchr(funcname, '/')) && nam[1]) {
+	char *dir;
+	nam = strrchr(funcname, '/');
+	if (nam == funcname) {
+	    dir = "/";
+	} else {
+	    *nam++ = '\0';
+	    dir = funcname;
+	}
+	shf->filename = ztrdup(dir);
+	shfunctab->addnode(shfunctab, ztrdup(nam), shf);
+    } else {
+	shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
+    }
+}
+
 /* Display or change the attributes of shell functions.   *
  * If called as autoload, it will define a new autoloaded *
  * (undefined) shell function.                            */
@@ -3195,7 +3216,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 			  "BUG: Calling autoload from empty function");
 		} else {
 		    shf = (Shfunc) zshcalloc(sizeof *shf);
-		    shfunctab->addnode(shfunctab, ztrdup(funcname), shf);
+		    add_autoload_function(shf, funcname);
 		}
 		shf->node.flags = on;
 		ret = eval_autoload(shf, funcname, ops, func);
@@ -3266,6 +3287,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 		printshfuncexpand(&shf->node, pflags, expand);
 	} else if (on & PM_UNDEFINED) {
 	    int signum = -1, ok = 1;
+	    char *nam;
 
 	    if (!strncmp(*argv, "TRAP", 4) &&
 		(signum = getsignum(*argv + 4)) != -1) {
@@ -3282,7 +3304,7 @@ bin_functions(char *name, char **argv, Options ops, int func)
 	    shf->node.flags = on;
 	    shf->funcdef = mkautofn(shf);
 	    shfunc_set_sticky(shf);
-	    shfunctab->addnode(shfunctab, ztrdup(*argv), shf);
+	    add_autoload_function(shf, *argv);
 
 	    if (signum != -1) {
 		if (settrap(signum, NULL, ZSIG_FUNC)) {
diff --git a/Src/exec.c b/Src/exec.c
index a439aec..6e4e99c 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -5155,12 +5155,23 @@ loadautofn(Shfunc shf, int fksh, int autol)
 {
     int noalias = noaliases, ksh = 1;
     Eprog prog;
-    char *fname;
+    char *fname, **alt_path, *spec_path[2];
 
     pushheap();
 
+    if (shf->filename && shf->filename[0] == '/')
+    {
+	spec_path[0] = dupstring(shf->filename);
+	spec_path[1] = NULL;
+	alt_path = spec_path;
+    }
+    else
+    {
+	alt_path = NULL;
+    }
+
     noaliases = (shf->node.flags & PM_UNALIASED);
-    prog = getfpfunc(shf->node.nam, &ksh, &fname);
+    prog = getfpfunc(shf->node.nam, &ksh, &fname, alt_path);
     noaliases = noalias;
 
     if (ksh == 1) {
@@ -5607,7 +5618,7 @@ runshfunc(Eprog prog, FuncWrap wrap, char *name)
 
 /**/
 Eprog
-getfpfunc(char *s, int *ksh, char **fname)
+getfpfunc(char *s, int *ksh, char **fname, char **alt_path)
 {
     char **pp, buf[PATH_MAX+1];
     off_t len;
@@ -5616,7 +5627,7 @@ getfpfunc(char *s, int *ksh, char **fname)
     Eprog r;
     int fd;
 
-    pp = fpath;
+    pp = alt_path ? alt_path : fpath;
     for (; *pp; pp++) {
 	if (strlen(*pp) + strlen(s) + 1 >= PATH_MAX)
 	    continue;
diff --git a/Src/parse.c b/Src/parse.c
index 50a0d5f..38c6da2 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -3324,7 +3324,7 @@ cur_add_func(char *nam, Shfunc shf, LinkList names, LinkList progs,
 	    return 1;
 	}
 	noaliases = (shf->node.flags & PM_UNALIASED);
-	if (!(prog = getfpfunc(shf->node.nam, NULL, NULL)) ||
+	if (!(prog = getfpfunc(shf->node.nam, NULL, NULL, NULL)) ||
 	    prog == &dummy_eprog) {
 	    noaliases = ona;
 	    zwarnnam(nam, "can't load function: %s", shf->node.nam);
diff --git a/Src/zsh.h b/Src/zsh.h
index f22d8b1..4fdd09a 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -1233,7 +1233,9 @@ struct cmdnam {
 
 struct shfunc {
     struct hashnode node;
-    char *filename;             /* Name of file located in */
+    char *filename;             /* Name of file located in.
+				   For not yet autoloaded file, name
+				   of explicit directory, if not NULL. */
     zlong lineno;		/* line number in above file */
     Eprog funcdef;		/* function definition    */
     Eprog redir;                /* redirections to apply */