autoload

autoload

[Ray Andrews]

I'm just reading that I can use autoload with my own functions (thought 
it was just for internal stuff ...).  Any words of wisdom on that?  Can 
we determine how much memory functions use when loaded, such that a 
practical determination might be made as to how much 'real' difference 
it might make to free memory vs. any speed considerations?  What do guys 
do in the real world?

. ./ray

Re: autoload

[Bart Schaefer]

On Dec 23,  7:51am, Ray Andrews wrote:
}
} Can we determine how much memory functions use when loaded, such that a 
} practical determination might be made as to how much 'real' difference 
} it might make to free memory vs. any speed considerations?

The in-memory size of a function is at least as large and probably a bit
larger than its on-disk size as a file, because it's stored in both
compiled and text form (the former for speed, the latter for output of
the verbose and xtrace options and/or the functions command).

} What do guys do in the real world?

I autoload practically everything.  I don't, however, zcompile things
very much.

Re: autoload

[Ray Andrews]

On 12/23/2014 09:36 AM, Bart Schaefer wrote:
> } What do guys do in the real world?
>
> I autoload practically everything.  I don't, however, zcompile things
> very much.
Ok, that's good enough reason to do it.

Announce of Zsh Navigation Tools

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

Hello,
I would like to announce Zsh Navigation Tools, a set of utilities based on
the tool's main component – n-list, a zcurses based selection list
supporting ANSI coloring and incremental search. The utilities are:

- n-aliases - browses aliases, relegates editing to vared
- n-cd - browses dirstack and bookmarked directories, allows to enter
selected directory
- n-env - browses environment, relegates editing to vared
- n-functions - browses functions, relegates editing to zed or vared
- n-history - browses history, allows to edit and run commands from it
- n-kill - browses processes list, allows to send signal to selected process
- n-options - browses options, allows to toggle their state
- n-panelize - loads output of given command into the list for browsing
- n-preview - a demonstration of running two lists at the same time and of
ANSI colors - syntax highlighted source browser

On GitHub page there is a link to video:

https://github.com/psprint/zsh-navigation-tools

The main idea behind the tools is that it is good to have navigation as
option side to completion.

Features not yet mentioned:
- initial grepping of what is browsed with "$1" given to utility
- making lists with non-selectable elements (to e.g. include headers in the
list)
- counting of how much elements there are (even when there are headers)
- jumping over predefined positions in the list with '[', ']' keys (used to
jump to bookmarks in 'n-cd' and through signals in 'n-kill')
- rich keys close to those of 'less' (like g, G, ctrl-u, ctrl-d, ctrl-p,
ctrl-n, /, j, k, home, end, etc.)

The code uses curses and regex modules. It is clean thanks to Zsh's rich
available math syntax when variable is declared as integer. N-preview will
show colored syntax if there is program "highlight" installed on the system.

Best regards,
Sebastian Gniazdowski

Re: Announce of Zsh Navigation Tools

[Ray Andrews]

On 09/08/2015 05:32 AM, Sebastian Gniazdowski wrote:
> Hello,
> I would like to announce Zsh Navigation Tools,
...

Thanks for those Sebastian, I'm playing with them now, several look very 
useful.  One thing, I had to create 
'/usr/local/share/zsh/site-functions'.  I wonder if there is a location 
for the files that is in keeping with my directory structure.  A trivial 
point: in the files, you call them 'functions' whereas they are scrips, 
I believe.  However, I'm tempted to convert them to functions anyway.

Re: Announce of Zsh Navigation Tools

[ZyX]

12.09.2015, 21:21, "Ray Andrews" <rayandrews@eastlink.ca>:
> On 09/08/2015 05:32 AM, Sebastian Gniazdowski wrote:
>>  Hello,
>>  I would like to announce Zsh Navigation Tools,
>
> ...
>
> Thanks for those Sebastian, I'm playing with them now, several look very
> useful. One thing, I had to create
> '/usr/local/share/zsh/site-functions'. I wonder if there is a location
> for the files that is in keeping with my directory structure. A trivial
> point: in the files, you call them 'functions' whereas they are scrips,
> I believe. However, I'm tempted to convert them to functions anyway.

They will work if located in *any* directory listed in $fpath. And they *are* functions, you need to read about autoload functions in man zshmisc, section AUTOLOADING FUNCTIONS:

       If  the  KSH_AUTOLOAD option is set, or the file contains only a simple definition of the function, the file's contents will be executed.  This will normally define the function in question, but may also per‐
       form initialization, which is executed in the context of the function execution, and may therefore define local parameters.  It is an error if the function is not defined by loading the file.

       Otherwise, the function body (with no surrounding `funcname() {...}') is taken to be the complete contents of the file.  This form allows the file to be used directly as an executable shell script.   If  pro‐
       cessing  of the file results in the function being re-defined, the function itself is not re-executed.  To force the shell to perform initialization and then call the function defined, the file should contain
       initialization code (which will be executed then discarded) in addition to a complete function definition (which will be retained for subsequent calls to the function), and  a  call  to  the  shell  function,
       including any arguments, at the end.

Re: Announce of Zsh Navigation Tools

[Ray Andrews]

On 09/12/2015 12:41 PM, ZyX wrote:
> They will work if located in *any* directory listed in $fpath. And they *are* functions, you need to read about autoload functions in man zshmisc, section AUTOLOADING FUNCTIONS:

I see.  So we very elegantly have it both ways--a script as it appears 
to be, but also as a function via autoload.
>
>         If  the  KSH_AUTOLOAD option is set, or the file contains only a simple definition of the function, the file's contents will be executed.  This will normally define the function in question, but may also per‐
>         form initialization, which is executed in the context of the function execution, and may therefore define local parameters.  It is an error if the function is not defined by loading the file.
>
>         Otherwise, the function body (with no surrounding `funcname() {...}') is taken to be the complete contents of the file.  This form allows the file to be used directly as an executable shell script.   If  pro‐
>         cessing  of the file results in the function being re-defined, the function itself is not re-executed.  To force the shell to perform initialization and then call the function defined, the file should contain
>         initialization code (which will be executed then discarded) in addition to a complete function definition (which will be retained for subsequent calls to the function), and  a  call  to  the  shell  function,
>         including any arguments, at the end.
That clarifies things greatly.  I've had a vague wondering what autoload 
was, just took it on faith that it was important.  What are the pros and 
cons of doing things that way vs. what might seem to be the more obvious 
function definition?  It's more compact, but then requires the special 
calling via autoload.  I expect there will be some substantial difference.

>

Re: Announce of Zsh Navigation Tools

[Bart Schaefer]

On Sep 12,  2:03pm, Ray Andrews wrote:
}
} What are the pros and cons of doing things that way vs. what might
} seem to be the more obvious function definition? It's more compact,
} but then requires the special calling via autoload. I expect there
} will be some substantial difference.

There isn't (intended to be) any execution-time difference.  If you
can find one, it's probably a bug, except for the variations that
are implied by (misapplication of) the -z / -k options of autoload.

The main benefit is that you don't have loaded into memory those
functions that you don't use. The not-yet-loaded form of an autoloaded
function is very small.  You also avoid a lot of the startup time of
parsing all those files.  E.g., the completion system is now nearly
10MB; why read all of that when most of it may never be used?

I suppose, conversely, the drawback is that there may be a delay for
parsing, the first time any such function is executed.

Another drawback is that you can't easily switch among different
emulation modes without taking special precautions, because the parse
will always be applied using the mode at the time of first execution
rather than the mode when the autoload command was issued.  We could
probably stand an autoload option to apply "sticky emulation."

Re: Announce of Zsh Navigation Tools

[Ray Andrews]

On 09/12/2015 03:10 PM, Bart Schaefer wrote:
> On Sep 12,  2:03pm, Ray Andrews wrote:
> }
> } What are the pros and cons of doing things that way vs. what might
> } seem to be the more obvious function definition? It's more compact,
> } but then requires the special calling via autoload. I expect there
> } will be some substantial difference.
>
> There isn't (intended to be) any execution-time difference.  If you
> can find one, it's probably a bug, except for the variations that
> are implied by (misapplication of) the -z / -k options of autoload.
>
> The main benefit is that you don't have loaded into memory those
> functions that you don't use. The not-yet-loaded form of an autoloaded
> function is very small.  You also avoid a lot of the startup time of
> parsing all those files.  E.g., the completion system is now nearly
> 10MB; why read all of that when most of it may never be used?
>
> I suppose, conversely, the drawback is that there may be a delay for
> parsing, the first time any such function is executed.
>
> Another drawback is that you can't easily switch among different
> emulation modes without taking special precautions, because the parse
> will always be applied using the mode at the time of first execution
> rather than the mode when the autoload command was issued.  We could
> probably stand an autoload option to apply "sticky emulation."
>
Ah ...

I load my own functions thusly:

     for aa in /aWorking/Zsh/Source/*;  do source $aa; done

... but as I understand you, autoload only 'really' loads things when 
they are actually called, so that has to be more efficient.  That 
immediately makes me want do something like this:

     for aa in /aWorking/Zsh/Source/*; do autoload $aa; done

... remembering of course that things need to be rewritten to look like 
scripts.  As to emulation, that's still way over my head.

BTW, looking at $fpath, it sure is laborious.  Do we, could wehave some 
sort of automatic subdirectory inclusion? Sorta: 
/usr/share/zsh/functions/**/  ... or something like that?

Re: Announce of Zsh Navigation Tools

[Bart Schaefer]

On Sep 12,  4:03pm, Ray Andrews wrote:
}
} immediately makes me want do something like this:
} 
}      for aa in /aWorking/Zsh/Source/*; do autoload $aa; done
} 
} ... remembering of course that things need to be rewritten to look like 
} scripts.

As long as there is only one function per file, you don't need to rewrite
them, just use

   ...; do autoload -k $aa; done

The "ksh-style" of autoloading is to have one complete function definition
in each file, including the "() { ... }" or "function { ... }" or whatever
wrapping syntax.

(Ksh does it this way because the syntax used to declare the function has
an effect on its semantics.)
 
} BTW, looking at $fpath, it sure is laborious.  Do we, could we have some 
} sort of automatic subdirectory inclusion?

I'm not sure what you mean by this, that is, are you trying to build up
the value of $fpath or are you trying to find all the files in $fpath
to autoload them?

If you're running an installed zsh, of course, then $fpath should have
all the necessary directories already (except e.g. personal ones in your
home dir).

If you're asking about constructing $fpath for a zsh build tree, then it
should suffice to do:

    fpath=( $build_dir/Completion/**/*(/) $build_dir/Functions/**/*(/)
            $fpath )

where you will need to define $build_dir appropriately, maybe it is $PWD.
Just be careful that you get the order right with respect to the default
$fpath -- the default should come FIRST if you are running an installed
shell.

If you're asking about autoloading everything after $fpath has been
defined, then this should work for the default paths:

    autoload $^fpath/*(N-.:t)

However, as I said you may want to add -k for your personal files, which
would need a second autoload command.

Some people also prefer to throw a (*) into the glob flags so as to only
autoload files that have been marked executable.

Putting this together, something like:

build_fpath=( $build_dir/Completion/**/*(/) $build_dir/Functions/**/*(/) )
my_fpath=( ~/zsh_functions/**/*(/) and so on )

autoload $^fpath/*(N-.:t)
autoload $^build_fpath/*(N-.:t)
autoload $^my_fpath/*(N-.:t)
fpath=( $my_fpath $build_fpath $fpath )

-- 
Barton E. Schaefer

Re: autoload

[Ray Andrews]

On 09/12/2015 04:43 PM, Bart Schaefer wrote:

Bart:

Tightening up my understanding of this a bit ...

>      autoload $^fpath/*(N-.:t)
... works fine, but what does the caret do?  Negation is counterintuitive.
Also I think I recognize the '(N-.:t)' as a globbing modifier, and I 
know the '.' and
the ':t' but I can't find any info on the 'N-' -- I don't know what 
phylum of animal it is
so as to research it.


    autoload /usr/local/share/zsh/site-functions/n-*(N-.:t)

... also works and is more literal.  Interesting that autoload must have 
the
path stripped off for it.  Why wouldn't it know to do that automatically?

Re: autoload

[Bart Schaefer]

On Sep 13,  7:14pm, Ray Andrews wrote:
}
} Tightening up my understanding of this a bit ...
} 
} >      autoload $^fpath/*(N-.:t)
} ... works fine, but what does the caret do?

Would it make more sense written like ${^fpath} ?  It's the operator
that causes an array (a b c) to expand to e.g. a/* b/* c/* instead of
to a b c/* -- cf. the rc_expand_param setopt.

} Also I think I recognize the '(N-.:t)' as a globbing modifier, and I 
} know the '.' and
} the ':t' but I can't find any info on the 'N-'

(N) means nonomatch, so empty directories disappear from the result
instead of causing a globbing error.  (-) is follow symlinks, so that
combined with (N) any dangling symlinks also disappear.  These both are
documented in the "Glob Qualifiers" subsection of the manual.

}     autoload /usr/local/share/zsh/site-functions/n-*(N-.:t)

Well, yes, but that only autoloads Sebastian's navigation functions
(and any others whose names happen to start with "n-").

} Interesting that autoload must have the
} path stripped off for it.  Why wouldn't it know to do that automatically?

You don't autoload from a location, you autoload a name and the fpath
is searched for it.  It would be misleading for autoload to discard
a directory which is passed to it in favor of one that appears earlier
in the fpath array, so you have to explicitly do so if that's what you
intend.

(zcompile on the other hand DOES load from locations rather than from
a path search, so if you use zcompile + autoload -w you get the path
stripping.)

Re: autoload

[Ray Andrews]

On 09/13/2015 07:38 PM, Bart Schaefer wrote:
> Would it make more sense written like ${^fpath} ?  It's the operator
> that causes an array (a b c) to expand to e.g. a/* b/* c/* instead of
> to a b c/* -- cf. the rc_expand_param setopt.
Yikes, that's brand new to me, I only know the caret as negation. So 
this is a feature
specific--it seems--to getting useful path expansion.  Just echoing some 
dirs, I get the
point loud and clear. That's powerful.
> (N) means nonomatch, so empty directories disappear from the result
> instead of causing a globbing error.  (-) is follow symlinks, so that
> combined with (N) any dangling symlinks also disappear.  These both are
> documented in the "Glob Qualifiers" subsection of the manual.

Nuts, I scanned that page top to bottom looking at random, and there it 
is near the
very bottom. 'NULL_GLOB' it says.  Thanks for the explanation.
> Well, yes, but that only autoloads Sebastian's navigation functions
> (and any others whose names happen to start with "n-").

Of course, I'm just experimenting for now.  Getting the feel ...
> You don't autoload from a location, you autoload a name and the fpath
> is searched for it.
Ah! ... I'm subconsciously thinking in terms of finding commands. No!

    autoload /usr/local/share/zsh/site-functions/n-*(N-.:t)


... is a kind of trick--we are grabbing the names of files and using 
them as the names of
functions to be found later, but we are NOT actually passing 
definitions.  Yes?

     autoload n-list n-cd n-env n-kill n-panelize n-options n-aliases 
n-functions n-history n-preview

... is actually more 'honest' since all autoload really knows at that 
point is names.  Definitions
will be retrieved later. Yes?

> (zcompile on the other hand DOES load from locations rather than from
> a path search, so if you use zcompile + autoload -w you get the path
> stripping.)
>
Sounds like a good idea, would that not be preferable?  Why not prime 
the pump with both the name and the location?  I'm over my head of 
course, but it seems that the '(...(N-.:t)' style command could notify 
autoload of both name and location in one go thus obviating the need to 
rummage through $fpath.  As always, if that's not done there will be a 
good reason.

Re: autoload

[Bart Schaefer]

On Sep 13,  8:34pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} On 09/13/2015 07:38 PM, Bart Schaefer wrote:
} > Would it make more sense written like ${^fpath} ?  It's the operator
} > that causes an array (a b c) to expand to e.g. a/* b/* c/* instead of
} > to a b c/* -- cf. the rc_expand_param setopt.
} Yikes, that's brand new to me, I only know the caret as negation. So 
} this is a feature
} specific--it seems--to getting useful path expansion.

No; it happens to involve path expansion in the specific example of
appending /* to every element of $fpath, but that's not really what
it means.  The second paragraph of the doc is useful:

${^SPEC}
     Turn on the RC_EXPAND_PARAM option for the evaluation of SPEC; if
     the `^' is doubled, turn it off.  When this option is set, array
     expansions of the form FOO${XX}BAR, where the parameter XX is set
     to (A B C), are substituted with `FOOABAR FOOBBAR FOOCBAR' instead
     of the default `FOOA B CBAR'.  Note that an empty array will
     therefore cause all arguments to be removed.

     Internally, each such expansion is converted into the equivalent
     list for brace expansion.  E.g., ${^var} becomes
     {$var[1],$var[2],...}, and is processed as described in *Note
     Brace Expansion:: below.  If word splitting is also in effect the
     $var[N] may themselves be split into different list elements.

} > (N) means nonomatch, so empty directories disappear from the result
} > instead of causing a globbing error.  (-) is follow symlinks, so that
} > combined with (N) any dangling symlinks also disappear.  These both are
} > documented in the "Glob Qualifiers" subsection of the manual.
} 
} Nuts, I scanned that page top to bottom looking at random, and there
} it is near the very bottom. 'NULL_GLOB' it says. Thanks for the
} explanation.

Yeah, I thinko nonomatch for nullglob periodically, don't know why.

}      autoload n-list n-cd n-env n-kill n-panelize n-options n-aliases 
} n-functions n-history n-preview
} 
} ... is actually more 'honest' since all autoload really knows at that 
} point is names.  Definitions will be retrieved later. Yes?

Yes.

} > (zcompile on the other hand DOES load from locations rather than from
} > a path search, so if you use zcompile + autoload -w you get the path
} > stripping.)
} >
} Sounds like a good idea, would that not be preferable?  Why not prime 
} the pump with both the name and the location?

Compatibility with the implementation chosen by other shells that had
autoloading earlier, mainly.  Less importantly, for .zshrc portability,
so you can autoload things without having to find them yourself in the
fpath first.  The inverse -- wanting to autoload a bunch of things
you know are in the fpath without knowing their function names -- was
not really contemplated as a common case when the command was invented.

Re: autoload

[Ray Andrews]

On 09/14/2015 01:21 PM, Bart Schaefer wrote:
> No; it happens to involve path expansion in the specific example of
> appending /* to every element of $fpath, but that's not really what
> it means.  The second paragraph of the doc is useful:
>
> ${^SPEC}
>       Turn on the RC_EXPAND_PARAM option for the evaluation of SPEC; if
>       the `^' is doubled, turn it off.  When this option is set, array
>       expansions of the form FOO${XX}BAR, where the parameter XX is set
>       to (A B C), are substituted with `FOOABAR FOOBBAR FOOCBAR' instead
>       of the default `FOOA B CBAR'.  Note that an empty array will
>       therefore cause all arguments to be removed.
Ok ... useful for path expansion but not specific to it.  It comes back 
to me that this RC_EXPAND_PARAM sounded great but had a gotcha attached 
to it.  But if the caret
turns it on for one particular expansion then that's very satisfactory.
> } > (zcompile on the other hand DOES load from locations rather than from
> } > a path search, so if you use zcompile + autoload -w you get the path
> } > stripping.)
> } >
>
One more thing if you would Bart: get me started with combining zcompile 
with autoload as you mentioned.  It sounds like a good idea but I can't 
find anything on it.  I know what zcompile is, but it makes so little 
difference in loading speed that I've not really bothered with it.  But 
if it works synergistically with autoload that might be interesting to 
play with.

Re: autoload

[Bart Schaefer]

On Sep 14,  7:48pm, Ray Andrews wrote:
}
} One more thing if you would Bart: get me started with combining zcompile 
} with autoload as you mentioned.  It sounds like a good idea but I can't 
} find anything on it.

I wrote about this in workers/36529 -- but I see that Sebastian had
redirected that thread from zsh-users to zsh-workers, so perhaps it
wasn't seen.  I'll reproduce here (so workers readers can stop now).


Take a look at the "zcompile" builtin and the "zrecompile" utility.
If you zcompile a collection of functions into a .zwc file, you can --

-- add the .zwc file name directly to $fpath and it will be searched
   like a directory at autoloaded-function execution time

-- specify the -z / -k behavior options at zcompile time so they don't
   have to be passed to autoload

-- easily autoload everything in the file:   autoload -w functions.zwc

ZWC files are machine-independent but zsh-version-dependent.  They get
memory-mapped when that's supported so the OS pages them efficiently.
The drawbacks are the version dependency if you frequently rebuild a
bleeding-edge shell (but see zrecompile) and that zcompile doesn't yet
support "sticky emulation" in any meaningful way.  Also, a ZWC file is
not like an archive file (e.g., a ZIP); you can't append or remove
individual functions, the whole file has to be rebuilt.

Finally, for Ray, you can for example replace your entire $fpath with
a single ZWC file:

    # Assume starting here with the default $fpath
    zsh_default_functions=~/.zsh-default-functions.zwc
    if ! zcompile -t $zsh_default_functions >&/dev/null
    then
      # File is missing or out of date.  Rebuild it.
      # Removes the file if any function cannot be compiled.
      zcompile $zsh_default_functions $^fpath/*(N.:A)
    fi
    if [[ -f $zsh_default_functions ]]
    then
      fpath=( $zsh_default_functions )
      autoload -w $zsh_default_functions
    fi

Re: autoload

[Ray Andrews]

On 09/14/2015 08:48 PM, Bart Schaefer wrote:
> On Sep 14,  7:48pm, Ray Andrews wrote:
> }
> } One more thing if you would Bart: get me started with combining zcompile
> } with autoload as you mentioned.  It sounds like a good idea but I can't
> } find anything on it.
Thanks, I'll throw myself at it.  This sounds like very satisfying stuff 
for philosophical
reasons--I prefer directness over convenience most of the time and an 
explicit
statement of where functions actually are turns my crank.  That's the 
reason I liked:

    autoload /usr/local/share/zsh/site-functions/n-*(N-.:t)

... because it's explicit.  Alas it turns out to be disharmonious with 
the nature of the
thing, like teaching a cat to fetch.


> Finally, for Ray, you can for example replace your entire $fpath with
> a single ZWC file:

Marvelous:

6 HP-w5--5-Debian1 /aWorking/Zsh 2$ echo $fpath
/usr/local/share/zsh/site-functions /usr/share/zsh/vendor-functions 
/usr/share/zsh/vendor-completions /usr/share/zsh/functions/Calendar 
/usr/share/zsh/functions/Chpwd /usr/share/zsh/functions/Completion 
/usr/share/zsh/functions/Completion/AIX 
/usr/share/zsh/functions/Completion/BSD 
/usr/share/zsh/functions/Completion/Base 
/usr/share/zsh/functions/Completion/Cygwin 
/usr/share/zsh/functions/Completion/Darwin 
/usr/share/zsh/functions/Completion/Debian 
/usr/share/zsh/functions/Completion/Linux 
/usr/share/zsh/functions/Completion/Mandriva 
/usr/share/zsh/functions/Completion/Redhat 
/usr/share/zsh/functions/Completion/Solaris 
/usr/share/zsh/functions/Completion/Unix 
/usr/share/zsh/functions/Completion/X 
/usr/share/zsh/functions/Completion/Zsh 
/usr/share/zsh/functions/Completion/openSUSE 
/usr/share/zsh/functions/Exceptions /usr/share/zsh/functions/MIME 
/usr/share/zsh/functions/Misc /usr/share/zsh/functions/Newuser 
/usr/share/zsh/functions/Prompts /usr/share/zsh/functions/TCP 
/usr/share/zsh/functions/VCS_Info 
/usr/share/zsh/functions/VCS_Info/Backends /usr/share/zsh/functions/Zftp 
/usr/share/zsh/functions/Zle

... is deeply disturbing on principal, I don't want my shell searching 
the world for my functions, I want it to *know* where they are and 
$fpath is a study in obfuscation.  It is information that subtracts from 
knowledge.

>
>      # Assume starting here with the default $fpath
>      zsh_default_functions=~/.zsh-default-functions.zwc
>      if ! zcompile -t $zsh_default_functions >&/dev/null
>      then
>        # File is missing or out of date.  Rebuild it.
>        # Removes the file if any function cannot be compiled.
>        zcompile $zsh_default_functions $^fpath/*(N.:A)
>      fi
>      if [[ -f $zsh_default_functions ]]
>      then
>        fpath=( $zsh_default_functions )
>        autoload -w $zsh_default_functions
>      fi
>
>
Deep magic ...
Arigato sensei

Re: autoload

[Ray Andrews]

Some first adventures with Bart's method.  Proof of concept only, no 
checking yet.

The kosher method in .zshrc:

#autoload -U compinit && compinit -d $ZSHBOOT/zcompdump
#autoload -U colors && colors
#autoload -U select-word-style && select-word-style shell

... gives:

     $ time zsh -i -c exit  0.14s user 0.06s system 98% cpu 0.211 total

... now this, after having moved all asked for files into my private 
ZSHBOOT dir
and cooked them up with zcompile:

fpath=$ZSHBOOT
autoload -w $ZSHBOOT/zsh-boot.zwc
compinit -d $ZSHBOOT/zcompdump
colors
select-word-style shell

... my thinking being that functions which are certain to be called are 
a good place to
start experimenting.

... gives:

     $ time zsh -i -c exit  0.05s user 0.06s system 97% cpu 0.111 total

... so with just those three autoloads it's twice as fast, saving a 
whopping 1/10th of
a second.  All of it in the 'user' category, I don't know if that's 
significant.  Most
interesting anyway, and so far it seems to work, tho I'd not be 
astonished to hear
that I've broken something.  It might be DOSthink, but I like having 
everything I need to
fire up my zsh in one place thusly:

3 HP-w5--5-Debian1 /aWorking/Zsh/Boot 3$ l

Sorting All file types Backwards by Modification Time ():

drwxr-xr-x 2   4096 2015-09-11_12:03:21 Backup/
-rw-r--r-- 1   2934 2014-12-02_20:26:53 zbindkey
-rw-r--r-- 1   1371 2015-09-13_15:55:51 aliases
-rw-r--r-- 1   2375 2015-09-14_08:27:29 zstyle
-rw-r--r-- 1  11403 2015-09-15_19:51:07 zmv
-rw-r--r-- 1   2230 2015-09-15_19:51:07 select-word-style
-rw-r--r-- 1  16435 2015-09-15_19:51:07 compinit
-rw-r--r-- 1   3417 2015-09-15_19:51:07 colors
-rw-r--r-- 1   5667 2015-09-15_20:25:25 compaudit
-rw-r--r-- 1   4015 2015-09-15_20:25:25 compdump
-r--r--r-- 1  57984 2015-09-15_20:25:50 zsh-boot.zwc
-rw-r--r-- 1    484 2015-09-15_20:26:11 zcompdump
-rw-r--r-- 1   5464 2015-09-15_20:43:10 zshrc
-rw-r--r-- 1    878 2015-09-15_20:55:39 dirstack
-rw------- 1 441938 2015-09-15_20:55:41 histfile

... of course there's more needed stuff as well no doubt, but this at 
least gives me
a bit of focus, and fpath is a single directory :-)

autoload

[Ray Andrews]

Bart:

I did this, which should be verbatim as you posted:


     # Assume starting here with the default $fpath
     zsh_default_functions=~/.zsh-default-functions.zwc
     if ! zcompile -t $zsh_default_functions >&/dev/null
     then
       # File is missing or out of date.  Rebuild it.
       # Removes the file if any function cannot be compiled.
       zcompile $zsh_default_functions $^fpath/*(N.:A)
     fi
     if [[ -f $zsh_default_functions ]]
     then
       fpath=( $zsh_default_functions )
       autoload -w $zsh_default_functions
     fi

... and I get:

     /aWorking/Zsh/Boot/ztest:5: parse error near `)'
     /aWorking/Zsh/Boot/ztest:zcompile:10: can't read file: /usr/share 
                 /zsh/functions/Completion /Base.zwc

('ztest' being sourced by .zshrc and where I do experiments.)

'Base.zwc' exists:

     $ ls -l Base.zwc
     -rw-r--r-- 1 root root 269472 Aug 31 16:26 Base.zwc

... and  "zcompile -t Base.zwc" doesn't report anything amiss.

The  " parse error near `)' " msg. is puzzling since it always reports 
line 5 regardless of anything added above it ,eg: "echo "this echo is 
only here to add a line of code"".

For example if I modify your code like this:

     line5
         # Assume starting here with the default $fpath
zsh_default_functions=~/.zsh-default-functions.zwc
     line9

... I get this:

     /aWorking/Zsh/Boot/ztest:5: command not found: line5
     /aWorking/Zsh/Boot/ztest:9: command not found: line9
     /aWorking/Zsh/Boot/ztest:5: parse error near `)'
     /aWorking/Zsh/Boot/ztest:zcompile:14: can't read file: /usr/share 
/zsh/functions/Completion/Base.zwc

... so the deliberate errors are as expected, yet she insists on finding 
an `)' error on line 5 as well, and that after having reached line 9, so 
I have no idea how to understand that.

Re: autoload

[Bart Schaefer]

On Sep 17,  8:54am, Ray Andrews wrote:
} Subject: autoload
}
}      # Assume starting here with the default $fpath

I think the problem here is that you've put a .zwc file IN a directory
that is itself listed in $fpath.  This is typically a no-no.  You want
a .zwc file to be treated as if it WERE a directory, NOT as if it were
one of the function definition files IN the directory.

If you must have the .zwc where you've presently got it, then you need
to exclude it from the file pattern $^fpath/*(N.:A) -- which is easiest
if you have extendedglob set, so you can write $^fpath/*~*.zwc(N.:A)
instead.

}      /aWorking/Zsh/Boot/ztest:5: parse error near `)'
}      /aWorking/Zsh/Boot/ztest:zcompile:10: can't read file: /usr/share 
}                  /zsh/functions/Completion /Base.zwc

See, zcompile (without the -t) is attempting to parse Base.zwc as a
function definition, and getting what looks like garbage to the parser.
(I have no idea where those spaces in the path are coming from.)

Re: autoload

[Ray Andrews]

On 09/17/2015 10:34 AM, Bart Schaefer wrote:
> On Sep 17,  8:54am, Ray Andrews wrote:
> } Subject: autoload
> }
> }      # Assume starting here with the default $fpath
>
> I think the problem here is that you've put a .zwc file IN a directory
> that is itself listed in $fpath.  This is typically a no-no.  You want
> a .zwc file to be treated as if it WERE a directory, NOT as if it were
> one of the function definition files IN the directory.

I'm close to certain I haven't, here's all .zwc on my system:


> /usr/share/zsh/functions/VCS_Info/Backends.zwc
> /usr/share/zsh/functions/Exceptions.zwc
> /usr/share/zsh/functions/VCS_Info.zwc
> /usr/share/zsh/functions/Prompts.zwc
> /usr/share/zsh/functions/MIME.zwc
> /usr/share/zsh/functions/Completion.zwc
> /usr/share/zsh/functions/TCP.zwc
> /usr/share/zsh/functions/Zle.zwc
> /usr/share/zsh/functions/Chpwd.zwc
> /usr/share/zsh/functions/Misc.zwc
> /usr/share/zsh/functions/Calendar.zwc
> /usr/share/zsh/functions/Completion/Debian.zwc
> /usr/share/zsh/functions/Completion/X.zwc
> /usr/share/zsh/functions/Completion/openSUSE.zwc
> /usr/share/zsh/functions/Completion/AIX.zwc
> /usr/share/zsh/functions/Completion/Mandriva.zwc
> /usr/share/zsh/functions/Completion/Base.zwc
> /usr/share/zsh/functions/Completion/Zsh.zwc
> /usr/share/zsh/functions/Completion/Linux.zwc
> /usr/share/zsh/functions/Completion/Cygwin.zwc
> /usr/share/zsh/functions/Completion/Solaris.zwc
> /usr/share/zsh/functions/Completion/Darwin.zwc
> /usr/share/zsh/functions/Completion/Redhat.zwc
> /usr/share/zsh/functions/Completion/BSD.zwc
> /usr/share/zsh/functions/Completion/Unix.zwc
> /usr/share/zsh/functions/Zftp.zwc
> /usr/share/zsh/functions/Newuser.zwc
>
>
> See, zcompile (without the -t) is attempting to parse Base.zwc as a
> function definition, and getting what looks like garbage to the parser.
> (I have no idea where those spaces in the path are coming from.)
>
It isn't possible I've somehow screwed up $fpath?  I understand it's 
hard coded at startup,
least it seems to always be restored. Anyhow I've not moved or played 
with Base.zwc.  I did try
creating a .zwc yesterday but it's long dead.

This seems to be the troublesome line:

      zcompile $zsh_default_functions $^fpath/*(N.:A)

... on a hunch:

      zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a 
new friend

... the previous error msgs are gone but:

     $ zsh
     /aWorking/Zsh/Boot/ztest:zcompile:13: can't open file: _bootctl

Re: autoload

[Bart Schaefer]

On Sep 17, 12:22pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} On 09/17/2015 10:34 AM, Bart Schaefer wrote:
} >
} > I think the problem here is that you've put a .zwc file IN a directory
} > that is itself listed in $fpath.
} 
} I'm close to certain I haven't, here's all .zwc on my system:

The issue isn't (just) what your .zwc files are, it's also what you
$fpath contains at the time that your .zshrc or whatever is loaded.

E.g. $^fpath/*(.) appears to include
    /usr/share/zsh/functions/Completion/Base.zwc
as one of the matching files.

Therefore /usr/share/zsh/functions/Completion must be in the default
$fpath, and what I'm saying is that you should not normally put Base.zwc
in a directory that is in $fpath.

} It isn't possible I've somehow screwed up $fpath?  I understand it's 
} hard coded at startup,

Yes, and the problem is that you've put .zwc files into the directories
in that hardcoded list, but then tried to use the hardcoded list to
(re)build yet another .zwc file.

I'm not really sure how further to explain this.  The sample code that
I provided takes EVERY DIRECTORY that's in (the default) $fpath and
packs ALL the files under them into ONE .zwc.

You on the other hand seem to have already built a separate .zwc out of
EACH directory of the default $fpath.  If you're using my example, you
don't need to do that - you should throw all those .zwc away and use
the single one that my sample builds.

Or conversely you should not use that example.

}       zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a 
} new friend

A different enemy, more like.

} ... the previous error msgs are gone but:
} 
}      $ zsh
}      /aWorking/Zsh/Boot/ztest:zcompile:13: can't open file: _bootctl

The :t discards /usr/share/zsh/... and leaves just a base file name;
that might work for autoload but won't for zcompile.  You need the
full paths, just not the .zwc files.

Re: autoload

[Ray Andrews]

On 09/17/2015 01:40 PM, Bart Schaefer wrote:
> Yes, and the problem is that you've put .zwc files into the directories
> in that hardcoded list, but then tried to use the hardcoded list to
> (re)build yet another .zwc file.
But I didn't create those files. They're dated August 31 which seems to 
be common to
most of the files in the distro as I downloaded it.  Is this something 
to do with Debian?

> I'm not really sure how further to explain this.  The sample code that
> I provided takes EVERY DIRECTORY that's in (the default) $fpath and
> packs ALL the files under them into ONE .zwc.
That's how I read it.

     You on the other hand seem to have already built a separate .zwc 
out of EACH
     directory of the default $fpath. If you're using my example, you 
don't need to
     do that - you should throw all those .zwc away and use the single 
one that my
     sample builds.


I tried deleting all of them which resulted in ~/.zsh-default-functions.zwc
being created, but zero length and of course nothing worked.  With 
original .zshrc code, things worked but a 'time' test was very slow, 
thus demonstrating the effectiveness of the .zwc
idea itself (by it's absence, in this case). I restored the .zwc files 
and I'm back to normal.
This seems to be a very tricky business.  Perhaps not worth wasting your 
time on.  It seems
a shame that zcompile doesn't know not to trip over itself, one might 
expect to find a .zwc
file in the same dir as the files from which it was built.  Or not.

> }       zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a
> } new friend
>
> } A different enemy, more like.
>

Sure, it was just experiment, I hardly expected it to be correct.

Re: autoload

[Bart Schaefer]

On Sep 17,  4:06pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} But I didn't create those files. They're dated August 31 which seems
} to be common to most of the files in the distro as I downloaded it. Is
} this something to do with Debian?

Possibly.  I've never installed a pure Debian distro, only Ubuntu.

} I tried deleting all of them which resulted in ~/.zsh-default-functions.zwc
} being created, but zero length and of course nothing worked.

That's a little confusing since with :t in there you at least got an
attempt to read _bootctl so there must be some function files.

What does

() { emulate -L zsh -o extendedglob; print -lR $^fpath/*~*.zwc(N.:A) | head }

output, if placed early in your startup files (before you do any of your
own changes to $fpath)?

Re: autoload

[Ray Andrews]

On 09/17/2015 04:20 PM, Bart Schaefer wrote:
> On Sep 17,  4:06pm, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } But I didn't create those files. They're dated August 31 which seems
> } to be common to most of the files in the distro as I downloaded it. Is
> } this something to do with Debian?
>
> Possibly.  I've never installed a pure Debian distro, only Ubuntu.
My first distro was Mint, and 'downgrading' to plain vanilla Debian was 
the smartest thing
I ever did--less baloney all 'round.   I don't even know what it is that 
I'm supposed to be
missing.  Anyway, those .zwc files are in the Debian package, I just 
checked.  Perhaps Axel
has a comment?
>
>
> That's a little confusing since with :t in there you at least got an
> attempt to read _bootctl so there must be some function files.
That's with 't' gone, and back to 'A'.
> What does
>
> () { emulate -L zsh -o extendedglob; print -lR $^fpath/*~*.zwc(N.:A) | head }
>
> output, if placed early in your startup files (before you do any of your
> own changes to $fpath)?
>
0 HP-w5--5-Debian1 /aWorking/Zsh 2$ zsh
/usr/local/share/zsh/site-functions/n-aliases
/usr/local/share/zsh/site-functions/n-cd
/usr/local/share/zsh/site-functions/n-env
/usr/local/share/zsh/site-functions/n-functions
/usr/local/share/zsh/site-functions/n-history
/usr/local/share/zsh/site-functions/n-kill
/usr/local/share/zsh/site-functions/n-list
/usr/local/share/zsh/site-functions/n-list-draw
/usr/local/share/zsh/site-functions/n-list-input
/usr/local/share/zsh/site-functions/n-options


... without 'head' it shows everything and it all looks fine.

Re: autoload

[Bart Schaefer]

On Sep 17,  6:20pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} Anyway, those .zwc files are in the Debian package, I just checked.

In that case you should probably just do:

    autoload -w $^fpath/*.zwc(N)

and forget about rebuilding from the source functions.

-- 
Barton E. Schaefer

Re: autoload

[Ray Andrews]

On 09/17/2015 09:04 PM, Bart Schaefer wrote:
> On Sep 17,  6:20pm, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } Anyway, those .zwc files are in the Debian package, I just checked.
>
> In that case you should probably just do:
>
>      autoload -w $^fpath/*.zwc(N)
>
> and forget about rebuilding from the source functions.
Yes, that was quite the red herring there.  Interesting that Debian does 
things
differently like that.  I suppose they are free to, but it still seems 
strange that
there'd be no doc to the effect that their package is different from 
std. like that.
Maybe it's not to be expected.  I guess the 'line 5 `)' error mystery is 
best
forgotten.  Very strange tho.

I'm trying to get the autoload system working with my own functions and I'm
almost there.  " /aWorking/Zsh/Source"  is where they live, and I've 
called them
like this:

     for aa in /aWorking/Zsh/Source/*; do source $aa; done

... which was the first thing I learned about such things.  They are all 
'normal'
functions and most files contain more than one.

Now I'm trying this:

$ fpath=( /aWorking/Zsh/Source "${fpath[@]}" )
     $ source miscfunctions
$ autoload -w /aWorking/Zsh/Source/Source.zwc
$ l
     [ nothing ]
     $ l
     [ now I get listing ]

... in the docs there's talk of this--loading vs. loading and 
executing--but I can't
figger it.  I've tried 'zcompile -k/z ...' and 'autoload k/z ...' but no 
combination seems
to prevent the first call to any function from being inert. 'zcompile -t 
...' always
shows me a list of files, not functions.  Strangely there seems
to be no trouble with multiple functions in one file except in the case of
'miscfunctions' so I have to source it separately.  (The other multiple 
functions
are all help functions only called by the main function in each file. I 
don't know
why that makes a difference, but it does.)  Also, I wonder why $fpath 
needs to
be modified since 'autoload -w' is given an explicit path to the .zwc 
file, so should
hardly be concerned with looking for it via $fpath.  I'd expect it to be 
hashed
and that's that.

Re: autoload

[Bart Schaefer]

On Sep 17, 10:00pm, Ray Andrews wrote:
}
} I guess the 'line 5 `)' error mystery is best
} forgotten.  Very strange tho.

It's not a mystery.  Try this:

    zsh -nf /usr/share/zsh/functions/Completion/Base.zwc

} $ fpath=( /aWorking/Zsh/Source "${fpath[@]}" )
} $ source miscfunctions
} $ autoload -w /aWorking/Zsh/Source/Source.zwc
} $ l
} [ nothing ]
} $ l
} [ now I get listing ]

The way autoload works is, the name of the file has to be the same as
the name of the function.  If you "source" the file, you execute the
file contents immediately, which in your case defines the functions.
If you "autoload" the function names, you NEITHER define NOR execute
them; you just mark them to be processed later.

Then when you try to execute, zsh looks for a file with the same name
as the function, parses it, and executes the result of the parse.
Except for handling of the positional parameters, this is *almost* the
same as

   functions[$filename:t]=$(<filename)
   eval "$functions[$filename:t]"

With "autoload -z" (the default when no option), zsh assumes the file
*was* the entire function body and does nothing further.

*If* the function was marked "autoload -k", then zsh assumes that the
eval-like step defined THAT function, and calls it by name a second time,
similar to:

   functions[$filename:t]=$(<filename)
   eval "$functions[$filename:t]"
   $filename:t

If we now throw zcompile into the mix, the -z / -k options to autoload
are ignored in favor of the -z / -k options to zcompile.  However, the
rule about the file name having to be the same as the function name
doesn't change.  Further "autoload -w" still doesn't define anything;
it just uses the index table in the zcompiled file to grab the list
of names to mark for later processing.

} Strangely there seems to be no trouble with multiple functions in one
} file except in the case of 'miscfunctions' so I have to source it
} separately. (The other multiple functions are all help functions only
} called by the main function in each file. I don't know why that makes
} a difference, but it does.)

Does what I just wrote, above, help?  In the "miscfunctions" case, the
name of the file matches NONE of the names of the functions inside it;
they can't be autoloaded.  The name of the file has to match the name
of AT LEAST ONE function inside the file, *OR* the entire file has to
be that single function (as if it were a script).

} Also, I wonder why $fpath needs to be modified since 'autoload -w' is
} given an explicit path to the .zwc file

The internal implementation of autoloaded functions is unrelated to
hashing and does not retain any file paths.  It keeps only the base
names.  Where that set of names came from does not change that.

This is important to the consistent operation of fpath searching.  If
you have

    fpath=(/tmp/testfunctions ...)

and you create a new file /tmp/testfunctions/tryme, and the function
is marked for autoloading, then that new file has to be found ahead
of anything else, even if you previously used "autoload -w" on a .zwc
file that contains a "tryme" file.  The contents of fpath at the
instant of execution ALWAYS takes precedence; there is no "never get
this function from anywhere except [here]" in the autoload scheme.

Re: autoload

[Ray Andrews]

On 09/17/2015 10:52 PM, Bart Schaefer wrote:
> It's not a mystery.  Try this:
>
>      zsh -nf /usr/share/zsh/functions/Completion/Base.zwc
Ah, so the error is really from some other file.  I assumed it was some 
sort of
second parsing or something.  Nice if the called file was identified tho.
Anyway at least that's clear.

> If we now throw zcompile into the mix, the -z / -k options to autoload
> are ignored in favor of the -z / -k options to zcompile.  However, the
> rule about the file name having to be the same as the function name
> doesn't change.  Further "autoload -w" still doesn't define anything;
> it just uses the index table in the zcompiled file to grab the list
> of names to mark for later processing.
This could be made clearer in the docs.
> } Strangely there seems to be no trouble with multiple functions in one
> } file except in the case of 'miscfunctions' so I have to source it
> } separately. (The other multiple functions are all help functions only
> } called by the main function in each file. I don't know why that makes
> } a difference, but it does.)
>
> Does what I just wrote, above, help?  In the "miscfunctions" case, the
> name of the file matches NONE of the names of the functions inside it;
> they can't be autoloaded.  The name of the file has to match the name
> of AT LEAST ONE function inside the file, *OR* the entire file has to
> be that single function (as if it were a script).
I see.  Nuts, there's nothing about that in the docs is there?

> ...  The contents of fpath at the
> instant of execution ALWAYS takes precedence; there is no "never get
> this function from anywhere except [here]" in the autoload scheme.
>
As always Bart you make things as clear as they can be, but it sure is 
obtuse.  I long for:

     $ autoload --just_DO_IT  Source.zwc

... that is functionally identical to:

     $ for aa in /aWorking/Zsh/Source/*; do source $aa; done

( Of course that's after having done:)

      $ zcompile --just_DO_IT  Source.zwc  /aWorking/Zsh/Source/*


... but much faster--all functions are pre-digested to word code and 
ready for action, only not loaded into memory until/unless actually 
called.  'zcompile --just_DO_IT' is smart enough to create function 
names after filenames in the case of scripts, and of course named 
functions are named functions.  No need for wandering around in $fpath, 
the names of all the functions found in '.../Source' (regardless of the 
name of any file except in the case of scripts) and their definition 
.zwc are tabled for instant access so the user can consider them as 
having been sourced already, it's just much faster and trimmer.  (Speed 
is the point of all this, right?) "zcompile -t --just_DO_IT Source.zwc" 
returns the list of all *functions* found or created (from scripts) in 
'Source.zwc'.

Never mind, like completion, this is not for mortals to dabble in. My 
'for' loop does it all in one line and there's no gotchas. Transparency 
is not a zsh virtue.

Re: autoload

[Bart Schaefer]

On Sep 18,  8:49am, Ray Andrews wrote:
}
} > If we now throw zcompile into the mix, the -z / -k options to autoload
} > are ignored in favor of the -z / -k options to zcompile.  However, the
} > rule about the file name having to be the same as the function name
} > doesn't change.  Further "autoload -w" still doesn't define anything;
} > it just uses the index table in the zcompiled file to grab the list
} > of names to mark for later processing.
} 
} This could be made clearer in the docs.
[...]
} I see.  Nuts, there's nothing about that in the docs is there?

There's an entire section for "Autoloading Functions" but it has no
cross-reference from the "autoload" builtin (autoload points to
functions points to typeset which finally does have the section
reference).  I'll rearrange this a bit.

There IS a cross-reference to that section from the zcompile builtin.

}      $ autoload --just_DO_IT  Source.zwc
} 
} ... that is functionally identical to:
} 
}      $ for aa in /aWorking/Zsh/Source/*; do source $aa; done

You can do that:

FPATH=/aWorking/Zsh/Source autoload +X -w /aWorking/Zsh/Source/Source.zwc

The +X option is documented and means "load this now, but don't run it."

Re: autoload

[Ray Andrews]

On 09/18/2015 09:52 AM, Bart Schaefer wrote:
> } I see.  Nuts, there's nothing about that in the docs is there?
>
> There's an entire section for "Autoloading Functions" but it has no
> cross-reference from the "autoload" builtin (autoload points to
> functions points to typeset which finally does have the section
> reference).  I'll rearrange this a bit.
What would be sweet is a self-contained HOWTO sort of doc that walks you 
through the
entire subject with real world examples of solutions for real 
situations.  IOW, instead of
focusing on a command, it would focus on the goal: "Say you want your 
functions to
load faster.  Up till now you've been sourcing them at startup, which is 
fine, but
you can do much better ...."  ... 20 minutes latter you are cooking.
>
> }
> }      $ for aa in /aWorking/Zsh/Source/*; do source $aa; done
>
> You can do that:
>
> FPATH=/aWorking/Zsh/Source autoload +X -w /aWorking/Zsh/Source/Source.zwc
>
> The +X option is documented and means "load this now, but don't run it."
>
Ha!  Just when I'd decided to give up.

Re: autoload

[Bart Schaefer]

On Sep 18, 11:29am, Ray Andrews wrote:
}
} What would be sweet is a self-contained HOWTO sort of doc that walks
} you through the entire subject with real world examples of solutions
} for real situations.

Yes, but there are an almost unlimited number of such possible HOWTO
topics and no volunteers who dedicate their time to writing this sort
of thing.

zcompile was even considered too obscure to get a mention in the
"From Bash to Zsh" book, though you'll find a lot of other HOWTO-type
stuff in there.

Re: autoload

[Ray Andrews]

On 09/18/2015 12:02 PM, Bart Schaefer wrote:
> On Sep 18, 11:29am, Ray Andrews wrote:
> }
> } What would be sweet is a self-contained HOWTO sort of doc that walks
> } you through the entire subject with real world examples of solutions
> } for real situations.
>
> Yes, but there are an almost unlimited number of such possible HOWTO
> topics and no volunteers who dedicate their time to writing this sort
> of thing.
I would if I ever mastered a topic.  Anyway I find most linux 
documentation to be
almost sadistically unhelpful. zsh's is already better than most.

BTW I just realized why I couldn't find "Autoloading
Functions" in zshmisc: because it's "AUTOLOADING FUNCTIONS"  and the 
search is
case sensitive :-(   It's the little things that drive ya nuts.

Re: autoload

[Bart Schaefer]

I decided I should clarify something:

On Sep 17, 10:34am, Bart Schaefer wrote:
}
} I think the problem here is that you've put a .zwc file IN a directory
} that is itself listed in $fpath.  This is typically a no-no.  You want
} a .zwc file to be treated as if it WERE a directory, NOT as if it were
} one of the function definition files IN the directory.

This is not quite correct.  I thought the situation was that we were
talking about was

    fpath=(/path/to/Directory ...)
    /path/to/Directory/Directory.zwc	# contains multiple functions

This would be wrong.  Instead it should be one of

    fpath=(/path/to/Directory ...)
    /path/to/Directory.zwc		# contains multiple functions
or
    fpath=(/path/to/Directory ...)
    /path/to/Directory/function.zwc	# contains one function only

In fact what Debian has done is the first of those two (correct) options,
so my "typically a no-no" was off base.

A third correct option would be

    fpath=(/path/to/Directory.zwc ...)

which I prefer, but which skips comparing the modification time of that
file to the files in /path/to/Directory/.  The advantage is that the
Directory itself need not exist, you can put the .zwc file anywhere.

Now, here's the interesting twist to what Debian has done.  They have

fpath=(... /usr/share/zsh/functions/Completion ...
       /usr/share/zsh/functions/Completion/Base ...)

with file /usr/share/zsh/functions/Completion/Base.zwc (among others).

For autoload, this is ambiguous -- if someone tries to run a command
named "Base", zsh searches /usr/share/zsh/functions/Completion/Base.zwc
for that function, because it might be the "one function only" option
described above.  It won't find the function there, of course, so it
will go on and look elsewhere and no one the wiser, and probably no one
ever runs "Base" anyway.

I don't want to claim there is anything wrong with Debian's choice, I
merly point it out as a probably-unintended side effect.

Re: autoload

[Ray Andrews]

On 09/18/2015 05:14 PM, Bart Schaefer wrote:
> I decided I should clarify something:
>
...
> I don't want to claim there is anything wrong with Debian's choice, I
> merly point it out as a probably-unintended side effect.
>
If I understand, you are saying that a .zwc can't be in the same dir as 
the files from which it
was created?  Even though that is where it will be created?  I dunno, 
autoload/zcompile
seems like such a good idea but the implementation seems deliberately 
obscure.  As I read
the doc, it looked for the most recent file in whatever dir.  We have 
every possible twist
and turn, what we don't have is a simple and direct way of precompiling 
functions:

autoload --functions-out /All-My-Functions/All-My-Functions.zwc

... precompiled, *and/but* only loaded when called.  Exactly like 
sourcing, only faster and better.

Could even make making 'em look like this, just to be symetrical:

autoload --functions-in  /All-My-Functions/All-My-Functions.zwc 
/All-My-Functions/*

.... scripts, functions, however many per file, whatever the name ... no 
matter, it's all
wrapped up in the .zwc ready to go, just remembering that functions from 
scripts
will be 'auto-named' after the containing file.  Too cute?  Na ... it's 
easy to understand
and rather elegant.  The whole thing could be explained in two minutes
and implemented in 30 seconds.  Why do things *have* to be difficult?

As for me, I took Sebastian's stuff and converted it all to plain 
vanilla good old fashioned
righteous Protestant functions and may I never have to think about 
$fpath again ;-)
Seriously, I'll just let my brain cool down for a while and try to learn 
something
that a mortal can learn.  Sheesh, even Bart forgets how it works ...

Re: autoload

[Bart Schaefer]

On Sep 19,  8:04am, Ray Andrews wrote:
}
} If I understand, you are saying that a .zwc can't be in the same dir
} as the files from which it was created?

No.  I'm saying that if fpath is a tree, (X X/Y X/Y/Z), having
function files at all levels of the tree, then a file X/Y/Z.zwc
containing all the functions from Z might be wrongly interpreted
as containing a single function named Z instead.

} autoload --functions-out /All-My-Functions/All-My-Functions.zwc
} 
} autoload --functions-in  /All-My-Functions/All-My-Functions.zwc \
} /All-My-Functions/*

I don't understand your intended semantics of "out" and "in" here,
which argues against "expained in two minutes," but the point of
shell builtins is to provide building blocks for solutions like you
seem to be thinking of here.
 
} Sheesh, even Bart forgets how it works ...

Actually that's because I don't use it.  I'm nearly always either
running everything out of a development tree where I recompile zsh
(or pull new functions from git) sometimes multiple times a day,
or else running the same shell nonstop for weeks at a time without
ever restarting it.  There's no point in trying to keep a compiled 
bundle around if I'm never going to reference it.

Re: autoload

[Ray Andrews]

On 09/19/2015 09:29 AM, Bart Schaefer wrote:
> On Sep 19,  8:04am, Ray Andrews wrote:
> }
>
> } autoload --functions-out /All-My-Functions/All-My-Functions.zwc
> }
> } autoload --functions-in  /All-My-Functions/All-My-Functions.zwc \
> } /All-My-Functions/*
>
> I don't understand your intended semantics of "out" and "in" here,
I'm thinking about how the creation of a zwc file, and it's subsequent 
use could
be considered to be part of the same builtin.  One being the 'inverse' 
of the other.

autoload --make-zwc /Functions/Functions.zwc /Functions/*
autoload --load-zwc /Functions/Functions.zwc

... it wouldn't take long to understand.


> Actually that's because I don't use it.
It's very relevant to what degree veterans like yourself use stuff. I'd 
provisionally
conclude that if you don't use it, then I don't want it.  There's more 
to be learned
from seeing how masters actually use a tool than from reading the tool's 
manual.
The book I'd long to read would be: "A *practical* guide to zsh".

Re: autoload

[Bart Schaefer]

On Sep 19, 11:13am, Ray Andrews wrote:
} Subject: Re: autoload
}
} I'm thinking about how the creation of a zwc file, and it's subsequent 
} use could be considered to be part of the same builtin.

But you don't say

    compiler -build something source/*
    compiler -load something

Rather, you build, and then you install, and then you run, and the way
all shells are designed, you use $path to bridge the install + run steps.

$fpath is the same idea; "autoload" is extra glue to tell the shell
that you DON'T want to use the default ($path) search for those names.

} autoload --make-zwc /Functions/Functions.zwc /Functions/*
} autoload --load-zwc /Functions/Functions.zwc
} 
} ... it wouldn't take long to understand.

As I've pointed out, these are exactly the same as

    zcompile /Functions/Functions.zwc /Functions/*
    FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc

Although the better practice would be to do

    zcompile /Functions.zwc /Functions/*
    FPATH=/Functions.zwc autoload +X -w /Functions.zwc

if not because of the way fpath is searched, then for the same reason that
you shouldn't do

    tar -c /Functions/Functions.tar /Functions/*

Anyway, just make yourself a little wrapper function:

    loadorcompile() {	# rename as suits you
    	local how=$1; shift
	case $how in
	(--make-zwc) zcompile "$@";;
	(--load-zwd) FPATH=$1 autoload +X -w $1;;
	(*) print -u2 "don't know $how";;
	esac
    }

} It's very relevant to what degree veterans like yourself use stuff.
} I'd provisionally conclude that if you don't use it, then I don't
} want it.

As I tried to point out, I'm not a very typical user.  I would also
never use Sebastian's navigation tools, or anything from oh-my-zsh;
but that doesn't mean anyone else should avoid them.

Re: autoload

[Ray Andrews]

On 09/19/2015 02:22 PM, Bart Schaefer wrote:
> On Sep 19, 11:13am, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } I'm thinking about how the creation of a zwc file, and it's subsequent
> } use could be considered to be part of the same builtin.
>
> But you don't say
>
>      compiler -build something source/*
>      compiler -load something
>
> Rather, you build, and then you install, and then you run, and the way
> all shells are designed, you use $path to bridge the install + run steps.
Sure, it wasn't to be taken very seriously, I'm just thinking about the 
idea that there
could be a sort of symmetry between creating a .zwc and then loading 
one--you
could think of them as so related that the command name could be the 
same.  IOW
the creation of a .zwc presupposes subsequent use of it the same way 
that making
dinner supposes eating dinner.  Command recall would be easier ... but 
that's
never going to happen.  I guess the parallel would be 'zip' vs. 
'unzip'--we're glad
that the names are related.

> As I've pointed out, these are exactly the same as
>
>      zcompile /Functions/Functions.zwc /Functions/*
>      FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc
>
> Although the better practice would be to do
>
>      zcompile /Functions.zwc /Functions/*
>      FPATH=/Functions.zwc autoload +X -w /Functions.zwc
But ... well, let me try it again, I may have busted something.
> } It's very relevant to what degree veterans like yourself use stuff.
> } I'd provisionally conclude that if you don't use it, then I don't
> } want it.
>
> As I tried to point out, I'm not a very typical user.  I would also
> never use Sebastian's navigation tools, or anything from oh-my-zsh;
> but that doesn't mean anyone else should avoid them.
No, OTOH in my judgment typicality is not of great value whereas decades of
experience is.  Not that I have the slightest hesitation in doing things 
my own
way, but I always want to know what the Old Masters think even if I 
choose to
differ ... until learning better.
>

Re: autoload

[Bart Schaefer]

On Sep 19,  3:12pm, Ray Andrews wrote:
}
} the creation of a .zwc presupposes subsequent use of it the same way 
} that making dinner supposes eating dinner.

Obviously you've never worked in a restaurant.

} I guess the parallel would be 'zip' vs.  'unzip'--we're glad
} that the names are related.

But that's not a parallel at all, because unzip doesn't do anything
with the data that it unpacks.  A better parallel might be, creating
a DVD vs. playing it back.

Re: autoload

[Ray Andrews]

On 09/19/2015 10:53 PM, Bart Schaefer wrote:
> On Sep 19,  3:12pm, Ray Andrews wrote:
> }
> } the creation of a .zwc presupposes subsequent use of it the same way
> } that making dinner supposes eating dinner.
>
> Obviously you've never worked in a restaurant.
>
> } I guess the parallel would be 'zip' vs.  'unzip'--we're glad
> } that the names are related.
>
> But that's not a parallel at all, because unzip doesn't do anything
> with the data that it unpacks.  A better parallel might be, creating
> a DVD vs. playing it back.
>
It's of philosophical interest onlybut this kind of question engages me 
quite a bit.
Your example of 'compiler --run-program' certainly demonstrates the 
reducto ad
absurdum of the idea--but one might compile something with a dozen different
compilers and compilers might compile any one of thousands of different 
programs,
and the compilers certainly don't care what the program does, and the 
program
might run under a dozen different OSon any compatible machine--
so their is no 'tight' link. Same with a DVD--one might burn with any 
one of several
programs and play back with any one of several with no link between the 
creator
and the player, and the contents of the product--the DVD--is irrelevant 
to the burner
and to the player.  (So long as the standards are maintained, of course.)

OTOH zcompile produces a filespecifically and only
usable by autoload, so one might rightly think of zcompile as a subsidiary
program to autoload.  I doubt you could change one without considering the
effect on the other. I'd say it's the same with zip/unzip--if one 
changes then
the other must change--they are a suite of linked programs. But whereas
unzip does not 'use' the content of whatever it unzips, autoload uses
a .zwc file and is the only program that does. Thusly, a mnemonic similarity
between names would be very mildly usefulthe same way that 'unzip'
is usefully similar to 'zip'.  In the real world of course, renaming 
zcompile
would do far more harm than good, so it's an academic discussion. Had
I been around 20 years ago, I'd have suggested naming it 'makezwc' or
something like that, or 'autoload --makezwc' or some such. One is given
a tiny but friendly hint that things are related, that's all. But unix 
is not
a friendly placeand seems not to want to be.  It delights in its obscurity.

Re: autoload

[Bart Schaefer]

On Sep 20,  8:37am, Ray Andrews wrote:
}
} OTOH zcompile produces a filespecifically and only
} usable by autoload

No, that's not true:  zcompiled files may be used by either autoload
(strictly speaking, by the deferred-loading procedure that USUALLY
happens long after the autoload command has been run) OR by source,
and the semantics of the two are not the same.

-- 
Barton E. Schaefer

Re: autoload

[Ray Andrews]

On 09/19/2015 02:22 PM, Bart Schaefer wrote:
> As I've pointed out, these are exactly the same as
>
>      zcompile /Functions/Functions.zwc /Functions/*
>      FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc
>
> Although the better practice would be to do
>
>      zcompile /Functions.zwc /Functions/*
>      FPATH=/Functions.zwc autoload +X -w /Functions.zwc

By gum it works if you get it exactly right:

-"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the 
same path. Fussy fussy.
-"autoload +X -w ..." works with Sebastian's, but not with mine (which 
are all functions).
-"autoload -w ..." works with mine but not Sebastian's.
-"source miscfunctions" *before* calling an autoloaded function to have 
it work the first time.
-Append new dirs to FPATH or other things break.

> if not because of the way fpath is searched, then for the same reason that
> you shouldn't do
>
>      tar -c /Functions/Functions.tar /Functions/*
... and most importantly, follow this advice, or if you rerun zcompile 
where there's already a
.zwc, it rather dumbly tries to eat it's own tail and build itself into 
itself and nothing works.

I guess the zcompile of my dreams would be able to process both scripts 
and functions in one
go, and be able to see the functions in 'miscfunctions' regardless of 
the filename, and not
try to digest existing .zwc's, and not worry about directories. And the 
autoload of my dreams
would be able to pass all of it back every time.  Best to leave it to 
zsh internals it's too
fragile for anything else.  Apologies to the list for all this.

Re: autoload

[Bart Schaefer]

On Sep 19,  5:58pm, Ray Andrews wrote:
}
} By gum it works if you get it exactly right:
} 
} -"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the 
} same path. Fussy fussy.

You *really* don't want that :t in there.

} -"autoload +X -w ..." works with Sebastian's, but not with mine (which 
} are all functions).

This is either because (a) you needed "zcompile -k ..." with yours, or
(b) you have multiple functions per file, which we've already discussed
at length with respect to autoload.  [More on this below.]

} -"autoload -w ..." works with mine but not Sebastian's.

I think "autoload" is entirely superfluous in this case, because:

} -"source miscfunctions" *before* calling an autoloaded function to have 
} it work the first time.

Er, no.  In that case it's 99% certain the function isn't autoloaded at
all, rather it's simply been fully defined during the "source" command.
The "source" command is probably reading miscfunctions.zwc, though.

} -Append new dirs to FPATH or other things break.

I have no idea whether that means "the FPATH has to contain some new
stuff" or also "... and the new stuff has to go at the end, it can't
be put anywhere else".  Either could be true or false depending on the
context.

} I guess the zcompile of my dreams would be able to process both
} scripts and functions in one go

This has nothing to do with zcompile output and everything to do with
how the result is read back.  Scripts you want to "source ..." and
functions you want to define (and possibly also call, if defining is
happening as part of deferred loading).

(Maybe we should never have created the "autoload" command, because
that name just muddles up the act of creating a placeholder for
deferred loading with the actual deferred loading event itself.)

[I just noticed that the doc for zcompile refers to "execution of
scripts".  This is misleading, it should say "sourcing of scripts".
Execution (as in spawning a new shell to run a file that has execute
mode bit set) does not involve loading .zwc files; only "source"
and "." do that.]

Also zcompile has no way to guess if the file it's being used on is a
source-able script or a function body.  You'd need to tell it which,
somehow.  So "all in one go" would become some convoluted thing like

zcompileOfMyDreams --script=miscfunctions --function=n-list ...

} and be able to see the functions in 'miscfunctions' regardless of
} the filename

This is for all practical purposes impossible to do automatically.
In order to know what functions are in "miscfunctions", zcompile
would have to first *run* (source) that file, which might have
other side effects besides just defining the functions themselves.

Neverthe less zcompile does have a way to do what you're asking,
assuming you know there are no side-effects:

    ( # subshell so as not to muck up the parent
      unfunction -m \*		# clean out all extra functions
      source miscfunctions	# load the functions we care about
      zcompile -a -c miscfunctions.zwc	# compile all the functions
    )

Note that here you are explicitly doing the "run that file" step.

} and not try to digest existing .zwc's,
} and not worry about directories.

This is a philosophical issue along the same lines as CSH_NULL_GLOB.
If I name a bunch of things and some of them are wrong, should the
shell complain about them or not, and either way, should it skip
them and continue working on the ones that are correct?  Which is
more likely to produce the final result that the user expected?

Since the user is perfectly capable of filtering these out when the
argument list is built, I think the decision to complain and stop
is the correct one.

} And the autoload of my dreams would be able to pass all of it back
} every time.

Maybe if you try replacing "autoload" with its synonym "functions -u"
(create an undefined function) in your mental map, you'd get a better
idea of what you're asking.

What would it mean to create an undefined script?  What event would
cause it to become defined?

Re: autoload

[Ray Andrews]

On 09/19/2015 10:41 PM, Bart Schaefer wrote:
> } -"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the
> } same path. Fussy fussy.
>
> You *really* don't want that :t in there.

Yup.  It turns out that it was the regular versions of the functions 
that were being found.
> This is for all practical purposes impossible to do automatically.

Sure, I can't argue, you know how it works.  My dream of simplicity may 
indeed
be impossible.  But you know what's in my head--something that's the 
equivalent of
'just sourcing' your functions, but doing it from precompiled code, and 
not pre-loading
anything at all, but doing it only when the function is actually 
called.  Nuts, it's so
elegant in theory.  I'm sorta in love with the idea but implementation 
becomes
a can of worms.
> } and not try to digest existing .zwc's,
> } and not worry about directories.
>
> This is a philosophical issue along the same lines as CSH_NULL_GLOB.
> If I name a bunch of things and some of them are wrong, should the
> shell complain about them or not, and either way, should it skip
> them and continue working on the ones that are correct?  Which is
> more likely to produce the final result that the user expected?
Indeed, and the right answer is not obvious.  Hand holding tends to end up
creating more problems that it solves.  I'd suppose that if there are 
built in
exceptions to the plain reading of a command it must be exceptions that
could never be desired ever, like, perhaps, including a .zwc inside it's own
replacement. Or not.  I always hated Microsoft for making 80% of things
easier and the remaining 20% impossible.  If it has to be tough, then it has
to be tough.

You know Bart, with all the time you've spent on this, you could have 
written
that HOWTO ;-)

Seriously, it requires something broader than just reading the manual.
>

Re: autoload

[Bart Schaefer]

On Sep 20,  4:21pm, Ray Andrews wrote:
}
} But you know what's in my head--something that's the equivalent of
} 'just sourcing' your functions, but doing it from precompiled code,
} and not pre-loading anything at all, but doing it only when the
} function is actually called.

Except for the point that "just sourcing" doesn't actually call the
function, everything you wrote is what DOES happen as long as the
functions are defined in separate files.  Because the language is
interpreted, the shell itself can't decompose a file into individual
function entry points the way e.g. a linker can for a shared object.
Requiring the file basename to be the same as the entry point name is
the most obvious way to handle this.

} You know Bart, with all the time you've spent on this, you could have 
} written that HOWTO ;-)

The problem is, I wouldn't have known what to write.

Re: autoload

[Ray Andrews]

On 09/20/2015 09:18 PM, Bart Schaefer wrote:
> Except for the point that "just sourcing" doesn't actually call the
> function, everything you wrote is what DOES happen as long as the
> functions are defined in separate files.  Because the language is
> interpreted, the shell itself can't decompose a file into individual
> function entry points the way e.g. a linker can for a shared object.
Really?  That's hard to fathom.  I source a file and it finds whatever 
functions may
be in there, but they can't be packed into a .zwc file without worrying 
about it's
name?  I'll take that on authority, but it blows my breakers.  I'd have 
thought it
would be as transparent as:

     for aa in /aWorking/Zsh/Source/*; do source $aa; done

... becoming:

zcompile --do_it  /not-the-same-directory-dummy\!/Source.zwc 
/aWorking/Zsh/Source/*

... but if you say that's not even doable I can't argue.
> } You know Bart, with all the time you've spent on this, you could have
> } written that HOWTO ;-)
>
> The problem is, I wouldn't have known what to write.
>
Yeah, the paradox of the expert.  Experts don't know what they know.  
The best
writer is the guy who just learned it so he remembers what he needed to 
learn,
where he went wrong, what was important, what was distraction.  Then the
expert edits it for factuality.  Once four or five learners have added their
lessons to the doc, it becomes really usable for the next guy.  Like the 
best
wikis.  The less of an expert you are, the better judge you are as to the
usefulness of a doc.

Re: autoload

[Bart Schaefer]

On Sep 21, 10:03am, Ray Andrews wrote:
} Subject: Re: autoload
}
} On 09/20/2015 09:18 PM, Bart Schaefer wrote:
} > Because the language is
} > interpreted, the shell itself can't decompose a file into individual
} > function entry points the way e.g. a linker can for a shared object.
} 
} Really? That's hard to fathom. I source a file and it finds whatever
} functions may be in there, but they can't be packed into a .zwc file
} without worrying about it's name?

Did you miss the email where I explained how to do that?

} I'd have thought it would be as transparent as:
} 
} zcompile --do_it  /not-the-same-directory-dummy\!/Source.zwc 
} /aWorking/Zsh/Source/*

Really think about "because the language is interpreted."

Suppose I have

    typeset some_critical_var_used_by_foo_and_bar=really_important_value

    foo() { ... }
    bar() { ... }

Now I want to compile this.  Sure, the shell could dump out foo and bar,
but where does it put that typeset?  Does it evaluate the assignment at
compile time?  (A C compiler would, but that's typically the wrong
semantics for a shell.)

What if really_important_value absolutely must only be interpreted once,
not each time foo and bar are individually loaded?

What if the names "foo" and "bar" are actually stored in other variables?

    function $foo { ... }

What if there's an "if" structure?

    if some_runtime_test
    then
	foo() { ... }
	bar() { ... }
    else
        bar() { ... }
	foo() { ... }
    fi

What if there's a runtime test with that crucial typeset INSIDE the test?

I can make up an infinite number of increasingly convoluted cases to
break any "simple, elegant" automation.  So unless you happen to have
"zmodload zsh/artificial-intellgence" lying around ready for use, I'm
done with this discussion.

Re: autoload

[Ray Andrews]

On 09/21/2015 11:17 AM, Bart Schaefer wrote:
> Did you miss the email where I explained how to do that?
On the first pass through a subject like this there is so much to learn
and so much error and misunderstanding and diversions and distractions
that one is lucky to come out of it alive.  On the second pass things will
gel.
>
> Really think about "because the language is interpreted."
Deep lessons ...
>
> I can make up an infinite number of increasingly convoluted cases to
> break any "simple, elegant" automation.  So unless you happen to have
> "zmodload zsh/artificial-intellgence" lying around ready for use, I'm
> done with this discussion.
>
I'd just expect the precompiled function to behave exactly as a sourced 
function
would behave, all that's different is that the file being parsed is ... 
is this where
I go off the rails? ... what I'd call 'object code' which is to say that 
it's somewhat
pre-digested as far as the interpreter goes.  But when it is actually 
called (as if
it were sourced) it would behave exactly identically to a normal sourced 
function.
It's just parsed quicker.  Very different from C, as you say.

I dunno, if I have a function ' test () { echo $PWD } ' I'm not 
expecting $PWD to
be hard captured at sourcing time, is there even a way to do that? It 
might be
cool if there was a way, but surely that's rather unusual.  So, on the 
contrary,
I'm thinking not of artificial-intelligence but rather of something
dumb as a donkey.  Autoloaded functions at present seem to behave as I'd
expect, so I don't know where the problem arises.  Your 'evaluate once'
scenario is valid of course, but I have no idea how that would be 
handled even
by a regular function.  It seems a separate issue to me.

To be honest, your stamina sticking with this so long has already 
astonished me,
so if you're done, that's overdue, but thanks.

Re: autoload

[Bart Schaefer]

OK, one last.

On Sep 21,  1:09pm, Ray Andrews wrote:
}
} I'd just expect the precompiled function to behave exactly as a
} sourced function would behave, all that's different is that the file
} being parsed is ... is this where I go off the rails?

Yes.

You're not asking for a precompiled function to act like a sourced
function.  A sourced function, to the extent that means anything at all,
does nothing but become defined.  But "sourcing" a function (which you
actually can't do, you can only source a *file*) and "calling" a
function are two completely distinct things.  Rather, you ask for
compiling a whole file full of defined functions, to behave like
creating a bunch of undefined ones.

An undefined function (created with "functions -u" aka autoload) does
both of the source and call operations, but it only knows the one name
(its own).  So the file (to be sourced) and the function (to be called)
both have to be identified by that single name.

Could we have created an entire distinct data structure that stores
both a file name and a function name, and means "oh by the way, when
you try to run foo, you actually have to go read the file bar" ?
Yes, we could.  We didn't.  We re-used the existing data structure
of a function, because one field for the name seemed like enough,
and the path-search mechanism seemed like a good existing model for
finding the file.

Further, the point of autoloading was to read into memory exactly
those functions that are actually used.  If to get foo you have to
source bar and do everything bar contains, you may end up reading
(and storing) a hundred functions in order to call one of them.
Not what we wanted to encourage.  So, one function per file.

zcompile lets you do two things:  compile a single file to be read by
source (*) or pack multiple files into an archive to be processed
by autoload.  But the latter doesn't change the fact that autoload
needs to match one file (one thing packed in the archive) to one
function name.  And the former doesn't magically grant the ability
to explode the single file into multiple archive entries. (**)

(*) But this is constrained in that both the original file and the
compiled file have to exist size-by-side, so that source can choose
the newer of the two.  This constraint isn't strictly necessary.

(**) "zcompile -c" can explode functions that are already in memory
into multiple "files" that then go into the archive, but that's as
close as it gets.

Re: autoload

[Ray Andrews]

On 09/21/2015 08:19 PM, Bart Schaefer wrote:
> You're not asking for a precompiled function to act like a sourced
> function.  A sourced function, to the extent that means anything at all,
> does nothing but become defined.  But "sourcing" a function (which you
> actually can't do, you can only source a *file*) and "calling" a
> function are two completely distinct things.
I understand, I just chose the wrong word.
>    Rather, you ask for
> compiling a whole file full of defined functions, to behave like
> creating a bunch of undefined ones.
>
> An undefined function (created with "functions -u" aka autoload) does
> both of the source and call operations, but it only knows the one name
> (its own).  So the file (to be sourced) and the function (to be called)
> both have to be identified by that single name.
>
> Could we have created an entire distinct data structure that stores
> both a file name and a function name, and means "oh by the way, when
> you try to run foo, you actually have to go read the file bar" ?
> Yes, we could.  We didn't.  We re-used the existing data structure
> of a function, because one field for the name seemed like enough,
> and the path-search mechanism seemed like a good existing model for
> finding the file.
Ok.  I admit that my notion does presume a table linking functions with 
their
.zwc:

raysfunc    /directory/raysfunc.zwc    line1
somefunc  /directory/all/allfunc.zwc  line40
otherfunc  /directory/all/allfunc.zwc  line200
...

I get a glimpse of how you did what you did within existing structures. 
If I understood
that fully I'd grok the limitations of the system.  I guess it's a bit 
outrageous of me,
but I presumed that in the hash table that links functions to where ever 
they are,
a sort of pointer to a line in a .zwc file would not be a huge problem.
>
> Further, the point of autoloading was to read into memory exactly
> those functions that are actually used.  If to get foo you have to
> source bar and do everything bar contains, you may end up reading
> (and storing) a hundred functions in order to call one of them.
> Not what we wanted to encourage.  So, one function per file.

That is so insightful, cuz I had exactly the opposite as the supposed 
goal--to
have functions available but not sourced *just in case* you might need them.
I never understood:

     autoload -U colors && colors

... because I'm thinking to myself: 'Why would you 'autoload' something 
that
you are just going to plain vanilla load anyway? Why not just load it 
and be
done with it?' (It doesn't work with compinit.  It seems because 
compinit will
be looking on FPATH for the rest of itself autoloaded, so it's committed to
that method--but you understand my thought.)
>
> zcompile lets you do two things:  compile a single file to be read by
> source (*) or pack multiple files into an archive to be processed
> by autoload.  But the latter doesn't change the fact that autoload
> needs to match one file (one thing packed in the archive) to one
> function name.

But ... if we have several functions packed in one .zwc it is not 
logically possible
for the name of the file to match the name of more than one of the 
functions which
breaks my head as far as this one-to-one function-file thing.  You tell 
me that
there is this function name - file name requirement but then violate that
commandment by packing in several functions to one .zwc.  I get that if 
a script
is autoloaded it will/must be 'named' after the file it came from. One 
script!
But with functions?  The rule is not followed.

> And the former doesn't magically grant the ability
> to explode the single file into multiple archive entries. (**)
>
> (*) But this is constrained in that both the original file and the
> compiled file have to exist size-by-side, so that source can choose
> the newer of the two.  This constraint isn't strictly necessary.
Well therewego.  Previously you're saying that these .zwc should not be 
in the same
dir, now they should be in the same dir. Nuts, they are created in the 
same dir.  I
tested that by touching the files--as you say, it finds the newest. But 
then you tell
me that best practice is to move the thing :-(  And, speaking of 
philosophy, here
is a case of hand-holding.  The first command found on the path is the 
one used,
and too bad for you if a newer version exists somewhere lower down. It 
doesn't
scan the whole path and load the newest file.  I'm expecting that if I 
autoload
something then it's active and too bad if there's a newer file:

     $ autoload my_function
     $ source /functions/myfunction

... I get the sourced version.

     $ source /functions/myfunction
     $ autoload my_function

... I get the version in the .zwc  found on FPATH.  If I want the newer 
version
then I should source it.

>
> (**) "zcompile -c" can explode functions that are already in memory
> into multiple "files" that then go into the archive, but that's as
> close as it gets.
>
When I revisit this, I'll read your posts backwards.  This one is the 
one that would come first
in the HOWTO.  The first thing to understand is why, when and where we'd 
want to
use autoload.  What is it for?  Why did God make it on the seventh day?  
Then, it's
constraints and why they are what they are, then the implementation with 
examples.

Looking back, I've misunderstood the whole point of the thing.  Like the 
C-head
that I am, I'm thinking: 'What a good idea!  We can boil down our 
functions to
'.obj' files for faster loading, pack them together into one file for 
more speed yet, and
best of all, have them unloaded in memory until they are actually called 
at which time
they are 'autoloaded' automagically.  How elegant, I says to myself, 
it's like a .zip--all
compressed and packed together, and just as you can extract a single 
file from a .zip
so you can source a single function from a .zwc and zsh will even do it 
automatically
for you if you autoload that function cuz zsh will keep a table of where 
that
function is found and source it only when you call for the function.

It's all Sebastian's fault ;-)

Seriously sir, I exhaust your patience.  For now the one thing I know is 
that I don't
want to autoload anything, it's too obscure.

Re: autoload

[Bart Schaefer]

Just so no one else is confused ...

On Sep 22, 10:33am, Ray Andrews wrote:
}
} > zcompile lets you do two things:  compile a single file to be read by
} > source (*) or pack multiple files into an archive to be processed
} > by autoload.
} >
} > (*) But this is constrained in that both the original file and the
} > compiled file have to exist size-by-side, so that source can choose
} > the newer of the two.  This constraint isn't strictly necessary.
} 
} Well therewego.  Previously you're saying that these .zwc should not be
} in the same dir, now they should be in the same dir.

Two different use cases:

(1) Packing up multiple files into a composite ZWC?  Don't put the
    composite alongside the files packed in it.  This is the case we
    started out talking about two dozen messages ago.  Makes sense
    only for autoload.

(2) Packing one file into one correspondingly-named ZWC?  Put the ZWC
    alongside the origin file.  Works for source or autoload, but in
    the autoload case this must be one-function/one-file/same-name.

The autoload search is straightforward:  $fpath order comes first.
If, while searching fpath in order, a pair of files "x" and "x.zwc"
is found, read the newer one.  As a special case of this, if "x" is
actually a directory listed in fpath, assume "x.zwc" contains all the
same files as "x/".

The "source" (and ".") search rule is similar, except using $path and
without the directory special case.

}      $ autoload my_function
}      $ source /functions/myfunction
} 
} ... I get the sourced version.

Yes, because in this case you clobbered the autoload declaration by
loading the full function definition on top of it with "source".

}      $ source /functions/myfunction
}      $ autoload my_function
} 
} ... I get the version in the .zwc found on FPATH.

No, you still get the one from the "source" command, because autoload
won't replace an existing function definition.  The only way to ever
get the version found on fpath is by never doing the "source" at all.

Re: autoload

[Ray Andrews]

On 09/22/2015 09:39 PM, Bart Schaefer wrote:
> Just so no one else is confused ...
>
>
Indeed.  But 3/4 of what we've discussed is not documented so others might
very well be unsure of these things too.  I did really jump to 
conclusions as
to how the thing would work, and I'm guilty of wishful thinking as always.
Still I do wish this thread could be gelled into even the outline of a 
howto
sort of doc. that way it would have served some purpose.  It seems one must
understand autoload from it's roots.