autoload

autoload

[Ray Andrews]

Bart:

I did this, which should be verbatim as you posted:


     # Assume starting here with the default $fpath
     zsh_default_functions=~/.zsh-default-functions.zwc
     if ! zcompile -t $zsh_default_functions >&/dev/null
     then
       # File is missing or out of date.  Rebuild it.
       # Removes the file if any function cannot be compiled.
       zcompile $zsh_default_functions $^fpath/*(N.:A)
     fi
     if [[ -f $zsh_default_functions ]]
     then
       fpath=( $zsh_default_functions )
       autoload -w $zsh_default_functions
     fi

... and I get:

     /aWorking/Zsh/Boot/ztest:5: parse error near `)'
     /aWorking/Zsh/Boot/ztest:zcompile:10: can't read file: /usr/share 
                 /zsh/functions/Completion /Base.zwc

('ztest' being sourced by .zshrc and where I do experiments.)

'Base.zwc' exists:

     $ ls -l Base.zwc
     -rw-r--r-- 1 root root 269472 Aug 31 16:26 Base.zwc

... and  "zcompile -t Base.zwc" doesn't report anything amiss.

The  " parse error near `)' " msg. is puzzling since it always reports 
line 5 regardless of anything added above it ,eg: "echo "this echo is 
only here to add a line of code"".

For example if I modify your code like this:

     line5
         # Assume starting here with the default $fpath
zsh_default_functions=~/.zsh-default-functions.zwc
     line9

... I get this:

     /aWorking/Zsh/Boot/ztest:5: command not found: line5
     /aWorking/Zsh/Boot/ztest:9: command not found: line9
     /aWorking/Zsh/Boot/ztest:5: parse error near `)'
     /aWorking/Zsh/Boot/ztest:zcompile:14: can't read file: /usr/share 
/zsh/functions/Completion/Base.zwc

... so the deliberate errors are as expected, yet she insists on finding 
an `)' error on line 5 as well, and that after having reached line 9, so 
I have no idea how to understand that.

Re: autoload

[Bart Schaefer]

On Sep 17,  8:54am, Ray Andrews wrote:
} Subject: autoload
}
}      # Assume starting here with the default $fpath

I think the problem here is that you've put a .zwc file IN a directory
that is itself listed in $fpath.  This is typically a no-no.  You want
a .zwc file to be treated as if it WERE a directory, NOT as if it were
one of the function definition files IN the directory.

If you must have the .zwc where you've presently got it, then you need
to exclude it from the file pattern $^fpath/*(N.:A) -- which is easiest
if you have extendedglob set, so you can write $^fpath/*~*.zwc(N.:A)
instead.

}      /aWorking/Zsh/Boot/ztest:5: parse error near `)'
}      /aWorking/Zsh/Boot/ztest:zcompile:10: can't read file: /usr/share 
}                  /zsh/functions/Completion /Base.zwc

See, zcompile (without the -t) is attempting to parse Base.zwc as a
function definition, and getting what looks like garbage to the parser.
(I have no idea where those spaces in the path are coming from.)

Re: autoload

[Ray Andrews]

On 09/17/2015 10:34 AM, Bart Schaefer wrote:
> On Sep 17,  8:54am, Ray Andrews wrote:
> } Subject: autoload
> }
> }      # Assume starting here with the default $fpath
>
> I think the problem here is that you've put a .zwc file IN a directory
> that is itself listed in $fpath.  This is typically a no-no.  You want
> a .zwc file to be treated as if it WERE a directory, NOT as if it were
> one of the function definition files IN the directory.

I'm close to certain I haven't, here's all .zwc on my system:


> /usr/share/zsh/functions/VCS_Info/Backends.zwc
> /usr/share/zsh/functions/Exceptions.zwc
> /usr/share/zsh/functions/VCS_Info.zwc
> /usr/share/zsh/functions/Prompts.zwc
> /usr/share/zsh/functions/MIME.zwc
> /usr/share/zsh/functions/Completion.zwc
> /usr/share/zsh/functions/TCP.zwc
> /usr/share/zsh/functions/Zle.zwc
> /usr/share/zsh/functions/Chpwd.zwc
> /usr/share/zsh/functions/Misc.zwc
> /usr/share/zsh/functions/Calendar.zwc
> /usr/share/zsh/functions/Completion/Debian.zwc
> /usr/share/zsh/functions/Completion/X.zwc
> /usr/share/zsh/functions/Completion/openSUSE.zwc
> /usr/share/zsh/functions/Completion/AIX.zwc
> /usr/share/zsh/functions/Completion/Mandriva.zwc
> /usr/share/zsh/functions/Completion/Base.zwc
> /usr/share/zsh/functions/Completion/Zsh.zwc
> /usr/share/zsh/functions/Completion/Linux.zwc
> /usr/share/zsh/functions/Completion/Cygwin.zwc
> /usr/share/zsh/functions/Completion/Solaris.zwc
> /usr/share/zsh/functions/Completion/Darwin.zwc
> /usr/share/zsh/functions/Completion/Redhat.zwc
> /usr/share/zsh/functions/Completion/BSD.zwc
> /usr/share/zsh/functions/Completion/Unix.zwc
> /usr/share/zsh/functions/Zftp.zwc
> /usr/share/zsh/functions/Newuser.zwc
>
>
> See, zcompile (without the -t) is attempting to parse Base.zwc as a
> function definition, and getting what looks like garbage to the parser.
> (I have no idea where those spaces in the path are coming from.)
>
It isn't possible I've somehow screwed up $fpath?  I understand it's 
hard coded at startup,
least it seems to always be restored. Anyhow I've not moved or played 
with Base.zwc.  I did try
creating a .zwc yesterday but it's long dead.

This seems to be the troublesome line:

      zcompile $zsh_default_functions $^fpath/*(N.:A)

... on a hunch:

      zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a 
new friend

... the previous error msgs are gone but:

     $ zsh
     /aWorking/Zsh/Boot/ztest:zcompile:13: can't open file: _bootctl

Re: autoload

[Bart Schaefer]

On Sep 17, 12:22pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} On 09/17/2015 10:34 AM, Bart Schaefer wrote:
} >
} > I think the problem here is that you've put a .zwc file IN a directory
} > that is itself listed in $fpath.
} 
} I'm close to certain I haven't, here's all .zwc on my system:

The issue isn't (just) what your .zwc files are, it's also what you
$fpath contains at the time that your .zshrc or whatever is loaded.

E.g. $^fpath/*(.) appears to include
    /usr/share/zsh/functions/Completion/Base.zwc
as one of the matching files.

Therefore /usr/share/zsh/functions/Completion must be in the default
$fpath, and what I'm saying is that you should not normally put Base.zwc
in a directory that is in $fpath.

} It isn't possible I've somehow screwed up $fpath?  I understand it's 
} hard coded at startup,

Yes, and the problem is that you've put .zwc files into the directories
in that hardcoded list, but then tried to use the hardcoded list to
(re)build yet another .zwc file.

I'm not really sure how further to explain this.  The sample code that
I provided takes EVERY DIRECTORY that's in (the default) $fpath and
packs ALL the files under them into ONE .zwc.

You on the other hand seem to have already built a separate .zwc out of
EACH directory of the default $fpath.  If you're using my example, you
don't need to do that - you should throw all those .zwc away and use
the single one that my sample builds.

Or conversely you should not use that example.

}       zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a 
} new friend

A different enemy, more like.

} ... the previous error msgs are gone but:
} 
}      $ zsh
}      /aWorking/Zsh/Boot/ztest:zcompile:13: can't open file: _bootctl

The :t discards /usr/share/zsh/... and leaves just a base file name;
that might work for autoload but won't for zcompile.  You need the
full paths, just not the .zwc files.

Re: autoload

[Ray Andrews]

On 09/17/2015 01:40 PM, Bart Schaefer wrote:
> Yes, and the problem is that you've put .zwc files into the directories
> in that hardcoded list, but then tried to use the hardcoded list to
> (re)build yet another .zwc file.
But I didn't create those files. They're dated August 31 which seems to 
be common to
most of the files in the distro as I downloaded it.  Is this something 
to do with Debian?

> I'm not really sure how further to explain this.  The sample code that
> I provided takes EVERY DIRECTORY that's in (the default) $fpath and
> packs ALL the files under them into ONE .zwc.
That's how I read it.

     You on the other hand seem to have already built a separate .zwc 
out of EACH
     directory of the default $fpath. If you're using my example, you 
don't need to
     do that - you should throw all those .zwc away and use the single 
one that my
     sample builds.


I tried deleting all of them which resulted in ~/.zsh-default-functions.zwc
being created, but zero length and of course nothing worked.  With 
original .zshrc code, things worked but a 'time' test was very slow, 
thus demonstrating the effectiveness of the .zwc
idea itself (by it's absence, in this case). I restored the .zwc files 
and I'm back to normal.
This seems to be a very tricky business.  Perhaps not worth wasting your 
time on.  It seems
a shame that zcompile doesn't know not to trip over itself, one might 
expect to find a .zwc
file in the same dir as the files from which it was built.  Or not.

> }       zcompile $zsh_default_functions $^fpath/*(N.:t)  << 't' being a
> } new friend
>
> } A different enemy, more like.
>

Sure, it was just experiment, I hardly expected it to be correct.

Re: autoload

[Bart Schaefer]

On Sep 17,  4:06pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} But I didn't create those files. They're dated August 31 which seems
} to be common to most of the files in the distro as I downloaded it. Is
} this something to do with Debian?

Possibly.  I've never installed a pure Debian distro, only Ubuntu.

} I tried deleting all of them which resulted in ~/.zsh-default-functions.zwc
} being created, but zero length and of course nothing worked.

That's a little confusing since with :t in there you at least got an
attempt to read _bootctl so there must be some function files.

What does

() { emulate -L zsh -o extendedglob; print -lR $^fpath/*~*.zwc(N.:A) | head }

output, if placed early in your startup files (before you do any of your
own changes to $fpath)?

Re: autoload

[Ray Andrews]

On 09/17/2015 04:20 PM, Bart Schaefer wrote:
> On Sep 17,  4:06pm, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } But I didn't create those files. They're dated August 31 which seems
> } to be common to most of the files in the distro as I downloaded it. Is
> } this something to do with Debian?
>
> Possibly.  I've never installed a pure Debian distro, only Ubuntu.
My first distro was Mint, and 'downgrading' to plain vanilla Debian was 
the smartest thing
I ever did--less baloney all 'round.   I don't even know what it is that 
I'm supposed to be
missing.  Anyway, those .zwc files are in the Debian package, I just 
checked.  Perhaps Axel
has a comment?
>
>
> That's a little confusing since with :t in there you at least got an
> attempt to read _bootctl so there must be some function files.
That's with 't' gone, and back to 'A'.
> What does
>
> () { emulate -L zsh -o extendedglob; print -lR $^fpath/*~*.zwc(N.:A) | head }
>
> output, if placed early in your startup files (before you do any of your
> own changes to $fpath)?
>
0 HP-w5--5-Debian1 /aWorking/Zsh 2$ zsh
/usr/local/share/zsh/site-functions/n-aliases
/usr/local/share/zsh/site-functions/n-cd
/usr/local/share/zsh/site-functions/n-env
/usr/local/share/zsh/site-functions/n-functions
/usr/local/share/zsh/site-functions/n-history
/usr/local/share/zsh/site-functions/n-kill
/usr/local/share/zsh/site-functions/n-list
/usr/local/share/zsh/site-functions/n-list-draw
/usr/local/share/zsh/site-functions/n-list-input
/usr/local/share/zsh/site-functions/n-options


... without 'head' it shows everything and it all looks fine.

Re: autoload

[Bart Schaefer]

On Sep 17,  6:20pm, Ray Andrews wrote:
} Subject: Re: autoload
}
} Anyway, those .zwc files are in the Debian package, I just checked.

In that case you should probably just do:

    autoload -w $^fpath/*.zwc(N)

and forget about rebuilding from the source functions.

-- 
Barton E. Schaefer

Re: autoload

[Ray Andrews]

On 09/17/2015 09:04 PM, Bart Schaefer wrote:
> On Sep 17,  6:20pm, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } Anyway, those .zwc files are in the Debian package, I just checked.
>
> In that case you should probably just do:
>
>      autoload -w $^fpath/*.zwc(N)
>
> and forget about rebuilding from the source functions.
Yes, that was quite the red herring there.  Interesting that Debian does 
things
differently like that.  I suppose they are free to, but it still seems 
strange that
there'd be no doc to the effect that their package is different from 
std. like that.
Maybe it's not to be expected.  I guess the 'line 5 `)' error mystery is 
best
forgotten.  Very strange tho.

I'm trying to get the autoload system working with my own functions and I'm
almost there.  " /aWorking/Zsh/Source"  is where they live, and I've 
called them
like this:

     for aa in /aWorking/Zsh/Source/*; do source $aa; done

... which was the first thing I learned about such things.  They are all 
'normal'
functions and most files contain more than one.

Now I'm trying this:

$ fpath=( /aWorking/Zsh/Source "${fpath[@]}" )
     $ source miscfunctions
$ autoload -w /aWorking/Zsh/Source/Source.zwc
$ l
     [ nothing ]
     $ l
     [ now I get listing ]

... in the docs there's talk of this--loading vs. loading and 
executing--but I can't
figger it.  I've tried 'zcompile -k/z ...' and 'autoload k/z ...' but no 
combination seems
to prevent the first call to any function from being inert. 'zcompile -t 
...' always
shows me a list of files, not functions.  Strangely there seems
to be no trouble with multiple functions in one file except in the case of
'miscfunctions' so I have to source it separately.  (The other multiple 
functions
are all help functions only called by the main function in each file. I 
don't know
why that makes a difference, but it does.)  Also, I wonder why $fpath 
needs to
be modified since 'autoload -w' is given an explicit path to the .zwc 
file, so should
hardly be concerned with looking for it via $fpath.  I'd expect it to be 
hashed
and that's that.

Re: autoload

[Bart Schaefer]

On Sep 17, 10:00pm, Ray Andrews wrote:
}
} I guess the 'line 5 `)' error mystery is best
} forgotten.  Very strange tho.

It's not a mystery.  Try this:

    zsh -nf /usr/share/zsh/functions/Completion/Base.zwc

} $ fpath=( /aWorking/Zsh/Source "${fpath[@]}" )
} $ source miscfunctions
} $ autoload -w /aWorking/Zsh/Source/Source.zwc
} $ l
} [ nothing ]
} $ l
} [ now I get listing ]

The way autoload works is, the name of the file has to be the same as
the name of the function.  If you "source" the file, you execute the
file contents immediately, which in your case defines the functions.
If you "autoload" the function names, you NEITHER define NOR execute
them; you just mark them to be processed later.

Then when you try to execute, zsh looks for a file with the same name
as the function, parses it, and executes the result of the parse.
Except for handling of the positional parameters, this is *almost* the
same as

   functions[$filename:t]=$(<filename)
   eval "$functions[$filename:t]"

With "autoload -z" (the default when no option), zsh assumes the file
*was* the entire function body and does nothing further.

*If* the function was marked "autoload -k", then zsh assumes that the
eval-like step defined THAT function, and calls it by name a second time,
similar to:

   functions[$filename:t]=$(<filename)
   eval "$functions[$filename:t]"
   $filename:t

If we now throw zcompile into the mix, the -z / -k options to autoload
are ignored in favor of the -z / -k options to zcompile.  However, the
rule about the file name having to be the same as the function name
doesn't change.  Further "autoload -w" still doesn't define anything;
it just uses the index table in the zcompiled file to grab the list
of names to mark for later processing.

} Strangely there seems to be no trouble with multiple functions in one
} file except in the case of 'miscfunctions' so I have to source it
} separately. (The other multiple functions are all help functions only
} called by the main function in each file. I don't know why that makes
} a difference, but it does.)

Does what I just wrote, above, help?  In the "miscfunctions" case, the
name of the file matches NONE of the names of the functions inside it;
they can't be autoloaded.  The name of the file has to match the name
of AT LEAST ONE function inside the file, *OR* the entire file has to
be that single function (as if it were a script).

} Also, I wonder why $fpath needs to be modified since 'autoload -w' is
} given an explicit path to the .zwc file

The internal implementation of autoloaded functions is unrelated to
hashing and does not retain any file paths.  It keeps only the base
names.  Where that set of names came from does not change that.

This is important to the consistent operation of fpath searching.  If
you have

    fpath=(/tmp/testfunctions ...)

and you create a new file /tmp/testfunctions/tryme, and the function
is marked for autoloading, then that new file has to be found ahead
of anything else, even if you previously used "autoload -w" on a .zwc
file that contains a "tryme" file.  The contents of fpath at the
instant of execution ALWAYS takes precedence; there is no "never get
this function from anywhere except [here]" in the autoload scheme.

Re: autoload

[Ray Andrews]

On 09/17/2015 10:52 PM, Bart Schaefer wrote:
> It's not a mystery.  Try this:
>
>      zsh -nf /usr/share/zsh/functions/Completion/Base.zwc
Ah, so the error is really from some other file.  I assumed it was some 
sort of
second parsing or something.  Nice if the called file was identified tho.
Anyway at least that's clear.

> If we now throw zcompile into the mix, the -z / -k options to autoload
> are ignored in favor of the -z / -k options to zcompile.  However, the
> rule about the file name having to be the same as the function name
> doesn't change.  Further "autoload -w" still doesn't define anything;
> it just uses the index table in the zcompiled file to grab the list
> of names to mark for later processing.
This could be made clearer in the docs.
> } Strangely there seems to be no trouble with multiple functions in one
> } file except in the case of 'miscfunctions' so I have to source it
> } separately. (The other multiple functions are all help functions only
> } called by the main function in each file. I don't know why that makes
> } a difference, but it does.)
>
> Does what I just wrote, above, help?  In the "miscfunctions" case, the
> name of the file matches NONE of the names of the functions inside it;
> they can't be autoloaded.  The name of the file has to match the name
> of AT LEAST ONE function inside the file, *OR* the entire file has to
> be that single function (as if it were a script).
I see.  Nuts, there's nothing about that in the docs is there?

> ...  The contents of fpath at the
> instant of execution ALWAYS takes precedence; there is no "never get
> this function from anywhere except [here]" in the autoload scheme.
>
As always Bart you make things as clear as they can be, but it sure is 
obtuse.  I long for:

     $ autoload --just_DO_IT  Source.zwc

... that is functionally identical to:

     $ for aa in /aWorking/Zsh/Source/*; do source $aa; done

( Of course that's after having done:)

      $ zcompile --just_DO_IT  Source.zwc  /aWorking/Zsh/Source/*


... but much faster--all functions are pre-digested to word code and 
ready for action, only not loaded into memory until/unless actually 
called.  'zcompile --just_DO_IT' is smart enough to create function 
names after filenames in the case of scripts, and of course named 
functions are named functions.  No need for wandering around in $fpath, 
the names of all the functions found in '.../Source' (regardless of the 
name of any file except in the case of scripts) and their definition 
.zwc are tabled for instant access so the user can consider them as 
having been sourced already, it's just much faster and trimmer.  (Speed 
is the point of all this, right?) "zcompile -t --just_DO_IT Source.zwc" 
returns the list of all *functions* found or created (from scripts) in 
'Source.zwc'.

Never mind, like completion, this is not for mortals to dabble in. My 
'for' loop does it all in one line and there's no gotchas. Transparency 
is not a zsh virtue.

Re: autoload

[Bart Schaefer]

On Sep 18,  8:49am, Ray Andrews wrote:
}
} > If we now throw zcompile into the mix, the -z / -k options to autoload
} > are ignored in favor of the -z / -k options to zcompile.  However, the
} > rule about the file name having to be the same as the function name
} > doesn't change.  Further "autoload -w" still doesn't define anything;
} > it just uses the index table in the zcompiled file to grab the list
} > of names to mark for later processing.
} 
} This could be made clearer in the docs.
[...]
} I see.  Nuts, there's nothing about that in the docs is there?

There's an entire section for "Autoloading Functions" but it has no
cross-reference from the "autoload" builtin (autoload points to
functions points to typeset which finally does have the section
reference).  I'll rearrange this a bit.

There IS a cross-reference to that section from the zcompile builtin.

}      $ autoload --just_DO_IT  Source.zwc
} 
} ... that is functionally identical to:
} 
}      $ for aa in /aWorking/Zsh/Source/*; do source $aa; done

You can do that:

FPATH=/aWorking/Zsh/Source autoload +X -w /aWorking/Zsh/Source/Source.zwc

The +X option is documented and means "load this now, but don't run it."

Re: autoload

[Ray Andrews]

On 09/18/2015 09:52 AM, Bart Schaefer wrote:
> } I see.  Nuts, there's nothing about that in the docs is there?
>
> There's an entire section for "Autoloading Functions" but it has no
> cross-reference from the "autoload" builtin (autoload points to
> functions points to typeset which finally does have the section
> reference).  I'll rearrange this a bit.
What would be sweet is a self-contained HOWTO sort of doc that walks you 
through the
entire subject with real world examples of solutions for real 
situations.  IOW, instead of
focusing on a command, it would focus on the goal: "Say you want your 
functions to
load faster.  Up till now you've been sourcing them at startup, which is 
fine, but
you can do much better ...."  ... 20 minutes latter you are cooking.
>
> }
> }      $ for aa in /aWorking/Zsh/Source/*; do source $aa; done
>
> You can do that:
>
> FPATH=/aWorking/Zsh/Source autoload +X -w /aWorking/Zsh/Source/Source.zwc
>
> The +X option is documented and means "load this now, but don't run it."
>
Ha!  Just when I'd decided to give up.

Re: autoload

[Bart Schaefer]

On Sep 18, 11:29am, Ray Andrews wrote:
}
} What would be sweet is a self-contained HOWTO sort of doc that walks
} you through the entire subject with real world examples of solutions
} for real situations.

Yes, but there are an almost unlimited number of such possible HOWTO
topics and no volunteers who dedicate their time to writing this sort
of thing.

zcompile was even considered too obscure to get a mention in the
"From Bash to Zsh" book, though you'll find a lot of other HOWTO-type
stuff in there.

Re: autoload

[Ray Andrews]

On 09/18/2015 12:02 PM, Bart Schaefer wrote:
> On Sep 18, 11:29am, Ray Andrews wrote:
> }
> } What would be sweet is a self-contained HOWTO sort of doc that walks
> } you through the entire subject with real world examples of solutions
> } for real situations.
>
> Yes, but there are an almost unlimited number of such possible HOWTO
> topics and no volunteers who dedicate their time to writing this sort
> of thing.
I would if I ever mastered a topic.  Anyway I find most linux 
documentation to be
almost sadistically unhelpful. zsh's is already better than most.

BTW I just realized why I couldn't find "Autoloading
Functions" in zshmisc: because it's "AUTOLOADING FUNCTIONS"  and the 
search is
case sensitive :-(   It's the little things that drive ya nuts.

Re: autoload

[Bart Schaefer]

I decided I should clarify something:

On Sep 17, 10:34am, Bart Schaefer wrote:
}
} I think the problem here is that you've put a .zwc file IN a directory
} that is itself listed in $fpath.  This is typically a no-no.  You want
} a .zwc file to be treated as if it WERE a directory, NOT as if it were
} one of the function definition files IN the directory.

This is not quite correct.  I thought the situation was that we were
talking about was

    fpath=(/path/to/Directory ...)
    /path/to/Directory/Directory.zwc	# contains multiple functions

This would be wrong.  Instead it should be one of

    fpath=(/path/to/Directory ...)
    /path/to/Directory.zwc		# contains multiple functions
or
    fpath=(/path/to/Directory ...)
    /path/to/Directory/function.zwc	# contains one function only

In fact what Debian has done is the first of those two (correct) options,
so my "typically a no-no" was off base.

A third correct option would be

    fpath=(/path/to/Directory.zwc ...)

which I prefer, but which skips comparing the modification time of that
file to the files in /path/to/Directory/.  The advantage is that the
Directory itself need not exist, you can put the .zwc file anywhere.

Now, here's the interesting twist to what Debian has done.  They have

fpath=(... /usr/share/zsh/functions/Completion ...
       /usr/share/zsh/functions/Completion/Base ...)

with file /usr/share/zsh/functions/Completion/Base.zwc (among others).

For autoload, this is ambiguous -- if someone tries to run a command
named "Base", zsh searches /usr/share/zsh/functions/Completion/Base.zwc
for that function, because it might be the "one function only" option
described above.  It won't find the function there, of course, so it
will go on and look elsewhere and no one the wiser, and probably no one
ever runs "Base" anyway.

I don't want to claim there is anything wrong with Debian's choice, I
merly point it out as a probably-unintended side effect.

Re: autoload

[Ray Andrews]

On 09/18/2015 05:14 PM, Bart Schaefer wrote:
> I decided I should clarify something:
>
...
> I don't want to claim there is anything wrong with Debian's choice, I
> merly point it out as a probably-unintended side effect.
>
If I understand, you are saying that a .zwc can't be in the same dir as 
the files from which it
was created?  Even though that is where it will be created?  I dunno, 
autoload/zcompile
seems like such a good idea but the implementation seems deliberately 
obscure.  As I read
the doc, it looked for the most recent file in whatever dir.  We have 
every possible twist
and turn, what we don't have is a simple and direct way of precompiling 
functions:

autoload --functions-out /All-My-Functions/All-My-Functions.zwc

... precompiled, *and/but* only loaded when called.  Exactly like 
sourcing, only faster and better.

Could even make making 'em look like this, just to be symetrical:

autoload --functions-in  /All-My-Functions/All-My-Functions.zwc 
/All-My-Functions/*

.... scripts, functions, however many per file, whatever the name ... no 
matter, it's all
wrapped up in the .zwc ready to go, just remembering that functions from 
scripts
will be 'auto-named' after the containing file.  Too cute?  Na ... it's 
easy to understand
and rather elegant.  The whole thing could be explained in two minutes
and implemented in 30 seconds.  Why do things *have* to be difficult?

As for me, I took Sebastian's stuff and converted it all to plain 
vanilla good old fashioned
righteous Protestant functions and may I never have to think about 
$fpath again ;-)
Seriously, I'll just let my brain cool down for a while and try to learn 
something
that a mortal can learn.  Sheesh, even Bart forgets how it works ...

Re: autoload

[Bart Schaefer]

On Sep 19,  8:04am, Ray Andrews wrote:
}
} If I understand, you are saying that a .zwc can't be in the same dir
} as the files from which it was created?

No.  I'm saying that if fpath is a tree, (X X/Y X/Y/Z), having
function files at all levels of the tree, then a file X/Y/Z.zwc
containing all the functions from Z might be wrongly interpreted
as containing a single function named Z instead.

} autoload --functions-out /All-My-Functions/All-My-Functions.zwc
} 
} autoload --functions-in  /All-My-Functions/All-My-Functions.zwc \
} /All-My-Functions/*

I don't understand your intended semantics of "out" and "in" here,
which argues against "expained in two minutes," but the point of
shell builtins is to provide building blocks for solutions like you
seem to be thinking of here.
 
} Sheesh, even Bart forgets how it works ...

Actually that's because I don't use it.  I'm nearly always either
running everything out of a development tree where I recompile zsh
(or pull new functions from git) sometimes multiple times a day,
or else running the same shell nonstop for weeks at a time without
ever restarting it.  There's no point in trying to keep a compiled 
bundle around if I'm never going to reference it.

Re: autoload

[Ray Andrews]

On 09/19/2015 09:29 AM, Bart Schaefer wrote:
> On Sep 19,  8:04am, Ray Andrews wrote:
> }
>
> } autoload --functions-out /All-My-Functions/All-My-Functions.zwc
> }
> } autoload --functions-in  /All-My-Functions/All-My-Functions.zwc \
> } /All-My-Functions/*
>
> I don't understand your intended semantics of "out" and "in" here,
I'm thinking about how the creation of a zwc file, and it's subsequent 
use could
be considered to be part of the same builtin.  One being the 'inverse' 
of the other.

autoload --make-zwc /Functions/Functions.zwc /Functions/*
autoload --load-zwc /Functions/Functions.zwc

... it wouldn't take long to understand.


> Actually that's because I don't use it.
It's very relevant to what degree veterans like yourself use stuff. I'd 
provisionally
conclude that if you don't use it, then I don't want it.  There's more 
to be learned
from seeing how masters actually use a tool than from reading the tool's 
manual.
The book I'd long to read would be: "A *practical* guide to zsh".

Re: autoload

[Bart Schaefer]

On Sep 19, 11:13am, Ray Andrews wrote:
} Subject: Re: autoload
}
} I'm thinking about how the creation of a zwc file, and it's subsequent 
} use could be considered to be part of the same builtin.

But you don't say

    compiler -build something source/*
    compiler -load something

Rather, you build, and then you install, and then you run, and the way
all shells are designed, you use $path to bridge the install + run steps.

$fpath is the same idea; "autoload" is extra glue to tell the shell
that you DON'T want to use the default ($path) search for those names.

} autoload --make-zwc /Functions/Functions.zwc /Functions/*
} autoload --load-zwc /Functions/Functions.zwc
} 
} ... it wouldn't take long to understand.

As I've pointed out, these are exactly the same as

    zcompile /Functions/Functions.zwc /Functions/*
    FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc

Although the better practice would be to do

    zcompile /Functions.zwc /Functions/*
    FPATH=/Functions.zwc autoload +X -w /Functions.zwc

if not because of the way fpath is searched, then for the same reason that
you shouldn't do

    tar -c /Functions/Functions.tar /Functions/*

Anyway, just make yourself a little wrapper function:

    loadorcompile() {	# rename as suits you
    	local how=$1; shift
	case $how in
	(--make-zwc) zcompile "$@";;
	(--load-zwd) FPATH=$1 autoload +X -w $1;;
	(*) print -u2 "don't know $how";;
	esac
    }

} It's very relevant to what degree veterans like yourself use stuff.
} I'd provisionally conclude that if you don't use it, then I don't
} want it.

As I tried to point out, I'm not a very typical user.  I would also
never use Sebastian's navigation tools, or anything from oh-my-zsh;
but that doesn't mean anyone else should avoid them.

Re: autoload

[Ray Andrews]

On 09/19/2015 02:22 PM, Bart Schaefer wrote:
> On Sep 19, 11:13am, Ray Andrews wrote:
> } Subject: Re: autoload
> }
> } I'm thinking about how the creation of a zwc file, and it's subsequent
> } use could be considered to be part of the same builtin.
>
> But you don't say
>
>      compiler -build something source/*
>      compiler -load something
>
> Rather, you build, and then you install, and then you run, and the way
> all shells are designed, you use $path to bridge the install + run steps.
Sure, it wasn't to be taken very seriously, I'm just thinking about the 
idea that there
could be a sort of symmetry between creating a .zwc and then loading 
one--you
could think of them as so related that the command name could be the 
same.  IOW
the creation of a .zwc presupposes subsequent use of it the same way 
that making
dinner supposes eating dinner.  Command recall would be easier ... but 
that's
never going to happen.  I guess the parallel would be 'zip' vs. 
'unzip'--we're glad
that the names are related.

> As I've pointed out, these are exactly the same as
>
>      zcompile /Functions/Functions.zwc /Functions/*
>      FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc
>
> Although the better practice would be to do
>
>      zcompile /Functions.zwc /Functions/*
>      FPATH=/Functions.zwc autoload +X -w /Functions.zwc
But ... well, let me try it again, I may have busted something.
> } It's very relevant to what degree veterans like yourself use stuff.
> } I'd provisionally conclude that if you don't use it, then I don't
> } want it.
>
> As I tried to point out, I'm not a very typical user.  I would also
> never use Sebastian's navigation tools, or anything from oh-my-zsh;
> but that doesn't mean anyone else should avoid them.
No, OTOH in my judgment typicality is not of great value whereas decades of
experience is.  Not that I have the slightest hesitation in doing things 
my own
way, but I always want to know what the Old Masters think even if I 
choose to
differ ... until learning better.
>

Re: autoload

[Bart Schaefer]

On Sep 19,  3:12pm, Ray Andrews wrote:
}
} the creation of a .zwc presupposes subsequent use of it the same way 
} that making dinner supposes eating dinner.

Obviously you've never worked in a restaurant.

} I guess the parallel would be 'zip' vs.  'unzip'--we're glad
} that the names are related.

But that's not a parallel at all, because unzip doesn't do anything
with the data that it unpacks.  A better parallel might be, creating
a DVD vs. playing it back.

Re: autoload

[Ray Andrews]

On 09/19/2015 10:53 PM, Bart Schaefer wrote:
> On Sep 19,  3:12pm, Ray Andrews wrote:
> }
> } the creation of a .zwc presupposes subsequent use of it the same way
> } that making dinner supposes eating dinner.
>
> Obviously you've never worked in a restaurant.
>
> } I guess the parallel would be 'zip' vs.  'unzip'--we're glad
> } that the names are related.
>
> But that's not a parallel at all, because unzip doesn't do anything
> with the data that it unpacks.  A better parallel might be, creating
> a DVD vs. playing it back.
>
It's of philosophical interest onlybut this kind of question engages me 
quite a bit.
Your example of 'compiler --run-program' certainly demonstrates the 
reducto ad
absurdum of the idea--but one might compile something with a dozen different
compilers and compilers might compile any one of thousands of different 
programs,
and the compilers certainly don't care what the program does, and the 
program
might run under a dozen different OSon any compatible machine--
so their is no 'tight' link. Same with a DVD--one might burn with any 
one of several
programs and play back with any one of several with no link between the 
creator
and the player, and the contents of the product--the DVD--is irrelevant 
to the burner
and to the player.  (So long as the standards are maintained, of course.)

OTOH zcompile produces a filespecifically and only
usable by autoload, so one might rightly think of zcompile as a subsidiary
program to autoload.  I doubt you could change one without considering the
effect on the other. I'd say it's the same with zip/unzip--if one 
changes then
the other must change--they are a suite of linked programs. But whereas
unzip does not 'use' the content of whatever it unzips, autoload uses
a .zwc file and is the only program that does. Thusly, a mnemonic similarity
between names would be very mildly usefulthe same way that 'unzip'
is usefully similar to 'zip'.  In the real world of course, renaming 
zcompile
would do far more harm than good, so it's an academic discussion. Had
I been around 20 years ago, I'd have suggested naming it 'makezwc' or
something like that, or 'autoload --makezwc' or some such. One is given
a tiny but friendly hint that things are related, that's all. But unix 
is not
a friendly placeand seems not to want to be.  It delights in its obscurity.

Re: autoload

[Bart Schaefer]

On Sep 20,  8:37am, Ray Andrews wrote:
}
} OTOH zcompile produces a filespecifically and only
} usable by autoload

No, that's not true:  zcompiled files may be used by either autoload
(strictly speaking, by the deferred-loading procedure that USUALLY
happens long after the autoload command has been run) OR by source,
and the semantics of the two are not the same.

-- 
Barton E. Schaefer

Re: autoload

[Ray Andrews]

On 09/19/2015 02:22 PM, Bart Schaefer wrote:
> As I've pointed out, these are exactly the same as
>
>      zcompile /Functions/Functions.zwc /Functions/*
>      FPATH=/Functions/Functions.zwc autoload +X -w /Functions/Functions.zwc
>
> Although the better practice would be to do
>
>      zcompile /Functions.zwc /Functions/*
>      FPATH=/Functions.zwc autoload +X -w /Functions.zwc

By gum it works if you get it exactly right:

-"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the 
same path. Fussy fussy.
-"autoload +X -w ..." works with Sebastian's, but not with mine (which 
are all functions).
-"autoload -w ..." works with mine but not Sebastian's.
-"source miscfunctions" *before* calling an autoloaded function to have 
it work the first time.
-Append new dirs to FPATH or other things break.

> if not because of the way fpath is searched, then for the same reason that
> you shouldn't do
>
>      tar -c /Functions/Functions.tar /Functions/*
... and most importantly, follow this advice, or if you rerun zcompile 
where there's already a
.zwc, it rather dumbly tries to eat it's own tail and build itself into 
itself and nothing works.

I guess the zcompile of my dreams would be able to process both scripts 
and functions in one
go, and be able to see the functions in 'miscfunctions' regardless of 
the filename, and not
try to digest existing .zwc's, and not worry about directories. And the 
autoload of my dreams
would be able to pass all of it back every time.  Best to leave it to 
zsh internals it's too
fragile for anything else.  Apologies to the list for all this.

Re: autoload

[Bart Schaefer]

On Sep 19,  5:58pm, Ray Andrews wrote:
}
} By gum it works if you get it exactly right:
} 
} -"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the 
} same path. Fussy fussy.

You *really* don't want that :t in there.

} -"autoload +X -w ..." works with Sebastian's, but not with mine (which 
} are all functions).

This is either because (a) you needed "zcompile -k ..." with yours, or
(b) you have multiple functions per file, which we've already discussed
at length with respect to autoload.  [More on this below.]

} -"autoload -w ..." works with mine but not Sebastian's.

I think "autoload" is entirely superfluous in this case, because:

} -"source miscfunctions" *before* calling an autoloaded function to have 
} it work the first time.

Er, no.  In that case it's 99% certain the function isn't autoloaded at
all, rather it's simply been fully defined during the "source" command.
The "source" command is probably reading miscfunctions.zwc, though.

} -Append new dirs to FPATH or other things break.

I have no idea whether that means "the FPATH has to contain some new
stuff" or also "... and the new stuff has to go at the end, it can't
be put anywhere else".  Either could be true or false depending on the
context.

} I guess the zcompile of my dreams would be able to process both
} scripts and functions in one go

This has nothing to do with zcompile output and everything to do with
how the result is read back.  Scripts you want to "source ..." and
functions you want to define (and possibly also call, if defining is
happening as part of deferred loading).

(Maybe we should never have created the "autoload" command, because
that name just muddles up the act of creating a placeholder for
deferred loading with the actual deferred loading event itself.)

[I just noticed that the doc for zcompile refers to "execution of
scripts".  This is misleading, it should say "sourcing of scripts".
Execution (as in spawning a new shell to run a file that has execute
mode bit set) does not involve loading .zwc files; only "source"
and "." do that.]

Also zcompile has no way to guess if the file it's being used on is a
source-able script or a function body.  You'd need to tell it which,
somehow.  So "all in one go" would become some convoluted thing like

zcompileOfMyDreams --script=miscfunctions --function=n-list ...

} and be able to see the functions in 'miscfunctions' regardless of
} the filename

This is for all practical purposes impossible to do automatically.
In order to know what functions are in "miscfunctions", zcompile
would have to first *run* (source) that file, which might have
other side effects besides just defining the functions themselves.

Neverthe less zcompile does have a way to do what you're asking,
assuming you know there are no side-effects:

    ( # subshell so as not to muck up the parent
      unfunction -m \*		# clean out all extra functions
      source miscfunctions	# load the functions we care about
      zcompile -a -c miscfunctions.zwc	# compile all the functions
    )

Note that here you are explicitly doing the "run that file" step.

} and not try to digest existing .zwc's,
} and not worry about directories.

This is a philosophical issue along the same lines as CSH_NULL_GLOB.
If I name a bunch of things and some of them are wrong, should the
shell complain about them or not, and either way, should it skip
them and continue working on the ones that are correct?  Which is
more likely to produce the final result that the user expected?

Since the user is perfectly capable of filtering these out when the
argument list is built, I think the decision to complain and stop
is the correct one.

} And the autoload of my dreams would be able to pass all of it back
} every time.

Maybe if you try replacing "autoload" with its synonym "functions -u"
(create an undefined function) in your mental map, you'd get a better
idea of what you're asking.

What would it mean to create an undefined script?  What event would
cause it to become defined?

Re: autoload

[Ray Andrews]

On 09/19/2015 10:41 PM, Bart Schaefer wrote:
> } -"zcompile ... (N-.:t)" ... or it barfs if there's a directory on the
> } same path. Fussy fussy.
>
> You *really* don't want that :t in there.

Yup.  It turns out that it was the regular versions of the functions 
that were being found.
> This is for all practical purposes impossible to do automatically.

Sure, I can't argue, you know how it works.  My dream of simplicity may 
indeed
be impossible.  But you know what's in my head--something that's the 
equivalent of
'just sourcing' your functions, but doing it from precompiled code, and 
not pre-loading
anything at all, but doing it only when the function is actually 
called.  Nuts, it's so
elegant in theory.  I'm sorta in love with the idea but implementation 
becomes
a can of worms.
> } and not try to digest existing .zwc's,
> } and not worry about directories.
>
> This is a philosophical issue along the same lines as CSH_NULL_GLOB.
> If I name a bunch of things and some of them are wrong, should the
> shell complain about them or not, and either way, should it skip
> them and continue working on the ones that are correct?  Which is
> more likely to produce the final result that the user expected?
Indeed, and the right answer is not obvious.  Hand holding tends to end up
creating more problems that it solves.  I'd suppose that if there are 
built in
exceptions to the plain reading of a command it must be exceptions that
could never be desired ever, like, perhaps, including a .zwc inside it's own
replacement. Or not.  I always hated Microsoft for making 80% of things
easier and the remaining 20% impossible.  If it has to be tough, then it has
to be tough.

You know Bart, with all the time you've spent on this, you could have 
written
that HOWTO ;-)

Seriously, it requires something broader than just reading the manual.
>

Re: autoload

[Bart Schaefer]

On Sep 20,  4:21pm, Ray Andrews wrote:
}
} But you know what's in my head--something that's the equivalent of
} 'just sourcing' your functions, but doing it from precompiled code,
} and not pre-loading anything at all, but doing it only when the
} function is actually called.

Except for the point that "just sourcing" doesn't actually call the
function, everything you wrote is what DOES happen as long as the
functions are defined in separate files.  Because the language is
interpreted, the shell itself can't decompose a file into individual
function entry points the way e.g. a linker can for a shared object.
Requiring the file basename to be the same as the entry point name is
the most obvious way to handle this.

} You know Bart, with all the time you've spent on this, you could have 
} written that HOWTO ;-)

The problem is, I wouldn't have known what to write.

Re: autoload

[Ray Andrews]

On 09/20/2015 09:18 PM, Bart Schaefer wrote:
> Except for the point that "just sourcing" doesn't actually call the
> function, everything you wrote is what DOES happen as long as the
> functions are defined in separate files.  Because the language is
> interpreted, the shell itself can't decompose a file into individual
> function entry points the way e.g. a linker can for a shared object.
Really?  That's hard to fathom.  I source a file and it finds whatever 
functions may
be in there, but they can't be packed into a .zwc file without worrying 
about it's
name?  I'll take that on authority, but it blows my breakers.  I'd have 
thought it
would be as transparent as:

     for aa in /aWorking/Zsh/Source/*; do source $aa; done

... becoming:

zcompile --do_it  /not-the-same-directory-dummy\!/Source.zwc 
/aWorking/Zsh/Source/*

... but if you say that's not even doable I can't argue.
> } You know Bart, with all the time you've spent on this, you could have
> } written that HOWTO ;-)
>
> The problem is, I wouldn't have known what to write.
>
Yeah, the paradox of the expert.  Experts don't know what they know.  
The best
writer is the guy who just learned it so he remembers what he needed to 
learn,
where he went wrong, what was important, what was distraction.  Then the
expert edits it for factuality.  Once four or five learners have added their
lessons to the doc, it becomes really usable for the next guy.  Like the 
best
wikis.  The less of an expert you are, the better judge you are as to the
usefulness of a doc.

Re: autoload

[Bart Schaefer]

On Sep 21, 10:03am, Ray Andrews wrote:
} Subject: Re: autoload
}
} On 09/20/2015 09:18 PM, Bart Schaefer wrote:
} > Because the language is
} > interpreted, the shell itself can't decompose a file into individual
} > function entry points the way e.g. a linker can for a shared object.
} 
} Really? That's hard to fathom. I source a file and it finds whatever
} functions may be in there, but they can't be packed into a .zwc file
} without worrying about it's name?

Did you miss the email where I explained how to do that?

} I'd have thought it would be as transparent as:
} 
} zcompile --do_it  /not-the-same-directory-dummy\!/Source.zwc 
} /aWorking/Zsh/Source/*

Really think about "because the language is interpreted."

Suppose I have

    typeset some_critical_var_used_by_foo_and_bar=really_important_value

    foo() { ... }
    bar() { ... }

Now I want to compile this.  Sure, the shell could dump out foo and bar,
but where does it put that typeset?  Does it evaluate the assignment at
compile time?  (A C compiler would, but that's typically the wrong
semantics for a shell.)

What if really_important_value absolutely must only be interpreted once,
not each time foo and bar are individually loaded?

What if the names "foo" and "bar" are actually stored in other variables?

    function $foo { ... }

What if there's an "if" structure?

    if some_runtime_test
    then
	foo() { ... }
	bar() { ... }
    else
        bar() { ... }
	foo() { ... }
    fi

What if there's a runtime test with that crucial typeset INSIDE the test?

I can make up an infinite number of increasingly convoluted cases to
break any "simple, elegant" automation.  So unless you happen to have
"zmodload zsh/artificial-intellgence" lying around ready for use, I'm
done with this discussion.

Re: autoload

[Ray Andrews]

On 09/21/2015 11:17 AM, Bart Schaefer wrote:
> Did you miss the email where I explained how to do that?
On the first pass through a subject like this there is so much to learn
and so much error and misunderstanding and diversions and distractions
that one is lucky to come out of it alive.  On the second pass things will
gel.
>
> Really think about "because the language is interpreted."
Deep lessons ...
>
> I can make up an infinite number of increasingly convoluted cases to
> break any "simple, elegant" automation.  So unless you happen to have
> "zmodload zsh/artificial-intellgence" lying around ready for use, I'm
> done with this discussion.
>
I'd just expect the precompiled function to behave exactly as a sourced 
function
would behave, all that's different is that the file being parsed is ... 
is this where
I go off the rails? ... what I'd call 'object code' which is to say that 
it's somewhat
pre-digested as far as the interpreter goes.  But when it is actually 
called (as if
it were sourced) it would behave exactly identically to a normal sourced 
function.
It's just parsed quicker.  Very different from C, as you say.

I dunno, if I have a function ' test () { echo $PWD } ' I'm not 
expecting $PWD to
be hard captured at sourcing time, is there even a way to do that? It 
might be
cool if there was a way, but surely that's rather unusual.  So, on the 
contrary,
I'm thinking not of artificial-intelligence but rather of something
dumb as a donkey.  Autoloaded functions at present seem to behave as I'd
expect, so I don't know where the problem arises.  Your 'evaluate once'
scenario is valid of course, but I have no idea how that would be 
handled even
by a regular function.  It seems a separate issue to me.

To be honest, your stamina sticking with this so long has already 
astonished me,
so if you're done, that's overdue, but thanks.

Re: autoload

[Bart Schaefer]

OK, one last.

On Sep 21,  1:09pm, Ray Andrews wrote:
}
} I'd just expect the precompiled function to behave exactly as a
} sourced function would behave, all that's different is that the file
} being parsed is ... is this where I go off the rails?

Yes.

You're not asking for a precompiled function to act like a sourced
function.  A sourced function, to the extent that means anything at all,
does nothing but become defined.  But "sourcing" a function (which you
actually can't do, you can only source a *file*) and "calling" a
function are two completely distinct things.  Rather, you ask for
compiling a whole file full of defined functions, to behave like
creating a bunch of undefined ones.

An undefined function (created with "functions -u" aka autoload) does
both of the source and call operations, but it only knows the one name
(its own).  So the file (to be sourced) and the function (to be called)
both have to be identified by that single name.

Could we have created an entire distinct data structure that stores
both a file name and a function name, and means "oh by the way, when
you try to run foo, you actually have to go read the file bar" ?
Yes, we could.  We didn't.  We re-used the existing data structure
of a function, because one field for the name seemed like enough,
and the path-search mechanism seemed like a good existing model for
finding the file.

Further, the point of autoloading was to read into memory exactly
those functions that are actually used.  If to get foo you have to
source bar and do everything bar contains, you may end up reading
(and storing) a hundred functions in order to call one of them.
Not what we wanted to encourage.  So, one function per file.

zcompile lets you do two things:  compile a single file to be read by
source (*) or pack multiple files into an archive to be processed
by autoload.  But the latter doesn't change the fact that autoload
needs to match one file (one thing packed in the archive) to one
function name.  And the former doesn't magically grant the ability
to explode the single file into multiple archive entries. (**)

(*) But this is constrained in that both the original file and the
compiled file have to exist size-by-side, so that source can choose
the newer of the two.  This constraint isn't strictly necessary.

(**) "zcompile -c" can explode functions that are already in memory
into multiple "files" that then go into the archive, but that's as
close as it gets.

Re: autoload

[Ray Andrews]

On 09/21/2015 08:19 PM, Bart Schaefer wrote:
> You're not asking for a precompiled function to act like a sourced
> function.  A sourced function, to the extent that means anything at all,
> does nothing but become defined.  But "sourcing" a function (which you
> actually can't do, you can only source a *file*) and "calling" a
> function are two completely distinct things.
I understand, I just chose the wrong word.
>    Rather, you ask for
> compiling a whole file full of defined functions, to behave like
> creating a bunch of undefined ones.
>
> An undefined function (created with "functions -u" aka autoload) does
> both of the source and call operations, but it only knows the one name
> (its own).  So the file (to be sourced) and the function (to be called)
> both have to be identified by that single name.
>
> Could we have created an entire distinct data structure that stores
> both a file name and a function name, and means "oh by the way, when
> you try to run foo, you actually have to go read the file bar" ?
> Yes, we could.  We didn't.  We re-used the existing data structure
> of a function, because one field for the name seemed like enough,
> and the path-search mechanism seemed like a good existing model for
> finding the file.
Ok.  I admit that my notion does presume a table linking functions with 
their
.zwc:

raysfunc    /directory/raysfunc.zwc    line1
somefunc  /directory/all/allfunc.zwc  line40
otherfunc  /directory/all/allfunc.zwc  line200
...

I get a glimpse of how you did what you did within existing structures. 
If I understood
that fully I'd grok the limitations of the system.  I guess it's a bit 
outrageous of me,
but I presumed that in the hash table that links functions to where ever 
they are,
a sort of pointer to a line in a .zwc file would not be a huge problem.
>
> Further, the point of autoloading was to read into memory exactly
> those functions that are actually used.  If to get foo you have to
> source bar and do everything bar contains, you may end up reading
> (and storing) a hundred functions in order to call one of them.
> Not what we wanted to encourage.  So, one function per file.

That is so insightful, cuz I had exactly the opposite as the supposed 
goal--to
have functions available but not sourced *just in case* you might need them.
I never understood:

     autoload -U colors && colors

... because I'm thinking to myself: 'Why would you 'autoload' something 
that
you are just going to plain vanilla load anyway? Why not just load it 
and be
done with it?' (It doesn't work with compinit.  It seems because 
compinit will
be looking on FPATH for the rest of itself autoloaded, so it's committed to
that method--but you understand my thought.)
>
> zcompile lets you do two things:  compile a single file to be read by
> source (*) or pack multiple files into an archive to be processed
> by autoload.  But the latter doesn't change the fact that autoload
> needs to match one file (one thing packed in the archive) to one
> function name.

But ... if we have several functions packed in one .zwc it is not 
logically possible
for the name of the file to match the name of more than one of the 
functions which
breaks my head as far as this one-to-one function-file thing.  You tell 
me that
there is this function name - file name requirement but then violate that
commandment by packing in several functions to one .zwc.  I get that if 
a script
is autoloaded it will/must be 'named' after the file it came from. One 
script!
But with functions?  The rule is not followed.

> And the former doesn't magically grant the ability
> to explode the single file into multiple archive entries. (**)
>
> (*) But this is constrained in that both the original file and the
> compiled file have to exist size-by-side, so that source can choose
> the newer of the two.  This constraint isn't strictly necessary.
Well therewego.  Previously you're saying that these .zwc should not be 
in the same
dir, now they should be in the same dir. Nuts, they are created in the 
same dir.  I
tested that by touching the files--as you say, it finds the newest. But 
then you tell
me that best practice is to move the thing :-(  And, speaking of 
philosophy, here
is a case of hand-holding.  The first command found on the path is the 
one used,
and too bad for you if a newer version exists somewhere lower down. It 
doesn't
scan the whole path and load the newest file.  I'm expecting that if I 
autoload
something then it's active and too bad if there's a newer file:

     $ autoload my_function
     $ source /functions/myfunction

... I get the sourced version.

     $ source /functions/myfunction
     $ autoload my_function

... I get the version in the .zwc  found on FPATH.  If I want the newer 
version
then I should source it.

>
> (**) "zcompile -c" can explode functions that are already in memory
> into multiple "files" that then go into the archive, but that's as
> close as it gets.
>
When I revisit this, I'll read your posts backwards.  This one is the 
one that would come first
in the HOWTO.  The first thing to understand is why, when and where we'd 
want to
use autoload.  What is it for?  Why did God make it on the seventh day?  
Then, it's
constraints and why they are what they are, then the implementation with 
examples.

Looking back, I've misunderstood the whole point of the thing.  Like the 
C-head
that I am, I'm thinking: 'What a good idea!  We can boil down our 
functions to
'.obj' files for faster loading, pack them together into one file for 
more speed yet, and
best of all, have them unloaded in memory until they are actually called 
at which time
they are 'autoloaded' automagically.  How elegant, I says to myself, 
it's like a .zip--all
compressed and packed together, and just as you can extract a single 
file from a .zip
so you can source a single function from a .zwc and zsh will even do it 
automatically
for you if you autoload that function cuz zsh will keep a table of where 
that
function is found and source it only when you call for the function.

It's all Sebastian's fault ;-)

Seriously sir, I exhaust your patience.  For now the one thing I know is 
that I don't
want to autoload anything, it's too obscure.

Re: autoload

[Bart Schaefer]

Just so no one else is confused ...

On Sep 22, 10:33am, Ray Andrews wrote:
}
} > zcompile lets you do two things:  compile a single file to be read by
} > source (*) or pack multiple files into an archive to be processed
} > by autoload.
} >
} > (*) But this is constrained in that both the original file and the
} > compiled file have to exist size-by-side, so that source can choose
} > the newer of the two.  This constraint isn't strictly necessary.
} 
} Well therewego.  Previously you're saying that these .zwc should not be
} in the same dir, now they should be in the same dir.

Two different use cases:

(1) Packing up multiple files into a composite ZWC?  Don't put the
    composite alongside the files packed in it.  This is the case we
    started out talking about two dozen messages ago.  Makes sense
    only for autoload.

(2) Packing one file into one correspondingly-named ZWC?  Put the ZWC
    alongside the origin file.  Works for source or autoload, but in
    the autoload case this must be one-function/one-file/same-name.

The autoload search is straightforward:  $fpath order comes first.
If, while searching fpath in order, a pair of files "x" and "x.zwc"
is found, read the newer one.  As a special case of this, if "x" is
actually a directory listed in fpath, assume "x.zwc" contains all the
same files as "x/".

The "source" (and ".") search rule is similar, except using $path and
without the directory special case.

}      $ autoload my_function
}      $ source /functions/myfunction
} 
} ... I get the sourced version.

Yes, because in this case you clobbered the autoload declaration by
loading the full function definition on top of it with "source".

}      $ source /functions/myfunction
}      $ autoload my_function
} 
} ... I get the version in the .zwc found on FPATH.

No, you still get the one from the "source" command, because autoload
won't replace an existing function definition.  The only way to ever
get the version found on fpath is by never doing the "source" at all.

Re: autoload

[Ray Andrews]

On 09/22/2015 09:39 PM, Bart Schaefer wrote:
> Just so no one else is confused ...
>
>
Indeed.  But 3/4 of what we've discussed is not documented so others might
very well be unsure of these things too.  I did really jump to 
conclusions as
to how the thing would work, and I'm guilty of wishful thinking as always.
Still I do wish this thread could be gelled into even the outline of a 
howto
sort of doc. that way it would have served some purpose.  It seems one must
understand autoload from it's roots.

wheels within wheels

[Ray Andrews]

I just learned that it's possible to declare a function within another 
function.  A strange liberty.  Why would one want to, and what are the 
gotchas of doing so?

BTW Bart, any further word on where to find the latest manual?  When we 
were hacking over autoload I'd have liked reading those edits you made 
hot of the press.  Is this  a 'git' sort of thing?

Re: wheels within wheels

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On Tue, Sep 29, 2015 at 4:16 PM, Ray Andrews <rayandrews@eastlink.ca> wrote:

> I just learned that it's possible to declare a function within another
> function.  A strange liberty.  Why would one want to, and what are the
> gotchas of doing so?
>

Many compiled languages support the concept of "closures". Even those, like
Python, which are compiled on the fly into byte-code as they are executed.
On the other hand zsh, and in general any shell, based on bash or csh do
not support closures or namespaces so I have no idea why someone would nest
function definitions in those shell languages. Google "zsh closures" and
read this answer:

http://unix.stackexchange.com/questions/94129/is-there-something-like-closures-for-zsh


In short: don't do it. You can see for yourself that nested, named,
function definitions are in fact globally visible (as opposed to visible
only to the function in which they're defined) by executing the following
script. Anonymous functions defined within another function are a different
matter but outside the scope of this question.

I've seen some pretty amazing bash/csh/ksh/zsh scripts in my life.
Including one that implemented an assembly language compiler. But if you
need that level of complexity you should be using a different language. If
the only tool you know how to use is a hammer then every problem looks like
a nail.

#!/bin/zsh

function foo {
    function bar {
        echo hello from bar
    }

    echo hello from foo
    bar
}

echo calling foo
foo
echo calling bar
bar

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: wheels within wheels

[Ray Andrews]

On 09/29/2015 07:55 PM, Kurtis Rader wrote:
> http://unix.stackexchange.com/questions/94129/is-there-something-like-closures-for-zsh
Never heard of it.  But thanks for the link.  I forget that shells 
aren't 'real' languages.
Nuts, I'm even getting used to not declaring variables.  As Bart once 
said, shells are
a glue for various command line operations.

> In short: don't do it.

I wouldn't, unless there was some sort of wonderful reason to, which is 
what I'm asking.

> You can see for yourself that nested, named,
> function definitions are in fact globally visible (as opposed to visible
> only to the function in which they're defined) by executing the following
> script. Anonymous functions defined within another function are a different
> matter but outside the scope of this question.

Yeah, I ran my own tests and it seems so.  One might think that this 
would be some sort of
localization mechanism, but it seems not to be the case.  Ok, thanks 
Kurtis, I'll not
try any such thing.  It seems very strange that it's even legal.

Re: wheels within wheels

[Bart Schaefer]

On Sep 29,  4:16pm, Ray Andrews wrote:
} Subject: wheels within wheels
}
} I just learned that it's possible to declare a function within another 
} function.  A strange liberty.

Interpreted language.  Defining a function is just like executing any
other command, so you can do it anywhere a command is allowed.  There
really isn't any such thing as "declaring" a function -- either it is
defined [the command that defines it has been *executed*, not merely
written out as part of some dormant code] or it isn't.

The "functions -u" aka "autoload" command is the closest there is to
"declaring" a function, but really it defines a function with an empty
body which is to be filled in later.  This is inaccurately referred to
as an "undefined" function because that's shorter to say.

} Why would one want to, and what are the gotchas of doing so?

Consider it a sort of conditional compilation, if you need a comparison.
Like the _git and tetris examples in the autoload thread -- various
helper functions get defined only if/when the main function that needs
them is actually called.

In addition to what Kurtis says, a gotcha with nesting functions is
that every time the outer function is called, it re-defines the inner
functions (deletes the old definitions and re-create them).  Remember,
interpreted language.  Everything interesting happens at run time.

Therefore do not nest function definitions unless you understand very
well what they are doing.

} BTW Bart, any further word on where to find the latest manual?

You should use the manual for the version of zsh that you have.  If you
start looking at the latest manual but you're not using the latest
shell, you may get just as much misinformation as if you're looking at
an older manual.

} hot of the press.  Is this  a 'git' sort of thing?

Yes.

On Sep 29,  7:55pm, Kurtis Rader wrote:
}
} In short: don't do it.

That's a bit too short.  Nesting function definitions is exactly what a
lot of autoloadable functions do.  The difference is that autoloadable
functions also redefine *themselves* this way, to change what happens
the second time the "same" function (in name only) is called.

} You can see for yourself that nested, named,
} function definitions are in fact globally visible (as opposed to visible
} only to the function in which they're defined) by executing the following
} script.

Here's a modification of that script:

    function foo {
      echo defining bar
      function bar {
	echo hello from bar
      }
      echo redefining foo
      function foo {
	echo hello from new foo
	bar
      }
      # this looks like infinite recursion,
      # but it is not, because foo was redefined
      foo
    }

    # This will say "command not found"
    echo calling bar before foo
    bar
    echo calling foo
    foo
    echo calling bar after foo
    bar
    echo calling foo again
    foo

Re: wheels within wheels

[Mikael Magnusson]

On Wed, Sep 30, 2015 at 5:40 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Sep 29,  7:55pm, Kurtis Rader wrote:
> }
> } In short: don't do it.
>
> That's a bit too short.  Nesting function definitions is exactly what a
> lot of autoloadable functions do.  The difference is that autoloadable
> functions also redefine *themselves* this way, to change what happens
> the second time the "same" function (in name only) is called.

Another example where this is needed is to do this:

function _zle_line_init() {
  zle set-local-history -n 1
  function _zle_line_init() {
    # do whatever it is you actually wanted to do in zle-line-init here
  }
  _zle_line_init
}

Ie, to set local history by default when you start the shell, because
it has to be run from within a widget, but you may want to toggle it
while running the shell so it can't stay in the line-init hook.
Therefore, do it the first time, and then redefine the hook to do
something else.

-- 
Mikael Magnusson

Re: wheels within wheels

[Ray Andrews]

On 09/29/2015 08:40 PM, Bart Schaefer wrote:
> }
> } I just learned that it's possible to declare a function within another
> } function.  A strange liberty.
>
> Interpreted language.

Yes.  It's so easy to forget that.
>      function foo {
>        echo defining bar
>        function bar {
> 	echo hello from bar
>        }
>        echo redefining foo
>        function foo {
> 	echo hello from new foo
> 	bar
>        }
>        # this looks like infinite recursion,
>        # but it is not, because foo was redefined
>        foo
>      }
>
>
Now that is mind expanding.  If you tried that in C you'd collapse the 
universe.
Interpreted .... sure, it can saw off the branch it's sitting on because 
there is no
tree.  Each command is past when it's past so that must mean that the 
address of
first foo ... is there ... yes of course there is, the thing is in 
memory ... just aborts
when second foo comes along.  It will expect grammatical completion of 
first foo
for the sake  of etiquette.  And when foo calls itself, it calls new foo 
which is
not recursive.  I am not yet able to snatch the pebble out of your hand, 
but that's
a zsh koan.

Re: wheels within wheels

[Bart Schaefer]

On Sep 29,  9:15pm, Ray Andrews wrote:
}
} Each command is past when it's past so that must mean that the address
} of first foo ... is there ... yes of course there is, the thing is in
} memory ... just aborts when second foo comes along.

No, the outer foo will happily keep executing until it returns, at
which point it is garbage-collected.  The old body has merely been
disconnected from the name.

    function foo {
      echo defining bar
      function bar {
	echo hello from bar
      }
      echo redefining foo
      function foo {
	echo hello from new foo
	bar
      }
      foo
      echo goodbye from the old foo
    }

Re: wheels within wheels

[Ray Andrews]

On 09/30/2015 12:05 AM, Bart Schaefer wrote:
> On Sep 29,  9:15pm, Ray Andrews wrote:
> }
> } Each command is past when it's past so that must mean that the address
> } of first foo ... is there ... yes of course there is, the thing is in
> } memory ... just aborts when second foo comes along.
>
> No, the outer foo will happily keep executing until it returns, at
> which point it is garbage-collected.  The old body has merely been
> disconnected from the name.
That's better.  It's not 'abandoned' it just becomes nameless.  Else 
we'd have
orphan lines of code and that's not sanitary.

Re: wheels within wheels

[ZyX]

30.09.2015, 07:16, "Ray Andrews" <rayandrews@eastlink.ca>:
> On 09/29/2015 08:40 PM, Bart Schaefer wrote:
>>  }
>>  } I just learned that it's possible to declare a function within another
>>  } function. A strange liberty.
>>
>>  Interpreted language.
>
> Yes. It's so easy to forget that.
>>       function foo {
>>         echo defining bar
>>         function bar {
>>          echo hello from bar
>>         }
>>         echo redefining foo
>>         function foo {
>>          echo hello from new foo
>>          bar
>>         }
>>         # this looks like infinite recursion,
>>         # but it is not, because foo was redefined
>>         foo
>>       }
>
> Now that is mind expanding. If you tried that in C you'd collapse the
> universe.
> Interpreted .... sure, it can saw off the branch it's sitting on because
> there is no
> tree. Each command is past when it's past so that must mean that the
> address of
> first foo ... is there ... yes of course there is, the thing is in
> memory ... just aborts
> when second foo comes along. It will expect grammatical completion of
> first foo
> for the sake of etiquette. And when foo calls itself, it calls new foo
> which is
> not recursive. I am not yet able to snatch the pebble out of your hand,
> but that's
> a zsh koan.

Nested *same* function definition is rather useful for initialization. Or when you need to alter the implementation based on some condition, and want to cache this to not check this condition again:

    # This will parse json array passed as argument to a shell array $reply.
    # Depending on whether zpython module is available it will either use built-in python interpreter support
    # which is faster or call python in a subshell.
    parse_json_array() {
        if zmodload libzpython 2>/dev/null ; then
            parse_json_array() {
                zpython 'import zsh, json; zsh.setvalue("reply", json.loads(zsh.getvalue("1")))'
            }
        else
            parse_json_array() {
                reply=( "${(@0)"$(python -c 'import sys, json; sys.stdout.write("\0".join(json.loads(sys.argv[1])))' "$1")"}" )
            }
        fi
        parse_json_array "$@"
    }

. Also this is a feature sometimes useful for metaprogramming: you can create a function which defines a function through eval. Will be needed if you for some reason want something like bash `export -f` feature.

And this is *not* a zsh koan. I have been using the same technique for the same reasons in Python: e.g. in https://github.com/ZyX-I/powerline/blob/5ebf0875b85f85e7cd4c1e07aa8cdebbcc3bd17e/powerline/segments/common/bat.py#L163. Many interpreted languages allow this. Specifically *all* POSIX-like shells I know allow this: https://github.com/ZyX-I/powerline/blob/5ebf0875b85f85e7cd4c1e07aa8cdebbcc3bd17e/powerline/bindings/shell/powerline.sh#L74-L99 is known to work in dash, [m]ksh, busybox ash and bash.

In C you may do something similar if you use function pointer as a public interface and store different address, but this makes call indirect. If you are OK with binding yourself to GCC you can even have nested definitions.

Re: wheels within wheels

[Ray Andrews]

On 09/30/2015 01:01 AM, ZyX wrote:
> Nested *same* function definition is rather useful for initialization. Or when you need to alter the implementation based on some condition, and want to cache this to not check this condition again:
Sure, I see that this sort of thing has it's power.  But this discussion 
has mostly involved redefining
a function within itself, not defining function A within function B so 
Kurtis' comments are still
cogent.  I'm looking at Sebastian's curses stuff and he does have nested 
definitions so I'm
curious as to the reason.

>    And this is *not* a zsh koan.

I just meant that Bart's example was a powerful and pointed 
demonstration of the right way of seeing
the issue.