Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

Hello,
with Zplugin I avoid setting FPATH by manually constructing autoload
function stub:

            eval "function $func {
                local FPATH="$PLUGIN_DIR":"${FPATH}"
                builtin autoload -X ${opts[*]}
            }"

This way one can have say 10 plugins loaded and still clean $FPATH.
What I astonishingly realized today is that functions autoloaded this
way can further use autoload builtin purely normally. FPATH visible
from the specially-autoloaded function is modified, has the required
added component, and builtin autoload works as expected (even with
<5.1 Zsh, it has a more specific manual autoload stub).

However once in 3 days I get "function definition file not found" not
found message. What could be the reason? I'm using non-optimized
zsh-5.2-dev-2 and could do some actions to track this, but what could
they be?

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Tue, Sep 27, 2016 at 00:07:37 +0200:
>             eval "function $func {
>                 local FPATH="$PLUGIN_DIR":"${FPATH}"
>                 builtin autoload -X ${opts[*]}
>             }"

The " signs in the first line of the function body need to be escaped.

The parameter expansions $foo need to be escaped, some as \$foo and
others as ${(q)foo}.

(No idea about your question; if I had an idea I'd have shared it.)

Cheers,

Daniel

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

On 27 September 2016 at 09:05, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Sebastian Gniazdowski wrote on Tue, Sep 27, 2016 at 00:07:37 +0200:
>>             eval "function $func {
>>                 local FPATH="$PLUGIN_DIR":"${FPATH}"
>>                 builtin autoload -X ${opts[*]}
>>             }"
>
> The " signs in the first line of the function body need to be escaped.
>
> The parameter expansions $foo need to be escaped, some as \$foo and
> others as ${(q)foo}.

As we discussed on IRC, the final version is like following – quoting
in 4 new places:
                eval "function ${(q)func} {
                    local FPATH=${(qqq)PLUGIN_DIR}:${(qqq)FPATH}
                    builtin autoload -X ${(q-)opts[@]}
                }"

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Bart Schaefer]

On Sep 27, 12:07am, Sebastian Gniazdowski wrote:
}
}             eval "function $func {
}                 local FPATH="$PLUGIN_DIR":"${FPATH}"
}                 builtin autoload -X ${opts[*]}
}             }"
} 
} What I astonishingly realized today is that functions autoloaded this
} way can further use autoload builtin purely normally. FPATH visible
} from the specially-autoloaded function is modified, has the required
} added component, and builtin autoload works as expected (even with
} <5.1 Zsh, it has a more specific manual autoload stub).

This will be the case as long as the autoload command AND the first
run of the normally-autoloaded function BOTH happen during the FIRST
run of the "specially-loaded" function.

E.g. suppose we have this silly example:

  --8<-- file "outer" --8<--
    if (( $+functions[inner2] )); then
      inner2
    else
      autoload inner1 inner2
      inner1
    fi
  -->8-- file "outer" -->8--

and we send func=outer through your eval above.  When "outer" is first
run, the call stack will look like

    outer # scope of local FPATH is here
     autoload -X
      outer # "normal autoloads" are here
       inner1 # modified FPATH is still in scope

In this case inner1 will load with the modified FPATH.  On the second
and subsequent runs of "outer", the call stack will look like:

    outer # no FPATH declared
     inner2 # no modified FPATH in scope

thus inner2 will load from the normal fpath.  It's the fpath at time of
function call that matters, not the fpath at time of autoload command.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

I will go deep into what you wrote, I'm just overloaded with work
currently (while having no day time job hehe) and cannot much divide
focus, however I now encountered the FPATH problem (first time from
the initial email), having a debug print in place. Function uizcm is
loaded via:

                eval "function ${(q)func} {
                    local FPATH=${(qqq)PLUGIN_DIR}:${(qqq)FPATH}
                    builtin autoload -X ${(q-)opts[@]}
                }"

It has:

echo "FPATH: $FPATH" >> /tmp/reply-fp

The FPATH printed always contained required component for
further-autoloading to work, however now – the case caught – it just
didn't. And, I think this is maybe interesting – doing:

FPATH="the required fpath"

in shell, then again invoking Zle widget uizcm via Ctrl-O-Ctrl-U,
yielded working setup. So it's like if once uizcm took FPATH from the
local "FPATH" line in the emulated autoload stub, and once took it
from global scope.

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

PS. Just one thing to add: further-autoloading always work for the
first time, and currently never for the second time, with the
"function definition file not found" effect as described.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

On 29 September 2016 at 09:10, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sep 27, 12:07am, Sebastian Gniazdowski wrote:
> }
> }             eval "function $func {
> }                 local FPATH="$PLUGIN_DIR":"${FPATH}"
> }                 builtin autoload -X ${opts[*]}
> }             }"
> }
> } What I astonishingly realized today is that functions autoloaded this
> } way can further use autoload builtin purely normally. FPATH visible
> } from the specially-autoloaded function is modified, has the required
> } added component, and builtin autoload works as expected (even with
> } <5.1 Zsh, it has a more specific manual autoload stub).
>
> This will be the case as long as the autoload command AND the first
> run of the normally-autoloaded function BOTH happen during the FIRST
> run of the "specially-loaded" function.

Now I can easily parse what you wrote (the call stacks were heavy)
after I've added fprintf() to builtin.c / typeset_single. Indeed, it
is called before first run of special-autoloaded functions. On second
run there's no "local FPATH" being run, and further-autoloaded
functions are not found. I simplified my test case and just run uizcm
twice. During second run new autoload functions aren't resolved. Turns
out a normal fpath-extending loading support is needed for Zplugin,
unless a creative solution is found

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Bart Schaefer]

On Sep 29,  3:58pm, Sebastian Gniazdowski wrote:
}
} Turns out a normal fpath-extending loading support is needed for
} Zplugin, unless a creative solution is found

Your autoload wrapper that sets up the "local FPATH" needs to store
the FPATH value for each module somewhere, e.g., in a has indexed
on the module name, and then recall that every time another autoload
occurs from inside a function that was loaded from that module.

Whether it's possible to find that the name of the module at the
necessary time, is a harder question.  In 5.3 "whence -v" will
reliably give you the path from which the function was loaded, but
it may not in earlier versions.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

On 29 September 2016 at 23:50, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sep 29,  3:58pm, Sebastian Gniazdowski wrote:
> }
> } Turns out a normal fpath-extending loading support is needed for
> } Zplugin, unless a creative solution is found
>
> Your autoload wrapper that sets up the "local FPATH" needs to store
> the FPATH value for each module somewhere, e.g., in a has indexed
> on the module name, and then recall that every time another autoload
> occurs from inside a function that was loaded from that module.

This would lead to aliases? The autoloaded function would have to have
"autoload" calls translated into "zplg_autoload" calls. -U flag causes
problems here. Loading function with aliases disabled.. I could use
$dis_aliases to disable every alias except for "autoload", and this
way in general provide what -U should, however it looks like restoring
$dis_aliases will be difficult after autoload -X, because following
content of autoload stub will not be run

I could also make autoload wrapper continuously active in Zsh session
but that doesn't look nice to me. Having fpath-extending loading
support in place doesn't mean that clean-FPATH goal will not be
reached. Few plugins use autoload -> autoload, so user would have to
load via fpath say 2 times, and 10 times via fpath-clean way. Fair
enough. And even this might not happen, because author of plugin only
has to do all autoloads in plugin.zsh file, not in autoloaded file, to
solve the problem.

> Whether it's possible to find that the name of the module at the
> necessary time, is a harder question.  In 5.3 "whence -v" will
> reliably give you the path from which the function was loaded, but
> it may not in earlier versions.

Supporting starting from 5.3 is fine to me in this case. Good to hear
about whence -v, this should open way to interesting tricks. Side, I
hope there is still some time till 5.3 because I want to make Zconvey
more robust, currently if I press Ctrl-R Ctrl-C continuously, rapidly
then often happens that every 2 seconds I get beep and new line –
zconvey's 2-second scheduled function is being apparently hit. Plan to
get on this with debugger today and to code as you had an idea of how
to make this less probable, will check that. Zconvey can be a cool
plugin, I currently e.g. switch all my sessions to current project
that I work on via "zcm -zz" (zcommodore -z -z, it uses zconvey). Plus
the take-session I mentioned and other possible ideas, I think having
sessions coupled together can get important say among users.

Idea "if you want functionality use latest" is fine to me if it's
about sophisticated functionality. So I exercised idea of extending
5.3 to make further-autoloading work. getfpfunc does:

    pp = fpath;
    for (; *pp; pp++) {

and it's about side-providing additional path to that loop. Had
multiple flawed ideas yesterday but now I'm rather blank. So maybe
actually no idea. Recalling something about if functions do have some
sort of meta-data. Then at some point additional fpath could be
extracted and provided to getfpfunc. Example problem: the state
save-restoration, one more data not serializable via declare -f. Other
flawed idea is autoload option to hold single path. It could store the
path somewhere so that getfpfunc could use it. But sorry I'm currently
not having any actual ideas.

Btw. I've stumbled upon following code in execcmd / exec.c:3054:

    } else if (is_shfunc || type == WC_AUTOFN) {
        Shfunc shf;
        if (is_shfunc)
            shf = (Shfunc)hn;
        else {
            shf = loadautofn(state->prog->shf, 1, 0);

Is this the place where autoload-marked functions are differentiated
from manual-autoload-stub functions?

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Daniel Shahaf]

Bart Schaefer wrote on Thu, Sep 29, 2016 at 14:50:22 -0700:
> On Sep 29,  3:58pm, Sebastian Gniazdowski wrote:
> }
> } Turns out a normal fpath-extending loading support is needed for
> } Zplugin, unless a creative solution is found
> 
> Your autoload wrapper that sets up the "local FPATH" needs to store
> the FPATH value for each module somewhere, e.g., in a has indexed
> on the module name, and then recall that every time another autoload
> occurs from inside a function that was loaded from that module.

Would it work to declare FPATH as a private in the wrapper?  (Using
zsh/param/private)

Sebastian said on IRC it wouldn't work but I don't see why.

> Whether it's possible to find that the name of the module at the
> necessary time, is a harder question.  In 5.3 "whence -v" will
> reliably give you the path from which the function was loaded, but
> it may not in earlier versions.
> 

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Bart Schaefer]

On Sep 30,  8:03am, Sebastian Gniazdowski wrote:
} Subject: Re: Surprising effect of fun() { local FPATH=...; autoload -X }, 
}
} > Your autoload wrapper that sets up the "local FPATH" needs to store
} > the FPATH value for each module somewhere
} 
} This would lead to aliases? The autoloaded function would have to have
} "autoload" calls translated into "zplg_autoload" calls.
[...]
} I could also make autoload wrapper continuously active in Zsh session
} but that doesn't look nice to me.

The wrapper would have to change so that it does the full monty only
when called in a known module context.  (Which returns to the question
of whether that can be discovered at the right time.)

} Having fpath-extending loading support in place doesn't mean that
} clean-FPATH goal will not be reached.

It's your software, make the design decision that you think best!

} Btw. I've stumbled upon following code in execcmd / exec.c:3054:
} 
}     } else if (is_shfunc || type == WC_AUTOFN) {
}         Shfunc shf;
}         if (is_shfunc)
}             shf = (Shfunc)hn;
}         else {
}             shf = loadautofn(state->prog->shf, 1, 0);
} 
} Is this the place where autoload-marked functions are differentiated
} from manual-autoload-stub functions?

It's the place where already-defined functions (stub or otherwise) are
differentiated from autoloads that still need to be pulled in.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

On 30 September 2016 at 17:31, Bart Schaefer <schaefer@brasslantern.com> wrote:
> It's your software, make the design decision that you think best!


Well yes but I've cooperated with this ML and also with IRC channel
and gained enormously. n-list supports 150k elements not 80k thanks to
your heap-optimization, Zplugin was created because of your autoload
stub idea, multiple *-list libraries and history searcher created
thanks to Mikael's *foo*~^*bar* pattern, Zconvey will use built-in
flock and not binary flock on 5.3 thanks to recent timeout-0 patch, my
multiple plugins will use Zcurses without problems and with 256 colors
thanks to my 3 patches, Zsnapshot will apparently save functions with
meta-data thanks to your zcompile-saving tip (didn't dive into this
yet), the possible fix to say ZTM_NONE in .recursiveedit that I hope
for will make Zconvey fully robust. Wonder if that's all. I think this
is a really cool way of development, to write software cooperating
with Zsh developers, it produced much good.

Best regards,
Sebastian Gniazdowski

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Sebastian Gniazdowski]

Ah and the ${(%):-%N} tip that allowed my plugins and z-sy-h to handle
nofunctionargzero.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Bart Schaefer]

On Sep 30,  6:49am, Daniel Shahaf wrote:
}
} Would it work to declare FPATH as a private in the wrapper?  (Using
} zsh/param/private)
} 
} Sebastian said on IRC it wouldn't work but I don't see why.

It would work, but it doesn't help -- there's a three-stage process:

1. autoload wrapper creates stub function containing local FPATH
2. stub function runs using that FPATH to load the real function
3. real function runs and calls real autoload instead of wrapper

The problem Sebastian wants to address is, how to get *this* real
function to call the wrapper autoload *without* forcing all usages
of autoload to call the wrapper.

Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug

[Bart Schaefer]

On Sep 30,  7:41pm, Sebastian Gniazdowski wrote:
} Subject: Re: Surprising effect of fun() { local FPATH=...; autoload -X }, 
}
} On 30 September 2016 at 17:31, Bart Schaefer <schaefer@brasslantern.com> wrote:
} > It's your software, make the design decision that you think best!
} 
} Well yes but I've cooperated with this ML and also with IRC channel

Of course I'm not suggesting that you change that practice, but in this
specific case there are two possible approaches and neither obviously
better.  I've never really understood why an "uncluttered FPATH" is an
important design goal, but you chose *that* even so. ;-)