* Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug @ 2016-09-26 22:07 Sebastian Gniazdowski 2016-09-27 7:05 ` Daniel Shahaf 2016-09-29 7:10 ` Bart Schaefer 0 siblings, 2 replies; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-26 22:07 UTC (permalink / raw) To: Zsh hackers list Hello, with Zplugin I avoid setting FPATH by manually constructing autoload function stub: eval "function $func { local FPATH="$PLUGIN_DIR":"${FPATH}" builtin autoload -X ${opts[*]} }" This way one can have say 10 plugins loaded and still clean $FPATH. What I astonishingly realized today is that functions autoloaded this way can further use autoload builtin purely normally. FPATH visible from the specially-autoloaded function is modified, has the required added component, and builtin autoload works as expected (even with <5.1 Zsh, it has a more specific manual autoload stub). However once in 3 days I get "function definition file not found" not found message. What could be the reason? I'm using non-optimized zsh-5.2-dev-2 and could do some actions to track this, but what could they be? Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-26 22:07 Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug Sebastian Gniazdowski @ 2016-09-27 7:05 ` Daniel Shahaf 2016-09-27 7:54 ` Sebastian Gniazdowski 2016-09-29 7:10 ` Bart Schaefer 1 sibling, 1 reply; 15+ messages in thread From: Daniel Shahaf @ 2016-09-27 7:05 UTC (permalink / raw) To: Sebastian Gniazdowski; +Cc: Zsh hackers list Sebastian Gniazdowski wrote on Tue, Sep 27, 2016 at 00:07:37 +0200: > eval "function $func { > local FPATH="$PLUGIN_DIR":"${FPATH}" > builtin autoload -X ${opts[*]} > }" The " signs in the first line of the function body need to be escaped. The parameter expansions $foo need to be escaped, some as \$foo and others as ${(q)foo}. (No idea about your question; if I had an idea I'd have shared it.) Cheers, Daniel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-27 7:05 ` Daniel Shahaf @ 2016-09-27 7:54 ` Sebastian Gniazdowski 0 siblings, 0 replies; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-27 7:54 UTC (permalink / raw) To: Daniel Shahaf; +Cc: Zsh hackers list On 27 September 2016 at 09:05, Daniel Shahaf <d.s@daniel.shahaf.name> wrote: > Sebastian Gniazdowski wrote on Tue, Sep 27, 2016 at 00:07:37 +0200: >> eval "function $func { >> local FPATH="$PLUGIN_DIR":"${FPATH}" >> builtin autoload -X ${opts[*]} >> }" > > The " signs in the first line of the function body need to be escaped. > > The parameter expansions $foo need to be escaped, some as \$foo and > others as ${(q)foo}. As we discussed on IRC, the final version is like following – quoting in 4 new places: eval "function ${(q)func} { local FPATH=${(qqq)PLUGIN_DIR}:${(qqq)FPATH} builtin autoload -X ${(q-)opts[@]} }" Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-26 22:07 Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug Sebastian Gniazdowski 2016-09-27 7:05 ` Daniel Shahaf @ 2016-09-29 7:10 ` Bart Schaefer 2016-09-29 9:42 ` Sebastian Gniazdowski 2016-09-29 13:58 ` Sebastian Gniazdowski 1 sibling, 2 replies; 15+ messages in thread From: Bart Schaefer @ 2016-09-29 7:10 UTC (permalink / raw) To: Zsh hackers list On Sep 27, 12:07am, Sebastian Gniazdowski wrote: } } eval "function $func { } local FPATH="$PLUGIN_DIR":"${FPATH}" } builtin autoload -X ${opts[*]} } }" } } What I astonishingly realized today is that functions autoloaded this } way can further use autoload builtin purely normally. FPATH visible } from the specially-autoloaded function is modified, has the required } added component, and builtin autoload works as expected (even with } <5.1 Zsh, it has a more specific manual autoload stub). This will be the case as long as the autoload command AND the first run of the normally-autoloaded function BOTH happen during the FIRST run of the "specially-loaded" function. E.g. suppose we have this silly example: --8<-- file "outer" --8<-- if (( $+functions[inner2] )); then inner2 else autoload inner1 inner2 inner1 fi -->8-- file "outer" -->8-- and we send func=outer through your eval above. When "outer" is first run, the call stack will look like outer # scope of local FPATH is here autoload -X outer # "normal autoloads" are here inner1 # modified FPATH is still in scope In this case inner1 will load with the modified FPATH. On the second and subsequent runs of "outer", the call stack will look like: outer # no FPATH declared inner2 # no modified FPATH in scope thus inner2 will load from the normal fpath. It's the fpath at time of function call that matters, not the fpath at time of autoload command. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 7:10 ` Bart Schaefer @ 2016-09-29 9:42 ` Sebastian Gniazdowski 2016-09-29 10:05 ` Sebastian Gniazdowski 2016-09-29 13:58 ` Sebastian Gniazdowski 1 sibling, 1 reply; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-29 9:42 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list I will go deep into what you wrote, I'm just overloaded with work currently (while having no day time job hehe) and cannot much divide focus, however I now encountered the FPATH problem (first time from the initial email), having a debug print in place. Function uizcm is loaded via: eval "function ${(q)func} { local FPATH=${(qqq)PLUGIN_DIR}:${(qqq)FPATH} builtin autoload -X ${(q-)opts[@]} }" It has: echo "FPATH: $FPATH" >> /tmp/reply-fp The FPATH printed always contained required component for further-autoloading to work, however now – the case caught – it just didn't. And, I think this is maybe interesting – doing: FPATH="the required fpath" in shell, then again invoking Zle widget uizcm via Ctrl-O-Ctrl-U, yielded working setup. So it's like if once uizcm took FPATH from the local "FPATH" line in the emulated autoload stub, and once took it from global scope. Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 9:42 ` Sebastian Gniazdowski @ 2016-09-29 10:05 ` Sebastian Gniazdowski 0 siblings, 0 replies; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-29 10:05 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list PS. Just one thing to add: further-autoloading always work for the first time, and currently never for the second time, with the "function definition file not found" effect as described. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 7:10 ` Bart Schaefer 2016-09-29 9:42 ` Sebastian Gniazdowski @ 2016-09-29 13:58 ` Sebastian Gniazdowski 2016-09-29 21:50 ` Bart Schaefer 1 sibling, 1 reply; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-29 13:58 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list On 29 September 2016 at 09:10, Bart Schaefer <schaefer@brasslantern.com> wrote: > On Sep 27, 12:07am, Sebastian Gniazdowski wrote: > } > } eval "function $func { > } local FPATH="$PLUGIN_DIR":"${FPATH}" > } builtin autoload -X ${opts[*]} > } }" > } > } What I astonishingly realized today is that functions autoloaded this > } way can further use autoload builtin purely normally. FPATH visible > } from the specially-autoloaded function is modified, has the required > } added component, and builtin autoload works as expected (even with > } <5.1 Zsh, it has a more specific manual autoload stub). > > This will be the case as long as the autoload command AND the first > run of the normally-autoloaded function BOTH happen during the FIRST > run of the "specially-loaded" function. Now I can easily parse what you wrote (the call stacks were heavy) after I've added fprintf() to builtin.c / typeset_single. Indeed, it is called before first run of special-autoloaded functions. On second run there's no "local FPATH" being run, and further-autoloaded functions are not found. I simplified my test case and just run uizcm twice. During second run new autoload functions aren't resolved. Turns out a normal fpath-extending loading support is needed for Zplugin, unless a creative solution is found Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 13:58 ` Sebastian Gniazdowski @ 2016-09-29 21:50 ` Bart Schaefer 2016-09-30 6:03 ` Sebastian Gniazdowski 2016-09-30 6:49 ` Daniel Shahaf 0 siblings, 2 replies; 15+ messages in thread From: Bart Schaefer @ 2016-09-29 21:50 UTC (permalink / raw) To: Zsh hackers list On Sep 29, 3:58pm, Sebastian Gniazdowski wrote: } } Turns out a normal fpath-extending loading support is needed for } Zplugin, unless a creative solution is found Your autoload wrapper that sets up the "local FPATH" needs to store the FPATH value for each module somewhere, e.g., in a has indexed on the module name, and then recall that every time another autoload occurs from inside a function that was loaded from that module. Whether it's possible to find that the name of the module at the necessary time, is a harder question. In 5.3 "whence -v" will reliably give you the path from which the function was loaded, but it may not in earlier versions. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 21:50 ` Bart Schaefer @ 2016-09-30 6:03 ` Sebastian Gniazdowski 2016-09-30 15:31 ` Bart Schaefer 2016-09-30 6:49 ` Daniel Shahaf 1 sibling, 1 reply; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-30 6:03 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list On 29 September 2016 at 23:50, Bart Schaefer <schaefer@brasslantern.com> wrote: > On Sep 29, 3:58pm, Sebastian Gniazdowski wrote: > } > } Turns out a normal fpath-extending loading support is needed for > } Zplugin, unless a creative solution is found > > Your autoload wrapper that sets up the "local FPATH" needs to store > the FPATH value for each module somewhere, e.g., in a has indexed > on the module name, and then recall that every time another autoload > occurs from inside a function that was loaded from that module. This would lead to aliases? The autoloaded function would have to have "autoload" calls translated into "zplg_autoload" calls. -U flag causes problems here. Loading function with aliases disabled.. I could use $dis_aliases to disable every alias except for "autoload", and this way in general provide what -U should, however it looks like restoring $dis_aliases will be difficult after autoload -X, because following content of autoload stub will not be run I could also make autoload wrapper continuously active in Zsh session but that doesn't look nice to me. Having fpath-extending loading support in place doesn't mean that clean-FPATH goal will not be reached. Few plugins use autoload -> autoload, so user would have to load via fpath say 2 times, and 10 times via fpath-clean way. Fair enough. And even this might not happen, because author of plugin only has to do all autoloads in plugin.zsh file, not in autoloaded file, to solve the problem. > Whether it's possible to find that the name of the module at the > necessary time, is a harder question. In 5.3 "whence -v" will > reliably give you the path from which the function was loaded, but > it may not in earlier versions. Supporting starting from 5.3 is fine to me in this case. Good to hear about whence -v, this should open way to interesting tricks. Side, I hope there is still some time till 5.3 because I want to make Zconvey more robust, currently if I press Ctrl-R Ctrl-C continuously, rapidly then often happens that every 2 seconds I get beep and new line – zconvey's 2-second scheduled function is being apparently hit. Plan to get on this with debugger today and to code as you had an idea of how to make this less probable, will check that. Zconvey can be a cool plugin, I currently e.g. switch all my sessions to current project that I work on via "zcm -zz" (zcommodore -z -z, it uses zconvey). Plus the take-session I mentioned and other possible ideas, I think having sessions coupled together can get important say among users. Idea "if you want functionality use latest" is fine to me if it's about sophisticated functionality. So I exercised idea of extending 5.3 to make further-autoloading work. getfpfunc does: pp = fpath; for (; *pp; pp++) { and it's about side-providing additional path to that loop. Had multiple flawed ideas yesterday but now I'm rather blank. So maybe actually no idea. Recalling something about if functions do have some sort of meta-data. Then at some point additional fpath could be extracted and provided to getfpfunc. Example problem: the state save-restoration, one more data not serializable via declare -f. Other flawed idea is autoload option to hold single path. It could store the path somewhere so that getfpfunc could use it. But sorry I'm currently not having any actual ideas. Btw. I've stumbled upon following code in execcmd / exec.c:3054: } else if (is_shfunc || type == WC_AUTOFN) { Shfunc shf; if (is_shfunc) shf = (Shfunc)hn; else { shf = loadautofn(state->prog->shf, 1, 0); Is this the place where autoload-marked functions are differentiated from manual-autoload-stub functions? Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-30 6:03 ` Sebastian Gniazdowski @ 2016-09-30 15:31 ` Bart Schaefer 2016-09-30 17:41 ` Sebastian Gniazdowski 0 siblings, 1 reply; 15+ messages in thread From: Bart Schaefer @ 2016-09-30 15:31 UTC (permalink / raw) To: Zsh hackers list On Sep 30, 8:03am, Sebastian Gniazdowski wrote: } Subject: Re: Surprising effect of fun() { local FPATH=...; autoload -X }, } } > Your autoload wrapper that sets up the "local FPATH" needs to store } > the FPATH value for each module somewhere } } This would lead to aliases? The autoloaded function would have to have } "autoload" calls translated into "zplg_autoload" calls. [...] } I could also make autoload wrapper continuously active in Zsh session } but that doesn't look nice to me. The wrapper would have to change so that it does the full monty only when called in a known module context. (Which returns to the question of whether that can be discovered at the right time.) } Having fpath-extending loading support in place doesn't mean that } clean-FPATH goal will not be reached. It's your software, make the design decision that you think best! } Btw. I've stumbled upon following code in execcmd / exec.c:3054: } } } else if (is_shfunc || type == WC_AUTOFN) { } Shfunc shf; } if (is_shfunc) } shf = (Shfunc)hn; } else { } shf = loadautofn(state->prog->shf, 1, 0); } } Is this the place where autoload-marked functions are differentiated } from manual-autoload-stub functions? It's the place where already-defined functions (stub or otherwise) are differentiated from autoloads that still need to be pulled in. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-30 15:31 ` Bart Schaefer @ 2016-09-30 17:41 ` Sebastian Gniazdowski 2016-09-30 17:53 ` Sebastian Gniazdowski 2016-09-30 20:13 ` Bart Schaefer 0 siblings, 2 replies; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-30 17:41 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list On 30 September 2016 at 17:31, Bart Schaefer <schaefer@brasslantern.com> wrote: > It's your software, make the design decision that you think best! Well yes but I've cooperated with this ML and also with IRC channel and gained enormously. n-list supports 150k elements not 80k thanks to your heap-optimization, Zplugin was created because of your autoload stub idea, multiple *-list libraries and history searcher created thanks to Mikael's *foo*~^*bar* pattern, Zconvey will use built-in flock and not binary flock on 5.3 thanks to recent timeout-0 patch, my multiple plugins will use Zcurses without problems and with 256 colors thanks to my 3 patches, Zsnapshot will apparently save functions with meta-data thanks to your zcompile-saving tip (didn't dive into this yet), the possible fix to say ZTM_NONE in .recursiveedit that I hope for will make Zconvey fully robust. Wonder if that's all. I think this is a really cool way of development, to write software cooperating with Zsh developers, it produced much good. Best regards, Sebastian Gniazdowski ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-30 17:41 ` Sebastian Gniazdowski @ 2016-09-30 17:53 ` Sebastian Gniazdowski 2016-09-30 20:13 ` Bart Schaefer 1 sibling, 0 replies; 15+ messages in thread From: Sebastian Gniazdowski @ 2016-09-30 17:53 UTC (permalink / raw) To: Bart Schaefer; +Cc: Zsh hackers list Ah and the ${(%):-%N} tip that allowed my plugins and z-sy-h to handle nofunctionargzero. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-30 17:41 ` Sebastian Gniazdowski 2016-09-30 17:53 ` Sebastian Gniazdowski @ 2016-09-30 20:13 ` Bart Schaefer 1 sibling, 0 replies; 15+ messages in thread From: Bart Schaefer @ 2016-09-30 20:13 UTC (permalink / raw) To: Zsh hackers list On Sep 30, 7:41pm, Sebastian Gniazdowski wrote: } Subject: Re: Surprising effect of fun() { local FPATH=...; autoload -X }, } } On 30 September 2016 at 17:31, Bart Schaefer <schaefer@brasslantern.com> wrote: } > It's your software, make the design decision that you think best! } } Well yes but I've cooperated with this ML and also with IRC channel Of course I'm not suggesting that you change that practice, but in this specific case there are two possible approaches and neither obviously better. I've never really understood why an "uncluttered FPATH" is an important design goal, but you chose *that* even so. ;-) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-29 21:50 ` Bart Schaefer 2016-09-30 6:03 ` Sebastian Gniazdowski @ 2016-09-30 6:49 ` Daniel Shahaf 2016-09-30 18:16 ` Bart Schaefer 1 sibling, 1 reply; 15+ messages in thread From: Daniel Shahaf @ 2016-09-30 6:49 UTC (permalink / raw) To: zsh-workers Bart Schaefer wrote on Thu, Sep 29, 2016 at 14:50:22 -0700: > On Sep 29, 3:58pm, Sebastian Gniazdowski wrote: > } > } Turns out a normal fpath-extending loading support is needed for > } Zplugin, unless a creative solution is found > > Your autoload wrapper that sets up the "local FPATH" needs to store > the FPATH value for each module somewhere, e.g., in a has indexed > on the module name, and then recall that every time another autoload > occurs from inside a function that was loaded from that module. Would it work to declare FPATH as a private in the wrapper? (Using zsh/param/private) Sebastian said on IRC it wouldn't work but I don't see why. > Whether it's possible to find that the name of the module at the > necessary time, is a harder question. In 5.3 "whence -v" will > reliably give you the path from which the function was loaded, but > it may not in earlier versions. > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug 2016-09-30 6:49 ` Daniel Shahaf @ 2016-09-30 18:16 ` Bart Schaefer 0 siblings, 0 replies; 15+ messages in thread From: Bart Schaefer @ 2016-09-30 18:16 UTC (permalink / raw) To: zsh-workers On Sep 30, 6:49am, Daniel Shahaf wrote: } } Would it work to declare FPATH as a private in the wrapper? (Using } zsh/param/private) } } Sebastian said on IRC it wouldn't work but I don't see why. It would work, but it doesn't help -- there's a three-stage process: 1. autoload wrapper creates stub function containing local FPATH 2. stub function runs using that FPATH to load the real function 3. real function runs and calls real autoload instead of wrapper The problem Sebastian wants to address is, how to get *this* real function to call the wrapper autoload *without* forcing all usages of autoload to call the wrapper. ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2016-10-02 15:39 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-09-26 22:07 Surprising effect of fun() { local FPATH=...; autoload -X }, and a bug Sebastian Gniazdowski 2016-09-27 7:05 ` Daniel Shahaf 2016-09-27 7:54 ` Sebastian Gniazdowski 2016-09-29 7:10 ` Bart Schaefer 2016-09-29 9:42 ` Sebastian Gniazdowski 2016-09-29 10:05 ` Sebastian Gniazdowski 2016-09-29 13:58 ` Sebastian Gniazdowski 2016-09-29 21:50 ` Bart Schaefer 2016-09-30 6:03 ` Sebastian Gniazdowski 2016-09-30 15:31 ` Bart Schaefer 2016-09-30 17:41 ` Sebastian Gniazdowski 2016-09-30 17:53 ` Sebastian Gniazdowski 2016-09-30 20:13 ` Bart Schaefer 2016-09-30 6:49 ` Daniel Shahaf 2016-09-30 18:16 ` Bart Schaefer
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).