zsh-workers
 help / color / mirror / code / Atom feed
* A serious bug in execution – where to debug?
@ 2019-07-30 17:00 Sebastian Gniazdowski
  2019-07-30 17:05 ` Sebastian Gniazdowski
                   ` (2 more replies)
  0 siblings, 3 replies; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 17:00 UTC (permalink / raw)
  To: Zsh hackers list

Hello,
I have a zle function that shadows the zle builtin. The bug is: the
function exits after a certain function get's called, if the arguments
are: -F {descriptor-#} _gitstatus_process_response_POWERLEVEL9K (I
didn't yet eliminate the second argument's so I'm pasting it as is).

The symptoms are:
- print before the call does output, after the call – doesn't (as
arranged as in here: http://psprint.blinkenshell.org/zle-exit-fun.png)
- the plugin's following zle -F {descriptor-#} call reports error:
--zplg-shadow-zle:zle:64: No handler installed for fd 21
meaning that the execution didn't really reach the point where an
actual builtin zle is being called by the shadowing function.

Now, what's interesting is the certain function that causes the exit
(https://github.com/zdharma/zplugin/blob/1ddc4a2c69adec7895b4acf53e3db837720663ac/zplugin.zsh#L1261-L1264).
If I remove the conditional from its third line, i.e. change the line
to:

ZPLG_REPORTS[_dtrace/_dtrace]+="$2"$'\n'

then the problem disappears. Not that this practically eliminates that
the bug report is something wrong with some other area of the code.

I find this impossible to narrow down, because e.g. doing the zle -F
call from some other, short plugin doesn't cause the problem. This
must be a lucky set of circumstances.

But I can debug it. Where to add debug printfs / breakpoints, what to
check in them?

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 17:00 A serious bug in execution – where to debug? Sebastian Gniazdowski
@ 2019-07-30 17:05 ` Sebastian Gniazdowski
  2019-07-30 17:41 ` Roman Perepelitsa
  2019-07-30 18:27 ` Bart Schaefer
  2 siblings, 0 replies; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 17:05 UTC (permalink / raw)
  To: Zsh hackers list

On Tue, 30 Jul 2019 at 19:00, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> Not that this practically eliminates that
> the bug report is something wrong with some other area of the code.

The sentence should be: NotE ...
-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 17:00 A serious bug in execution – where to debug? Sebastian Gniazdowski
  2019-07-30 17:05 ` Sebastian Gniazdowski
@ 2019-07-30 17:41 ` Roman Perepelitsa
  2019-07-30 17:55   ` Sebastian Gniazdowski
  2019-07-30 18:27 ` Bart Schaefer
  2 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 17:41 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 7:01 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> Hello,
> I have a zle function that shadows the zle builtin. The bug is: the
> function exits after a certain function get's called, if the arguments
> are: -F {descriptor-#} _gitstatus_process_response_POWERLEVEL9K (I
> didn't yet eliminate the second argument's so I'm pasting it as is).
>
> The symptoms are:
> - print before the call does output, after the call – doesn't (as
> arranged as in here: http://psprint.blinkenshell.org/zle-exit-fun.png)
> - the plugin's following zle -F {descriptor-#} call reports error:
> --zplg-shadow-zle:zle:64: No handler installed for fd 21
> meaning that the execution didn't really reach the point where an
> actual builtin zle is being called by the shadowing function.

Perhaps because of this?

https://github.com/romkatv/powerlevel10k/blob/f14497918f0a70955f6d227d1e002ad2a3f94cc8/gitstatus/gitstatus.plugin.zsh#L302-L307

Feel free to send me a PR replacing `zle` calls in this file with
`builtin zle`. I suppose you aren't shadowing `builtin` for good
measure?

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 17:41 ` Roman Perepelitsa
@ 2019-07-30 17:55   ` Sebastian Gniazdowski
  2019-07-30 18:12     ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 17:55 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 19:42, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
> > - print before the call does output, after the call – doesn't (as
> > arranged as in here: http://psprint.blinkenshell.org/zle-exit-fun.png)
> > - the plugin's following zle -F {descriptor-#} call reports error:
> > --zplg-shadow-zle:zle:64: No handler installed for fd 21
> > meaning that the execution didn't really reach the point where an
> > actual builtin zle is being called by the shadowing function.
>
> Perhaps because of this?
>
> https://github.com/romkatv/powerlevel10k/blob/f14497918f0a70955f6d227d1e002ad2a3f94cc8/gitstatus/gitstatus.plugin.zsh#L302-L307

Nice argument, however this doesn't explain:
- why the actual zle -F isn't called in the later of the prematurely
exited function,
- why changing the meaningless line in the return-causing function
cancels the effect.

> Feel free to send me a PR replacing `zle` calls in this file with
> `builtin zle`. I suppose you aren't shadowing `builtin` for good
> measure?

Yes, implementation of plugin effects withdrawal (i.e. unloading)
bases on this. I saw that your initialization of the prompt is called
from precmd, where the shadowing isn't active. So (today) I've
implemented: on-demand wrapping of any function call with the
shadowing-enabling code, and then after a single call unwrapping. This
way I'm able to unload your plugin.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 17:55   ` Sebastian Gniazdowski
@ 2019-07-30 18:12     ` Roman Perepelitsa
  2019-07-30 18:16       ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 18:12 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 7:55 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> Yes, implementation of plugin effects withdrawal (i.e. unloading)
> bases on this. I saw that your initialization of the prompt is called
> from precmd, where the shadowing isn't active. So (today) I've
> implemented: on-demand wrapping of any function call with the
> shadowing-enabling code, and then after a single call unwrapping. This
> way I'm able to unload your plugin.

When you unload plugins, you also remove temporary files and kill
background daemons, right?

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:12     ` Roman Perepelitsa
@ 2019-07-30 18:16       ` Sebastian Gniazdowski
  2019-07-30 18:22         ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 18:16 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 20:12, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 7:55 PM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> > Yes, implementation of plugin effects withdrawal (i.e. unloading)
> > bases on this. I saw that your initialization of the prompt is called
> > from precmd, where the shadowing isn't active. So (today) I've
> > implemented: on-demand wrapping of any function call with the
> > shadowing-enabling code, and then after a single call unwrapping. This
> > way I'm able to unload your plugin.
>
> When you unload plugins, you also remove temporary files and kill
> background daemons, right?
>
> Roman.

No. I guess that the temp-files removal can be easily added – by
shadowing mktemp call. I didn't think about killing of the background
processes. I guess this too can be added, but I suspect that most of
them will be invisible in $jobtexts, i.e. disowned, so it might get
hard. Thanks for the suggestions. How are you running the gitstatus
process in the plugin?

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:16       ` Sebastian Gniazdowski
@ 2019-07-30 18:22         ` Roman Perepelitsa
  2019-07-30 18:53           ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 18:22 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 8:16 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Tue, 30 Jul 2019 at 20:12, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > On Tue, Jul 30, 2019 at 7:55 PM Sebastian Gniazdowski
> > <sgniazdowski@gmail.com> wrote:
> > > Yes, implementation of plugin effects withdrawal (i.e. unloading)
> > > bases on this. I saw that your initialization of the prompt is called
> > > from precmd, where the shadowing isn't active. So (today) I've
> > > implemented: on-demand wrapping of any function call with the
> > > shadowing-enabling code, and then after a single call unwrapping. This
> > > way I'm able to unload your plugin.
> >
> > When you unload plugins, you also remove temporary files and kill
> > background daemons, right?
> >
> > Roman.
>
> No. I guess that the temp-files removal can be easily added – by
> shadowing mktemp call. I didn't think about killing of the background
> processes. I guess this too can be added, but I suspect that most of
> them will be invisible in $jobtexts, i.e. disowned, so it might get
> hard. Thanks for the suggestions. How are you running the gitstatus
> process in the plugin?

Do you think it's possible to implement clean shutdown and reentrant
initialization for a piece of code as blackbox?

gitstatusd and p10k both have clean shutdown in their public APIs. It
was quite difficult to implement. By using this API you release all
resources and can initialize the plugins once again within the same
shell. However, if you unset internal variables or close file
descriptors, or fail to kill daemons, you are in undefined behavior
land and all bets are off.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 17:00 A serious bug in execution – where to debug? Sebastian Gniazdowski
  2019-07-30 17:05 ` Sebastian Gniazdowski
  2019-07-30 17:41 ` Roman Perepelitsa
@ 2019-07-30 18:27 ` Bart Schaefer
  2019-07-30 18:46   ` Sebastian Gniazdowski
  2 siblings, 1 reply; 33+ messages in thread
From: Bart Schaefer @ 2019-07-30 18:27 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 10:01 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> If I remove the conditional from its third line, i.e. change the line
> to:
>
> ZPLG_REPORTS[_dtrace/_dtrace]+="$2"$'\n'
>
> then the problem disappears.

If you change the [[ ... ]] && ... to an if/then, does the behavior change?
If you add a "return" or "return $?" at the end of the function, does it change?

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:27 ` Bart Schaefer
@ 2019-07-30 18:46   ` Sebastian Gniazdowski
  2019-07-30 21:02     ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 18:46 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 20:28, Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Tue, Jul 30, 2019 at 10:01 AM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > If I remove the conditional from its third line, i.e. change the line
> > to:
> >
> > ZPLG_REPORTS[_dtrace/_dtrace]+="$2"$'\n'
> >
> > then the problem disappears.
>
> If you change the [[ ... ]] && ... to an if/then, does the behavior change?
> If you add a "return" or "return $?" at the end of the function, does it change?

a) yes, the problem cancels with if-then
b) no, adding one of the return's doesn't change the behavior

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:22         ` Roman Perepelitsa
@ 2019-07-30 18:53           ` Sebastian Gniazdowski
  2019-07-30 19:23             ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 18:53 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 20:22, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 8:16 PM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > On Tue, 30 Jul 2019 at 20:12, Roman Perepelitsa
> > <roman.perepelitsa@gmail.com> wrote:
> > >
> > > On Tue, Jul 30, 2019 at 7:55 PM Sebastian Gniazdowski
> > > <sgniazdowski@gmail.com> wrote:
> > No. I guess that the temp-files removal can be easily added – by
> > shadowing mktemp call. I didn't think about killing of the background
> > processes. I guess this too can be added, but I suspect that most of
> > them will be invisible in $jobtexts, i.e. disowned, so it might get
> > hard. Thanks for the suggestions. How are you running the gitstatus
> > process in the plugin?
>
> Do you think it's possible to implement clean shutdown and reentrant
> initialization for a piece of code as blackbox?

What do you mean exactly? As a blackbox - as a regular, repeatable method?

> gitstatusd and p10k both have clean shutdown in their public APIs. It
> was quite difficult to implement. By using this API you release all
> resources and can initialize the plugins once again within the same
> shell. However, if you unset internal variables or close file
> descriptors, or fail to kill daemons, you are in undefined behavior
> land and all bets are off.

This might be a good use case for the proposed unload function
(http://zdharma.org/Zsh-100-Commits-Club/Zsh-Plugin-Standard.html#unload-fun).
If you would provide a function powerlevel10k_unload_plugin, then I
would call it in Zplugin when unloading the plugin, allowing the
function to close the background process.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:53           ` Sebastian Gniazdowski
@ 2019-07-30 19:23             ` Roman Perepelitsa
  2019-07-30 19:34               ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 19:23 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 8:53 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Tue, 30 Jul 2019 at 20:22, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > Do you think it's possible to implement clean shutdown and reentrant
> > initialization for a piece of code as blackbox?
>
> What do you mean exactly? As a blackbox - as a regular, repeatable method?

In order to implement clean shutdown for a piece of code with
non-trivial capabilities you have to rely on its implementation
details. You cannot implement generic clean shutdown that will work
with any code.

Here's an example. Suppose a plugin you are monitoring spawns a
process. Should you kill it when you unload the plugin? It depends. If
this plugin spawns a process every time it is loaded and kills it
every time you call its public `shutdown` function, then you should
probably kill it to avoid leaking processes. You don't want to have a
horde of stray processes if the plugin is loaded and unloaded multiple
times. On the other hand, if this plugin spawns a one-per-user daemon
(could also be one-per-machine), then another shell might already be
using it, and if you kill the daemon when the plugin is unloaded in
one shell, you can break that same plugin in another shell. Plugins
don't expect that their internal processes will be randomly killed,
files randomly deleted and internal variables randomly unset, so if
you start doing it, things will break. To implement clean shutdown in
this case, you need to check whether this plugin is the last instance
for this user/machine and only then kill the daemon. In other words,
in both cases you need to implement the same logic the plugin already
implements in its shutdown routine. Once you implement it, you have to
keep it in sync with internal implementation changes of the plugin.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 19:23             ` Roman Perepelitsa
@ 2019-07-30 19:34               ` Sebastian Gniazdowski
  2019-07-30 19:41                 ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 19:34 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 21:24, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 8:53 PM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > On Tue, 30 Jul 2019 at 20:22, Roman Perepelitsa
> > <roman.perepelitsa@gmail.com> wrote:
> > >
> > > Do you think it's possible to implement clean shutdown and reentrant
> > > initialization for a piece of code as blackbox?
> >
> > What do you mean exactly? As a blackbox - as a regular, repeatable method?
>
> In order to implement clean shutdown for a piece of code with
> non-trivial capabilities you have to rely on its implementation
> details. You cannot implement generic clean shutdown that will work
> with any code.

Yeah but this is solved by the unload function that I've described in
the previous post.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 19:34               ` Sebastian Gniazdowski
@ 2019-07-30 19:41                 ` Roman Perepelitsa
  2019-07-30 19:59                   ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 19:41 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

[-- Attachment #1: Type: text/plain, Size: 1583 bytes --]

On Tue, Jul 30, 2019 at 9:34 PM Sebastian Gniazdowski <
sgniazdowski@gmail.com> wrote:

> On Tue, 30 Jul 2019 at 21:24, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > On Tue, Jul 30, 2019 at 8:53 PM Sebastian Gniazdowski
> > <sgniazdowski@gmail.com> wrote:
> > >
> > > On Tue, 30 Jul 2019 at 20:22, Roman Perepelitsa
> > > <roman.perepelitsa@gmail.com> wrote:
> > > >
> > > > Do you think it's possible to implement clean shutdown and reentrant
> > > > initialization for a piece of code as blackbox?
> > >
> > > What do you mean exactly? As a blackbox - as a regular, repeatable
> method?
> >
> > In order to implement clean shutdown for a piece of code with
> > non-trivial capabilities you have to rely on its implementation
> > details. You cannot implement generic clean shutdown that will work
> > with any code.
>
> Yeah but this is solved by the unload function that I've described in
> the previous post.
>

I think we are in agreement. If there is a public function to unload, you
can call it. If there isn't, it's a missing feature that cannot be provided
by anyone other than the code's author. it's not possible to safely unload
code as a blackbox without knowing all of its implementation details.

Both gitstatusd and p10k do provide public functions for unloading. You are
welcome to call them. They guarantee that repeated load+unload won't leak
resources. That is, as long as you don't unset any of their internal
variables. They do leave a handful of global parameters after unloading;
without them they cannot be loaded correctly again.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 19:41                 ` Roman Perepelitsa
@ 2019-07-30 19:59                   ` Sebastian Gniazdowski
  2019-07-30 20:08                     ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 19:59 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 21:42, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 9:34 PM Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
>>
>> On Tue, 30 Jul 2019 at 21:24, Roman Perepelitsa
>> <roman.perepelitsa@gmail.com> wrote:
> I think we are in agreement. If there is a public function to unload, you can call it. If there isn't, it's a missing feature that cannot be provided by anyone other than the code's author. it's not possible to safely unload code as a blackbox without knowing all of its implementation details.

Yes, but that's in general. I have very positive experiences with
unloading of prompts. Currently in my setup by setting parameter
MYPROMPT=1..8 I can choose the prompt that's active, as described in:
http://zdharma.org/zplugin/site/Multiple-prompts/. In general it is
like you've said, e.g. syntax-highlighting plugin's won't unload
correctly, but only due to a final glitch and as I tested now, it
*did* unload correctly (due to changes in my setup, most probably).
But in general yes, plugins often do not unload fully clean, so I
think that we are in agreement.

> Both gitstatusd and p10k do provide public functions for unloading. You are welcome to call them. They guarantee that repeated load+unload won't leak resources. That is, as long as you don't unset any of their internal variables. They do leave a handful of global parameters after unloading; without them they cannot be loaded correctly again.

Could the function powerlevel10k_unload_plugin() be provided? I could
deduce it from the plugin's name and call it in a general manner.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 19:59                   ` Sebastian Gniazdowski
@ 2019-07-30 20:08                     ` Roman Perepelitsa
  2019-07-30 20:38                       ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 20:08 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Zsh hackers list

On Tue, Jul 30, 2019 at 9:59 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Tue, 30 Jul 2019 at 21:42, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > On Tue, Jul 30, 2019 at 9:34 PM Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> >>
> >> On Tue, 30 Jul 2019 at 21:24, Roman Perepelitsa
> >> <roman.perepelitsa@gmail.com> wrote:
> > I think we are in agreement. If there is a public function to unload, you can call it. If there isn't, it's a missing feature that cannot be provided by anyone other than the code's author. it's not possible to safely unload code as a blackbox without knowing all of its implementation details.
>
> Yes, but that's in general. I have very positive experiences with
> unloading of prompts. Currently in my setup by setting parameter
> MYPROMPT=1..8 I can choose the prompt that's active, as described in:
> http://zdharma.org/zplugin/site/Multiple-prompts/. In general it is
> like you've said, e.g. syntax-highlighting plugin's won't unload
> correctly, but only due to a final glitch and as I tested now, it
> *did* unload correctly (due to changes in my setup, most probably).
> But in general yes, plugins often do not unload fully clean, so I
> think that we are in agreement.

I'm not above some dirty hacks myself but you have to be clear with
your messaging if you are inviting others to use your code. If your
code may or may not work, can leak resources and corrupt files
(corrupting .git is pretty bad), can work today and stop working
tomorrow after an update, -- you might want to mention it.

>
> > Both gitstatusd and p10k do provide public functions for unloading. You are welcome to call them. They guarantee that repeated load+unload won't leak resources. That is, as long as you don't unset any of their internal variables. They do leave a handful of global parameters after unloading; without them they cannot be loaded correctly again.
>
> Could the function powerlevel10k_unload_plugin() be provided? I could
> deduce it from the plugin's name and call it in a general manner.

I have no objections against your providing this function, although I
would prefer that you didn't use "powerlevel10k" prefix to avoid
potential name clashes in the future. As long as it calls only the
public functions of p10k, it won't break when p10k evolves.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 20:08                     ` Roman Perepelitsa
@ 2019-07-30 20:38                       ` Sebastian Gniazdowski
  0 siblings, 0 replies; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 20:38 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Tue, 30 Jul 2019 at 22:09, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
> I'm not above some dirty hacks myself but you have to be clear with
> your messaging if you are inviting others to use your code. If your
> code may or may not work, can leak resources and corrupt files
> (corrupting .git is pretty bad), can work today and stop working
> tomorrow after an update, -- you might want to mention it.

The unloading was a tentative, good-natured initiatory and it still
didn't found its full resolution. But it's a tool in the toolbox and
it's already proving its usefulness (e.g. the MYPROMPT=1..8 setup).

> I have no objections against your providing this function, although I
> would prefer that you didn't use "powerlevel10k" prefix to avoid
> potential name clashes in the future. As long as it calls only the
> public functions of p10k, it won't break when p10k evolves.

Follow-up off-list.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 18:46   ` Sebastian Gniazdowski
@ 2019-07-30 21:02     ` Roman Perepelitsa
  2019-07-30 21:38       ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 21:02 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Bart Schaefer, Zsh hackers list

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Tue, Jul 30, 2019 at 11:00 PM Sebastian Gniazdowski <
sgniazdowski@gmail.com> wrote:

> On Tue, 30 Jul 2019 at 20:28, Bart Schaefer <schaefer@brasslantern.com>
> wrote:
> >
> > On Tue, Jul 30, 2019 at 10:01 AM Sebastian Gniazdowski
> > <sgniazdowski@gmail.com> wrote:
> > >
> > > If I remove the conditional from its third line, i.e. change the line
> > > to:
> > >
> > > ZPLG_REPORTS[_dtrace/_dtrace]+="$2"$'\n'
> > >
> > > then the problem disappears.
> >
> > If you change the [[ ... ]] && ... to an if/then, does the behavior
> change?
> > If you add a "return" or "return $?" at the end of the function, does it
> change?
>
> a) yes, the problem cancels with if-then
> b) no, adding one of the return's doesn't change the behavior
>

I think Bart solved it. Do you have `emulate -L zsh` in your `zle` function?

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 21:02     ` Roman Perepelitsa
@ 2019-07-30 21:38       ` Sebastian Gniazdowski
  2019-07-30 21:45         ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 21:38 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Bart Schaefer, Zsh hackers list

On Tue, 30 Jul 2019 at 23:02, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>> On Tue, 30 Jul 2019 at 20:28, Bart Schaefer <schaefer@brasslantern.com> wrote:
>> > If you change the [[ ... ]] && ... to an if/then, does the behavior change?
>> > If you add a "return" or "return $?" at the end of the function, does it change?
>>
>> a) yes, the problem cancels with if-then
>> b) no, adding one of the return's doesn't change the behavior
>
> I think Bart solved it. Do you have `emulate -L zsh` in your `zle` function?

No, but adding it helps. Any emulation (sh, ksh) does. Aah, so this is
the errreturn option set within your plugin! and the if-then was
setting the return code to 0. A very interesting situation, having
your code called in foreign context, to know what options to defend
from.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 21:38       ` Sebastian Gniazdowski
@ 2019-07-30 21:45         ` Roman Perepelitsa
  2019-07-30 21:54           ` Sebastian Gniazdowski
  2019-07-30 22:18           ` Daniel Shahaf
  0 siblings, 2 replies; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 21:45 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Bart Schaefer, Zsh hackers list

On Tue, Jul 30, 2019 at 11:38 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Tue, 30 Jul 2019 at 23:02, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >> On Tue, 30 Jul 2019 at 20:28, Bart Schaefer <schaefer@brasslantern.com> wrote:
> >> > If you change the [[ ... ]] && ... to an if/then, does the behavior change?
> >> > If you add a "return" or "return $?" at the end of the function, does it change?
> >>
> >> a) yes, the problem cancels with if-then
> >> b) no, adding one of the return's doesn't change the behavior
> >
> > I think Bart solved it. Do you have `emulate -L zsh` in your `zle` function?
>
> No, but adding it helps. Any emulation (sh, ksh) does. Aah, so this is
> the errreturn option set within your plugin! and the if-then was
> setting the return code to 0. A very interesting situation, having
> your code called in foreign context, to know what options to defend
> from.

You should call `emulate -L zsh` from every functions that can be
called by any code other than yours. It's almost impossible to write
code that works the same way with all combinations of options. In my
public code I also unset aliases. These defensive measures reduce the
number of bug reports by *a lot*. People really seem to like creating
global aliases with single-letter names. I'll probably start calling
`disable -f` for all builtins I use.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 21:45         ` Roman Perepelitsa
@ 2019-07-30 21:54           ` Sebastian Gniazdowski
  2019-07-30 22:11             ` Roman Perepelitsa
  2019-07-30 22:18           ` Daniel Shahaf
  1 sibling, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-30 21:54 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Bart Schaefer, Zsh hackers list

On Tue, 30 Jul 2019 at 23:45, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 11:38 PM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > On Tue, 30 Jul 2019 at 23:02, Roman Perepelitsa
> > <roman.perepelitsa@gmail.com> wrote:
> > >> On Tue, 30 Jul 2019 at 20:28, Bart Schaefer <schaefer@brasslantern.com> wrote:
> > >> > If you change the [[ ... ]] && ... to an if/then, does the behavior change?
> > >> > If you add a "return" or "return $?" at the end of the function, does it change?
> > >>
> > >> a) yes, the problem cancels with if-then
> > >> b) no, adding one of the return's doesn't change the behavior
> > >
> > > I think Bart solved it. Do you have `emulate -L zsh` in your `zle` function?
> >
> > No, but adding it helps. Any emulation (sh, ksh) does. Aah, so this is
> > the errreturn option set within your plugin! and the if-then was
> > setting the return code to 0. A very interesting situation, having
> > your code called in foreign context, to know what options to defend
> > from.
>
> You should call `emulate -L zsh` from every functions that can be
> called by any code other than yours. It's almost impossible to write

I avoided localoptions for performance – it consumes some time, as it
(probably, but the performance effect is confirmed) allocates a new
table of options and fills it with defaults. So I've decided to write
an option-transparent code. But turns out that this might be hard in
case of such options as errreturn. I'll think about whether to replace
inline conditions with if-then or to do emulate -L.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 21:54           ` Sebastian Gniazdowski
@ 2019-07-30 22:11             ` Roman Perepelitsa
  0 siblings, 0 replies; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 22:11 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Bart Schaefer, Zsh hackers list

On Tue, Jul 30, 2019 at 11:54 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Tue, 30 Jul 2019 at 23:45, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > On Tue, Jul 30, 2019 at 11:38 PM Sebastian Gniazdowski
> > <sgniazdowski@gmail.com> wrote:
> I avoided localoptions for performance – it consumes some time, as it
> (probably, but the performance effect is confirmed) allocates a new
> table of options and fills it with defaults. So I've decided to write
> an option-transparent code. But turns out that this might be hard in
> case of such options as errreturn. I'll think about whether to replace
> inline conditions with if-then or to do emulate -L.

Keep in mind that 5.4 (I think it was 5.4) had a bug where a false
condition under `if` was causing the function to return if err_return
is set. Same with `while` -- it would return from the function at the
end of iteration. 5.4 is still very popular.

My solution for the performance problem is to call `emulate -L zsh`
only at the points of entry (public API) and then avoid calling too
many small functions.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 21:45         ` Roman Perepelitsa
  2019-07-30 21:54           ` Sebastian Gniazdowski
@ 2019-07-30 22:18           ` Daniel Shahaf
  2019-07-30 22:32             ` Roman Perepelitsa
  1 sibling, 1 reply; 33+ messages in thread
From: Daniel Shahaf @ 2019-07-30 22:18 UTC (permalink / raw)
  To: zsh-workers

Roman Perepelitsa wrote on Tue, 30 Jul 2019 21:46 +00:00:
> You should call `emulate -L zsh` from every functions that can be
> called by any code other than yours. It's almost impossible to write
> code that works the same way with all combinations of options. In my
> public code I also unset aliases. These defensive measures reduce the
> number of bug reports by *a lot*. People really seem to like creating
> global aliases with single-letter names.

z-sy-h also does 'emulate -L' and unsets aliases.  I wonder if there
should be a built-in way to do this, in order to make it easier to write
plugins?  Perhaps a «source -U foo.zsh» syntax, as in autoload?

> I'll probably start calling `disable -f` for all builtins I use.

I wonder if you might be throwing the baby out with the bathwater here.
Aren't there legitimate use-cases for writing shadowing functions?  For
example, what if somebody does «zle() { typeset -p funcstack >&2; zle "$@" }»
for debugging purposes?

(And yes, I'm aware compsys and z-sy-h both use 'command' and 'builtin' in places..)

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 22:18           ` Daniel Shahaf
@ 2019-07-30 22:32             ` Roman Perepelitsa
  2019-07-31  1:30               ` Sebastian Gniazdowski
  2019-07-31  1:42               ` Bart Schaefer
  0 siblings, 2 replies; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-30 22:32 UTC (permalink / raw)
  To: Daniel Shahaf; +Cc: Zsh hackers list

On Wed, Jul 31, 2019 at 12:19 AM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Roman Perepelitsa wrote on Tue, 30 Jul 2019 21:46 +00:00:
> > You should call `emulate -L zsh` from every functions that can be
> > called by any code other than yours. It's almost impossible to write
> > code that works the same way with all combinations of options. In my
> > public code I also unset aliases. These defensive measures reduce the
> > number of bug reports by *a lot*. People really seem to like creating
> > global aliases with single-letter names.
>
> z-sy-h also does 'emulate -L' and unsets aliases.

By the way, why does it deal with aliases manually instead of
`unsetopt aliases`?

>  I wonder if there
> should be a built-in way to do this, in order to make it easier to write
> plugins?  Perhaps a «source -U foo.zsh» syntax, as in autoload?

The only problem I have with `unsetopt aliases` is that it has to be
done before a function is parsed. This means using one of two
patterns:

1. Remember at the top of the file if `aliases` options is set, then
unset the option. At the bottom of the file, restore aliases.
2. Use two files. The first contains the real meet. The second has an
anonymous function with `emulate -L zsh && unsetopt alizes && source
first.zsh`.

The first is bound to sometimes fail to restore the options, which
causes catastrophic effects. The second requires two files makes
things a bit slower.

I use (2) with large files as it gives me the additional benefit of
being able to prevent extra loads of the file. The small file acts
like a header guard in C. Having the guarantee that my file is sourced
at most once gives many advantages.

> > I'll probably start calling `disable -f` for all builtins I use.
>
> I wonder if you might be throwing the baby out with the bathwater here.
> Aren't there legitimate use-cases for writing shadowing functions?  For
> example, what if somebody does «zle() { typeset -p funcstack >&2; zle "$@" }»
> for debugging purposes?

The flip side is that users often define functions with any name they
feel like and are none the wiser if they are shadowing some obscure
builtin your code needs. If you want to call the builtin, call the
builtin.

I'm on the fence on this issue. As long as I don't get too many bug
reports because of over-ambitious plugin managers, I'm fine. It does
get on my nerves sometimes though. If I could defend against functions
shadowing builtins easily, I would. I just don't know how. If
`builtin`, `disable`, `setopt`, `unsetopt`, `unset` and `unfunction`
are all hidden, escaping the sandbox is tricky.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 22:32             ` Roman Perepelitsa
@ 2019-07-31  1:30               ` Sebastian Gniazdowski
  2019-07-31  7:23                 ` Roman Perepelitsa
  2019-07-31  1:42               ` Bart Schaefer
  1 sibling, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-31  1:30 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, 31 Jul 2019 at 00:39, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:

> I'm on the fence on this issue. As long as I don't get too many bug
> reports because of over-ambitious plugin managers, I'm fine.

So Zplugin is considered an over-ambitious plugin manager by you? :)
I'll take that as a compliment. Besides, isn't your prompt little
over-ambitious too? ;) With the binary, background daemon? ;D

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-30 22:32             ` Roman Perepelitsa
  2019-07-31  1:30               ` Sebastian Gniazdowski
@ 2019-07-31  1:42               ` Bart Schaefer
  1 sibling, 0 replies; 33+ messages in thread
From: Bart Schaefer @ 2019-07-31  1:42 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Daniel Shahaf, Zsh hackers list

[-- Attachment #1: Type: text/plain, Size: 256 bytes --]

On Tue, Jul 30, 2019, 3:33 PM Roman Perepelitsa <roman.perepelitsa@gmail.com>
wrote:

> The only problem I have with `unsetopt aliases` is that it has to be
> done before a function is parsed.
>

That's what autoloading is supposed to be able to avoid.

>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31  1:30               ` Sebastian Gniazdowski
@ 2019-07-31  7:23                 ` Roman Perepelitsa
  2019-07-31 11:41                   ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-31  7:23 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, Jul 31, 2019 at 3:30 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Wed, 31 Jul 2019 at 00:39, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
>
> > I'm on the fence on this issue. As long as I don't get too many bug
> > reports because of over-ambitious plugin managers, I'm fine.
>
> So Zplugin is considered an over-ambitious plugin manager by you? :)

I meant over-ambitious in the specific sense mentioned above --
advertising features that are not and cannot be implemented. There is
no way for users to tell whether unloading is safe to use zplugin to
unload X and what the consequences are if it isn't. Features whose
user-observable behavior cannot be described even in theory, and for
which an honest specification is tightly tied to their implementation
details, are over-ambitious. Saying that zplugin can unload arbitrary
code is inaccurate at best and deceptive at worst. What it can do is
unset parameters/functions/etc set by the code. Note how this refers
to the implementation and not the user-observable behavior. What the
user will actually see is unpredictable.

> Besides, isn't your prompt little
> over-ambitious too? ;) With the binary, background daemon? ;D

Spawning long-lived background processes that opt-out of job control
isn't uncommon among zsh themes and plugins. Powerline, p10k, p9k,
zinc, pure all do it. So does zplugin, I believe. The level of
daemonization and sharing differ: powerline detaches its child process
and shares it among multiple shells; p10k and everything else using
gitstatusd double-forks and creates a new session with setsid; pure
and zplugin leave their processes as children. Technical differences
in background process management make it impossible to implement clean
shutdown for these themes/plugins externally but they don't make them
over-ambitious. Powerline does what it claims to do -- give you a
prompt with lots of colors and ornaments. No leaky abstractions, no
deception.

Having said that, I have no illusions about the utility of my public
zsh code. If all zsh plugins and themes would disappear overnight, I
don't know if the world would be better or worse off. This ecosystem
feeds on customization addiction after all.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31  7:23                 ` Roman Perepelitsa
@ 2019-07-31 11:41                   ` Sebastian Gniazdowski
  2019-07-31 12:40                     ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-31 11:41 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, 31 Jul 2019 at 09:23, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Wed, Jul 31, 2019 at 3:30 AM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > On Wed, 31 Jul 2019 at 00:39, Roman Perepelitsa
> > <roman.perepelitsa@gmail.com> wrote:
> >
> > > I'm on the fence on this issue. As long as I don't get too many bug
> > > reports because of over-ambitious plugin managers, I'm fine.
> >
> > So Zplugin is considered an over-ambitious plugin manager by you? :)
>
> I meant over-ambitious in the specific sense mentioned above --
> advertising features that are not and cannot be implemented.
> ...
> Features whose
> user-observable behavior cannot be described even in theory, and for
> which an honest specification is tightly tied to their implementation
> details, are over-ambitious.

I'm not sure if this will not be "honest specification" (that is) "is
tightly tied to their implementation
details", but somewhat user-observable effects of zplugin unload
procedure "described in theory" are:

1. Withdrawal of any bindkey call, i.e. if a plugin did bindkey ^T
widget, then the binding will be deleted (restoring of previous
binding is a TODO), including range bindings (bindkey -R).
2. Deletion of created keymaps (bindkey -N) and unlinking of the main
keymap (bindkey -A).
3. Withdrawal of any zstyles (again, only deletion).
4. Withdrawal (i.e. deletion) and restoration (i.e. set up of the
previous value) of aliases, including suffix and global aliases.
5. Withdrawal of zle created widgets (zle -N) – restoration and deletion.
6. Restoration of changed options.
7. Removal of added PATH and FPATH elements (adding of removed element
is ready to be implemented, yet it wasn't needed)
8. Deletion of created functions
9. Removal of any added hooks
10. Unset of created global parameters (restoration of previous type's
is a niche TODO).

> Saying that zplugin can unload arbitrary
> code is inaccurate at best and deceptive at worst.

Maybe.. I think that it's an untold common truth shared by any user
that Zplugin cannot be that smart to withdraw *any* sophisticated
effect of a potential plugin. Claiming in an straightforward manner
"Zplugin can unload plugins" is like a statement that encourages to
acknowledge that *some serious* unloading is possible, that the shell
isn't a black hole where things undergo untracked, in one direction
only. Didn't you revise your opinion a little on unloading of plugins
after reading the list from the previous paragraph?

> What the user will actually see is unpredictable.
I think that it's actually possible to predict to a large extent by
looking at the list of unload-actions.

> Spawning long-lived background processes that opt-out of job control
> isn't uncommon among zsh themes and plugins.

To be clear: I like the sophistication of the theme and see it as the
right path.

> Powerline, p10k, p9k,
> zinc, pure all do it. So does zplugin, I believe.

Zplugin forks (if you're talking about the `services' feature) or
loads a binary zshell module zplugin.so.

> Technical differences
> in background process management make it impossible to implement clean
> shutdown for these themes/plugins externally but they don't make them
> over-ambitious.

Ok, agreed.

> Powerline does what it claims to do -- give you a
> prompt with lots of colors and ornaments. No leaky abstractions, no
> deception.

I hope that by the second paragraph I've lessen the "deception"
argument to a significant degree.

> Having said that, I have no illusions about the utility of my public
> zsh code. If all zsh plugins and themes would disappear overnight, I
> don't know if the world would be better or worse off. This ecosystem
> feeds on customization addiction after all.
>
> Roman.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 11:41                   ` Sebastian Gniazdowski
@ 2019-07-31 12:40                     ` Roman Perepelitsa
  2019-07-31 13:10                       ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-31 12:40 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, Jul 31, 2019 at 1:41 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> I'm not sure if this will not be "honest specification" (that is) "is
> tightly tied to their implementation
> details", but somewhat user-observable effects of zplugin unload
> procedure "described in theory" are:
>
> [...]

When users are told they can unload plugins without restarting ZSH,
they hear the following story. Normally, to disable a plugin you must
remove it from your ~/.zshrc and restart ZSH. But if you are using
zplugin you can achieve the same result without restarting your ZSH
simply by typing a command.

This is an easy-to-understand feature. If it could be implemented, it
would be mildly useful. Unfortunately, it cannot be implemented even
for simple plugins.

Suppose I have three plugins specified in ~/.zshrc called foo, bar and
baz, and they are loaded in the specified order. I want to unload foo.
That is, I want to get the same shell state that I would get if I
removed foo from ~/.zshrc and restarted ZSH. While loading, foo
defined an alias for grep: alias grep='grep --color=auto'. When foo is
unloaded, this alias is unset, and this causes two unexpected
consequences. The first is that when I type `grep` in my prompt I get
output without colors. The second is that calling a function from bar
produces colorful output. Once I restart zsh with just bar and baz in
~/.zshrc, these cases swap: interactive grep is colorful while a
function in bar isn't.

How is this possible?

  foo.zsh:  alias grep='grep --color=auto'
  bar.zsh:  function bar() { grep hello <<<hello }
  baz.zsh  alias grep='grep --color=auto'

You can get the same problems when reverting changes to any resource
with a shared name: aliases, traps, options, widgets. All code that
ran after the plugin you are attempting could've been affected by the
value you are reverting.

Removing widgets is very likely to break your prompt due to the way
widgets are commonly hooked. When a decent plugin wants to hook a zle
widget, it will copy the previous widget and install its own. The
installed widget will do something and call the previous widget. Now,
if *this* widget gets wrapped too, and you remove it, the chain of
hooks is broken. All plugins that have wrapped that widget before
won't be called. Plugins that have wrapped the widget after will spew
errors.

> I think that it's actually possible to predict to a large extent by
> looking at the list of unload-actions.

You can predict which aliases will be unset and which widgets removed
but you cannot predict how it will affect shell. Things can break a
little or a lot. Things may look OK but do damage behind the scenes.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 12:40                     ` Roman Perepelitsa
@ 2019-07-31 13:10                       ` Sebastian Gniazdowski
  2019-07-31 13:34                         ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-31 13:10 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, 31 Jul 2019 at 14:40, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
> ...
> How is this possible?
>
>   foo.zsh:  alias grep='grep --color=auto'
>   bar.zsh:  function bar() { grep hello <<<hello }
>   baz.zsh  alias grep='grep --color=auto'

Thanks for bringing this up into the light. It's a good example of the
problems with unloading. It also hints an extension to zplugin – to
test whether the alias' value is the one expected to be and hold the
unsetting/restoration of the alias if it isn't. So, if baz would set
alias grep='grep -B1 --color=auto', then things would work as
expected. And that's how it is with the unloading – it *often* works,
sometimes doesn't. But still, it's an useful tool that can be often
used.

> You can get the same problems when reverting changes to any resource
> with a shared name: aliases, traps, options, widgets. All code that
> ran after the plugin you are attempting could've been affected by the
> value you are reverting.

Ok, but the unloading can still be useful. For example, besides the
MYPROMPT=1..8 use case, I'm also using it often when developing a
plugin, instead of the shell restart. It allows for a quick
unfunction/redeclaration of autoload functions, for example.

> Removing widgets is very likely to break your prompt due to the way
> widgets are commonly hooked. When a decent plugin wants to hook a zle
> widget, it will copy the previous widget and install its own.
> ...
> if *this* widget gets wrapped too, and you remove it, the chain of
> hooks is broken.

This particular case will be solved if the plugin will use
add-zle-hook-widget instead of wrapping (support for this isn't yet in
zplugin, but it'll be there soon).

> > I think that it's actually possible to predict to a large extent by
> > looking at the list of unload-actions.
>
> You can predict which aliases will be unset and which widgets removed
> but you cannot predict how it will affect shell. Things can break a
> little or a lot. Things may look OK but do damage behind the scenes.

To sum up, your opinion is a mathematical-like proof that:

* You cannot implement an unloading that just works as expected.

While my opinion is a practical-view -like point that:

* You can often get good results with unloading, just try & test it
first with the plugin that you need to unload.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 13:10                       ` Sebastian Gniazdowski
@ 2019-07-31 13:34                         ` Roman Perepelitsa
  2019-07-31 13:40                           ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-31 13:34 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, Jul 31, 2019 at 3:10 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> This particular case will be solved if the plugin will use
> add-zle-hook-widget instead of wrapping (support for this isn't yet in
> zplugin, but it'll be there soon).

Unfortunately, using add-zle-hook-widget is no-no. If you use it while
other plugins are still wrapping widgets old-school, you can cause
infinite loops. All plugins have to be switched to add-zle-hook-widget
at the same time, which is pretty much impossible. Hopefully someone
will care enough to fix add-zle-hook-widget but the benefits are
difficult to see. Wrapping widgets works fine. It breaks zplugin's
unloading code but the same can be said about pretty much everything.

> To sum up, your opinion is a mathematical-like proof that:
>
> * You cannot implement an unloading that just works as expected.
>
> While my opinion is a practical-view -like point that:
>
> * You can often get good results with unloading, just try & test it
> first with the plugin that you need to unload.

Both statements are true.

What I don't like is the implication of offering a
maybe-works-maybe-doesn't unloading mechanism. It creates an
expectation that plugins must be unloadable not via their public APIs
but through brutal and unceremonious deletion of their internal
parameters, widgets and so on. If something breaks during unloading,
users may not even realize they've grown to rely on hacks and that
their shell configuration is unsupported. They can reasonably reach
out to the "malfunctioning" plugin's developers and ask them to fix
the "issue". It's like selling rocks as a tool to turn off TV sets.
Just throw a rock at the TV and it'll turn off! If it breaks your TV,
please don't complain to your TV manufacturer.

I also have intense emotional reaction to this kind of unloading. It
just feels rude to modify internal implementation details of software.
It's one thing to do it with code that only you yourself use, or to
apply this sort of dirty patching to a product you are intimately
familiar with, but it's quite another to distribute this crude tool as
a feature. It's disrespectful to the developers whose code is
brutalized, and it causes extra strain on them due to bug reports by
the affected users.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 13:34                         ` Roman Perepelitsa
@ 2019-07-31 13:40                           ` Sebastian Gniazdowski
  2019-07-31 14:11                             ` Roman Perepelitsa
  0 siblings, 1 reply; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-31 13:40 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, 31 Jul 2019 at 15:34, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
> a feature. It's disrespectful to the developers whose code is
> brutalized, and it causes extra strain on them due to bug reports by
> the affected users.

I see. For me the topic isn't associated with brutalization. It's a
removal of plugin's widget together with its binding, so that it's not
exposed anymore. A simple withdrawal of plugin's presence.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 13:40                           ` Sebastian Gniazdowski
@ 2019-07-31 14:11                             ` Roman Perepelitsa
  2019-07-31 17:56                               ` Sebastian Gniazdowski
  0 siblings, 1 reply; 33+ messages in thread
From: Roman Perepelitsa @ 2019-07-31 14:11 UTC (permalink / raw)
  To: Sebastian Gniazdowski; +Cc: Daniel Shahaf, Zsh hackers list

On Wed, Jul 31, 2019 at 3:40 PM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Wed, 31 Jul 2019 at 15:34, Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> > a feature. It's disrespectful to the developers whose code is
> > brutalized, and it causes extra strain on them due to bug reports by
> > the affected users.
>
> I see. For me the topic isn't associated with brutalization. It's a
> removal of plugin's widget together with its binding, so that it's not
> exposed anymore. A simple withdrawal of plugin's presence.

Once upon a time there lived Jack -- the sole designer and
manufacturer of TV sets that bore his name. He took pride in the
functional and aesthetic properties of his product. Jack TVs had a
slick screen that looked almost weightless, and a nice remote for
turning the TV on and off. Then one day an enterprising distributor
showed up in town and started bundling Jack TV together with other
home appliances and advertising rocks as a general tool for turning
off electronic devices. "It often works", he said. Jack shook his head
every time he saw a scratched or broken screen with his name on the
frame. He no longer told people he made those TVs as it became
difficult to look at the brutalized hunks of steel and glass with
pride. "I'm not brutalizing this TV set, I'm just throwing rocks at
it", one customer was heard to say. He didn't know there was a remote.
The distributor didn't tell him.

Roman.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: A serious bug in execution – where to debug?
  2019-07-31 14:11                             ` Roman Perepelitsa
@ 2019-07-31 17:56                               ` Sebastian Gniazdowski
  0 siblings, 0 replies; 33+ messages in thread
From: Sebastian Gniazdowski @ 2019-07-31 17:56 UTC (permalink / raw)
  To: Roman Perepelitsa; +Cc: Zsh hackers list

On Wed, 31 Jul 2019 at 16:11, Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
> Once upon a time there lived Jack -- the sole designer and
> manufacturer of TV sets that bore his name. He took pride in the
> functional and aesthetic properties of his product. Jack TVs had a
> slick screen that looked almost weightless, and a nice remote for
> turning the TV on and off. Then one day an enterprising distributor
> showed up in town and started bundling Jack TV together with other
> home appliances and advertising rocks as a general tool for turning
> off electronic devices. "It often works", he said. Jack shook his head
> ...

Once upon a time there was Garry that had been the designer of the
Hi-Fi that worn his name. He was providing his users with a remote for
the Hi-Fi. Then an enterprising distributor showed up in town and
started bundling Garry Hi-Fi together with a remote, that had a
special function: it allowed to record the music to a pendrive
provided that the hour was even. Each new user at the beginning was
little puzzled with the feature:

1. It provided an amount of uncertainty – some customers when holding
the remote in their hands had small but causing discomfort part of
their minds focused on the question: "Is the hour odd or even?"
2. It caused disappointments when user wanted to record the music
currently being played, but the hour was odd.
3. The user's were questioning the meaningfulness of such feature
(side note: this recalls me the reactions to Intel's hyperthreading,
the "not full hardware threads")

However, over the time, point (1) basically dropped. The anxiety
simply dispersed. The point (2) resolved with "it's not that big of a
deal, with calm attitude all I need to do is simply wait approx 1 hour
to record". (3) resolved with "the feature is a must-have, it allows
me to effectively move my music to the computer or mp3 player". Who
would today want to resign from hyperthreads?


--
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

^ permalink raw reply	[flat|nested] 33+ messages in thread

end of thread, other threads:[~2019-07-31 17:57 UTC | newest]

Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-30 17:00 A serious bug in execution – where to debug? Sebastian Gniazdowski
2019-07-30 17:05 ` Sebastian Gniazdowski
2019-07-30 17:41 ` Roman Perepelitsa
2019-07-30 17:55   ` Sebastian Gniazdowski
2019-07-30 18:12     ` Roman Perepelitsa
2019-07-30 18:16       ` Sebastian Gniazdowski
2019-07-30 18:22         ` Roman Perepelitsa
2019-07-30 18:53           ` Sebastian Gniazdowski
2019-07-30 19:23             ` Roman Perepelitsa
2019-07-30 19:34               ` Sebastian Gniazdowski
2019-07-30 19:41                 ` Roman Perepelitsa
2019-07-30 19:59                   ` Sebastian Gniazdowski
2019-07-30 20:08                     ` Roman Perepelitsa
2019-07-30 20:38                       ` Sebastian Gniazdowski
2019-07-30 18:27 ` Bart Schaefer
2019-07-30 18:46   ` Sebastian Gniazdowski
2019-07-30 21:02     ` Roman Perepelitsa
2019-07-30 21:38       ` Sebastian Gniazdowski
2019-07-30 21:45         ` Roman Perepelitsa
2019-07-30 21:54           ` Sebastian Gniazdowski
2019-07-30 22:11             ` Roman Perepelitsa
2019-07-30 22:18           ` Daniel Shahaf
2019-07-30 22:32             ` Roman Perepelitsa
2019-07-31  1:30               ` Sebastian Gniazdowski
2019-07-31  7:23                 ` Roman Perepelitsa
2019-07-31 11:41                   ` Sebastian Gniazdowski
2019-07-31 12:40                     ` Roman Perepelitsa
2019-07-31 13:10                       ` Sebastian Gniazdowski
2019-07-31 13:34                         ` Roman Perepelitsa
2019-07-31 13:40                           ` Sebastian Gniazdowski
2019-07-31 14:11                             ` Roman Perepelitsa
2019-07-31 17:56                               ` Sebastian Gniazdowski
2019-07-31  1:42               ` Bart Schaefer

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).