bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Axel Beckert]

Hi,

I have the following snippet in my .zshrc:

  autoload -U url-quote-magic
  zle -N self-insert url-quote-magic
  autoload -Uz bracketed-paste-magic
  zle -N bracketed-paste bracketed-paste-magic

The latter two lines are only necessary for zsh ≥ 5.1, but don't seem
to do any harm for earlier versions.

Wth this configuration I noticed the following regression which has
probably been added with zsh 5.1: I can't reproduce it with 5.0.7 in
Debian Stable, but can reproduce it with 5.2 in Debian Unstable as
well as with the current git HEAD:

Pasting an URL, e.g.

  https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=zsh

into zsh with url-quote-magic activated resulted in the the following
commandlines with 5.0.7 (pasting starts with the URL, all other
characters were typed):

  % echo https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh

  % echo "https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=zsh

  % echo 'https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=zsh

This is exactly what I expect: backslashes for quoting were only added
if no quoting using single or double quotes has been started already.

But with zsh 5.2 (and zsh git HEAD), it looks like this:

  % echo https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh

  % echo "https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh

  % echo 'https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh

While the first example still works as expected, I consider the latter
two examples to be some kind of regression. No backslashes should be
added in these cases.

But it only happens if I paste the full URL. It doesn't happen if I
just type the URL and it doesn't happen if just paste the "?src=zsh".
So I can imagine that this possibly is just a missing feature of
bracketed-paste-magic and not a real regression in url-quote-magic.

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Bart Schaefer]

On May 6, 10:53pm, Axel Beckert wrote:
}
} But with zsh 5.2 (and zsh git HEAD), it looks like this:
} 
}   % echo https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh
} 
}   % echo "https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh
} 
}   % echo 'https://bugs.debian.org/cgi-bin/pkgreport.cgi\?src\=zsh
} 
} While the first example still works as expected, I consider the latter
} two examples to be some kind of regression. No backslashes should be
} added in these cases.

You want

    zstyle :bracketed-paste-magic paste-init backward-extend-paste

so that the quote marks that are already on the line are treated as part
of the pasted text.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Vincent Lefevre]

On 2016-05-08 11:50:26 -0700, Bart Schaefer wrote:
> You want
> 
>     zstyle :bracketed-paste-magic paste-init backward-extend-paste
> 
> so that the quote marks that are already on the line are treated as
> part of the pasted text.

Shouldn't this be the default behavior?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Bart Schaefer]

On May 9,  4:13pm, Vincent Lefevre wrote:
} Subject: Re: bracket-paste-magic adds backslashes inside a quoted string i
}
} On 2016-05-08 11:50:26 -0700, Bart Schaefer wrote:
} > You want
} > 
} >     zstyle :bracketed-paste-magic paste-init backward-extend-paste
} > 
} > so that the quote marks that are already on the line are treated as
} > part of the pasted text.
} 
} Shouldn't this be the default behavior?

We had quite a bit of discussion on related topics when bracketed-paste
was added.  The general idea is that a paste is treated as a unit, and
the default behavior of bracketed-paste-magic is to restrict its action
to remain "inside" that unit.

backward-extend-paste is appropriate when bracketed-paste-magic combines
with url-quote-magic, but not necessarily so in other cases.  paste-init
(and paste-finish) can be used for this kind of thing generically; the
backward-extend action is included by default, but has to be turned on.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Vincent Lefevre]

On 2016-05-09 08:41:43 -0700, Bart Schaefer wrote:
> On May 9,  4:13pm, Vincent Lefevre wrote:
> } Subject: Re: bracket-paste-magic adds backslashes inside a quoted string i
> }
> } On 2016-05-08 11:50:26 -0700, Bart Schaefer wrote:
> } > You want
> } > 
> } >     zstyle :bracketed-paste-magic paste-init backward-extend-paste
> } > 
> } > so that the quote marks that are already on the line are treated as
> } > part of the pasted text.
> } 
> } Shouldn't this be the default behavior?
> 
> We had quite a bit of discussion on related topics when bracketed-paste
> was added.  The general idea is that a paste is treated as a unit, and
> the default behavior of bracketed-paste-magic is to restrict its action
> to remain "inside" that unit.

I'm not sure what you call "unit", but IMHO, the notion of unit
should be different from the notion of argument. For instance,
I would agree that a URL could be regarded as a unit, and after

  echo "The URL is: 

it makes sense to paste a URL. So, the context should matter.

> backward-extend-paste is appropriate when bracketed-paste-magic combines
> with url-quote-magic, but not necessarily so in other cases.  paste-init
> (and paste-finish) can be used for this kind of thing generically; the
> backward-extend action is included by default, but has to be turned on.

Or perhaps there should be a function that does all the work for
URL handling, e.g. currently

  autoload -Uz url-quote-magic
  zle -N self-insert url-quote-magic
  autoload -Uz bracketed-paste-magic
  zle -N bracketed-paste bracketed-paste-magic
  zstyle :bracketed-paste-magic paste-init backward-extend-paste

in case there would be future changes.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Bart Schaefer]

On May 10, 10:58am, Vincent Lefevre wrote:
}
} > We had quite a bit of discussion on related topics when bracketed-paste
} > was added.  The general idea is that a paste is treated as a unit, and
} > the default behavior of bracketed-paste-magic is to restrict its action
} > to remain "inside" that unit.
} 
} I'm not sure what you call "unit"

A single keystroke.  A single widget event.  As if the entire paste, no
matter how large, is one indivisible object.

} but IMHO, the notion of unit
} should be different from the notion of argument.

"Argument" has no bearing here; these are units of user input, like,
say, pressing a function key; not units of shell syntax or command
line semantics.

Of course the reason bracketed-paste-magic exists is because sometimes
one might not want that.  The opposite end of the spectrum would be to
explode the paste completely into its individual keystrokes, but that's
no different than unsetting zle_bracketed_paste to turn it off.  So
with bracketed-paste-magic I took the route of keeping the pasted text
self-contained, like doing a recursive-edit or using the minibuffer,
but providing hooks to bring in surrounding context if desired.

bracketed-paste-url-magic takes a different approach, examining the
paste to see if it starts with a URI scheme and if so quoting the
whole thing, and otherwise not interfering with the default paste
behavior at all (so you get even less context sensitivity).

A third (so far unimplemented) approach would be to examine LBUFFER
for an unbalanced quote mark and treat the paste accordingly, but
what exactly does "accordingly" mean?  Escape any quotes of the same
type that are in the paste, or assume the user is intentionally
pasting the closing quote?

} Or perhaps there should be a function that does all the work for
} URL handling

There's more than one way to do it even now, so someone would have to
make a judgement call as to which way was "the" way in order to create
such a function.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Vincent Lefevre]

On 2016-05-10 12:58:48 -0700, Bart Schaefer wrote:
> On May 10, 10:58am, Vincent Lefevre wrote:
> }
> } > We had quite a bit of discussion on related topics when bracketed-paste
> } > was added.  The general idea is that a paste is treated as a unit, and
> } > the default behavior of bracketed-paste-magic is to restrict its action
> } > to remain "inside" that unit.
> } 
> } I'm not sure what you call "unit"
> 
> A single keystroke.  A single widget event.  As if the entire paste, no
> matter how large, is one indivisible object.

But the main point is that there is a context. For instance,
a context can be:

  echo http://localhost/foo

or

  echo 'http://localhost/foo

If I type "?", I expect "\?" to be entered in the first context,
and "?" in the second case. Ditto if "?" is pasted. In both cases,
the action remains inside the unit. So, I don't understand why
bracketed-paste-magic doesn't behave that way by default.

> bracketed-paste-url-magic takes a different approach, examining the
> paste to see if it starts with a URI scheme and if so quoting the
> whole thing, and otherwise not interfering with the default paste
> behavior at all (so you get even less context sensitivity).

bracketed-paste-url-magic is something new to me. I currently have:

  autoload -Uz url-quote-magic
  zle -N self-insert url-quote-magic
  autoload -Uz bracketed-paste-magic
  zle -N bracketed-paste bracketed-paste-magic
  zstyle :bracketed-paste-magic paste-init backward-extend-paste

the only reason of the zstyle line being to get the above expected
behavior when pasting "?". Do you mean that instead of the zstyle
line, bracketed-paste-url-magic would do the work? Does this replace
bracketed-paste-magic or should bracketed-paste-url-magic be used
in addition to bracketed-paste-magic?

bracketed-paste-url-magic is not described in the zsh man pages.

> A third (so far unimplemented) approach would be to examine LBUFFER
> for an unbalanced quote mark and treat the paste accordingly, but
> what exactly does "accordingly" mean?  Escape any quotes of the same
> type that are in the paste, or assume the user is intentionally
> pasting the closing quote?

I would say: the same behavior as url-quote-magic.

URL's normally don't have quotes. For instance, when searching for

  "foo"

or

  'foo'

(with quotes) on Google, Firefox displays the URL's with quotes,
but when I copy-paste these URL's to Emacs, I get:

  https://www.google.fr/#q=%22foo%22
  https://www.google.fr/#q=%27foo%27

and I assume that's Firefox that escapes the quote characters (there
may be a RFC requirement concerning allowed characters in URL's, but
I haven't checked). Note that such escaping is not just useful for
shells, but also for pasting URL's to an XML document in an editor,
since XML attributes also use " and ' quote characters as delimiters.

So, I think that a quote should have its usual shell meaning like
with url-quote-magic (though this could also be configurable).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Bart Schaefer]

On Fri, May 13, 2016 at 2:23 AM, Vincent Lefevre <vincent@vinc17.net> wrote:
> On 2016-05-10 12:58:48 -0700, Bart Schaefer wrote:
>> On May 10, 10:58am, Vincent Lefevre wrote:
>> }
>> } > We had quite a bit of discussion on related topics when bracketed-paste
>> } > was added.  The general idea is that a paste is treated as a unit, and
>> } > the default behavior of bracketed-paste-magic is to restrict its action
>> } > to remain "inside" that unit.
>> }
>> } I'm not sure what you call "unit"
>>
>> A single keystroke.  A single widget event.  As if the entire paste, no
>> matter how large, is one indivisible object.
>
> But the main point is that there is a context.

We're talking past each other a little bit here, I think.

There are two entirely separate things happening here:  (1) special
handling of the paste event; (2) special handling of text that looks
like URLs.

The proponents of "bracketing" the paste event take the position that
a paste is one indivisible action and should be handled as such.
bracketed-paste is like self-insert except that "self" is the entire
pasted text at once.  Context is irrelevant.  Further, treating the
paste this way is expected to "protect" the user from any unexpected
content such as control characters that might have side-effects, and
also to make reverting the paste be a single "undo" step.

This is in direct conflict with special handling of URLs (or any other
kind of possibly-context-dependent action) where, by definition, there
are side-effects from what would otherwise be ordinary input.  Several
choices have to be made in order to resolve this conflict.
bracketed-paste-magic is designed to be a generalized paste handler
that allows the user to specify what happens for each of those
choices.  Consequently the default behavior is to minimally extend the
plain un-magical bracketed-paste, and the more magic you want the more
incantations you must utter.

I apologize if the use of -magic as a suffix here is confusing.  It
was not intended to imply a close connection between
bracketed-paste-magic and url-quote-magic; it's more that both are
inspired by the "magic-space" built-in widget.

Stepping outside the box a little here -- when url-quote-magic was
written there was no such thing as the zle-line-pre-redraw hook.  It
might be that a better approach would be to examine the entire buffer
in that hook after the paste has finished and apply the quoting
retroactively, rather than to do it character by character during the
paste.  (On the other hand zle-line-pre-redraw is thoroughly co-opted
by popular plugins like zsh-syntax-highlighting.)


> Do you mean that instead of the zstyle
> line, bracketed-paste-url-magic would do the work?

No, quite the opposite.  bracketed-paste-url-magic was written because
its author felt that bracketed-paste-magic is too generalized, but he
went far the other way:  Context already on the line is entirely
ignored, and nothing happens unless the entire paste looks like a
single complete URL.

> Does this replace
> bracketed-paste-magic or should bracketed-paste-url-magic be used
> in addition to bracketed-paste-magic?

They're each intended as a bracketed-paste replacement, they don't
work together.

> bracketed-paste-url-magic is not described in the zsh man pages.

We usually leave that up to the writer of the function.  See comments
at top of function source file itself.

>> A third (so far unimplemented) approach would be to examine LBUFFER
>> for an unbalanced quote mark and treat the paste accordingly, but
>> what exactly does "accordingly" mean?  Escape any quotes of the same
>> type that are in the paste, or assume the user is intentionally
>> pasting the closing quote?
>
> I would say: the same behavior as url-quote-magic.

Again, I'm asking what might be done in the general case of someone
using bracketed-paste-magic but not necessarily using url-quote-magic.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Vincent Lefevre]

On 2016-05-13 04:06:45 -0700, Bart Schaefer wrote:
> The proponents of "bracketing" the paste event take the position that
> a paste is one indivisible action and should be handled as such.
> bracketed-paste is like self-insert except that "self" is the entire
> pasted text at once.  Context is irrelevant.  Further, treating the
> paste this way is expected to "protect" the user from any unexpected
> content such as control characters that might have side-effects, and
> also to make reverting the paste be a single "undo" step.

I see. BTW, bracketed-paste-magic breaks the "undo": it clears
the whole line.

> This is in direct conflict with special handling of URLs (or any other
> kind of possibly-context-dependent action) where, by definition, there
> are side-effects from what would otherwise be ordinary input.  Several
> choices have to be made in order to resolve this conflict.
> bracketed-paste-magic is designed to be a generalized paste handler
> that allows the user to specify what happens for each of those
> choices.  Consequently the default behavior is to minimally extend the
> plain un-magical bracketed-paste, and the more magic you want the more
> incantations you must utter.
> 
> I apologize if the use of -magic as a suffix here is confusing.  It
> was not intended to imply a close connection between
> bracketed-paste-magic and url-quote-magic; it's more that both are
> inspired by the "magic-space" built-in widget.

OK. Since the only effect of bracketed-paste-magic I was seeing
was the url-quote-magic one, I thought that this was the equivalent
of url-quote-magic, but with paste instead of keys entered with the
keyboard.

So, the behavior I expect *on the buffer* seems to be the one with
backward-extend-paste (except that undo and paste highlighting are
broken).

> > Do you mean that instead of the zstyle
> > line, bracketed-paste-url-magic would do the work?
> 
> No, quite the opposite.  bracketed-paste-url-magic was written because
> its author felt that bracketed-paste-magic is too generalized, but he
> went far the other way:  Context already on the line is entirely
> ignored, and nothing happens unless the entire paste looks like a
> single complete URL.

Indeed, I've tried, and it works badly. The paste must be a single
complete URL and form a single argument.

> >> A third (so far unimplemented) approach would be to examine LBUFFER
> >> for an unbalanced quote mark and treat the paste accordingly, but
> >> what exactly does "accordingly" mean?  Escape any quotes of the same
> >> type that are in the paste, or assume the user is intentionally
> >> pasting the closing quote?
> >
> > I would say: the same behavior as url-quote-magic.
> 
> Again, I'm asking what might be done in the general case of someone
> using bracketed-paste-magic but not necessarily using url-quote-magic.

IMHO, the behavior should be similar to the case where the user enters
(non-control) characters one by one instead of pasting them.

Now, even url-quote-magic seems incomplete: if I start to type:

  echo $(echo http://foo

then a question mark, it is not quoted.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

undo problems/crashes (was Re: bracket-paste-magic ...)

[Bart Schaefer]

On May 13,  2:19pm, Vincent Lefevre wrote:
} Subject: Re: bracket-paste-magic adds backslashes inside a quoted string i
}
} BTW, bracketed-paste-magic breaks the "undo": it clears
} the whole line.

Hmm, I think this is a relatively new thing; it also core dumps for me
on multiple undo-ings.

Oliver's latest patch in 38540 stops the core dump, but it also stops
the undo from happening at all; in fact if I attempt ESC x undo RET
then not only does the undo not happen, but the RET is interpreted as
accept-line instead of as the end of the execute-named-command input.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Daniel Shahaf]

Bart Schaefer wrote on Fri, May 13, 2016 at 04:06:45 -0700:
> Stepping outside the box a little here -- when url-quote-magic was
> written there was no such thing as the zle-line-pre-redraw hook.  It
> might be that a better approach would be to examine the entire buffer
> in that hook after the paste has finished and apply the quoting
> retroactively, rather than to do it character by character during the
> paste.  (On the other hand zle-line-pre-redraw is thoroughly co-opted
> by popular plugins like zsh-syntax-highlighting.)

So perhaps it's time to permit registering multiple functions to be
invoked at pre-redraw, as proposed in 37639?  This way, z-sy-h and
url-quote-magic could each register a pre-redraw hook function.

zle-line-pre-redraw has not yet been released: its interface can be
changed arbitrarily.  z-sy-h will be updated to use whatever interface
zsh-5.3 will be released with.

Cheers,

Daniel

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Mikael Magnusson]

On Sat, May 14, 2016 at 12:20 AM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Bart Schaefer wrote on Fri, May 13, 2016 at 04:06:45 -0700:
>> Stepping outside the box a little here -- when url-quote-magic was
>> written there was no such thing as the zle-line-pre-redraw hook.  It
>> might be that a better approach would be to examine the entire buffer
>> in that hook after the paste has finished and apply the quoting
>> retroactively, rather than to do it character by character during the
>> paste.  (On the other hand zle-line-pre-redraw is thoroughly co-opted
>> by popular plugins like zsh-syntax-highlighting.)
>
> So perhaps it's time to permit registering multiple functions to be
> invoked at pre-redraw, as proposed in 37639?  This way, z-sy-h and
> url-quote-magic could each register a pre-redraw hook function.
>
> zle-line-pre-redraw has not yet been released: its interface can be
> changed arbitrarily.  z-sy-h will be updated to use whatever interface
> zsh-5.3 will be released with.

It's already possible to do

zle-line-pre-redraw() {
  whatever code
  other code
  z-sy-h-hook
  yet more code
}

Surely there has to be some limit to how lazy a user can be?

Also, if multiple things register for this automatically, it feels
pretty likely more than one of them will be trying to do highlighting
and then things will be wonky. And if you modify the buffer in one of
them like in the GP example, the order _definitely_ matters or you
will have highlights in the wrong place, so the user will have to
intervene in that case. If "plugins" are automatically registering
your more generic hook interface, it is very difficult for the user to
arrange for the order to be correct.

This has a weak NAK from me.

-- 
Mikael Magnusson

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Daniel Shahaf]

Mikael Magnusson wrote on Sat, May 14, 2016 at 03:33:46 +0200:
> On Sat, May 14, 2016 at 12:20 AM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> > Bart Schaefer wrote on Fri, May 13, 2016 at 04:06:45 -0700:
> >> Stepping outside the box a little here -- when url-quote-magic was
> >> written there was no such thing as the zle-line-pre-redraw hook.  It
> >> might be that a better approach would be to examine the entire buffer
> >> in that hook after the paste has finished and apply the quoting
> >> retroactively, rather than to do it character by character during the
> >> paste.  (On the other hand zle-line-pre-redraw is thoroughly co-opted
> >> by popular plugins like zsh-syntax-highlighting.)
> >
> > So perhaps it's time to permit registering multiple functions to be
> > invoked at pre-redraw, as proposed in 37639?  This way, z-sy-h and
> > url-quote-magic could each register a pre-redraw hook function.
> >
> > zle-line-pre-redraw has not yet been released: its interface can be
> > changed arbitrarily.  z-sy-h will be updated to use whatever interface
> > zsh-5.3 will be released with.
> 
> It's already possible to do
> 
> zle-line-pre-redraw() {
>   whatever code
>   other code
>   z-sy-h-hook
>   yet more code
> }
> 
> Surely there has to be some limit to how lazy a user can be?
> 

You're assuming the user knows what is all the code that needs to be run
from pre-redraw.  That need not be the case: the user may have N plugins
and not know which ones of them use pre-redraw, since the use of pre-redraw
by a given plugin might be an implementation detail of that plugin.

Asking the user to explicitly invoke from pre-redraw all plugins that
need to be invoked from pre-redraw would be a compatibility / upgrades
nightmare: whenever a plugin that previously didn't use pre-redraw would
want to start using pre-redraw, all that plugin's end users would have
to edit the Single Master Pre-Redraw Function in their .zshrc's, in
lockstep with installing the updated plugin.  This would simply not
scale.

What would scale is an interface similar to the one add-zsh-hook
provides for non-zle hooks: an interface that allows an arbitrary number
of registrations, where the various registrants — be they the user's
.zshrc or plugins — need not know or care whether there are other
registrants.

z-sy-h uses add-zsh-hook, but I doubt many users of z-sy-h even know
that add-zsh-hook exists.  I'd like the same to be true regarding
pre-redraw.

> Also, if multiple things register for this automatically, it feels
> pretty likely more than one of them will be trying to do highlighting
> and then things will be wonky. And if you modify the buffer in one of
> them like in the GP example, the order _definitely_ matters or you
> will have highlights in the wrong place, so the user will have to
> intervene in that case. If "plugins" are automatically registering
> your more generic hook interface, it is very difficult for the user to
> arrange for the order to be correct.
> 

Yes, if one pre-redraw hook modifies the buffer and the other updates
$region_highlight, then they should run in this order.  How do you
propose that this be achieved?

I think there are two ways: either (1) have plugins document their
ordering requirements, and leave it to the user to 'autoload X && X' the
plugins in a valid order; or (2) extend the widgets/hooks API so plugins
can tell zle what relative order to run them in.

Approach (1) is what z-sy-h uses today under zsh≤5.2 (which of course
lacks zle-line-pre-redraw, but still has a problem analogous to this
"inter-plugin hook ordering" problem).

> This has a weak NAK from me.
> 
> -- 
> Mikael Magnusson

I respect your opinion, but I wish you would suggest an alternative.  As
I said, asking the user to invoke all his plugins from pre-redraw
wouldn't scale.

Cheers,

Daniel
(offline for the next few days)

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Bart Schaefer]

On Fri, May 13, 2016 at 8:00 PM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>
> Asking the user to explicitly invoke from pre-redraw all plugins that
> need to be invoked from pre-redraw would be a compatibility / upgrades
> nightmare: whenever a plugin that previously didn't use pre-redraw would
> want to start using pre-redraw, all that plugin's end users would have
> to edit the Single Master Pre-Redraw Function in their .zshrc's, in
> lockstep with installing the updated plugin.  This would simply not
> scale.
>
> What would scale is an interface similar to the one add-zsh-hook
> provides for non-zle hooks: an interface that allows an arbitrary number
> of registrations, where the various registrants — be they the user's
> .zshrc or plugins — need not know or care whether there are other
> registrants.

I was never very happy with the built-in variables for lists of hooks,
particularly because they needed an externally-defined helper
(add-zsh-hook) to make them practical.  I think add-zsh-hook could
have been written entirely in terms of the original precmd et al.
functions without the corresponding variables.

Hence I also think that a plugin registry for the special zle hook
widgets ought to be implemented externally, and the plugins agree to
make use of it -- because if the plugins don't follow such a
convention then the user has to update .zshrc in lockstep with the
plugin installation anyway.

This convention could include allowing plugins to describe (or at
least suggest) the order in which their hook should be called.

Re: bracket-paste-magic adds backslashes inside a quoted string if URL is pasted ("regression" compared to pre-5.1 url-quote-magic)

[Daniel Shahaf]

Bart Schaefer wrote on Sun, May 15, 2016 at 04:59:27 -0700:
> On Fri, May 13, 2016 at 8:00 PM, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> >
> > Asking the user to explicitly invoke from pre-redraw all plugins that
> > need to be invoked from pre-redraw would be a compatibility / upgrades
> > nightmare: whenever a plugin that previously didn't use pre-redraw would
> > want to start using pre-redraw, all that plugin's end users would have
> > to edit the Single Master Pre-Redraw Function in their .zshrc's, in
> > lockstep with installing the updated plugin.  This would simply not
> > scale.
> >
> > What would scale is an interface similar to the one add-zsh-hook
> > provides for non-zle hooks: an interface that allows an arbitrary number
> > of registrations, where the various registrants — be they the user's
> > .zshrc or plugins — need not know or care whether there are other
> > registrants.
> 
> I was never very happy with the built-in variables for lists of hooks,
> particularly because they needed an externally-defined helper
> (add-zsh-hook) to make them practical.  I think add-zsh-hook could
> have been written entirely in terms of the original precmd et al.
> functions without the corresponding variables.
> 

Fair enough.

> Hence I also think that a plugin registry for the special zle hook
> widgets ought to be implemented externally, and the plugins agree to
> make use of it -- because if the plugins don't follow such a
> convention then the user has to update .zshrc in lockstep with the
> plugin installation anyway.

I don't follow your reasoning: whether .zshrc needs to be updated
manually is orthogonal to how the plugin hooks into zle.  It is about
the mechanism used by zsh to find and run the code that _installs_ the
zle hook, not about whether said installation is done by directly
calling builtins and defining specially-named shell functions or by
calling some registrar function.

(That is: even if a plugin foo uses a "plugin registry", the user would
still need to run «echo 'autoload foo && foo' >> .zshrc».)

I'm happy to keep this external, although if widgets shipped with zsh
used this "external" plugin, it'd probably become de facto official.

Cheers,

Daniel

> This convention could include allowing plugins to describe (or at
> least suggest) the order in which their hook should be called.

zle hook conventions (was Re: bracket-paste-magic ...)

[Bart Schaefer]

On May 19,  9:14pm, Daniel Shahaf wrote:
}
} Bart Schaefer wrote on Sun, May 15, 2016 at 04:59:27 -0700:
} > Hence I also think that a plugin registry for the special zle hook
} > widgets ought to be implemented externally, and the plugins agree to
} > make use of it -- because if the plugins don't follow such a
} > convention then the user has to update .zshrc in lockstep with the
} > plugin installation anyway.
} 
} I don't follow your reasoning: whether .zshrc needs to be updated
} manually is orthogonal to how the plugin hooks into zle.

If you have two or more plugins referenced from .zshrc, and not all of
them follow the same convention for how to hook into zle, then any time
one of them changes the user has to make sure it hasn't now interfered
with the hooks of the others.

} It is about the mechanism used by zsh to find and run the code that
} _installs_ the zle hook, not about whether said installation is done
} by [...]

That's exactly my point: A convention no one uses is no better than no
convention at all, even if there's specialty C code instead of shell
code.

Further, an externally-defined convention can be made backward-compatible
to at least some older versions of zsh.

} I'm happy to keep this external, although if widgets shipped with zsh
} used this "external" plugin, it'd probably become de facto official.

Currently there aren't any functions shipped with zsh that would need
this.  The only references to the zle-* hooks in the shipped code are
tests for whether zle-line-init is currently in progress, not for using
any hooks to set up specific behavior.

Crudely sketched, I'm thinking of something like this:

    zmodload zsh/parameter || {
	print -u2 "Need parameter module for zle hooks"
	return 1
    }
    for hook in isearch-exit isearch-update \
		line-pre-redraw line-init line-finish \
		history-line-set keymap-select
    do
	function zle-$hook {
	    local -a hook_functions hook_widgets
	    local hook
	    # Values of these styles look like number:name
	    # and we run them in number order
	    # $funcstack is more reliable than $0
	    # Also, ksh_arrays is annoying
	    emulate zsh -c '
		zstyle -a $funcstack[2] functions hook_functions
		zstyle -a $funcstack[2] widgets hook_widgets
	    '
	    for hook in "${(@)${(@on)hook_functions}#*:}"
	    do
		"$hook" || return
	    done
	    for hook in "${@${(@on)hook_widgets}#*:}"
	    do
		zle "$hook" -Nw || return
	    done
	    return 0
	}
    done

    # This stuff with "is it a function or is it a widget?" is
    # another reason we didn't define internal arrays for this.
    # Maybe just require widgets, and discard the functions bit.

    add-zle-hook-function() {
	local -aU extant_hooks
	local hook="$1"
	shift
	zstyle -g extant_hooks "$hook" functions
	extant_hooks+=("$@")
	zstyle -- "$hook" functions "${extant_hooks[@]}"
	if [[ -z "${widgets[$hook]}" ]]; then
	    zle -N $hook
	fi
    }

    add-zle-hook-widget() {
	local -aU extant_hooks
	local hook="$1"
	shift
	zstyle -g extant_hooks "$hook" widgets
	extant_hooks+=("$@")
	zstyle -- "$hook" widgets "${extant_hooks[@]}"
	if [[ -z "${widgets[$hook]}" ]]; then
	    zle -N $hook
	fi
    }

Usage would be e.g. (inventing a function for example purposes):

    add-zle-hook-widget zle-line-pre-redraw 30:z-sy-h-redraw

If you leave out the number: prefix, it's assumed you don't care in
what order it executes, but it'll always be after the ones that do
have number prefixes.  Left up to the plugins to work out what number
they need/get.  Note you can call the same widget more than once by
adding it with two different numbers.

Re: zle hook conventions (was Re: bracket-paste-magic ...)

[Daniel Shahaf]

Bart Schaefer wrote on Thu, Jun 02, 2016 at 00:06:00 -0700:
> On May 19,  9:14pm, Daniel Shahaf wrote:
> }
> } Bart Schaefer wrote on Sun, May 15, 2016 at 04:59:27 -0700:
> } > Hence I also think that a plugin registry for the special zle hook
> } > widgets ought to be implemented externally, and the plugins agree to
> } > make use of it -- because if the plugins don't follow such a
> } > convention then the user has to update .zshrc in lockstep with the
> } > plugin installation anyway.
> } 
> } I don't follow your reasoning: whether .zshrc needs to be updated
> } manually is orthogonal to how the plugin hooks into zle.
> 
> If you have two or more plugins referenced from .zshrc, and not all of
> them follow the same convention for how to hook into zle, then any time
> one of them changes the user has to make sure it hasn't now interfered
> with the hooks of the others.
> 

I think it is possible for a plugin to hook into zle without interfering
with other plugins' hooks, without assuming anything about those other
plugins.  z-sy-h's hooking code attempts to achieve that (modulo a known
bug†):

https://github.com/zsh-users/zsh-syntax-highlighting/blob/47144ad9742a948337c6a0e315474b5fea94a6d3/zsh-syntax-highlighting.zsh#L243-L272

That said, I agree that having some an array of functions that are to be
called by a central dispatcher — as with add-zsh-hook and
add-zle-widget-hook — would sidestep this "devise a perfectly composable
wrapper" problem entirely.

> } It is about the mechanism used by zsh to find and run the code that
> } _installs_ the zle hook, not about whether said installation is done
> } by [...]
> 
> That's exactly my point: A convention no one uses is no better than no
> convention at all, even if there's specialty C code instead of shell
> code.
> 

I think you're saying: if one plugin uses add-zle-hook-widget(), then
other plugins can coexist with that plugin by also using
add-zle-hook-widget().

I'm a bit wary of imposing dependencies on other plugins.  Might there
be a valid reason for those fourth-party plugins to prefer not to use
add-zle-hook-widget? 

> Further, an externally-defined convention can be made backward-compatible
> to at least some older versions of zsh.
> 

Good point.

> } I'm happy to keep this external, although if widgets shipped with zsh
> } used this "external" plugin, it'd probably become de facto official.
> 
> Currently there aren't any functions shipped with zsh that would need
> this.  The only references to the zle-* hooks in the shipped code are
> tests for whether zle-line-init is currently in progress, not for using
> any hooks to set up specific behavior.
> 

Currently, no.  However, this subthread started when you mentioned (in
38485) that url-quote-magic considered hooking to zle-line-pre-redraw.

> Crudely sketched, I'm thinking of something like this:
> ⋮
> Usage would be e.g. (inventing a function for example purposes):
> 
>     add-zle-hook-widget zle-line-pre-redraw 30:z-sy-h-redraw
> 
> If you leave out the number: prefix, it's assumed you don't care in
> what order it executes, but it'll always be after the ones that do
> have number prefixes.  Left up to the plugins to work out what number
> they need/get.  Note you can call the same widget more than once by
> adding it with two different numbers.

This appears to be a straightforward registrar API.

I would suggest making this idempotent by uniquifying the list of hooks
(in order to support 'source ~/.zshrc').

With my plugin author perspective, if I were to add a dependency on this
function, I'd want it to be easily available for users to install.
Where would add-zle-hook-widget's implementation live?  In Functions/ in
the zsh tree, or as a third-party project?

> 	zstyle -g extant_hooks "$hook" widgets
> 	extant_hooks+=("$@")
> 	zstyle -- "$hook" widgets "${extant_hooks[@]}"

Why use zstyle's instead of declaring a global array and appending to it?

Many thanks,

Daniel

† The linked excerpt's infinite recursion guard doesn't currently catch
the case that $cur_widget is a user-defined widget and ${widgets[$cur_widget]#user:}
is a function that calls «_zsh_highlight_widget_${cur_widget}»; however,
this should be simple to fix by embedding a nonce (e.g., $EPOCHREALTIME)
in the name of the wrapper functions and widgets.

Re: zle hook conventions (was Re: bracket-paste-magic ...)

[Bart Schaefer]

On Jun 3,  8:40pm, Daniel Shahaf wrote:
}
} I think it is possible for a plugin to hook into zle without interfering
} with other plugins' hooks, without assuming anything about those other
} plugins.  z-sy-h's hooking code attempts to achieve that

Yes, but (a) it's convoluted (b) it doesn't solve the ordering problem
[whichever plugin is sourced last gets its wrapper outermost] (c) it
is fragile in cases such as reloading a plugin [load A, load B, then
load A again, and now you have A() { B() { A() { ... } } } wrapping].

Plus, I thought the whole point of this conversation was to consider
ways to avoid having to do that kind of thing, because most plugins
are a lot more naive than z-sy-h.

} I think you're saying: if one plugin uses add-zle-hook-widget(), then
} other plugins can coexist with that plugin by also using
} add-zle-hook-widget().

Again yes, but also: if no plugin uses add-zle-hook-widget because it
requires zsh-5.X for some X >=3 and those plugins want to keep working
in zsh-4+, then we might as well never have introduced the feature.
If we're taking the position that the feature is useful, we should
design it in such a way that plugin writers want to adopt it.

} I'm a bit wary of imposing dependencies on other plugins.  Might there
} be a valid reason for those fourth-party plugins to prefer not to use
} add-zle-hook-widget?

You seem to be arguing in circles here.  This is back to supporting the
position of throwing up our hands and doing nothing because we can't
(or shouldn't) make everyone do the same thing.

Plugins that don't want to use the convention are in no different boat
than they always were:  They either must be clever like z-sy-h, or they
simply break (or cause breakage) by being naive, and it's then up to
the user who's installing them to work it out (which was my original
point about paying attention to ~/.zshrc when updating a plugin).

} > Usage would be e.g. (inventing a function for example purposes):
} > 
} >     add-zle-hook-widget zle-line-pre-redraw 30:z-sy-h-redraw
} 
} This appears to be a straightforward registrar API.

Intentionally comparable to add-zsh-hook.
 
} I would suggest making this idempotent by uniquifying the list of hooks
} (in order to support 'source ~/.zshrc').

Cf. "crudely sketched", but it in fact does so, the extant_hooks local
is declared with -U, so when assigned it will discard duplicates.

} With my plugin author perspective, if I were to add a dependency on this
} function, I'd want it to be easily available for users to install.
} Where would add-zle-hook-widget's implementation live?  In Functions/ in
} the zsh tree, or as a third-party project?

I'd presume that if the convention were agreed upon, we'd ship it with
the "standard" set of autoloadable functions alongside (or even packed
with, in some way) add-zsh-hook.

} > 	zstyle -g extant_hooks "$hook" widgets
} > 	extant_hooks+=("$@")
} > 	zstyle -- "$hook" widgets "${extant_hooks[@]}"
} 
} Why use zstyle's instead of declaring a global array and appending to it?

Mostly to be able to use the widget names directly without having to
remove hyphens or convert to underscores.  Partly, to avoid polluting
the parameter namespace further.  Also to demonstrate that zstyle is
not just a quaint side-effect of the completion system.

-- 
Barton E. Schaefer

Re: zle hook conventions (was Re: bracket-paste-magic ...)

[Daniel Shahaf]

Bart Schaefer wrote on Fri, Jun 03, 2016 at 15:51:54 -0700:
> On Jun 3,  8:40pm, Daniel Shahaf wrote:
> }
> } I think it is possible for a plugin to hook into zle without interfering
> } with other plugins' hooks, without assuming anything about those other
> } plugins.  z-sy-h's hooking code attempts to achieve that
> 
> Yes, but (a) it's convoluted (b) it doesn't solve the ordering problem
> [whichever plugin is sourced last gets its wrapper outermost] (c) it
> is fragile in cases such as reloading a plugin [load A, load B, then
> load A again, and now you have A() { B() { A() { ... } } } wrapping].
> 

Agreed on all counts.  (Your point (c) is identical to the one
I mentioned in the footnote of my previous mail.)

> Plus, I thought the whole point of this conversation was to consider
> ways to avoid having to do that kind of thing, because most plugins
> are a lot more naive than z-sy-h.
> 

Of course.  I once considered abstracting away the hooking strategy
z-sy-h uses into a separate, reusable component.  I'm already convinced
that add-zle-hook-widget is a far better design for such a reusable
component, but that doesn't stop me from mentioning which other
alternatives I had considered.

> } I think you're saying: if one plugin uses add-zle-hook-widget(), then
> } other plugins can coexist with that plugin by also using
> } add-zle-hook-widget().
> 
> Again yes, but also: if no plugin uses add-zle-hook-widget because it
> requires zsh-5.X for some X >=3 and those plugins want to keep working
> in zsh-4+, then we might as well never have introduced the feature.
> If we're taking the position that the feature is useful, we should
> design it in such a way that plugin writers want to adopt it.
> 

Of course.  That implies, for example, that add-zle-hook-widget should
be written to be backwards compatible with older versions of zsh, even
if its source lives in the zsh-5.3 source tree; and we could recommend
to downstream packagers to backport it to their zsh-5.(≤2) packages,
etc..

> } I'm a bit wary of imposing dependencies on other plugins.  Might there
> } be a valid reason for those fourth-party plugins to prefer not to use
> } add-zle-hook-widget?
> 
> You seem to be arguing in circles here.  This is back to supporting the
> position of throwing up our hands and doing nothing because we can't
> (or shouldn't) make everyone do the same thing.
> 

I'm not proposing that we do nothing.  I am simply trying to ensure
that our outcome will be a design that plugin authors will in fact
adopt.

> Plugins that don't want to use the convention are in no different boat
> than they always were:  They either must be clever like z-sy-h, or they
> simply break (or cause breakage) by being naive, and it's then up to
> the user who's installing them to work it out (which was my original
> point about paying attention to ~/.zshrc when updating a plugin).
> 

Indeed.  For what it's worth, this concern is not academic; there is in
fact a third-party plugin that causes segfaults because of a bug in its
zle hooking.  (I've sent them a patch.)

> } With my plugin author perspective, if I were to add a dependency on this
> } function, I'd want it to be easily available for users to install.
> } Where would add-zle-hook-widget's implementation live?  In Functions/ in
> } the zsh tree, or as a third-party project?
> 
> I'd presume that if the convention were agreed upon, we'd ship it with
> the "standard" set of autoloadable functions alongside (or even packed
> with, in some way) add-zsh-hook.
> 

Perfect.  I'd be happy to agree on add-zle-widget-hook.  (I have a few
localised changes to propose, by the way, once it loses its "crude
sketch" status.)

> } > 	zstyle -g extant_hooks "$hook" widgets
> } > 	extant_hooks+=("$@")
> } > 	zstyle -- "$hook" widgets "${extant_hooks[@]}"
> } 
> } Why use zstyle's instead of declaring a global array and appending to it?
> 
> Mostly to be able to use the widget names directly without having to
> remove hyphens or convert to underscores.  Partly, to avoid polluting
> the parameter namespace further.  Also to demonstrate that zstyle is
> not just a quaint side-effect of the completion system.

The last point resonates with me: I've felt for some time that I could
put zstyle to use, but haven't quite pinned how.

Cheers,

Daniel

Re: zle hook conventions (was Re: bracket-paste-magic ...)

[Daniel Shahaf]

Bart Schaefer wrote on Thu, Jun 02, 2016 at 00:06:00 -0700:
> Crudely sketched, I'm thinking of something like this:

Do you plan to take this further, i.e., to polish it and commit it?
It would be a useful interface to have...

Cheers,

Daniel

>     add-zle-hook-function() {
> 	local -aU extant_hooks
> 	local hook="$1"
> 	shift
> 	zstyle -g extant_hooks "$hook" functions
> 	extant_hooks+=("$@")
> 	zstyle -- "$hook" functions "${extant_hooks[@]}"
> 	if [[ -z "${widgets[$hook]}" ]]; then
> 	    zle -N $hook
> 	fi
>     }
> 
>     add-zle-hook-widget() {
> 	local -aU extant_hooks
> 	local hook="$1"
> 	shift
> 	zstyle -g extant_hooks "$hook" widgets
> 	extant_hooks+=("$@")
> 	zstyle -- "$hook" widgets "${extant_hooks[@]}"
> 	if [[ -z "${widgets[$hook]}" ]]; then
> 	    zle -N $hook
> 	fi
>     }

Re: zle hook conventions (was Re: bracket-paste-magic ...)

[Bart Schaefer]

On Jun 10,  5:36pm, Daniel Shahaf wrote:
}
} Do you plan to take this further, i.e., to polish it and commit it?

Yes, but I'm still catching up from having been out of the country for
three weeks, so limiting my zsh involvement to things I can do quickly.