bracketed paste - chopping trailing newlines

bracketed paste - chopping trailing newlines

[Carsten Hey]

Hi,

35834 adds stripping a final newline from pasted text in order to enable
distinguishing a running command from a paste.

Pasting a command with one trailing space now looks exactly like typing
the command without the trailing space, which is a very intuitive way to
prompt the user to press return to execute the command.

If the paste contains more than one trailing newline, this rationale
does not apply anymore, i.e., stripping one of these newlines does not
imply that the user should press return to execute the command.  Pasting
multiple trailing newlines can also be intentional, e.g., to build
a here document.  Therefore I think, that a trailing newline should only
be stripped if it does not immediately follow an other newline (what the
correct behaviour is if only one newline and no other character is
pasted is debatable - and the behaviour of this corner case could be
adapted for later releases).

Ideally, a stripped newline of a previous paste would be added again if
the user pastes again without acknowledging the previous paste first.
If this would be implemented, pasting $'foo\n' would result in $'foo'
and pasting $'foo\n' and $'bar\n' would result in $'foo\nbar'.


The other natural choice to enable distinguishing a running command from
a paste is using zle_highlight.  This could also avoid the need for the
possibly unexpected newline stripping feature.

Using paste:standout has been suggested, but in 36160 it has been
mentioned that region highlight already uses standout and that there
could be some problems with rendering the cursor on basic terminals.

An alternative to paste:standout is paste:underline.  Given that this
one does not suffer from similar possible problems as paste:standout,
I would prefer using paste:underline as default and dropping the
stripping of trailing newlines (using both would also be an option).


I think at least the 'only chop single newlines' change should be
implemented (this would need to check if n > 2 and to check the value of
wpaste[n-2]).  It would be great if this could already be part of zsh
5.1 in to order to minimize the behavioural changes between releases.


Regards
Carsten

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

Reordering things a bit for response ...

On Aug 29,  3:24am, Carsten Hey wrote:
}
} The other natural choice to enable distinguishing a running command from
} a paste is using zle_highlight.  This could also avoid the need for the
} possibly unexpected newline stripping feature.
} 
} An alternative to paste:standout is paste:underline.  Given that this
} one does not suffer from similar possible problems as paste:standout,

Hm, standout, boldface, and underline are all used already for different
defaults.  Underline is for isearch, though, so I suppose it's quite
unlikely to clash with paste highlighting.

} [...] I think, that a trailing newline should only
} be stripped if it does not immediately follow an other newline

Another way to approach this would be for a trailing newline to actually
be taken as accept-line, provided that RBUFFER is empty.  This would be
the least different from previous (unbracketed) paste behavior, and
could be independent of the number of trailing newlines.  In contexts
other than PS1, e.g., a here-document, this is identical to keeping
the trailing newline (except that the PS2 prompt is repeated).

Here's a way to try that on for size:

    autoload -Uz bracketed-paste-magic
    zle -N bracketed-paste bracketed-paste-magic
    nl-accept () {
      if [[ "$PASTED" = *$'\n' && -z "$RBUFFER" ]]
      then
        PASTED="${PASTED%?}" 
        zle -U $'\n'
      fi
      return 0
    }
    zstyle :bracketed-paste-magic active-widgets ''
    zstyle :bracketed-paste-magic paste-finish nl-accept

Re: bracketed paste - chopping trailing newlines

[Carsten Hey]

% fgrep 5.0.9 *
bracketed-paste-magic:# Starting with zsh-5.0.9, ZLE began to recognize the "bracketed paste"
url-quote-magic:# As of zsh-5.0.9, the following may also be necessary in order to apply


Hi,

to get the most intuitive behaviour possible the bracketed paste code
would need to consider what the parser thinks about the current line.
If we assume, that this information is not available, then, stripping
a newline if the previous character is not a newline (even though
I mentioned that I would prefer not stripping them) in combination with
the described possible behaviour if a paste immediately follows an other
one, might lead to the most intuitive behaviour that can be archived
easily.


* Bart Schaefer [2015-08-28 22:00 -0700]:
> On Aug 29,  3:24am, Carsten Hey wrote:
> } The other natural choice to enable distinguishing a running command from
> } a paste is using zle_highlight.  This could also avoid the need for the
> } possibly unexpected newline stripping feature.
> }
> } An alternative to paste:standout is paste:underline.  Given that this
> } one does not suffer from similar possible problems as paste:standout,
>
> Hm, standout, boldface, and underline are all used already for different
> defaults.  Underline is for isearch, though, so I suppose it's quite
> unlikely to clash with paste highlighting.

I noticed that GNU screen does not support bracketed paste.  Expecting
average zsh users to know which zsh version they run and if they
disabled bracketed paste might be reasonable, but also expecting them to
know which of their terminals support bracketed paste seems to be a bit
too much, given that many people only paste seldom.  If it is thus not
that unusual that users might not know if bracketed paste is currently
enabled, then being able to distinguish between pastes and running
commands in an obvious way by enabling highlighting by default seems to
be more important to me.


> } [...] I think, that a trailing newline should only
> } be stripped if it does not immediately follow an other newline
>
> Another way to approach this would be for a trailing newline to actually
> be taken as accept-line, provided that RBUFFER is empty.  ...

This is a very interesting idea and I might use a variant of it in my
~/.zshrc.  I don't think that this would be adequate as default
behaviour, though, since it breaks the assumption users could have that
a pasted command never runs without being acknowledged first, but YMMV.

> ...  In contexts other than PS1, e.g., a here-document, this is
> identical to keeping the trailing newline (except that the PS2 prompt
> is repeated).
>
> Here's a way to try that on for size:

I assume your prototype was not meant to handle all corner cases in
a perfect way, but other than that, it's a good fit for my personal
needs, thanks :)


Regards
Carsten

[patch] 5.0.9 vs 5.1 in source code comments (was: bracketed paste - chopping trailing newlines)

[Axel Beckert]

Hi,

On Sun, Aug 30, 2015 at 10:25:30PM +0200, Carsten Hey wrote:
> % fgrep 5.0.9 *
> bracketed-paste-magic:# Starting with zsh-5.0.9, ZLE began to recognize the "bracketed paste"
> url-quote-magic:# As of zsh-5.0.9, the following may also be necessary in order to apply

Oops. Should we change that to 5.1 now? :-)

I'd apply following patch to git:

diff --git a/Functions/Zle/bracketed-paste-magic b/Functions/Zle/bracketed-paste-magic
index daf5aec..49f4b66 100644
--- a/Functions/Zle/bracketed-paste-magic
+++ b/Functions/Zle/bracketed-paste-magic
@@ -1,4 +1,4 @@
-# Starting with zsh-5.0.9, ZLE began to recognize the "bracketed paste"
+# Starting with zsh-5.1, ZLE began to recognize the "bracketed paste"
 # capability of terminal emulators, that is, the sequences $'\e[200~' to
 # start a paste and $'\e[201~' to indicate the end of the pasted text.
 # Pastes are handled by the bracketed-paste widget and insert literally
diff --git a/Functions/Zle/url-quote-magic b/Functions/Zle/url-quote-magic
index 362d15c..0e49573 100644
--- a/Functions/Zle/url-quote-magic
+++ b/Functions/Zle/url-quote-magic
@@ -8,7 +8,7 @@
 #       autoload -Uz url-quote-magic
 #       zle -N self-insert url-quote-magic
 
-# As of zsh-5.0.9, the following may also be necessary in order to apply
+# As of zsh-5.1, the following may also be necessary in order to apply
 # quoting to copy-pasted URLs:
 #       autload -Uz bracketed-paste-magic
 #       zle -N bracketed-paste bracketed-paste-magic

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: bracketed paste - chopping trailing newlines

[Axel Beckert]

Hi,

On Sun, Aug 30, 2015 at 10:25:30PM +0200, Carsten Hey wrote:
> I noticed that GNU screen does not support bracketed paste.

JFTR: This will be supported in GNU Screen with version 5 (currentlt
under development) onwards:
http://git.savannah.gnu.org/cgit/screen.git/commit/?id=c389013e89e7b04ce43872f2e72d43f77461a3c0

I don't expect to see that in the current 4.x series, though.

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

Carsten Hey wrote on Sun, Aug 30, 2015 at 22:25:30 +0200:
> Hi,
> 
> to get the most intuitive behaviour possible the bracketed paste code
> would need to consider what the parser thinks about the current line.
> If we assume, that this information is not available, then, stripping
> a newline if the previous character is not a newline (even though
> I mentioned that I would prefer not stripping them) in combination with
> the described possible behaviour if a paste immediately follows an other
> one, might lead to the most intuitive behaviour that can be archived
> easily.
> 

So, to clarify, you are proposing:

- A trailing newline will be stripped if preceded by a non-newline and
  RBUFFER is not empty...
- but will be re-added if immediately followed by another paste.

I don't like the sound of this, for several reasons:

- The second clause constitutes state: <paste>f<backspace><paste> and
  <paste><paste> behave differently.  I think state should be minimized.

- This particular state may cause a \n byte to be added at the _start_
  of the paste, which is an area I never look at: after I do a paste, my
  eyes are on the cursor, at the end of the paste.  So I fear this \n
  might be added without the user noticing it.  This has the potential
  to change the command-line's meaning.

- I think designs should have as few rules and exceptions as possible.
  Saying "Newlines will be stripped if X and Y, unless some third
  condition Z trumps" raises my mental alarm bells, due to the number of
  involved conditions.  (Especially when this is the initial draft of
  a design, rather than complexity that had developed organically over
  time.)

As I've stated in other threads, my preferred option is to simply never
munge the user data at all, to minimize surprising behaviour and
maximize compatibility with other places that accept pastes.  The
problem of user not realizing the command hasn't started to run could be
solved by having zle_highlight=(paste:…) and/or a 'zle -M' message by
default.

We could even make <accept-line> strip the final newline for cosmetic
reasons if it had been inserted by a paste and is syntactically
whitespace.  (The user would know the command has started running
because the pasted text will cease to be highlighted.)

Is there any other place accepting pastes that removes the final
newline?  Everywhere I can think of just pastes the final newline if it
was part of the copied text.

Cheers,

Daniel

> 
> * Bart Schaefer [2015-08-28 22:00 -0700]:
> > On Aug 29,  3:24am, Carsten Hey wrote:
> > } The other natural choice to enable distinguishing a running command from
> > } a paste is using zle_highlight.  This could also avoid the need for the
> > } possibly unexpected newline stripping feature.
> > }
> > } An alternative to paste:standout is paste:underline.  Given that this
> > } one does not suffer from similar possible problems as paste:standout,
> >
> > Hm, standout, boldface, and underline are all used already for different
> > defaults.  Underline is for isearch, though, so I suppose it's quite
> > unlikely to clash with paste highlighting.
> 
> I noticed that GNU screen does not support bracketed paste.  Expecting
> average zsh users to know which zsh version they run and if they
> disabled bracketed paste might be reasonable, but also expecting them to
> know which of their terminals support bracketed paste seems to be a bit
> too much, given that many people only paste seldom.  If it is thus not
> that unusual that users might not know if bracketed paste is currently
> enabled, then being able to distinguish between pastes and running
> commands in an obvious way by enabling highlighting by default seems to
> be more important to me.
> 
> 
> > } [...] I think, that a trailing newline should only
> > } be stripped if it does not immediately follow an other newline
> >
> > Another way to approach this would be for a trailing newline to actually
> > be taken as accept-line, provided that RBUFFER is empty.  ...
> 
> This is a very interesting idea and I might use a variant of it in my
> ~/.zshrc.  I don't think that this would be adequate as default
> behaviour, though, since it breaks the assumption users could have that
> a pasted command never runs without being acknowledged first, but YMMV.
> 
> > ...  In contexts other than PS1, e.g., a here-document, this is
> > identical to keeping the trailing newline (except that the PS2 prompt
> > is repeated).
> >
> > Here's a way to try that on for size:
> 
> I assume your prototype was not meant to handle all corner cases in
> a perfect way, but other than that, it's a good fit for my personal
> needs, thanks :)
> 
> 
> Regards
> Carsten

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Aug 31,  5:47am, Daniel Shahaf wrote:
}
} So, to clarify, you are proposing:
} 
} - A trailing newline will be stripped if preceded by a non-newline and
}   RBUFFER is not empty...

I think stripped if RBUFFER *is* empty, rather than not, because if
RBUFFER is left sitting there not empty then it's obvious that the
paste did not result in a command line being accepted.

} - but will be re-added if immediately followed by another paste.
} 
} I don't like the sound of this, for several reasons:
} 
} - The second clause constitutes state: <paste>f<backspace><paste> and
}   <paste><paste> behave differently.  I think state should be minimized.

I agree.

} As I've stated in other threads, my preferred option is to simply never
} munge the user data at all, to minimize surprising behaviour and
} maximize compatibility with other places that accept pastes.

Of course the problem is that "least surprise" for a current user of zsh
is for the newlines to be accept-line, whereas for a person expecting
the behavior of a text editor the newlines should just be inserted.

My suggestion of accept-line for a final newline only when RBUFFER is
empty was meant as the least objectionable hybrid of the two.

} The problem of user not realizing the command hasn't started to run
} could be solved by having zle_highlight=(paste:...) and/or a 'zle -M'
} message by default.

On further reflection I don't think highlighting the paste is sufficient
to solve this.  Yes, it lets you know that a paste has occurred, but it
doesn't help with realizing that a paste doesn't result in accept-line
if you aren't already familiar with that.

} We could even make <accept-line> strip the final newline for cosmetic
} reasons if it had been inserted by a paste and is syntactically
} whitespace.

I find myself ambivalent about that.

} Is there any other place accepting pastes that removes the final
} newline?  Everywhere I can think of just pastes the final newline if it
} was part of the copied text.

This is also considered by my hybrid suggestion -- the newline is not
removed; it is part of the paste.  It just means something different at
the end of the line (it means what a newline usually means at the end
of a line).

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

Bart Schaefer wrote on Tue, Sep 01, 2015 at 16:44:48 -0700:
> On Aug 31,  5:47am, Daniel Shahaf wrote:
> } The problem of user not realizing the command hasn't started to run
> } could be solved by having zle_highlight=(paste:...) and/or a 'zle -M'
> } message by default.
> 
> On further reflection I don't think highlighting the paste is sufficient
> to solve this.  Yes, it lets you know that a paste has occurred, but it
> doesn't help with realizing that a paste doesn't result in accept-line
> if you aren't already familiar with that.
> 

The purpose of the standout isn't to tell people that a paste occurred,
but to alert them to the changed behaviour of the shell.  The standout
should serve as a clue that the shell is waiting for input.  It may also
serve as a "concomitant unexplained phenomenon" — "My command isn't
executing, and the text I pasted is in reverse video" — which might
assist the user in debugging the unforeseen behaviour.

> } As I've stated in other threads, my preferred option is to simply never
> } munge the user data at all, to minimize surprising behaviour and
> } maximize compatibility with other places that accept pastes.
> 
> Of course the problem is that "least surprise" for a current user of zsh
> is for the newlines to be accept-line, whereas for a person expecting
> the behavior of a text editor the newlines should just be inserted.
> 

An argument that the "least surprise" for a current user of zsh is the
status quo can be used to argue against any change whatsoever.

> My suggestion of accept-line for a final newline only when RBUFFER is
> empty was meant as the least objectionable hybrid of the two.
> 
> } Is there any other place accepting pastes that removes the final
> } newline?  Everywhere I can think of just pastes the final newline if it
> } was part of the copied text.
> 
> This is also considered by my hybrid suggestion -- the newline is not
> removed; it is part of the paste.  It just means something different at
> the end of the line (it means what a newline usually means at the end
> of a line).

I actually quite like the New World Order, in which nothing gets
executed until I press <Enter>.  The only time it irks me is when I run

    zsh -f
    source ./minimal-repro-recipe.zsh

repeatedly, which I do when debugging the shell... but that doesn't
count as a use-case :)

---

In the meantime, I've reverted the strip-final-newline patch (35834)
locally, and I'll keep an eye on my use patterns to see how well
Carsten's suggestion would work for me.  As to Bart's idea, I expect
I will sometimes try to paste a multiline string and _not_ want it to be
executed (e.g., because I'll want to edit it first); perhaps <^V><paste>
could be used to disable newline-stripping.

Cheers,

Daniel

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Aug 30, 10:25pm, Carsten Hey wrote:
}
} * Bart Schaefer [2015-08-28 22:00 -0700]:
} > Another way to approach this would be for a trailing newline to actually
} > be taken as accept-line, provided that RBUFFER is empty.  ...
} 
} This is a very interesting idea and I might use a variant of it in my
} ~/.zshrc.  I don't think that this would be adequate as default
} behaviour, though, since it breaks the assumption users could have that
} a pasted command never runs without being acknowledged first, but YMMV.

Except for the circumstance  that we shipped 5.1 in this state, users
of any previous version of zsh would never have that assumption; it
has never been true, before.

} > Here's a way to try that on for size:
} 
} I assume your prototype was not meant to handle all corner cases in
} a perfect way

It was meant to do exactly what I described, and that's all.  If there
are corner cases that doesn't cover, are they worth the effort?

Re: bracketed paste - chopping trailing newlines

[Carsten Hey]

Hi,

in my previous mail I preferred an extended newline stripping variant.
I think this was wrong.

The proposed "strip a trailing \n only if it is a syntactic whitespace"
variant would solve all possible issues I wanted to solve by this in
a better way.  Even without this syntactic whitespace variant, I would
now prefer not to strip trailing newlines at all with highlighing by
default over my extended newline stripping variant.

This syntactic whitespace variant could be applied after pasting, before
the command gets added to the history, or not at all.  Without some
testing, I don't know which of those I would prefer.


* Bart Schaefer [2015-09-01 16:48 -0700]:
> On Aug 30, 10:25pm, Carsten Hey wrote:
> } > Here's a way to try that on for size:
> }
> } I assume your prototype was not meant to handle all corner cases in
> } a perfect way
>
> It was meant to do exactly what I described, and that's all.

Ok, then this was more than a prototype (with some definitions of this
word in mind).

If used with zsh 5.1, the below quoted part of the description does not
match the behaviour, i.e., a trailing newline is stripped, but you might
have intended it to be used with a zsh that has the newline stripping
commit reverted instead, and then, I think, it should match the
description:

| ...  In contexts other than PS1, e.g., a here-document, this is
| identical to keeping the trailing newline (except that the PS2 prompt
| is repeated).

I should have tried the other part with only one trailing newline too.

> If there are corner cases that doesn't cover, ...

The highlighting also felt wrong somehow.

> ... are they worth the effort?

Only if they are not fixed by reverting the newline stripping and if you
consider this as reasonable alternative for default behaviour in the
next zsh release.

If someone wonders how if this behaviour different from the behaviour of
5.0.8 (not tested, but I don't see anything that could have broken
this), try for example pasting $'echo foo\recho bar'.

An other main paste feature, not running a pasted command without user
interaction, would be disabled by this (which makes it, for my private
setup, a good candidate for being activated if needed by some currently
unused function key).

Btw., I also think that highlighting is sufficient to alert the user
that no command is running - that it vanishes if a command runs makes
this pretty clear for the second time, and the first time the previously
unseen highlighing should catch the attention of the users and make them
check what happend.  zle -M ... seems to be a bit too verbose for
a default behaviour.


Regards
Carsten

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Carsten Hey wrote on Fri, Sep 04, 2015 at 01:59:36 +0200:
> An other main paste feature, not running a pasted command without user
> interaction, would be disabled by this (which makes it, for my private
> setup, a good candidate for being activated if needed by some currently
> unused function key).
> 
> Btw., I also think that highlighting is sufficient to alert the user
> that no command is running - that it vanishes if a command runs makes
> this pretty clear for the second time, and the first time the previously
> unseen highlighing should catch the attention of the users and make them
> check what happend.  zle -M ... seems to be a bit too verbose for
> a default behaviour.

I'm attaching a patch and a zshrc that together demonstrate the proposed
behaviour.  To test them, apply the patch, run the resulting 'zsh -f',
and source the script.

I propose to make the _resulting behaviour_ the default in 5.1.1.  However,
before I spend time on rewriting the attached patch entirely in C, I'd
like to ensure we have consensus for making this change.

The behaviour is: (a) pastes are never executed until <Enter> is pressed;
(b) zle_highlight is set; (c) newlines are removed only at <accept-line>.

Cheers,

Daniel

[-- Attachment #2: paste.diff --]
[-- Type: text/x-patch, Size: 950 bytes --]

diff --git a/ChangeLog b/ChangeLog
index 7b98bfe..af3a4a9 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -366,6 +366,7 @@
 
 	* 36125: Src/Zle/zle_hist.c: don't set history context in get-line
 
+	[reverted post-5.1]
 	* 35834 (tweaked): Src/Zle/zle_misc.c: strip a final newline from
 	pasted text: inserting is hard to tell apart from accepting it
 
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index 2d18628..c1005dc 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -787,12 +787,6 @@ bracketedpaste(char **args)
 	    zmult = 1;
 	    if (region_active)
 		killregion(zlenoargs);
-	    /* Chop a final newline if its insertion would be hard to
-	     * distinguish by the user from the line being accepted. */
-	    else if (n > 1 && zlecontext != ZLCON_VARED &&
-		    (zlecs + (insmode ? 0 : n - 1)) >= zlell &&
-		    wpaste[n-1] == ZWC('\n'))
-		n--;
 	    yankcs = yankb = zlecs;
 	    doinsert(wpaste, n);
 	    yanke = zlecs;

[-- Attachment #3: paste.zshrc --]
[-- Type: text/plain, Size: 349 bytes --]

accept-line accept-and-hold() {
  if [[ $LASTWIDGET == bracketed-paste ]] && 
     [[ $BUFFER[-1] == $'\n' ]]; then
    zle .backward-delete-char
  fi
  zle .$WIDGET -- "$@"
}
zle -N accept-line
zle -N accept-and-hold

# Maybe do the same for the other accept-* widgets.

(( $+zle_highlight )) || zle_highlight=()
zle_highlight+=( paste:standout )

Re: bracketed paste - chopping trailing newlines

[Oliver Kiddle]

Daniel Shahaf wrote:
> The behaviour is: (a) pastes are never executed until <Enter> is pressed;
> (b) zle_highlight is set; (c) newlines are removed only at <accept-line>.

I really don't see the point of doing (c). Perhaps the reason for
removing newlines in the current 5.1 behaviour has got lost in all this.
>From pre-5.1, users are accustomed to pasted newlines being executed.
With a trailing newline, it is now hard to discern that the line hasn't
been executed: the cursor advances to the next line whether the newline
is accepted or inserted so without the stripping the user is left
waiting until they lose patience. I think that the default behaviour
should cater to normal users who don't follow this list.

I agree that the newline stripping isn't ideal but it seemed the best
option for the short term. And it is possible to work around it with a
custom widget.

Oliver

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

On Sep 6, 2015 9:57 AM, "Oliver Kiddle" <okiddle@yahoo.co.uk> wrote:
>
> I think that the default behaviour
> should cater to normal users who don't follow this list.

OK, so who is a normal user who doesn't follow this list?  A zsh user from
past versions who expects the paste to run magic-space and accept-line?
That clearly isn't what you mean, but I'd like to de-confuse them too.

Re: bracketed paste - chopping trailing newlines

[Oliver Kiddle]

Bart wrote:
> OK, so who is a normal user who doesn't follow this list?  A zsh user from
> past versions who expects the paste to run magic-space and accept-line?
> That clearly isn't what you mean, but I'd like to de-confuse them too.

Not getting magic-space in the middle of a paste is fairly apparent. The
same for intervening accept-lines because the prompt is not reprinted
for each line. So in these cases a user expecting old behaviour would
see that there has been a change and hopefully be pleasantly surprised
rather than confused. A trailing newline is not so apparent. It seems
like I'm just repeating myself on the rationale for the blank stripping;
wasn't the case made adequately back in 35794?

I'm all in favour of alternatives to the blank removal that address the
original issue. Simply waiting until user expectations have adapted and
is one alternative. Blank removal in accept-line is a quite separate
notion.

Daniel Shahaf wrote:
> 3. Avoid saving trailing newlines into history.

I'm no more keen on accept-line altering text before it goes into
history than on bracketed-paste altering pasted text.

Oliver

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Sep 8, 12:39pm, Oliver Kiddle wrote:
} 
} Not getting magic-space in the middle of a paste is fairly apparent. The
} same for intervening accept-lines because the prompt is not reprinted
} for each line. So in these cases a user expecting old behaviour would
} see that there has been a change and hopefully be pleasantly surprised
} rather than confused.

Unfortunately this also means that failure of url-quote-magic is also
fairly apparent and is both an unpleasant surprise AND confusing, but
we're already past that.

} A trailing newline is not so apparent.  It seems
} like I'm just repeating myself on the rationale for the blank stripping;
} wasn't the case made adequately back in 35794?

I don't see rationale in that specific message and I re-read a few of the
follow-ups without ever finding it explained all in one place, so maybe
that is the problem.  Or in Carsten's case, it was adequately explained
but he simply disagrees.

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

Oliver Kiddle wrote on Tue, Sep 08, 2015 at 12:39:37 +0200:
> Bart wrote:
> > OK, so who is a normal user who doesn't follow this list?  A zsh user from
> > past versions who expects the paste to run magic-space and accept-line?
> > That clearly isn't what you mean, but I'd like to de-confuse them too.
> 
> Not getting magic-space in the middle of a paste is fairly apparent. The
> same for intervening accept-lines because the prompt is not reprinted
> for each line. So in these cases a user expecting old behaviour would
> see that there has been a change and hopefully be pleasantly surprised
> rather than confused. A trailing newline is not so apparent. It seems
> like I'm just repeating myself on the rationale for the blank stripping;
> wasn't the case made adequately back in 35794?

Yes, I also feel I'm repeating myself.  I've already made my case a few
times.  I've also suggested fixes and implemented them.  I have been
trying to reach consensus, but so far haven't succeeded.  As you can
imagine, by now I'm rather frustrated.

Looking back over the threads, my RFE consisted of three parts:

1. Don't strip newlines at paste
2. Highlight pastes
3. Strip newlines at accept-line that immediately follows a paste

I think no one actually objected to (1), so I'll push it after the 5.1.1
tag, so we have the entire 5.1.2 release cycle to make tweaks.

I think all objections to (2) have been addressed, apart from the
question of whether to use standout or underline.  I think standout is
easier to spot and doesn't clash with zle_highlight[region]'s default
(because the user knows whether she just pasted or not), but I wouldn't
mind underline being the default, if that'll get us out of the impasse.

As to (3), I don't feel strongly about it.  If people don't want it in
the core, I'll just effect it in my dotfiles (as in 36429).

So, to summarize: I'll push the 0001 and 0002 patches from 36443 after
the 5.1.1 tag, and if we want to later change it to underline, we can.

I hope this thread will end soon :-)

Cheers,

Daniel

> I'm all in favour of alternatives to the blank removal that address the
> original issue. Simply waiting until user expectations have adapted and
> is one alternative. Blank removal in accept-line is a quite separate
> notion.
> 
> Daniel Shahaf wrote:
> > 3. Avoid saving trailing newlines into history.
> 
> I'm no more keen on accept-line altering text before it goes into
> history than on bracketed-paste altering pasted text.
> 
> Oliver

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Sep 10,  7:11pm, Daniel Shahaf wrote:
}
} Yes, I also feel I'm repeating myself.  I've already made my case a few
} times.  I've also suggested fixes and implemented them.  I have been
} trying to reach consensus, but so far haven't succeeded.  As you can
} imagine, by now I'm rather frustrated.

This may be a situation where it's not possible to get consensus.  We
have I think three different approaches (Oliver's, Carsten's, yours)
none of the three agrees with either of the others, and we haven't
tried conducting any sort of poll.

} 1. Don't strip newlines at paste
} 2. Highlight pastes
} 3. Strip newlines at accept-line that immediately follows a paste
} 
} I think all objections to (2) have been addressed, apart from the
} question of whether to use standout or underline.  I think standout is
} easier to spot and doesn't clash with zle_highlight[region]'s default

I'm of two minds here.  One, there is a relationship between paste and
the region -- if the region is active, the paste replaces it.  However,
if there is no active region, the paste does not make it active (even
though it sets MARK/CURSOR to the opposite ends and e.g. kill-region
will kill the pasted text).  So two, making the paste look the same as
an active region may be misleading ... or, maybe it's OK because it's
almost the same thing.

I guess I agree with Oliver about the newline stripping.

Re: bracketed paste - chopping trailing newlines

[Mikael Magnusson]

On Sat, Sep 12, 2015 at 1:07 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Sep 10,  7:11pm, Daniel Shahaf wrote:
> }
> } Yes, I also feel I'm repeating myself.  I've already made my case a few
> } times.  I've also suggested fixes and implemented them.  I have been
> } trying to reach consensus, but so far haven't succeeded.  As you can
> } imagine, by now I'm rather frustrated.
>
> This may be a situation where it's not possible to get consensus.  We
> have I think three different approaches (Oliver's, Carsten's, yours)
> none of the three agrees with either of the others, and we haven't
> tried conducting any sort of poll.

If three developers on the mailing list can't agree, chances are at
least one user will be unhappy with whatever we decide.

> } 1. Don't strip newlines at paste
> } 2. Highlight pastes
> } 3. Strip newlines at accept-line that immediately follows a paste
> }
> } I think all objections to (2) have been addressed, apart from the
> } question of whether to use standout or underline.  I think standout is
> } easier to spot and doesn't clash with zle_highlight[region]'s default
>
> I'm of two minds here.  One, there is a relationship between paste and
> the region -- if the region is active, the paste replaces it.  However,
> if there is no active region, the paste does not make it active (even
> though it sets MARK/CURSOR to the opposite ends and e.g. kill-region
> will kill the pasted text).  So two, making the paste look the same as
> an active region may be misleading ... or, maybe it's OK because it's
> almost the same thing.

I personally find it very surprising that pasting would interact with
the region in any way at all, yanking normally doesn't. If this is
some special vi thing, maybe it should only be replaced in vi mode. I
don't think the region and pasted text have anything in common or are
"almost the same thing", and I don't think they should be, unless we
also make yanked text act the same way, which I also don't think we
should.

(My interest here is mostly academic since I won't use any of this
builtin behavior. I would personally prefer if nothing gets deleted,
nothing gets stripped, and generally nothing in general happens except
the text gets pasted, like I asked it to be. I don't think anyone ever
complained to me about the behavior of the widget I made, but then
again, they all added it on purpose).

> I guess I agree with Oliver about the newline stripping.

We have at least two options that decide if and how we print a single
newline before drawing the prompt, maybe there is room for some here
as well.

-- 
Mikael Magnusson

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Sep 12,  2:17am, Mikael Magnusson wrote:
}
} I personally find it very surprising that pasting would interact with
} the region in any way at all, yanking normally doesn't.

Yes, I was a little surprised that Oliver implemented it that way, as
it doesn't seem to be what Emacs does (well, except possibly Aquamacs
on MacOS, which tries to emulate MacOS cmd-c/cmd-v behavior).

However, that adds yet another thing to the list of disagreements, so
I wasn't going to go there.

} > I guess I agree with Oliver about the newline stripping.
} 
} We have at least two options that decide if and how we print a single
} newline before drawing the prompt, maybe there is room for some here
} as well.

Problems with that are (a) for newlines before the prompt, the initial
state is outside the editor, but with paste we're inside the editor
the whole time so any extra stuff we output (other than highlighting)
could be confused with $BUFFER contents; and (b) we're trying to move
away from having module-specific setopts in the main shell.

Using RPS1 with TRANSIENT_RPROMPT might cover a lot of this, as
the prompt would still be there if the line has not been accepted.
Unfortunately RPS1 stays on the first line and is transient based on
the legth of that first line.  It might be nicer if it acted more
like POSTDISPLAY.

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

Bart Schaefer wrote on Fri, Sep 11, 2015 at 16:07:02 -0700:
> On Sep 10,  7:11pm, Daniel Shahaf wrote:
> }
> } Yes, I also feel I'm repeating myself.  I've already made my case a few
> } times.  I've also suggested fixes and implemented them.  I have been
> } trying to reach consensus, but so far haven't succeeded.  As you can
> } imagine, by now I'm rather frustrated.
> 
> This may be a situation where it's not possible to get consensus.  We
> have I think three different approaches (Oliver's, Carsten's, yours)
> none of the three agrees with either of the others, and we haven't
> tried conducting any sort of poll.

I've spoken privately with both Oliver and Carsten and I actually think
there _is_ consensus for both patches that I committed (strip newlines
and enable highlighting).  It's the "strip newlines at accept-line" part
that's not in consensus.

As to interaction with the region: in my workflow, I sometimes do
<set-mark-command>, <typing>, <quote-region>.  During that sequence,
even I happen to do a paste during the <typing> part, I wouldn't want
MARK to move to the start of the paste.  I don't know <paste> should
default to moving MARK.

Cheers,

Daniel

> } 1. Don't strip newlines at paste
> } 2. Highlight pastes
> } 3. Strip newlines at accept-line that immediately follows a paste
> } 
> } I think all objections to (2) have been addressed, apart from the
> } question of whether to use standout or underline.  I think standout is
> } easier to spot and doesn't clash with zle_highlight[region]'s default
> 
> I'm of two minds here.  One, there is a relationship between paste and
> the region -- if the region is active, the paste replaces it.  However,
> if there is no active region, the paste does not make it active (even
> though it sets MARK/CURSOR to the opposite ends and e.g. kill-region
> will kill the pasted text).  So two, making the paste look the same as
> an active region may be misleading ... or, maybe it's OK because it's
> almost the same thing.
> 
> I guess I agree with Oliver about the newline stripping.

Re: bracketed paste - chopping trailing newlines

[Bart Schaefer]

On Sep 14,  8:35pm, Daniel Shahaf wrote:
}
} As to interaction with the region: in my workflow, I sometimes do
} <set-mark-command>, <typing>, <quote-region>.  During that sequence,
} even I happen to do a paste during the <typing> part, I wouldn't want
} MARK to move to the start of the paste.  I don't know <paste> should
} default to moving MARK.

Hm.  This is down to whether pasting is an actual editor action, or
just another form of user input.  For example MARK and CURSOR are
reset by the yank widget in exactly the circumstance you describe,
and bracketed-paste is currently implemented mostly 'as if' the pasted
text were cut into the kill ring and then yanked.

Also, just as an aside, if the region is highlighted ("active") but a
paste does NOT change the region, then how do you decide what ends up
highlighted after the paste?  And if the answer is "both", then how
can the default paste highlight be the same as the default region
highlight without being really confusing?

We still seem to have three competing desires here, depending on who
you ask:

(1) Insert exactly what I pasted, as if it were a single keystroke,
    without further interpretation or change to the editor state.

(2) Handle paste as an atomic editor command similar to yanking from
    an external buffer instead of from the kill ring (as currently).

(3) Treat a paste like multiple keystrokes, so that url-quote-magic
    and other input-related rebindings apply, but don't execute any
    non-input bindings (accept-line, motions, yanks, kills, etc.)

Obviously there can be some overlap, i.e., you might want it treated
as multiple keystrokes but still without change to the editor state.

Maybe we should have stuck with (1) in the C code and built out (2)
in script code the way bracketed-paste-magic builds out (3).  Or
maybe e.g. $zle_bracketed_paste should have more fields to specify
which bits of (2) should be performed.

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

Oliver Kiddle wrote on Sun, Sep 06, 2015 at 15:51:05 +0200:
> Daniel Shahaf wrote:
> > The behaviour is: (a) pastes are never executed until <Enter> is pressed;
> > (b) zle_highlight is set; (c) newlines are removed only at <accept-line>.
> 
> I really don't see the point of doing (c).

Several reasons:

1. So that making several pastes in a row, which each paste a single line
including a \n, will retain the newlines.  In 5.1, the newlines will be
eaten (and users might not know how to insert literal newlines "by hand"
at a place where <accept-line> would simply execute the input-so-far).

2. Cosmetic: when I have a screenful of prompts, I'd like all of them to
have the same number of empty lines between them, regardless of whether
the command was typed manually or pasted.  In other words, once I've
decided to execute the command, I'm happy to remove any trailing \n's it
may have.

3. Avoid saving trailing newlines into history.

> Perhaps the reason for
> removing newlines in the current 5.1 behaviour has got lost in all this.
> From pre-5.1, users are accustomed to pasted newlines being executed.
> With a trailing newline, it is now hard to discern that the line hasn't
> been executed: the cursor advances to the next line whether the newline
> is accepted or inserted so without the stripping the user is left
> waiting until they lose patience.

Thanks for reminding us of that.

I do agree that we shouldn't confuse users.  However, I also think that
most useful handling of pastes is to never strip newlines.  I do not
think we should be handling pastes in a suboptimal manner in order to
avoid risking confusing users.  I think we should handle pastes
optimally — which, to me, means the behaviour described in my previous
mail — and find some other way to ensure we don't confuse users.

How about trying to collect some actual data about whether or not
newline retention confuses users?  For example, we could make the
5.1-test-2 release behave as I'm suggesting, and then wait to see
whether users who test that release complain about pasted commands
hanging indefinitely.

> I think that the default behaviour
> should cater to normal users who don't follow this list.

Agreed.

> I agree that the newline stripping isn't ideal but it seemed the best
> option for the short term. And it is possible to work around it with a
> custom widget.

I'm thinking of 'zsh -f' and of the long term.  I think the behaviour
I'm proposing would make perfect sense to someone who has never pasted
into a shell in his life.

Cheers,

Daniel

Re: bracketed paste - chopping trailing newlines

[Carsten Hey]

* Oliver Kiddle [2015-09-06 15:51 +0200]:
> Daniel Shahaf wrote:
> > The behaviour is: (a) pastes are never executed until <Enter> is
> > pressed; (b) zle_highlight is set; ...

It is still an open question, or at least one without a clear consensus,
whether using paste:standout or paste:underline would be a more
reasonable choice if highlighting will be enabled by default.
I consider paste:standout to be a more natural choice, but there were
concerns regarding using it as default.  I have no clear preference for
any of these two options.


> > ... (c) newlines are removed only at <accept-line>.
>
> I really don't see the point of doing (c).

For example, if this is not done, and assuming the default configuration
using the emacs keymap, you'd have to press <up> three times to move two
entries back in history list after pasting and accepting a command with
a trailing newline.


> With a trailing newline, it is now hard to discern that the line hasn't
> been executed: the cursor advances to the next line whether the newline
> is accepted or inserted so without the stripping the user is left
> waiting until they lose patience.

This is a very intuitive way if users paste exactly one newline and know
that they did so.  Additional highlighting could further improve this
intuitiveness, but it is not necessary for this example.


If you paste a command with two trailing newlines into zsh 5.1 running
in an xterm, the result looks like this (an underline is used as
cursor):

% cp -a /data /mnt/foo
_

If you paste a command with one trailing newline into zsh 5.1 running in
a screen (the latest release, not the 5.x development branch), the
result looks like this:

% cp -a /data /mnt/foo
_

As you notice, both are indistinguishable, but only the first one shows
a running command, and the second one shows a paste waiting to be
accepted.


> I think that the default behaviour
> should cater to normal users who don't follow this list.

> I agree that the newline stripping isn't ideal but it seemed the best
> option for the short term.

I fully agree, and I consider 5.1 to be adequate for such a short term
solution.  5.1-test-1-1 is already tagged in git, and the long term
solution is still being discussed, thus it could not have been part of
a non-delayed 5.1.

For the next release, or at least the one following the next one, a long
term solution could be targeted.


> And it is possible to work around it with a custom widget.

I think we all agree that we are only discussing the default behaviour.


Regards
Carsten

Re: bracketed paste - chopping trailing newlines

[Carsten Hey]

* Carsten Hey [2015-09-09 01:33 +0200]:
> If you paste a command with two trailing newlines into zsh 5.1 running
> in an xterm, the result looks like this (an underline is used as
> cursor):
>
> % cp -a /data /mnt/foo
> _
>
> If you paste a command with one trailing newline into zsh 5.1 running in
> a screen (the latest release, not the 5.x development branch), the
> result looks like this:
>
> % cp -a /data /mnt/foo
> _

If you paste a command with two trailing newlines into screen, the
result looks the same.  The additional newline does not influence
whether the command is run.

> As you notice, both are indistinguishable, but only the first one shows
> a running command, and the second one shows a paste waiting to be
> accepted.

This was wrong.  The first one shows a paste waiting to be accepted and
the second one shows a running command (I did not make this error on
purpose).

Re: bracketed paste - chopping trailing newlines

[Peter Stephenson]

I'm away next week, so I'd like to make 5.1.1 tonight with bug fixes
only (apart from function tweaks).  If this isn't uncontroversial enough
to go in a bug-fix-only release it can wait till next time; the current
code isn't actually broken.

pws

Re: bracketed paste - chopping trailing newlines

[Daniel Shahaf]

[-- Attachment #1: Type: text/plain, Size: 1426 bytes --]

Daniel Shahaf wrote on Sun, Sep 06, 2015 at 09:52:24 +0000:
> Carsten Hey wrote on Fri, Sep 04, 2015 at 01:59:36 +0200:
> > An other main paste feature, not running a pasted command without user
> > interaction, would be disabled by this (which makes it, for my private
> > setup, a good candidate for being activated if needed by some currently
> > unused function key).
> > 
> > Btw., I also think that highlighting is sufficient to alert the user
> > that no command is running - that it vanishes if a command runs makes
> > this pretty clear for the second time, and the first time the previously
> > unseen highlighing should catch the attention of the users and make them
> > check what happend.  zle -M ... seems to be a bit too verbose for
> > a default behaviour.
> 
> I'm attaching a patch and a zshrc that together demonstrate the proposed
> behaviour.  To test them, apply the patch, run the resulting 'zsh -f',
> and source the script.
> 
> I propose to make the _resulting behaviour_ the default in 5.1.1.  However,
> before I spend time on rewriting the attached patch entirely in C, I'd
> like to ensure we have consensus for making this change.
> 
> The behaviour is: (a) pastes are never executed until <Enter> is pressed;
> (b) zle_highlight is set; (c) newlines are removed only at <accept-line>.

I've gone ahead and prepared a patch series that implementes that behaviour
entirely in C.  It's attached.


[-- Attachment #2: 0001-Revert-35834-strip-a-final-newline-from-pasted-text-.patch --]
[-- Type: text/x-patch, Size: 1460 bytes --]

>From 5fff3ba4c26ec7105a54f0189a53145d4a287541 Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <d.s@daniel.shahaf.name>
Date: Wed, 2 Sep 2015 12:16:39 +0000
Subject: [PATCH 1/3] Revert "35834: strip a final newline from pasted text:
 inserting is hard to tell apart from accepting it"

This reverts commit f17eb26a34af69a2238a3d8b46079445e09c096e.

Conflicts:
	ChangeLog
	Src/Zle/zle_misc.c
---
 ChangeLog          | 1 +
 Src/Zle/zle_misc.c | 6 ------
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index cdaed18..bfb71f4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -371,6 +371,7 @@
 
 	* 36125: Src/Zle/zle_hist.c: don't set history context in get-line
 
+	[reverted in NNNNN]
 	* 35834 (tweaked): Src/Zle/zle_misc.c: strip a final newline from
 	pasted text: inserting is hard to tell apart from accepting it
 
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index 2d18628..c1005dc 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -787,12 +787,6 @@ bracketedpaste(char **args)
 	    zmult = 1;
 	    if (region_active)
 		killregion(zlenoargs);
-	    /* Chop a final newline if its insertion would be hard to
-	     * distinguish by the user from the line being accepted. */
-	    else if (n > 1 && zlecontext != ZLCON_VARED &&
-		    (zlecs + (insmode ? 0 : n - 1)) >= zlell &&
-		    wpaste[n-1] == ZWC('\n'))
-		n--;
 	    yankcs = yankb = zlecs;
 	    doinsert(wpaste, n);
 	    yanke = zlecs;
-- 
2.1.4


[-- Attachment #3: 0002-Set-zle_highlight-paste-standout-by-default.patch --]
[-- Type: text/x-patch, Size: 1284 bytes --]

>From 2266d4e4e56dcad8c35ec7f57973f69b0b278ff9 Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <d.s@daniel.shahaf.name>
Date: Mon, 7 Sep 2015 15:50:26 +0000
Subject: [PATCH 2/3] Set zle_highlight=(paste:standout) by default.

---
 Doc/Zsh/zle.yo        | 2 +-
 Src/Zle/zle_refresh.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
index 4e93695..05bae3c 100644
--- a/Doc/Zsh/zle.yo
+++ b/Doc/Zsh/zle.yo
@@ -2656,7 +2656,7 @@ If tt(zle_highlight) is not set or no value applies to a particular
 context, the defaults applied are equivalent to
 
 example(zle_highlight=LPAR()region:standout special:standout
-suffix:bold isearch:underline+RPAR())
+suffix:bold isearch:underline paste:standout+RPAR())
 
 i.e. both the region and special characters are shown in standout mode.
 
diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index 0c28c0a..6facff4 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -372,7 +372,8 @@ zle_set_highlight(void)
 	region_highlights[1].atr = TXTUNDERLINE;
     if (!suffix_atr_on_set)
 	region_highlights[2].atr = TXTBOLDFACE;
-        /* paste defaults to 0 */
+    if (!paste_atr_on_set)
+	region_highlights[3].atr = TXTSTANDOUT;
 
     allocate_colour_buffer();
 }
-- 
2.1.4


[-- Attachment #4: 0003-accept-line-accept-and-hold-accept-and-infer-next-hi.patch --]
[-- Type: text/x-patch, Size: 3246 bytes --]

>From a0e7bfb791703f140fdbe3aee5c24f490945228c Mon Sep 17 00:00:00 2001
From: Daniel Shahaf <d.s@daniel.shahaf.name>
Date: Mon, 7 Sep 2015 17:10:03 +0000
Subject: [PATCH 3/3] accept-line, accept-and-hold,
 accept-and-infer-next-history: Strip trailing newline after paste.

---
 Src/Zle/zle_hist.c |  4 ++++
 Src/Zle/zle_misc.c | 25 +++++++++++++++++++++++++
 Src/utils.c        | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)

diff --git a/Src/Zle/zle_hist.c b/Src/Zle/zle_hist.c
index c61b4ef..c3d7c1a 100644
--- a/Src/Zle/zle_hist.c
+++ b/Src/Zle/zle_hist.c
@@ -1728,12 +1728,16 @@ infernexthist(Histent he, UNUSED(char **args))
     return NULL;
 }
 
+/* Defined in zle_misc.c */
+void zle_strip_trailing_newline_if_after_paste(void);
+
 /**/
 int
 acceptandinfernexthistory(char **args)
 {
     Histent he;
 
+    zle_strip_trailing_newline_if_after_paste();
     if (!(he = infernexthist(hist_ring, args)))
 	return 1;
     zpushnode(bufstack, ztrdup(he->node.nam));
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index c1005dc..dac31bf 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -395,10 +395,34 @@ poundinsert(UNUSED(char **args))
     return 0;
 }
 
+/* Declare non-static so zle_hist.c can use it */
+void zle_strip_trailing_newline_if_after_paste(void);
+
+/* If the last widget is a paste, and the last byte is a newline, strip it.
+ *
+ * If it was syntactically whitespace, that's cleaner; and if it was
+ * significant, it will be re-added latre.
+ */
+void zle_strip_trailing_newline_if_after_paste(void)
+{
+    if (lbindk == Th(z_bracketedpaste)) {
+	if (zlemetaline ? zlemetall : zlell) {
+	    char last = zlemetaline ? ztr_last(zlemetaline) : zleline[zlell-1];
+	    if (last == ZWC('\n')) {
+		/* backwarddeletechar() doesn't use ARGS, but be conservative
+		 * and pass them empty. */
+		static char *empty_args[] = { NULL };
+		backwarddeletechar(empty_args);
+	    }
+	}
+    }
+}
+
 /**/
 int
 acceptline(UNUSED(char **args))
 {
+    zle_strip_trailing_newline_if_after_paste();
     done = 1;
     return 0;
 }
@@ -407,6 +431,7 @@ acceptline(UNUSED(char **args))
 int
 acceptandhold(UNUSED(char **args))
 {
+    zle_strip_trailing_newline_if_after_paste();
     zpushnode(bufstack, zlelineasstring(zleline, zlell, 0, NULL, NULL, 0));
     stackcs = zlecs;
     done = 1;
diff --git a/Src/utils.c b/Src/utils.c
index ca68eae..325997a 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -4483,6 +4483,38 @@ unmeta(const char *file_name)
     return fn;
 }
 
+/* Return the last character of a metafied non-empty string */
+
+/**/
+mod_export char
+ztr_last(char const *s)
+{
+    char last;
+
+    if (!*s) {
+	DPUTS(1, "BUG: ztr_last() called on empty string\n");
+
+	/* We must return some value.  Hopefully NUL is good enough. */
+	return '\0';
+    }
+
+    while (*s) {
+	if (*s == Meta) {
+	    ++s;
+	    if (! *s) {
+		DPUTS(1, "BUG: unexpected end of string in ztr_last()\n");
+
+		/* We must return some value.  Hopefully NUL is good enough. */
+		return '\0';
+	    }
+	    last = *s++ ^ 32;
+	}
+	else
+	    last = *s++;
+    }
+    return last;
+}
+
 /*
  * Unmetafy and compare two strings, comparing unsigned character values.
  * "a\0" sorts after "a".
-- 
2.1.4