bracketed paste

bracketed paste

[Yuri D'Elia]

If you remember, I sent a patch for urxvt and bracketed paste in Debian
a while ago:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787628

I now realized that we could avoid all the escaping madness if we
re-define what bracketed paste should do for us. I briefly checked and
found no use in the wild of bracketed paste (scanning for the escapes in
google and github).

This made me think what I've written before in the bug report: what if
the terminal just encodes into base64 the block to be pasted? What we'd
need to just is just wait for the end sequence, decode the block, and we
have no other issues with break processing. It would even allow to send
binary content in the way (d&d in the terminal, who-ho!).

Heck, I'll write a patch to xterm myself if I'd need to push this through.

Re: bracketed paste

[Bart Schaefer]

On Nov 18,  8:57pm, Yuri D'Elia wrote:
} Subject: bracketed paste
}
} https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787628
} 
} This made me think what I've written before in the bug report: what if
} the terminal just encodes into base64 the block to be pasted?

This would at the least require a different set of start/end sequences.
(Not necessarily the ones sent from the app to the terminal to initiate
and end the mode, but the ones sent by the terminal to indicate when a
paste is happening.)  Whether you can find usages in the wild or not,
the current pair of escapes already has different semantics.

But I can't really muster enthusiasm for this idea.  It means that an
app that doesn't want to play the game can't simply throw away the
escapes and still receive sensible pasted content.  The consequences
if the mode is unintentionally activated (or remains activated, as was
happening with vim invoked from our edit-command-line widget) are at
least as unpleasant for the app and arguably worse for the user.

And it means building a base64 decoder into everything.  Yuck.

In the vast majority of cases the only problematic characters that
appear in a paste are line breaks.  Having bracketed-paste activated
by default in zsh seems only to be causing more issues than it solved.  
Having an extra keystroke to turn it on (and then automatically turn
off when the "closing bracket" is seen) might have been a better plan.

Re: bracketed paste

[Yuri D'Elia]

On 19/11/15 07:05, Bart Schaefer wrote:
> } https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787628
> } 
> } This made me think what I've written before in the bug report: what if
> 
> This would at the least require a different set of start/end sequences.
> (Not necessarily the ones sent from the app to the terminal to initiate
> and end the mode, but the ones sent by the terminal to indicate when a
> paste is happening.)  Whether you can find usages in the wild or not,
> the current pair of escapes already has different semantics.

Not a big problem IMHO.

> But I can't really muster enthusiasm for this idea.  It means that an
> app that doesn't want to play the game can't simply throw away the
> escapes and still receive sensible pasted content.  The consequences
> if the mode is unintentionally activated (or remains activated, as was
> happening with vim invoked from our edit-command-line widget) are at
> least as unpleasant for the app and arguably worse for the user.

Yes, it's something that we need to release in more cases, but I do not
expect so many weird cases popping up. Actually, I would expect the
majority to be found very soon due to the encoding being more invasive.

I'd "almost" want the proper bracketed mode being more invasive, as it
would clearly require support in the client, and require an active
interest as well.

> And it means building a base64 decoder into everything.  Yuck.

base64 is really the first thing that popped up in my mind, since it
would allow for transparent encoding of any binary also across a remote
pty. It's trivial to implement compared to more efficient encodings, and
popular enough you can just grab a prebuilt package most of the times.

But we can define our own encoding, really. Arguably, we could just
encode unwanted [8bit] sequences as an argument of a new control
sequence entirely. This would degrade much better, for example, but
deciding what gets encoded and what not would not be as clean as a flat
base64.

> In the vast majority of cases the only problematic characters that
> appear in a paste are line breaks.  Having bracketed-paste activated
> by default in zsh seems only to be causing more issues than it solved.  
> Having an extra keystroke to turn it on (and then automatically turn
> off when the "closing bracket" is seen) might have been a better plan.

For me this would be a step backwards. By fiddling with the widget I was
finally able to enable URL quoting only when pasting, I do whitespace
stripping as well, and in general I'm quite happy that I can
differentiate between the paste and typing.

In fact, I'd like the same behavior in vim, for example, so that I don't
have to enter/exit paste mode, and so that I can ensure a paste will not
exit paste mode itself.

Re: bracketed paste

[Oliver Kiddle]

Bart wrote:
> } (note that there seems to be a bug that we aren't doing unmeta() on
> } postedit so you might need the patch below for these).
> 
> Ah, I thought that was intentional to assure that the postedit string
> was output verbatim.  Are param values stored metafied?  I don't see any

I'd assume they are. var=$'one\0two' works so surely it is needed for
that.

It's a pity our function prototypes aren't explicit on this.

> unmeta() in the getsparam() call chain, which is used to retrieve the
> values of e.g. PROMPT_EOL_MARK, TMPPREFIX, EDITOR, HISTFILE, ...  Do
> all of those need unmetafying too?

TMPPREFIX+=$'\u2584'
noglob echo =(echo)
/tmp/zshâ>#â>#clEH3p
That seems to be picking up extra garbage.

hist.c around line 2721, explicitly does open(unmeta(fn), ....
but the not HAVE_SYMLINK branch in lockhistfile doesn't use unmeta.
And none of the zerr calls bother.

I think PROMPT_EOL_MARK is passed through promptexpand which I think
is fixing it. EDITOR is fine when exported. FCEDIT seems to work because
it is passed through word splitting.

Oliver

Re: bracketed paste

[Bart Schaefer]

On Jul 18,  2:05pm, Yuri D'Elia wrote:
} Subject: Re: bracketed paste
}
} On 16/07/15 22:15, Bart Schaefer wrote:
} > There are a couple of features that might be useful if we could come up
} > with a way to do them.  One would be an indication of whether bracketed
} > paste is supported at all, so that no one is taken by surprise when the
} > shell starts executing what they pasted.  Another is some indication of
} > what text was pasted (maybe highlight-color it differently?).
} 
} Is there a setting to highlight yanked text?

Could be done in a custom wrapper widget with region_highlight, I think.
Otherwise, by default, no, but perhaps we could add one.

} What makes you think that the command was sent is that the terminal
} scrolls upward and the cursor is on the last line, which looks empty.

Oliver's POSTEDIT suggestion is helpful here, but for commands that
produce only a small amount of output (not enough to cover up the
"Please wait..." text) it's suboptimal.

Re: bracketed paste

[Bart Schaefer]

On Jul 18,  4:33am, Oliver Kiddle wrote:
}
} Like Bart, I'd rather the text is not altered. Sometimes I really want
} it verbatim and I can't write a widget to unalter it. Stripping just a
} single final newline wouldn't bother me greatly, however and it would
} mostly solve to concern I described above. There may be other solutions.

I could live with having only a trailing newline removed.

} (note that there seems to be a bug that we aren't doing unmeta() on
} postedit so you might need the patch below for these).

Ah, I thought that was intentional to assure that the postedit string
was output verbatim.  Are param values stored metafied?  I don't see any
unmeta() in the getsparam() call chain, which is used to retrieve the
values of e.g. PROMPT_EOL_MARK, TMPPREFIX, EDITOR, HISTFILE, ...  Do
all of those need unmetafying too?

Re: bracketed paste

[Yuri D'Elia]

On 16/07/15 22:15, Bart Schaefer wrote:
> There are a couple of features that might be useful if we could come up
> with a way to do them.  One would be an indication of whether bracketed
> paste is supported at all, so that no one is taken by surprise when the
> shell starts executing what they pasted.  Another is some indication of
> what text was pasted (maybe highlight-color it differently?).

Is there a setting to highlight yanked text?
If such a setting exists, I would respect it in order to make a
bracketed paste equivalent to a regular paste.

I'm partly in favor of stripping a final newline. Partly because you
might still want that newline in a quote. In that sense, I still prefer
let the user implement his own widget to control the paste behavior.

Without stripping trailing whitespace itself before the newline you
still won't notice.

What makes you think that the command was sent is that the terminal
scrolls upward and the cursor is on the last line, which looks empty.

A newline followed by few spaces still tricks me.

Re: bracketed paste

[Yuri D'Elia]

On 18/07/15 04:33, Oliver Kiddle wrote:
> My concern is the average user that doesn't configure this. At the
> moment people are used to pasted newlines being accepted. If with 5.0.9,
> they paste a single line command including a trailing newline, there is
> no visible clue to tell them that things have changed: the cursor sits
> at the beginning of the next line and the user wonders for a few seconds
> why their command is taking so long.
> 
> Like Bart, I'd rather the text is not altered. Sometimes I really want
> it verbatim and I can't write a widget to unalter it. Stripping just a
> single final newline wouldn't bother me greatly, however and it would
> mostly solve to concern I described above. There may be other solutions.

I agree here, I also wouldn't want text munging by default.

> Anyway, I'm not sure that a style achieves much because it isn't hard
> to write a custom widget and a widget is more flexible. The trick of
> using a prefix key also means you can have a selection: raw, quoted,
> whitespace removed, ${(Z.C.)var} for comment removal etc.

I'm still not sure if I can catch the prefix argument when using the new
widget (?)

Re: bracketed paste

[Oliver Kiddle]

Yuri D'Elia wrote:
> I would say a better improvement would be to remove any trailing empty
> lines. Something like /(\n\s*$)+/.

> In that scenario, maybe a "paste_trimmed" style would be better. It
> avoids the kludges, and it's obvious: either you want literal paste, or
> you don't want excess whitespace around.

My concern is the average user that doesn't configure this. At the
moment people are used to pasted newlines being accepted. If with 5.0.9,
they paste a single line command including a trailing newline, there is
no visible clue to tell them that things have changed: the cursor sits
at the beginning of the next line and the user wonders for a few seconds
why their command is taking so long.

Like Bart, I'd rather the text is not altered. Sometimes I really want
it verbatim and I can't write a widget to unalter it. Stripping just a
single final newline wouldn't bother me greatly, however and it would
mostly solve to concern I described above. There may be other solutions.

For example I quite like the effect you get by putting normal characters
in POSTEDIT followed by a carriage return, some examples:
  POSTEDIT+=$' Please wait...\r'
  POSTEDIT+=$'\u2588\u231b\r'
  POSTEDIT+=$'\u2584\b'
(note that there seems to be a bug that we aren't doing unmeta() on
postedit so you might need the patch below for these).

Anyway, I'm not sure that a style achieves much because it isn't hard
to write a custom widget and a widget is more flexible. The trick of
using a prefix key also means you can have a selection: raw, quoted,
whitespace removed, ${(Z.C.)var} for comment removal etc.

Oliver

diff --git a/Src/Zle/zle_main.c b/Src/Zle/zle_main.c
index c13e3a0..e610ae1 100644
--- a/Src/Zle/zle_main.c
+++ b/Src/Zle/zle_main.c
@@ -1865,7 +1865,7 @@ trashzle(void)
 	    clearflag = listshown = 0;
 	}
 	if (postedit)
-	    fprintf(shout, "%s", postedit);
+	    fprintf(shout, "%s", unmeta(postedit));
 	fflush(shout);
 	resetneeded = 1;
 	if (!(zlereadflags & ZLRF_NOSETTY))

Re: bracketed paste

[Bart Schaefer]

On Jul 16,  7:19am, Oliver Kiddle wrote:
} Subject: Re: bracketed paste
}
} > Also, I realized sometimes a final newline was tricking me into thinking
} > that the command was already enter-ed to the shell (but of course, it
} > wasn't). Stripping any final newlines is an improvement.
} 
} Should we perhaps do this in the default widget? That is strip off the
} last character if it is a newline.

I'd rather that the text is not altered while pasting.  If this were
not a built-in widget, I'd say it'd be OK to have a zstyle for it, but
I also don't want to start having the zstyle mechanism leak into the
builtins.

There are a couple of features that might be useful if we could come up
with a way to do them.  One would be an indication of whether bracketed
paste is supported at all, so that no one is taken by surprise when the
shell starts executing what they pasted.  Another is some indication of
what text was pasted (maybe highlight-color it differently?).

Re: bracketed paste

[Yuri D'Elia]

On 16/07/15 07:19, Oliver Kiddle wrote:
> Yuri D'Elia wrote:
>>
>> Also, I realized sometimes a final newline was tricking me into thinking
>> that the command was already enter-ed to the shell (but of course, it
>> wasn't). Stripping any final newlines is an improvement.
> 
> Should we perhaps do this in the default widget? That is strip off the
> last character if it is a newline. With a default setup, it isn't
> actually possible to see the difference between the newline being
> inserted and the command being executed but doing nothing for a long
> time. In either case, the cursor position is the same.

I don't think that removing it only if it's the final character is
enough. A string like "\n " is also tricky to spot somehow.

I would say a better improvement would be to remove any trailing empty
lines. Something like /(\n\s*$)+/.

But then again, if you're pasting spaces, the cursor might also end-up
being exactly at the beginning of the new line.

It kind of depends on what are your expectations on paste. If you're
pasting spaces on purpose (for example, to concatenate more commands),
you kinda expect paste to be literal. I do this from time to time: paste
beginning from here, rest from there, or paste whitespaces in a literal
quote (yes!). Terminal-to-terminal pastes are often containing
intentional blanks and newlines. What brakes it is mostly paste from a
rich web page, where the selection is often aleatory.

I'm quite happy with the whitespace trimming on both ends. The fact that
I could do the same with the new widget is also good.

Although I'm not sure I would need to unquote the string before
stripping when a prefix argument is given: in that case my stripping
function would break, and I need to know if the argument was given or
not to perform the unquote/quote.

In that scenario, maybe a "paste_trimmed" style would be better. It
avoids the kludges, and it's obvious: either you want literal paste, or
you don't want excess whitespace around.

Re: bracketed paste

[Oliver Kiddle]

Yuri D'Elia wrote:
> 
> Also, I realized sometimes a final newline was tricking me into thinking
> that the command was already enter-ed to the shell (but of course, it
> wasn't). Stripping any final newlines is an improvement.

Should we perhaps do this in the default widget? That is strip off the
last character if it is a newline. With a default setup, it isn't
actually possible to see the difference between the newline being
inserted and the command being executed but doing nothing for a long
time. In either case, the cursor position is the same.

Oliver

Re: bracketed paste

[Yuri D'Elia]

On 15/07/15 20:15, Mikael Magnusson wrote:
>> And yes, that whitespace trimming looks ugly.
>> Is there a better way?
> 
> Here's my local .zshrc diff that adapts to the new widget, if that

Sending a diff or .zshrc doesn't make it easy to analyze, but looks
overall *much* better. All the plumbing's gone.

> helps, and yeah, you'll have to wait for 5.0.9.

Also, I still didn't try to implement that fix for handling SIGINT
directly into urxvt.

One of these days.. after 5.0.9 is out ;)

Re: bracketed paste

[Mikael Magnusson]

On Wed, Jul 15, 2015 at 7:33 PM, Yuri D'Elia <wavexx@thregr.org> wrote:
> On 15/07/15 18:51, Yuri D'Elia wrote:
>> I'd like to do the same using the final widget if possible.
>
> To give an example, if you still remember the code I posted earlier:
>
> function _end_paste() {
>   bindkey -e
>   setopt localoptions extendedglob
>   _paste_content=${${_paste_content%%[[:space:]]#}##[[:space:]]#}
>   [[ $_paste_quoted == 1 ]] && LBUFFER+=${(q)_paste_content} || LBUFFER+=$_paste_content
>   unset _paste_content _paste_quoted
> }
>
> And yes, that whitespace trimming looks ugly.
> Is there a better way?

Here's my local .zshrc diff that adapts to the new widget, if that
helps, and yeah, you'll have to wait for 5.0.9.

diff --git a/.zshrc b/.zshrc
index 7fc4091..b972f40 100644
--- a/.zshrc
+++ b/.zshrc
@@ -443,9 +443,7 @@ zle -N _hextochar
 zle -N _chartohex
 zle -N zle-line-init _zle_line_init
 zle -N zle-line-finish _zle_line_finish
-zle -N _start_paste
-zle -N _end_paste
-zle -N paste-insert _paste_insert
+zle -N bracketed-paste _start_paste
 zle -N _spaceafterpastequote
 zle -N self-insert-unmeta-right _self_insert_unmeta_right
 zle -N overwrite-mode _overwrite_mode
@@ -547,13 +545,6 @@ bindkey "^[j"  backward-char
 bindkey "^[k"  forward-char
 bindkey "^[ "  self-insert-unmeta-right

-# Hooks
-bindkey -N paste
-bindkey -R -M paste "^@"-"\M-^?" paste-insert
-bindkey '^[[200~' _start_paste
-bindkey -M paste '^[[201~' _end_paste
-bindkey -M paste -s '^M' '^J'
-
 # Unbind defaults that conflict with keychains
 bindkey -r "^_"

diff --git a/hooks.zsh b/hooks.zsh
index 5916774..d579aa3 100644
--- a/hooks.zsh
+++ b/hooks.zsh
@@ -486,8 +486,6 @@ function _zle_line_init() {
   zle set-local-history -n 1
   function _zle_line_init() {
     zle -N zle-line-pre-redraw _line_redraw
-    # Tell urxvt to send escape codes around a paste
-    [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004h'
     bindkey -e
   }
   _zle_line_init
@@ -495,8 +493,6 @@ function _zle_line_init() {

 function _zle_line_finish() {
   [[ $CONTEXT = vared ]] || ZLE_LINE_ABORTED=$BUFFER
-  # Tell it to stop
-  [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004l'
 }

 function _recover_line_or_else() {
@@ -510,18 +506,13 @@ function _recover_line_or_else() {
 zle -N up-history _recover_line_or_else

 function _start_paste() {
-  unset _paste_content
-  bindkey -A paste main
-}
-
-function _end_paste() {
-  bindkey -e
+  local content
+  zle .$WIDGET -N content
   if [[ $_SPACE_AFTER_PASTE_QUOTE = 1 ]]; then
-    LBUFFER+=${(q-)_paste_content}' '
+    LBUFFER+=${(q-)content}' '
   else
-    LBUFFER+=$_paste_content
+    LBUFFER+=$content
   fi
-  unset _paste_content
 }

 function _spaceafterpastequote() {
@@ -721,10 +712,6 @@ function _chartohex() {
   (( CURSOR = pos + $#new - 1 ))
 }

-function _paste_insert() {
-  _paste_content+=$KEYS
-}
-

-- 
Mikael Magnusson

Re: bracketed paste

[Yuri D'Elia]

On 15/07/15 18:51, Yuri D'Elia wrote:
> I'd like to do the same using the final widget if possible.

To give an example, if you still remember the code I posted earlier:

function _end_paste() {
  bindkey -e
  setopt localoptions extendedglob
  _paste_content=${${_paste_content%%[[:space:]]#}##[[:space:]]#}
  [[ $_paste_quoted == 1 ]] && LBUFFER+=${(q)_paste_content} || LBUFFER+=$_paste_content
  unset _paste_content _paste_quoted
}

And yes, that whitespace trimming looks ugly.
Is there a better way?

bracketed paste

[Yuri D'Elia]

Did the bracketedpaste widget made into 5.0.8?

The detailed release notes: http://zsh.sourceforge.net/releases.html are
missing, and I'm too lazy to diff ;)

In my current code, I decided to strip any blank from the string before
quoting and inserting into command line.

This has the effect of removing any leading/trailing newlines (or gobs
of whitespace, if you're pasting from a web page) without destroying the
content.

Also, I realized sometimes a final newline was tricking me into thinking
that the command was already enter-ed to the shell (but of course, it
wasn't). Stripping any final newlines is an improvement.

I'd like to do the same using the final widget if possible.

Re: bracketed paste

[Mikael Magnusson]

On 7 November 2011 23:31, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Has anyone tried enabling the bracketed paste feature of xterm?

I have this for urxvt (now, this is a bit spread out in my config, so
I might miss some bits here, hope not).

bindkey -N paste
bindkey -R -M paste "^@"-"\M-^?" paste-insert
bindkey '^[[200~' _start_paste
bindkey -M paste '^[[201~' _end_paste
bindkey -M paste -s '^M' '^J'

function _zle_line_init() {
  # Tell urxvt to send escape codes around a paste
  [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004h'
}

function _zle_line_finish() {
  # Tell it to stop
  [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004l'
}

function _start_paste() {
  bindkey -A paste main
}

function _end_paste() {
  bindkey -e
  if [[ $_SPACE_AFTER_PASTE_QUOTE = 1 ]]; then
    LBUFFER+=${(q)_paste_content}' '
  else
    LBUFFER+=$_paste_content
  fi
  unset _paste_content
}

function _spaceafterpastequote() {
  if [[ $_SPACE_AFTER_PASTE_QUOTE = 1 ]]; then
    _SPACE_AFTER_PASTE_QUOTE=0
    zle -M "Not inserting a space after pastes, not quoting"
  else
    _SPACE_AFTER_PASTE_QUOTE=1
    zle -M "Inserting a space after pastes and quoting"
  fi
}

function _paste_insert() {
  _paste_content+=$KEYS
}

zle -N zle-line-init _zle_line_init
zle -N zle-line-finish _zle_line_finish
zle -N _start_paste
zle -N _end_paste
zle -N paste-insert _paste_insert
zle -N _spaceafterpastequote

-- 
Mikael Magnusson

bracketed paste

[Oliver Kiddle]

Has anyone tried enabling the bracketed paste feature of xterm?

Basically, when you paste text it is preceded by \e[200~ and terminated
with \e[201~. This has the potential to solve some irritations with
paste: undo can undo all the pasted text in one go, pasted tab
characters won't invoke completion and in vi command mode, pasted text
mightn't be treated as vi commands.

To handle this, what I've tried so far is putting \e[?2004h at the end
of PS1 and PS2 to enable it and \e[?2004l in POSTEDIT so it doesn't
remain enabled for other software. I then put together the following
zle widget which is, of course, bound to \e[200~.

bracketed-paste() { 
 local REPLY paste
 local end=$'\e[201~' idx=1
 while (( idx <= $#end )) && read -k 1; do
   paste+="${REPLY/$'\r'/
}"
   [[ $REPLY = $end[idx++] ]] || (( idx = 1 + (#REPLY == #end) ))
 done
 if (( NUMERIC )); then
   LBUFFER+="${(q)paste%$end}"
 else
   LBUFFER+="${paste%$end}"
 fi
}

The main problem I've found is that you can't paste into the mini-
buffer. Is there any way around that?

Does it make sense to include this function in Functions/Zle? It would
still require people to enable bracketed paste in their xterm. Any other
thoughts on what you might do with the pasted text besides pasting it.
$LASTWIDGET might be useful here. The function above quotes the text if
you supply a numeric argument. This can be useful when pasting, e.g.
URLs. In the past, I've used a separate widget that tries to get the
buffer with xclip, xselection or xprop.

Oliver