bracketed paste

bracketed paste

[Yuri D'Elia]

Did the bracketedpaste widget made into 5.0.8?

The detailed release notes: http://zsh.sourceforge.net/releases.html are
missing, and I'm too lazy to diff ;)

In my current code, I decided to strip any blank from the string before
quoting and inserting into command line.

This has the effect of removing any leading/trailing newlines (or gobs
of whitespace, if you're pasting from a web page) without destroying the
content.

Also, I realized sometimes a final newline was tricking me into thinking
that the command was already enter-ed to the shell (but of course, it
wasn't). Stripping any final newlines is an improvement.

I'd like to do the same using the final widget if possible.

Re: bracketed paste

[Yuri D'Elia]

On 15/07/15 18:51, Yuri D'Elia wrote:
> I'd like to do the same using the final widget if possible.

To give an example, if you still remember the code I posted earlier:

function _end_paste() {
  bindkey -e
  setopt localoptions extendedglob
  _paste_content=${${_paste_content%%[[:space:]]#}##[[:space:]]#}
  [[ $_paste_quoted == 1 ]] && LBUFFER+=${(q)_paste_content} || LBUFFER+=$_paste_content
  unset _paste_content _paste_quoted
}

And yes, that whitespace trimming looks ugly.
Is there a better way?

Re: bracketed paste

[Mikael Magnusson]

On Wed, Jul 15, 2015 at 7:33 PM, Yuri D'Elia <wavexx@thregr.org> wrote:
> On 15/07/15 18:51, Yuri D'Elia wrote:
>> I'd like to do the same using the final widget if possible.
>
> To give an example, if you still remember the code I posted earlier:
>
> function _end_paste() {
>   bindkey -e
>   setopt localoptions extendedglob
>   _paste_content=${${_paste_content%%[[:space:]]#}##[[:space:]]#}
>   [[ $_paste_quoted == 1 ]] && LBUFFER+=${(q)_paste_content} || LBUFFER+=$_paste_content
>   unset _paste_content _paste_quoted
> }
>
> And yes, that whitespace trimming looks ugly.
> Is there a better way?

Here's my local .zshrc diff that adapts to the new widget, if that
helps, and yeah, you'll have to wait for 5.0.9.

diff --git a/.zshrc b/.zshrc
index 7fc4091..b972f40 100644
--- a/.zshrc
+++ b/.zshrc
@@ -443,9 +443,7 @@ zle -N _hextochar
 zle -N _chartohex
 zle -N zle-line-init _zle_line_init
 zle -N zle-line-finish _zle_line_finish
-zle -N _start_paste
-zle -N _end_paste
-zle -N paste-insert _paste_insert
+zle -N bracketed-paste _start_paste
 zle -N _spaceafterpastequote
 zle -N self-insert-unmeta-right _self_insert_unmeta_right
 zle -N overwrite-mode _overwrite_mode
@@ -547,13 +545,6 @@ bindkey "^[j"  backward-char
 bindkey "^[k"  forward-char
 bindkey "^[ "  self-insert-unmeta-right

-# Hooks
-bindkey -N paste
-bindkey -R -M paste "^@"-"\M-^?" paste-insert
-bindkey '^[[200~' _start_paste
-bindkey -M paste '^[[201~' _end_paste
-bindkey -M paste -s '^M' '^J'
-
 # Unbind defaults that conflict with keychains
 bindkey -r "^_"

diff --git a/hooks.zsh b/hooks.zsh
index 5916774..d579aa3 100644
--- a/hooks.zsh
+++ b/hooks.zsh
@@ -486,8 +486,6 @@ function _zle_line_init() {
   zle set-local-history -n 1
   function _zle_line_init() {
     zle -N zle-line-pre-redraw _line_redraw
-    # Tell urxvt to send escape codes around a paste
-    [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004h'
     bindkey -e
   }
   _zle_line_init
@@ -495,8 +493,6 @@ function _zle_line_init() {

 function _zle_line_finish() {
   [[ $CONTEXT = vared ]] || ZLE_LINE_ABORTED=$BUFFER
-  # Tell it to stop
-  [[ $TERM == rxvt-unicode || $TERM = xterm ]] && printf '\e[?2004l'
 }

 function _recover_line_or_else() {
@@ -510,18 +506,13 @@ function _recover_line_or_else() {
 zle -N up-history _recover_line_or_else

 function _start_paste() {
-  unset _paste_content
-  bindkey -A paste main
-}
-
-function _end_paste() {
-  bindkey -e
+  local content
+  zle .$WIDGET -N content
   if [[ $_SPACE_AFTER_PASTE_QUOTE = 1 ]]; then
-    LBUFFER+=${(q-)_paste_content}' '
+    LBUFFER+=${(q-)content}' '
   else
-    LBUFFER+=$_paste_content
+    LBUFFER+=$content
   fi
-  unset _paste_content
 }

 function _spaceafterpastequote() {
@@ -721,10 +712,6 @@ function _chartohex() {
   (( CURSOR = pos + $#new - 1 ))
 }

-function _paste_insert() {
-  _paste_content+=$KEYS
-}
-

-- 
Mikael Magnusson

Re: bracketed paste

[Yuri D'Elia]

On 15/07/15 20:15, Mikael Magnusson wrote:
>> And yes, that whitespace trimming looks ugly.
>> Is there a better way?
> 
> Here's my local .zshrc diff that adapts to the new widget, if that

Sending a diff or .zshrc doesn't make it easy to analyze, but looks
overall *much* better. All the plumbing's gone.

> helps, and yeah, you'll have to wait for 5.0.9.

Also, I still didn't try to implement that fix for handling SIGINT
directly into urxvt.

One of these days.. after 5.0.9 is out ;)

Re: bracketed paste

[Oliver Kiddle]

Yuri D'Elia wrote:
> 
> Also, I realized sometimes a final newline was tricking me into thinking
> that the command was already enter-ed to the shell (but of course, it
> wasn't). Stripping any final newlines is an improvement.

Should we perhaps do this in the default widget? That is strip off the
last character if it is a newline. With a default setup, it isn't
actually possible to see the difference between the newline being
inserted and the command being executed but doing nothing for a long
time. In either case, the cursor position is the same.

Oliver

Re: bracketed paste

[Yuri D'Elia]

On 16/07/15 07:19, Oliver Kiddle wrote:
> Yuri D'Elia wrote:
>>
>> Also, I realized sometimes a final newline was tricking me into thinking
>> that the command was already enter-ed to the shell (but of course, it
>> wasn't). Stripping any final newlines is an improvement.
> 
> Should we perhaps do this in the default widget? That is strip off the
> last character if it is a newline. With a default setup, it isn't
> actually possible to see the difference between the newline being
> inserted and the command being executed but doing nothing for a long
> time. In either case, the cursor position is the same.

I don't think that removing it only if it's the final character is
enough. A string like "\n " is also tricky to spot somehow.

I would say a better improvement would be to remove any trailing empty
lines. Something like /(\n\s*$)+/.

But then again, if you're pasting spaces, the cursor might also end-up
being exactly at the beginning of the new line.

It kind of depends on what are your expectations on paste. If you're
pasting spaces on purpose (for example, to concatenate more commands),
you kinda expect paste to be literal. I do this from time to time: paste
beginning from here, rest from there, or paste whitespaces in a literal
quote (yes!). Terminal-to-terminal pastes are often containing
intentional blanks and newlines. What brakes it is mostly paste from a
rich web page, where the selection is often aleatory.

I'm quite happy with the whitespace trimming on both ends. The fact that
I could do the same with the new widget is also good.

Although I'm not sure I would need to unquote the string before
stripping when a prefix argument is given: in that case my stripping
function would break, and I need to know if the argument was given or
not to perform the unquote/quote.

In that scenario, maybe a "paste_trimmed" style would be better. It
avoids the kludges, and it's obvious: either you want literal paste, or
you don't want excess whitespace around.

Re: bracketed paste

[Bart Schaefer]

On Jul 16,  7:19am, Oliver Kiddle wrote:
} Subject: Re: bracketed paste
}
} > Also, I realized sometimes a final newline was tricking me into thinking
} > that the command was already enter-ed to the shell (but of course, it
} > wasn't). Stripping any final newlines is an improvement.
} 
} Should we perhaps do this in the default widget? That is strip off the
} last character if it is a newline.

I'd rather that the text is not altered while pasting.  If this were
not a built-in widget, I'd say it'd be OK to have a zstyle for it, but
I also don't want to start having the zstyle mechanism leak into the
builtins.

There are a couple of features that might be useful if we could come up
with a way to do them.  One would be an indication of whether bracketed
paste is supported at all, so that no one is taken by surprise when the
shell starts executing what they pasted.  Another is some indication of
what text was pasted (maybe highlight-color it differently?).

Re: bracketed paste

[Oliver Kiddle]

Yuri D'Elia wrote:
> I would say a better improvement would be to remove any trailing empty
> lines. Something like /(\n\s*$)+/.

> In that scenario, maybe a "paste_trimmed" style would be better. It
> avoids the kludges, and it's obvious: either you want literal paste, or
> you don't want excess whitespace around.

My concern is the average user that doesn't configure this. At the
moment people are used to pasted newlines being accepted. If with 5.0.9,
they paste a single line command including a trailing newline, there is
no visible clue to tell them that things have changed: the cursor sits
at the beginning of the next line and the user wonders for a few seconds
why their command is taking so long.

Like Bart, I'd rather the text is not altered. Sometimes I really want
it verbatim and I can't write a widget to unalter it. Stripping just a
single final newline wouldn't bother me greatly, however and it would
mostly solve to concern I described above. There may be other solutions.

For example I quite like the effect you get by putting normal characters
in POSTEDIT followed by a carriage return, some examples:
  POSTEDIT+=$' Please wait...\r'
  POSTEDIT+=$'\u2588\u231b\r'
  POSTEDIT+=$'\u2584\b'
(note that there seems to be a bug that we aren't doing unmeta() on
postedit so you might need the patch below for these).

Anyway, I'm not sure that a style achieves much because it isn't hard
to write a custom widget and a widget is more flexible. The trick of
using a prefix key also means you can have a selection: raw, quoted,
whitespace removed, ${(Z.C.)var} for comment removal etc.

Oliver

diff --git a/Src/Zle/zle_main.c b/Src/Zle/zle_main.c
index c13e3a0..e610ae1 100644
--- a/Src/Zle/zle_main.c
+++ b/Src/Zle/zle_main.c
@@ -1865,7 +1865,7 @@ trashzle(void)
 	    clearflag = listshown = 0;
 	}
 	if (postedit)
-	    fprintf(shout, "%s", postedit);
+	    fprintf(shout, "%s", unmeta(postedit));
 	fflush(shout);
 	resetneeded = 1;
 	if (!(zlereadflags & ZLRF_NOSETTY))

Re: bracketed paste

[Yuri D'Elia]

On 18/07/15 04:33, Oliver Kiddle wrote:
> My concern is the average user that doesn't configure this. At the
> moment people are used to pasted newlines being accepted. If with 5.0.9,
> they paste a single line command including a trailing newline, there is
> no visible clue to tell them that things have changed: the cursor sits
> at the beginning of the next line and the user wonders for a few seconds
> why their command is taking so long.
> 
> Like Bart, I'd rather the text is not altered. Sometimes I really want
> it verbatim and I can't write a widget to unalter it. Stripping just a
> single final newline wouldn't bother me greatly, however and it would
> mostly solve to concern I described above. There may be other solutions.

I agree here, I also wouldn't want text munging by default.

> Anyway, I'm not sure that a style achieves much because it isn't hard
> to write a custom widget and a widget is more flexible. The trick of
> using a prefix key also means you can have a selection: raw, quoted,
> whitespace removed, ${(Z.C.)var} for comment removal etc.

I'm still not sure if I can catch the prefix argument when using the new
widget (?)

Re: bracketed paste

[Yuri D'Elia]

On 16/07/15 22:15, Bart Schaefer wrote:
> There are a couple of features that might be useful if we could come up
> with a way to do them.  One would be an indication of whether bracketed
> paste is supported at all, so that no one is taken by surprise when the
> shell starts executing what they pasted.  Another is some indication of
> what text was pasted (maybe highlight-color it differently?).

Is there a setting to highlight yanked text?
If such a setting exists, I would respect it in order to make a
bracketed paste equivalent to a regular paste.

I'm partly in favor of stripping a final newline. Partly because you
might still want that newline in a quote. In that sense, I still prefer
let the user implement his own widget to control the paste behavior.

Without stripping trailing whitespace itself before the newline you
still won't notice.

What makes you think that the command was sent is that the terminal
scrolls upward and the cursor is on the last line, which looks empty.

A newline followed by few spaces still tricks me.

Re: bracketed paste

[Bart Schaefer]

On Jul 18,  4:33am, Oliver Kiddle wrote:
}
} Like Bart, I'd rather the text is not altered. Sometimes I really want
} it verbatim and I can't write a widget to unalter it. Stripping just a
} single final newline wouldn't bother me greatly, however and it would
} mostly solve to concern I described above. There may be other solutions.

I could live with having only a trailing newline removed.

} (note that there seems to be a bug that we aren't doing unmeta() on
} postedit so you might need the patch below for these).

Ah, I thought that was intentional to assure that the postedit string
was output verbatim.  Are param values stored metafied?  I don't see any
unmeta() in the getsparam() call chain, which is used to retrieve the
values of e.g. PROMPT_EOL_MARK, TMPPREFIX, EDITOR, HISTFILE, ...  Do
all of those need unmetafying too?

Re: bracketed paste

[Bart Schaefer]

On Jul 18,  2:05pm, Yuri D'Elia wrote:
} Subject: Re: bracketed paste
}
} On 16/07/15 22:15, Bart Schaefer wrote:
} > There are a couple of features that might be useful if we could come up
} > with a way to do them.  One would be an indication of whether bracketed
} > paste is supported at all, so that no one is taken by surprise when the
} > shell starts executing what they pasted.  Another is some indication of
} > what text was pasted (maybe highlight-color it differently?).
} 
} Is there a setting to highlight yanked text?

Could be done in a custom wrapper widget with region_highlight, I think.
Otherwise, by default, no, but perhaps we could add one.

} What makes you think that the command was sent is that the terminal
} scrolls upward and the cursor is on the last line, which looks empty.

Oliver's POSTEDIT suggestion is helpful here, but for commands that
produce only a small amount of output (not enough to cover up the
"Please wait..." text) it's suboptimal.

Re: bracketed paste

[Oliver Kiddle]

Bart wrote:
> } (note that there seems to be a bug that we aren't doing unmeta() on
> } postedit so you might need the patch below for these).
> 
> Ah, I thought that was intentional to assure that the postedit string
> was output verbatim.  Are param values stored metafied?  I don't see any

I'd assume they are. var=$'one\0two' works so surely it is needed for
that.

It's a pity our function prototypes aren't explicit on this.

> unmeta() in the getsparam() call chain, which is used to retrieve the
> values of e.g. PROMPT_EOL_MARK, TMPPREFIX, EDITOR, HISTFILE, ...  Do
> all of those need unmetafying too?

TMPPREFIX+=$'\u2584'
noglob echo =(echo)
/tmp/zshâ>#â>#clEH3p
That seems to be picking up extra garbage.

hist.c around line 2721, explicitly does open(unmeta(fn), ....
but the not HAVE_SYMLINK branch in lockhistfile doesn't use unmeta.
And none of the zerr calls bother.

I think PROMPT_EOL_MARK is passed through promptexpand which I think
is fixing it. EDITOR is fine when exported. FCEDIT seems to work because
it is passed through word splitting.

Oliver

PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> There are a couple of features that might be useful if we could come up
> with a way to do them.  One would be an indication of whether bracketed
> paste is supported at all, so that no one is taken by surprise when the

That's not possible as far as I know. If it was possible to detect that,
we'd be able to make the sending of the start/end strings conditional.

> shell starts executing what they pasted.  Another is some indication of
> what text was pasted (maybe highlight-color it differently?).

How about the following patch? This adds another special token to
zle_highlight. So you can do, e.g: zle_highlight+=( paste:bg=87 )
This applies also for vi put and emacs yank commands.

We were already tracking start/end positions of a paste for the purposes
of the yank-pop widget. Using those, leads to some further changes:

Text from a bracketed paste is now added to the cutbuffer. As far as
I can tell, this seems to actually also be what the GUI mode of emacs
does. Though emacs is perhaps not putting duplicates in. It'd be good if
some actual emacs users could check that it all makes sense, however.

I had been pondering how to interface bracketed-paste with vi registers,
anyway. This will now also put the bracketed paste in the numbered
registers. If you explicitly name a vi register, the pasted text is now
NOT inserted: it is only assigned to the register. This seems quite
useful if not the most vim compatible. For a middle-mouse click after
naming a register, gvim will paste the named register at the position
under the mouse.

Oliver

diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
index da8ee47..ef73f4d 100644
--- a/Doc/Zsh/zle.yo
+++ b/Doc/Zsh/zle.yo
@@ -2510,6 +2510,9 @@ a directory name.  Note that suffix removal is configurable; the
 circumstances under which the suffix will be removed may differ
 for different completions.
 )
+item(tt(paste))(
+Following a command to paste text, the characters that were inserted.
+)
 enditem()
 
 tt(zle_highlight) may contain additional fields for controlling how
diff --git a/Src/Zle/iwidgets.list b/Src/Zle/iwidgets.list
index 6a07212..ebcf317 100644
--- a/Src/Zle/iwidgets.list
+++ b/Src/Zle/iwidgets.list
@@ -28,7 +28,7 @@
 "beginning-of-history", beginningofhistory, 0
 "beginning-of-line", beginningofline, 0
 "beginning-of-line-hist", beginningoflinehist, 0
-"bracketed-paste", bracketedpaste, ZLE_MENUCMP | ZLE_KEEPSUFFIX
+"bracketed-paste", bracketedpaste, ZLE_MENUCMP | ZLE_KEEPSUFFIX | ZLE_YANKBEFORE
 "capitalize-word", capitalizeword, 0
 "clear-screen", clearscreen, ZLE_MENUCMP | ZLE_KEEPSUFFIX | ZLE_LASTCOL | ZLE_NOTCOMMAND
 "complete-word", completeword, ZLE_MENUCMP | ZLE_KEEPSUFFIX | ZLE_ISCOMP
diff --git a/Src/Zle/zle.h b/Src/Zle/zle.h
index ab2428e..59f4591 100644
--- a/Src/Zle/zle.h
+++ b/Src/Zle/zle.h
@@ -429,8 +429,9 @@ struct region_highlight {
  * 0: region between point and mark
  * 1: isearch region
  * 2: suffix
+ * 3: pasted text
  */
-#define N_SPECIAL_HIGHLIGHTS	(3)
+#define N_SPECIAL_HIGHLIGHTS	(4)
 
 
 #ifdef MULTIBYTE_SUPPORT
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index d350688..b040b97 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -517,10 +517,12 @@ copyregionaskill(char **args)
 
 /*
  * kct: index into kill ring, or -1 for original cutbuffer of yank.
- * yankb, yanke: mark the start and end of last yank in editing buffer.
  * yankcs marks the cursor position preceding the last yank
  */
-static int kct, yankb, yanke, yankcs;
+static int kct, yankcs;
+
+/**/
+int yankb, yanke; /* mark the start and end of last yank in editing buffer. */
 
 /* The original cutbuffer, either cutbuf or one of the vi buffers. */
 static Cutbuffer kctbuf;
@@ -778,10 +780,17 @@ bracketedpaste(char **args)
 	ZLE_STRING_T wpaste;
 	wpaste = stringaszleline((zmult == 1) ? pbuf :
 	    quotestring(pbuf, NULL, QT_BACKSLASH), 0, &n, NULL, NULL);
-	zmult = 1;
-	if (region_active)
-	    killregion(zlenoargs);
-	doinsert(wpaste, n);
+	cuttext(wpaste, n, CUT_REPLACE);
+	if (!(zmod.flags & MOD_VIBUF)) {
+	    kct = -1;
+	    kctbuf = &cutbuf;
+	    zmult = 1;
+	    if (region_active)
+		killregion(zlenoargs);
+	    yankcs = yankb = zlecs;
+	    doinsert(wpaste, n);
+	    yanke = zlecs;
+	}
 	free(pbuf); free(wpaste);
     }
     return 0;
diff --git a/Src/Zle/zle_refresh.c b/Src/Zle/zle_refresh.c
index fe33799..5b3b89b 100644
--- a/Src/Zle/zle_refresh.c
+++ b/Src/Zle/zle_refresh.c
@@ -318,6 +318,7 @@ zle_set_highlight(void)
     int region_atr_on_set = 0;
     int isearch_atr_on_set = 0;
     int suffix_atr_on_set = 0;
+    int paste_atr_on_set = 0;
     struct region_highlight *rhp;
 
     special_atr_on = default_atr_on = 0;
@@ -337,7 +338,8 @@ zle_set_highlight(void)
 	for (; *atrs; atrs++) {
 	    if (!strcmp(*atrs, "none")) {
 		/* reset attributes for consistency... usually unnecessary */
-		special_atr_on = default_atr_on = 0;
+		special_atr_on = default_atr_on =
+		    paste_atr_on_set = 0;
 		special_atr_on_set = region_atr_on_set =
 		    isearch_atr_on_set = suffix_atr_on_set = 1;
 	    } else if (strpfx("default:", *atrs)) {
@@ -354,6 +356,9 @@ zle_set_highlight(void)
 	    } else if (strpfx("suffix:", *atrs)) {
 		match_highlight(*atrs + 7, &(region_highlights[2].atr));
 		suffix_atr_on_set = 1;
+	    } else if (strpfx("paste:", *atrs)) {
+		match_highlight(*atrs + 6, &(region_highlights[3].atr));
+		paste_atr_on_set = 1;
 	    }
 	}
     }
@@ -367,6 +372,7 @@ zle_set_highlight(void)
 	region_highlights[1].atr = TXTUNDERLINE;
     if (!suffix_atr_on_set)
 	region_highlights[2].atr = TXTBOLDFACE;
+        /* paste defaults to 0 */
 
     allocate_colour_buffer();
 }
@@ -1073,6 +1079,14 @@ zrefresh(void)
 	region_highlights[2].start = region_highlights[2].end = -1;
     }
 
+    if (lastcmd & ZLE_YANK) {
+	region_highlights[3].start = yankb;
+	region_highlights[3].end = yanke;
+    } else {
+	region_highlights[3].start = region_highlights[3].end = -1;
+    }
+
+
     if (clearlist && listshown > 0) {
 	if (tccan(TCCLEAREOD)) {
 	    int ovln = vln, ovcs = vcs;

Unmetafy of getsparam()

[Bart Schaefer]

On Jul 19,  1:28am, Oliver Kiddle wrote:
}
} Bart wrote:
} > unmeta() in the getsparam() call chain, which is used to retrieve the
} > values of e.g. PROMPT_EOL_MARK, TMPPREFIX, EDITOR, HISTFILE, ...  Do
} > all of those need unmetafying too?
} 
} TMPPREFIX+=$'\u2584'
} noglob echo =(echo)
} /tmp/zsha>#a>#clEH3p
} That seems to be picking up extra garbage.
} 
} hist.c around line 2721, explicitly does open(unmeta(fn), ....
} but the not HAVE_SYMLINK branch in lockhistfile doesn't use unmeta.
} And none of the zerr calls bother.

Here's a patch that attempts to clean up [un]metafication of non-special
parameter values used internally to the shell.  Well, some of them.

I'm unsure what to do with:

- ENV in init.c -- for parsestr() is a tokenized string also metafied?
- untok_and_escape() in subst.c
- spckword() in utils.c -- patcompile expects a metafied string?
- all of hist.c is a mess with respect to handling of HISTFILE
- getcvar() in math.c
- savesession() in zftp.c
- LISTPROMPT in complistmatches()
- MENUSELECT, MENUSCROLL, MENUMODE, MENUPROMPT in domenuselect()

Specifically with respect to hist.c, it's the caller of gettempfile()
and by extension gettempname() so if metafication changes there it
may need to follow through into changes in utils.c.

Please check my work.  The unmeta() calls in argumets of setlang() et al.
are probably superfluous since it's doubtful that any valid LC_* or LANG
string is going to need metafication.


diff --git a/Src/Modules/newuser.c b/Src/Modules/newuser.c
index 71902da..efdb2ab 100644
--- a/Src/Modules/newuser.c
+++ b/Src/Modules/newuser.c
@@ -67,7 +67,7 @@ check_dotfile(const char *dotdir, const char *fname)
 int
 boot_(UNUSED(Module m))
 {
-    const char *dotdir = getsparam("ZDOTDIR");
+    const char *dotdir = getsparam_u("ZDOTDIR");
     const char *spaths[] = {
 #ifdef SITESCRIPT_DIR
 	SITESCRIPT_DIR,
diff --git a/Src/Modules/zftp.c b/Src/Modules/zftp.c
index 09d4bd7..30f5176 100644
--- a/Src/Modules/zftp.c
+++ b/Src/Modules/zftp.c
@@ -731,7 +731,7 @@ zfgetmsg(void)
     stopit = (*ptr++ != '-');
 
     queue_signals();
-    if (!(verbose = getsparam("ZFTP_VERBOSE")))
+    if (!(verbose = getsparam_u("ZFTP_VERBOSE")))
 	verbose = "";
     if (strchr(verbose, lastcodestr[0])) {
 	/* print the whole thing verbatim */
@@ -1785,7 +1785,7 @@ zftp_open(char *name, char **args, int flags)
 	char *hname;
 	alarm(0);
 	queue_signals();
-	if ((hname = getsparam("ZFTP_HOST")) && *hname) 
+	if ((hname = getsparam_u("ZFTP_HOST")) && *hname) 
 	    zwarnnam(name, "timeout connecting to %s", hname);
 	else
 	    zwarnnam(name, "timeout on host name lookup");
@@ -3077,7 +3077,7 @@ bin_zftp(char *name, char **args, UNUSED(Options ops), UNUSED(int func))
     }
 
     queue_signals();
-    if ((prefs = getsparam("ZFTP_PREFS"))) {
+    if ((prefs = getsparam_u("ZFTP_PREFS"))) {
 	zfprefs = 0;
 	for (ptr = prefs; *ptr; ptr++) {
 	    switch (toupper(STOUC(*ptr))) {
diff --git a/Src/Zle/complist.c b/Src/Zle/complist.c
index f37a432..fd90ccb 100644
--- a/Src/Zle/complist.c
+++ b/Src/Zle/complist.c
@@ -507,8 +507,8 @@ getcols()
     max_caplen = lr_caplen = 0;
     mcolors.flags = 0;
     queue_signals();
-    if (!(s = getsparam("ZLS_COLORS")) &&
-	!(s = getsparam("ZLS_COLOURS"))) {
+    if (!(s = getsparam_u("ZLS_COLORS")) &&
+	!(s = getsparam_u("ZLS_COLOURS"))) {
 	for (i = 0; i < NUM_COLS; i++)
 	    mcolors.files[i] = filecol("");
 	mcolors.pats = NULL;
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index d350688..556ce5b 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -1552,13 +1552,13 @@ makesuffix(int n)
 {
     char *suffixchars;
 
-    if (!(suffixchars = getsparam("ZLE_REMOVE_SUFFIX_CHARS")))
+    if (!(suffixchars = getsparam_u("ZLE_REMOVE_SUFFIX_CHARS")))
 	suffixchars = " \t\n;&|";
 
     addsuffixstring(SUFTYP_POSSTR, 0, suffixchars, n);
 
     /* Do this second so it takes precedence */
-    if ((suffixchars = getsparam("ZLE_SPACE_SUFFIX_CHARS")) && *suffixchars)
+    if ((suffixchars = getsparam_u("ZLE_SPACE_SUFFIX_CHARS")) && *suffixchars)
 	addsuffixstring(SUFTYP_POSSTR, SUFFLAGS_SPACE, suffixchars, n);
 
     suffixlen = n;
diff --git a/Src/init.c b/Src/init.c
index 0fe4d75..2ef9099 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -1426,7 +1426,7 @@ sourcehome(char *s)
     char *h;
 
     queue_signals();
-    if (EMULATION(EMULATE_SH|EMULATE_KSH) || !(h = getsparam("ZDOTDIR"))) {
+    if (EMULATION(EMULATE_SH|EMULATE_KSH) || !(h = getsparam_u("ZDOTDIR"))) {
 	h = home;
 	if (!h)
 	    return;
diff --git a/Src/params.c b/Src/params.c
index 7d0c852..efbe92b 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -2639,6 +2639,15 @@ getsparam(char *s)
     return getstrvalue(v);
 }
 
+/**/
+mod_export char *
+getsparam_u(char *s)
+{
+    if ((s = getsparam(s)))
+	return unmetafy(s, NULL);
+    return s;
+}
+
 /* Retrieve an array parameter */
 
 /**/
@@ -3971,7 +3980,7 @@ setlang(char *x)
     struct localename *ln;
     char *x2;
 
-    if ((x2 = getsparam("LC_ALL")) && *x2)
+    if ((x2 = getsparam_u("LC_ALL")) && *x2)
 	return;
 
     /*
@@ -3985,10 +3994,10 @@ setlang(char *x)
      * from this is meaningless.  So just all $LANG to show through in
      * that case.
      */
-    setlocale(LC_ALL, x ? x : "");
+    setlocale(LC_ALL, x ? unmeta(x) : "");
     queue_signals();
     for (ln = lc_names; ln->name; ln++)
-	if ((x = getsparam(ln->name)) && *x)
+	if ((x = getsparam_u(ln->name)) && *x)
 	    setlocale(ln->category, x);
     unqueue_signals();
 }
@@ -4004,7 +4013,7 @@ lc_allsetfn(Param pm, char *x)
      * that with any LC_* that are set.
      */
     if (!x || !*x) {
-	x = getsparam("LANG");
+	x = getsparam_u("LANG");
 	if (x && *x) {
 	    queue_signals();
 	    setlang(x);
@@ -4012,7 +4021,7 @@ lc_allsetfn(Param pm, char *x)
 	}
     }
     else
-	setlocale(LC_ALL, x);
+	setlocale(LC_ALL, unmeta(x));
 }
 
 /**/
@@ -4020,7 +4029,7 @@ void
 langsetfn(Param pm, char *x)
 {
     strsetfn(pm, x);
-    setlang(x);
+    setlang(unmeta(x));
 }
 
 /**/
@@ -4046,7 +4055,7 @@ lcsetfn(Param pm, char *x)
     if (x && *x) {
 	for (ln = lc_names; ln->name; ln++)
 	    if (!strcmp(ln->name, pm->node.nam))
-		setlocale(ln->category, x);
+		setlocale(ln->category, unmeta(x));
     }
     unqueue_signals();
 }
diff --git a/Src/utils.c b/Src/utils.c
index 8ff575f..ba90564 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -248,7 +248,7 @@ VA_DCL
 
     VA_START(ap, message);
     VA_GET_ARG(ap, message, const char *);
-    if ((filename = getsparam("ZSH_DEBUG_LOG")) != NULL &&
+    if ((filename = getsparam_u("ZSH_DEBUG_LOG")) != NULL &&
 	(file = fopen(filename, "a")) != NULL) {
 	zerrmsg(file, message, ap);
 	fclose(file);
@@ -1949,7 +1949,8 @@ extern char *_mktemp(char *);
 /* Get a unique filename for use as a temporary file.  If "prefix" is
  * NULL, the name is relative to $TMPPREFIX; If it is non-NULL, the
  * unique suffix includes a prefixed '.' for improved readability.  If
- * "use_heap" is true, we allocate the returned name on the heap. */
+ * "use_heap" is true, we allocate the returned name on the heap.
+ * The string passed as "prefix" is expected to be metafied. */
 
 /**/
 mod_export char *
@@ -1976,6 +1977,9 @@ gettempname(const char *prefix, int use_heap)
     return ret;
 }
 
+/* The gettempfile() "prefix" is expected to be metafied, see hist.c
+ * and gettempname(). */
+
 /**/
 mod_export int
 gettempfile(const char *prefix, int use_heap, char **tempname)
@@ -3585,7 +3589,7 @@ zbeep(void)
 {
     char *vb;
     queue_signals();
-    if ((vb = getsparam("ZBEEP"))) {
+    if ((vb = getsparam_u("ZBEEP"))) {
 	int len;
 	vb = getkeystring(vb, &len, GETKEYS_BINDKEY, NULL);
 	write_loop(SHTTY, vb, len);
diff --git a/Src/watch.c b/Src/watch.c
index fe409f9..e1bdaa4 100644
--- a/Src/watch.c
+++ b/Src/watch.c
@@ -566,7 +566,7 @@ dowatch(void)
 	return;
     }
     queue_signals();
-    if (!(fmt = getsparam("WATCHFMT")))
+    if (!(fmt = getsparam_u("WATCHFMT")))
 	fmt = DEFAULT_WATCHFMT;
     while ((uct || wct) && !errflag)
 	if (!uct || (wct && ucmp(uptr, wptr) > 0))

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 19,  4:50am, Oliver Kiddle wrote:
}
} How about the following patch? This adds another special token to
} zle_highlight. So you can do, e.g: zle_highlight+=( paste:bg=87 )
} This applies also for vi put and emacs yank commands.
} 
} Text from a bracketed paste is now added to the cutbuffer.
} 
} I had been pondering how to interface bracketed-paste with vi registers,
} anyway. This will now also put the bracketed paste in the numbered
} registers.

This all sounds fine to me.  I'm not certain of the real emacs behavior;
I suspect it may have morphed over time and I don't know the most recent
incarnation.

Re: PATCH: highlight pasted text

[Yuri D'Elia]

On 19/07/15 04:50, Oliver Kiddle wrote:
> How about the following patch? This adds another special token to
> zle_highlight. So you can do, e.g: zle_highlight+=( paste:bg=87 )
> This applies also for vi put and emacs yank commands.

This is sweet. I can safely say this gives a great indication of the
state of the shell, and fixes the behavior of the literal paste in any
condition.

But chopping off that newline if it's the last character... hmm, I still
think that's a good idea.

> Text from a bracketed paste is now added to the cutbuffer. As far as
> I can tell, this seems to actually also be what the GUI mode of emacs
> does. Though emacs is perhaps not putting duplicates in. It'd be good if
> some actual emacs users could check that it all makes sense, however.

It matches the default behavior of 24.* at least.

Re: PATCH: highlight pasted text

[Daniel Shahaf]

Yuri D'Elia wrote on Sun, Jul 19, 2015 at 14:13:39 +0200:
> On 19/07/15 04:50, Oliver Kiddle wrote:
> > How about the following patch? This adds another special token to
> > zle_highlight. So you can do, e.g: zle_highlight+=( paste:bg=87 )
> > This applies also for vi put and emacs yank commands.
> 
> This is sweet. I can safely say this gives a great indication of the
> state of the shell, and fixes the behavior of the literal paste in any
> condition.
> 
> But chopping off that newline if it's the last character... hmm, I still
> think that's a good idea.

I'd rather text be pasted verbatim (for a number of reasons, I can
elaborate if needed).  How about doing

    zle -M "Press <Enter> to execute"

if the pasted text ends with a newline?

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Yuri D'Elia wrote:
> But chopping off that newline if it's the last character... hmm, I still
> think that's a good idea.

How about the following? It only strips the newline when pasting at
the end of the buffer with region inactive and only after putting the
unadulterated string in the cut buffer.

I've also added a note to the NEWS file about bracketed paste.

> > does. Though emacs is perhaps not putting duplicates in. It'd be good if
> > some actual emacs users could check that it all makes sense, however.
> It matches the default behavior of 24.* at least.

Good, thanks.

Oliver

diff --git a/NEWS b/NEWS
index d515a60..6e78e30 100644
--- a/NEWS
+++ b/NEWS
@@ -18,6 +18,10 @@ creates a local "scalar" containing the text "one word"
 and an array "array" containing the words "several"
 "words".
 
+- The bracketed paste mode of newer terminal emulators is now supported.
+  This feature allows the shell to differentiate between pasted text and
+  typed-in characters that may include keys with associated functions.
+
 Changes from 5.0.0 to 5.0.8
 ---------------------------
 
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index b040b97..1012fee 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -787,6 +787,11 @@ bracketedpaste(char **args)
 	    zmult = 1;
 	    if (region_active)
 		killregion(zlenoargs);
+	    /* chop a final newline if it's insertion would be hard to
+	     * distinguish by the user from the line being accepted */
+	    else if ((zlecs + (insmode ? 0 : n - 1)) >= zlell &&
+		    wpaste[n-1] == ZWC('\n'))
+		n--;
 	    yankcs = yankb = zlecs;
 	    doinsert(wpaste, n);
 	    yanke = zlecs;

Re: PATCH: highlight pasted text

[Yuri D'Elia]

On 19/07/15 19:57, Oliver Kiddle wrote:
> Yuri D'Elia wrote:
>> But chopping off that newline if it's the last character... hmm, I still
>> think that's a good idea.
> 
> How about the following? It only strips the newline when pasting at
> the end of the buffer with region inactive and only after putting the
> unadulterated string in the cut buffer.

I like it, that's a nice touch.

To reply to Daniel as well, I'm ambivalent about the behavior.
If you feel that's "not right" (and I see the reasons - no need to
elaborate for me), I'm equally fine with literal pasting, so I cannot
help you decide here.

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 19,  5:12pm, Daniel Shahaf wrote:
} Subject: Re: PATCH: highlight pasted text
}
} Yuri D'Elia wrote on Sun, Jul 19, 2015 at 14:13:39 +0200:
} > But chopping off that newline if it's the last character... hmm, I still
} > think that's a good idea.
} 
} I'd rather text be pasted verbatim (for a number of reasons, I can
} elaborate if needed).  How about doing
} 
}     zle -M "Press <Enter> to execute"
} 
} if the pasted text ends with a newline?

The question is what to do in the default (built-in widget) case.  Yes,
the built-in widget could do something equivalent to that, but the
message is context-dependent; if you're pasting into a here-document,
for example, pressing enter won't execute anything.

It'd make more sense to (for example) always show the PS2 prompt after
a paste that contains a newline, because that can both describe the
context and indicate that the shell is waiting for more input.

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> } On Jul 19,  5:12pm, Daniel Shahaf wrote:
> } Yuri D'Elia wrote on Sun, Jul 19, 2015 at 14:13:39 +0200:
> } > But chopping off that newline if it's the last character... hmm, I still
> } > think that's a good idea.
> } 
> } I'd rather text be pasted verbatim (for a number of reasons, I can
> } elaborate if needed).  How about doing
> } 
> }     zle -M "Press <Enter> to execute"

I'd prefer something more subtle but as a more lateral thinking approach
this is going in the right direction.

> It'd make more sense to (for example) always show the PS2 prompt after
> a paste that contains a newline, because that can both describe the
> context and indicate that the shell is waiting for more input.

I like that.

One disadvantage is that it wouldn't be compatible with the new feature
of highlighting pasted text; at least not having the highlighting
removed on the next action.

What would you use for the parser status that you get with %^ or %_?
The pasted text might leave an open parser status or might not.

Another disadvantage is that PS2 in general is irritating for the
fact that it restarts zle and push-line-or-edit tricks are needed on
up-cursor to allow editing the prebuffer.

As an aside, I recommend an empty $PS2 and transient_rprompt for RPS2
so the PS2 prompt doesn't get in the way of attempts to copy multi-line
commands with the mouse.

Oliver

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 21,  5:23pm, Oliver Kiddle wrote:
} Subject: Re: PATCH: highlight pasted text
}
} Bart wrote:
} > It'd make more sense to (for example) always show the PS2 prompt after
} > a paste that contains a newline, because that can both describe the
} > context and indicate that the shell is waiting for more input.
} 
} One disadvantage is that it wouldn't be compatible with the new feature
} of highlighting pasted text; at least not having the highlighting
} removed on the next action.
} 
} Another disadvantage is that PS2 in general is irritating for the
} fact that it restarts zle and push-line-or-edit tricks are needed on
} up-cursor to allow editing the prebuffer.

I didn't say "restart zle at the PS2 prompt" ... I just said to display
it.  It could be handled like RPS2 and immediately removed as soon as
more keystrokes are received, at the time that the highlighting goes.

} What would you use for the parser status that you get with %^ or %_?
} The pasted text might leave an open parser status or might not.

Hm.  Well, until there actually is an accept-line, the parser status
hasn't changed from when the paste began.  Perhaps just append a "+"
or a "?" to the current value of %_ to indicate that the parser has
yet to determine its state?

} As an aside, I recommend an empty $PS2 and transient_rprompt for RPS2
} so the PS2 prompt doesn't get in the way of attempts to copy multi-line
} commands with the mouse.

Yeah, this is a good idea.  I think I'll add

    RPS2="<${PS2%> }"
    PS2=''
    setopt transient_rprompt

or something akin to it, to prompt_bart_setup in the distribution.

Re: Unmetafy of getsparam()

[Peter Stephenson]

On Sun, 19 Jul 2015 01:13:54 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Here's a patch that attempts to clean up [un]metafication of non-special
> parameter values used internally to the shell.  Well, some of them.

Seems to make sense.  You can commit it now.  (Oops...)

> I'm unsure what to do with:
> 
> - ENV in init.c -- for parsestr() is a tokenized string also metafied?

The intention must surely be yes: parsestr() operates on internal
strings way after the input stage, and indeed metafication occurs early
in that, as witness zleread() returns a metafied line back to input.c
(and so does shingetline() if ZLE is not in use), so it's already
metafied by the time it hits the input stack --- so the string to be
parsed needs to be metafied as it's put back on the input stack.

Whether all pathways are correct is another question.

> - untok_and_escape() in subst.c

I think this has internal (metafied) strings at both input and output.

> - spckword() in utils.c -- patcompile expects a metafied string?

Yes, it does: I think spckword() gets a metafied string: zle_tricky.c
isn't easily penestrable but that's because it's dealing with shell
internal string, which should be metafied.

> - all of hist.c is a mess with respect to handling of HISTFILE

Yes, Mikael and I had extreme fun with this together with heckles from
the crowd for added effect about our efforts to keep it consistent.
(No doping allegations, at least...)  Probably best left alone until
something else turns up.

> - getcvar() in math.c

That's supposed to be handling it already within the function.

> - savesession() in zftp.c

This is all internal state, isn't it?  So as long as it's consistent it
doesn't matter.

> - LISTPROMPT in complistmatches()

That turns ultimately into the fmt used in compprintfmt, I think.
This already seems to be prepared to handle metafied strings,
though again whether it does so consistently may well not have been
checked.

> - MENUSELECT, MENUSCROLL, MENUMODE, MENUPROMPT in domenuselect()

Look to me like they all go to places that have some sort of a metafied
feel about them.  I'm guessing something customisable like this is
likely to have had a fair amount of kanji or katakana thrown at it by
now.

pws

Re: Unmetafy of getsparam()

[Bart Schaefer]

On Jul 21,  9:07pm, Peter Stephenson wrote:
} Subject: Re: Unmetafy of getsparam()
}
} On Sun, 19 Jul 2015 01:13:54 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Here's a patch that attempts to clean up [un]metafication of non-special
} > parameter values used internally to the shell.  Well, some of them.
} 
} Seems to make sense.  You can commit it now.  (Oops...)

Oops, indeed.

Someone should probably test using a character that needs metafication
in ZLE_REMOVE_SUFFIX_CHARS and/or ZLE_SPACE_SUFFIX_CHARS.

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> } Another disadvantage is that PS2 in general is irritating for the
> } fact that it restarts zle and push-line-or-edit tricks are needed on
> } up-cursor to allow editing the prebuffer.
> 
> I didn't say "restart zle at the PS2 prompt" ... I just said to display
> it.  It could be handled like RPS2 and immediately removed as soon as
> more keystrokes are received, at the time that the highlighting goes.

That's not quite so easy.
 
> Yeah, this is a good idea.  I think I'll add
> 
>     RPS2="<${PS2%> }"

Note, however, that where %_ is used in PS2, you might prefer %^ in
RPS2. That's the same thing but with the order reversed and I've just
realised that it isn't in the documentation; hence the small patch.

Oliver

diff --git a/Doc/Zsh/prompt.yo b/Doc/Zsh/prompt.yo
index 3075876..c4e2014 100644
--- a/Doc/Zsh/prompt.yo
+++ b/Doc/Zsh/prompt.yo
@@ -92,6 +92,10 @@ print as many as there are.  This is most useful in prompts tt(PS2) for
 continuation lines and tt(PS4) for debugging with the tt(XTRACE) option; in
 the latter case it will also work non-interactively.
 )
+item(tt(%^))(
+The status of the parser in reverse. This is the same as `tt(%_)' other than
+the order of strings.  It is often used in tt(RPS2).
+)
 xitem(tt(%d))
 item(tt(%/))(
 Current working directory.  If an integer follows the `tt(%)',

Re: PATCH: highlight pasted text

[Mikael Magnusson]

On Sun, Jul 19, 2015 at 4:50 AM, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Bart wrote:
>> There are a couple of features that might be useful if we could come up
>> with a way to do them.  One would be an indication of whether bracketed
>> paste is supported at all, so that no one is taken by surprise when the
>
> That's not possible as far as I know. If it was possible to detect that,
> we'd be able to make the sending of the start/end strings conditional.
>
>> shell starts executing what they pasted.  Another is some indication of
>> what text was pasted (maybe highlight-color it differently?).
>
> How about the following patch? This adds another special token to
> zle_highlight. So you can do, e.g: zle_highlight+=( paste:bg=87 )
> This applies also for vi put and emacs yank commands.
>
> We were already tracking start/end positions of a paste for the purposes
> of the yank-pop widget. Using those, leads to some further changes:
>
> Text from a bracketed paste is now added to the cutbuffer. As far as
> I can tell, this seems to actually also be what the GUI mode of emacs
> does. Though emacs is perhaps not putting duplicates in. It'd be good if
> some actual emacs users could check that it all makes sense, however.
>
> I had been pondering how to interface bracketed-paste with vi registers,
> anyway. This will now also put the bracketed paste in the numbered
> registers. If you explicitly name a vi register, the pasted text is now
> NOT inserted: it is only assigned to the register. This seems quite
> useful if not the most vim compatible. For a middle-mouse click after
> naming a register, gvim will paste the named register at the position
> under the mouse.

Is there any way to use any of this (the highlighting) if you wrap the
widget to manipulate the text first?

-- 
Mikael Magnusson

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Mikael Magnusson wrote:
> Is there any way to use any of this (the highlighting) if you wrap the
> widget to manipulate the text first?

Currently, no.

Apart from using a bindkey -s sequence that finishes with a widget that
has ZLE_YANK set.

Oliver

Re: Unmetafy of getsparam()

[Peter Stephenson]

On Sun, 19 Jul 2015 01:13:54 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> @@ -4020,7 +4029,7 @@ void
>  langsetfn(Param pm, char *x)
>  {
>      strsetfn(pm, x);
> -    setlang(x);
> +    setlang(unmeta(x));
>  }

This is giving me an instant segmentation violation on one system.

I think the best fix is for unmeta() to adopt the behaviour of checking
and returning NULL immediately.

pws

diff --git a/Src/utils.c b/Src/utils.c
index ba90564..0acab88 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -4401,7 +4401,10 @@ unmeta(const char *file_name)
     char *p;
     const char *t;
     int newsz, meta;
-    
+
+    if (!file_name)
+	return NULL;
+
     meta = 0;
     for (t = file_name; *t; t++) {
 	if (*t == Meta)

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 23,  8:23am, Oliver Kiddle wrote:
} Subject: Re: PATCH: highlight pasted text
}
} Mikael Magnusson wrote:
} > Is there any way to use any of this (the highlighting) if you wrap the
} > widget to manipulate the text first?

Like this:

   zle yank
   zle -R
   zle read-command && zle $REPLY

} Apart from using a bindkey -s sequence that finishes with a widget that
} has ZLE_YANK set.

It could be enabled by poking into region_highlight if the yankb / yanke
internal variables were exposed via zsh/zleparam.

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 23, 10:06pm, Bart Schaefer wrote:
}
}    zle yank
}    zle -R
}    zle read-command && zle $REPLY

That doesn't work with yank-pop, sadly, but can anyone point out a
problem with this tiny patch?


diff --git a/Src/Zle/iwidgets.list b/Src/Zle/iwidgets.list
index ebcf317..657e4ef 100644
--- a/Src/Zle/iwidgets.list
+++ b/Src/Zle/iwidgets.list
@@ -94,7 +94,7 @@
 "quoted-insert", quotedinsert, ZLE_MENUCMP | ZLE_KEEPSUFFIX
 "quote-line", quoteline, 0
 "quote-region", quoteregion, 0
-"read-command", readcommand, 0
+"read-command", readcommand, ZLE_NOTCOMMAND
 "recursive-edit", recursiveedit, ZLE_MENUCMP | ZLE_KEEPSUFFIX | ZLE_LASTCOL
 "redisplay", redisplay, ZLE_MENUCMP | ZLE_KEEPSUFFIX | ZLE_LASTCOL
 "redo", redo, ZLE_KEEPSUFFIX


Then you can do:

    zle yank
    zle -R
    while zle read-command
    do
      # This looks odd but "zle $REPLY" might change $REPLY
      # so we have to test it before doing anything else
      if [[ $REPLY = (.|)yank-pop ]]
      then
        zle $REPLY
      else
        zle $REPLY
        break
      fi
    done

Alternately,

    zle yank
    zle -R
    zle recursive-edit

Of course that requires an extra accept-line to escape from the
recursive-edit.  You could fix that with some fancy keymap stuff
passed with -K to recursive-edit, see Functions/Zle/keymap+widget
for possible inspiration.

Vim special marks - Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> } Apart from using a bindkey -s sequence that finishes with a widget that
> } has ZLE_YANK set.
> 
> It could be enabled by poking into region_highlight if the yankb / yanke
> internal variables were exposed via zsh/zleparam.

Vim has special marks for jumping to the start and end of the most
recent change: `[ and `]. There's also `. which is similar. In effect,
this is much the same as yankb and yanke after a paste. I think it'd
be better to make something more general like that available. It's
a feature I use quite a bit in vim. Is there anything in emacs like
yankb/yanke?

The following is an implementation that works by digging in the undo
information. This has the advantage of keeping the code related to this
feature contained within vigotomark rather than having to sprinkle code
everywhere to update the special marks. There's one very unfortunate
effect of this design choice, however: if the start of some inserted
text looks the same as what was there before, the undo mechanism will
identify the later characters as having been inserted rather than the
earlier ones.

Any ideas: stick with this and ignore that bug, hack in some way of
hinting to undo where it should look for changes, add global variables
that are updated everywhere or something else?

Oliver

diff --git a/Src/Zle/zle_move.c b/Src/Zle/zle_move.c
index d751c43..1b6177b 100644
--- a/Src/Zle/zle_move.c
+++ b/Src/Zle/zle_move.c
@@ -876,24 +876,81 @@ int
 vigotomark(UNUSED(char **args))
 {
     ZLE_INT_T ch;
+    int *markcs, *markhist = 0;
     int oldcs = zlecs;
     int oldline = histline;
+    struct change *current;
+    int tmpcs, tmphist, found = 0;
 
     ch = getfullchar(0);
-    if (ch == ZWC('\'') || ch == ZWC('`'))
-	ch = 26;
-    else {
-	if (ch < ZWC('a') || ch > ZWC('z'))
+    if (ch == ZWC('\'') || ch == ZWC('`')) {
+	markhist = vimarkline + 26;
+	markcs = vimarkcs + 26;
+    } else if (ch == ZWC('[')) {
+	current = curchange->prev;
+	tmpcs = zlell;
+	tmphist = histline;
+	/* find lowest change offset in current undo change */
+	for (; current; current = current->prev) {
+	    if (current->hist != tmphist && found)
+		break;
+            if (current->off < tmpcs || current->hist != tmphist)
+		tmpcs = current->off;
+		tmphist = current->hist;
+	    if (!found && (current->insl || current->dell))
+		found = 1;
+	    if (!(current->flags & CH_PREV))
+		break;
+	}
+	if (!found)
 	    return 1;
-	ch -= ZWC('a');
-    }
-    if (!vimarkline[ch])
-	return 1;
-    if (curhist != vimarkline[ch] && !zle_goto_hist(vimarkline[ch], 0, 0)) {
-	vimarkline[ch] = 0;
+	markcs = &tmpcs;
+	markhist = &tmphist;
+    } else if (ch == ZWC(']')) {
+	for (current = curchange->prev;current && current->flags & CH_PREV;
+		current = current->prev) {}
+	if (!current)
+	    return 1;
+	tmpcs = current->off + current->insl;
+	tmphist = histline;
+	/* highest offset after insert in current undo change */
+        for (;current != curchange;current = current->next) {
+	    if (!found && (current->insl || current->dell))
+		found = 1;
+	    if (current->off + current->insl >= tmpcs ||
+		    current->hist != tmphist) {
+		tmpcs = current->off;
+		tmphist = current->hist;
+	        if (current->insl)
+		    tmpcs += current->insl - 1;
+	    } else /* adjust tmpcs following this change */
+		tmpcs += current->insl - current->dell;
+	}
+	if (!found)
+	    return 1;
+	markcs = &tmpcs;
+	markhist = &tmphist;
+    } else if (ch == ZWC('.') && curchange->prev) {
+	/* position cursor where it was after the last change. not exactly
+	 * what vim does but close enough */
+	tmpcs = curchange->prev->new_cs;
+	tmphist = curchange->prev->hist;
+	markcs = &tmpcs;
+	markhist = &tmphist;
+    } else if (ch >= ZWC('a') && ch <= ZWC('z')) {
+	markhist = vimarkline + (ch - ZWC('a'));
+	markcs = vimarkcs + (ch - ZWC('a'));
+    } else
 	return 1;
+    if (markhist) {
+	if (!*markhist)
+	    return 1;
+	if (histline != *markhist && !zle_goto_hist(*markhist, 0, 0)) {
+	    *markhist = 0;
+	    return 1;
+	}
     }
-    zlecs = vimarkcs[ch];
+    zlecs = *markcs;
     vimarkcs[26] = oldcs;
     vimarkline[26] = oldline;
     if (zlecs > zlell)
diff --git a/Test/X02zlevi.ztst b/Test/X02zlevi.ztst
index 14bc02e..21d22b1 100644
--- a/Test/X02zlevi.ztst
+++ b/Test/X02zlevi.ztst
@@ -289,6 +289,22 @@
 >BUFFER: one wo
 >CURSOR: 2
 
+  zletest $'one two three\e`[~wcenew\e`[~$p`[~$rO`['
+0:move to start of change with `[
+>BUFFER: One New threeTwO
+>CURSOR: 15
+
+  zletest $'one\eyyPj`.~``cetwo\e0`.'
+0:return to position after last change
+>BUFFER: One
+>two
+>CURSOR: 6
+
+  zletest $'one two three\e0`]~bbcenew\e0`]~0P`]~0rO`]'
+0:move to end of change with `]
+>BUFFER: OwOone neW threE
+>CURSOR: 0
+
   zletest $'keepnot\eo  unwanted\ekhhcvj '
 0:force character-wise change to join lines
 >BUFFER: keep wanted

Re: Vim special marks - Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 24,  9:06pm, Oliver Kiddle wrote:
} Subject: Vim special marks - Re: PATCH: highlight pasted text
}
} Vim has special marks for jumping to the start and end of the most
} recent change: `[ and `]. There's also `. which is similar. In effect,
} this is much the same as yankb and yanke after a paste. I think it'd
} be better to make something more general like that available. It's
} a feature I use quite a bit in vim. Is there anything in emacs like
} yankb/yanke?

Emacs moves MARK to the position of CURSOR from before the yank, and
then places CURSOR at the end of the yank.  This seems to apply whether
using internal commands or pasting from the X clipboard.  There are no
other special marks as far as I know.

However, it does not highlight the text between MARK and CURSOR as a
region unless you e.g. exchange-point-and-mark after pasting.  Region
operations like indent-region act on the pasted text, though.

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> }
> }    zle yank
> }    zle -R
> }    zle read-command && zle $REPLY

That might want a split-undo in the middle there.

> That doesn't work with yank-pop, sadly, but can anyone point out a
> problem with this tiny patch?
> 
> -"read-command", readcommand, 0
> +"read-command", readcommand, ZLE_NOTCOMMAND

Seems like a sensible change to me. And I can't spot anything it might
break. read-command only really makes sense when used in a user-defined
widget.

Oliver

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Jul 25, 12:22am, Oliver Kiddle wrote:
} Subject: Re: PATCH: highlight pasted text
}
} Bart wrote:
} > }
} > }    zle yank
} > }    zle -R
} > }    zle read-command && zle $REPLY
} 
} That might want a split-undo in the middle there.

Before or after read-command?

Upon further thought, it might actually be preferable to do

    zle -U "$KEYS"

instead of "zle $REPLY".

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> } > }    zle yank
> } > }    zle -R
> } > }    zle read-command && zle $REPLY
> } 
> } That might want a split-undo in the middle there.
> Before or after read-command?

Shouldn't matter since read-command neither inserts or deletes text nor
changes histline.
It would need to be after the 'zle yank' and before the 'zle $REPLY'. So
in practical terms, you'd put it before because of the &&.

> Upon further thought, it might actually be preferable to do
> 
>     zle -U "$KEYS"
> 
> instead of "zle $REPLY".

Yes, I think that is preferable. You might want zle -U - "$KEYS" just in
case the next key is -, however.

Oliver

zle options (was Re: PATCH: highlight pasted text)

[Bart Schaefer]

On Jul 25,  9:49am, Oliver Kiddle wrote:
}
} Yes, I think that is preferable. You might want zle -U - "$KEYS" just in
} case the next key is -, however.

Wow, so incremental-complete-word has had a bug all these years.

What's the point of allowing other options after -U?  Just a side-effect
of using the standard option parsing mechanism?  There's no sensible
combination of other options with -U (or with -K or -M or ...), they
all just generate "too many arguments" or "incompatible" errors.

Re: zle options (was Re: PATCH: highlight pasted text)

[Oliver Kiddle]

On 25 Jul, Bart wrote:
> }
> } Yes, I think that is preferable. You might want zle -U - "$KEYS" just in
> } case the next key is -, however.
> 
> Wow, so incremental-complete-word has had a bug all these years.
> 
> What's the point of allowing other options after -U?  Just a side-effect
> of using the standard option parsing mechanism?  There's no sensible
> combination of other options with -U (or with -K or -M or ...), they
> all just generate "too many arguments" or "incompatible" errors.

It would be possible to fix this without losing the standard option
parsing mechanism by making the argument an argument to -U rather than
one of the standard arguments. So a colon would be added after U where
the builtin is defined in zle_main.c.

Only problem is that it would break any case where someone has done
zle -U - "$KEYS"

Oliver

Re: PATCH: highlight pasted text

[Daniel Shahaf]

Oliver Kiddle wrote on Sun, Jul 19, 2015 at 19:57:19 +0200:
> Yuri D'Elia wrote:
> > But chopping off that newline if it's the last character... hmm, I still
> > think that's a good idea.
> 
> How about the following? It only strips the newline when pasting at
> the end of the buffer with region inactive and only after putting the
> unadulterated string in the cut buffer.
...
> +	    /* chop a final newline if it's insertion would be hard to
> +	     * distinguish by the user from the line being accepted */

Over here I'm using zle_highlight=(paste:standout).  It's unobtrusive,
doesn't munge the pasted text, and on at least one occasion reminded me
to invoke <accept-line> after pasting.

Maybe set that by default instead?

---

Alternatively, could I do something like:
   bracketed-paste() {
     zle .bracketed-paste
     zle .yank-pop
   }
   zle -N bracketed-paste
to disable newline trimming, if this patch is applied?

Re: Vim special marks - Re: PATCH: highlight pasted text

[Oliver Kiddle]

On 24 Jul, I wrote:
> There's one very unfortunate
> effect of this design choice, however: if the start of some inserted
> text looks the same as what was there before, the undo mechanism will
> identify the later characters as having been inserted rather than the
> earlier ones.

I've not come up with a solution to this. One part of the patch was
actually a bug fix: the use of histline rather than curhist when
checking whether the zle_goto_hist call is needed. This prevented you
from returning to the original line.

The following is a cut down version of the patch. It includes the addition
of the . mark which uses the cursor position from the last change rather
than the offset so is unaffected by the problem,

Oliver

diff --git a/Src/Zle/zle_move.c b/Src/Zle/zle_move.c
index d751c43..208839e 100644
--- a/Src/Zle/zle_move.c
+++ b/Src/Zle/zle_move.c
@@ -876,24 +876,36 @@ int
 vigotomark(UNUSED(char **args))
 {
     ZLE_INT_T ch;
+    int *markcs, *markhist = 0;
     int oldcs = zlecs;
     int oldline = histline;
+    int tmpcs, tmphist;
 
     ch = getfullchar(0);
-    if (ch == ZWC('\'') || ch == ZWC('`'))
-	ch = 26;
-    else {
-	if (ch < ZWC('a') || ch > ZWC('z'))
-	    return 1;
-	ch -= ZWC('a');
-    }
-    if (!vimarkline[ch])
-	return 1;
-    if (curhist != vimarkline[ch] && !zle_goto_hist(vimarkline[ch], 0, 0)) {
-	vimarkline[ch] = 0;
+    if (ch == ZWC('\'') || ch == ZWC('`')) {
+	markhist = vimarkline + 26;
+	markcs = vimarkcs + 26;
+    } else if (ch == ZWC('.') && curchange->prev) {
+	/* position cursor where it was after the last change. not exactly
+	 * what vim does but close enough */
+	tmpcs = curchange->prev->new_cs;
+	tmphist = curchange->prev->hist;
+	markcs = &tmpcs;
+	markhist = &tmphist;
+    } else if (ch >= ZWC('a') && ch <= ZWC('z')) {
+	markhist = vimarkline + (ch - ZWC('a'));
+	markcs = vimarkcs + (ch - ZWC('a'));
+    } else
 	return 1;
+    if (markhist) {
+	if (!*markhist)
+	    return 1;
+	if (histline != *markhist && !zle_goto_hist(*markhist, 0, 0)) {
+	    *markhist = 0;
+	    return 1;
+	}
     }
-    zlecs = vimarkcs[ch];
+    zlecs = *markcs;
     vimarkcs[26] = oldcs;
     vimarkline[26] = oldline;
     if (zlecs > zlell)

Re: PATCH: highlight pasted text

[Daniel Shahaf]

Mikael Magnusson wrote on Thu, Jul 23, 2015 at 07:00:40 +0200:
> Is there any way to use any of this (the highlighting) if you wrap the
> widget to manipulate the text first?

Here's an attempt to summarize the solutions proposed later in the thread:

paste-and-munge() {
  local unmunged munged
  zle .bracketed-paste unmunged
  munged=":::${unmunged}:::"
  LBUFFER+=$munged
  (( MARK = $#LBUFFER - $#munged ))
  zle kill-region
  zle yank
  zle -R
  zle read-command && zle -U - $KEYS
}
bindkey -s $zle_bracketed_paste[2] paste-and-munge
zle -N bracketed-paste paste-and-munge
zle_highlight=(paste:standout)

I haven't incorporated split-undo.

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Aug 13, 11:14pm, Daniel Shahaf wrote:
}
} paste-and-munge() {
}   local unmunged munged
}   zle .bracketed-paste unmunged
}   munged=":::${unmunged}:::"
}   LBUFFER+=$munged
}   (( MARK = $#LBUFFER - $#munged ))
}   zle kill-region
}   zle yank
}   zle -R
}   zle read-command && zle -U - $KEYS
} }
} bindkey -s $zle_bracketed_paste[2] paste-and-munge

What's this?  That's not how bindkey -s works.  Unless you're attempting
to have the string "paste-and-munge" inserted on the command line, which
I suspect you aren't.  Also $zle_bracketed_paste[2] is the sequence that
is *output* for disabling bracketed paste, which is going to be eaten by
the terminal emulator so you'd never see it as input.

Hmm, it just occurred to me that although the starting *input* sequence
for bracketed-paste can be changed by using "bindkey", the *ending*
input sequence is hardwired in bracketedstring().  If the terminal
might have different start/end mode sequences for output, it might
also *send* different start/end sequences around the paste, and as it
stands we can't discern the ending one.

It might actually be nice to have two widgets, one called at each end
of the paste.  Then things like applying the quoting could appear in
the end-of-paste widget.

Anyway, back to the topic at hand: this --

} zle -N bracketed-paste paste-and-munge

-- should take care of calling paste-and-munge when the appropriate
start-of-a-paste sequence is seen.

} zle_highlight=(paste:standout)
} 
} I haven't incorporated split-undo.

-- 
Barton E. Schaefer

Re: PATCH: highlight pasted text

[Oliver Kiddle]

On 8 Aug, Daniel Shahaf wrote:
> Over here I'm using zle_highlight=(paste:standout).  It's unobtrusive,
> doesn't munge the pasted text, and on at least one occasion reminded me
> to invoke <accept-line> after pasting.
> 
> Maybe set that by default instead?

I don't have a strong opinion on this. I didn't want to impose anything
and it seemed an easy thing to change later.

Standout is also used for an active region by default and this isn't the
region. For a basic terminal there are limited attributes available:
with the region it can be confusing if your terminal renders your cursor
as standout as doubled standout puts things back to normal. I use
subtler colours if I've got an 88 or 256 colour terminal which would be
my normal environment and disable it otherwise.

Oliver

Re: PATCH: highlight pasted text

[Oliver Kiddle]

Bart wrote:
> Hmm, it just occurred to me that although the starting *input* sequence
> for bracketed-paste can be changed by using "bindkey", the *ending*
> input sequence is hardwired in bracketedstring().  If the terminal

I did actually mention that early on. Extra elements can be added to
zle_bracketed_paste but to make it generic you'd have to change the
string matching C code to something less efficient that would allow
substrings of the string prefix within the end string.

In reality, every modern terminal emulator is just implementing things
to be xterm compatible and I don't see much point getting our knickers
in a twist about this.

> It might actually be nice to have two widgets, one called at each end
> of the paste.  Then things like applying the quoting could appear in
> the end-of-paste widget.

How would you select different end-of-paste widgets for corresponding
start widgets.

Mikael's shell function solution effectively had that because it was
using a separate keymap. I considered that approach at the time but
there are advantages either way.

Oliver

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Aug 14,  4:09am, Oliver Kiddle wrote:
}
} In reality, every modern terminal emulator is just implementing things
} to be xterm compatible and I don't see much point getting our knickers
} in a twist about this.

I can live with that.

} > It might actually be nice to have two widgets, one called at each end
} > of the paste.
} 
} How would you select different end-of-paste widgets for corresponding
} start widgets.

Examine $LASTWIDGET, I presume.  I wasn't necessarily thinking about
needing more than one of each.  We've already run into problems with
highlighting when having "zle .bracketed-paste" embedded in a user-
defined widget; maybe that would be easier to deal with if there were
a separate end-of-paste widget.  Or maybe not.

Re: PATCH: highlight pasted text

[Bart Schaefer]

On Aug 14,  3:38am, Oliver Kiddle wrote:
} Subject: Re: PATCH: highlight pasted text
}
} On 8 Aug, Daniel Shahaf wrote:
} > Over here I'm using zle_highlight=(paste:standout).
} > 
} > Maybe set that by default instead?
} 
} I don't have a strong opinion on this. I didn't want to impose anything
} and it seemed an easy thing to change later.

I'm not even sure I like having bracketed-paste itself enabled by default.

} Standout is also used for an active region by default and this isn't the
} region.

Or it could be both the region and a paste, see other thread about what
happens if you set the mark before pasting.