zsh-workers
 help / color / mirror / code / Atom feed
* Turning display attributes on / off in prompt strings
@ 2022-12-16 19:45 Bart Schaefer
  2022-12-17  0:43 ` Oliver Kiddle
  2022-12-21 17:12 ` Oliver Kiddle
  0 siblings, 2 replies; 9+ messages in thread
From: Bart Schaefer @ 2022-12-16 19:45 UTC (permalink / raw)
  To: Zsh hackers list

Reviving another old discussion ...

On Tue, May 11, 2021 at 3:18 AM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> On 5/9/21, Bart Schaefer <schaefer@brasslantern.com> wrote:
> >
> > Does anyone know why
> > putpromptchar() has this?
> >             case 'b':
> >                 txtchangeset(txtchangep, TXTNOBOLDFACE, TXTBOLDFACE);
> >                 txtunset(TXTBOLDFACE);
> >                 tsetcap(TCALLATTRSOFF, TSC_PROMPT|TSC_DIRTY);
> >                 break;
> > That is, why TCALLATTRSOFF ?  That isn't done for %s or %u ... why is
> > there no TCBOLDFACEEND defined?
>
> I think that's a shortcoming in termcap and/or terminfo. In termcap
> there is md (1m) and me (0m) that should correspond to bold, but 0m
> turns off all attributes. Compare with us (4m) and ue (24m).

So ... I've been playing with this, and on my "xterm-color" terminal
se and ue are both $'\e[m' -- and yet "print -P" and ${(%)...} and so
on do "the right thing" and output extra control sequences depending
on what other attributes are active.  For example

  print -P %U%B%S...%u...

outputs $'\e[4m\e[1m\e[7m...\e[m\e[1m\e[7m...' -- it has recorded
(txtchangeset) the three attributes and then properly restores the
ones that are not intended to be turned off.

So the problem from the original thread (workers/48800 and refs) is
that you can't mix in "raw" sequences for attributes that prompt.c
doesn't "know about" lest those attributes become disabled every time
%s, %u, or definitely %b on any terminal, is used.

> https://www.gnu.org/software/termutils/manual/termcap-1.3/html_chapter/termcap_4.html
> also implies that there is no specific termcap sequence to only turn
> off specific appearance modes.
[...]
> Since we already assume ANSI for things like colors, I don't think we
> would lose a lot of compatibility in practice if we just use 22
> instead of 0 for %b, but we could potentially put it behind a setopt?

In connection with Oliver's patch to eat CSI sequences ... perhaps we
should abandon looking up termcap strings for these attributes, or at
least have a fallback to the ANSI set when we encounter a termcap
result that's not specific enough?


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-16 19:45 Turning display attributes on / off in prompt strings Bart Schaefer
@ 2022-12-17  0:43 ` Oliver Kiddle
  2022-12-17  7:50   ` Bart Schaefer
  2022-12-21 17:12 ` Oliver Kiddle
  1 sibling, 1 reply; 9+ messages in thread
From: Oliver Kiddle @ 2022-12-17  0:43 UTC (permalink / raw)
  To: Zsh hackers list

Bart Schaefer wrote:
> So the problem from the original thread (workers/48800 and refs) is
> that you can't mix in "raw" sequences for attributes that prompt.c
> doesn't "know about" lest those attributes become disabled every time
> %s, %u, or definitely %b on any terminal, is used.

It can also get confused when you assign the results of ${(%)...} to a
variable without printing the results. My .zshrc contains instances of
  : ${(%):-%u%k}

> > https://www.gnu.org/software/termutils/manual/termcap-1.3/html_chapter/termcap_4.html
> > also implies that there is no specific termcap sequence to only turn
> > off specific appearance modes.
> [...]
> > Since we already assume ANSI for things like colors, I don't think we
> > would lose a lot of compatibility in practice if we just use 22
> > instead of 0 for %b, but we could potentially put it behind a setopt?

I'd be reluctant to create an extra setopt without first having evidence
that terminals that need it are still in use. And if we need a way to
override terminal settings, something more generic may be better. I like
tmux's terminal-overrides[] but that's tied to terminfo and the premise
of this problem is that there is no termcap/info entry for end of bold.
Maybe #ifdef around it and add a configure option that we can remove
later.

> In connection with Oliver's patch to eat CSI sequences ... perhaps we
> should abandon looking up termcap strings for these attributes, or at
> least have a fallback to the ANSI set when we encounter a termcap
> result that's not specific enough?

If we allow ourselves more assumptions about escape sequences, we could
also make zsh an easier experience for new users by binding a few more
keys by default. There even is termcap entries for some. For a very long
time, I've taken the approach of binding nearly all sequences for every
terminal I use rather than using case $TERM and the superfluous bindings
are harmless.

Oliver


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-17  0:43 ` Oliver Kiddle
@ 2022-12-17  7:50   ` Bart Schaefer
  2022-12-17  9:47     ` Oliver Kiddle
  0 siblings, 1 reply; 9+ messages in thread
From: Bart Schaefer @ 2022-12-17  7:50 UTC (permalink / raw)
  To: Oliver Kiddle; +Cc: Zsh hackers list

On Fri, Dec 16, 2022 at 4:45 PM Oliver Kiddle <opk@zsh.org> wrote:
>
> Bart Schaefer wrote:
> > you can't mix in "raw" sequences for attributes that prompt.c
> > doesn't "know about"
>
> It can also get confused when you assign the results of ${(%)...} to a
> variable without printing the results.

A consideration is that if we simply replaced $"\e[m' with $'\e[24m'
and so on, the present prompt code would still emit the
then-unnecessary sequences to restore the other attributes.  Is it
worth the effort to rework that?

> I'd be reluctant to create an extra setopt without first having evidence
> that terminals that need it are still in use.

Agreed, I was excerpting Mikael mostly for "would [not] lose a lot of
compatibility".

> And if we need a way to
> override terminal settings, something more generic may be better.

Would populating a writable special hash parameter be a security issue?

As you implied, though, the hash key names could be tricky to choose.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-17  7:50   ` Bart Schaefer
@ 2022-12-17  9:47     ` Oliver Kiddle
  2022-12-17 19:45       ` Bart Schaefer
  0 siblings, 1 reply; 9+ messages in thread
From: Oliver Kiddle @ 2022-12-17  9:47 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh hackers list

Bart Schaefer wrote:
> A consideration is that if we simply replaced $"\e[m' with $'\e[24m'
> and so on, the present prompt code would still emit the
> then-unnecessary sequences to restore the other attributes.  Is it
> worth the effort to rework that?

It's probably not worth replacing the escape sequence without also
allowing it to avoid that because the original problem would only be
partially solved.

> > And if we need a way to
> > override terminal settings, something more generic may be better.
>
> Would populating a writable special hash parameter be a security issue?

I can't think of anything it would open up that isn't already the case.
It can depend on what you regard as untrusted data.

> As you implied, though, the hash key names could be tricky to choose.

We do already have an interface in the form of zle -T transformation func
Transformation is currently only "tc" for termcap but something for end
bold* or general attribute changing could be added. Prompt handling is in
the main part of zsh rather than zle but the zle module could define a
callback. The existing tc transformation interface doesn't make it easy
to modify only some capabilities, you'd need to reimplement tparm(3) in
shell code. A function provides maximum flexibility but could be slow
and a string would suffice for most uses. A hash seems like a nicer
interface but we can have a transformation function do the hash lookup
and keep that in shell code.

Oliver

* \e[22m selects normal, disabling also faint so naming indicating end
of bold would be a poor choice.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-17  9:47     ` Oliver Kiddle
@ 2022-12-17 19:45       ` Bart Schaefer
  2022-12-21 16:46         ` Oliver Kiddle
  0 siblings, 1 reply; 9+ messages in thread
From: Bart Schaefer @ 2022-12-17 19:45 UTC (permalink / raw)
  To: Oliver Kiddle; +Cc: Zsh hackers list

On Sat, Dec 17, 2022 at 1:47 AM Oliver Kiddle <opk@zsh.org> wrote:
>
> Bart Schaefer wrote:
> > the present prompt code would still emit the
> > then-unnecessary sequences to restore the other attributes.
>
> It's probably not worth replacing the escape sequence without also
> allowing it to avoid that because the original problem would only be
> partially solved.

Hm, I was thinking of it more as a new problem of optimization (we go
to some lengths in other places to send the shortest possible control
sequences).  In what way would the original problem not be solved?  An
attribute might be turned back on after having been turned off?

> We do already have an interface in the form of zle -T transformation func
> Transformation is currently only "tc" for termcap but something for end
> bold* or general attribute changing could be added. Prompt handling is in
> the main part of zsh rather than zle but the zle module could define a
> callback.

Hm, that's a bit more heavyweight than I was thinking, but would be
the most flexible.

I presume you mean to move the transformation to the main shell and
have zle call it?  Loading zle in non-interactive context would be
less desirable?

> * \e[22m selects normal, disabling also faint so naming indicating end
> of bold would be a poor choice.

That's true, outside of prompt context (which doesn't currently have a
way to express "faint" without using raw sequences).  But all the
other attributes are additive whereas faint/normal/bold are mutually
exclusive even at the terminal level, we're not going to be able to
make a general solution for (faint on)(bold on)(bold off) that ends in
(faint on) state.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-17 19:45       ` Bart Schaefer
@ 2022-12-21 16:46         ` Oliver Kiddle
  2022-12-21 21:28           ` Bart Schaefer
  0 siblings, 1 reply; 9+ messages in thread
From: Oliver Kiddle @ 2022-12-21 16:46 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh hackers list

On 17 Dec, Bart Schaefer wrote:
> On Sat, Dec 17, 2022 at 1:47 AM Oliver Kiddle <opk@zsh.org> wrote:
> >
> > Bart Schaefer wrote:
> > > the present prompt code would still emit the
> > > then-unnecessary sequences to restore the other attributes.
> >
> > It's probably not worth replacing the escape sequence without also
> > allowing it to avoid that because the original problem would only be
> > partially solved.
>
> Hm, I was thinking of it more as a new problem of optimization (we go
> to some lengths in other places to send the shortest possible control
> sequences).  In what way would the original problem not be solved?  An
> attribute might be turned back on after having been turned off?

The original problem was due to mixing raw sequences with prompt
sequences, right? Any time we produce \e[0m we lose everything from
the raw sequences. We could do more to send the shortest possible
control sequence. Something like %u in a prompt will always send the
off sequence regardless of whether underline was previously on. If
we optimise that, we actually make the situation worse for mixtures:
underline on might have been done as a raw sequence. The only way to
really solve that would be to parse the raw sequences.

If you search for TSC_DIRTY in prompt.c, you'll see that we restore
other attributes after not only bold off but for all of %s, %u, %B and,
of course %b. This code path is only used for prompts. You can see this
in the output of:
  print -P '%Sone%utwo'|sed -n l

The zle code does things differently. It only restores other attributes
where the use of bold ends. The implementation there is rather cleaner
in my opinion. settextattributes() is passed a zattr parameter for all
desired attributes and compares against the existing known state. Simply
by handling TXTNOBOLDFACE first, it ensures others are restored.
It'd be good to use the same interface for prompts, preferably also
dropping the "NO" forms of the flags to free up some bits to add more
attributes.

The glitch handling using the "sg" and "ug" termcap sequences also seems
to only be done for prompts but not ZLE. Do you know more about this?
Can it also be condemned to ancient history?

To optimise the control sequences to the maximum, generating sequences
should be delayed until actual text is printed but that either needs
to be added in many places or the output routines need to be wrapped.
Are there other control sequences that the current attributes affect
(scrolling up perhaps)?

> Hm, that's a bit more heavyweight than I was thinking, but would be
> the most flexible.

It is heavyweight but it already exists. I don't think the tc interface
itself is ideal because it is too low-level. Extending the interface
with something like zle -T attr doesn't necessarily need to be followed
by a function. It's a question of what sort of interface would be
useful? I think it would be best if we can assume a modern terminal
(i.e. just use \e[22m) but allow old or broken terminals to be worked
around from a shell interface. The hard part is designing that
interface.

> I presume you mean to move the transformation to the main shell and
> have zle call it?  Loading zle in non-interactive context would be
> less desirable?

The zle module could register a callback which the main shell would call
when changing attributes. Or it could be moved to a separate module.

> > * \e[22m selects normal, disabling also faint so naming indicating end
> > of bold would be a poor choice.
>
> That's true, outside of prompt context (which doesn't currently have a
> way to express "faint" without using raw sequences).  But all the
> other attributes are additive whereas faint/normal/bold are mutually
> exclusive even at the terminal level, we're not going to be able to
> make a general solution for (faint on)(bold on)(bold off) that ends in
> (faint on) state.

If you consider double underline, underline no underline or slow
blink, rapid blink, no blink then the font weight is not actually
unique in this regard. It wouldn't be a good idea for a variety of
reasons but you could in theory have:

  %bfaint%Bnormal%Bbold

While less terse, it is probably wiser to go with:

  %A{faint}faint%A{normal}normal%Bbold

Oliver


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-16 19:45 Turning display attributes on / off in prompt strings Bart Schaefer
  2022-12-17  0:43 ` Oliver Kiddle
@ 2022-12-21 17:12 ` Oliver Kiddle
  2022-12-21 21:03   ` Bart Schaefer
  1 sibling, 1 reply; 9+ messages in thread
From: Oliver Kiddle @ 2022-12-21 17:12 UTC (permalink / raw)
  To: Bart Schaefer; +Cc: Zsh workers

On 16 Dec, you wrote:
> So ... I've been playing with this, and on my "xterm-color" terminal
> se and ue are both $'\e[m' -- and yet "print -P" and ${(%)...} and so
> on do "the right thing" and output extra control sequences depending
> on what other attributes are active.  For example

Just noticing these details again, it would seem that zle is currently
broken for your case:

% TERM=xterm-color zsh -df
% zle-line-init() {
region_highlight=( 'P 1 2 underline' 'P 1 4 standout' )
LBUFFER=1234567890
}
% zle -N zle-line-init

Only the 2 is in standout.

Are you sure xterm-color is the right $TERM value to use? Are you
getting that by default or have you set it? Just running xterm gives me
a TERM value of xterm. Setting it to xterm-256color would also work. I
use rxvt-unicode which has display glitches in compinit menus that I
think are zsh issues.

So perhaps we do need to invalidate attributes after turning off
standout and underline if we stick to termcap. I'm still sceptical about
our prompt code doing it for turning bold on.

Oliver


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-21 17:12 ` Oliver Kiddle
@ 2022-12-21 21:03   ` Bart Schaefer
  0 siblings, 0 replies; 9+ messages in thread
From: Bart Schaefer @ 2022-12-21 21:03 UTC (permalink / raw)
  To: Oliver Kiddle; +Cc: Zsh workers

On Wed, Dec 21, 2022 at 9:12 AM Oliver Kiddle <opk@zsh.org> wrote:
>
> Are you sure xterm-color is the right $TERM value to use? Are you
> getting that by default or have you set it?

It's what gets set automatically when I ssh from MacOS Terminal.app
(xterm-256color) to Ubuntu.

> % zle-line-init() {
> region_highlight=( 'P 1 2 underline' 'P 1 4 standout' )
> LBUFFER=1234567890
> }
> % zle -N zle-line-init
>
> Only the 2 is in standout.

That's correct.

> So perhaps we do need to invalidate attributes after turning off
> standout and underline if we stick to termcap.

I think the discussion started with the premise that we would only use
termcap if ANSI sequences don't work, and ANSI sequences can turn
on/off specific attributes with the exception of faint/normal/bold ...
so theoretically we could optimize out the tracking/restoration of
most attributes most of the time?

> I'm still sceptical about
> our prompt code doing it for turning bold on.

Agree.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Turning display attributes on / off in prompt strings
  2022-12-21 16:46         ` Oliver Kiddle
@ 2022-12-21 21:28           ` Bart Schaefer
  0 siblings, 0 replies; 9+ messages in thread
From: Bart Schaefer @ 2022-12-21 21:28 UTC (permalink / raw)
  To: Oliver Kiddle; +Cc: Zsh hackers list

On Wed, Dec 21, 2022 at 8:46 AM Oliver Kiddle <opk@zsh.org> wrote:
>
> The original problem was due to mixing raw sequences with prompt
> sequences, right? Any time we produce \e[0m we lose everything from
> the raw sequences.

Sure, but the idea is that we'd stop producing \e[0m and instead
output \e[24m to specifically turn off underlining.  However, if
suppose the prompt code has seen %S%U and then standout (reverse) was
disabled by a raw \e[27m, but for %u we output \e[24m\e[07m because we
chose not to re-optimize the current code, then we've broken things in
the other direction.

Ultimately there's no way to keep track of what any raw sequences have
done so the best outcome is to do as little as necessary to maintain
the state implied by the %-codes encountered so far while also making
the state change requested by the next %-code we find.

> Something like %u in a prompt will always send the
> off sequence regardless of whether underline was previously on. If
> we optimise that, we actually make the situation worse for mixtures

Right, we always need to send something.  We can't optimize (e.g.) %u
to less than \e[24m.

> underline on might have been done as a raw sequence. The only way to
> really solve that would be to parse the raw sequences.

Yeah, I'm not suggesting that.  It still doesn't work for e.g. using
${(%)%u} to find the underline-off code.

> The zle code does things differently.  [...]
> It'd be good to use the same interface for prompts, preferably also
> dropping the "NO" forms of the flags to free up some bits to add more
> attributes.

Zle has the advantage of always being in control of the entire state,
it never has to worry about attributes being toggled "behind it's
back" or to figure out what to do with ${(%)...}.

> The glitch handling using the "sg" and "ug" termcap sequences also seems
> to only be done for prompts but not ZLE. Do you know more about this?

I do not know anything about it other than that it's for slow
terminals that might not be able to keep up with the control
sequences.  Wayne would be the expert, I think.


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-12-21 21:29 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-16 19:45 Turning display attributes on / off in prompt strings Bart Schaefer
2022-12-17  0:43 ` Oliver Kiddle
2022-12-17  7:50   ` Bart Schaefer
2022-12-17  9:47     ` Oliver Kiddle
2022-12-17 19:45       ` Bart Schaefer
2022-12-21 16:46         ` Oliver Kiddle
2022-12-21 21:28           ` Bart Schaefer
2022-12-21 17:12 ` Oliver Kiddle
2022-12-21 21:03   ` Bart Schaefer

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).