Reviving another old discussion ... On Tue, May 11, 2021 at 3:18 AM Mikael Magnusson <mikachu@gmail.com> wrote: > > On 5/9/21, Bart Schaefer <schaefer@brasslantern.com> wrote: > > > > Does anyone know why > > putpromptchar() has this? > > case 'b': > > txtchangeset(txtchangep, TXTNOBOLDFACE, TXTBOLDFACE); > > txtunset(TXTBOLDFACE); > > tsetcap(TCALLATTRSOFF, TSC_PROMPT|TSC_DIRTY); > > break; > > That is, why TCALLATTRSOFF ? That isn't done for %s or %u ... why is > > there no TCBOLDFACEEND defined? > > I think that's a shortcoming in termcap and/or terminfo. In termcap > there is md (1m) and me (0m) that should correspond to bold, but 0m > turns off all attributes. Compare with us (4m) and ue (24m). So ... I've been playing with this, and on my "xterm-color" terminal se and ue are both $'\e[m' -- and yet "print -P" and ${(%)...} and so on do "the right thing" and output extra control sequences depending on what other attributes are active. For example print -P %U%B%S...%u... outputs $'\e[4m\e[1m\e[7m...\e[m\e[1m\e[7m...' -- it has recorded (txtchangeset) the three attributes and then properly restores the ones that are not intended to be turned off. So the problem from the original thread (workers/48800 and refs) is that you can't mix in "raw" sequences for attributes that prompt.c doesn't "know about" lest those attributes become disabled every time %s, %u, or definitely %b on any terminal, is used. > https://www.gnu.org/software/termutils/manual/termcap-1.3/html_chapter/termcap_4.html > also implies that there is no specific termcap sequence to only turn > off specific appearance modes. [...] > Since we already assume ANSI for things like colors, I don't think we > would lose a lot of compatibility in practice if we just use 22 > instead of 0 for %b, but we could potentially put it behind a setopt? In connection with Oliver's patch to eat CSI sequences ... perhaps we should abandon looking up termcap strings for these attributes, or at least have a fallback to the ANSI set when we encounter a termcap result that's not specific enough?
Bart Schaefer wrote: > So the problem from the original thread (workers/48800 and refs) is > that you can't mix in "raw" sequences for attributes that prompt.c > doesn't "know about" lest those attributes become disabled every time > %s, %u, or definitely %b on any terminal, is used. It can also get confused when you assign the results of ${(%)...} to a variable without printing the results. My .zshrc contains instances of : ${(%):-%u%k} > > https://www.gnu.org/software/termutils/manual/termcap-1.3/html_chapter/termcap_4.html > > also implies that there is no specific termcap sequence to only turn > > off specific appearance modes. > [...] > > Since we already assume ANSI for things like colors, I don't think we > > would lose a lot of compatibility in practice if we just use 22 > > instead of 0 for %b, but we could potentially put it behind a setopt? I'd be reluctant to create an extra setopt without first having evidence that terminals that need it are still in use. And if we need a way to override terminal settings, something more generic may be better. I like tmux's terminal-overrides[] but that's tied to terminfo and the premise of this problem is that there is no termcap/info entry for end of bold. Maybe #ifdef around it and add a configure option that we can remove later. > In connection with Oliver's patch to eat CSI sequences ... perhaps we > should abandon looking up termcap strings for these attributes, or at > least have a fallback to the ANSI set when we encounter a termcap > result that's not specific enough? If we allow ourselves more assumptions about escape sequences, we could also make zsh an easier experience for new users by binding a few more keys by default. There even is termcap entries for some. For a very long time, I've taken the approach of binding nearly all sequences for every terminal I use rather than using case $TERM and the superfluous bindings are harmless. Oliver
On Fri, Dec 16, 2022 at 4:45 PM Oliver Kiddle <opk@zsh.org> wrote: > > Bart Schaefer wrote: > > you can't mix in "raw" sequences for attributes that prompt.c > > doesn't "know about" > > It can also get confused when you assign the results of ${(%)...} to a > variable without printing the results. A consideration is that if we simply replaced $"\e[m' with $'\e[24m' and so on, the present prompt code would still emit the then-unnecessary sequences to restore the other attributes. Is it worth the effort to rework that? > I'd be reluctant to create an extra setopt without first having evidence > that terminals that need it are still in use. Agreed, I was excerpting Mikael mostly for "would [not] lose a lot of compatibility". > And if we need a way to > override terminal settings, something more generic may be better. Would populating a writable special hash parameter be a security issue? As you implied, though, the hash key names could be tricky to choose.
Bart Schaefer wrote: > A consideration is that if we simply replaced $"\e[m' with $'\e[24m' > and so on, the present prompt code would still emit the > then-unnecessary sequences to restore the other attributes. Is it > worth the effort to rework that? It's probably not worth replacing the escape sequence without also allowing it to avoid that because the original problem would only be partially solved. > > And if we need a way to > > override terminal settings, something more generic may be better. > > Would populating a writable special hash parameter be a security issue? I can't think of anything it would open up that isn't already the case. It can depend on what you regard as untrusted data. > As you implied, though, the hash key names could be tricky to choose. We do already have an interface in the form of zle -T transformation func Transformation is currently only "tc" for termcap but something for end bold* or general attribute changing could be added. Prompt handling is in the main part of zsh rather than zle but the zle module could define a callback. The existing tc transformation interface doesn't make it easy to modify only some capabilities, you'd need to reimplement tparm(3) in shell code. A function provides maximum flexibility but could be slow and a string would suffice for most uses. A hash seems like a nicer interface but we can have a transformation function do the hash lookup and keep that in shell code. Oliver * \e[22m selects normal, disabling also faint so naming indicating end of bold would be a poor choice.
On Sat, Dec 17, 2022 at 1:47 AM Oliver Kiddle <opk@zsh.org> wrote: > > Bart Schaefer wrote: > > the present prompt code would still emit the > > then-unnecessary sequences to restore the other attributes. > > It's probably not worth replacing the escape sequence without also > allowing it to avoid that because the original problem would only be > partially solved. Hm, I was thinking of it more as a new problem of optimization (we go to some lengths in other places to send the shortest possible control sequences). In what way would the original problem not be solved? An attribute might be turned back on after having been turned off? > We do already have an interface in the form of zle -T transformation func > Transformation is currently only "tc" for termcap but something for end > bold* or general attribute changing could be added. Prompt handling is in > the main part of zsh rather than zle but the zle module could define a > callback. Hm, that's a bit more heavyweight than I was thinking, but would be the most flexible. I presume you mean to move the transformation to the main shell and have zle call it? Loading zle in non-interactive context would be less desirable? > * \e[22m selects normal, disabling also faint so naming indicating end > of bold would be a poor choice. That's true, outside of prompt context (which doesn't currently have a way to express "faint" without using raw sequences). But all the other attributes are additive whereas faint/normal/bold are mutually exclusive even at the terminal level, we're not going to be able to make a general solution for (faint on)(bold on)(bold off) that ends in (faint on) state.
On 17 Dec, Bart Schaefer wrote: > On Sat, Dec 17, 2022 at 1:47 AM Oliver Kiddle <opk@zsh.org> wrote: > > > > Bart Schaefer wrote: > > > the present prompt code would still emit the > > > then-unnecessary sequences to restore the other attributes. > > > > It's probably not worth replacing the escape sequence without also > > allowing it to avoid that because the original problem would only be > > partially solved. > > Hm, I was thinking of it more as a new problem of optimization (we go > to some lengths in other places to send the shortest possible control > sequences). In what way would the original problem not be solved? An > attribute might be turned back on after having been turned off? The original problem was due to mixing raw sequences with prompt sequences, right? Any time we produce \e[0m we lose everything from the raw sequences. We could do more to send the shortest possible control sequence. Something like %u in a prompt will always send the off sequence regardless of whether underline was previously on. If we optimise that, we actually make the situation worse for mixtures: underline on might have been done as a raw sequence. The only way to really solve that would be to parse the raw sequences. If you search for TSC_DIRTY in prompt.c, you'll see that we restore other attributes after not only bold off but for all of %s, %u, %B and, of course %b. This code path is only used for prompts. You can see this in the output of: print -P '%Sone%utwo'|sed -n l The zle code does things differently. It only restores other attributes where the use of bold ends. The implementation there is rather cleaner in my opinion. settextattributes() is passed a zattr parameter for all desired attributes and compares against the existing known state. Simply by handling TXTNOBOLDFACE first, it ensures others are restored. It'd be good to use the same interface for prompts, preferably also dropping the "NO" forms of the flags to free up some bits to add more attributes. The glitch handling using the "sg" and "ug" termcap sequences also seems to only be done for prompts but not ZLE. Do you know more about this? Can it also be condemned to ancient history? To optimise the control sequences to the maximum, generating sequences should be delayed until actual text is printed but that either needs to be added in many places or the output routines need to be wrapped. Are there other control sequences that the current attributes affect (scrolling up perhaps)? > Hm, that's a bit more heavyweight than I was thinking, but would be > the most flexible. It is heavyweight but it already exists. I don't think the tc interface itself is ideal because it is too low-level. Extending the interface with something like zle -T attr doesn't necessarily need to be followed by a function. It's a question of what sort of interface would be useful? I think it would be best if we can assume a modern terminal (i.e. just use \e[22m) but allow old or broken terminals to be worked around from a shell interface. The hard part is designing that interface. > I presume you mean to move the transformation to the main shell and > have zle call it? Loading zle in non-interactive context would be > less desirable? The zle module could register a callback which the main shell would call when changing attributes. Or it could be moved to a separate module. > > * \e[22m selects normal, disabling also faint so naming indicating end > > of bold would be a poor choice. > > That's true, outside of prompt context (which doesn't currently have a > way to express "faint" without using raw sequences). But all the > other attributes are additive whereas faint/normal/bold are mutually > exclusive even at the terminal level, we're not going to be able to > make a general solution for (faint on)(bold on)(bold off) that ends in > (faint on) state. If you consider double underline, underline no underline or slow blink, rapid blink, no blink then the font weight is not actually unique in this regard. It wouldn't be a good idea for a variety of reasons but you could in theory have: %bfaint%Bnormal%Bbold While less terse, it is probably wiser to go with: %A{faint}faint%A{normal}normal%Bbold Oliver
On 16 Dec, you wrote:
> So ... I've been playing with this, and on my "xterm-color" terminal
> se and ue are both $'\e[m' -- and yet "print -P" and ${(%)...} and so
> on do "the right thing" and output extra control sequences depending
> on what other attributes are active. For example
Just noticing these details again, it would seem that zle is currently
broken for your case:
% TERM=xterm-color zsh -df
% zle-line-init() {
region_highlight=( 'P 1 2 underline' 'P 1 4 standout' )
LBUFFER=1234567890
}
% zle -N zle-line-init
Only the 2 is in standout.
Are you sure xterm-color is the right $TERM value to use? Are you
getting that by default or have you set it? Just running xterm gives me
a TERM value of xterm. Setting it to xterm-256color would also work. I
use rxvt-unicode which has display glitches in compinit menus that I
think are zsh issues.
So perhaps we do need to invalidate attributes after turning off
standout and underline if we stick to termcap. I'm still sceptical about
our prompt code doing it for turning bold on.
Oliver
On Wed, Dec 21, 2022 at 9:12 AM Oliver Kiddle <opk@zsh.org> wrote: > > Are you sure xterm-color is the right $TERM value to use? Are you > getting that by default or have you set it? It's what gets set automatically when I ssh from MacOS Terminal.app (xterm-256color) to Ubuntu. > % zle-line-init() { > region_highlight=( 'P 1 2 underline' 'P 1 4 standout' ) > LBUFFER=1234567890 > } > % zle -N zle-line-init > > Only the 2 is in standout. That's correct. > So perhaps we do need to invalidate attributes after turning off > standout and underline if we stick to termcap. I think the discussion started with the premise that we would only use termcap if ANSI sequences don't work, and ANSI sequences can turn on/off specific attributes with the exception of faint/normal/bold ... so theoretically we could optimize out the tracking/restoration of most attributes most of the time? > I'm still sceptical about > our prompt code doing it for turning bold on. Agree.
On Wed, Dec 21, 2022 at 8:46 AM Oliver Kiddle <opk@zsh.org> wrote: > > The original problem was due to mixing raw sequences with prompt > sequences, right? Any time we produce \e[0m we lose everything from > the raw sequences. Sure, but the idea is that we'd stop producing \e[0m and instead output \e[24m to specifically turn off underlining. However, if suppose the prompt code has seen %S%U and then standout (reverse) was disabled by a raw \e[27m, but for %u we output \e[24m\e[07m because we chose not to re-optimize the current code, then we've broken things in the other direction. Ultimately there's no way to keep track of what any raw sequences have done so the best outcome is to do as little as necessary to maintain the state implied by the %-codes encountered so far while also making the state change requested by the next %-code we find. > Something like %u in a prompt will always send the > off sequence regardless of whether underline was previously on. If > we optimise that, we actually make the situation worse for mixtures Right, we always need to send something. We can't optimize (e.g.) %u to less than \e[24m. > underline on might have been done as a raw sequence. The only way to > really solve that would be to parse the raw sequences. Yeah, I'm not suggesting that. It still doesn't work for e.g. using ${(%)%u} to find the underline-off code. > The zle code does things differently. [...] > It'd be good to use the same interface for prompts, preferably also > dropping the "NO" forms of the flags to free up some bits to add more > attributes. Zle has the advantage of always being in control of the entire state, it never has to worry about attributes being toggled "behind it's back" or to figure out what to do with ${(%)...}. > The glitch handling using the "sg" and "ug" termcap sequences also seems > to only be done for prompts but not ZLE. Do you know more about this? I do not know anything about it other than that it's for slow terminals that might not be able to keep up with the control sequences. Wayne would be the expert, I think.