[bug report] prompt can erase messages written on the terminal by background processes

[bug report] prompt can erase messages written on the terminal by background processes

[Millian Poquet]

[-- Attachment #1.1: Type: text/plain, Size: 3020 bytes --]

Hi,

I think that I found a zsh prompt display bug that can hide the messages 
printed by some processes on the terminal to end users.
(Please tell me if I should report the bug elsewhere, I sent it to this 
mailing list because 1. I could not find an active zsh bug tracker and 
2. both https://github.com/zsh-users/zsh's README and 
https://sourceforge.net/p/zsh/bugs/ told me to write the bug report here.)

I am no zsh expert so please tell me if this behavior is expected and if 
a configuration option exists to prevent such message loss.


*# Bug description*
It seems that the way zsh displays its prompt enables the loss of 
messages written on the terminal in some cases.
As the bug is visual, I recorded what has been printed on terminal 
session there: https://asciinema.org/a/543461

The attached MWE program `nowait.c` can be compiled with `gcc -std=c11 
-D_POSIX_C_SOURCE=199309L -Wall -Wextra -O2 -o nowait nowait.c` and 
enables to reproduce the error. It has the following behavior.
- The initial process writes a message on the terminal, then creates a 
subprocess (via libc `fork()`), then exits.
- The subprocess sleeps for a given number of milliseconds (or does not 
sleep at all if `argv[1] == 0`), then writes another message on the 
terminal. The written message is terminated by a newline if and only if 
`argv[2] == 1`.

If the subprocess writes on the terminal between the termination of the 
initial process and the writing of the prompt, the content of its 
message is erased by the prompt.
If the message is newline-terminated this is not a big deal, but 
otherwise the whole message will be hidden from the end user of the 
terminal.

I had the issue on the theme I use regularly (that uses `prompt_subst`), 
but I could also reproduce the issue from the default theme.
To do so, I made the prompt message generation slow by setting 
`PROMPT="prompt $(sleep 2)"` and by enabling `prompt_subst`.

*# Desired behavior*
I think that messages displayed by processes on the terminal should not 
be hidden by the shell.

IMHO keeping the printed messages before the prompt would be reasonable, 
but anything that prevents message loss would be an improvement (e.g., 
to copy the messages after erasing the prompt then writing the prompt if 
you think this is better).

*# Execution environment*
I could reproduce the problem on several environments.

On my laptop (dell latitude 3420, x86-64, linux-5.15.72, NixOS 22.05, 
zsh-5.9) both on a tty and on the kitty-0.26.2 graphical terminal 
emulator. Zsh used my configuration there.

On a [Grid'5000 econome 
machine](https://www.grid5000.fr/w/Nantes:Hardware#econome), x86-64, 
linux-5.10.0-19-amd64, Debian 5.10.149-2 (2022-10-21), zsh-5.9. This 
machine has an empty zsh configuration. I was connected on the machine 
remotely via ssh.


Best regards,

-- 
Millian Poquet, MCF (Associate Prof), Univ. Toulouse III
FSI, Dpt. INFO ; IRIT, Sepia team
https://mpoquet.github.io
Room 469, IRIT2, cours Rose Dieng-Kuntz, Campus Univ. Toulouse III

[-- Attachment #1.2: Type: text/html, Size: 4166 bytes --]

[-- Attachment #2: nowait.c --]
[-- Type: text/x-csrc, Size: 789 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
  if (argc != 3) {
    fprintf(stderr, "usage: nowait <DELAY-MILLISECONDS> <CHILD-NEWLINE>\n");
    exit(1);
  }

  unsigned int delay_ms = atoi(argv[1]);
  int child_newline = atoi(argv[2]);

  struct timespec delay;
  delay.tv_sec = delay_ms / 1000;
  delay.tv_nsec = (delay_ms % 1000) * 1000000;

  printf("msg from parent process\n");
  fflush(stdout);

  switch(fork()) {
    case -1:
      perror("fork");
      return 1;
    case 0:
      if (delay_ms > 0)
        nanosleep(&delay, NULL);
      if (child_newline)
        printf("msg from child process\n");
      else
        printf("msg from child process");
      fflush(stdout);
      return 0;
  }

  return 0;
}

Re: [bug report] prompt can erase messages written on the terminal by background processes

[Roman Perepelitsa]

On Wed, Dec 7, 2022 at 8:04 PM Millian Poquet <millian.poquet@irit.fr> wrote:
>
> prompt can erase messages written on the terminal by background processes

Not just prompt but zle in general, or any program really that moves
the cursor (try `top`, for example, or `less`). There is no way around
it. Zle assumes that nobody else prints to the terminal. If this
assumption is violated, the outcome is unspecified mess.

Here's a simpler demonstration. Run this in interactive zsh:

  ( sleep 3; print -rn hello ) &!

Then immediately type "ls" and wait without hitting enter. A moment
later "hello" will appear on the command line. Try editing it and
observe that zle is essentially broken.

You can break any terminal program that moves the cursor by using this
trick. Try running `top` immediately after the command listed above,
or after you C program, which is essentially the same.

Roman.

Re: [bug report] prompt can erase messages written on the terminal by background processes

[Bart Schaefer]

On Wed, Dec 7, 2022 at 2:57 PM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Wed, Dec 7, 2022 at 8:04 PM Millian Poquet <millian.poquet@irit.fr> wrote:
> >
> > prompt can erase messages written on the terminal by background processes
>
> There is no way around it. Zle assumes that nobody else prints to the terminal.

Perhaps try this.

get_cursor_pos() {
  print -n $'\e[6n'
  IFS=$'\e[;' read -s -d R -A ${1:-reply}
  shift 3 ${1:-reply}
}
precmd() {
  local rowcol
  get_cursor_pos rowcol
  if (( $rowcol[1] > 1 ))
  then print
  fi
}

Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Stephane Chazelas]

2022-12-07 19:46:37 -0800, Bart Schaefer:
[...]
> get_cursor_pos() {
>   print -n $'\e[6n'

Note that "print" already does \expansions without -r.

>   IFS=$'\e[;' read -s -d R -A ${1:-reply}

The query sequence could be sent using the prompt string (on
stderr):

IFS=$'\e[;' read -sd R -A ${1-reply}$'?\e[6n'

Then assuming "read -s" does the right thing and disables
line discipline echo before printing the prompt, that would
remove the risk of the terminal's reply coming before the echo
is disabled.

curpos() {
  IFS=$'\e[;' read -rsdR -t0.2 1$'?\e[6n' 1 ${2-y} ${1-x} 1 <> /dev/tty 2>&0
}
curpos col row


>   shift 3 ${1:-reply}

Shouldn't that be shift 2?

terminal should reply \e[y;xR, which would be split into ('' '' y x).

Or do a strict parsing of the reply:

curpos() {
  set -o localoptions -o extendedglob
  local match answer
  IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
    [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
    eval "${1-x}=\$match[2] ${2-y}=\$match[1]"

-- 
Stephane

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Roman Perepelitsa]

On Thu, Dec 8, 2022 at 9:21 AM Stephane Chazelas <stephane@chazelas.org> wrote:
>
> curpos() {
>   set -o localoptions -o extendedglob
>   local match answer
>   IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
>     [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
>     eval "${1-x}=\$match[2] ${2-y}=\$match[1]"
> }

Functions that accept output parameter names as arguments are tricky.
The following won't work:

    # Which line is the cursor on?
    curpos _ answer
    print -r -- "The cursor is on the line number $answer"

Roman.

Re: [bug report] prompt can erase messages written on the terminal by background processes

[Roman Perepelitsa]

On Thu, Dec 8, 2022 at 4:46 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Wed, Dec 7, 2022 at 2:57 PM Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > On Wed, Dec 7, 2022 at 8:04 PM Millian Poquet <millian.poquet@irit.fr> wrote:
> > >
> > > prompt can erase messages written on the terminal by background processes
> >
> > There is no way around it. Zle assumes that nobody else prints to the terminal.
>
> Perhaps try this.
>
> get_cursor_pos() {
>   print -n $'\e[6n'
>   IFS=$'\e[;' read -s -d R -A ${1:-reply}
>   shift 3 ${1:-reply}
> }
> precmd() {
>   local rowcol
>   get_cursor_pos rowcol
>   if (( $rowcol[1] > 1 ))
>   then print
>   fi
> }

Here's the simplified version of Millian's test:

    % setopt prompt_subst; PS1='%# $(sleep 2)'; ( sleep 1; print -n hello ) &!

This prints "hello" and then erases it. The precmd hook runs before
prompt expansion, so it has no effect.

Even if this somehow worked for this test case, it would merely mask
the problem. Zle has to assume that nobody else writes to the TTY.
It's user error to violate this assumption.

Roman.

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Stephane Chazelas]

2022-12-08 09:34:41 +0100, Roman Perepelitsa:
[...]
> > curpos() {
> >   set -o localoptions -o extendedglob
> >   local match answer
> >   IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
> >     [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
> >     eval "${1-x}=\$match[2] ${2-y}=\$match[1]"
> > }
> 
> Functions that accept output parameter names as arguments are tricky.
> The following won't work:
> 
>     # Which line is the cursor on?
>     curpos _ answer
>     print -r -- "The cursor is on the line number $answer"
[...]

Yes, I thought of using 3 as the intermediary variable, but
decided against it as we can't do anything about match,
mbegin... anyway.

typeset -g "${1-x}=$match[2]" "${2-y}=$match[1]"

Doesn't work either. I guess there's no way to access variables
by the same name in that parent scope or to unlocal a variable?

(bash does by exploiting some bug (calling unset -v in a
severate function peels one layer of scoping instead of
unsetting in the current scope)).

-- 
Stephane

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Stephane Chazelas]

2022-12-08 10:02:15 +0000, Stephane Chazelas:
[...]
> (bash does by exploiting some bug (calling unset -v in a
> severate function peels one layer of scoping instead of
> unsetting in the current scope)).
[...]

Not sure how "separate" became "severate" there. Sometimes it
feels like my fingers have a life of their own... Sorry.

unlocal in bash (or yash) can be written:

unlocal() { unset -v "$@"; }

(that's one of the main reasons POSIX couldn't specify "local"
btw).

It's worse in mksh where unset is always unlocal.

$ bash -c 'unlocal() { unset -v "$@"; }; f() { local a=1; g; echo "f: $a"; }; g() { local a=2; unlocal a; echo "g: $a"; a=3; }; a=0; f'
g: 1
f: 3
$ yash -c 'unlocal() { unset -v "$@"; }; f() { local a=1; g; echo "f: $a"; }; g() { local a=2; unlocal a; echo "g: $a"; a=3; }; a=0; f'
g: 1
f: 3

$ zsh -c 'unlocal() { unset -v "$@"; }; f() { local a=1; g; echo "f: $a"; }; g() { local a=2; unlocal a; echo "g: $a"; a=3; }; a=0; f'
g:
f: 1
$ dash -c 'unlocal() { unset -v "$@"; }; f() { local a=1; g; echo "f: $a"; }; g() { local a=2; unlocal a; echo "g: $a"; a=3; }; a=0; f'
g:
f: 1
$ bosh -c 'unlocal() { unset -v "$@"; }; f() { local a=1; g; echo "f: $a"; }; g() { local a=2; unlocal a; echo "g: $a"; a=3; }; a=0; f'
g:
f: 1

-- 
Stephane

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Mikael Magnusson]

On 12/8/22, Stephane Chazelas <stephane@chazelas.org> wrote:
> 2022-12-08 09:34:41 +0100, Roman Perepelitsa:
> [...]
>> > curpos() {
>> >   set -o localoptions -o extendedglob
>> >   local match answer
>> >   IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
>> >     [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
>> >     eval "${1-x}=\$match[2] ${2-y}=\$match[1]"
>> > }
>>
>> Functions that accept output parameter names as arguments are tricky.
>> The following won't work:
>>
>>     # Which line is the cursor on?
>>     curpos _ answer
>>     print -r -- "The cursor is on the line number $answer"
> [...]
>
> Yes, I thought of using 3 as the intermediary variable, but
> decided against it as we can't do anything about match,
> mbegin... anyway.
>
> typeset -g "${1-x}=$match[2]" "${2-y}=$match[1]"
>
> Doesn't work either. I guess there's no way to access variables
> by the same name in that parent scope or to unlocal a variable?

I have a patch for it, but I never sent it because I couldn't think of
a usecase. http://comm.it.cx/cgit/zsh-cvs/patch/?id=cf1ba693c9f3ed80ff085bfed74d854796349a0d
(I'm not 100% sure all the error checks are correct, and cover all
cases, especially wrt bart's private module).

-- 
Mikael Magnusson

Re: [bug report] prompt can erase messages written on the terminal by background processes

[Oliver Kiddle]

Bart Schaefer wrote:
> > There is no way around it. Zle assumes that nobody else prints to the terminal.
>
> Perhaps try this.
>
> get_cursor_pos() {
>   print -n $'\e[6n'
>   IFS=$'\e[;' read -s -d R -A ${1:-reply}

Aside from only working if the timing of the background process involves
writing to the terminal before the prompt is printed this approach also
has another disadvantage: the read can swallow any text that has been
typed before it prints the prompt. To see that, run something like sleep
3 and type your next command while it sleeps.

I've used a similar approach as a fallback mechanism[1] to detect a dark or
light background by printing '\e]11;?\a'. Typeahead is rarer for a new
window but I did add handling to try to grab the typeahead and restore
it with print -z but it still gets messed up occasionally; and I can't
be sure that all the typeahead was intended for zsh.

It might be better to report a bug for whatever programme you have that
forks and continues to write to the terminal from the child process
while the parent quits.

Oliver

[1] The primary mechanism is to look at $COLORFGBG which I typically add
to the list of variables that ssh/sshd will pass/accept.

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Bart Schaefer]

On Thu, Dec 8, 2022 at 12:21 AM Stephane Chazelas <stephane@chazelas.org> wrote:
>
> 2022-12-07 19:46:37 -0800, Bart Schaefer:
> >   IFS=$'\e[;' read -s -d R -A ${1:-reply}
> >   shift 3 ${1:-reply}
>
> Shouldn't that be shift 2?

Yeah, I changed the function to only return the column and then
neglected to edit all the names before copy-pasting.  Note precmd
referenced subscript [1] not [2].

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Bart Schaefer]

On Thu, Dec 8, 2022 at 2:02 AM Stephane Chazelas <stephane@chazelas.org> wrote:
>
> typeset -g "${1-x}=$match[2]" "${2-y}=$match[1]"
>
> Doesn't work either. I guess there's no way to access variables
> by the same name in that parent scope or to unlocal a variable?

trap "typeset -g '${1-x}=$match[2]' '${2-y}=$match[1]'" EXIT

Of course that only works if the assignments are going to be the last
thing in the function.  Other quoting games are probably needed to
generalize.

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Philippe Altherr]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

>
> typeset -g "${1-x}=$match[2]" "${2-y}=$match[1]"


> Doesn't work either. I guess there's no way to access variables
> by the same name in that parent scope or to unlocal a variable?


I just stumbled on the module zsh/param/private
<https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-zsh_002fparam_002fprivate-Module>,
which makes it possible (but I won't claim that it's very practical):

assign() {
>     print -v "$1" "$2"
> }
>
> curpos() {
>     set -o localoptions -o extendedglob
>     # Initialize the return variables. Without this, print -v fails if
>     # the variable is the same as one of the private variables.
>     eval "${1-x}= ${2-y}="
>     zmodload zsh/param/private
>     local -P -a match mbegin mend
>     local -P answer
>     IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
>         [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
>         assign ${1-x} "$match[2]" &&
>         assign ${2-y} "$match[1]"
> }
>
> curpos answer match
> echo x=$answer y=$match
>

It's a bit strange that the initialization is needed. Private variables
don't hide global variables of the same name, nor prevent their assignment,
but prevent their initial declaration.

Philippe

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Bart Schaefer]

On Fri, Dec 9, 2022 at 4:46 AM Philippe Altherr
<philippe.altherr@gmail.com> wrote:
>
> I just stumbled on the module zsh/param/private, which makes it possible (but I won't claim that it's very practical):
>[...]
> It's a bit strange that the initialization is needed. Private variables don't hide global variables of the same name, nor prevent their assignment, but prevent their initial declaration.

This isn't exactly an effect of private, it's the previously observed
effect that there is no way to "unlocal" a parameter in the current
scope, so you have to add it to the desired scope before declaring it
local.

> assign() {
>     print -v "$1" "$2"
> }

You don't actually need a named function or print -v here, although I
guess "print -v" is shorter to write than "typeset -g".

curpos() {
    set -o localoptions -o extendedglob
    eval "${1-x}= ${2-y}="
    zmodload zsh/param/private
    local -P -a match mbegin mend
    local -P answer
    IFS= read -rsdR -t0.2 answer$'?\e[6n' &&
        [[ $answer = (#b)$'\e['(<->)';'(<->) ]] &&
        () { typeset -g "$1"="$2" } ${1-x} "$match[2]" &&
        () { typeset -g "$1"="$2" } ${2-y} "$match[1]"
}

I'm torn about that eval-as-delcarator.  My first thought would be to use

  typeset -gi ${1-x} ${2-y}

but the eval avoids changing any declared flags on existing parameters
in the enclosing scope.  On the other hand both that eval and the use
of "print -v" force the variables to be scalar, whereas "typeset -g"
throws errors on assignment mismatch.

Private variables not private enough? (Was: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes))

[Philippe Altherr]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

My code was mainly a proof of concept. I didn't put much thought into
questions like "print -v" vs "typeset -g". I was just pleasantly
surprised that the whole thing was at all possible :-)

> It's a bit strange that the initialization is needed. Private variables
> don't hide global variables of the same name, nor prevent their assignment,
> but prevent their initial declaration.


> This isn't exactly an effect of private, it's the previously observed
> effect that there is no way to "unlocal" a parameter in the current
> scope, so you have to add it to the desired scope before declaring it
> local.


Private variables are supposed to have a scope that is limited to the
function where they are declared. Intuitively that implies that they should
have no impact on the behavior of callees. However, that's not always the
case. Consider the following example:

zmodload zsh/param/private


> (() { v=;               () { typeset -g v=z } }; echo 1:v=$v)
> (() { v=;               () { print   -v v z } }; echo 2:v=$v)
> (() {                   () { typeset -g v=z } }; echo 3:v=$v)
> (() {                   () { print   -v v z } }; echo 4:v=$v)


> (() { v=; local -P v=x; () { typeset -g v=z } }; echo 5:v=$v)
> (() { v=; local -P v=x; () { print   -v v z } }; echo 6:v=$v)
> (() {     local -P v=x; () { typeset -g v=z } }; echo 7:v=$v)
> (() {     local -P v=x; () { print   -v v z } }; echo 8:v=$v)


Here is the result

1:v=z
> 2:v=z
> 3:v=z
> 4:v=z
> 5:v=z
> 6:v=z
> 7:v=
> (anon): v: attempt to assign private in nested scope


Case 1 to 4 show that both "typeset -g" and "print -v" are able to set a
global variable independent of whether the variable already exists (case
1-2) or not (case 3-4).

Case 5 and 6 show that "typeset -g" and "print -v" are able to set an
already existing global variable even if there is a private variable with
the same name in a parent scope. In these cases, the private variable is
truly private and doesn't affect the behavior of the callee.

Case 7 and 8 show that the same isn't true if the global variable doesn't
already exist. I find this very counterintuitive. With truly private
variables, I don't see why case 7 and 8 should exhibit a different behavior
than case 3 and 4.

Philippe

[-- Attachment #2: Type: text/html, Size: 4023 bytes --]

Re: Private variables not private enough? (Was: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes))

[Bart Schaefer]

On Sat, Dec 10, 2022 at 7:03 AM Philippe Altherr
<philippe.altherr@gmail.com> wrote:
>
> Case 1 to 4 show that both "typeset -g" and "print -v" are able to set a global variable independent of whether the variable already exists (case 1-2) or not (case 3-4).

This is true if the variable is not already declared local at some
enclosing scope.  Shell variables are dynamically scoped, defaulting
to global except when in the argument list of "typeset" and related
declarations.  Despite the apparent mnemonic, the only effect of -g is
to skip the current scope before selecting the enclosing dynamic
scope.

> Case 7 and 8 show that the same isn't true if the global variable doesn't already exist. I find this very counterintuitive. With truly private variables, I don't see why case 7 and 8 should exhibit a different behavior than case 3 and 4.
>> 7:v=
>> (anon): v: attempt to assign private in nested scope

Case 7 was a bug fixed by workers/49456.  Case 8 has the wrong error
message.  In zsh-5.9, both 7 and 8 produce the error

(anon): v: can't change parameter attribute

This is documented (last sentence of this paragraph):

   * Within any other function called by the declaring function, the
     private parameter does _NOT_ hide other parameters of the same
     name, so for example a global parameter of the same name is visible
     and may be assigned or unset.  This includes calls to anonymous
     functions, although that may also change in the future.  However,
     the private name may not be created outside the local scope when it
     was not previously declared.

This limitation is enforced by the internal implementation of dynamic
scoping in zsh as a stack of parameter declarations.  There's no way
to "unshift" a new element onto the stack at an enclosing scope.

Re: Private variables not private enough? (Was: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes))

[Bart Schaefer]

On Sat, Dec 10, 2022 at 9:36 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> Despite the apparent mnemonic, the only effect of [typeset] -g is
> to skip the current scope before selecting the enclosing dynamic
> scope.

To clarify, the chosen dynamic scope is the closest one in which the
parameter name already exists, or the global scope if the parameter
name is not present in any other scope.

Re: Get cursor position (Was: [bug report] prompt can erase messages written on the terminal by background processes)

[Stephane Chazelas]

2022-12-09 13:46:21 +0100, Philippe Altherr:
[...]
> assign() {
> >     print -v "$1" "$2"
> > }
[...]

Note that that "assign" function has a command injection
vulnerability.

Even more  so than for other  commands, -- (- also works) should
always be used for "print" to separate options from non-options
at least when the first non-option argument is not guaranteed
not to start with - or +.

Try for instance:

assign var '-va[1$(reboot)]'

So:

assign() print -rv "$1" -- "$2"

Or the Bourne-compatible:

assign() eval "$1=\$2"

Or POSIX:

assign() { eval "$1=\$2"; }

A bit ironic that people often go to great lengths to avoid
using "eval" but end up coming up with unsafe solutions.

-- 
Stephane