one time in 20 error

one time in 20 error

[Ray Andrews]

I have a function that uses zcurses that works fine about 19 times out 
of 20 but occasionally, on returning, it leaves something like this on 
the command line:

#zsh: permission denied: 1
#^[[<0;24;4m#
$ 0;24;4m

... the first number is always zero, the second is always two digits and 
the third one can be one or two digits.  The semi-colons and the 'm' 
never change.  Does anyone have the slightest clue what might be going 
on?  They look like color codes sorta.  When it happens seems completely 
random and it's frustratingly rare, so hard to even begin to diagnose.  
When it happens nothing seems amiss subsequently -- nothing seems to be 
broken.

Re: one time in 20 error

[Roman Perepelitsa]

On Mon, Nov 28, 2022 at 6:41 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I have a function that uses zcurses that works fine about 19 times out
> of 20 but occasionally, on returning, it leaves something like this on
> the command line:
>
> #zsh: permission denied: 1
> #^[[<0;24;4m#
> $ 0;24;4m

This might be a response by the TTY to a query that your script has
sent to it. Basically, if the script queries the TTY and exits before
reading the response, then the response goes to zle and you see this
kind of garbage on the command line.

^[[<0;24;4m is a CSI sequence. I don't remember its meaning from the
top of my head.

Roman.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-28 09:54, Roman Perepelitsa wrote:
>
> This might be a response by the TTY to a query that your script has
> sent to it. Basically, if the script queries the TTY and exits before
> reading the response, then the response goes to zle and you see this
> kind of garbage on the command line.
>
> ^[[<0;24;4m is a CSI sequence. I don't remember its meaning from the
> top of my head.

I thought it might be something of that general nature.  I wonder, is 
there some strategic place where I might put a 'sleep' to let things 
sort themselves out?  Probably only would take a millisecond or two.  
Like after 'zcurses end' or something?  It's always on return, never 
within the executing program.  As to the strings, within the rules I 
mentioned the numbers vary:  0;22;3m 0;18;10m 0;15;14m... and so on, so 
any exact meaning isn't likely to be found.  It seems random, like a 
dangling pointer error.

Re: one time in 20 error

[Roman Perepelitsa]

On Mon, Nov 28, 2022 at 9:34 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> On 2022-11-28 09:54, Roman Perepelitsa wrote:
> >
> > This might be a response by the TTY to a query that your script has
> > sent to it. Basically, if the script queries the TTY and exits before
> > reading the response, then the response goes to zle and you see this
> > kind of garbage on the command line.
> >
> > ^[[<0;24;4m is a CSI sequence. I don't remember its meaning from the
> > top of my head.
>
> I thought it might be something of that general nature.  I wonder, is
> there some strategic place where I might put a 'sleep' to let things
> sort themselves out?  Probably only would take a millisecond or two.

You can try this:

    () {
      zmodload zsh/system
      local REPLY
      while sysread -t 0.1; do done
    }

This reminds me of a student in a C programming class who was
frustrated by the pesky "core dumped" message printed after his
program succeeded. He redirected stderr to /dev/null
and--Voilà--problem solved!

The teacher wasn't convinced by this "solution" and neither should you.

Roman.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-28 12:46, Roman Perepelitsa wrote:
> You can try this:
>      () {
>        zmodload zsh/system
>        local REPLY
>        while sysread -t 0.1; do done
>      }
>
How do I use that?

> The teacher wasn't convinced by this "solution" and neither should you.

Not my style.  It's not fixed until it's fixed.  I like your idea that 
it's a timing issue tho, why else would it be trouble free 95% of the 
time?  As you say, something it getting bothered vis a vis the terminal.

BTW, Bart was right (sooprise!) about 'err_exit' sheesh, it nukes the 
entire terminal.  I was sorta thinking 'err_return'.

Re: one time in 20 error

[Roman Perepelitsa]

On Mon, Nov 28, 2022 at 11:16 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> On 2022-11-28 12:46, Roman Perepelitsa wrote:
> > You can try this:
> >      () {
> >        zmodload zsh/system
> >        local REPLY
> >        while sysread -t 0.1; do done
> >      }
> >
> How do I use that?

You invoke it right after the script that has the particular bug
you've described above.

> > The teacher wasn't convinced by this "solution" and neither should you.
>
> Not my style.

Then don't use it. It's not a fix for the bug, it just masks it.

Roman.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-28 14:24, Roman Perepelitsa wrote:
> Then don't use it. It's not a fix for the bug, it just masks it. 

I'll hang on to it anyway.  Next time the bug surfaces I'll throw your 
code at it and maybe learn something.  Never used that module so I'm 
bound to learn something.


> Roman.
>

Re: one time in 20 error

[Risto Laitinen]

It is a mouse event (SGR report mode)
CSI < p1 ; p2 ; p3 m   button release
CSI < p1 ; p2 ; p3 M   button press or move
p1 = button + modifiers
p2 = column
p3 = row

https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Mouse-Tracking

(Sent this just to Ray first by accident, apologies for the dupe.)

On Mon, Nov 28, 2022 at 7:41 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I have a function that uses zcurses that works fine about 19 times out
> of 20 but occasionally, on returning, it leaves something like this on
> the command line:
>
> #zsh: permission denied: 1
> #^[[<0;24;4m#
> $ 0;24;4m
>
> ... the first number is always zero, the second is always two digits and
> the third one can be one or two digits.  The semi-colons and the 'm'
> never change.  Does anyone have the slightest clue what might be going
> on?  They look like color codes sorta.  When it happens seems completely
> random and it's frustratingly rare, so hard to even begin to diagnose.
> When it happens nothing seems amiss subsequently -- nothing seems to be
> broken.
>
>
>
>

Re: one time in 20 error

[Ray Andrews]

On 2022-11-28 18:48, Risto Laitinen wrote:
> It is a mouse event (SGR report mode)
> CSI < p1 ; p2 ; p3 m   button release
> CSI < p1 ; p2 ; p3 M   button press or move
> p1 = button + modifiers
> p2 = column
> p3 = row,

Well  that makes sense.  Using the mouse is exactly what I'm doing 
there.  But all mouse events are completing normally.  It's always 'm', 
I wonder what that tells us.  Anyway, just trying now to force the error 
with slow clicks and such, but no luck. Anyway that's a hot tip, many 
thanks.

Re: one time in 20 error

[Roman Perepelitsa]

On Tue, Nov 29, 2022 at 3:49 AM Risto Laitinen <risto.laitinen@gmail.com> wrote:
>
> It is a mouse event (SGR report mode)

Thanks!

So here's what's probably happening. The script is enabling mouse
tracking, the user clicks with the mouse, the script disables mouse
tracking and exits before reading the mouse event, zle reads the mouse
event.

The script needs to read from the TTY after disabling mouse tracking.
Reading with timeout is suboptimal because there is no upper bound on
input latency (imagine working over SSH). A better solution is to send
a query and read the response. Something like this perhaps:

    function cleanup() {
      emulate -L zsh -o extended_glob
      # Disable mouse tracking.
      print -n '\e[?1000l'
      # Disable SGR mouse mode.
      print -n '\e[?1006l'
      # Query the TTY (Primary Device Identification).
      print -n '\e[0c'
      # Read stdin until we find the response to our query.
      local resp
      while [[ $resp != *$'\e[?'(<->\;)#<->c ]] &&
            read -skr 'resp[$#resp+1]'; do
        # Do nothing.
      done
    }

This function needs to be invoked before the script exits.

There is probably a better solution to this problem.

Roman.

P.S.

I have the following in my .zshrc:

    function skip-csi-sequence() {
     local key
     while read -sk key && (( $((#key)) < 0x40 || $((#key)) > 0x7E )); do
       # empty body
     done
    }

    zle -N skip-csi-sequence
    bindkey '\e[' skip-csi-sequence

With this binding a buggy script that leaks mouse events into zle
won't have any effect.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-29 02:13, Roman Perepelitsa wrote:
>
> The script needs to read from the TTY after disabling mouse tracking.
> Reading with timeout is suboptimal because there is no upper bound on
> input latency (imagine working over SSH). ...

Yikes, that was an order of magnitude over my head.  Grasping at straws 
I just added a 'zcurses timeout my_window' before 'zcurses end' and it 
ran 60 cycles with nothing written to zle ... so I'm calling it good.  
Dunno, I think my code is clean, it's all according to Hoyle as far as 
the docs go; I don't think I've done anything wrong, perhaps 'zcurses 
end' should take more care to clean up?  Flush whatever buffers just in 
case?

Re: one time in 20 error

[Bart Schaefer]

On Tue, Nov 29, 2022 at 5:29 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I just added a 'zcurses timeout my_window' before 'zcurses end'

This seems entirely appropriate.

> perhaps 'zcurses
> end' should take more care to clean up?

I suspect that your program is taking action on mouse-down, so it
really is up to you to also handle the following mouse-up.  There's no
way for curses endwin() or by extension "zcurses end" to know that the
extra characters were generated by the terminal or were typed by you
(with the expectation that they'd go to the regular shell input).

Re: one time in 20 error

[Ray Andrews]

On 2022-11-29 20:32, Bart Schaefer wrote:
> perhaps 'zcurses
>> end' should take more care to clean up?
> I suspect that your program is taking action on mouse-down, so it
> really is up to you to also handle the following mouse-up.
I'm capturing both PRESSED and CLICKED, the former being a slow click in 
practice.  I had the same idea tho, the mouse-up is dangling.  What's 
puzzling is that I can't force the error.  I can press and hold the 
button, wait for the program to end (so far mouse clicks always make a 
selection from a list and return) and then release the button and, 
again, maybe one time in 20 I'll get the bogey.  I've tried variation on 
click speed but nothing  makes it better or worse.   You don't suppose 
it could be a mechanical issue with the mouse?  You know how they 
sometimes get sticky and mouse clicks are poorly registered?  I've had 
mice that tended to return two clicks when only one was made.  A 
mechanically unclean release seems to confuse the computer.
> There's no
> way for curses endwin() or by extension "zcurses end" to know that the
> extra characters were generated by the terminal or were typed by you
> (with the expectation that they'd go to the regular shell input).

Well, so far the 'timeout' is working 100%, still I myself would sorta 
build that in to 'zcurses end'.  While in zcurses,it owns all inputs so 
on quitting the idea that some input would be 'forwarded' to the 
terminal seems strange.  Dunno, maybe such a thing could be wanted but 
... well, could it ever be wanted?  As you say there are these 
'generated by the terminal' ... but I don't know anything about such 
things.  Seems to me the terminal receives input, not makes input but I 
suppose an xterm is itself a program that writes to hardware so ... But 
this is the only time I've ever seen ghost-writing to zle.

BTW you were right about ERR_EXIT, sheesh, it nukes the whole terminal.  
I had something like ERR_RETURN in  mind but I guess we don't have that.

Re: one time in 20 error

[Roman Perepelitsa]

On Wed, Nov 30, 2022 at 2:57 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> I can press and hold the button, wait for the program to end (so far mouse
> clicks always make a selection from a list and return) and then release the
> button and, again, maybe one time in 20 I'll get the bogey.  I've tried
> variation on click speed but nothing  makes it better or worse.   You don't
> suppose it could be a mechanical issue with the mouse?  You know how they
> sometimes get sticky and mouse clicks are poorly registered?  I've had mice
> that tended to return two clicks when only one was made.  A mechanically
> unclean release seems to confuse the computer.

What happens is that you click while the script is running and
listening to mouse events, this causes the terminal to send you data,
which can take arbitrary long to arrive to your script's stdin. Then
the script deactivates mouse tracking and exists, and then the data
from the past click reaches stdin. Clicks after the script exits won't
generate any data because mouse tracking is already disabled.

> Well, so far the 'timeout' is working 100%

It won't work if you run your script over SSH with high latency.
Moreover, it adds lag every time the script exists. A better solution
is what I posted above. It doesn't add more latency than necessary and
will correctly clear stdin with any amount of latency between your
script and the terminal. The code I posted needs to be run when the
script ends. If your script disables mouse tracking on its own, you
can remove the first two `print` statements from my code snippet.

Roman.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-30 06:10, Roman Perepelitsa wrote:
> Well, so far the 'timeout' is working 100%
> It won't work if you run your script over SSH with high latency.

Very unlikely!  Anyway I've  saved your code for future experiments.  
I'll also see what can be done executing on button-up vs. button-down.

Re: one time in 20 error

[Ray Andrews]

If I use:

zcurses mouse delay 0

... then I get the characters written to zle every time. According to 
the doc, if I set a long delay then I should have longer between press 
and release and still have it interpreted as a click, but no luck 
there.  And I can't get 'RELEASE' to show up.  How to force waiting for 
the release?

Re: one time in 20 error

[Bart Schaefer]

On Wed, Nov 30, 2022 at 3:17 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> And I can't get 'RELEASE' to show up.  How to force waiting for
> the release?

This has ceased to be a zsh question and become a curses question.

Re: one time in 20 error

[Bart Schaefer]

On Wed, Nov 30, 2022 at 5:56 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Well, so far the 'timeout' is working 100%, still I myself would sorta
> build that in to 'zcurses end'.  While in zcurses,it owns all inputs so
> on quitting the idea that some input would be 'forwarded' to the
> terminal seems strange.

Suppose you have a curses program that's controlled from the keyboard,
and when you type shift-Q that program calls "zcurses end".  Now, you
happen to be a 300-words-per-minute typist, so upon hitting Q you
instinctively begin typing the first word of the next command that you
expect to execute, without waiting for a prompt.  Should zcurses
swallow up all or part of that command because it "owns" the input?

> As you say there are these
> 'generated by the terminal' ... but I don't know anything about such
> things.  Seems to me the terminal receives input, not makes input

Everything about ANSI terminal control is unfortunately "in-band":
There is only one data stream between whatever program is using the
terminal (zsh in this case) and the terminal itself, which
(importantly) includes the keyboard.  All those weird sequences
starting with ^[[ or similar are still sent to (and from!) the
terminal via the same stream as the prompt or the output of "ls" or
your typing or copy-pasting or whatever.  In the "to" direction, the
terminal notices ^[[ followed by something, and interprets it as a
command (change the color, move the cursor, etc.) instead of as
"normal" characters to be displayed.  Some of those commands ask the
terminal for data (tell me the cursor position, tell me where the
mouse was clicked, etc.).  In that case data is sent in the "from"
direction:  the terminal writes ^[[ followed by the requested data,
and the program using the terminal is expected to recognize the ^[[ as
"not normal" and read it -- but there's nothing else to distinguish
that from something that was typed on the keyboard, it's all on the
same stream.  So unless the program is waiting for one of those ^[[
sequences, it responds as if you typed it.

One of Roman's hacks (his skip-csi-sequence function) is to make zsh
ALWAYS be "waiting for" one of those sequences, so as to throw it away
if it wasn't really requested.

Re: one time in 20 error

[Ray Andrews]

On 2022-11-30 16:03, Bart Schaefer wrote:
> but there's nothing else to distinguish
> that from something that was typed on the keyboard, it's all on the
> same stream.

I understand.  I know it's complicated.  Still one should be able to 
choose to dump whatever buffer might have some pending stream. But as I 
said, the 'timeout' seems to do exactly that.  It's quite functional, 
the rest is curiosity at this point ... like why I never see 'RELEASE'.  
It has to happen, so I'll figure out why I can't see it.


> This has ceased to be a zsh question and become a curses question.

I don't know how much of zcurses is different from whatever it descends from, but it seems to be within scope as a question.  Nevermind tho, if anyone's interested that's fine, if not, I'm functional.

Re: one time in 20 error

[Roman Perepelitsa]

On Thu, Dec 1, 2022 at 4:21 AM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Still one should be able to choose to dump whatever buffer
> might have some pending stream.

One is able to do exactly that. It's honestly quite frustrating trying
to help you.

Roman.

Re: one time in 20 error

[Ray Andrews]

On 2022-12-01 01:19, Roman Perepelitsa wrote:
>
> One is able to do exactly that. It's honestly quite frustrating trying
> to help you.

Still you almost always succeed.

Re: one time in 20 error

[Philippe Altherr]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

On Wed, Nov 30, 2022 at 2:56 PM Ray Andrews <rayandrews@eastlink.ca> wrote:

> BTW you were right about ERR_EXIT, sheesh, it nukes the whole terminal.
> I had something like ERR_RETURN in  mind but I guess we don't have that.


Actually there is an ERR_RETURN, it's described here
<https://zsh.sourceforge.io/Doc/Release/Options.html#index-ERRRETURN>.

Philippe

[-- Attachment #2: Type: text/html, Size: 684 bytes --]

Re: one time in 20 error

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]


On 2022-12-01 06:42, Philippe Altherr wrote:
> On Wed, Nov 30, 2022 at 2:56 PM Ray Andrews <rayandrews@eastlink.ca> 
> wrote:
>
>     BTW you were right about ERR_EXIT, sheesh, it nukes the whole
>     terminal.
>     I had something like ERR_RETURN in  mind but I guess we don't have
>     that.
>
>
> Actually there is an ERR_RETURN, it's described here 
> <https://zsh.sourceforge.io/Doc/Release/Options.html#index-ERRRETURN>.

That's embarrassing, I could have checked!  Thanks Philippe


>
> Philippe
>

[-- Attachment #2: Type: text/html, Size: 1751 bytes --]