[BUG?] Very strange behavior, execution of a function is interrupted

[BUG?] Very strange behavior, execution of a function is interrupted

[Sebastian Gniazdowski]

Hello,
it's quite a lost game to explain this, but I'll try:

- there is a scheduler called from sched +1 and once from zle -F
- it detects timed-out tasks, moves them to ZPLG_RUN, then executes
them reading them from this array

Problem: sometimes execution never reaches this line and beyond:
https://github.com/zdharma/zplugin/blob/7a24478e5862a1359b6cfb2887a792213be07e3a/zplugin.zsh#L1485

Some cycles of the preceding loop are then also skipped.

I feel very helpless in explaining this, maybe there's standard
questions sheet for situation where bug in exec.c is suspected?

I know `break 2' inside a function will break out of loop that
contains the function call. But I don't know what could exit a
function. I have debug prints and observe this clearly. At the
mentioned line, no execution occurs, ZPLG_RUN isn't cleared, I get to
run some tasks twice.

Twice when I checked, it was happening when zle -F was called so near
next second, that timeout'd tasks detection included two time slots:
"0" (0 seconds after first precmd, handled by zle -F) and "1" (1
second after first precmd, handled by sched +1).
-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] Very strange behavior, execution of a function is interrupted

[Sebastian Gniazdowski]

I've found the cause. `compinit' does `read -q' when occurring
insecure directories. If this happens from `sched' with possibly Zle
being active, then following call inside bin_read() uses Zle's
raw_getbyte() to read a character:

zleentry(ZLE_CMD_GET_KEY, izle_timeout, NULL, &val);

This means that `read' can make all the rich side-effects of the regular
command-line to activate. I think this has significant importance, is
severe. I wouldn't expect the `read' builtin to bring up whole `zle -F'
handling, waiting for inputs, invoking callbacks.

Below test code confirms the above thesis. Its execution is recorded
here: https://asciinema.org/a/199265 Basically, my problem is that
Zplugin's scheduler gets invoked in nested manner because of
compinit's `read -q' (I was doing stress tests with compaudit
activating) and data structures go to improper state.


FD=1337; setup() {
    exec {FD}< <( sleep 1; echo run )
    zle -F $FD -my-scheduler
    echo "Sched call reporting being called"
    read -q "?[y/n]?"
    echo "Sched call reporting exiting"
}
-my-scheduler() {
    local THEFD="$1"; zle -F "$THEFD"; exec {THEFD}<&- # remove zle -F handler
    echo "Zle -F handler reporting being called"
}
sched +1 setup


On Sat, 1 Sep 2018 at 00:01, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> Hello,
> it's quite a lost game to explain this, but I'll try:
>
> - there is a scheduler called from sched +1 and once from zle -F
> - it detects timed-out tasks, moves them to ZPLG_RUN, then executes
> them reading them from this array
>
> Problem: sometimes execution never reaches this line and beyond:
> https://github.com/zdharma/zplugin/blob/7a24478e5862a1359b6cfb2887a792213be07e3a/zplugin.zsh#L1485
>
> Some cycles of the preceding loop are then also skipped.
>
> I feel very helpless in explaining this, maybe there's standard
> questions sheet for situation where bug in exec.c is suspected?
>
> I know `break 2' inside a function will break out of loop that
> contains the function call. But I don't know what could exit a
> function. I have debug prints and observe this clearly. At the
> mentioned line, no execution occurs, ZPLG_RUN isn't cleared, I get to
> run some tasks twice.
>
> Twice when I checked, it was happening when zle -F was called so near
> next second, that timeout'd tasks detection included two time slots:
> "0" (0 seconds after first precmd, handled by zle -F) and "1" (1
> second after first precmd, handled by sched +1).
> --
> Sebastian Gniazdowski
> News: https://twitter.com/ZdharmaI
> IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> Blog: http://zdharma.org



--
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] Very strange behavior, execution of a function is interrupted

[Sebastian Gniazdowski]

Guys to state this more simply but also FUD-like: the little `read -q`
brings up front whole Zle-command-line machinery, with sched, zle -F,
waiting for inputs (select) on multiple file descriptors, running
callbacks. Author of a script that asks y/n with `read -q` would never
expect that he will run `sched' by it, embedding foreign scripts in
middle of his script. This should have some attention, i.e. should be
fixed.
On Sat, 1 Sep 2018 at 21:12, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> I've found the cause. `compinit' does `read -q' when occurring
> insecure directories. If this happens from `sched' with possibly Zle
> being active, then following call inside bin_read() uses Zle's
> raw_getbyte() to read a character:
>
> zleentry(ZLE_CMD_GET_KEY, izle_timeout, NULL, &val);
>
> This means that `read' can make all the rich side-effects of the regular
> command-line to activate. I think this has significant importance, is
> severe. I wouldn't expect the `read' builtin to bring up whole `zle -F'
> handling, waiting for inputs, invoking callbacks.
>
> Below test code confirms the above thesis. Its execution is recorded
> here: https://asciinema.org/a/199265 Basically, my problem is that
> Zplugin's scheduler gets invoked in nested manner because of
> compinit's `read -q' (I was doing stress tests with compaudit
> activating) and data structures go to improper state.
>
>
> FD=1337; setup() {
>     exec {FD}< <( sleep 1; echo run )
>     zle -F $FD -my-scheduler
>     echo "Sched call reporting being called"
>     read -q "?[y/n]?"
>     echo "Sched call reporting exiting"
> }
> -my-scheduler() {
>     local THEFD="$1"; zle -F "$THEFD"; exec {THEFD}<&- # remove zle -F handler
>     echo "Zle -F handler reporting being called"
> }
> sched +1 setup
>
>
> On Sat, 1 Sep 2018 at 00:01, Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > Hello,
> > it's quite a lost game to explain this, but I'll try:
> >
> > - there is a scheduler called from sched +1 and once from zle -F
> > - it detects timed-out tasks, moves them to ZPLG_RUN, then executes
> > them reading them from this array
> >
> > Problem: sometimes execution never reaches this line and beyond:
> > https://github.com/zdharma/zplugin/blob/7a24478e5862a1359b6cfb2887a792213be07e3a/zplugin.zsh#L1485
> >
> > Some cycles of the preceding loop are then also skipped.
> >
> > I feel very helpless in explaining this, maybe there's standard
> > questions sheet for situation where bug in exec.c is suspected?
> >
> > I know `break 2' inside a function will break out of loop that
> > contains the function call. But I don't know what could exit a
> > function. I have debug prints and observe this clearly. At the
> > mentioned line, no execution occurs, ZPLG_RUN isn't cleared, I get to
> > run some tasks twice.
> >
> > Twice when I checked, it was happening when zle -F was called so near
> > next second, that timeout'd tasks detection included two time slots:
> > "0" (0 seconds after first precmd, handled by zle -F) and "1" (1
> > second after first precmd, handled by sched +1).
> > --
> > Sebastian Gniazdowski
> > News: https://twitter.com/ZdharmaI
> > IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> > Blog: http://zdharma.org
>
>
>
> --
> Sebastian Gniazdowski
> News: https://twitter.com/ZdharmaI
> IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> Blog: http://zdharma.org



-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: [BUG?] Very strange behavior, execution of a function is interrupted

[Peter Stephenson]

On Sun, 2 Sep 2018 13:53:40 +0200
Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> Guys to state this more simply but also FUD-like: the little `read -q`
> brings up front whole Zle-command-line machinery, with sched, zle -F,
> waiting for inputs (select) on multiple file descriptors, running
> callbacks.

This should stop the function execution feature if we're just reading a
key, but I haven't checked the details work.

diff --git a/Src/Zle/zle_keymap.c b/Src/Zle/zle_keymap.c
index 2e96ac7..a5cf101 100644
--- a/Src/Zle/zle_keymap.c
+++ b/Src/Zle/zle_keymap.c
@@ -1518,7 +1518,7 @@ getrestchar_keybuf(void)
 	     * arrive together.  If we don't do this the input can
 	     * get stuck if an invalid byte sequence arrives.
 	     */
-	    inchar = getbyte(1L, &timeout);
+	    inchar = getbyte(1L, &timeout, 1);
 	    /* getbyte deliberately resets lastchar_wide_valid */
 	    lastchar_wide_valid = 1;
 	    if (inchar == EOF) {
@@ -1673,7 +1673,7 @@ addkeybuf(int c)
 static int
 getkeybuf(int w)
 {
-    int c = getbyte((long)w, NULL);
+    int c = getbyte((long)w, NULL, 1);
 
     if(c < 0)
 	return EOF;
diff --git a/Src/Zle/zle_main.c b/Src/Zle/zle_main.c
index db70e7d..3487b5d 100644
--- a/Src/Zle/zle_main.c
+++ b/Src/Zle/zle_main.c
@@ -452,7 +452,7 @@ struct ztmout {
  */
 
 static void
-calc_timeout(struct ztmout *tmoutp, long do_keytmout)
+calc_timeout(struct ztmout *tmoutp, long do_keytmout, int full)
 {
     if (do_keytmout && (keytimeout > 0 || do_keytmout < 0)) {
 	if (do_keytmout < 0)
@@ -465,7 +465,7 @@ calc_timeout(struct ztmout *tmoutp, long do_keytmout)
     } else
 	tmoutp->tp = ZTM_NONE;
 
-    if (timedfns) {
+    if (full && timedfns) {
 	for (;;) {
 	    LinkNode tfnode = firstnode(timedfns);
 	    Timedfn tfdat;
@@ -504,7 +504,7 @@ calc_timeout(struct ztmout *tmoutp, long do_keytmout)
 /* see calc_timeout for use of do_keytmout */
 
 static int
-raw_getbyte(long do_keytmout, char *cptr)
+raw_getbyte(long do_keytmout, char *cptr, int full)
 {
     int ret;
     struct ztmout tmout;
@@ -519,7 +519,7 @@ raw_getbyte(long do_keytmout, char *cptr)
 # endif
 #endif
 
-    calc_timeout(&tmout, do_keytmout);
+    calc_timeout(&tmout, do_keytmout, full);
 
     /*
      * Handle timeouts and watched fd's.  If a watched fd or a function
@@ -684,7 +684,7 @@ raw_getbyte(long do_keytmout, char *cptr)
 		     * reconsider the key timeout from scratch.
 		     * The effect of this is microscopic.
 		     */
-		    calc_timeout(&tmout, do_keytmout);
+		    calc_timeout(&tmout, do_keytmout, full);
 		    break;
 		}
 		/*
@@ -810,7 +810,7 @@ raw_getbyte(long do_keytmout, char *cptr)
 # endif
 	    }
 	    /* If looping, need to recalculate timeout */
-	    calc_timeout(&tmout, do_keytmout);
+	    calc_timeout(&tmout, do_keytmout, full);
 	}
 # ifdef HAVE_POLL
 	zfree(fds, sizeof(struct pollfd) * nfds);
@@ -852,7 +852,7 @@ raw_getbyte(long do_keytmout, char *cptr)
 
 /**/
 mod_export int
-getbyte(long do_keytmout, int *timeout)
+getbyte(long do_keytmout, int *timeout, int full)
 {
     char cc;
     unsigned int ret;
@@ -877,7 +877,7 @@ getbyte(long do_keytmout, int *timeout)
 	for (;;) {
 	    int q = queue_signal_level();
 	    dont_queue_signals();
-	    r = raw_getbyte(do_keytmout, &cc);
+	    r = raw_getbyte(do_keytmout, &cc, full);
 	    restore_queue_signals(q);
 	    if (r == -2) {
 		/* timeout */
@@ -956,7 +956,7 @@ getbyte(long do_keytmout, int *timeout)
 mod_export ZLE_INT_T
 getfullchar(int do_keytmout)
 {
-    int inchar = getbyte((long)do_keytmout, NULL);
+    int inchar = getbyte((long)do_keytmout, NULL, 1);
 
 #ifdef MULTIBYTE_SUPPORT
     return getrestchar(inchar, NULL, NULL);
@@ -1021,7 +1021,7 @@ getrestchar(int inchar, char *outstr, int *outcount)
 	 * arrive together.  If we don't do this the input can
 	 * get stuck if an invalid byte sequence arrives.
 	 */
-	inchar = getbyte(1L, &timeout);
+	inchar = getbyte(1L, &timeout, 1);
 	/* getbyte deliberately resets lastchar_wide_valid */
 	lastchar_wide_valid = 1;
 	if (inchar == EOF) {
@@ -2139,7 +2139,7 @@ zle_main_entry(int cmd, va_list ap)
 	do_keytmout = va_arg(ap, long);
 	timeout = va_arg(ap, int *);
 	chrp = va_arg(ap, int *);
-	*chrp = getbyte(do_keytmout, timeout);
+	*chrp = getbyte(do_keytmout, timeout, 0);
 	break;
     }
 
diff --git a/Src/Zle/zle_misc.c b/Src/Zle/zle_misc.c
index 898b552..612ac21 100644
--- a/Src/Zle/zle_misc.c
+++ b/Src/Zle/zle_misc.c
@@ -756,7 +756,7 @@ bracketedstring(void)
     while (endesc[endpos]) {
 	if (current + 1 >= psize)
 	    pbuf = zrealloc(pbuf, psize *= 2);
-	if ((next = getbyte(1L, &timeout)) == EOF)
+	if ((next = getbyte(1L, &timeout, 1)) == EOF)
 	    break;
 	if (!endpos || next != endesc[endpos++])
 	    endpos = (next == *endesc);
@@ -970,7 +970,7 @@ universalargument(char **args)
      *
      * Hence for now this remains byte-by-byte.
      */
-    while ((gotk = getbyte(0L, NULL)) != EOF) {
+    while ((gotk = getbyte(0L, NULL, 1)) != EOF) {
 	if (gotk == '-' && !digcnt) {
 	    minus = -1;
 	    digcnt++;
diff --git a/Src/Zle/zle_vi.c b/Src/Zle/zle_vi.c
index e0923db..a5ff920 100644
--- a/Src/Zle/zle_vi.c
+++ b/Src/Zle/zle_vi.c
@@ -131,7 +131,7 @@ vigetkey(void)
     char m[3], *str;
     Thingy cmd;
 
-    if (getbyte(0L, NULL) == EOF)
+    if (getbyte(0L, NULL, 1) == EOF)
 	return ZLEEOF;
 
     m[0] = lastchar;