typeahead problem

typeahead problem

[Bernd Eggink]

A sticking 'y' key on my keyboard revealed a weird zle behaviour which I
first assumed to be a bug, but after glancing through the sources turned
out to be a feature. In a script I have to process a long list of items,
and for each item there is a statement like

  read -q "REPLY?Yes or no: " && do_something

Now if "do_something" takes some time and you type at least one
character in advance, 'read -q' behaves as if you are constantly typing
'n', until you consume the pending character by a normal 'read'. In
other words, if you do NOT issue a normal read, EVERY following 'read
-q' will behave as if you had typed 'n', until the end of the script.

Is there a rationale for this feature  (which I still consider a bug,
because it makes 'read -q' nearly unusable, at least in scripts)?

Bernd

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

RE: typeahead problem

[Andrej Borsenkow]

It is a "feature" of your particular OS. There is CLOBBERS_TYPEAHEAD define,
that tries to correct this. It gets set in configure for some systems. You
can try to recompile with this define set, and if it corrects your problem,
add setting for your system to configure.in.

/andrej

Re: typeahead problem

[Bernd Eggink]

Andrej Borsenkow wrote:
> 
> It is a "feature" of your particular OS. There is CLOBBERS_TYPEAHEAD define,
> that tries to correct this. It gets set in configure for some systems. You
> can try to recompile with this define set, and if it corrects your problem,
> add setting for your system to configure.in.
> 
> /andrej

In Linux and AIX 4.2 there is no difference in the behaviour whether or
not I define CLOBBERS_TYPEAHEAD in config.h. Hm, is that the right
location? I also added "x-*-linux*" to the switch in the "TYPEAHEAD
KLUDGE" section in configure.in, but that didn't change anything either.

Bernd

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

RE: typeahead problem

[Andrej Borsenkow]

>
> In Linux and AIX 4.2 there is no difference in the behaviour whether or
> not I define CLOBBERS_TYPEAHEAD in config.h. Hm, is that the right
> location? I also added "x-*-linux*" to the switch in the "TYPEAHEAD
> KLUDGE" section in configure.in, but that didn't change anything either.
>


Sorry, it is unrelated to typeahead probleam, I had to read better :(

I tried it here with 3.1.5 + patches. It looks, like ZSH takes the first
character on the line and ignores the rest.

while read -q && sleep 10
do
echo YES
done
y <- cursor immediately springs to the next line
yyyYES
      ^^^ output by ZSH
<- note newline
YES

Only first 'y' from 'yyy' is taken. Is it what you've seen?

Well, manual says "read -q reads only one character" so it is really
confusing. Currently it "reads the line and takes the first character".
Who's wrong - binary or manual?

/andrej

Re: typeahead problem

[Bernd Eggink]

Andrej Borsenkow wrote:
> 
> >
> > In Linux and AIX 4.2 there is no difference in the behaviour whether or
> > not I define CLOBBERS_TYPEAHEAD in config.h. Hm, is that the right
> > location? I also added "x-*-linux*" to the switch in the "TYPEAHEAD
> > KLUDGE" section in configure.in, but that didn't change anything either.
> >
> 
> Sorry, it is unrelated to typeahead probleam, I had to read better :(
> 
> I tried it here with 3.1.5 + patches. It looks, like ZSH takes the first
> character on the line and ignores the rest.
> 
> while read -q && sleep 10
> do
> echo YES
> done
> y <- cursor immediately springs to the next line
> yyyYES
>       ^^^ output by ZSH
> <- note newline
> YES
> 
> Only first 'y' from 'yyy' is taken. Is it what you've seen?

No. For example, try this script:

  for i in {1..}
  do if read -q "REPLY?y/n: "
     then print yes
     else print no
     fi
     sleep 1
  done

After the first prompt, type aa quickly (or any other characters). On
Linux and AIX, I get

  y/n: a
  no
  ay/n: n
  no
  ay/n: n
  no
  ay/n: n
  no
  ay/n: n
  no
  $ a

where $ stands for the shell prompt.

I guess the problem is getzlequery() in Zle/zle_utils.c. The comment
says: "If there are any characters in the buffer, this is taken as a
negative response, and no characters are read." This sounds like an
exact description of what happens, and doesn't seem to make much sense.
Why aren't the characters in the buffer read?? OTOH, there is no such
bug on HP-UP, so I hesitate to hack the sources blindly...

Bernd

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@rrz.uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

RE: typeahead problem

[Andrej Borsenkow]

>
> No. For example, try this script:
>
>   for i in {1..}
>   do if read -q "REPLY?y/n: "
>      then print yes
>      else print no
>      fi
>      sleep 1
>   done
>
> After the first prompt, type aa quickly (or any other characters). On
> Linux and AIX, I get
>
>   y/n: a
>   no
>   ay/n: n
>   no
>   ay/n: n
>   no
>   ay/n: n
>   no
>   ay/n: n
>   no
>   $ a
>

On 3.1.5-patched I cannot reproduce it. It behaves exactly as I described.
It _smells_ like CLOBBERS_TYPEAHED though ...

> where $ stands for the shell prompt.
>
> I guess the problem is getzlequery() in Zle/zle_utils.c. The comment
> says: "If there are any characters in the buffer, this is taken as a
> negative response, and no characters are read."

No, what is called is getquery(); zlegetquery() is used only when you have
too long completion list.

 guess, this happens exactly because your system does _not_ clobbers
typeahed :)) getquery() can be told to purge input - but read -q does not do
it; it sees typeahed, takes it for "no", and then is called again - but
nothing was ever read from terminal, so it sees the same input again ...

As to why it is taken for "no" - imagine, you typed something _before_
read -q, and that was not completely consumed - you definitely does not want
some leftover "y" to remove your valuable files :) so it tries to play safe.

The only thing I don't understand, why it works on my system ...

This sounds like an
> exact description of what happens, and doesn't seem to make much sense.
> Why aren't the characters in the buffer read?? OTOH, there is no such
> bug on HP-UP, so I hesitate to hack the sources blindly...
>

there is not on my system as well ... but funnily enough, it probably means
a bug on my system :))  The ultimate place to correct it is bin_read(); call
getquery with last argument 1.

/andrej

Re: typeahead problem

[Bernd Eggink]

Andrej Borsenkow wrote:
 
> As to why it is taken for "no" - imagine, you typed something _before_
> read -q, and that was not completely consumed - you definitely does not want
> some leftover "y" to remove your valuable files :) so it tries to play safe.

But of course the prompt could as well have been "Do you want to keep
the file?"! IMHO it would be safer to discard all previously typed
characters before querying.

> The only thing I don't understand, why it works on my system ...

You're lucky. I don't understand a LOT of things! ;)

> The ultimate place to correct it is bin_read(); call
> getquery with last argument 1.

Yeah, that fixes it! Thanks! 
Hm, is there any reason to call getquery(x, 0) at all?

Bernd 

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

Re: typeahead problem

[Bart Schaefer]

On Dec 8,  1:38pm, Bernd Eggink wrote:
} Subject: typeahead problem
}
}   read -q "REPLY?Yes or no: " && do_something
} 
} Now if "do_something" takes some time and you type at least one
} character in advance, 'read -q' behaves as if you are constantly typing
} 'n', until you consume the pending character by a normal 'read'. In
} other words, if you do NOT issue a normal read, EVERY following 'read
} -q' will behave as if you had typed 'n', until the end of the script.
} 
} Is there a rationale for this feature  (which I still consider a bug,
} because it makes 'read -q' nearly unusable, at least in scripts)?

Andrej has been telling you mostly the right things ...

On Dec 8,  6:54pm, Andrej Borsenkow wrote:
} As to why it is taken for "no" - imagine, you typed something _before_
} read -q, and that was not completely consumed - you definitely does not want
} some leftover "y" to remove your valuable files :) so it tries to play safe.

The getquery() function is called for the `rm *` check (the one DISabled
by `setopt rmstarsilent`) and for spell checking (`setopt correct`) as
well as for read -q.  In those cases, typeahead is considered undesirable,
and in the spell-check case zsh wants to leave it available in case the
user intended it as input for the command.  In the `rm *` case, on systems
that support ioctl(FIONREAD), zsh consumes all the typeahead before it
prints the query.

On systems that don't support FIONREAD, zsh always consumes one character.

The question is which of those two cases `read -q` should emulate.  The
decision made was to treat it like spell checking, leaving the typeahead
alone when possible so that other commands may consume it.  The way you
get the "always consume one character" behavior is to use

	read -k1 "REPLY?Yes or no: " && do_something

so having `read -q` available as an alternative that does NOT consume any
characters is more flexible.  The problem this leaves that there's no way
to emulate "consume typeahead" from a script (because all other reads are
blocking and wait for a newline).

On Dec 8,  4:01pm, Andrej Borsenkow wrote:
} Subject: RE: typeahead problem
}
} It is a "feature" of your particular OS. There is CLOBBERS_TYPEAHEAD define,
} that tries to correct this.

Unfortunately, CLOBBERS_TYPEAHEAD only applies when zle is setting up the
terminal with zsetterm(), which doesn't happen during getquery().

On Dec 8,  5:30pm, Andrej Borsenkow wrote:
} Subject: RE: typeahead problem
}
} I tried it here with 3.1.5 + patches. It looks, like ZSH takes the first
} character on the line and ignores the rest.
} 
} while read -q && sleep 10
} do
} echo YES
} done
} y <- cursor immediately springs to the next line
} yyyYES
}       ^^^ output by ZSH
} <- note newline
} YES
} 
} Only first 'y' from 'yyy' is taken. Is it what you've seen?

I see the following in a patched 3.0.5 and in both patched and unpatched
3.1.5:

zagzig% while read -q && sleep 5
while> do echo YES; done
y				<-- newline echoed immediately
yyyYES				<-- I typed yyy, YES echoed after 5 sec.
n				<-- echoed immediately after YES
zagzig% yyy			<-- typeahead now appears at the prompt

} Well, manual says "read -q reads only one character" so it is really
} confusing. Currently it "reads the line and takes the first character".
} Who's wrong - binary or manual?

On my system, at least, zsh does not consume the line, only the first y
of the four that I typed.  That agrees with the manual, except that the
manual doesn't discuss the typeahead behavior.

On Dec 8,  6:54pm, Andrej Borsenkow wrote:
} Subject: RE: typeahead problem
}
} The ultimate place to correct it is bin_read(); call
} getquery with last argument 1.

That would indeed change the behavior, but only when FIONREAD is supported.

On Dec 8,  6:39pm, Bernd Eggink wrote:
} Subject: Re: typeahead problem
}
} Hm, is there any reason to call getquery(x, 0) at all?

I think the choice to pass purge==0 was partly to get consistent script
behavior regardless of the availablilty of FIONREAD.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: typeahead problem

[Andrej Borsenkow]

> -----Original Message-----
> From: Bart Schaefer [mailto:schaefer@brasslantern.com]
>
>
> I see the following in a patched 3.0.5 and in both patched and unpatched
> 3.1.5:
>
> zagzig% while read -q && sleep 5
> while> do echo YES; done
> y				<-- newline echoed immediately
> yyyYES				<-- I typed yyy, YES echoed
> after 5 sec.
> n				<-- echoed immediately after YES
> zagzig% yyy			<-- typeahead now appears at the prompt
>
> } Well, manual says "read -q reads only one character" so it is really
> } confusing. Currently it "reads the line and takes the first character".
> } Who's wrong - binary or manual?
>
> On my system, at least, zsh does not consume the line, only the first y
> of the four that I typed.  That agrees with the manual, except that the
> manual doesn't discuss the typeahead behavior.
>

I know now, what happens here. See later

> I think the choice to pass purge==0 was partly to get consistent script
> behavior regardless of the availablilty of FIONREAD.
>

Unfortunately, it's not consistent. Consistent would be to read the first
character of typeahead.

About why it works and does not work.

getquery() changes tty modes, and on some systems that flushes unread input
("clobber typeahead") Looks, like my system is so fast, that even if I type
several characters, the first one is read by _first_ read -q, then the
second is read by the _second_ read -q and the others are simply lost.

In other words, if system does have FIONREAD but clobbers typeahead, zsh
always reads the first character and forgets the rest. CLOBBER_TYPEAHED is
used in zsetterm() but not in getquery() ... I think it is quite possible
... Well, do we need to change modes (setcbreak()) _before_ doing read? that
is, if FIONREAD is defined we check and either output NL
and return or read up typeahead and go on. the same, if we read the first
character of typeahead instead of assuming "no".

/andrej

Re: typeahead problem

[Bernd Eggink]

Bart Schaefer wrote:
 
> The question is which of those two cases `read -q` should emulate.  The
> decision made was to treat it like spell checking, leaving the typeahead
> alone when possible so that other commands may consume it.  

IMHO this decision was wrong, but it's probabely too late now. The
situation when I noticed that behaviour was in a loop consisting of
about 100 successive 'read -q' queries. ONE false move, and ALL
remaining questions get the answer 'no'. This can have very harmful
consequences, whereas not being able to type something in advance for
the next command is just a minor inconvenience.

> The way you
> get the "always consume one character" behavior is to use
> 
>         read -k1 "REPLY?Yes or no: " && do_something

Hm, no. You have to write something like

	typeset -u REPLY
	read -k1 "REPLY?Yes or no: "
	[[ $REPLY == Y ]] && do_something

But apart from that you're right, of course.

Regards,
Bernd 

--
Bernd Eggink
Regionales Rechenzentrum der Uni Hamburg
eggink@uni-hamburg.de
http://www.rrz.uni-hamburg.de/eggink/BEggink.html

Re: typeahead problem

[Bart Schaefer]

On Dec 8,  9:38pm, Andrej Borsenkow wrote:
> Subject: RE: typeahead problem
> 
> > I think the choice to pass purge==0 was partly to get consistent script
> > behavior regardless of the availablilty of FIONREAD.
> 
> Unfortunately, it's not consistent. Consistent would be to read the first
> character of typeahead.
> 
> About why it works and does not work.
> 
> getquery() changes tty modes, and on some systems that flushes unread input
> ("clobber typeahead") Looks, like my system is so fast, that even if I type
> several characters, the first one is read by _first_ read -q, then the
> second is read by the _second_ read -q and the others are simply lost.

Your machine executes "sleep 10" so quickly that you can't type before it
finishes?

> In other words, if system does have FIONREAD but clobbers typeahead, zsh
> always reads the first character and forgets the rest.

That shouldn't be how it works.  If the system clobbers typeahead in the
way described in Src/Zle/zle_main.c, then the typeahead should get lost
before getquery() has a chance to read even the first character.

However, it is the case that zsetterm() and getquery() are inconsistent
about their use of FIONREAD.  zsetterm() calls ioctl(FIONREAD) _before_
attaching to the TTY, whereas getquery() attaches first and then calls
ioctl().  If the zsetterm() code doesn't cause a problem with spurious
SIGTTIN and SIGTTOU signals, it should be safe to move the FIONREAD code
in getquery() to above the attachtty() and setcbreak().

> CLOBBER_TYPEAHED is used in zsetterm() but not in getquery() ...

Yes, but that doesn't make any difference [except for the inconsistency
I just noted], because getquery() already performs an equivalent test on
the value obtained from ioctl(FIONREAD).

> ... Well, do we need to change modes (setcbreak()) _before_ doing read?

Yes, but apparently not before doing ioctl(FIONREAD).

On Dec 8,  7:38pm, Bernd Eggink wrote:
> Subject: Re: typeahead problem
> Bart Schaefer wrote:
>  
> > The question is which of those two cases `read -q` should emulate.  The
> > decision made was to treat it like spell checking, leaving the typeahead
> > alone when possible so that other commands may consume it.  
> 
> IMHO this decision was wrong, but it's probabely too late now.

Actually, I don't get the impression that `read -q` is all that widely
used, and as it's already inconsistent depending on FIONREAD, this might
be a reasonable change.  We should probably float it on zsh-users first,
though.

> > get the "always consume one character" behavior is to use
> > 
> >         read -k1 "REPLY?Yes or no: " && do_something
> 
> Hm, no. You have to write something like
> 
> 	typeset -u REPLY
> 	read -k1 "REPLY?Yes or no: "
> 	[[ $REPLY == Y ]] && do_something

Well, yes, but you should still test for success of read.  I assumed you
were already testing $REPLY inside "do_something" ... if you don't need
the answer captured in $REPLY, you can just do

	read -q "?Yes or no: " && ...