Unexpected stdin-behavior

Unexpected stdin-behavior

[Tycho Kirchner]

Dear zsh-maintainers,

first of all, I'm not a mail-subscriber but please respond to this email 
anyway ^_^

I almost finished the (yet unpublished) zsh-integration of my 
shell-tracker shournal[1] and noticed some inconsistency on the handling 
of stdin during an integration test.
While bash -i and zsh -s show the expected behavior (stdin is "consumed" 
command by command) zsh -s -i seems to "consume" the whole stdin 
beforehand (see below). I there a rationale behind this?
What I actually try to do is running automated tests on an interactive 
zsh with prompts, preexec, etc by piping commands to it similar to the 
snippet below.

Thanks in advance
Tycho Kirchner

[1] https://github.com/tycho-kirchner/shournal

____________________________________________________________

debian-dell% echo $ZSH_VERSION
5.8
debian-dell% stdintest(){ printf 'echo one\nsh -c "while read -r row; do 
echo got row \$row; done" \necho two\n'; }
debian-dell% stdintest | bash  --norc --noprofile -i
bash-5.0$ echo one
one
bash-5.0$ sh -c "while read -r row; do echo got row \$row; done"
got row echo two
bash-5.0$ exit
debian-dell% stdintest | zsh -f -s
one
got row echo two
debian-dell% stdintest | zsh -f -s -i
debian-dell% one
debian-dell% %
debian-dell% two
debian-dell% %
debian-dell%

Re: Unexpected stdin-behavior

[Bart Schaefer]

On Thu, Oct 21, 2021 at 5:40 AM Tycho Kirchner <tychokirchner@mail.de> wrote:
>
> While bash -i and zsh -s show the expected behavior (stdin is "consumed"
> command by command) zsh -s -i seems to "consume" the whole stdin
> beforehand (see below). I there a rationale behind this?

Zsh is using the stdio library to read its input.  (This has actually
changed in the current development version but still buffers in an
equivalent way.)

Re: Unexpected stdin-behavior

[Bart Schaefer]

On Thu, Oct 21, 2021 at 9:47 AM Tycho Kirchner <tychokirchner@mail.de> wrote:
>
> thanks for your response. Could you please elaborate how your answer
> explains the difference in the output between the commands
> zsh -s and
> zsh -s -i
> ?

When -i is NOT present, stdin is set to line buffered for the stdio library.

I don't actually see any difference between -s and -s -i except for
the printing of the prompt, when I try it with the latest development
version, and the only code difference is the removal of stdio.  This
is probably an unintentional behavior change in the new code, and may
bear looking into.

For zsh-workers (particularly PWS), I'm referring to this bit of code in init.c:

    /*
     * Finish setting up SHIN and its relatives.
     */
    shinbufalloc();
    if (isset(SHINSTDIN) && !SHIN && unset(INTERACTIVE)) {
#ifdef _IONBF
        setvbuf(stdin, NULL, _IONBF, 0);
#else
        setlinebuf(stdin);
#endif
    }

We either don't need the set*buf business at all, or we need its
equivalent for shinbuf, I think.

Re: Unexpected stdin-behavior

[Tycho Kirchner]

Am 21.10.21 um 21:14 schrieb Bart Schaefer:
> On Thu, Oct 21, 2021 at 9:47 AM Tycho Kirchner <tychokirchner@mail.de> wrote:
>>
>> thanks for your response. Could you please elaborate how your answer
>> explains the difference in the output between the commands
>> zsh -s and
>> zsh -s -i
>> ?
> 
> When -i is NOT present, stdin is set to line buffered for the stdio library.
> 
> I don't actually see any difference between -s and -s -i except for
> the printing of the prompt, when I try it with the latest development
> version, and the only code difference is the removal of stdio.  This
> is probably an unintentional behavior change in the new code, and may
> bear looking into.
> 
> For zsh-workers (particularly PWS), I'm referring to this bit of code in init.c:
> 
>      /*
>       * Finish setting up SHIN and its relatives.
>       */
>      shinbufalloc();
>      if (isset(SHINSTDIN) && !SHIN && unset(INTERACTIVE)) {
> #ifdef _IONBF
>          setvbuf(stdin, NULL, _IONBF, 0);
> #else
>          setlinebuf(stdin);
> #endif
>      }
> 
> We either don't need the set*buf business at all, or we need its
> equivalent for shinbuf, I think.
> 

Indeed, in the latest dev-version both behave the same. I think *now 
both behave wrong*, instead zsh -s -i should behave the same as zsh -s 
did previously, not the other way round.

Re: Unexpected stdin-behavior

[Peter Stephenson]

> On 21 October 2021 at 20:14 Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Thu, Oct 21, 2021 at 9:47 AM Tycho Kirchner <tychokirchner@mail.de> wrote:
> > thanks for your response. Could you please elaborate how your answer
> > explains the difference in the output between the commands
> > zsh -s and
> > zsh -s -i
> > ?
> 
> When -i is NOT present, stdin is set to line buffered for the stdio library.
> 
> I don't actually see any difference between -s and -s -i except for
> the printing of the prompt, when I try it with the latest development
> version, and the only code difference is the removal of stdio.  This
> is probably an unintentional behavior change in the new code, and may
> bear looking into.
> 
> For zsh-workers (particularly PWS), I'm referring to this bit of code in init.c:
> 
>     /*
>      * Finish setting up SHIN and its relatives.
>      */
>     shinbufalloc();
>     if (isset(SHINSTDIN) && !SHIN && unset(INTERACTIVE)) {
> #ifdef _IONBF
>         setvbuf(stdin, NULL, _IONBF, 0);
> #else
>         setlinebuf(stdin);
> #endif
>     }
> 
> We either don't need the set*buf business at all, or we need its
> equivalent for shinbuf, I think.

That _IONBF is (and has always been) inconsistent with setlinebuf(), surely?
It means no buffering.

As far as shinbuf itself is concerned, is the most logical behaviour
line buffering (i.e. read ahead only up to a \n) if interactive?

pws

Re: Unexpected stdin-behavior

[Tycho Kirchner]

I also sent an email to the dash-developers regarding the same issue, 
you may be interested in Chet's response:


On 10/22/21 7:11 AM, Tycho Kirchner wrote:
 > Dear DASH developers,
 > I think stdin should be consumed line by line in order to make 
passing to
 > other commands possible. Please consider the following difference 
between
 > the stdin "consumption" between bash and dash

If you're curious, POSIX specifies the bash behavior. From

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117

"When the shell is using standard input and it invokes a command that also
uses standard input, the shell shall ensure that the standard input file
pointer points directly after the command it has read when the command
begins execution. It shall not read ahead in such a manner that any
characters intended to be read by the invoked command are consumed by the
shell (whether interpreted by the shell or not) or that characters that are
not read by the invoked command are not seen by the shell."

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/