Further wide stdio issues [Was: Re: [PATCH] fix fgetwc when decoding a character that crosses buffer boundary]

[Rich Felker <dalias@libc.org>]

On Sat, Nov 18, 2017 at 05:51:48PM +0100, Szabolcs Nagy wrote:
> Update the buffer position according to the bytes consumed into st when
> decoding an incomplete character at the end of the buffer.

Further related problems:

1. fgetws is unable to accurately determine whether fgetwc failed due
   to hitting EOF on a clean stream vs hitting EOF in the middle of a
   truncated character. The latter is an EILSEQ condition and should
   cause fgetws to return a null pointer rather than a line not ending
   in newline.

2. fgetwc attempts to leave all but the first byte unread on EILSEQ so
   that calling fgetwc again will resync, but doesn't do this
   consistently when it has to call getc_unlocked to get new bytes
   rather than reading from the buffer.

I think (1) can be fixed in an ugly way by setting errno to something
other than EILSEQ before calling fgetwc, then checking if EILSEQ upon
failure.

As for (2), it's not fixable without some pushback (ungetc). I'm not
sure if pushback here on an unbuffered stream is even conforming; if
not, no fix is possible, which is quite unfortunate because it makes
it impossible to resync after error. Assuming pushback is acceptable,
however, we'd still need up to MB_LEN_MAX-1 bytes of pushback to give
the same behavior as the buffered case now. It might be better to have
both the buffered and unbuffered cases consume as many bytes as yield
(size_t)-2 from mbrtowc (i.e. as many as would be a prefix of a valid
character) and only leave unread (pushed back) the first byte that
determines failure (which could be invalid itself, or the first byte
of the next valid character). This would require slightly more work in
the buffered case, but it could instead be handled by just falling
through to the slow unbuffered-case loop, with st reset to {0} and
without advancing the buffer position, when mbrtowc fails. In fact
then we could even use mbtowc instead of mbrtowc in the fast path,
making it slightly faster.

Thoughts?

Rich