From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/12114 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Further wide stdio issues [Was: Re: [PATCH] fix fgetwc when decoding a character that crosses buffer boundary] Date: Sun, 19 Nov 2017 20:48:53 -0500 Message-ID: <20171120014853.GI1627@brightrain.aerifal.cx> References: <20171118165148.GF15263@port70.net> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1511142550 9140 195.159.176.226 (20 Nov 2017 01:49:10 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 20 Nov 2017 01:49:10 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-12130-gllmg-musl=m.gmane.org@lists.openwall.com Mon Nov 20 02:49:06 2017 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1eGbCp-0001xG-BI for gllmg-musl@m.gmane.org; Mon, 20 Nov 2017 02:49:03 +0100 Original-Received: (qmail 25898 invoked by uid 550); 20 Nov 2017 01:49:07 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 25851 invoked from network); 20 Nov 2017 01:49:06 -0000 Content-Disposition: inline In-Reply-To: <20171118165148.GF15263@port70.net> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:12114 Archived-At: On Sat, Nov 18, 2017 at 05:51:48PM +0100, Szabolcs Nagy wrote: > Update the buffer position according to the bytes consumed into st when > decoding an incomplete character at the end of the buffer. Further related problems: 1. fgetws is unable to accurately determine whether fgetwc failed due to hitting EOF on a clean stream vs hitting EOF in the middle of a truncated character. The latter is an EILSEQ condition and should cause fgetws to return a null pointer rather than a line not ending in newline. 2. fgetwc attempts to leave all but the first byte unread on EILSEQ so that calling fgetwc again will resync, but doesn't do this consistently when it has to call getc_unlocked to get new bytes rather than reading from the buffer. I think (1) can be fixed in an ugly way by setting errno to something other than EILSEQ before calling fgetwc, then checking if EILSEQ upon failure. As for (2), it's not fixable without some pushback (ungetc). I'm not sure if pushback here on an unbuffered stream is even conforming; if not, no fix is possible, which is quite unfortunate because it makes it impossible to resync after error. Assuming pushback is acceptable, however, we'd still need up to MB_LEN_MAX-1 bytes of pushback to give the same behavior as the buffered case now. It might be better to have both the buffered and unbuffered cases consume as many bytes as yield (size_t)-2 from mbrtowc (i.e. as many as would be a prefix of a valid character) and only leave unread (pushed back) the first byte that determines failure (which could be invalid itself, or the first byte of the next valid character). This would require slightly more work in the buffered case, but it could instead be handled by just falling through to the slow unbuffered-case loop, with st reset to {0} and without advancing the buffer position, when mbrtowc fails. In fact then we could even use mbtowc instead of mbrtowc in the fast path, making it slightly faster. Thoughts? Rich