From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/13231 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: string-backed FILEs mess Date: Wed, 12 Sep 2018 12:33:45 -0400 Message-ID: <20180912163345.GX1878@brightrain.aerifal.cx> References: <20180912140239.GV1878@brightrain.aerifal.cx> <20180912150941.GB13976@voyager> <20180912154306.GW1878@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1536769912 29789 195.159.176.226 (12 Sep 2018 16:31:52 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 12 Sep 2018 16:31:52 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-13247-gllmg-musl=m.gmane.org@lists.openwall.com Wed Sep 12 18:31:48 2018 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1g083Q-0007fH-4O for gllmg-musl@m.gmane.org; Wed, 12 Sep 2018 18:31:48 +0200 Original-Received: (qmail 26277 invoked by uid 550); 12 Sep 2018 16:33:57 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 26259 invoked from network); 12 Sep 2018 16:33:56 -0000 Content-Disposition: inline In-Reply-To: <20180912154306.GW1878@brightrain.aerifal.cx> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:13231 Archived-At: On Wed, Sep 12, 2018 at 11:43:06AM -0400, Rich Felker wrote: > > The __string_read approach shouldn't be all that much slower, though. It > > only adds overhead when the read buffer is empty, which is going to be > > on the first call and then every 256 characters. And the overhead is > > mostly a memchr()... on second thought, that might take a while. > > Ah, I forgot that __string_read is still doing the hack of having > buffer pointers point directly to the string, not copying it, so that > ungetc is unsafe (in general; the intscan/floatscan/shgetc framework > has a way to handle it). So it's not a copy, but it is a memchr of N > to N+(-N&255) bytes, where N is the length of the input item. Even if > N is short, the input item might lie in a longer string, and if you > have a sequence of short numbers (worst-case, most single-digit) in a > long string of numbers, you'll end up repeatedly scanning for null up > to 255 bytes past the end of the number. Just lowering the scanahead > from 256 to something like 16-24 should make this largely irrelevant, > though. Such tuning should probably be specific to caller (strto* vs > vsscanf, since the latter is likely to be reading multiple fields and > possibly non-numeric fields). OK, some measurements. The test is making a string of " 1" repeated 10M times and looping through it with strtol updating end pointer. With existing strtol, it takes 0.91s. With __string_read and existing 256-byte unit, it takes 5.76s. That's a pretty huge slowdown. Lowering the step from 256 to 24, it takes 2.92s. Better but still unacceptably slower. Inlining the buffer setup code and memchr so that __string_read doesn't have to get called unless it's exhausted (and doesn't get called in the test) gets it down to 2.00s with a step of 24, and 1.75s with a step of 8. OK, I've been properly initializing the FILE rather than leaving it uninitialized except for the important fields like the old code did. Changing that, it's 1.44s with step 8, 1.60s with step 24. I also confirmed that this version of the code is almost as fast as the existing code with the memchr removed (just assuming it can read ahead). I'll see what else I can find but it doesn't look like there's a way to fix this without either making it a minimum of ~1.5x slower at this particular worst-case, or redesigning the backend not to use a fake FILE but some different abstraction. Rich