From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/10556 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general,gmane.comp.version-control.git Subject: Re: Regression: git no longer works with musl libc's regex impl Date: Tue, 4 Oct 2016 11:40:45 -0400 Message-ID: <20161004154045.GT19318@brightrain.aerifal.cx> References: <20161004150848.GA7949@brightrain.aerifal.cx> <20161004152722.ex2nox43oj5ak4yi@sigill.intra.peff.net> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1475595679 6848 195.159.176.226 (4 Oct 2016 15:41:19 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 4 Oct 2016 15:41:19 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) Cc: git@vger.kernel.org, musl@lists.openwall.com To: Jeff King Original-X-From: musl-return-10569-gllmg-musl=m.gmane.org@lists.openwall.com Tue Oct 04 17:41:13 2016 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1brRq3-0008UL-5m for gllmg-musl@m.gmane.org; Tue, 04 Oct 2016 17:41:03 +0200 Original-Received: (qmail 13434 invoked by uid 550); 4 Oct 2016 15:41:03 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 13405 invoked from network); 4 Oct 2016 15:41:01 -0000 Content-Disposition: inline In-Reply-To: <20161004152722.ex2nox43oj5ak4yi@sigill.intra.peff.net> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:10556 gmane.comp.version-control.git:306077 Archived-At: On Tue, Oct 04, 2016 at 11:27:22AM -0400, Jeff King wrote: > On Tue, Oct 04, 2016 at 11:08:48AM -0400, Rich Felker wrote: > > > This commit broke support for using git with musl libc: > > > > https://github.com/git/git/commit/2f8952250a84313b74f96abb7b035874854cf202 > > Yep. The idea is that you would compile git with NO_REGEX=1, and it > would use the included compat routines. This is really obnoxious to have to do manually. Why can't it simply be auto-detected based on non-definition of REG_STARDEND? > Is there something in particular you want to get out of using musl's > regex that is not supported in the compat library? It's always nice not to link extra implementations of the same thing. This comes up all the time with gratuitous gnulib replacement functions too. > > Rather than depending on non-portable GNU regex extensions, there is a > > simple portable fix for the issue this code was added to work around: > > When a text file is being mmapped for use with string functions which > > depend on null termination, if the file size: > > > > 1. is nonzero mod page size, it just works; the remainder of the last > > page reads as zero bytes when mmapped. > > Is that a portable assumption? Yes. Per POSIX: "The system shall always zero-fill any partial page at the end of an object. Further, the system shall never write out any modified portions of the last page of an object which are beyond its end. References within the address range starting at pa and continuing for len bytes to whole pages following the end of an object shall result in delivery of a SIGBUS signal." Source: http://pubs.opengroup.org/onlinepubs/9699919799/functions/mmap.html The same or similar text appears at least back to SUSv2; I did not look further back. > > 2. if an exact multiple of the page size, then instead of directly > > mmapping the file, first mmap a mapping 1 byte (thus 1 page) larger > > with MAP_ANON, then use MAP_FIXED to map the file over top of all > > but the last page. Now the mmapped buffer can safely be used as a C > > string. > > I'm not sure whether all of our compat layers for mmap would be happy > with that (e.g., see compat/win32mmap.c). The nice thing about this approach is that if the MAP_FIXED fails due to a broken system you can just read() into the anonymous memory. > So it seems like any mmap-related solutions would have to be > conditional, too. And then regexec_buf() would have to become something > like: > > int regexec_buf(...) > { > #if defined(REG_STARTEND) > ... set up match ... > return regexec(..., REG_STARTEND); > #elif defined(MMAP_ALWAYS_HAS_NUL) > /* > * We assume that every buffer we see is always NUL-terminated > * eventually, either because it comes from xmallocz() or our > * mmap layer always ensures an extra NUL. > */ > return regexec(...); > #else > #error "Nope, you need either NO_REGEX or USE_MMAP_NUL" > #endif > } > > The assumption in the middle case feels pretty hacky, though. It fails I agree. I would prefer just falling back to read() after MAP_ANON if the MAP_FIXED fails. This would work for all systems. > if we get a buffer from somewhere besides those two sources. It fails if > somebody calls regexec_buf() on a subset of a string. > > It also doesn't handle matching past embedded NULs in the string. That's > not something we're relying on yet, but it would be nice to support > consistently in the long run. This is going to be even more non-portable and implementation-specific. There's not even a good spec for how embedded nuls should be treated in regex. > If there's a compelling reason, it might be worth making that tradeoff. > But I am not sure what the compelling reason is to use musl's regex > (aside from the obvious of "less code in the resulting executable"). The compelling reason is just being portable by default rather than requiring manual overrides to use a replacement library with weird extensions for something that could have been done portably to begin with. Rich