From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/8385 Path: news.gmane.org!not-for-mail From: Khem Raj Newsgroups: gmane.linux.lib.musl.general Subject: Re: Adjustments to roadmap Date: Thu, 27 Aug 2015 20:38:16 -0700 Message-ID: References: <20150828024347.GA7624@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1440733160 4835 80.91.229.3 (28 Aug 2015 03:39:20 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 28 Aug 2015 03:39:20 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-8397-gllmg-musl=m.gmane.org@lists.openwall.com Fri Aug 28 05:39:05 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1ZVAVJ-0007S4-58 for gllmg-musl@m.gmane.org; Fri, 28 Aug 2015 05:39:01 +0200 Original-Received: (qmail 23685 invoked by uid 550); 28 Aug 2015 03:38:59 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 23660 invoked from network); 28 Aug 2015 03:38:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=4BDRrnggEo37nXRp860bDP0qtGQY/BGY2J4Td9HSqo8=; b=M3VL0sz0LgjCHppV6TXdcJtt7i+uVGsuU7F8zOaOY+rcCfzqNxxgTZRPw86SoRY29v UUhNMS0nk/V/1I98PEj3NleMZn48j4ItCEwCOUV1J7ZA3Tcy05s5KDhNuvCp0TLKJASw XLpgQRatt5A8ApBkc2ZmXoXzqKfcXzHPvU9EH5EfSAdCz1Ee1M5mELZt+7J2rZC2eoPn FRq/l9Zw0/9dJ4DvicY3lKauoSQJcQgd+RGA4CJ/8MFsevun4Whte1MddjFfz6d9bW0+ 4HVju1DV+zjgOl/XDGQqytek8vf6q8EviP3r9WG34yQT5wyoWKi3AHEWBnqgMYSs8NQX O8Iw== X-Received: by 10.107.168.214 with SMTP id e83mr11922070ioj.112.1440733126543; Thu, 27 Aug 2015 20:38:46 -0700 (PDT) In-Reply-To: <20150828024347.GA7624@brightrain.aerifal.cx> Xref: news.gmane.org gmane.linux.lib.musl.general:8385 Archived-At: On Thu, Aug 27, 2015 at 7:43 PM, Rich Felker wrote: > Hey everyone, > > Between travel for a CII meeting last month (from which there's > exciting stuff I still need to follow up on, especially tlsify), work, > and making personal time to enjoy the summer, I've gotten rather > behind on musl releases. I think I'm caught up with everything > important for 1.1.11 though, and plan to release it ASAP. > > During this period of release delay, I haven't been updating the > roadmap on the wiki, but lots of new ideas have come up for enhancing > musl that may be appropriate to prioritize somewhere in the next few > release cycles. I'd like to share some of them here and get feedback > on what the community/users would like to see: > > ---------------------------------------------------------------------- > > 1. Unifying static/shared libc .o files. > > Right now, all the files in musl are built twice, once as .o for > inclusion in libc.a, and once as ".lo" (my naming convention) for .lo is conflicting with libtool generated objects btw. > inclusion in libc.so. This of course requires twice the time and space > to build, and also has semi-gratuitously separate code paths that need > to be maintained. I would like to eliminate that. > > One of the obvious big reasons for building libc.so's files separately > is that they need to be PIC. But for static-PIE support, which musl > now has and which I'm trying to get upstream in GCC/binutils, libc.a > needs to be built as PIC too. I did some tests, and the size increase > of libc.a by using PIC is about 1-3% on "important" archs and still > only 4-6% on some of the worst ones. I don't have figures on > performance cost, though, and it's harder to measure. > > If we do get rid of the dual-build, it will still be possible to build > a non-PIC libc.a via something like --disable-pic, which would imply > --disable-shared. It would just be a separate run of the build, which > is somewhat inconvenient now but much less so if we add out-of-tree > build support. And of course getting rid of the double-build in the > makefile would simplify it and make developing out-of-tree build > support easier. This seems a good proposal. We need some data on arches like mips > > I have more detailed notes on this topic I'll post later. > > 2. Rewriting malloc > > It came to my attention during this last release cycle that musl's > strategies for avoiding contention in malloc were leading to excessive > heap growth/fragmentation, contrary to my explicit intent. This was > the issue Timo Ter=C3=A4s reported for which I made a mitigation as commi= t > c3761622e8168b0c6453637ac82e70b09af3e8e9. > > I don't yet have a proposed long-term direction for malloc. Some brief > testing determined that the fine-grained locking we have now, despite > being theoretically of dubious benefit, actually does help; switching > to a single lock resulted in significant performance regressions. > > In addition, I've lost a good deal of faith in the whole "dlmalloc" > style heap-management approach. Worst-case fragmentation is expressed > by the fact that allocations as small as M/N can fail, where M is the > "total free memory" (total minus used) and N is the number of live > allocations. It should be possible to achieve much better bounds by > avoiding excessive mixing of different-sized chunks, and there are > known highly-inefficient ways to do this, but I haven't found any good > ones. > > What I'd like to do at the same time is eliminate most or all of the > need for merging of free chunks -- this is the main bottleneck to > concurrency. > > 3. Symbol versioning > > Right now musl's dynamic linker ignores symbol versioning information > except to select the "current" version of a symbol the same way ld > does. This is sufficient for most things, if you don't want to support > old library ABIs and mixed (i.e. broken) ABIs, but it breaks some > hacks GCC is doing in their target libs (libgcc_s.so, ...), and some > setups that arguably "should" work, especially for C++ apps. What's > probably worse is that you get silent breakage when an app is trying > to use symbol versioning, expecting it to work, and it doesn't. > > I don't think the status quo is a reasonable option. We should either > teach GCC that musl targets don't support symbol versioning, and make > sure apps/libs' build systems detect this, or we should make them > work. My leaning is towards the latter. we can detect using triplets so apps should be able to conflgure for musl. I think symbol versioning could be a good thing for musl for backward compatibilty with itself who knows what future holds for musl, we might have to deal with own past. but it should be a build time option, for glibc if we disable versioned symbols code size reduces significantly. > > This is not an endorsement of symbol versioning. It's a poor > approximation of the best possible solution to a problem that's not > fully solvable, and it DOES introduce the possibility of silent > breakage in most places where it's used, but there are ways to use it > safely for some limited purposes, and supporting these seems to be the > path of least headache. > > 4. Dynamic linker library chain changes > > There is some ambiguity of global symbol definition priority when a > library is first loaded as RTLD_LOCAL then later moved to the global > namespace with RTLD_GLOBAL, but basically I think what musl is doing > right now is "wrong" short of an official interpretation of the > standard to the contrary. > > If this is resolved, I'd like to rework how musl does its linked list > of shared libraries. Right now there's just one list with a flag for > each library indicating whether it's in the global namespace or not. > This forces searches for symbols to step through libraries that are > not relevant, and also forces TLS initialization for new threads to > step through all the libraries even if only a few actually have TLS. > > What I'd like to do is have 3 or more separate lists: one for all the > libraries (for gdb and dl_iterate_phdr to use), one representing the > global namespace, and one for just the libraries with TLS. This trades > a very small amount of memory in 'struct dso' for significant > performance improvements in programs with large numbers of libraries. > > ---------------------------------------------------------------------- > > In addition to the above, there are all of the existing roadmap items > on the wiki which are open for discussion of how they should be > prioritized. The big projects are roughly: > > - Atomics refactorization/deduplication > - Bits refactorization/deduplication > - Out-of-tree builds > - LC_COLLATE > - IDN > - Advanced glibc ABI-compat features in dynamic linker > - Documentation how about nommu ? > > Apologies for the slow progress lately. Don't worry though, there's > still lots more good stuff to come for musl. Some goals around code size might be interesting too. > > Rich