From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/8384 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Adjustments to roadmap Date: Thu, 27 Aug 2015 22:43:48 -0400 Message-ID: <20150828024347.GA7624@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: ger.gmane.org 1440729852 25509 80.91.229.3 (28 Aug 2015 02:44:12 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 28 Aug 2015 02:44:12 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-8396-gllmg-musl=m.gmane.org@lists.openwall.com Fri Aug 28 04:44:12 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1ZV9eE-0002TV-7A for gllmg-musl@m.gmane.org; Fri, 28 Aug 2015 04:44:10 +0200 Original-Received: (qmail 13561 invoked by uid 550); 28 Aug 2015 02:44:08 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 13515 invoked from network); 28 Aug 2015 02:44:01 -0000 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:8384 Archived-At: Hey everyone, Between travel for a CII meeting last month (from which there's exciting stuff I still need to follow up on, especially tlsify), work, and making personal time to enjoy the summer, I've gotten rather behind on musl releases. I think I'm caught up with everything important for 1.1.11 though, and plan to release it ASAP. During this period of release delay, I haven't been updating the roadmap on the wiki, but lots of new ideas have come up for enhancing musl that may be appropriate to prioritize somewhere in the next few release cycles. I'd like to share some of them here and get feedback on what the community/users would like to see: ---------------------------------------------------------------------- 1. Unifying static/shared libc .o files. Right now, all the files in musl are built twice, once as .o for inclusion in libc.a, and once as ".lo" (my naming convention) for inclusion in libc.so. This of course requires twice the time and space to build, and also has semi-gratuitously separate code paths that need to be maintained. I would like to eliminate that. One of the obvious big reasons for building libc.so's files separately is that they need to be PIC. But for static-PIE support, which musl now has and which I'm trying to get upstream in GCC/binutils, libc.a needs to be built as PIC too. I did some tests, and the size increase of libc.a by using PIC is about 1-3% on "important" archs and still only 4-6% on some of the worst ones. I don't have figures on performance cost, though, and it's harder to measure. If we do get rid of the dual-build, it will still be possible to build a non-PIC libc.a via something like --disable-pic, which would imply --disable-shared. It would just be a separate run of the build, which is somewhat inconvenient now but much less so if we add out-of-tree build support. And of course getting rid of the double-build in the makefile would simplify it and make developing out-of-tree build support easier. I have more detailed notes on this topic I'll post later. 2. Rewriting malloc It came to my attention during this last release cycle that musl's strategies for avoiding contention in malloc were leading to excessive heap growth/fragmentation, contrary to my explicit intent. This was the issue Timo Teräs reported for which I made a mitigation as commit c3761622e8168b0c6453637ac82e70b09af3e8e9. I don't yet have a proposed long-term direction for malloc. Some brief testing determined that the fine-grained locking we have now, despite being theoretically of dubious benefit, actually does help; switching to a single lock resulted in significant performance regressions. In addition, I've lost a good deal of faith in the whole "dlmalloc" style heap-management approach. Worst-case fragmentation is expressed by the fact that allocations as small as M/N can fail, where M is the "total free memory" (total minus used) and N is the number of live allocations. It should be possible to achieve much better bounds by avoiding excessive mixing of different-sized chunks, and there are known highly-inefficient ways to do this, but I haven't found any good ones. What I'd like to do at the same time is eliminate most or all of the need for merging of free chunks -- this is the main bottleneck to concurrency. 3. Symbol versioning Right now musl's dynamic linker ignores symbol versioning information except to select the "current" version of a symbol the same way ld does. This is sufficient for most things, if you don't want to support old library ABIs and mixed (i.e. broken) ABIs, but it breaks some hacks GCC is doing in their target libs (libgcc_s.so, ...), and some setups that arguably "should" work, especially for C++ apps. What's probably worse is that you get silent breakage when an app is trying to use symbol versioning, expecting it to work, and it doesn't. I don't think the status quo is a reasonable option. We should either teach GCC that musl targets don't support symbol versioning, and make sure apps/libs' build systems detect this, or we should make them work. My leaning is towards the latter. This is not an endorsement of symbol versioning. It's a poor approximation of the best possible solution to a problem that's not fully solvable, and it DOES introduce the possibility of silent breakage in most places where it's used, but there are ways to use it safely for some limited purposes, and supporting these seems to be the path of least headache. 4. Dynamic linker library chain changes There is some ambiguity of global symbol definition priority when a library is first loaded as RTLD_LOCAL then later moved to the global namespace with RTLD_GLOBAL, but basically I think what musl is doing right now is "wrong" short of an official interpretation of the standard to the contrary. If this is resolved, I'd like to rework how musl does its linked list of shared libraries. Right now there's just one list with a flag for each library indicating whether it's in the global namespace or not. This forces searches for symbols to step through libraries that are not relevant, and also forces TLS initialization for new threads to step through all the libraries even if only a few actually have TLS. What I'd like to do is have 3 or more separate lists: one for all the libraries (for gdb and dl_iterate_phdr to use), one representing the global namespace, and one for just the libraries with TLS. This trades a very small amount of memory in 'struct dso' for significant performance improvements in programs with large numbers of libraries. ---------------------------------------------------------------------- In addition to the above, there are all of the existing roadmap items on the wiki which are open for discussion of how they should be prioritized. The big projects are roughly: - Atomics refactorization/deduplication - Bits refactorization/deduplication - Out-of-tree builds - LC_COLLATE - IDN - Advanced glibc ABI-compat features in dynamic linker - Documentation Apologies for the slow progress lately. Don't worry though, there's still lots more good stuff to come for musl. Rich