From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 13804 invoked from network); 10 May 2020 22:09:33 -0000 Received-SPF: pass (mother.openwall.net: domain of lists.openwall.com designates 195.42.179.200 as permitted sender) receiver=inbox.vuxu.org; client-ip=195.42.179.200 envelope-from= Received: from mother.openwall.net (195.42.179.200) by inbox.vuxu.org with ESMTPUTF8; 10 May 2020 22:09:33 -0000 Received: (qmail 28570 invoked by uid 550); 10 May 2020 22:09:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28539 invoked from network); 10 May 2020 22:09:28 -0000 Date: Sun, 10 May 2020 18:09:15 -0400 From: Rich Felker To: musl@lists.openwall.com Message-ID: <20200510220915.GA17662@brightrain.aerifal.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: [musl] upstreaming mallocng Finishing initial development of and upstreaming mallocng is the main item for this release cycle, and I think we're close to there. The main question is whether there will be any serious performance or memory usage regressions relative to oldmalloc, which may be problematic for some users. With that in mind, I'm considering supporting a build-time choice of oldmalloc for at least the first couple releases with mallocng, so that there's an easy path back for users who are hit by any such regression that can't be solved immediately. If offering oldmalloc as an option, I think I'd like to apply the patch for the runaway heap expansion issue first: https://www.openwall.com/lists/musl/2019/04/12/4 This does cost some level of performance, but the loss should be well-understood. Arguably this should have been applied at the time I wrote it, but I wasn't really excited about the performance hit without having something better to offer in place of it. With that said, here are some of the things users should expect to see from mallocng: - More predictable behavior. Availability of memory should be influenced a lot less by history of allocator activity within the process's lifetime, especially on 64-bit archs where virtual address space fragmentation is not an issue. Non-essential patterns of ballooning fragmentation, like in glibc issue https://sourceware.org/bugzilla/show_bug.cgi?id=14581 which musl oldmalloc also exhibited, should also not happen anymore. - Memory getting returned to the system, aggressively unless it bounces back and forth enough that we back off to avoid pathologically bad performance. Software that builds then frees large data structures should expect to mostly return to usage levels from before the allocation after it's all freed, rather than getting stuck at near-peak usage like what happens with dlmalloc type allocator. - No more runaway heap expansion (see above thread). - Differences in performance characteristics. Expected to perform better at high usage, anywhere between noticably better and noticably worse at low usage levels. (Where worse is severe, efforts will be made to mitigate it as much as possible, but only if it's reported.) - Lower memory overhead at high usage. Ideally similar, but possibly somewhat higher, memory overhead at low usage. Tuning out all of the causes of gratuitous excess usage and over-eagar preallocation when the nominal usage is low, without clobbering performance, has been the big focus of recent development. - Byte-accurate malloc_usable_size. This is necessary to make accesses out to malloc_usable_size offset not produce UB in the eyes of the compiler/fortify, which may generate traps (e.g. for memset(p, 42, malloc_usable_size(p))). Now would be a great time for feedback on large-scale use of mallocng, if anyone's up for it. You can put it in LD_PRELOAD via .profile or even in init for system-wide use. Rich