From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/4772 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: malloc & uncommitting memory Date: Wed, 2 Apr 2014 17:08:05 -0400 Message-ID: <20140402210805.GA852@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1396526513 7980 80.91.229.3 (3 Apr 2014 12:01:53 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 3 Apr 2014 12:01:53 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-4774-gllmg-musl=m.gmane.org@lists.openwall.com Thu Apr 03 14:01:46 2014 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1WVez5-0002EN-Cs for gllmg-musl@plane.gmane.org; Thu, 03 Apr 2014 12:34:59 +0200 Original-Received: (qmail 15565 invoked by uid 550); 2 Apr 2014 21:08:17 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 15551 invoked from network); 2 Apr 2014 21:08:17 -0000 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Xref: news.gmane.org gmane.linux.lib.musl.general:4772 Archived-At: Adding the ability for malloc to "uncommit" memory that it's already obtained has been a goal for a long time, but I haven't found a way to make it efficient. On the kernel side there's been the whole "vrange" idea floating around for a while, but it's only going to work on really new kernels, and only on mainstream archs, even if it gets adopted, and so far the interface seems ugly and unlikely to work very well. Here's a potential alternative approach: Whenever a chunk is added (merged, updated, and re-added) to the bin-63 free list (the largest bin, usually not used directly for satisfying allocation requests but as a source for trimming when no smaller-size bins have free chunks that can satisfy the allocation), move it to the end of the bin rather than the beginning if it's larger than the chunk currently at the end. This allows us to track the largest free area. If the number of free chunks in bin 63 exceeds a certain threshold (possibly very small), allocate the largest one (unbin it), mmap a new PROT_NONE zero region over top of the middle of it, and store it in a new linked list of "frozen" memory regions. When no free chunks are available in bin 63 and one is needed, before trying to expand the brk or use mmap, remove a chunk from the frozen list, mprotect it accessible, and use it. If it's excessively large, split it and only mprotect a reasonable-size part, re-adding the remainder to the frozen list. (Note: mprotect can fail here due to commit limit, and in that case, the frozen chunk would be stuck frozen and malloc would have to fail the allocation.) As long as thresholds are chosen right, I don't think this approach would be costly from a performance standpoint. But it might have consequences for fragmentation. So I'd like to discuss it further, see what ideas emerge, and whether it looks reasonable before trying to implement anything. It's also possible that this whole goal is undesirable, since it would create _more_ situations where malloc could fail in the app that's being nice about "uncommitting" its memory; the benefit is that it reduces the chance of malloc failure in other apps when one app has momentarily used huge amounts of memory (allocated in small units, so that mmap wasn't used) then returned it. Rich