From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/11737 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: [PATCH] malloc/expand_heap.c: really try to use all available memory Date: Tue, 18 Jul 2017 13:27:54 -0400 Message-ID: <20170718172754.GR1627@brightrain.aerifal.cx> References: <20170718125259.19932-1-stevenrwalter@gmail.com> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1500398920 12201 195.159.176.226 (18 Jul 2017 17:28:40 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 18 Jul 2017 17:28:40 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-11750-gllmg-musl=m.gmane.org@lists.openwall.com Tue Jul 18 19:28:31 2017 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1dXWIN-0002UI-7T for gllmg-musl@m.gmane.org; Tue, 18 Jul 2017 19:28:27 +0200 Original-Received: (qmail 26332 invoked by uid 550); 18 Jul 2017 17:28:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 26181 invoked from network); 18 Jul 2017 17:28:06 -0000 Content-Disposition: inline In-Reply-To: <20170718125259.19932-1-stevenrwalter@gmail.com> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:11737 Archived-At: On Tue, Jul 18, 2017 at 08:52:59AM -0400, Steven Walter wrote: > Previously expand_heap would ask for increasingly larger mmap areas > every time, in order to avoid allocating a bunch of small areas and > fragmenting memory. However, after a point, we may ask for an mmap area > so large that it can't possibly succeed, even though there is still > system memory available. This is particularly likely on a 32-bit system > with gigs of RAM, or else on a no-MMU system with pretty much any amount > of RAM. Without an MMU to make physically-discontiguous pages appear > contigious, the chance of any large mmap succeeding are very low. This is partly intentional, since beyond this point fragmentation is expected to be catastrophic to the point that it would prevent mmap (and on nommu, bring down the whole system). But if it's seriously capping how much memory you can get with malloc on a 32-bit system (esp one with mmu), some tweaking is probably needed. > To fix this, support decreasing mmap_step once we hit an allocation > failure. We'll try smaller and smaller amounts until we either ask for > a single page or exactly as many pages as we currently need. Only if > that fails do we fail the overall request. This is probably a really bad idea, though I don't have data to back that up; it's just an intuition about what happens. Would it work to instead decrease the growth rate on requests? If you're at the point where a single page (or a few pages) makes the difference between success and failure, it's already highly unpredictable (due to aslr & other memory layout factors) whether your job is going to succeed either way. If you get in a situation where there are hundreds of megs free but you can't use them, that's a real problem, but I think it's one we can solve without dropping down to small requests. Note that there's also a bad fragmentation behavior from your proposal if a large expand_heap is only temporarily blocked, e.g. by a short-lived mmap. In that case, dropping down to smaller requests will just fragment memory horribly so that subsequent requests (after the mmap is released) will fail even worse. Rich