From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/4777 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: malloc & uncommitting memory Date: Thu, 3 Apr 2014 15:17:53 -0400 Message-ID: <20140403191753.GE26358@brightrain.aerifal.cx> References: <20140402210805.GA852@brightrain.aerifal.cx> <20140403044323.GR26358@brightrain.aerifal.cx> <20140403201725.7509341f@MG> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1396552694 31638 80.91.229.3 (3 Apr 2014 19:18:14 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 3 Apr 2014 19:18:14 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-4781-gllmg-musl=m.gmane.org@lists.openwall.com Thu Apr 03 21:18:07 2014 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1WVn9K-0005bE-7Q for gllmg-musl@plane.gmane.org; Thu, 03 Apr 2014 21:18:06 +0200 Original-Received: (qmail 19534 invoked by uid 550); 3 Apr 2014 19:18:05 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 19526 invoked from network); 3 Apr 2014 19:18:05 -0000 Content-Disposition: inline In-Reply-To: <20140403201725.7509341f@MG> User-Agent: Mutt/1.5.21 (2010-09-15) Xref: news.gmane.org gmane.linux.lib.musl.general:4777 Archived-At: On Thu, Apr 03, 2014 at 08:17:25PM +0200, M. Ziebell wrote: > As far as i understand "uncommitted" memory is allocated address space. > A region in memory which is not used by the program, but the address > space expanded to make a smaller region available to the programm for > use. > Please correct me if I am wrong. I'm not clear on exactly what you're asking so I'll try to just clarify the issue. There are basically three types of memory usage that matter: - Virtual address space: Process-local resource. Aside from relatively low overhead in kernel bookkeeping, this only matters in that, if you run out of addresses, you can't allocate anything else. - Anonymous pages: Global resource. These are pages that actually consume (rather than just temporarily using as cache) physical storage (ram or swap) because they're not backed by a file or device or the zero page or shared (via COW) with another mapping. - Commit charge: Global resource. A superset of anonymous pages, also counting mapped pages that are presently shared (COW) with the zero page, a file, another process, etc. but which will need their own storage if/when they're written. At present, musl's malloc can deallocate only the following: * All three, only when resizing or freeing an allocation that was large enough to be serviced individually by mmap. * Anonymous pages only, by calling madvise with MADV_DONTNEED to reset them to COW copies of the zero page when a large free range coalesces in the heap. In particular, neither commit charge nor virtual address space from the heap used for small allocations is ever freed. Avoiding freeing the virtual address space is generally a good thing, since doing so could result in pathological fragmentation. But not freeing the commit charge means that an application which momentarily used a large amount of memory by allocating a small bit at a time (think hideous tree/list structures used in C++ and glib-based code) then freeing it all will continue to tie part of the commit limit and reduce the amount of memory available to other programs (note: this only applies with overcommit disabled). > Based on that understanding, the idea or your proposal sounds not > really desirable. > You have to take care of tons of exceptions and special cases. > I'm pretty sure I don't understand what uncommitted memory is for, but > I'm heary is hundreds of lines of complex code, if-else constructs > everywhere and bugs. The whole of malloc is under 600 lines, so anything measured in "hundreds of lines" is definitely inappropriate as an addition. However the basic concept (frozen chunk list, described in the first mail) is at most tens of lines of code, and the strategies 1-3 for avoiding fragmentation would each be somewhere on the same order of magnitude, I think. Rich