From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/13381 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: musl: about malloc 'expand heap' issue Date: Tue, 30 Oct 2018 11:00:39 -0400 Message-ID: <20181030150039.GT5150@brightrain.aerifal.cx> References: Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1540911528 5376 195.159.176.226 (30 Oct 2018 14:58:48 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 30 Oct 2018 14:58:48 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-13397-gllmg-musl=m.gmane.org@lists.openwall.com Tue Oct 30 15:58:44 2018 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1gHVTf-0001KC-Bm for gllmg-musl@m.gmane.org; Tue, 30 Oct 2018 15:58:43 +0100 Original-Received: (qmail 9482 invoked by uid 550); 30 Oct 2018 15:00:52 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 9464 invoked from network); 30 Oct 2018 15:00:51 -0000 Content-Disposition: inline In-Reply-To: Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:13381 Archived-At: On Tue, Oct 30, 2018 at 11:11:07AM +0000, zhangwentao (M) wrote: > > Hi all, > I am using musl in my project and I found an issue about the malloc function in musl: > > Issue Description: > * When in muti-threads environment, malloc/free are called in high concurrency. > > Malloc: > Will find 'struct bin' from bitmap(without lock), and allocate memory from the bin (with lock). > > Free: > Will merge the chunk together if the free memory is 'connected' to the existing chunk. > > ? It will remove the old chunk first then combine the chunk to a larger one.. > > ? After merge operation done, insert the chunk to the bin list. > > ? Each of the chunk operation is locked while merging, but the whole steps aren't within a lock. > > So here is the issue: > > 1. There is only one chunk in largest bin list, and Free is on process, just remove the largest bins chunk from bin, the bitmap(mal.binmap) on that bit will be zero. > > 2. A malloc comes, the bitmap is zero, and goes to expand heap. (Actually there is enough memories in process) > > 3. Free operation goes on, and put the merged big chunk to bins. > > But in operation 2, the process has expand heap. > > If we have a loop on step 1-3, the process will expand heap frequently. > So it will cost more Virtual Memory (of course, physical memory would be freed by calling '__madvise' if the chunk is big enough) > > In my environment , we do not have that much virtual memory. I think stop expand heap would a better choice. > > Do you have plan to fix it ?? This is a known issue, and intended to be fixed in the complete redesign of malloc. Fixing it right in the current design seems to impose significant performance costs that I thought were equivalent to, or worse than, just using one global lock. However if it's causing major problems I may be able to make a quick fix that's not too expensive -- I'll take a look again today or tomorrow. Rich