From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/205 Path: news.gmane.org!not-for-mail From: Vasiliy Kulikov Newsgroups: gmane.linux.lib.musl.general Subject: Re: holywar: malloc() vs. OOM Date: Sun, 24 Jul 2011 22:25:33 +0400 Message-ID: <20110724182533.GB6429@albatros> References: <20110724103325.GA24069@albatros> <20110724124034.GI132@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: dough.gmane.org 1311532123 10791 80.91.229.12 (24 Jul 2011 18:28:43 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 24 Jul 2011 18:28:43 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-289-gllmg-musl=m.gmane.org@lists.openwall.com Sun Jul 24 20:28:36 2011 Return-path: Envelope-to: gllmg-musl@lo.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by lo.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1Ql3Pj-0004gJ-PP for gllmg-musl@lo.gmane.org; Sun, 24 Jul 2011 20:28:31 +0200 Original-Received: (qmail 3977 invoked by uid 550); 24 Jul 2011 18:28:31 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 3969 invoked from network); 24 Jul 2011 18:28:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=YGWaD3rZAFVOkEYO+3W0n7171iU0Y2PNqAFtRChPWtw=; b=SalbtSpl8b2Ls7V9ntyMvgjm2N4ea3C79phHwNd19/Xn4h4vZHot3QZnVfA6HFCoXO v3wOGlUYJnJovv9HWUyi3JYGnT7HH5kssWSFTxe25N0+yKnER/E9q60Bc9QuX6S2YSjJ 6ym8KaXelvxiYphnc0Y3vzSHWZaULeSYkX8w0= Original-Sender: Vasiliy Kulikov Content-Disposition: inline In-Reply-To: <20110724124034.GI132@brightrain.aerifal.cx> User-Agent: Mutt/1.5.20 (2009-06-14) Xref: news.gmane.org gmane.linux.lib.musl.general:205 Archived-At: Rich, On Sun, Jul 24, 2011 at 08:40 -0400, Rich Felker wrote: > On Sun, Jul 24, 2011 at 02:33:25PM +0400, Vasiliy Kulikov wrote: > > Rich, > > > > This is more a question about your malloc() failure policy for musl than > > an actual proposal. > > > > [...] > > > > In theory, these are bugs of applications and not of libc, and they > > should be fully handled in programs, not in libc. Period. > > > > But looking at the problem from the pragmatic point of view we'll see > > that libc is actually the easiest place where the problem may be > > workarounded (not fixed, surely). The workaround would be simply > > raising SIGKILL if malloc() fails (either because of brk() or mmap()). > > For the rare programs craving to handle OOM such code should be used: > > This is absolutely wrong and non-conformant. It will also ruin all > robust programs and result in massive data loss, deadlock with shared > locks due to failure to release locks before termination, and all > sorts of ills. Oh, I forgot one major detail - the kernel by default have memory overcommit enabled (sysctl vm.overcommit_memory=0). It means that even root owned program may be killed by OOM killer in case of system global OOM :-) There are procfs adjustments for such processes, but the history shows that OOM killer logic is often somehow unexpected (if not broken). Also it was rewritten almost from scratch in the latest kernels, so I'd expect new bugs in it. For overcommit disabled OOM graceful handling should be possible, but I'm not sure it is _guaranteed_ that memory allocated by brk() and mmap() will be really available in the future. So, yes, if the program guarantees that it gracefully handle OOM *for sure*, then the workaround is indeed a breakage. But I'm sure such programs are hell rare. BTW, do you know such programs, except DBUS? :) > The only common situation I can think of where it > might happen to initially access a high offset first is when calling > glibc's memcpy which sometimes chooses to copy backwards. musl's > memcpy does not take this liberty, even if it might be faster in some > cases, for that very reason - it's dangerous to access high offsets > first if a program was not careful about checking the return value of > malloc. Also the program/libs might (re)implement such functions for the performance gain. > A better solution might be to have a gcc option to generate a read > from the base address the first time a function performs arithmetic on > a pointer it has not already checked. This is valid because the C > language does not allow pointer arithmetic to cross object boundaries, > and this approach could be made 100% correct rather than being a > heuristic that breaks correct applications. A good idea. It would be interesting to show actual numbers of the slowdown. However, most of the time it would be a slowdown for no actual gain. Thanks, -- Vasiliy