From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,NICE_REPLY_A, RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 17909 invoked from network); 19 Sep 2022 21:02:32 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 21:02:32 -0000 Received: (qmail 28290 invoked by uid 550); 19 Sep 2022 21:02:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28270 invoked from network); 19 Sep 2022 21:02:29 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date; bh=uAytRNdFlQKin4lW50V+/hKtxEpupzp7sT5Rt7uoLG8=; b=iXp5c1B8VFNnXXxu6u5mGXEXA9cM3VPxjFxfcc920rLaR9jRZALlP6/mBp9J838+wp HJsCqKlQECyaixk6lumOaaCmpvILW3tU87hiW+unvpw6TGVo6oO+gRbyTyXi3wcdF9Ef p8jxtQxSskY/AaHKZaFt9BQb6q3G0ZAHqAPdIXVw9Bx3eA10NgwNUH+CD6z98bMwZjXi 0bG0eYXh0oaL6koeJr6c/9O5fTKTxsTXeyPogfGNiprBzIvr2qZb5ws8l49GB9tSjsKQ e9ZyysNm3TYhlDW1fG7727oRkwc51wl9rIu0Vhp4uWR0OpuKCsv+QrbZmppOQB/uxNE4 uBpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=uAytRNdFlQKin4lW50V+/hKtxEpupzp7sT5Rt7uoLG8=; b=lqSI9mlSso3U6Ws00Fsm7My7h7Q9J0zpxJUR0mak5/U8b1YYngUtti38Z7cjl8JmRi zR07rM+YlV1WZS1Foz2ki2lKb3Y/j390a1Fmz4Zdjte3sBP1zhO/GVivBX0tmdAc3f9X HqAbXGUkY+rJe5Z55X0XrxiAxvJHcdHR/8nxWN6aGPmWyZRYwQi4hEp01TrcGmCiUgqv 3/5cR5P7euUTBXRTQYurcJSviIZR68ehW2IfuVe4SYh6oTgTyLb84kHTZagFXgZVs8xS QyyzUg1YS0/DuZnpNXEHhJ9TkbeY55Z4zNCG5LvB69PqW03W+a4xMTQu7R8bjGRUPclp ZWHw== X-Gm-Message-State: ACrzQf1jS6VA5fd8xbeFEeC0y1DTaEDFl4dJvC4ZqWlwpQhdr2IgA9Bw Eks1tVb93unBnMUDeQK+imxRw7/VAFSrrQ== X-Google-Smtp-Source: AMsMyM7QucYgOWb5W+56P68w1UhCO3Vp5Pk2Aw+ruhK/Si5lsQob7PpLQhi1KRMoNZfJwOTFzvgHnQ== X-Received: by 2002:a05:6000:110d:b0:228:db67:75c4 with SMTP id z13-20020a056000110d00b00228db6775c4mr12076074wrw.217.1663621337510; Mon, 19 Sep 2022 14:02:17 -0700 (PDT) Message-ID: <83f1e705-22b2-79b1-123c-1403d3e2281c@gmail.com> Date: Mon, 19 Sep 2022 23:02:16 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.1 Content-Language: en-US To: Rich Felker Cc: baiyang , James Y Knight , musl , Florian Weimer References: <2022091915532777412615@gmail.com> <20220919110829.GA2158779@port70.net> <874jx3h76u.fsf@oldenburg.str.redhat.com> <20220919134659.GO9709@brightrain.aerifal.cx> <2022092001404698842815@gmail.com> <20220919181441.GC2158779@port70.net> <20220919192126.GV9709@brightrain.aerifal.cx> From: Gabriel Ravier In-Reply-To: <20220919192126.GV9709@brightrain.aerifal.cx> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) On 9/19/22 21:21, Rich Felker wrote: > On Mon, Sep 19, 2022 at 09:07:57PM +0200, Gabriel Ravier wrote: >> On 9/19/22 20:14, Szabolcs Nagy wrote: >>> * baiyang [2022-09-20 01:40:48 +0800]: >>>> I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ). >>>> >>>> On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1499 >>>> >>>> I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault. >>> obviously internally the implementation can use the internal chunk size... >>> >>> GetSize(p) is not the exact size (that the user allocated) but an internal >>> size (which may be larger) and that must not be exposed *outside* of the >>> malloc implementation (other than for diagnostic purposes). >>> >>> you can have 2 views: >>> >>> (1) tcmalloc and jemalloc are buggy because they expose an internal >>> that must not be exposed (becaues it can break user code). >>> >>> (2) user code is buggy if it uses malloc_usable_size for any purpose >>> other than diagnostic/statistics (because other uses are broken >>> on many implementations). >>> >>> either way the brokenness you want to support is a security hazard >>> and you are lucky that musl saves the day: it works hard not to >>> expose internal sizes so the code you seem to care about can operate >>> safely (which is not true on tcmalloc and jemalloc: the compiler >>> may break that code). >> While I would agree that using malloc_usable_size is generally not a >> great idea (it's at most acceptable as a small micro-optimization, >> but I would only ever expect it to be seen in very well-tested code >> in very hot loops, as it is indeed quite easily misused), it seems >> like a bit of a stretch to say that all of: >> >> - sqlite3 (https://github.com/sqlite/sqlite/blob/master/src/mem1.c) >> >> - systemd >> (https://github.com/systemd/systemd/blob/main/src/basic/alloc-util.h >> , along with all files using MALLOC_SIZEOF_SAFE, i.e. >> src/basic/alloc-util.c, src/basic/compress.c, src/basic/fileio.c, >> src/basic/memory-util.h, src/basic/recurse-dir.c, >> src/basic/string-util.c, src/libsystemd/sd-netlink/netlink-socket.c, >> src/shared/journal-importer.c, src/shared/varlink.c, >> src/test/test-alloc-util.c and src/test/test-compress.c) >> >> - rocksdb (https://github.com/facebook/rocksdb/blob/main/table/block_based/filter_policy.cc >> , along with at least 20 other uses) >> >> - folly (https://github.com/facebook/folly/blob/main/folly/small_vector.h) >> >> - lzham_codec (https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_mem.cpp) >> >> - quickjs >> (https://raw.githubusercontent.com/bellard/quickjs/master/quickjs.c) >> >> - redis >> (https://github.com/redis/redis/blob/unstable/src/networking.c, >> along with a few other uses elsewhere) >> >> along with so many more well-known projects that I've given up on >> listing them, are all buggy because of their usage of >> malloc_usable_size... > Depending on how you interpret the contract of malloc_usable_size > (which was historically ambigious), either (1) or (2) above is > *necessarily* true. It's not a matter of opinion just logical > consequences of the choice you make. Hmmmm... to me none of the two options you have given are true. Instead, I'd argue that, as seen by the C abstract machine, invocation of malloc_usable_size actually increases the size of a memory block, in place, as if by magic. This may seem stupid to some degree, but to me it is the only way one can make sense of the documentation for malloc_usable_size in glibc, as it has been present in glibc since malloc_usable_size was first added in 1996:     This routine tells you how many bytes you can actually use in an     allocated chunk, which may be more than you requested (although     often not). You can use this many bytes without worrying about     overwriting other allocated objects. Not a particularly great     programming practice, but still sometimes useful. To me, this note directly indicates that the function is intended to work, within the context of the abstract machine, as if it somehow increased the size of the block at the given address. It could perhaps have been better named as something like malloc_try_extend_in_place w.r.t. interpretation of the function's behavior as relating to the abstract machine, but the name it was given probably made more sense at the time, as one may have assumed that the other name would have meant that it might do some more work that just returning the internal allocated size. This interpretation is also the one followed by pretty much every single program out there that uses malloc_usable_size, and the opposite interpretation (that malloc_usable_size is allowed to return a completely useless number that can't be used for anything useful) seems rather asinine to me Such effects on the C abstract machine may seem ludicrous, as though nothing in C should be able to do things like this, but when you search for other examples of extensions to C that do weird things with the abstract machine, you can find some quite quickly: - POSIX makes it so that storing into a function pointer by aliasing into it with a data pointer shall result in defined behavior - GNU attributes like __attribute__((may_alias)) make it so that access through pointers with types with that attribute are defined even if they try to alias normally incompatible object types - x86 intrinsics that access pointers are defined as having defined behavior even if they try to alias normally incompatible object types (the may_alias attribute above is used in GCC to implement them) - POSIX makes it so that calling signal and then making the program multi-threaded shall result in defined behavior - POSIX makes it so that calling fprintf with numbered argument conversion specifications (%n$), the apostrophe flag character (along with a few other things I'm too lazy to list) shall result in defined behavior - Many more... and yet all of these are quite often used and it is generally accepted that an implementation implementing such features should also make it so that the rules of the abstract machine, as implemented, are correctly affected and that the implementation does not suddenly make it so that using these result in UB where they claim that they implement the corresponding features. You might say that you accept the changes POSIX makes to the abstract machine, but not those made by malloc_usable_size, but I would say that to define that function is to implicitly accept the semantics of it as defined by the implementation you're trying to follow: otherwise, it seems to me as though you're implementing the function wrongly, intentionally violating the specification of the function as originally written. And yes, I would also argue that this does indeed make it so that _FORTIFY_SOURCE=3 as currently implemented is wrong (as shown in https://www.openwall.com/lists/musl/2022/09/19/23), instead of malloc_usable_size being a useless function Also, the previously cited e-mail ended up leading me to a glibc bugzilla issue, in which it looks as though glibc developers have explicitly stated they consider the value returned by malloc_usable_size to be validly usable as the object size corresponding to the pointer: https://sourceware.org/bugzilla/show_bug.cgi?id=24775#c6 > > Moreover, it's not at all a stretch to say 7+ popular projects have > gigantic UB they don't care to fix. The whole story of musl has been > finding *hundreds* of such projects, and eventually getting lots of > them fixed. > > Rich