From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 6334 invoked from network); 19 Sep 2022 19:21:41 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 19:21:41 -0000 Received: (qmail 29935 invoked by uid 550); 19 Sep 2022 19:21:39 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 29909 invoked from network); 19 Sep 2022 19:21:38 -0000 Date: Mon, 19 Sep 2022 15:21:27 -0400 From: Rich Felker To: Gabriel Ravier Cc: baiyang , James Y Knight , musl , Florian Weimer Message-ID: <20220919192126.GV9709@brightrain.aerifal.cx> References: <2022091915532777412615@gmail.com> <20220919110829.GA2158779@port70.net> <874jx3h76u.fsf@oldenburg.str.redhat.com> <20220919134659.GO9709@brightrain.aerifal.cx> <2022092001404698842815@gmail.com> <20220919181441.GC2158779@port70.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) On Mon, Sep 19, 2022 at 09:07:57PM +0200, Gabriel Ravier wrote: > On 9/19/22 20:14, Szabolcs Nagy wrote: > >* baiyang [2022-09-20 01:40:48 +0800]: > >>I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ). > >> > >>On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1499 > >> > >>I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault. > >obviously internally the implementation can use the internal chunk size... > > > >GetSize(p) is not the exact size (that the user allocated) but an internal > >size (which may be larger) and that must not be exposed *outside* of the > >malloc implementation (other than for diagnostic purposes). > > > >you can have 2 views: > > > >(1) tcmalloc and jemalloc are buggy because they expose an internal > > that must not be exposed (becaues it can break user code). > > > >(2) user code is buggy if it uses malloc_usable_size for any purpose > > other than diagnostic/statistics (because other uses are broken > > on many implementations). > > > >either way the brokenness you want to support is a security hazard > >and you are lucky that musl saves the day: it works hard not to > >expose internal sizes so the code you seem to care about can operate > >safely (which is not true on tcmalloc and jemalloc: the compiler > >may break that code). > > While I would agree that using malloc_usable_size is generally not a > great idea (it's at most acceptable as a small micro-optimization, > but I would only ever expect it to be seen in very well-tested code > in very hot loops, as it is indeed quite easily misused), it seems > like a bit of a stretch to say that all of: > > - sqlite3 (https://github.com/sqlite/sqlite/blob/master/src/mem1.c) > > - systemd > (https://github.com/systemd/systemd/blob/main/src/basic/alloc-util.h > , along with all files using MALLOC_SIZEOF_SAFE, i.e. > src/basic/alloc-util.c, src/basic/compress.c, src/basic/fileio.c, > src/basic/memory-util.h, src/basic/recurse-dir.c, > src/basic/string-util.c, src/libsystemd/sd-netlink/netlink-socket.c, > src/shared/journal-importer.c, src/shared/varlink.c, > src/test/test-alloc-util.c and src/test/test-compress.c) > > - rocksdb (https://github.com/facebook/rocksdb/blob/main/table/block_based/filter_policy.cc > , along with at least 20 other uses) > > - folly (https://github.com/facebook/folly/blob/main/folly/small_vector.h) > > - lzham_codec (https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_mem.cpp) > > - quickjs > (https://raw.githubusercontent.com/bellard/quickjs/master/quickjs.c) > > - redis > (https://github.com/redis/redis/blob/unstable/src/networking.c, > along with a few other uses elsewhere) > > along with so many more well-known projects that I've given up on > listing them, are all buggy because of their usage of > malloc_usable_size... Depending on how you interpret the contract of malloc_usable_size (which was historically ambigious), either (1) or (2) above is *necessarily* true. It's not a matter of opinion just logical consequences of the choice you make. Moreover, it's not at all a stretch to say 7+ popular projects have gigantic UB they don't care to fix. The whole story of musl has been finding *hundreds* of such projects, and eventually getting lots of them fixed. Rich