From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,NICE_REPLY_A, RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 4827 invoked from network); 19 Sep 2022 19:08:15 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 19:08:15 -0000 Received: (qmail 20082 invoked by uid 550); 19 Sep 2022 19:08:12 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 20062 invoked from network); 19 Sep 2022 19:08:12 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date; bh=3azjJLop/jwcTEMYXb5Xmk9BufKFlFv+Ob9taLVjDuY=; b=Unmy6k+7VsMQ28j2czyMd/KlpqWORJz0Jd6pK0d5jeb6u8zD9MlRoWWNty9mqwv/eU B6tvmqz6srtwhCYzhfGTgGR2y/hoxDyUvL6Vbw1A2fF3uSfvg33uq1G/xRyjkYXmI2nw X7WkGRSxdlbR6U+W27x/GjfPHlf5PyAxruZD5Fl+24OcCXcjweR9ADuDKw3KMvpLNAzu 0os2cXTISqrbk4ENsVLTxvAu+V72NtJWsEhU6R1m3GIT+VsXnqnIzvac138wns5YYfbX 1n6jpLqmH9CZo9KwpCe74GTS19jRTwi/CQJ7V8KN/B6PTqbC5wBb86Gla5YEqfB0yC+5 gSag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=3azjJLop/jwcTEMYXb5Xmk9BufKFlFv+Ob9taLVjDuY=; b=4frgGmcCPDDcD9AkGoLffhPn6Hv2zsR8q73aYKr+WS8UDpVF+RvN3DrsiArpY3+qAC v5VfaPRZxBOvXSaMd2OyumgKFVEgfWz0SNy748bp15/ZSWyhpS9foX3q+/nR1TxfyXOZ h/8cCe/kTyigYamzcIuG5PRi+Y6aULjOlTOvJ80qCJTEzo5eYsVBBVf7Fc2MOGqMcYfo lOjh3g19UEFjA+3Gz1IhqmU0HYBcoaEAWdCQDZ7zZNkwwlD+Tf5jYldHnp51HcO4E47f LmB16SAos2uY6M3CZng+1B6zSHS1o+NfhZoRH1wzDfOQt+25CDFF2uAott1/Aq0sHOU+ dLVw== X-Gm-Message-State: ACrzQf3acpyrE7JnVNW/1ZcTfaJ9kyTMmTm2sfaP58yGoIe/U1yrZzcO XggRngMAHEqoW7WyLDk6f9k= X-Google-Smtp-Source: AMsMyM4/c8ILa6KA5AXEG7TdaeQxXsWHivQQAKjPVRfb9KcsQFAYQmlyDX6dvBa0s/mcMD0eamprtQ== X-Received: by 2002:adf:e4ca:0:b0:228:d8b7:48a7 with SMTP id v10-20020adfe4ca000000b00228d8b748a7mr11393047wrm.300.1663614480818; Mon, 19 Sep 2022 12:08:00 -0700 (PDT) Message-ID: Date: Mon, 19 Sep 2022 21:07:57 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.1 Content-Language: en-US To: baiyang , James Y Knight , musl , Florian Weimer , dalias@libc.org References: <2022091915532777412615@gmail.com> <20220919110829.GA2158779@port70.net> <874jx3h76u.fsf@oldenburg.str.redhat.com> <20220919134659.GO9709@brightrain.aerifal.cx> <2022092001404698842815@gmail.com> <20220919181441.GC2158779@port70.net> From: Gabriel Ravier In-Reply-To: <20220919181441.GC2158779@port70.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) On 9/19/22 20:14, Szabolcs Nagy wrote: > * baiyang [2022-09-20 01:40:48 +0800]: >> I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ). >> >> On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1499 >> >> I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault. > obviously internally the implementation can use the internal chunk size... > > GetSize(p) is not the exact size (that the user allocated) but an internal > size (which may be larger) and that must not be exposed *outside* of the > malloc implementation (other than for diagnostic purposes). > > you can have 2 views: > > (1) tcmalloc and jemalloc are buggy because they expose an internal > that must not be exposed (becaues it can break user code). > > (2) user code is buggy if it uses malloc_usable_size for any purpose > other than diagnostic/statistics (because other uses are broken > on many implementations). > > either way the brokenness you want to support is a security hazard > and you are lucky that musl saves the day: it works hard not to > expose internal sizes so the code you seem to care about can operate > safely (which is not true on tcmalloc and jemalloc: the compiler > may break that code). While I would agree that using malloc_usable_size is generally not a great idea (it's at most acceptable as a small micro-optimization, but I would only ever expect it to be seen in very well-tested code in very hot loops, as it is indeed quite easily misused), it seems like a bit of a stretch to say that all of: - sqlite3 (https://github.com/sqlite/sqlite/blob/master/src/mem1.c) - systemd (https://github.com/systemd/systemd/blob/main/src/basic/alloc-util.h , along with all files using MALLOC_SIZEOF_SAFE, i.e. src/basic/alloc-util.c, src/basic/compress.c, src/basic/fileio.c, src/basic/memory-util.h, src/basic/recurse-dir.c, src/basic/string-util.c, src/libsystemd/sd-netlink/netlink-socket.c, src/shared/journal-importer.c, src/shared/varlink.c, src/test/test-alloc-util.c and src/test/test-compress.c) - rocksdb (https://github.com/facebook/rocksdb/blob/main/table/block_based/filter_policy.cc , along with at least 20 other uses) - folly (https://github.com/facebook/folly/blob/main/folly/small_vector.h) - lzham_codec (https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_mem.cpp) - quickjs (https://raw.githubusercontent.com/bellard/quickjs/master/quickjs.c) - redis (https://github.com/redis/redis/blob/unstable/src/networking.c, along with a few other uses elsewhere) along with so many more well-known projects that I've given up on listing them, are all buggy because of their usage of malloc_usable_size...