From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 6008 invoked from network); 19 Sep 2022 19:18:35 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 19:18:35 -0000 Received: (qmail 26216 invoked by uid 550); 19 Sep 2022 19:18:33 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 26184 invoked from network); 19 Sep 2022 19:18:32 -0000 Date: Mon, 19 Sep 2022 15:18:20 -0400 From: Rich Felker To: baiyang Cc: musl Message-ID: <20220919191820.GU9709@brightrain.aerifal.cx> References: <2022091915532777412615@gmail.com> <20220919134319.GN9709@brightrain.aerifal.cx> <202209200132289145679@gmail.com> <20220919181556.GT9709@brightrain.aerifal.cx> <2022092002445709017731@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2022092002445709017731@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) On Tue, Sep 20, 2022 at 02:44:58AM +0800, baiyang wrote: > > Is there a reason you're relying on an unreliable and nonstandard > > function (malloc_usable_size) to do this rather than your program > > keeping track of its own knowledge of the allocated size? This is what > > the C language expects you to do. For example if you have a structure > > that contains a pointer to a dynamically sized buffer, normally you > > store the size in a size_t member right next to that pointer, allowing > > you to make these kind of decisions without having to probe anything. > > Yes, as I have been said, by comparing the number of bytes that > realloc needs to copy in the worst case (the return value of > malloc_usable_size), and the number of bytes we actually need to > copy, we can optimize the performance of realloc in real scenarios > and avoid unnecessary memory copies. You can do exactly the same keeping track of the size yourself. The only correct value malloc_usable_size can return is the value you passed to the allocator. If it's returning a different size, your malloc implementation has a problem that will make you commit UB when you use the result of malloc_usable_size. Many real-world ones do have this problem. > In fact, in scenarios including glibc, tcmalloc, windows crt, mac os > x, uclibc and musl 1.1, we did achieve good optimization results. > > On the other hand, of course we keep the number of bytes actually > allocated, but it doesn't really reflect objectively the number of > bytes to be copied by realloc when the memcpy actually occurs. And > malloc_usable_size() more accurately reflects how many bytes realloc > needs to copy when it degenerates back to malloc-memcpy-free mode. I don't understand your claim here. The size you would store is exactly the size that realloc would have to copy. It if copies more, that's just realloc being inefficient, but the difference is not going to be material anyway. > So our expectation is as mentioned in the man page for linux, mac os > or windows: "The value returned by malloc_usable_size() may be > **greater than** the requested size of the allocation" or "The > memory block size is always at least as large as the allocation it > backs, **and may be larger**." - We expect to get its internal size > to evaluate the cost of memory copying. It's sounding more and more like you did premature optimization without measuring any of this, since there is *no way* the possible amount of excess copying a realloc implementation might make internally could cost more than an extra external function call to malloc_usable_size (even if it did nothing but return). Rich