From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_FONT_FACE_BAD,HTML_MESSAGE, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,T_KAM_HTML_FONT_INVALID autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 15166 invoked from network); 19 Sep 2022 20:38:21 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 20:38:21 -0000 Received: (qmail 9871 invoked by uid 550); 19 Sep 2022 20:38:18 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 9832 invoked from network); 19 Sep 2022 20:38:17 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:mime-version:references:subject:cc:to:from:date:from:to :cc:subject:date; bh=X6HyvcQ5pQnCm3Wx0QBPduxJJMTwkOh2AE+6Y94sEak=; b=ZMIYGhGDgtj2Vu3jvpN35VgHSQM6mr793g6HnBgvameDwvKPYPpUJ1e9bD5wNaNtRc S4ue1C4KXcvmBj+QiY75b1LGwFdhMZRIEkbEufJDKBbWonkWDTBvcVIrRJXlPSqk4Tlr kTdnNcVXPXQjT8+pF9qtsJoVdG8BRNxUQ7EMpyB5SGmmy2ZlzzEbGCMv5G8Ux7mA+Igq eJSH91QO5vIMUfULY7W//WWLOf8FwwYLTd/W5Uwtj4oulaxPfNqB4u3hyF6PUTEZtH5Y bJgxDjaW838tMN2ettSKyJhiom48mhfjZkdBEocxMm++oK3INSDm0lvyRklbpL980SnL Y9hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=message-id:mime-version:references:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date; bh=X6HyvcQ5pQnCm3Wx0QBPduxJJMTwkOh2AE+6Y94sEak=; b=kRyX7Isy5SaSQy//wYFQ25sdF+wEGUANIEgB5vtNtK+Tr9rxu7rgbgEgpzl5DjADms Qc6PVvFShlhEK097F+VFnxfAfoivFTNg1JEj1WXHymG+aW8PJIWdkQIuHHMBkfnyvJaz i0XPuamS6FS/t4I9uFOwornIGBH5RtMb5T1nOTJoupdxo9dg1AURot6cyN2TpXiW1BYW uVMVSfbpZBkeEdR4Xgj7oFtitkdG9n47WIInoJ2ULn04PxJiBdGzq8RxluvYjRARobZ1 GeWIqtHNzCsVUam1J27n2bpKql6mqJVPB9/PLucQ+SXGv5+AWrGbwthIQMC8D7D6l9ZQ wxzQ== X-Gm-Message-State: ACrzQf1l74mAPDFCTLqgIG9/0/jTqjKz3CSajbMbVaBGM/Esp9qnwEi4 49OK1eA5QGSHy6USmTC1JjRp9gITJSE6Lg== X-Google-Smtp-Source: AMsMyM5kUtBCG4seJv+nYZ/Mj0dgAb/9QPIP2YQWXvio8NvwmZt1HHU18ePHs+vYPW/NbZqg4kfwBA== X-Received: by 2002:a17:90a:d585:b0:1f4:f9a5:22a9 with SMTP id v5-20020a17090ad58500b001f4f9a522a9mr1106pju.49.1663619885783; Mon, 19 Sep 2022 13:38:05 -0700 (PDT) Date: Tue, 20 Sep 2022 04:38:08 +0800 From: baiyang To: "Rich Felker" Cc: musl References: <2022091915532777412615@gmail.com>, <20220919134319.GN9709@brightrain.aerifal.cx>, <202209200132289145679@gmail.com>, <20220919181556.GT9709@brightrain.aerifal.cx>, <2022092002445709017731@gmail.com>, <20220919191820.GU9709@brightrain.aerifal.cx>, <2022092003453382350548@gmail.com>, <20220919200744.GW9709@brightrain.aerifal.cx>, <2022092004171908873459@gmail.com>, <20220919202838.GY9709@brightrain.aerifal.cx> X-Priority: 3 X-GUID: AB07286D-C649-4CB1-B3DC-6BA7BE5F3479 X-Has-Attach: no X-Mailer: Foxmail 7.2.23.116[cn] Mime-Version: 1.0 Message-ID: <2022092004380716331869@gmail.com> Content-Type: multipart/alternative; boundary="----=_001_NextPart358841201457_=----" Subject: Re: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) This is a multi-part message in MIME format. ------=_001_NextPart358841201457_=---- Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 PiBUbyBteSBrbm93bGVkZ2UgdGhlcmUgaXMgbm8gYWxsb2NhdG9yIHdoaWNoIGRvZXMgY2h1bmsg bWVyZ2luZyBvbiA1MDBrIGNodW5rcyBpbiB0aGUgZGVmYXVsdCBjb25maWd1cmF0aW9uLg0KSXQg Y2FuIGJlIEFOWSBzaXplLCAiNTAwS0IiIGlzIGp1c3QgYW4gZXhhbXBsZS4NCg0KPiBtdXNsIDEu MS54IG5ldmVyIGRpZCBibG9jayBtZXJnaW5nIG9uIG9iamVjdHMgb2YgdGhpcyBzaXplLg0KWWVz LCB3ZSBrbm93biBpdCwgYW5kIHdlIHdpbGwgZXZhbHVhdGUgYWNjb3JkaW5nIHRvIHRoZSBzcGVj aWZpYyBzaXR1YXRpb24gb2YgdGhlIG9wZXJhdGluZyBzeXN0ZW0sIGxpYmMgaW1wbGVtZW50YXRp b24gYW5kIGFsbG9jYXRpb24gc2l6ZS4gQWdhaW4sIHRoaXMgaXMgbm90IHRoZSBwb2ludCBvZiB0 aGlzIGRpc2N1c3Npb24uDQoNCj4gSW4gdGhlIGFic2VuY2Ugb2YgYSBjb25jcmV0ZSwgcXVhbnRp dGF0aXZlIHJlcG9ydCBvZiBob3cgaXQncyBpbXBhY3RpbmcgcGVyZm9ybWFuY2UsIGRlZmluaXRl bHkgbm8uDQpPSywgSSB1bmRlcnN0b29kIGFuZCB0aGFua3MgZm9yIHlvdXIgdGltZS4NCiANCi0t DQoNCiAgIEJlc3QgUmVnYXJkcw0KICBCYWlZYW5nDQogIGJhaXlhbmdAZ21haWwuY29tDQogIGh0 dHA6Ly9pLmJhaXkuY24NCioqKiogPCBFTkQgT0YgRU1BSUwgPiAqKioqIA0KIA0KIA0KRnJvbTog UmljaCBGZWxrZXINCkRhdGU6IDIwMjItMDktMjAgMDQ6MjgNClRvOiBiYWl5YW5nDQpDQzogbXVz bA0KU3ViamVjdDogUmU6IFJlOiBbbXVzbF0gVGhlIGhlYXAgbWVtb3J5IHBlcmZvcm1hbmNlICht YWxsb2MvZnJlZS9yZWFsbG9jKSBpcyBzaWduaWZpY2FudGx5IGRlZ3JhZGVkIGluIG11c2wgMS4y IChjb21wYXJlZCB0byAxLjEpDQpPbiBUdWUsIFNlcCAyMCwgMjAyMiBhdCAwNDoxNzoyMEFNICsw ODAwLCBiYWl5YW5nIHdyb3RlOg0KPiA+IENsZWFybHkgeW91IGRpZCBub3QgbWVhc3VyZSB0aGlz LCBiZWNhdXNlIHdpdGggYmFzaWNhbGx5IGFueQ0KPiA+IHJlYWwtd29ybGQgbWFsbG9jLCBpdCB3 aWxsIGNhbGwgbXJlbWFwIGFuZCBtb3ZlIHRoZSBtZW1vcnkgdmlhDQo+ID4gTU1VLWxldmVsIHJl bWFwcGluZywgd2l0aCBubyBjb3B5aW5nIGludm9sdmVkIHdoYXRzb2V2ZXIuDQo+IA0KPiBPSywg SSd2ZSBiZWVuIHNhaWQgbXVsdGlwbGUgdGltZXM6IA0KPiA+IC4uLmFuZCB0aGUgY2hhbmNlIG9m IG1lcmdpbmcgY2h1bmtzIChzbWFsbCBibG9ja3MpIG9yICoqbXJlbWFwKioNCj4gPiAobGFyZ2Ug YmxvY2tzKSBpbiB0aGUgdW5kZXJsYXllciByZWFsbG9jLg0KIA0KVG8gbXkga25vd2xlZGdlIHRo ZXJlIGlzIG5vIGFsbG9jYXRvciB3aGljaCBkb2VzIGNodW5rIG1lcmdpbmcgb24gNTAwaw0KY2h1 bmtzIGluIHRoZSBkZWZhdWx0IGNvbmZpZ3VyYXRpb24uDQogDQo+ID4gLi4uZXZhbHVhdGUgdGhl IGNvc3QgKGluY2x1ZGluZyB0aGUgcG9zc2liaWxpdHkgb2YgcmVhbGxvYyB1c2luZw0KPiA+IGJs b2NrIG1lcmdpbmcgbGlrZSBpbiBtdXNsIDEuMSwgYW5kIHRlY2huaXF1ZXMgbGlrZSAqKm1yZW1h cCoqIHRvDQo+ID4gYXZvaWQgY29weWluZykNCiANCm11c2wgMS4xLnggbmV2ZXIgZGlkIGJsb2Nr IG1lcmdpbmcgb24gb2JqZWN0cyBvZiB0aGlzIHNpemUuDQogDQo+IFRoZXJlZm9yZSwgd2UgZGV0 ZXJtaW5lZCB0aGF0IHRoZSBwb3NzaWJpbGl0eSBvZiBlYWNoIG1lbW9yeQ0KPiBhbGxvY2F0b3Ig Y2FsbGluZyBtcmVtYXAgaW4gZGlmZmVyZW50IHNpdHVhdGlvbnMgd2FzIHNwZWNpZmljYWxseQ0K PiBjb25zaWRlcmVkIG9uIHRoZSBMSU5VWCBwbGF0Zm9ybS4NCj4gDQo+IEFuZCBJIG1lbnRpb25l ZCBpdCBiZWZvcmU6IHdlIGRpZCBtYXNzaXZlbHkgb3B0aW1pemUgcGVyZm9ybWFuY2UgaW4NCj4g cmVhbC13b3JsZCBhcHBsaWNhdGlvbnMuIFRoZXNlIGFyZSBub3QgdGhlIGZvY3VzIG9mIG91ciBk aXNjdXNzaW9uLg0KPiANCj4gU28gbm93IGl0IGlzIGNlcnRhaW4gdGhhdCBpbiBtdXNsIG1hbGxv Y25nOg0KPiAxLiBtYWxsb2NfdXNhYmxlX3NpemUgd2lsbCBhbHdheXMganVzdCByZXR1cm4gdGhl IHNpemUgc3VibWl0dGVkIGJ5DQo+IHRoZSB1c2VyIHRvIG1hbGxvYywgbm90IHRoZSBhY3R1YWwg c2l6ZSBhbGxvY2F0ZWQgaW5zaWRlIGl0LCByaWdodD8NCiANClllcy4NCiANCj4gMi4gV2UgaGF2 ZSBubyBwbGFucyB0byBpbXByb3ZlIG1hbGxvY191c2FibGVfc2l6ZSBwZXJmb3JtYW5jZSB5ZXQs IHJpZ2h0Pw0KIA0KSW4gdGhlIGFic2VuY2Ugb2YgYSBjb25jcmV0ZSwgcXVhbnRpdGF0aXZlIHJl cG9ydCBvZiBob3cgaXQncw0KaW1wYWN0aW5nIHBlcmZvcm1hbmNlLCBkZWZpbml0ZWx5IG5vLg0K IA0KRXZlbiBpZiB0aGVyZSBpcyBzdWNoIGEgcmVwb3J0LCBpZiB0aGUgc291cmNlIG9mIHRoZSBw cm9ibGVtIGlzIHRoYXQNCnlvdSdyZSBncmF0dWl0b3VzbHkgdHJ5aW5nIHRvIHNlY29uZC1ndWVz cyByZWFsbG9jIGFuZCBtYWtpbmcgeW91cg0KcHJvZ3JhbSBzbG93ZXIsIHJhdGhlciB0aGFuIGp1 c3QgY2FsbGluZyByZWFsbG9jLCBpdCdzIG5vdCBhIHZlcnkNCmludGVyZXN0aW5nIHJlcG9ydC4g bWFsbG9jX3VzYWJsZV9zaXplIGlzICpub3QqIGEgZnVuY3Rpb24gd2hvc2UgdXNlDQppcyByZWNv bW1lbmRlZCBhbmQgaXQncyBvbmx5IHRoZXJlIGF0IGFsbCBiZWNhdXNlIHdlIGNhbid0IHJlbW92 ZSBhbg0KaW50ZXJmYWNlIG9uY2UgYWRkaW5nIGl0Lg0KIA0KSWYgdGhlIHByb2JsZW0gaXMgYWN0 dWFsbHkgdGhlIHNpemUgZGV0ZXJtaW5hdGlvbiBpbiByZWFsbG9jIGFuZCBmcmVlLA0KYW5kIGlm IHlvdSd2ZSAqbWVhc3VyZWQqIHRoYXQgYW5kIGNhbiBtYWtlIGEgcXVhbnRpdGF0aXZlIHJlcG9y dCBvbg0KaG93IGl0IGFmZmVjdGVkICpyZWFsIHdvcmxkIHVzYWdlKiBub3QgYSBtYWRlLXVwIGJl bmNobWFyaywgdGhlbiB0aGF0DQppcyBwb3RlbnRpYWxseSBhY3Rpb25hYmxlLg0KIA0KUmljaA0K ------=_001_NextPart358841201457_=---- Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable =0A
> To my knowledge there is no allocator = which does chunk merging on 500k chu= nks in the default configuration.
It can be ANY size, "500KB" is just an exampl= e.

> musl 1.1.x never did block merging on object= s of this size.
Yes, we known it, and we will evaluate according to the specific= situation of the operating system, libc implementation and allocation siz= e. Again, this = is not the point of this discussion.

> In the absence of a concrete, quantitative = report of how it's impacting performance, definitely no.
<= div>OK, I understood and thanks for your ti= me.
 =0A
--
   Best Regards
&n= bsp; BaiYang
  baiyang@g= mail.com
  http://i.baiy.cn<= /a>
**** <= ; END OF EMAIL > **** 
 
 
Fro= m: Rich Felker
<= b>Date: 2022-09-20 04:28
To: baiyang
CC: musl
Subject:&nb= sp;Re: Re: [musl] The heap memory performance (malloc/free/realloc) is sig= nificantly degraded in musl 1.2 (compared to 1.1)
On Tue, Sep 20, 2022 at 04:17:20AM +0800, baiyang wrote:
=0A
&= gt; > Clearly you did not measure this, because with basically any=0A
> > real-world malloc, it will call mremap and move the mem= ory via
=0A
> > MMU-level remapping, with no copying involv= ed whatsoever.
=0A
>
=0A
> OK, I've been said mul= tiple times:
=0A
> > ...and the chance of merging chunks (= small blocks) or **mremap**
=0A
> > (large blocks) in the u= nderlayer realloc.
=0A
 
=0A
To my knowledge there = is no allocator which does chunk merging on 500k
=0A
chunks in th= e default configuration.
=0A
 
=0A
> > ...eva= luate the cost (including the possibility of realloc using
=0A
&g= t; > block merging like in musl 1.1, and techniques like **mremap** to<= /div>=0A
> > avoid copying)
=0A
 
=0A
mus= l 1.1.x never did block merging on objects of this size.
=0A
&nbs= p;
=0A
> Therefore, we determined that the possibility of each= memory
=0A
> allocator calling mremap in different situations= was specifically
=0A
> considered on the LINUX platform.=0A
>
=0A
> And I mentioned it before: we did massive= ly optimize performance in
=0A
> real-world applications. Thes= e are not the focus of our discussion.
=0A
>
=0A
>= ; So now it is certain that in musl mallocng:
=0A
> 1. malloc_= usable_size will always just return the size submitted by
=0A
>= ; the user to malloc, not the actual size allocated inside it, right?=0A
 
=0A
Yes.
=0A
 
=0A
> 2.= We have no plans to improve malloc_usable_size performance yet, right?=0A
 
=0A
In the absence of a concrete, quantitative = report of how it's
=0A
impacting performance, definitely no.=0A
 
=0A
Even if there is such a report, if the source= of the problem is that
=0A
you're gratuitously trying to second-= guess realloc and making your
=0A
program slower, rather than jus= t calling realloc, it's not a very
=0A
interesting report. malloc= _usable_size is *not* a function whose use
=0A
is recommended and= it's only there at all because we can't remove an
=0A
interface = once adding it.
=0A
 
=0A
If the problem is actuall= y the size determination in realloc and free,
=0A
and if you've *= measured* that and can make a quantitative report on
=0A
how it a= ffected *real world usage* not a made-up benchmark, then that
=0Ais potentially actionable.
=0A
 
=0A
Rich
=0A=
=0A ------=_001_NextPart358841201457_=------