From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_FONT_FACE_BAD,HTML_FONT_LOW_CONTRAST, HTML_MESSAGE,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, T_KAM_HTML_FONT_INVALID autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 27100 invoked from network); 19 Sep 2022 17:41:03 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 17:41:03 -0000 Received: (qmail 28378 invoked by uid 550); 19 Sep 2022 17:41:00 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28358 invoked from network); 19 Sep 2022 17:40:59 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:mime-version:references:subject:cc:to:from:date:from:to :cc:subject:date; bh=+rsVJzVL1t2MT99IeYSTjL23Ev0A46xJQCmFhCZ893k=; b=SEgOxV3VnWFspsmJMFaOxxKWxZon1+sjN+ESDAGzGnMhn7/LNKEK1lu5nRTXY/ckWi gwfWx51sojJ56ATPsfgQaxxky/T9fOg9HheVUTojEub1LvHSGV/sHBfhH7B6FJ8L5Qra tE+SdihOykxwmmcE6ICVbadErEtkWBARcVr/XquIuaNwLsgd07Fd+ppdHGGHnpcrF8mc bY2yy1YcumWNH/+W3rawUCyF7t07hXtyUjd5TVixr8OqAVKE47r8tU53mtRw3rOOyDGN jLspXFIycViCe7KLKGMQMiTISbKUkMgzoUw18bEFjTxIgQ/9Ooi94n3FbZrmrmAYQegG t6ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=message-id:mime-version:references:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date; bh=+rsVJzVL1t2MT99IeYSTjL23Ev0A46xJQCmFhCZ893k=; b=gjBMxgwcHxP7FKNAkS3wjsxp3EM94u6QBV+apeIgdC+tCzfAM/RQ2jS/ZPPEdFtsbV 1RGPRZbKAaQr0Xi2QUTUT2MSW3EFr0VnKSz3Fzr03BI0I3sz9weCWUEhBkZMJJrlP+9e FVMygnD8E71AfQSmy/4LJKJJbvfTFQZdS5B5GZvpcqXVdSde2/Dc8PInNQbbgbag3nga 22mNZyIWV59H0dtwcRq/1S2bfiDtlg7G6+WwrsDY+CVpPe9TCwV2/IN3SWmXQAIaLcKR OQjOdwd8zE3ZTaWQQtJXt3ARDrASNJ78Jp2A5hc6d1I0qPanW98iKUgrxW0Nk7suKnri t5tQ== X-Gm-Message-State: ACrzQf0c4WTOKcvzanGYWirR16UcTYoJ7K07ySmog+Ib2K66cRZjgn/V s4wRMLLVcoqABqZMCEDCrJSIpWZ4MkelWA== X-Google-Smtp-Source: AMsMyM7RmPK7NOfgdX8f1slbTIgiW4HTERet+Fa/Kys2dofUT7h4yyWz7INn+8kZOcmQ8m7gUeG0fw== X-Received: by 2002:a17:902:c951:b0:176:d421:7502 with SMTP id i17-20020a170902c95100b00176d4217502mr859527pla.72.1663609247598; Mon, 19 Sep 2022 10:40:47 -0700 (PDT) Date: Tue, 20 Sep 2022 01:40:48 +0800 From: baiyang To: "James Y Knight" , musl Cc: "Florian Weimer" References: <2022091915532777412615@gmail.com>, <20220919110829.GA2158779@port70.net>, <874jx3h76u.fsf@oldenburg.str.redhat.com>, <20220919134659.GO9709@brightrain.aerifal.cx>, X-Priority: 3 X-GUID: CD8366B9-85A0-4F0B-B1EA-46807FC3194F X-Has-Attach: no X-Mailer: Foxmail 7.2.23.116[cn] Mime-Version: 1.0 Message-ID: <2022092001404698842815@gmail.com> Content-Type: multipart/alternative; boundary="----=_001_NextPart407476881701_=----" Subject: Re: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) This is a multi-part message in MIME format. ------=_001_NextPart407476881701_=---- Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 SGkgSmFtZXMsDQoNCkkgbG9va2VkIGF0IHRoZSBjb2RlIG9mIHRjbWFsbG9jLCBidXQgSSBkaWRu J3QgZmluZCBhbnkgb2YgdGhlIHByb2JsZW1zIHlvdSBtZW50aW9uZWQgaW4gdGhlIGltcGxlbWVu dGF0aW9uIG9mIG1hbGxvY191c2FibGVfc2l6ZSAoc2VlOiBodHRwczovL2dpdGh1Yi5jb20vZ29v Z2xlL3RjbWFsbG9jL2Jsb2IvOTE3OWJiODg0ODQ4YzMwNjE2NjY3YmExMjliY2Y5YWZlZTExNGMz Mi90Y21hbGxvYy90Y21hbGxvYy5jYyNMMTA5OSApLg0KDQpPbiB0aGUgY29udHJhcnksIHNpbWls YXIgdG8gbXVzbCwgdGNtYWxsb2MgYWxzbyBkaXJlY3RseSB1c2VzIHRoZSByZXR1cm4gdmFsdWUg b2YgbWFsbG9jX3VzYWJsZV9zaXplIGluIGl0cyByZWFsbG9jIGltcGxlbWVudGF0aW9uIHRvIGRl dGVybWluZSB3aGV0aGVyIG1lbW9yeSBuZWVkcyB0byBiZSByZWFsbG9jYXRlZDogaHR0cHM6Ly9n aXRodWIuY29tL2dvb2dsZS90Y21hbGxvYy9ibG9iLzkxNzliYjg4NDg0OGMzMDYxNjY2N2JhMTI5 YmNmOWFmZWUxMTRjMzIvdGNtYWxsb2MvdGNtYWxsb2MuY2MjTDE0OTkNCg0KSSB0aGluayB0aGlz IGlzIGVub3VnaCB0byBzaG93IHRoYXQgdGhlIHJldHVybiB2YWx1ZSBvZiBtYWxsb2NfdXNhYmxl X3NpemUgaW4gdGNtYWxsb2MgaXMgYWNjdXJhdGUgYW5kIHJlbGlhYmxlLCBvdGhlcndpc2UgaXRz IG93biByZWFsbG9jIHdpbGwgY2F1c2UgYSBzZWdtZW50IGZhdWx0Lg0KDQpUaGFua3MgOi0pDQoN Ci0tDQoNCiAgIEJlc3QgUmVnYXJkcw0KICBCYWlZYW5nDQogIGJhaXlhbmdAZ21haWwuY29tDQog IGh0dHA6Ly9pLmJhaXkuY24NCioqKiogPCBFTkQgT0YgRU1BSUwgPiAqKioqIA0KIA0KIA0KRnJv bTogSmFtZXMgWSBLbmlnaHQNCkRhdGU6IDIwMjItMDktMTkgMjE6NTMNClRvOiBtdXNsDQpDQzog RmxvcmlhbiBXZWltZXI7IGJhaXlhbmcNClN1YmplY3Q6IFJlOiBbbXVzbF0gVGhlIGhlYXAgbWVt b3J5IHBlcmZvcm1hbmNlIChtYWxsb2MvZnJlZS9yZWFsbG9jKSBpcyBzaWduaWZpY2FudGx5IGRl Z3JhZGVkIGluIG11c2wgMS4yIChjb21wYXJlZCB0byAxLjEpDQpJbmRlZWQuIFJlZEhhdCBtZW50 aW9uZWQgdGhhdCBwcm9ibGVtIGluIHRoZWlyIHJlY2VudCBwb3N0IGFib3V0IF9GT1JUSUZZX1NP VVJDRT0zLCBoZXJlDQpodHRwczovL2RldmVsb3BlcnMucmVkaGF0LmNvbS9hcnRpY2xlcy8yMDIy LzA5LzE3L2djY3MtbmV3LWZvcnRpZmljYXRpb24tbGV2ZWwNCg0KIiIiDQpfRk9SVElGWV9TT1VS Q0U9MyByZXZlYWxlZCBhbm90aGVyIHBhdHRlcm4uIEFwcGxpY2F0aW9ucyBzdWNoIGFzIHN5c3Rl bWQgdXNlZCBtYWxsb2NfdXNhYmxlX3NpemUgdG8gZGV0ZXJtaW5lIGF2YWlsYWJsZSBzcGFjZSBp biBvYmplY3RzIGFuZCB0aGVuIHVzZWQgdGhlIHJlc2lkdWFsIHNwYWNlLiBUaGUgZ2xpYmMgbWFu dWFsIGRpc2NvdXJhZ2VzIHRoaXMgdHlwZSBvZiB1c2FnZSwgZGljdGF0aW5nIHRoYXQgbWFsbG9j X3VzYWJsZV9zaXplIGlzIGZvciBkaWFnbm9zdGljIHB1cnBvc2VzIG9ubHkuIEJ1dCBhcHBsaWNh dGlvbnMgdXNlIHRoZSBmdW5jdGlvbiBhcyBhIGhhY2sgdG8gYXZvaWQgcmVhbGxvY2F0aW5nIGJ1 ZmZlcnMgd2hlbiB0aGVyZSBpcyBzcGFjZSBpbiB0aGUgdW5kZXJseWluZyBtYWxsb2MgY2h1bmsu IFRoZSBpbXBsZW1lbnRhdGlvbiBvZiBtYWxsb2NfdXNhYmxlX3NpemUgbmVlZHMgdG8gYmUgZml4 ZWQgdG8gcmV0dXJuIHRoZSBhbGxvY2F0ZWQgb2JqZWN0IHNpemUgaW5zdGVhZCBvZiB0aGUgY2h1 bmsgc2l6ZSBpbiBub24tZGlhZ25vc3RpYyB1c2UuIEFsdGVybmF0aXZlbHksIGFub3RoZXIgc29s dXRpb24gaXMgdG8gZGVwcmVjYXRlIHRoZSBmdW5jdGlvbi4gQnV0IHRoYXQgaXMgYSB0b3BpYyBm b3IgZGlzY3Vzc2lvbiBieSB0aGUgZ2xpYmMgY29tbXVuaXR5Lg0KIiIiDQoNCk9uIE1vbiwgU2Vw IDE5LCAyMDIyIGF0IDk6NDcgQU0gUmljaCBGZWxrZXIgPGRhbGlhc0BsaWJjLm9yZz4gd3JvdGU6 DQpPbiBNb24sIFNlcCAxOSwgMjAyMiBhdCAwMjozNjo0MVBNICswMjAwLCBGbG9yaWFuIFdlaW1l ciB3cm90ZToNCj4gKiBTemFib2xjcyBOYWd5Og0KPiANCj4gPiB1bmxpa2UgbXVzbCB0aG9zZSBp bXBsZW1lbnRhdGlvbnMgZG9uJ3QgcmV0dXJuIGV4YWN0IHNpemUgbm9yIGhhdmUgdGhlDQo+ID4g c2FtZSBzZWN1cml0eSBhbmQgbWVtb3J5IGZyYWdtZW50YXRpb24gZ3VhcmFudGVlcywgc28gYmFk IGNvbXBhcmlzaW9uLg0KPiA+DQo+ID4gdGNtYWxsb2M6DQo+ID4gICAvLyBSZXR1cm5zIHRoZSBh Y3R1YWwgbnVtYmVyIE4gb2YgYnl0ZXMgcmVzZXJ2ZWQgYnkgdGNtYWxsb2MgZm9yIHRoZSBwb2lu dGVyDQo+ID4gICAvLyBwLiAgVGhpcyBudW1iZXIgbWF5IGJlIGVxdWFsIHRvIG9yIGdyZWF0ZXIg dGhhbiB0aGUgbnVtYmVyIG9mIGJ5dGVzDQo+ID4gICAvLyByZXF1ZXN0ZWQgd2hlbiBwIHdhcyBh bGxvY2F0ZWQuDQo+ID4gICAvLw0KPiA+ICAgLy8gVGhpcyBmdW5jdGlvbiBpcyBqdXN0IHVzZWZ1 bCBmb3Igc3RhdGlzdGljcyBjb2xsZWN0aW9uLiAgVGhlIGNsaWVudCBtdXN0DQo+ID4gICAvLyAq bm90KiByZWFkIG9yIHdyaXRlIGZyb20gdGhlIGV4dHJhIGJ5dGVzIHRoYXQgYXJlIGluZGljYXRl ZCBieSB0aGlzIGNhbGwuDQo+ID4NCj4gPiBqZW1hbGxvYzoNCj4gPiAgICAgICA8cGFyYT5UaGUg PGZ1bmN0aW9uPm1hbGxvY191c2FibGVfc2l6ZSgpPC9mdW5jdGlvbj4gZnVuY3Rpb24NCj4gPiAg ICAgICByZXR1cm5zIHRoZSB1c2FibGUgc2l6ZSBvZiB0aGUgYWxsb2NhdGlvbiBwb2ludGVkIHRv IGJ5DQo+ID4gICAgICAgPHBhcmFtZXRlcj5wdHI8L3BhcmFtZXRlcj4uICBUaGUgcmV0dXJuIHZh bHVlIG1heSBiZSBsYXJnZXIgdGhhbiB0aGUgc2l6ZQ0KPiA+ICAgICAgIHRoYXQgd2FzIHJlcXVl c3RlZCBkdXJpbmcgYWxsb2NhdGlvbi4gIFRoZQ0KPiA+ICAgICAgIDxmdW5jdGlvbj5tYWxsb2Nf dXNhYmxlX3NpemUoKTwvZnVuY3Rpb24+IGZ1bmN0aW9uIGlzIG5vdCBhDQo+ID4gICAgICAgbWVj aGFuaXNtIGZvciBpbi1wbGFjZSA8ZnVuY3Rpb24+cmVhbGxvYygpPC9mdW5jdGlvbj47IHJhdGhl cg0KPiA+ICAgICAgIGl0IGlzIHByb3ZpZGVkIHNvbGVseSBhcyBhIHRvb2wgZm9yIGludHJvc3Bl Y3Rpb24gcHVycG9zZXMuICBBbnkNCj4gPiAgICAgICBkaXNjcmVwYW5jeSBiZXR3ZWVuIHRoZSBy ZXF1ZXN0ZWQgYWxsb2NhdGlvbiBzaXplIGFuZCB0aGUgc2l6ZSByZXBvcnRlZA0KPiA+ICAgICAg IGJ5IDxmdW5jdGlvbj5tYWxsb2NfdXNhYmxlX3NpemUoKTwvZnVuY3Rpb24+IHNob3VsZCBub3Qg YmUNCj4gPiAgICAgICBkZXBlbmRlZCBvbiwgc2luY2Ugc3VjaCBiZWhhdmlvciBpcyBlbnRpcmVs eSBpbXBsZW1lbnRhdGlvbi1kZXBlbmRlbnQuDQo+IA0KPiBUaGVzZSBpbXBsZW1lbnRhdGlvbnMg YXJlIGJ1Z2d5IG9yIGF0IGxlYXN0IG1pcy1kb2N1bWVudGVkLiAgVGhlDQo+IGludGVyZmFjZSBj b250cmFjdCBpcyBjbGVhcmx5IHRoYXQgZm9yIHRoYXQgcGFydGljdWxhciBvYmplY3QsIHRoZSBl eHRyYQ0KPiBieXRlcyBpbiB0aGUgYWxsb2NhdGlvbiBhcmUgYXZhaWxhYmxlIGZvciByZWFkaW5n IGFuZCB3cml0aW5nLiAgSXQgaXMNCj4gbm90IGd1YXJhbnRlZWQgdGhhdCB0aGUgYWxsb2NhdG9y IHdpbGwgYWx3YXlzIHByb3ZpZGUgdGhlIHNhbWUgbnVtYmVyIG9mDQo+IGV4dHJhIGJ5dGVzIGZv ciB0aGUgc2FtZSByZXF1ZXN0ZWQgc2l6ZSwgYnV0IHRoZXkgbXVzdCBiZSB0aGVyZSBmb3IgdGhl DQo+IGFsbG9jYXRpb24gYmVpbmcgZXhhbWluZWQuICBJdCdzIGV2ZW4gaW4gdGhlIG5hbWUgb2Yg dGhlIGZ1bmN0aW9uIQ0KDQpJJ20gbm90IHN1cmUgSSB1bmRlcnN0YW5kIHdoYXQgeW91J3JlIHNh eWluZywgYnV0IHRoZSBjb3JlIHByb2JsZW0NCnRoYXQgcmVhbGx5IGNhbid0IGJlIHNvbHZlZCBp cyBwb3RlbnRpYWwgZGlzY3JlcGFuY3kgYmV0d2VlbiB0aGUNCm1hbGxvYyBpbXBsZW1lbnRhdGlv bidzIGlkZWEgb2YgdXNhYmxlIGFuZCB0aGUgY29tcGlsZXIncy4gRm9yDQpleGFtcGxlOg0KDQog ICAgICAgIGNoYXIgKnAgPSBtYWxsb2MoMSk7DQogICAgICAgIGlmIChtYWxsb2NfdXNhYmxlX3Np emUocCk+MSkgcFsxXSA9IDQyOw0KDQp3aWxsIGNhdXNlIGEgY29tcGlsZXIgdGhhdCdzIGFjdGl2 ZWx5IGRldGVjdGluZyBVQiB0byBhYm9ydCB0aGUNCnByb2dyYW0gd2hlbiBtYWxsb2NfdXNhYmxl X3NpemUgcmV0dXJucyBhIHZhbHVlIGxhcmdlciB0aGFuIDEuDQoNClJpY2gNCg== ------=_001_NextPart407476881701_=---- Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0A
Hi James,

= I looked at the code of tcmalloc, but I didn't find any of the problems yo= u mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179= bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ).

On the contrary, similar to musl, tcmalloc also direct= ly uses the return value of malloc_usable_size in its realloc implementati= on to determine whether memory needs to be reallocated: https://github.com= /google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tc= malloc.cc#L1499
=0A

I think this is enough to show that the return value o= f malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault.

<= /div>
Thanks :-)

=0A
--

 <= img src=3D"http://baiy.cn/image/portrait_100px.jpg" border=3D"0">  Best Regards
  BaiYang
 =  baiyang@gmail.com
 =  http://i.baiy.cn
**** < END OF EMAIL > **= ** 
 
<= div> 
Date:&nbs= p;2022-09-19 21:53
To: musl
Subject: Re: [musl] The heap memory per= formance (malloc/free/realloc) is significantly degraded in musl 1.2 (comp= ared to 1.1)
Indeed. RedHat mentioned that problem in their recent = post about _FORTIFY_SOURCE=3D3, here
https://develope= rs.redhat.com/articles/2022/09/17/gccs-new-fortification-level

"""
_FORTIFY_SOURCE=3D3 revealed another p= attern. Applications such as systemd used malloc_usable_size to determine = available space in objects and then used the residual space. The glibc man= ual discourages this type of usage, dictating that malloc_usable_size is f= or diagnostic purposes only. But applications use the function as a hack t= o avoid reallocating buffers when there is space in the underlying malloc = chunk. The implementation of malloc_usable_size needs to be fixed to retur= n the allocated object size instead of the chunk size in non-diagnostic us= e. Alternatively, another solution is to deprecate the function. But that = is a topic for discussion by the glibc community.
"""

On Mon, Sep= 19, 2022 at 9:47 AM Rich Felker <da= lias@libc.org> wrote:
On Mon, Sep 19, 2022 at 02:36:41PM +0200, Florian Wei= mer wrote:
=0A> * Szabolcs Nagy:
=0A>
=0A> > unlike = musl those implementations don't return exact size nor have the
=0A>= > same security and memory fragmentation guarantees, so bad comparisio= n.
=0A> >
=0A> > tcmalloc:
=0A> >   /= / Returns the actual number N of bytes reserved by tcmalloc for the pointe= r
=0A> >   // p.  This number may be equal to or g= reater than the number of bytes
=0A> >   // requested w= hen p was allocated.
=0A> >   //
=0A> >  =  // This function is just useful for statistics collection.  The= client must
=0A> >   // *not* read or write from the e= xtra bytes that are indicated by this call.
=0A> >
=0A> >= ; jemalloc:
=0A> >       <para>The <= function>malloc_usable_size()</function> function
=0A> >=        returns the usable size of the allocation point= ed to by
=0A> >       <parameter>ptr<= ;/parameter>.  The return value may be larger than the size
=0A= > >       that was requested during allocation.&= nbsp; The
=0A> >       <function>malloc= _usable_size()</function> function is not a
=0A> >  &n= bsp;    mechanism for in-place <function>realloc()</fun= ction>; rather
=0A> >       it is provided= solely as a tool for introspection purposes.  Any
=0A> >&nb= sp;      discrepancy between the requested allocation size = and the size reported
=0A> >       by <fun= ction>malloc_usable_size()</function> should not be
=0A> &g= t;       depended on, since such behavior is entirely = implementation-dependent.
=0A>
=0A> These implementations are= buggy or at least mis-documented.  The
=0A> interface contract= is clearly that for that particular object, the extra
=0A> bytes in= the allocation are available for reading and writing.  It is
=0A&= gt; not guaranteed that the allocator will always provide the same number = of
=0A> extra bytes for the same requested size, but they must be th= ere for the
=0A> allocation being examined.  It's even in the n= ame of the function!
=0A
=0AI'm not sure I understand what you're sa= ying, but the core problem
=0Athat really can't be solved is potential = discrepancy between the
=0Amalloc implementation's idea of usable and t= he compiler's. For
=0Aexample:
=0A
=0A       = char *p =3D malloc(1);
=0A        if (malloc_usabl= e_size(p)>1) p[1] =3D 42;
=0A
=0Awill cause a compiler that's act= ively detecting UB to abort the
=0Aprogram when malloc_usable_size retu= rns a value larger than 1.
=0A
=0ARich
=0A
=0A<= /div>
=0A ------=_001_NextPart407476881701_=------