From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 31632 invoked from network); 19 Sep 2022 13:54:26 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 19 Sep 2022 13:54:26 -0000 Received: (qmail 19662 invoked by uid 550); 19 Sep 2022 13:54:24 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 19639 invoked from network); 19 Sep 2022 13:54:23 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=K2rENCkGXqABLNEfaQnnLUP1mwI6rKPfpHugDPBp258=; b=a/6P1LGUxCHCYFtwD33Qh8EDq11cbldxPhN6ReLHWq6eQJYZyP8iVBunwS5KbE5qYQ BuawDR4wWVPKk6bvt02iYmwjOAbb/7a1q2oLUVM4E9ZdG+dpz/zH7ra3VengXiiepqQ4 OOM06XTIhs5CjhLlh3Y0732DkjSBDbGbXzTLV3C2z2jznDyiq0vDFXJjnGbSLj+NCzHC HZKy7E9YerjMuL/7qOeS9ISgJPlSLeSBf4Xa2rfNm6XQdA3NHv3bQrr4UXgri+jRWkF4 Q0kSJUuKPQRpRUX82TnZjvkTcBBvzLB8PSF0kIwwIF5MVOgwoiCzyaB+90XIQDFt94Oc kjlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=K2rENCkGXqABLNEfaQnnLUP1mwI6rKPfpHugDPBp258=; b=mKiGo1U2eAuwhWn7GXi/e0OgAkH0A+lxlEtOg37/D5PbKfM9tLhyQSDrfDzN/CVV69 RdCPpA4DlN5w3r57gdrEWGZIGgw82prgIjpy3iWGthyqr/gau+iO9iBp3pD5DBLQx1Ho Bfv5RsgF7NQ3zXCNBSBkaOgsUxyly2Ey2uDyEqGOcPgTqXMlr1jBCT58Y1TEQ4P4rn7k 0kgKXTQrAPZK5n30O3clJOstf2fY2Oki+xSCJ42LgFk3+5fOSQrMAoEOGep0jivF+L25 dZxA54cI9w4YzVLbNO3IFM1urpol07l4tQBdzalVKIghJmGDzXAEXofi6gpl7W4xYdyP kv6Q== X-Gm-Message-State: ACrzQf1r9iE+6e0Q4AnuE5J3f0lszoU3F5PlAA4u0AuM93E038qwSqZU jKbstwzCpIFRgNkzIiULS2kUQ+Mf5dnHdmNx/ql0faxN3VwDGQ== X-Google-Smtp-Source: AMsMyM6QORbQqHFg9/NTDed9Q/pesUqobutCAP5G6/NCHWF6WUDhDV8o3yc2tg22E+9C1OKHiBT8nmy9siCx7tATN8s= X-Received: by 2002:a25:7485:0:b0:6b0:c0d:2f5b with SMTP id p127-20020a257485000000b006b00c0d2f5bmr13479067ybc.587.1663595651285; Mon, 19 Sep 2022 06:54:11 -0700 (PDT) MIME-Version: 1.0 References: <2022091915532777412615@gmail.com> <20220919110829.GA2158779@port70.net> <874jx3h76u.fsf@oldenburg.str.redhat.com> <20220919134659.GO9709@brightrain.aerifal.cx> In-Reply-To: <20220919134659.GO9709@brightrain.aerifal.cx> From: James Y Knight Date: Mon, 19 Sep 2022 09:53:45 -0400 Message-ID: To: musl@lists.openwall.com Cc: Florian Weimer , baiyang Content-Type: multipart/alternative; boundary="000000000000d8d5b805e9080b3e" Subject: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) --000000000000d8d5b805e9080b3e Content-Type: text/plain; charset="UTF-8" Indeed. RedHat mentioned that problem in their recent post about _FORTIFY_SOURCE=3, here https://developers.redhat.com/articles/2022/09/17/gccs-new-fortification-level """ _FORTIFY_SOURCE=3 revealed another pattern. Applications such as systemd used malloc_usable_size to determine available space in objects and then used the residual space. The glibc manual discourages this type of usage, dictating that malloc_usable_size is for diagnostic purposes only. But applications use the function as a hack to avoid reallocating buffers when there is space in the underlying malloc chunk. The implementation of malloc_usable_size needs to be fixed to return the allocated object size instead of the chunk size in non-diagnostic use. Alternatively, another solution is to deprecate the function. But that is a topic for discussion by the glibc community. """ On Mon, Sep 19, 2022 at 9:47 AM Rich Felker wrote: > On Mon, Sep 19, 2022 at 02:36:41PM +0200, Florian Weimer wrote: > > * Szabolcs Nagy: > > > > > unlike musl those implementations don't return exact size nor have the > > > same security and memory fragmentation guarantees, so bad comparision. > > > > > > tcmalloc: > > > // Returns the actual number N of bytes reserved by tcmalloc for the > pointer > > > // p. This number may be equal to or greater than the number of > bytes > > > // requested when p was allocated. > > > // > > > // This function is just useful for statistics collection. The > client must > > > // *not* read or write from the extra bytes that are indicated by > this call. > > > > > > jemalloc: > > > The malloc_usable_size() function > > > returns the usable size of the allocation pointed to by > > > ptr. The return value may be larger than > the size > > > that was requested during allocation. The > > > malloc_usable_size() function is not a > > > mechanism for in-place realloc(); rather > > > it is provided solely as a tool for introspection purposes. Any > > > discrepancy between the requested allocation size and the size > reported > > > by malloc_usable_size() should not be > > > depended on, since such behavior is entirely > implementation-dependent. > > > > These implementations are buggy or at least mis-documented. The > > interface contract is clearly that for that particular object, the extra > > bytes in the allocation are available for reading and writing. It is > > not guaranteed that the allocator will always provide the same number of > > extra bytes for the same requested size, but they must be there for the > > allocation being examined. It's even in the name of the function! > > I'm not sure I understand what you're saying, but the core problem > that really can't be solved is potential discrepancy between the > malloc implementation's idea of usable and the compiler's. For > example: > > char *p = malloc(1); > if (malloc_usable_size(p)>1) p[1] = 42; > > will cause a compiler that's actively detecting UB to abort the > program when malloc_usable_size returns a value larger than 1. > > Rich > --000000000000d8d5b805e9080b3e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Indeed. RedHat mentioned that problem in their recent post= about _FORTIFY_SOURCE=3D3, here
=
"""
_FORTIFY_SOURCE=3D3 revealed an= other pattern. Applications such as systemd used malloc_usable_size to dete= rmine available space in objects and then used the residual space. The glib= c manual discourages this type of usage, dictating that malloc_usable_size = is for diagnostic purposes only. But applications use the function as a hac= k to avoid reallocating buffers when there is space in the underlying mallo= c chunk. The implementation of malloc_usable_size needs to be fixed to retu= rn the allocated object size instead of the chunk size in non-diagnostic us= e. Alternatively, another solution is to deprecate the function. But that i= s a topic for discussion by the glibc community.
"""

On Mon, Sep 19, 2022 at 9:47 AM Rich Felker <dalias@libc.org> wrote:
On Mon, Sep 19, 2022 at 02:36:41PM +0200, Florian W= eimer wrote:
> * Szabolcs Nagy:
>
> > unlike musl those implementations don't return exact size nor= have the
> > same security and memory fragmentation guarantees, so bad compari= sion.
> >
> > tcmalloc:
> >=C2=A0 =C2=A0// Returns the actual number N of bytes reserved by t= cmalloc for the pointer
> >=C2=A0 =C2=A0// p.=C2=A0 This number may be equal to or greater th= an the number of bytes
> >=C2=A0 =C2=A0// requested when p was allocated.
> >=C2=A0 =C2=A0//
> >=C2=A0 =C2=A0// This function is just useful for statistics collec= tion.=C2=A0 The client must
> >=C2=A0 =C2=A0// *not* read or write from the extra bytes that are = indicated by this call.
> >
> > jemalloc:
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<para>The <function>malloc_= usable_size()</function> function
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0returns the usable size of the allocati= on pointed to by
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<parameter>ptr</parameter>.= =C2=A0 The return value may be larger than the size
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0that was requested during allocation.= =C2=A0 The
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<function>malloc_usable_size()<= ;/function> function is not a
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0mechanism for in-place <function>= realloc()</function>; rather
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0it is provided solely as a tool for int= rospection purposes.=C2=A0 Any
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0discrepancy between the requested alloc= ation size and the size reported
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0by <function>malloc_usable_size()= </function> should not be
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0depended on, since such behavior is ent= irely implementation-dependent.
>
> These implementations are buggy or at least mis-documented.=C2=A0 The<= br> > interface contract is clearly that for that particular object, the ext= ra
> bytes in the allocation are available for reading and writing.=C2=A0 I= t is
> not guaranteed that the allocator will always provide the same number = of
> extra bytes for the same requested size, but they must be there for th= e
> allocation being examined.=C2=A0 It's even in the name of the func= tion!

I'm not sure I understand what you're saying, but the core problem<= br> that really can't be solved is potential discrepancy between the
malloc implementation's idea of usable and the compiler's. For
example:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 char *p =3D malloc(1);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (malloc_usable_size(p)>1) p[1] =3D 42;
will cause a compiler that's actively detecting UB to abort the
program when malloc_usable_size returns a value larger than 1.

Rich
--000000000000d8d5b805e9080b3e--