From: baiyang <baiyang@gmail.com>
To: "James Y Knight" <jyknight@google.com>, musl <musl@lists.openwall.com>
Cc: "Florian Weimer" <fweimer@redhat.com>
Subject: Re: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1)
Date: Tue, 20 Sep 2022 01:40:48 +0800 [thread overview]
Message-ID: <2022092001404698842815@gmail.com> (raw)
In-Reply-To: <CAA2zVHpUTRjPy2C-LK6pPJeCZyv=vtmbqQGfdE-qUe7XbX086A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4264 bytes --]
Hi James,
I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ).
On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1499
I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault.
Thanks :-)
--
Best Regards
BaiYang
baiyang@gmail.com
http://i.baiy.cn
**** < END OF EMAIL > ****
From: James Y Knight
Date: 2022-09-19 21:53
To: musl
CC: Florian Weimer; baiyang
Subject: Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1)
Indeed. RedHat mentioned that problem in their recent post about _FORTIFY_SOURCE=3, here
https://developers.redhat.com/articles/2022/09/17/gccs-new-fortification-level
"""
_FORTIFY_SOURCE=3 revealed another pattern. Applications such as systemd used malloc_usable_size to determine available space in objects and then used the residual space. The glibc manual discourages this type of usage, dictating that malloc_usable_size is for diagnostic purposes only. But applications use the function as a hack to avoid reallocating buffers when there is space in the underlying malloc chunk. The implementation of malloc_usable_size needs to be fixed to return the allocated object size instead of the chunk size in non-diagnostic use. Alternatively, another solution is to deprecate the function. But that is a topic for discussion by the glibc community.
"""
On Mon, Sep 19, 2022 at 9:47 AM Rich Felker <dalias@libc.org> wrote:
On Mon, Sep 19, 2022 at 02:36:41PM +0200, Florian Weimer wrote:
> * Szabolcs Nagy:
>
> > unlike musl those implementations don't return exact size nor have the
> > same security and memory fragmentation guarantees, so bad comparision.
> >
> > tcmalloc:
> > // Returns the actual number N of bytes reserved by tcmalloc for the pointer
> > // p. This number may be equal to or greater than the number of bytes
> > // requested when p was allocated.
> > //
> > // This function is just useful for statistics collection. The client must
> > // *not* read or write from the extra bytes that are indicated by this call.
> >
> > jemalloc:
> > <para>The <function>malloc_usable_size()</function> function
> > returns the usable size of the allocation pointed to by
> > <parameter>ptr</parameter>. The return value may be larger than the size
> > that was requested during allocation. The
> > <function>malloc_usable_size()</function> function is not a
> > mechanism for in-place <function>realloc()</function>; rather
> > it is provided solely as a tool for introspection purposes. Any
> > discrepancy between the requested allocation size and the size reported
> > by <function>malloc_usable_size()</function> should not be
> > depended on, since such behavior is entirely implementation-dependent.
>
> These implementations are buggy or at least mis-documented. The
> interface contract is clearly that for that particular object, the extra
> bytes in the allocation are available for reading and writing. It is
> not guaranteed that the allocator will always provide the same number of
> extra bytes for the same requested size, but they must be there for the
> allocation being examined. It's even in the name of the function!
I'm not sure I understand what you're saying, but the core problem
that really can't be solved is potential discrepancy between the
malloc implementation's idea of usable and the compiler's. For
example:
char *p = malloc(1);
if (malloc_usable_size(p)>1) p[1] = 42;
will cause a compiler that's actively detecting UB to abort the
program when malloc_usable_size returns a value larger than 1.
Rich
[-- Attachment #2: Type: text/html, Size: 8271 bytes --]
next prev parent reply other threads:[~2022-09-19 17:41 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-19 7:53 baiyang
2022-09-19 11:08 ` Szabolcs Nagy
2022-09-19 12:36 ` Florian Weimer
2022-09-19 13:46 ` Rich Felker
2022-09-19 13:53 ` James Y Knight
2022-09-19 17:40 ` baiyang [this message]
2022-09-19 18:14 ` Szabolcs Nagy
2022-09-19 18:40 ` baiyang
2022-09-19 19:07 ` Gabriel Ravier
2022-09-19 19:21 ` Rich Felker
2022-09-19 21:02 ` Gabriel Ravier
2022-09-19 21:47 ` Rich Felker
2022-09-19 22:31 ` Gabriel Ravier
2022-09-19 22:46 ` baiyang
2022-09-19 20:46 ` Nat!
2022-09-20 8:51 ` Szabolcs Nagy
2022-09-20 0:13 ` James Y Knight
2022-09-20 0:25 ` baiyang
2022-09-20 0:38 ` Rich Felker
2022-09-20 0:47 ` baiyang
2022-09-20 1:00 ` Rich Felker
2022-09-20 1:18 ` baiyang
2022-09-20 2:15 ` Rich Felker
2022-09-20 2:35 ` baiyang
2022-09-20 3:28 ` Rich Felker
2022-09-20 3:53 ` baiyang
2022-09-20 5:41 ` Rich Felker
2022-09-20 5:56 ` baiyang
2022-09-20 12:16 ` Rich Felker
2022-09-20 17:21 ` baiyang
2022-09-20 8:33 ` Florian Weimer
2022-09-20 13:54 ` Siddhesh Poyarekar
2022-09-20 16:59 ` James Y Knight
2022-09-20 17:34 ` Szabolcs Nagy
2022-09-20 19:53 ` James Y Knight
2022-09-24 8:55 ` Fangrui Song
2022-09-20 17:39 ` baiyang
2022-09-20 18:12 ` Quentin Rameau
2022-09-20 18:19 ` Rich Felker
2022-09-20 18:26 ` Alexander Monakov
2022-09-20 18:35 ` baiyang
2022-09-20 20:33 ` Gabriel Ravier
2022-09-20 20:45 ` baiyang
2022-09-21 8:42 ` NRK
2022-09-20 18:37 ` Quentin Rameau
2022-09-21 10:15 ` [musl] " 王志强
2022-09-21 16:11 ` [musl] " 王志强
2022-09-21 17:15 ` [musl] " Rich Felker
2022-09-21 17:58 ` Rich Felker
2022-09-22 3:34 ` [musl] " 王志强
2022-09-22 9:10 ` [musl] " 王志强
2022-09-22 9:39 ` [musl] " 王志强
2022-09-20 17:28 ` baiyang
2022-09-20 17:44 ` Siddhesh Poyarekar
2022-10-10 14:13 ` Florian Weimer
2022-09-19 13:43 ` Rich Felker
2022-09-19 17:32 ` baiyang
2022-09-19 18:15 ` Rich Felker
2022-09-19 18:44 ` baiyang
2022-09-19 19:18 ` Rich Felker
2022-09-19 19:45 ` baiyang
2022-09-19 20:07 ` Rich Felker
2022-09-19 20:17 ` baiyang
2022-09-19 20:28 ` Rich Felker
2022-09-19 20:38 ` baiyang
2022-09-19 22:02 ` Quentin Rameau
2022-09-19 20:17 ` Joakim Sindholt
2022-09-19 20:33 ` baiyang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2022092001404698842815@gmail.com \
--to=baiyang@gmail.com \
--cc=fweimer@redhat.com \
--cc=jyknight@google.com \
--cc=musl@lists.openwall.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/musl/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).