mailing list of musl libc
 help / color / mirror / code / Atom feed
From: 王志强 <00107082@163.com>
To: musl@lists.openwall.com
Cc: dalias@libc.org, "Quentin Rameau" <quinq@fifth.space>,
	 "Florian Weimer" <fweimer@redhat.com>
Subject: [musl] Re:[musl] Re:[musl] Re:Re: [musl] Re:Re: [musl] The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1)
Date: Thu, 22 Sep 2022 17:39:53 +0800 (CST)	[thread overview]
Message-ID: <1dac8fac.4b1c.18364916533.Coremail.00107082@163.com> (raw)
In-Reply-To: <adc8516.46d8.18364764f17.Coremail.00107082@163.com>

[-- Attachment #1: Type: text/plain, Size: 6389 bytes --]




Sorry, I found I made a wrong reading about profiling in last mail about malloc_usable_size,  I think malloc_usable_size is indeed has no impact.


The symbol "aligned_allocmalloc_usable_size" should be aligned_alloc<?>malloc_usable_size,  means a function address not symbolized, and the address belongs to neither aligned_alloc nor  malloc_usable_size, but is between   aligned_alloc and malloc_usable_size, not sure what it belongs to  (Most likely, it is some static function, somewhere)  Compared with old profiling report which did not call malloc_usable_size, there were also significant addresses around  that range, so...



Please ignore my last mail about malloc_usable_size.....sorry....













At 2022-09-22 17:10:18, "王志强" <00107082@163.com> wrote:

update:
I change the test code to include malloc_usable_size,  just to verify the impact of this function call

#define MAXF 4096
void* tobefree[MAXF];
int main() {
    long long i;
    int v, k, j;
    size_t s, c=0;
    char *p;
    for (i=0; i<100000000L; i++) {
        v = rand();   
        s = ((v%126)+1)*1024;
        p = (char*) malloc(s);
    for (j=0; j+1<s; j+=1024) p[j+1]=j;  ////<<--poke pages
    s = malloc_usable_size(p);   ////<<-----------------add here
        if (c>=MAXF) {
            k = v%c;
            free(tobefree[k]);
            tobefree[k]=tobefree[--c];
        }
        tobefree[c++]=p;
    }
    return 0;
}
# time ./m.alpine
real    7m 19.07s
user    4m 53.61s
sys    2m 25.22s
It took about 440 seconds to finish the whole 100 million iterations (on average 4.4microseconds per malloc&free,  while with glibc it took 96 seconds, average 1microsecond) the profiling report is as following
_init__libc_start_main(59.374% 202450/340976)
main(79.104% 160147/202450)
aligned_allocmalloc_usable_size(20.098% 40689/202450)
madvise(33.323% 113623/340976)
aligned_allocmalloc_usable_size(6.534% 22279/340976)


aligned_allocmalloc_usable_size got picked by profiler about (40689+22279)/340976 = 18.4%,  should this raise any concern? (Still,  madvise calls has major impact, 33.3%)



(The profiling report is different from last few because I add code to poke several memory addresses after malloc)



Here is the profiler report for glibc, running same code, malloc_usable_size got picked only 1% of total 75568 samples

_dl_catch_error??(62.487% 47220/75568)
__libc_start_main(100.000% 47220/47220)
main(80.064% 37806/47220)
cfree(17.274% 8157/47220)
malloc_usable_size(1.355% 640/47220)
pthread_attr_setschedparam?__libc_malloc(36.795% 27805/75568)


My profiler code is here,  https://github.com/zq-david-wang/linux-tools/tree/main/perf/profiler, it take a pid as parameter and would profile all the pids within same perf_event cgroup.




















At 2022-09-22 11:34:59, "王志强" <00107082@163.com> wrote:

Hi Rich,


Thanks for your time.


Totally agreed that in realworld application, it would take way more time to process that huge bulk of memory, compared with the  average 3 microsecond per malloc&free.





At 2022-09-22 01:58:17, "Rich Felker" <dalias@libc.org> wrote:

>
>
>> Your test case, with the completely random size distribution across

>> various large sizes, is likely a worst case. The mean size you're

>> allocating is 128k, which is the threshold for direct mmap/munmap of

>> each allocation, so at least half of the allocations you're making can

>> *never* be reused, and will always be immediately unmapped on free. It
>> might be interesting to change the scaling factor from 1k to 256 bytes

>> so that basically all of the allocation sizes are in the >> malloc-managed range.

>

>One observation if this change is made: it looks like at least 70% of

>the time is spent performing madvise(MADV_FREE), and that a large

>portion of the rest (just looking at strace) seems to be repeatedly

>mapping and freeing a 17-page (68k) block, probably because this size

>happens to be at the boundary of some threshold where bounce

>protection isn't happening. I think we should look at both of these in

>more detail, since they both suggest opportunities for large

>performance improvements at low cost.

>

I have made several profiling, the report indeed show that as the size decreased, performance went up significantly and madvise now take major portion of time, as you suggested,

also madviser's portion decrease as size decrease, when average size reach to 2K, madviser was only picked by profiler less than 2%:
1. average 64K(1K~128K) malloc/free
# time ./m.alpine
real    1m 50.12s
user    0m 39.80s
sys    1m 10.17s

madvise(61.945% 52926/85440)
__libc_start_main?(22.158% 18932/85440)
malloc_usable_size?(82.870% 15689/18932)
asm_exc_page_fault(2.766% 434/15689)
main(16.781% 3177/18932)
asm_exc_page_fault(2.487% 79/3177)
malloc_usable_size?(10.969% 9372/85440)
asm_exc_page_fault(6.519% 611/9372)
munmap(2.449% 2092/85440)
exit?(1.540% 1316/85440)
2. average 32K (1K~64K) malloc/free
# time ./m.alpine
real    1m 12.89s
user    0m 30.62s
sys    0m 41.91s

madvise(60.835% 34282/56352)
__libc_start_main?(27.410% 15446/56352)
malloc_usable_size?(78.558% 12134/15446)
main(20.996% 3243/15446)
malloc_usable_size?(9.354% 5271/56352)
exit?(1.888% 1064/56352)


3. average 8K (1K~16K)

# time ./m.alpine
real    0m 42.35s
user    0m 22.94s
sys    0m 19.27s

madvise(49.338% 16169/32772)
__libc_start_main?(36.592% 11992/32772)
malloc_usable_size?(79.244% 9503/11992)
main(20.480% 2456/11992)
malloc_usable_size?(10.921% 3579/32772)
exit?(2.591% 849/32772)
4. average 4K (1k~8K)
# time ./m.debian
real    0m32.477s
user    0m31.829s
sys    0m0.596s

__libc_start_main?(44.474% 9279/20864)
malloc_usable_size?(81.410% 7554/9279)
main(17.987% 1669/9279)
madvise(37.720% 7870/20864)
malloc_usable_size?(13.986% 2918/20864)
exit?(3.350% 699/20864)
5 average 2K(128B~4096B) (madviser only about 1.7%)

# time ./m.alpine
real    0m 13.02s
user    0m 12.68s
sys    0m 0.26s

__libc_start_main?(69.538% 6974/10029)
malloc_usable_size?(80.786% 5634/6974)
main(18.569% 1295/6974)
malloc_usable_size?(21.538% 2160/10029)
exit?(7.060% 708/10029)
madvise(1.715% 172/10029)
6. average 1K (128B~2048B)

# time ./m.alpine
real    0m 10.75s
user    0m 10.68s
sys    0m 0.01s

__libc_start_main?(72.495% 6012/8293)
malloc_usable_size?(76.630% 4607/6012)
main(22.904% 1377/6012)
malloc_usable_size?(18.823% 1561/8293)
exit?(8.610% 714/8293)




David

[-- Attachment #2: Type: text/html, Size: 14461 bytes --]

  reply	other threads:[~2022-09-22  9:41 UTC|newest]

Thread overview: 68+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-19  7:53 baiyang
2022-09-19 11:08 ` Szabolcs Nagy
2022-09-19 12:36   ` Florian Weimer
2022-09-19 13:46     ` Rich Felker
2022-09-19 13:53       ` James Y Knight
2022-09-19 17:40         ` baiyang
2022-09-19 18:14           ` Szabolcs Nagy
2022-09-19 18:40             ` baiyang
2022-09-19 19:07             ` Gabriel Ravier
2022-09-19 19:21               ` Rich Felker
2022-09-19 21:02                 ` Gabriel Ravier
2022-09-19 21:47                   ` Rich Felker
2022-09-19 22:31                     ` Gabriel Ravier
2022-09-19 22:46                       ` baiyang
2022-09-19 20:46             ` Nat!
2022-09-20  8:51               ` Szabolcs Nagy
2022-09-20  0:13           ` James Y Knight
2022-09-20  0:25             ` baiyang
2022-09-20  0:38               ` Rich Felker
2022-09-20  0:47                 ` baiyang
2022-09-20  1:00                   ` Rich Felker
2022-09-20  1:18                     ` baiyang
2022-09-20  2:15                       ` Rich Felker
2022-09-20  2:35                         ` baiyang
2022-09-20  3:28                           ` Rich Felker
2022-09-20  3:53                             ` baiyang
2022-09-20  5:41                               ` Rich Felker
2022-09-20  5:56                                 ` baiyang
2022-09-20 12:16                                   ` Rich Felker
2022-09-20 17:21                                     ` baiyang
2022-09-20  8:33       ` Florian Weimer
2022-09-20 13:54         ` Siddhesh Poyarekar
2022-09-20 16:59           ` James Y Knight
2022-09-20 17:34             ` Szabolcs Nagy
2022-09-20 19:53               ` James Y Knight
2022-09-24  8:55               ` Fangrui Song
2022-09-20 17:39             ` baiyang
2022-09-20 18:12               ` Quentin Rameau
2022-09-20 18:19                 ` Rich Felker
2022-09-20 18:26                   ` Alexander Monakov
2022-09-20 18:35                     ` baiyang
2022-09-20 20:33                       ` Gabriel Ravier
2022-09-20 20:45                         ` baiyang
2022-09-21  8:42                           ` NRK
2022-09-20 18:37                     ` Quentin Rameau
2022-09-21 10:15                   ` [musl] " 王志强
2022-09-21 16:11                     ` [musl] " 王志强
2022-09-21 17:15                     ` [musl] " Rich Felker
2022-09-21 17:58                       ` Rich Felker
2022-09-22  3:34                         ` [musl] " 王志强
2022-09-22  9:10                           ` [musl] " 王志强
2022-09-22  9:39                             ` 王志强 [this message]
2022-09-20 17:28           ` baiyang
2022-09-20 17:44             ` Siddhesh Poyarekar
2022-10-10 14:13           ` Florian Weimer
2022-09-19 13:43 ` Rich Felker
2022-09-19 17:32   ` baiyang
2022-09-19 18:15     ` Rich Felker
2022-09-19 18:44       ` baiyang
2022-09-19 19:18         ` Rich Felker
2022-09-19 19:45           ` baiyang
2022-09-19 20:07             ` Rich Felker
2022-09-19 20:17               ` baiyang
2022-09-19 20:28                 ` Rich Felker
2022-09-19 20:38                   ` baiyang
2022-09-19 22:02                 ` Quentin Rameau
2022-09-19 20:17             ` Joakim Sindholt
2022-09-19 20:33               ` baiyang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1dac8fac.4b1c.18364916533.Coremail.00107082@163.com \
    --to=00107082@163.com \
    --cc=dalias@libc.org \
    --cc=fweimer@redhat.com \
    --cc=musl@lists.openwall.com \
    --cc=quinq@fifth.space \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/musl/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).