From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 29312 invoked by alias); 11 Oct 2015 16:49:20 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: X-Seq: 36835 Received: (qmail 12397 invoked from network); 11 Oct 2015 16:49:18 -0000 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, T_DKIM_INVALID autolearn=ham autolearn_force=no version=3.4.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=AW2Q93jrWVeLWC0/ZWYDpwava+g6Any7uandE8xacqU=; b=pHIm1P2z7r/wM0kAC+PixBexn79h6ky496TYOiXiwAb++LhjzCrJDAafD5/MWWmgn7 3YZ8Covb8wk6GHoJa/Lg0D2KsZEpKtBA3Mjgz/Jr9hIV6Ru3u8GPsRajpR1UqPQ8t1zD P8F+ovPq4cx/BdAmPYwLwpTCn1WwJhmy22i7shtjFstPPMo0+T8deUiJWcUi0sQiKlYZ 3tD1HTNP4r3DhD6BUZ8Y5yHDWaUhVzygaVlfZw2xovqzUWtaR+R8rYUQGXOBQ+c/nZMA uDZS7dnRi+xcIyiljmBCvyYflukpmnK3AjlU1x3pxlkxxtH9/BhIPyUpmZL8AKvyb2lw RBHw== X-Received: by 10.112.167.101 with SMTP id zn5mr2580736lbb.18.1444582156002; Sun, 11 Oct 2015 09:49:16 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <151011091757.ZM27755@torch.brasslantern.com> References: <151010105849.ZM10144@torch.brasslantern.com> <151010170623.ZM16166@torch.brasslantern.com> <151010232045.ZM12931@torch.brasslantern.com> <151011091757.ZM27755@torch.brasslantern.com> From: Sebastian Gniazdowski Date: Sun, 11 Oct 2015 18:48:56 +0200 Message-ID: Subject: Re: Slowdown around 5.0.5-dev-0 To: Bart Schaefer Cc: zsh-workers@zsh.org Content-Type: multipart/mixed; boundary=001a11c25fce42a2580521d6fe65 --001a11c25fce42a2580521d6fe65 Content-Type: text/plain; charset=UTF-8 It's now much faster, zprof time 11851 vs 3433. That's still strongly lagging behavior. One other trait is that it slows down after a while. Best regards, Sebastian Gniazdowski On 11 October 2015 at 18:17, Bart Schaefer wrote: > On Oct 11, 10:39am, Sebastian Gniazdowski wrote: > } > } Zprof says there is no change: > > Well, that's not entirely surprising. It probably just means that there > is some space in a lot of arenas, so fpop doesn't move far from the head > of the list. Or it could mean that the real problem is popheap() rather > than freeheap(). > > } BTW, the patch didn't apply to head or to 23f98c3, > > Curious. "git diff master origin/master" shows no diffs for my local > repository, and the patch was just a straight "git diff". > > There were additional changes committed after 23f98c3 so there's no > reason to have expected it would apply to that. > > In any case here's another stab at it. Should apply to 83a1757. This > should help if there's a lot of freeheap() happening, but not so much > if there's a lot of push/pop (in fact it may be worse in that case). > > If it's push/pop that's the issue, the next step would be to try to > identify the crucial pushheap() and replace it with NEWHEAPS() if the > scope permits. > > > diff --git a/Src/mem.c b/Src/mem.c > index b9569ea..d49f685 100644 > --- a/Src/mem.c > +++ b/Src/mem.c > @@ -294,7 +294,7 @@ pushheap(void) > #endif > > for (h = heaps; h; h = h->next) { > - DPUTS(!h->used, "BUG: empty heap"); > + DPUTS(!h->used && h->next, "BUG: empty heap"); > hs = (Heapstack) zalloc(sizeof(*hs)); > hs->next = h->sp; > h->sp = hs; > @@ -334,17 +334,15 @@ freeheap(void) > * > * Whenever fheap is NULL here, the loop below sweeps back over the > * entire heap list again, resetting the free space in every arena to > - * the amount stashed by pushheap() and finding the first arena with > + * the amount stashed by pushheap() and finding the arena with the most > * free space to optimize zhalloc()'s next search. When there's a lot > * of stuff already on the heap, this is an enormous amount of work, > * and performance goes to hell. > * > - * However, if the arena to which fheap points is unused, we want to > - * free it, so we have no choice but to do the sweep for a new fheap. > - */ > - if (fheap && !fheap->sp) > - fheap = NULL; /* We used to do this unconditionally */ > - /* > + * Therefore, we defer freeing the most recently allocated arena until > + * we reach popheap(). This may fail to reclaim some space in earlier > + * arenas. > + * > * In other cases, either fheap is already correct, or it has never > * been set and this loop will do it, or it'll be reset from scratch > * on the next popheap(). So all that's needed here is to pick up > @@ -361,7 +359,11 @@ freeheap(void) > memset(arena(h) + h->sp->used, 0xff, h->used - h->sp->used); > #endif > h->used = h->sp->used; > - if (!fheap && h->used < ARENA_SIZEOF(h)) > + if (!fheap) { > + if (h->used < ARENA_SIZEOF(h)) > + fheap = h; > + } else if (ARENA_SIZEOF(h) - h->used > > + ARENA_SIZEOF(fheap) - fheap->used) > fheap = h; > hl = h; > #ifdef ZSH_HEAP_DEBUG > @@ -384,6 +386,26 @@ freeheap(void) > VALGRIND_MEMPOOL_TRIM((char *)h, (char *)arena(h), h->used); > #endif > } else { > + if (h->next) { > + /* We want to cut this out of the arena list if we can */ > + if (h == heaps) > + hl = heaps = h->next; > + else if (hl && hl->next == h) > + hl->next = h->next; > + else { > + DPUTS(hl, "hl->next != h when freeing"); > + hl = h; > + continue; > + } > + h->next = NULL; > + } else { > + /* Leave an empty arena at the end until popped */ > + h->used = 0; > + fheap = hl = h; > + break; > + } > + if (fheap == h) > + fheap = NULL; > #ifdef USE_MMAP > munmap((void *) h, h->size); > #else > @@ -441,12 +463,29 @@ popheap(void) > #ifdef ZSH_VALGRIND > VALGRIND_MEMPOOL_TRIM((char *)h, (char *)arena(h), h->used); > #endif > - if (!fheap && h->used < ARENA_SIZEOF(h)) > + if (!fheap) { > + if (h->used < ARENA_SIZEOF(h)) > + fheap = h; > + } else if (ARENA_SIZEOF(h) - h->used > > + ARENA_SIZEOF(fheap) - fheap->used) > fheap = h; > zfree(hs, sizeof(*hs)); > > hl = h; > } else { > + if (h->next) { > + /* We want to cut this out of the arena list if we can */ > + if (h == heaps) > + hl = heaps = h->next; > + else if (hl && hl->next == h) > + hl->next = h->next; > + else { > + DPUTS(hl, "hl->next != h when popping"); > + hl = h; > + continue; > + } > + h->next = NULL; > + } > #ifdef USE_MMAP > munmap((void *) h, h->size); > #else > @@ -524,7 +563,7 @@ zheapptr(void *p) > mod_export void * > zhalloc(size_t size) > { > - Heap h; > + Heap h, hp = NULL; > size_t n; > #ifdef ZSH_VALGRIND > size_t req_size = size; > @@ -546,6 +585,7 @@ zhalloc(size_t size) > for (h = ((fheap && ARENA_SIZEOF(fheap) >= (size + fheap->used)) > ? fheap : heaps); > h; h = h->next) { > + hp = h; > if (ARENA_SIZEOF(h) >= (n = size + h->used)) { > void *ret; > > @@ -566,7 +606,6 @@ zhalloc(size_t size) > } > } > { > - Heap hp; > /* not found, allocate new heap */ > #if defined(ZSH_MEM) && !defined(USE_MMAP) > static int called = 0; > @@ -575,7 +614,6 @@ zhalloc(size_t size) > #endif > > n = HEAP_ARENA_SIZE > size ? HEAPSIZE : size + sizeof(*h); > - for (hp = NULL, h = heaps; h; hp = h, h = h->next); > > #ifdef USE_MMAP > h = mmap_heap_alloc(&n); > @@ -607,6 +645,7 @@ zhalloc(size_t size) > VALGRIND_MEMPOOL_ALLOC((char *)h, (char *)arena(h), req_size); > #endif > > + DPUTS(hp && hp->next, "failed to find end of chain in zhalloc"); > if (hp) > hp->next = h; > else --001a11c25fce42a2580521d6fe65 Content-Type: text/plain; charset=US-ASCII; name="result2.txt" Content-Disposition: attachment; filename="result2.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ifmqtehh0 cGF0Y2hlZDoKbnVtICBjYWxscyAgICAgICAgICAgICAgICB0aW1lICAgICAgICAgICAgICAgICAg ICAgICBzZWxmICAgICAgICAgICAgbmFtZQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQogMSkg ICAgMSAgICAgICAgMzQzMyw5OCAgMzQzMyw5OCAgIDk5LDYyJSAgIDI0MzksMzggIDI0MzksMzgg ICA3MCw3NyUgIG4tbGlzdAogMikgIDE2MSAgICAgICAgIDYzOCwyMSAgICAgMyw5NiAgIDE4LDUy JSAgICA2MzgsMjEgICAgIDMsOTYgICAxOCw1MiUgIF9ubGlzdF9wcmludF93aXRoX2Fuc2kKIDMp ICAgIDcgICAgICAgICA5MjgsNzIgICAxMzIsNjcgICAyNiw5NCUgICAgMjkwLDUxICAgIDQxLDUw ICAgIDgsNDMlICBuLWxpc3QtZHJhdwogNCkgICAgNyAgICAgICAgICA1MCwyOSAgICAgNywxOCAg ICAxLDQ2JSAgICAgNTAsMjAgICAgIDcsMTcgICAgMSw0NiUgIG4tbGlzdC1pbnB1dAogNSkgICAg MiAgICAgICAgICAxMSwzOSAgICAgNSw2OSAgICAwLDMzJSAgICAgMTEsMzkgICAgIDUsNjkgICAg MCwzMyUgIF9ubGlzdF9jdXJzb3JfdmlzaWJpbGl0eQogNikgICAgNyAgICAgICAgICAxMCw3OSAg ICAgMSw1NCAgICAwLDMxJSAgICAgMTAsNjIgICAgIDEsNTIgICAgMCwzMSUgIF9ubGlzdF9zZXR1 cF91c2VyX3ZhcnMKIDcpICAgIDEgICAgICAgICAgMTIsOTYgICAgMTIsOTYgICAgMCwzOCUgICAg ICA2LDA5ICAgICA2LDA5ICAgIDAsMTglICBfbmxpc3RfZXhpdAogOCkgICAgNyAgICAgICAgICAg MCwyOSAgICAgMCwwNCAgICAwLDAxJSAgICAgIDAsMjkgICAgIDAsMDQgICAgMCwwMSUgIF9ubGlz dF9zdGF0dXNfbXNnCiggbi1wYW5lbGl6ZSA8IH4vbHNvZmxzb2Y7ICkgIDUsMThzIHVzZXIgMCwx OHMgc3lzdGVtIDg5JSBjcHUgNSw5NTkgdG90YWwKCmNsZWFuOgpudW0gIGNhbGxzICAgICAgICAg ICAgICAgIHRpbWUgICAgICAgICAgICAgICAgICAgICAgIHNlbGYgICAgICAgICAgICBuYW1lCi0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiAxKSAgMTYxICAgICAgICA1MjY2LDQ2ICAgIDMyLDcx ICAgNDQsMzUlICAgNTI2Niw0NiAgICAzMiw3MSAgIDQ0LDM1JSAgX25saXN0X3ByaW50X3dpdGhf YW5zaQogMikgICAgMSAgICAgICAxMTg1MSwxOSAxMTg1MSwxOSAgIDk5LDc5JSAgIDM3MDYsMTIg IDM3MDYsMTIgICAzMSwyMSUgIG4tbGlzdAogMykgICAgNyAgICAgICAgNzc1NCw4NCAgMTEwNyw4 MyAgIDY1LDMwJSAgIDI0ODgsMzggICAzNTUsNDggICAyMCw5NSUgIG4tbGlzdC1kcmF3CiA0KSAg ICA3ICAgICAgICAgMzE2LDQwICAgIDQ1LDIwICAgIDIsNjYlICAgIDMxMCw5NiAgICA0NCw0MiAg ICAyLDYyJSAgbi1saXN0LWlucHV0CiA1KSAgICA3ICAgICAgICAgIDM4LDM1ICAgICA1LDQ4ICAg IDAsMzIlICAgICAzMCw1OSAgICAgNCwzNyAgICAwLDI2JSAgX25saXN0X3NldHVwX3VzZXJfdmFy cwogNikgICAgNyAgICAgICAgICAyNiw2MCAgICAgMyw4MCAgICAwLDIyJSAgICAgMjYsNjAgICAg IDMsODAgICAgMCwyMiUgIF9ubGlzdF9zdGF0dXNfbXNnCiA3KSAgICAyICAgICAgICAgIDIzLDg5 ICAgIDExLDk1ICAgIDAsMjAlICAgICAyMyw4OSAgICAxMSw5NSAgICAwLDIwJSAgX25saXN0X2N1 cnNvcl92aXNpYmlsaXR5CiA4KSAgICAxICAgICAgICAgIDI0LDM1ICAgIDI0LDM1ICAgIDAsMjEl ICAgICAgOSwzNCAgICAgOSwzNCAgICAwLDA4JSAgX25saXN0X2V4aXQKKCBuLXBhbmVsaXplIDwg fi9sc29mbHNvZjsgKSAgMTMsNzdzIHVzZXIgMCwyMXMgc3lzdGVtIDk1JSBjcHUgMTQsNjYwIHRv dGFsCgo= --001a11c25fce42a2580521d6fe65--