Re: Improving zcompdump (Re: A patch with hashtable optimization, which doesn't work)

Re: Improving zcompdump (Re: A patch with hashtable optimization, which doesn't work)

[Sebastian Gniazdowski]

On 20 maja 2017 at 19:08:09, Bart Schaefer (schaefer@brasslantern.com) wrote:
> } Tried to optimize mkautofn, to speed up sourcing zcompdump.
>
> How much does zcompdump actually help? Have you compared startup with
> and without it?
>
> There's a bunch of stuff in .zcompdump. Have you investigated whether
> certain parts of it are slower than others?
>
> One lesson learned with Completion/Base/Utility/_store_cache is that
> parsing array assignments is expensive.

I've wrapped sourcing zcompdump in compinit this way:

      zmodload zsh/zprof
      () {
          builtin . "$_comp_dumpfile"
      }
      zprof | head -n 14

Then I tried with a) normal .zcompdump, and b) with modification – with _comps=( ), i.e. empty. Results seem to confirm what you said:

num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    1          58,93    58,93  100,00%     58,93    58,93  100,00%  (anon)

vs.

 1)    1          12,81    12,81  100,00%     12,81    12,81  100,00%  (anon)

There's 58-12=46 ms to win, a significant value when thinking in terms of instant Zsh startup, which today is rather a melody of the past, with zsh-syntax-highlighting and zsh-autosuggestions overloading all $widgets entries during startup, in a loop.

I would go in direction of implementing new trivial parser that would read key-value pairs and put them to hash. It might even predict required size for 1562 _comps elements in the hash (it's x4 AFAIR, saw in addhashnode2), so that no expandhashtable() will be called. There would be .zcompdump_comps file with the pairs. Nothing will break, old .zcompdump will work.

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Improving zcompdump (Re: A patch with hashtable optimization, which doesn't work)

[Bart Schaefer]

On Jun 3,  4:11pm, Sebastian Gniazdowski wrote:
} Subject: Re: Improving zcompdump (Re: A patch with hashtable optimization,
}
} On 20 maja 2017 at 19:08:09, Bart Schaefer (schaefer@brasslantern.com) wrote:
} > How much does zcompdump actually help? Have you compared startup with
} > and without it?

Haven't seen that yet (nor tried it myself, tho).

} > One lesson learned with Completion/Base/Utility/_store_cache is that
} > parsing array assignments is expensive.
} 
} Results seem to confirm what you said

I also tried something similar ... using the just the default set of
completions, more wall-clock time is spent restoring the autoloads
than loading the _comps array, at least on my system.

Re: Improving zcompdump (Re: A patch with hashtable optimization, which doesn't work)

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]



On 4 czerwca 2017 at 02:54:57, Bart Schaefer (schaefer@brasslantern.com) wrote:
> On Jun 3, 4:11pm, Sebastian Gniazdowski wrote:
> } Subject: Re: Improving zcompdump (Re: A patch with hashtable optimization,
> }
> } On 20 maja 2017 at 19:08:09, Bart Schaefer (schaefer@brasslantern.com) wrote:
> } > How much does zcompdump actually help? Have you compared startup with
> } > and without it?
> 
> Haven't seen that yet (nor tried it myself, tho).

It wasn't me in the "How does .."

> } > One lesson learned with Completion/Base/Utility/_store_cache is that
> } > parsing array assignments is expensive.
> }
> } Results seem to confirm what you said
> 
> I also tried something similar ... using the just the default set of
> completions, more wall-clock time is spent restoring the autoloads
> than loading the _comps array, at least on my system.

Could you try patch with single-allocation in mkautofn()? Chances are low, but who knows. The patch spans also to other non-heap Eprog allocation places.

--
Sebastian Gniazdowski
psprint /at/ zdharma.org

[-- Attachment #2: ef_real_single.diff.txt --]
[-- Type: text/plain, Size: 4194 bytes --]

diff --git a/Src/builtin.c b/Src/builtin.c
index 063644e..680b19a 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -3483,15 +3483,15 @@ mkautofn(Shfunc shf)
 {
     Eprog p;
 
-    p = (Eprog) zalloc(sizeof(*p));
+    p = (Eprog) zalloc(sizeof(*p) + 5 * sizeof(wordcode));
     p->len = 5 * sizeof(wordcode);
-    p->prog = (Wordcode) zalloc(p->len);
+    p->prog = (Wordcode) (((char*)p)+sizeof(*p));
     p->strs = NULL;
     p->shf = shf;
     p->npats = 0;
     p->nref = 1; /* allocated from permanent storage */
     p->pats = (Patprog *) p->prog;
-    p->flags = EF_REAL;
+    p->flags = EF_REAL_SINGLE; /* no allocation for p->prog */
     p->dump = NULL;
 
     p->prog[0] = WCB_LIST((Z_SYNC | Z_END), 0);
diff --git a/Src/exec.c b/Src/exec.c
index debb0ae..088080b 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -4973,7 +4973,13 @@ execfuncdef(Estate state, Eprog redir_prog)
 	    prog = (Eprog) zhalloc(sizeof(*prog));
 	    prog->nref = -1; /* on the heap */
 	} else {
-	    prog = (Eprog) zalloc(sizeof(*prog));
+            if (state->prog->dump || !names) {
+                prog = (Eprog) zalloc(sizeof(*prog));
+            } else {
+                /* The EF_REAL path below */
+                prog = (Eprog) zalloc(sizeof(*prog) + len);
+                prog->pats = pp = (Patprog *) (((char*)prog)+sizeof(*prog));
+            }
 	    prog->nref = 1; /* allocated from permanent storage */
 	}
 	prog->npats = npats;
@@ -4992,8 +4998,11 @@ execfuncdef(Estate state, Eprog redir_prog)
 	    prog->prog = state->pc;
 	    prog->strs = state->strs + sbeg;
 	} else {
-	    prog->flags = EF_REAL;
-	    prog->pats = pp = (Patprog *) zalloc(len);
+            /* The EF_REAL path */
+	    prog->flags = EF_REAL_SINGLE;
+	    // prog->pats = pp = (Patprog *) zalloc(len);
+            // -->
+            // prog->pats = pp = (((char*)prog)+sizeof(*prog));
 	    prog->prog = (Wordcode) (prog->pats + npats);
 	    prog->strs = (char *) (prog->prog + nprg);
 	    prog->dump = NULL;
diff --git a/Src/parse.c b/Src/parse.c
index 8769baa..1e1c528 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -504,18 +504,28 @@ bld_eprog(int heap)
 
     ecadd(WCB_END());
 
-    ret = heap ? (Eprog) zhalloc(sizeof(*ret)) : (Eprog) zalloc(sizeof(*ret));
-    ret->len = ((ecnpats * sizeof(Patprog)) +
-		(ecused * sizeof(wordcode)) +
-		ecsoffs);
-    ret->npats = ecnpats;
-    ret->nref = heap ? -1 : 1;
-    ret->pats = heap ? (Patprog *) zhalloc(ret->len) :
-	(Patprog *) zshcalloc(ret->len);
+    if ( heap ) {
+        ret = (Eprog) zhalloc(sizeof(*ret));
+        ret->len = ((ecnpats * sizeof(Patprog)) +
+                (ecused * sizeof(wordcode)) +
+                ecsoffs);
+        ret->npats = ecnpats;
+        ret->nref = -1;
+        ret->pats = (Patprog *) zhalloc(ret->len);
+    } else {
+        int len = ((ecnpats * sizeof(Patprog)) +
+                (ecused * sizeof(wordcode)) +
+                ecsoffs);
+        ret = (Eprog) zalloc(sizeof(*ret) + len);
+        ret->len = len;
+        ret->npats = ecnpats;
+        ret->nref = 1;
+        ret->pats = (Patprog *) (((char*)ret)+sizeof(*ret));
+    }
     ret->prog = (Wordcode) (ret->pats + ecnpats);
     ret->strs = (char *) (ret->prog + ecused);
     ret->shf = NULL;
-    ret->flags = heap ? EF_HEAP : EF_REAL;
+    ret->flags = heap ? EF_HEAP : EF_REAL_SINGLE;
     ret->dump = NULL;
     for (l = 0; l < ecnpats; l++)
 	ret->pats[l] = dummy_patprog1;
@@ -2709,9 +2719,14 @@ freeeprog(Eprog p)
 	    if (p->dump) {
 		decrdumpcount(p->dump);
 		zfree(p->pats, p->npats * sizeof(Patprog));
-	    } else
-		zfree(p->pats, p->len);
-	    zfree(p, sizeof(*p));
+	    } else {
+                if ( (p->flags & EF_REAL_SINGLE) == 0 )
+                    zfree(p->pats, p->len);
+            }
+            if ( (p->flags & EF_REAL_SINGLE) )
+                zfree(p, sizeof(*p) + p->len);
+            else
+                zfree(p, sizeof(*p));
 	}
     }
 }
diff --git a/Src/zsh.h b/Src/zsh.h
index 22f73f8..ab70cb7 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -797,6 +797,7 @@ struct eprog {
 #define EF_HEAP 2
 #define EF_MAP  4
 #define EF_RUN  8
+#define EF_REAL_SINGLE 16
 
 typedef struct estate *Estate;
 

Improving zcompdump (Re: A patch with hashtable optimization, which doesn't work)

[Bart Schaefer]

On May 20,  7:06am, Sebastian Gniazdowski wrote:
}
} There are no performance gains from keeping the collision list sorted.

Yes, I was going to reply to your earlier message to remark that I would
only expect sorting the collision list to *slow down* insertion ops, not
speed up lookups.

} Tried to optimize mkautofn, to speed up sourcing zcompdump.

How much does zcompdump actually help?  Have you compared startup with
and without it?

There's a bunch of stuff in .zcompdump.  Have you investigated whether
certain parts of it are slower than others?

One lesson learned with Completion/Base/Utility/_store_cache is that
parsing array assignments is expensive.  _store_cache now writes out
the array contents as a here-document and splits with ${(Q)${(z)...}}
to avoid parser overhead.  The _comps array is large; maybe there is
a big win to be had by changing the format of the file.