Status of optimizations – can have native buffers

Status of optimizations – can have native buffers

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

Hello,
with the optimizations for array (Zsh 5.3.1-dev-0):

    typeset -a array
    array[100]=""
    array[20,29]=( 1 2 3 4 5 6 7 8 9 10 )

we can now have somewhat "native" buffers in zsh. Above will not do
anything with elements outside <20,30> range.

One use case are 2-dimensional arrays, with e.g. the library I sent to
zsh-users (ID: 21926). Created are e.g. 10 fixed rows of 100 elements.
Then it's possible to fill rows by storing to computed indexes. Or to
fill entire row by storing to <row_begin,row_end>. 

The same has been done with strings (Zsh 5.3), and string can be a
native buffer too.

That said, when I test with my screen-saver zsh-morpho
(https://github.com/psprint/zsh-morpho), which uses hash tables for its
pixel native buffer, arrays are still slower. It is 55.9 s vs. 27.3 s.
Callgrind reveals the cause:

59,930,247,427  arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2]
15,675,075,695  szone_free_definite_size
[/usr/lib/system/libsystem_malloc.dylib]
11,983,363,082  tiny_malloc_from_free_list
[/usr/lib/system/libsystem_malloc.dylib]
 8,037,180,402  tiny_free_list_add_ptr
 [/usr/lib/system/libsystem_malloc.dylib]
 7,181,545,835  _platform_memmove$VARIANT$Nehalem
 [/usr/lib/system/libsystem_platform.dylib]
(full data in attachement)

The buffer has size 34x158=5372 elements. Without the optimizations that
are the topic, time is 367 s.
-- 
  Sebastian Gniazdowski
  psprint3@fastmail.com

[-- Attachment #2: callgrind_annotate.txt --]
[-- Type: text/plain, Size: 9089 bytes --]

--------------------------------------------------------------------------------
Profile data file 'callgrind.out.9896' (creator: callgrind-3.12.0)
--------------------------------------------------------------------------------
I1 cache: 
D1 cache: 
LL cache: 
Timerange: Basic block 0 - 44213777818
Trigger: Program termination
Profiled target:  zsh-5.3.1-dev-0_O2 ./zmorpho (PID 9896, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:     
User annotated:   
Auto-annotation:  off

--------------------------------------------------------------------------------
             Ir 
--------------------------------------------------------------------------------
210,464,472,644  PROGRAM TOTALS

--------------------------------------------------------------------------------
            Ir  file:function
--------------------------------------------------------------------------------
59,930,247,427  ???:arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2]
15,675,075,695  ???:szone_free_definite_size [/usr/lib/system/libsystem_malloc.dylib]
11,983,363,082  ???:tiny_malloc_from_free_list [/usr/lib/system/libsystem_malloc.dylib]
 8,037,180,402  ???:tiny_free_list_add_ptr [/usr/lib/system/libsystem_malloc.dylib]
 7,181,545,835  ???:_platform_memmove$VARIANT$Nehalem [/usr/lib/system/libsystem_platform.dylib]
 6,887,423,013  ???:szone_size [/usr/lib/system/libsystem_malloc.dylib]
 6,701,641,389  ???:itype_end [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 6,668,004,434  ???:arrlen [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 5,979,575,631  ???:tiny_free_list_remove_ptr [/usr/lib/system/libsystem_malloc.dylib]
 5,583,741,132  ???:szone_malloc_should_clear [/usr/lib/system/libsystem_malloc.dylib]
 3,676,908,734  ???:strcpy [/usr/lib/dyld]
 3,229,526,238  ???:free [/usr/lib/dyld]
 3,006,353,708  ???:get_tiny_free_size [/usr/lib/system/libsystem_malloc.dylib]
 2,690,849,561  ???:strlen [/usr/lib/dyld]
 2,677,510,801  ???:stringsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 2,558,606,524  ???:stringsubst'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 2,370,009,449  ???:ingetc [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 2,103,253,219  ???:zalloc [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 2,010,130,192  ???:malloc_zone_malloc [/usr/lib/system/libsystem_malloc.dylib]
 1,980,835,207  ???:strtod_l [/usr/lib/system/libsystem_c.dylib]
 1,871,413,725  ???:dquote_parse [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,651,513,719  ???:zhalloc [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,641,875,732  ???:_platform_strcmp [/usr/lib/system/libsystem_platform.dylib]
 1,604,082,741  ???:get_tiny_previous_free_msize [/usr/lib/system/libsystem_malloc.dylib]
 1,574,144,473  ???:zzlex [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,435,808,720  ???:malloc [/usr/lib/dyld]
 1,334,946,291  ???:ztrdup [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,255,063,534  ???:__quorem_D2A [/usr/lib/system/libsystem_c.dylib]
 1,221,146,102  ???:__multadd_D2A [/usr/lib/system/libsystem_c.dylib]
 1,165,383,862  ???:untokenize [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,098,371,849  ???:lexconstant [/usr/local/bin/zsh-5.3.1-dev-0_O2]
 1,069,952,015  ???:fetchvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   982,451,064  ???:remnulargs [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   965,483,295  ???:paramsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   853,269,280  ???:getarg [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   792,779,000  ???:op [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   790,688,907  ???:hasher [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   771,110,306  ???:checkunary [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   763,084,062  ???:setlocale [/usr/lib/system/libsystem_c.dylib]
   762,984,113  ???:setarrvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   740,301,374  ???:mathparse'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   717,789,930  ???:_os_lock_spin_lock [/usr/lib/system/libsystem_platform.dylib]
   687,889,980  ???:__Balloc_D2A [/usr/lib/system/libsystem_c.dylib]
   661,099,642  ???:__mult_D2A [/usr/lib/system/libsystem_c.dylib]
   645,284,640  ???:paramsubst'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   644,208,465  ???:__lshift_D2A [/usr/lib/system/libsystem_c.dylib]
   628,587,537  ???:gethashnode2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   623,241,531  ???:addvars [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   595,811,848  ???:__dtoa [/usr/lib/system/libsystem_c.dylib]
   585,980,177  ???:__Bfree_D2A [/usr/lib/system/libsystem_c.dylib]
   585,163,299  ???:loadlocale [/usr/lib/system/libsystem_c.dylib]
   582,166,785  ???:mathparse [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   478,719,744  ???:getparamnode [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   464,243,313  ???:mathevall [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   457,242,648  ???:add [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   444,824,636  ???:zsfree [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   438,807,694  ???:getindex [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   427,077,934  ???:__vfprintf [/usr/lib/system/libsystem_c.dylib]
   418,216,827  ???:dupstring [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   406,725,998  ???:fetchvalue'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   383,005,195  ???:zstrtol_underscore [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   380,580,653  ???:_platform_strchr$VARIANT$Generic [/usr/lib/system/libsystem_platform.dylib]
   379,564,975  ???:__numeric_load_locale [/usr/lib/system/libsystem_c.dylib]
   377,953,347  ???:inpoptop [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   358,951,860  ???:szone_malloc [/usr/lib/system/libsystem_malloc.dylib]
   316,823,215  ???:__cmp_D2A [/usr/lib/system/libsystem_c.dylib]
   305,846,720  ???:freearray [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   303,615,681  ???:hist_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   297,508,538  ???:watchlog_match [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   295,769,198  ???:__vsnprintf_chk [/usr/lib/system/libsystem_c.dylib]
   287,116,172  ???:os_lock_unlock [/usr/lib/system/libsystem_platform.dylib]
   287,116,172  ???:os_lock_lock [/usr/lib/system/libsystem_platform.dylib]
   287,115,972  ???:_os_lock_spin_unlock [/usr/lib/system/libsystem_platform.dylib]
   272,071,205  ???:hist_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   268,128,151  ???:parse_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   263,544,579  ???:execlist'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   260,249,491  ???:hbegin [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   258,703,816  ???:__vsnprintf_chk'2 [/usr/lib/system/libsystem_c.dylib]
   257,234,183  ???:_platform_bzero$VARIANT$Merom [/usr/lib/system/libsystem_platform.dylib]
   245,441,859  ???:getmathparam [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   242,921,408  ???:prefork [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   240,386,850  ???:prefork'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   237,246,060  ???:__s2b_D2A [/usr/lib/system/libsystem_c.dylib]
   236,583,660  ???:parse_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   228,942,730  ???:__diff_D2A [/usr/lib/system/libsystem_c.dylib]
   224,754,367  ???:inpush [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   221,552,781  ???:getstrvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   215,349,190  ???:get_node_from_uniquing_table [/usr/lib/system/libsystem_malloc.dylib]
   209,424,935  ???:convbase_ptr [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   203,022,100  ???:dupstrpfx [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   197,153,050  ???:lex_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   195,374,559  ???:strncpy [/usr/lib/dyld]
   193,209,994  ???:zcontext_save_partial [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   190,469,954  ???:__d2b_D2A [/usr/lib/system/libsystem_c.dylib]
   187,983,320  ???:getnumvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   187,398,894  ???:arithsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   185,323,872  ???:zcontext_restore_partial [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   181,385,636  ???:hend [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   181,170,554  ???:assignstrvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   177,453,442  ???:inputsetline [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   176,710,920  ???:assignsparam [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   168,822,313  ???:ecgetstr [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   165,608,562  ???:lex_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   165,164,992  ???:execsimple'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   158,197,295  ???:__error [/usr/lib/system/libsystem_kernel.dylib]
   155,692,436  ???:__sfvwrite [/usr/lib/system/libsystem_c.dylib]
   154,124,892  ???:strcatsub [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   153,573,732  ???:__b2d_D2A [/usr/lib/system/libsystem_c.dylib]
   141,574,370  ???:__pow5mult_D2A [/usr/lib/system/libsystem_c.dylib]
   140,403,054  ???:parse_subscript [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   132,539,796  ???:zshlex_raw_add [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   131,809,956  ???:filesubstr [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   130,952,151  ???:filesub [/usr/local/bin/zsh-5.3.1-dev-0_O2]
   130,475,899  ???:localeconv_l [/usr/lib/system/libsystem_c.dylib]
   110,035,356  ???:pthread_getspecific [/usr/lib/system/libsystem_pthread.dylib]
   106,487,955  ???:inungetc [/usr/local/bin/zsh-5.3.1-dev-0_O2]
    94,891,375  ???:__detect_path_locale [/usr/lib/system/libsystem_c.dylib]

Re: Status of optimizations – can have native buffers

[Sebastian Gniazdowski]

On Sat, Mar 4, 2017, at 05:49 AM, Sebastian Gniazdowski wrote:
> Callgrind reveals the cause:
> 
> 59,930,247,427  arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2]

I thought that badarrays branch would have chances for merge if a
vallgrind-based testing suite would have been developed. To reassure all
people that code is safe. With many edge cases and stress tests, and
valgrind's infallibility, it could be a no-risk merge.

-- 
  Sebastian Gniazdowski
  psprint3@fastmail.com

Re: Status of optimiza tions - can have n ative buffers

[Bart Schaefer]

On Mar 4,  5:49am, Sebastian Gniazdowski wrote:
}
} Callgrind reveals the cause:
} 
} 59,930,247,427  arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2]

Yeah, I think we've already determined that we're not getting away from
that without a significant rewrite to stop managing arrays as simple
char ** pointers.

Although the following might help in some cases -- (v->start < 0) can't
be true very often or arrlen() would have shown up nearly as much as
arrlen_le(), but maybe this shaves a few unnecessary recounts?

Anyway changing from arrlen_le() to arrlen_gt() removes a couple of
tail calls, so this might get faster, and will also reveal how many
of those arrlen_le() are coming through here.

diff --git a/Src/params.c b/Src/params.c
index 8942fef..6164ba6 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -2101,10 +2101,14 @@ getstrvalue(Value v)
 	if (v->isarr)
 	    s = sepjoin(ss, NULL, 1);
 	else {
-	    if (v->start < 0)
-		v->start += arrlen(ss);
-	    s = (arrlen_le(ss, v->start) || v->start < 0) ?
-		(char *) hcalloc(1) : ss[v->start];
+	    if (v->start < 0) {
+		if ((v->start += arrlen(ss)) < 0)
+		    s = (char *) hcalloc(1);
+		else
+		    s = ss[v->start];
+	    } else
+		s = arrlen_gt(ss, v->start) ?
+		     ss[v->start] : (char *) hcalloc(1);
 	}
 	return s;
     case PM_INTEGER:

Re: Status of optimiza tions - can have n ative buffers

[Sebastian Gniazdowski]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Sat, Mar 4, 2017, at 04:00 PM, Bart Schaefer wrote:
> Although the following might help in some cases -- (v->start < 0) can't
> be true very often or arrlen() would have shown up nearly as much as
> arrlen_le(), but maybe this shaves a few unnecessary recounts?
> 
> Anyway changing from arrlen_le() to arrlen_gt() removes a couple of
> tail calls, so this might get faster, and will also reveal how many
> of those arrlen_le() are coming through here.

Time is still the same, 25 s. The zrealloc-patch that was pushed helped,
before: 37 s, after: the 25 s. Hashes are 17 s. Attached callgrind
result from with-patch current-head run:

29,965,124,064  arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
29,965,124,049  arrlen_gt [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 6,700,764,538  itype_end [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 6,668,004,434  arrlen [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 4,552,766,774  _platform_memmove$VARIANT$Nehalem
 [/usr/lib/system/libsystem_platform.dylib]
 2,677,827,871  stringsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]

-- 
Sebastian Gniazdowski
psprint3@fastmail.com

[-- Attachment #2: callgrind_annotate_arrlen_le.txt --]
[-- Type: text/plain, Size: 9927 bytes --]

--------------------------------------------------------------------------------
Profile data file 'callgrind.out.96783' (creator: callgrind-3.12.0)
--------------------------------------------------------------------------------
I1 cache: 
D1 cache: 
LL cache: 
Timerange: Basic block 0 - 33571987355
Trigger: Program termination
Profiled target:  zsh-5.3.1-dev-0_O2_bopt ./zmorpho (PID 96783, part 1)
Events recorded:  Ir
Events shown:     Ir
Event sort order: Ir
Thresholds:       99
Include dirs:     
User annotated:   
Auto-annotation:  off

--------------------------------------------------------------------------------
             Ir 
--------------------------------------------------------------------------------
140,296,535,115  PROGRAM TOTALS

--------------------------------------------------------------------------------
            Ir  file:function
--------------------------------------------------------------------------------
29,965,124,064  ???:arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
29,965,124,049  ???:arrlen_gt [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 6,700,764,538  ???:itype_end [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 6,668,004,434  ???:arrlen [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 4,552,766,774  ???:_platform_memmove$VARIANT$Nehalem [/usr/lib/system/libsystem_platform.dylib]
 2,677,827,871  ???:stringsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 2,558,606,524  ???:stringsubst'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 2,370,011,695  ???:ingetc [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,994,378,494  ???:strcpy [/usr/lib/system/libsystem_c.dylib]
 1,975,384,760  ???:strtod_l [/usr/lib/system/libsystem_c.dylib]
 1,871,413,725  ???:dquote_parse [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,763,948,314  ???:szone_free_definite_size [/usr/lib/system/libsystem_malloc.dylib]
 1,651,427,451  ???:zhalloc [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,641,937,107  ???:_platform_strcmp [/usr/lib/system/libsystem_platform.dylib]
 1,574,721,628  ???:zzlex [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,274,353,832  ???:tiny_free_list_add_ptr [/usr/lib/system/libsystem_malloc.dylib]
 1,254,904,343  ???:__quorem_D2A [/usr/lib/system/libsystem_c.dylib]
 1,226,489,880  ???:__multadd_D2A [/usr/lib/system/libsystem_c.dylib]
 1,188,316,071  ???:strlen [/usr/lib/system/libsystem_c.dylib]
 1,176,570,715  ???:tiny_malloc_from_free_list [/usr/lib/system/libsystem_malloc.dylib]
 1,165,383,862  ???:untokenize [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,098,752,923  ???:lexconstant [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
 1,069,738,177  ???:fetchvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   983,039,508  ???:remnulargs [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   971,165,723  ???:szone_size [/usr/lib/system/libsystem_malloc.dylib]
   965,483,295  ???:paramsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   853,269,280  ???:getarg [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   816,363,124  ???:szone_malloc_should_clear [/usr/lib/system/libsystem_malloc.dylib]
   792,748,080  ???:op [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   790,606,736  ???:hasher [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   771,054,754  ???:checkunary [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   763,084,062  ???:setlocale [/usr/lib/system/libsystem_c.dylib]
   740,303,435  ???:mathparse'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   690,597,783  ???:__Balloc_D2A [/usr/lib/system/libsystem_c.dylib]
   669,049,573  ???:__mult_D2A [/usr/lib/system/libsystem_c.dylib]
   645,284,640  ???:paramsubst'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   642,438,873  ???:__lshift_D2A [/usr/lib/system/libsystem_c.dylib]
   628,523,356  ???:gethashnode2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   622,923,429  ???:addvars [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   596,886,151  ???:__dtoa [/usr/lib/system/libsystem_c.dylib]
   588,286,824  ???:__Bfree_D2A [/usr/lib/system/libsystem_c.dylib]
   585,163,299  ???:loadlocale [/usr/lib/system/libsystem_c.dylib]
   582,074,808  ???:mathparse [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   479,005,053  ???:free [/usr/lib/system/libsystem_malloc.dylib]
   478,653,459  ???:getparamnode [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   464,179,856  ???:mathevall [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   457,242,648  ???:add [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   438,807,694  ???:getindex [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   427,244,753  ???:__vfprintf [/usr/lib/system/libsystem_c.dylib]
   418,205,697  ???:dupstring [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   406,725,998  ???:fetchvalue'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   392,086,757  ???:tiny_free_list_remove_ptr [/usr/lib/system/libsystem_malloc.dylib]
   383,005,195  ???:zstrtol_underscore [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   380,526,825  ???:_platform_strchr$VARIANT$Generic [/usr/lib/system/libsystem_platform.dylib]
   379,564,975  ???:__numeric_load_locale [/usr/lib/system/libsystem_c.dylib]
   377,953,347  ???:inpoptop [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   316,294,171  ???:__cmp_D2A [/usr/lib/system/libsystem_c.dylib]
   303,615,681  ???:hist_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   298,694,844  ???:malloc_zone_malloc [/usr/lib/system/libsystem_malloc.dylib]
   272,071,205  ???:hist_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   268,128,151  ???:parse_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   263,647,209  ???:execlist'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   260,249,562  ???:hbegin [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   259,535,774  ???:__vsnprintf_chk'2 [/usr/lib/system/libsystem_c.dylib]
   256,465,400  ???:_platform_bzero$VARIANT$Merom [/usr/lib/system/libsystem_platform.dylib]
   245,406,915  ???:getmathparam [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   242,868,938  ???:prefork [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   240,386,850  ???:prefork'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   237,275,094  ???:__s2b_D2A [/usr/lib/system/libsystem_c.dylib]
   236,583,660  ???:parse_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   234,598,487  ???:__vsnprintf_chk [/usr/lib/system/libsystem_c.dylib]
   229,297,192  ???:__diff_D2A [/usr/lib/system/libsystem_c.dylib]
   224,754,367  ???:inpush [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   220,048,614  ???:getstrvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   213,354,900  ???:malloc [/usr/lib/system/libsystem_malloc.dylib]
   209,424,935  ???:convbase_ptr [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   208,449,767  ???:zalloc [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   203,083,989  ???:get_tiny_free_size [/usr/lib/system/libsystem_malloc.dylib]
   202,999,700  ???:dupstrpfx [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   199,346,429  ???:get_tiny_previous_free_msize [/usr/lib/system/libsystem_malloc.dylib]
   197,153,050  ???:lex_context_save [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   195,301,367  ???:strncpy [/usr/lib/system/libsystem_c.dylib]
   193,209,994  ???:zcontext_save_partial [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   190,435,519  ???:__d2b_D2A [/usr/lib/system/libsystem_c.dylib]
   187,954,648  ???:getnumvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   187,239,378  ???:arithsubst [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   185,323,872  ???:zcontext_restore_partial [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   181,385,682  ???:hend [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   181,077,329  ???:assignstrvalue [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   177,453,489  ???:inputsetline [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   176,615,209  ???:assignsparam [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   168,755,279  ???:ecgetstr [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   165,608,562  ???:lex_context_restore [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   165,163,514  ???:execsimple'2 [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   158,197,505  ???:__error [/usr/lib/system/libsystem_kernel.dylib]
   155,426,633  ???:__sfvwrite [/usr/lib/system/libsystem_c.dylib]
   154,124,892  ???:strcatsub [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   151,322,700  ???:__b2d_D2A [/usr/lib/system/libsystem_c.dylib]
   145,111,600  ???:__pow5mult_D2A [/usr/lib/system/libsystem_c.dylib]
   140,403,054  ???:parse_subscript [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   132,539,922  ???:zshlex_raw_add [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   131,795,646  ???:filesubstr [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   130,934,661  ???:filesub [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   130,475,899  ???:localeconv_l [/usr/lib/system/libsystem_c.dylib]
   114,177,546  ???:watchlog_match [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
   110,387,904  ???:pthread_getspecific [/usr/lib/system/libsystem_pthread.dylib]
   106,565,100  ???:_os_lock_spin_lock [/usr/lib/system/libsystem_platform.dylib]
   106,487,955  ???:inungetc [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    94,891,375  ???:__detect_path_locale [/usr/lib/system/libsystem_c.dylib]
    90,138,378  ???:__hi0bits_D2A [/usr/lib/system/libsystem_c.dylib]
    87,155,328  ???:parsestrnoerr [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    82,869,602  ???:matheval [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    80,705,915  ???:setunderscore [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    74,920,775  ???:zfree [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    70,977,006  ???:getsparam [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    70,975,062  ???:init_parse_status [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    70,653,912  ???:ugetnode [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    67,451,051  ???:vsprintf_l [/usr/lib/system/libsystem_c.dylib]
    63,593,894  ???:isident [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    60,058,960  ???:convfloat [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
    59,307,895  ???:spin_lock [/usr/lib/system/libsystem_platform.dylib]
    56,113,944  ???:__ratio_D2A [/usr/lib/system/libsystem_c.dylib]
    55,202,826  ???:inpop [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]

Re: Status of optimiza tions - can have n ative buffers

[Bart Schaefer]

On Mar 5,  1:32am, Sebastian Gniazdowski wrote:
}
} result from with-patch current-head run:
} 
} 29,965,124,064  arrlen_le [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
} 29,965,124,049  arrlen_gt [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]
}  6,668,004,434  arrlen [/usr/local/bin/zsh-5.3.1-dev-0_O2_bopt]

Well now THAT is interesting.  Same total number of calls but now
they're split exactly half _le and half _gt.

This in turn strongly suggests that the calls are the one my patch
changed in params.c, and the one currently at line 2567 in subst.c.

If we could figure out whether these two are actually examining the
length of the same array each time, there might be a way (cache in
the struct value, as with the "arr" field?) to cut this in half.

Re: Status of optimizations – can have native buffers

[Sebastian Gniazdowski]

Just encountered a positive effect of what has been done:

https://asciinema.org/a/aexw8dvhuejq5w704ybfadhrn

I'm writing (D)HTML-like library for zcurses and created a history app.
89000 entries. Zsh-5.1.1 is slow, Zsh-5.3.1-dev-0 is fast :)

(Anyone interested in the ZUI library, here's a short demo:
https://asciinema.org/a/4r5g7wd61uwbvkcskj391i2uq )
-- 
  Sebastian Gniazdowski
  psprint3@fastmail.com

Re: Status of optimizations – can have native buffers

[Sebastian Gniazdowski]

On Mon, Mar 13, 2017, at 02:01 AM, Sebastian Gniazdowski wrote:
> Just encountered a positive effect of what has been done:
> 
> https://asciinema.org/a/aexw8dvhuejq5w704ybfadhrn

Turned out I was testing -O0 builds (both). With -O2, zsh-5.1.1 is still
slow, basically the same, while zsh-5.3.1-dev-0 is even faster – instant
startup time, compared to 1.5-2 seconds of 5.1.1. Also, searching is
very usable with 89k elements, while 5.1.1 is like a turtle ;)

-- 
  Sebastian Gniazdowski
  psprint3@fastmail.com