Re: Slowdown around 5.0.5-dev-0

[Sebastian Gniazdowski <sgniazdowski@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2540 bytes --]

I ran the performance tests again with script formatting them
differently – they are aggregated on test function, not on zsh
version. This allows easier interpretation. A new conclusion is that
zhalloc patch doesn't help for string_test. So the first patch (36834)
speeds up string operations and also introduces increased memory
usage.

Best regards,
Sebastian Gniazdowski

On 22 October 2015 at 14:49, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> On 19 October 2015 at 19:21, Bart Schaefer <schaefer@brasslantern.com> wrote:
>> We leave that empty arena at the end, but then on the next zhalloc()
>> it's still not big enough, so another one gets allocated.
>>
>> You could try removing that "else" to let the trailing block be freed
>> again, but then we're back to needing to compare speed as well as space
>> across shell versions.
>
> I removed the else, code works longer and memory is still allocated.
> This is seen in mem-test3.txt, where there are many samples and large
> numbers, and in perf-test.txt, where there noted is long running time
> (numbers are milliseconds)
>
> I wrote perf-test.zsh to automatically perform performance tests:
> https://github.com/psprint/zsh-tools/
>
> With it and with mem-test.zsh I tested following zsh variants:
> - 5.0.2 osx
> - newheaps patch (zsh-newheaps-only)
> - newheaps and zhalloc patch 36836 (zsh-newheaps-zhalloc)
> - zhalloc and the previous patch 36834 (zsh-head-both-patches)
> - all three patches (zsh-newheaps-three-patches)
> - a 5.1.1-dev-0 clean zsh
> - the empty arena else removed
>
> My conclusions:
> - 36834 introduces high memory usage; I would suggest to use only two
> optimizations – newheaps and zhalloc; my znavtools are instant fast
> with them, what's slower is searching (wrote search_test function,
> results attached); basically half of pattern search speedup is lost
> with 36834, however what's gained is beautiful memory usage, without
> firefox-like values for RES (~700 MB); newheaps is responsible for
> instant-responsivity of my script, 36836 for much faster searching –
> that's a good compromise
>
> - searching with patterns through very large arrays (700k) allocates
> 264 MB of memory regardless of zsh version; this probably tells
> something
>
> - still not sure why 36834 allocates much memory for the string test
>
> - newheaps makes function calls longer, but it's not a substantial
> difference – 21196 ms vs. 18549 ms
>
> Best regards,
> Sebastian Gniazdowski

[-- Attachment #2: perf-test2.txt --]
[-- Type: text/plain, Size: 1880 bytes --]

Running [zsh-empty-arena-else]:          string_test      24933,14
Running [zsh-newheaps-zhalloc]:          string_test      10045,79
Running [zsh-newheaps-only]:             string_test      10053,55
Running [zsh-5.0.2-osx]:                 string_test       9726,88
Running [zsh-head-both-patches]:         string_test       4204,30
Running [zsh-newheaps-three-patches]:    string_test       4406,47
Running [zsh-5.1.1-dev-0-clean]:         string_test       9600,21

Running [zsh-empty-arena-else]:          array_test       12689,70
Running [zsh-newheaps-zhalloc]:          array_test       12042,77
Running [zsh-newheaps-only]:             array_test       11985,55
Running [zsh-5.0.2-osx]:                 array_test       12206,27
Running [zsh-head-both-patches]:         array_test       12320,56
Running [zsh-newheaps-three-patches]:    array_test       11788,41
Running [zsh-5.1.1-dev-0-clean]:         array_test       12814,19

Running [zsh-empty-arena-else]:          function_test    21774,49
Running [zsh-newheaps-zhalloc]:          function_test    21981,39
Running [zsh-newheaps-only]:             function_test    21268,29
Running [zsh-5.0.2-osx]:                 function_test    18539,70
Running [zsh-head-both-patches]:         function_test    18774,33
Running [zsh-newheaps-three-patches]:    function_test    21171,02
Running [zsh-5.1.1-dev-0-clean]:         function_test    18940,36

Running [zsh-empty-arena-else]:          search_test       3934,60
Running [zsh-newheaps-zhalloc]:          search_test       7007,60
Running [zsh-newheaps-only]:             search_test      11653,22
Running [zsh-5.0.2-osx]:                 search_test      11136,82
Running [zsh-head-both-patches]:         search_test       3872,39
Running [zsh-newheaps-three-patches]:    search_test       3960,67
Running [zsh-5.1.1-dev-0-clean]:         search_test      11281,82


[-- Attachment #3: perf-test.zsh --]
[-- Type: application/octet-stream, Size: 2219 bytes --]

#!/bin/zsh

emulate -L zsh
setopt extendedglob

zmodload zsh/zprof

zshs=( zsh-empty-arena-else zsh-newheaps-zhalloc zsh-newheaps-only )
zshs+=( zsh-5.0.2-osx zsh-head-both-patches zsh-newheaps-three-patches zsh-5.1.1-dev-0-clean )

#
# Children administration
#

trap "finished" SIGUSR1

finished() {
    FINISHED=1
}

# Waits for signal from child process
wait_for_end_of_test() {
    while [ "$FINISHED" -eq 0 ]; do
        sleep 1
    done
    kill -15 "$SUB_PID"
}

_finished_signal_wait() {
    kill -SIGUSR1 "$MAIN_PID"
    sleep 60
}

#
# Tests
#

tests=( string_test array_test function_test search_test )

float multiplier=0.5

string_test() {
    local a=""
    integer i=$(( 150000*multiplier ))
    repeat $i; do a+="$i"; done
}

array_test() {
    typeset -a a
    integer i=$(( 25000*multiplier ))
    repeat $i; do a+=( $i ); done
}

function_test() {
    local count
    integer i=$(( 10000*multiplier ))

    if [ -z "$1" ]; then
        repeat $i; do function_test 100; done
    else
        count="$1"
    fi

    if (( count -- > 0 )); then
        function_test "$count"
    fi
}

search_test() {
    integer elements=$(( 800000 * multiplier ))
    a="${(r:elements:: _:)b}"
    a=( $=a )
    a=( "${(@M)a:#(#i)*_*}" )
    a=( "${(@)a//(#mi)(_|a)/-${MATCH}-}" )
    a=( "${(@)a//(#bi)(_|-)/|${match[1]}|}" )
}

#
# Main code
#

# Detect main vs. for-test invocation
if [ -z "$1" ]; then

    for test in "$tests[@]"; do
        for current_zsh in "$zshs[@]"; do

            type "$current_zsh" 2>/dev/null 1>&2 || { echo >&2 "Skipping non-accessible $current_zsh"; continue }
            zsh_binary="${current_zsh##*/}"
            FINISHED=0
            TEST="$test"

            "$current_zsh" -c "source ./$0 $$ \"$current_zsh\" $test" &

            SUB_PID=$!
            wait_for_end_of_test
        done

        echo

    done

else
    MAIN_PID="$1"
    zsh_binary="${2##*/}"
    shift
    shift
    txt="Running [$zsh_binary]: "
    echo -n "${(r:40:: :)txt}" "${(r:15:: :)*}"

    # Run the test
    zprof -c
    "$@"
    zprof_out=( "${(@f)"$( zprof )"}" )
    zprof_out="$zprof_out[3]"
    zprof_out=( $=zprof_out )
    echo "${(l:10:: :)zprof_out[3]}"

    _finished_signal_wait
fi