Slow parsing of large array assignments

[Bart Schaefer <schaefer@brasslantern.com>]

On Feb 8, 12:27pm, Bart Schaefer wrote:
}
} The slowdown here appears to be with compiling the source'd file into
} the internal wordcode format before executing it.  Even dumping the
} whole assignment as a single string and then using "eval" on that, is
} faster than allowing "source" to parse the words directly.  We may
} want to dig further into why that is the case.

The "slow" routine is parse.c:par_nl_wordlist(), which calls zshlex()
repeatedly [once for every word in the array].  This is slower when
reading from a file than when reading from a string in memory, which
explains the difference between "source" and "eval" for me.

It's also a lot slower than bufferwords(), which implements the (z)
flag for parameter splitting.

Finally, par_nl_wordlist() in the current revision of zsh seems to take
longer than in 4.2.0 or 5.0.7 for the same input.

Over 10 runs I get these average times sourcing a file that assigns
approximately 55000 words to an array:

zsh 4.2.0:                    3.91s
zsh-5.0.7 (release version):  4.85s
zsh-5.0.7-289-g7d15b9a:       7.47s

I believe the two zsh-5.0.7 were compiled with identical flags, though
the 5.0.7 binary is stripped; I don't know about the old 4.2.0.  If I
re-compile with -O in CFLAGS, g7d15b9a speeds up to 4.95s average.