On Thu, Nov 10, 2016, at 04:31 AM, Peter Stephenson wrote:
> To do a good job optimising here, we really need state information
> outside the functions --- in an experiment with my start up files, only
> 16% of calls to untokenize() actually had any effect.  But recording the
> state generally is a very big change.
> 
> Some possible optimisations are along the following lines, although a
> bit of care it's needed as it's not necessarily the case on all
> architectures that the bit test used by itok() is necessarily faster
> than the range test the following replaces it with.  It did seem faster
> on this fairly standard Intel CPU.

Tested this and no big change, maybe 14 ms – running times are 2135 vs
2149, but that can be just instability. However callgrind reports
851,712,174 instructions instead of 1,177,994,701 for untokenize, while
other instruction counts are kept the same so the test seems valid.

My motivation is parsing of long Zsh code – would be a cool thing to
iterate long (z)-splitted input in say 400 ms instead of 2 seconds – a
dreamed result, maybe actually impossible, as disabling multibyte yields
1560 ms. State recording might seem bad but at least there is room for
improvement condensed in apparently few places, better than counting
cycles along whole Zsh code.

-- 
  Sebastian Gniazdowski
  psprint@fastmail.com