Regular expression performance in emacs, number of cached regexps

[gsstark@mit.edu (Gregory S. Stark)]

In profiling an elisp program I was working on I found that an inordinate
quantity of string memory was being allocated in a function I thought I had
optimized extensively. Using the source, I eventually found that the memory
was allocated by the regular expression searching functions to cache a small
constant number of compiled regular expressions.

Unfortunately in my code, which was fairly short and simple, the behaviour was
pathological. If I recall correctly, Emacs cached the five most recently used
compiled regular expressions, and I was looping through about six or seven. So
Emacs was repeatedly recompiling the same six or seven regular expressions and
allocating memory only to be thrown away as garbage shortly later, causing
excessive regular expression compilations and garbage collections.

I've since optimized my package to use less than five regular expressions, but
my concern is that several other large packages including W3 and Gnus use
re-search-forward and looking-at liberally in code that is otherwise quite
carefully optimized (more or less). Reducing the number of regular expressions
these packages use in loops is likely to be quite hard. I suspect increasing
the compile-time configuration constant governing the size of this cache will
be a big speed and memory boost for these packages.

greg

In case people are curious, here is the code I use to profile memory usage in
my packages. I've suggested something like this be integrated in elp.el, which
I don't think would really be all that hard, but I don't think elp.el is under
active development, which is a shame, it's a great tool.

(defadvice the-function-to-profile (around profile-memory)
  (let (before after delta)
    (setq before (memory-use-counts))
    ad-do-it
    (setq after (memory-use-counts))
    (while before
    (push (- (pop after) (pop before)) delta))
    (apply 'message
           "MEMORY USE cons:%d fl:%d vec:%d sym:%d str:%d misc:%d int:%d"
           (nreverse delta))))