Re: adaptive word scoring

[Holger Franz <hfranz@physik.rwth-aachen.de>]

Just to toss in my two cents: How about adding the exponent feature
from the procmail scoring mechanism? It provides a flexible method to
keep scores from frequent matches low.

>From `man procmailsc`:
---8<---
Weighted regular expression conditions
       The  first  time  the regular expression is found, it will
       add w to the score.  The second time it is found, w*x will
       be  added.   The  third  time  it  is found, w*x*x will be
       added.  The fourth time w*x*x*x will  be  added.   And  so
       forth.

       This can be described by the following concise formula:

                                   n
                   n   k-1        x - 1
              w * Sum x    = w * -------
                  k=1             x - 1

       It  represents the total added score for this condition if
       n matches are found.

       Note that the following case distinctions can be made:

       x=0     Only the first match  will  contribute  w  to  the
               score.  Any subsequent matches are ignored.

       x=1     Every  match  will  contribute  the  same w to the
               score.  The score grows linearly with  the  number
               of matches found.

       0<x<1   Every match will contribute less to the score than
               the previous one.  The score  will  asymptotically
               approach  a  certain  value (see the NOTES section
               below).

       1<x     Every match will contribute more to the score than
               the  previous  one.   The score will grow exponen-
               tionally.

       x<0     Can be utilised to favour odd or  even  number  of
               matches.
---8<---

I think that could be implemented easily if one added to each rule a
counter that reflects how often the rule was already 'adapted'. There
is one major problem though: with an exponent 0<x<1 the first adaption
is likely to dominate over later adaptions. So maybe there will have
to be separate counters for positive and negative adaptions. If
positive and negative adaptions were tuned in such a way that they had
roughly the same asymptotic value, 'meaningless' words that are raised
and lowered randomly would gain a comparatively low net score.

All this sounds very alpha, but may be worth some more thought.

Holger 

-- 
Holger Franz <hfranz@physik.rwth-aachen.de>                             

Caution: feeding Gnus to your XEmacs will make it fat.