Re: adaptive word scoring

[Janne Sinkkonen <janne@avocado.pc.helsinki.fi>]

Sean Lynch <seanl@Internex.NET> writes:

> the way to go.  The fundamental theorem of information theory tells us
> that the value of any piece of information is inversely proportional
> to its probability of occurrence.

To the logarithm of the probability, actually. This holds as long as
the events are independent. The occurrence of words depend on the
context, but we get an approximation anyway.

> The score of the word in the database would be adjusted by adding
> (old score - new score)/c to it, where c is the speed of light.

This makes sense (given c is in appropriate units).

> C could decrease over time so that scores would stabilize, though
> this would cause scores to stop adapting eventually.

I vote against decreasing C. Instead, it should be a constant small
value, say something between 0.05 and 0.001. Reading pattern changes
etc. - the scores should adapt all the time. 

> Obviously, there would be some sort of thresholding function to drop
> words with a large probability of occurrence.

And words with small probabilities should not be in the calculations
because the probability estimates are unstable.

-- 
Janne Sinkkonen      <janne@iki.fi>      <URL: http://www.iki.fi/~janne/ >