Re: adaptive word scoring

[Sean Lynch <seanl@Internex.NET>]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

I remember reading earlier in this thread about the possibility of
rating words based on interestingness, and I think this is probably
the way to go.  The fundamental theorem of information theory tells us
that the value of any piece of information is inversely proportional
to its probability of occurrence.  Therefore, we should keep some sort
of history of the number of occurrences of each word in the adaptive
scoring criteria (i.e. the subject lines) and estimate the probability
of each word's occurrence, weighting the affect of each word on the
final score by the inverse of the probability.

Multiple occurrences of the same word in a given line probably should
not be counted, because I can't think of any situation where we'd want
to score an article differently just because a word occurred more than
once in the subject line.  However, I guess maybe people would want to
score an article lower if it had the word "free" more than once in the
subject line.

The final score given to a word would be, using the example Robert
gave, -50*(probability of the occurrence of this word)/(sum of
probabilities of all distinct words in this line)

The score of the word in the database would be adjusted by adding
(old score - new score)/c to it, where c is the speed of light.  C
could decrease over time so that scores would stabilize, though this
would cause scores to stop adapting eventually.  

Obviously, there would be some sort of thresholding function to drop
words with a large probability of occurrence.

>>>>> Sometime around 05 Dec 1996 19:50:56 +0100,
>>>>> in article <m2ral4rfjj.fsf@proletcult.slip.ifi.uio.no>,
>>>>> someone posing as Lars Magne Ingebrigtsen <larsi@ifi.uio.no> wrote:

Lars> Robert Bihlmeyer <e9426626@student.tuwien.ac.at> writes:
>> (I think I've already suggested this once:) IMHO word-scoring
>> should derive it's scores from the normal adaptive (line-)scores
>> (by deviding the line-score by the number of words in the
>> subject). So if killed articles would get -50 for their subject,
>> words from the subject should be scored with -50/(number of words)
>> each (ignored words are not include in this count). In this way,
>> word-scoring mimics the effects for line-scoring for single
>> subjects, but has the intended side-effects on others. Furthermore,
>> the scores would by a magnitude smaller than the line-scores,
>> giving the latter priority.

Lars> I think that sounds reasonable...  Anybody have any thoughts on
Lars> this?

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

Sean Lynch,  Internex Network Operations                 <noc@internex.net>
Voice: +1 408 327 2200  Fax: +1 408 496 5484  <URL:http://www.internex.net>
Technical support: <support@internex.net> <URL:http://support.internex.net>