Email filing.

[Scott A Crosby <scrosby@cs.rice.edu>]

On Mon, 19 Aug 2002 01:44:05 -0400, prj@po.cwru.edu (Paul Jarc) writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Gro~johann) wrote:
> > There is a research field known as "information filtering" or
> > "(automatic) text classification" or "text categorization".  I don't
> > know the details of the theory, but folks in that community are
> > speaking of "naive Bayes classifiers" as one of the ways to do it --
> > maybe that's similar to his approach.
> 
> Sounds like it.  Anyone know if this (or another) method generalizes
> to more than two categories (spam/nonspam)?  If so, it could be used

Yup.

> for all mail splitting.  We wouldn't have to manually craft split
> rules; we'd just seed a new group with the mails we have so far that

Been done. Look at ifile. I've heard its slow though. 

And that although it works, it doesn't always work as nicely as you'd
like, in that it classifies right only about 80-90% of the time, but
that 10% will annoy you. IMHO, its only really useful when you have
email thats uncatagorizable by any other means.

It could try to identify mailing lists by noting list-headers but I
wouldn't want to bet on perfect reliability.

Personally, I can accept writing mailing list filters right now.  Then
I sort out some emails by putting it into folders by-author, and
finally use gnus-split-with-parent to put project emails with their
parents in the folder automatically. [1]

This is easy and has perfect reliability. However, if I had gobs of
uncatagorizable email (whether mailing list or not), I may consider
trying to use such a bayesean classifier.[2]

For spam-checking, I'm, doing an implementation of something that does
naive bayesean, but is flexible enough to be used for this. A *very
fast* implementation.... my benchmark right now for the statistics
building is 5 seconds on a 35mb, 7500 message corpus. V2 should be 30%
faster.)

> belong there, and their contents would let the computer guess which
> new mails belong with them.

Scott



[2] It wouldn't be *too* hard to link in to something like gnus.
Whenever a user manually classifies an email into a folder, tack on a
header 'manually moved here'. My core, when building statistics,
ignores all messages without that header.  Yeah, its a brute-scan, but
its fast. :) To classify, have a split rule pipe the email to the core
which outputs the suggested catagory.

[1] Extracts from my .gnus:

Among nice features. If I'm in a project/sender folder and send a
message, it gets stored there autommatically (by gcc-self). For all
non-mailing list emails, they obey my-split-fancy-with-parent.



;; Where to store responses:
;;  I use GCC-self as a topic parameter in sender.*/project.*, which
;;  are not marked total-expire. For everything else, put it by-date.
(setq gnus-message-archive-group
      '(;; Sent list email messages to list email.
;	("blah.*" "nnml:store_folder") ;Example for constant regexp matches
        (lambda (x) (cond 
		     ;; Store in existing group. (Done above with gcc-self)
;		      ((string-match "mail.*" group) (concat "nnml:" group))
;		      ((string-match "project.*" group) (concat "nnml:" group))
;		      ((string-match "sender.*" group) (concat "nnml:" group))
		      ;; If no match, just dump 
		      (t 
		       (concat "nnml:sent-mesg." (format-time-string "%Y-%m")))))))


(defun my-split-fancy-with-parent ()
  "Do a split-with-parent, however, ignore the result if it wants to put it in sent-mesg. This way, we put followups in the same group, however, we never put followups into sent-mesg."

  (let ((my-split (nnmail-split-fancy-with-parent)))
    ;;(message "Doing-my-split-fancy-with-parent")
    ;;(message my-split)
    (if (or (null my-split) (string-match "sent-mesg" my-split))
	nil
      my-split)))



(setq nnmail-split-fancy
      '(| 
	(| ;; Loop/Spam/special rules
	 ("X-Spam-Status" "Yes" "junk.spam")
	 ("Loop" "s?crosby@.*\.edu" "inbox.loop")
	 ("X-Delivery-Agent" "TMDA" "junk.tmda")
	 ("Delivery-Agent" "TMDA" "junk.tmda")
	 )
	(| ;; LIST RULES
	 ("Sender" "spamassassin-talk-admin@lists.sourceforge.net" "list.dev.spamassassin.talk")
	 ("Sender" "spamassassin-devel-admin@lists.sourceforge.net" "list.dev.spamassassin.dev")
	 ("Delivered-To" "alias-ding@gnus.org" "list.dev.gnus.ding")
	(: my-split-fancy-with-parent)
	(| ;; PROJECT RULES
	(| ;; SENDER RULES
	 ("from" "XXXXXXXXXX" "sender.family.mom")
	 ("from" "YYYYY" "sender.FOOBAR")
	 )
	(| ;; INBOX RULES
	 ;; Dump duplicates into dups. 	  
	 ("Gnus-warning" "duplicat\\(e\\|ion\\) of message" "junk.duplicates")
	 ("X-Spam-Level" "\\*\\*\\*\\*" "inbox.spam")
	 ("Precedence" "bulk\\|list" "inbox.bulk")
	 )
	"inbox.misc"
	)
      )

##

This below doesn't work pre-oorts, but it gives you an idea of what I wanted my group parameters to be. (I do it now by setting topic parameters.)


'(setq gnus-group-parameters
      '(
	(".*"
	 (display . all))
;; Projects and by-sender boxes get responses stored in same.
	("^\\(project|sender\\)\\..*"
         (gcc-self . t))