spam*.el and ifile-gnus.el?

spam*.el and ifile-gnus.el?

[Kai Großjohann]

Has anyone connected spam*.el and ifile-gnus.el?

Right now, I move messages into the spam group to tell ifile that
they are spam, but maybe spam*.el can do that for me?

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: spam*.el and ifile-gnus.el?

[Alex Schroeder]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Has anyone connected spam*.el and ifile-gnus.el?
>
> Right now, I move messages into the spam group to tell ifile that
> they are spam, but maybe spam*.el can do that for me?

Here is what I do.  I filter using the BBDB as a whitelist.  I filter
away all HTML-only and Korean mail into mail.spam.filtered, because I
don't want to train spam-stat.el on them.  Some computer challenged
people send me HTML-only mail, so I need them in the list as well.  :)
The rest is either for mail.misc or mail.spam.  I use spam-stat.el to
make this decision for me.  And from time to time, I retrain
spam-stat.el on mail.misc and mail.spam only.

Thus, the splitting and training works fine using spam-stat.el.  This
approach is not compatible with the ideas in spam.el.  spam.el could
use spam-stat.el as well, but the glue code is missing (and no new
spam-stat.el being available does not improve the situation...).

Alex.

PS: At the moment spam-stat.el still must have some bugs in it, but it
happens very rarely.  This is why I have not yet released a new
version.  I would like to release it as soon as possible, so if
anybody wants to help debugging it, just ask.  :)

(setq nnmail-crosspost nil
      nnmail-split-methods 'nnmail-split-fancy
      bbdb/gnus-split-default-group nil
      bbdb/gnus-split-nomatch-function nil
      bbdb/gnus-split-myaddr-regexp gnus-ignored-from-addresses
      nnmail-expiry-wait 28
      gnus-total-expirable-newsgroups
      (regexp-opt '("mail.guile"
		    "mail.gnutella"
		    "mail.emacs.devel"
		    "mail.emacs.ilisp"
		    "mail.frieden.jetzt"
		    "mail.baeckeranlage"))
      nnmail-split-fancy
      `(| ("Gnus-Warning" "This is a duplicate" "mail.spam.duplicates")
	  ;; spam filtering based on spam assassin
	  ;; ("X-Spam-Flag" "YES" "mail.spam")
	  ;; ("X-Spam-Level" "\\*\\*\\*\\*\\*\\*\\*" "mail.spam.filtered")
	  ;; computer challenged people I know sending me HTML mails
	  ("From" "Sarah Leiken" "mail.family")
	  ;; remaining HTML only mail is spam
	  ("Content-Type" "text/html" "mail.spam.filtered")
	  ;; weird character sets are spam, too
	  ("Subject" "=?ks_c_5601-1987" "mail.spam.filtered")
	  ;; virus detection is spam
	  ;; ("Subject" "^Norton AntiVirus detected" "mail.spam.filtered")
	  ;; temporary spam stuff
	  ;; (any "machine365.com" "mail.spam.filtered")
	  ;; now use the BBDB to split
	  (: (lambda ()
	       (car (bbdb/gnus-split-method))))
	  ;; some of the packages I maintain
	  ("Subject" "\\(color-theme\\|ansi-color\\)" "mail.emacs")
	  ;; spam filtering based on statistics
	  (: spam-stat-split-fancy)
	  ;; mail from and to me, mail to any of the emacswiki addresses
	  ;; (any ,gnus-ignored-from-addresses "mail.misc")
	  ;; (any ".*@emacswiki.org" "mail.emacs.wiki")
	  ;; anonymous mail, probably spam
	  ;; "mail.spam.anonymous"
	  "mail.misc"))

Re: spam*.el and ifile-gnus.el?

[Kai Großjohann]

Alex Schroeder <alex@emacswiki.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> Has anyone connected spam*.el and ifile-gnus.el?
>>
>> Right now, I move messages into the spam group to tell ifile that
>> they are spam, but maybe spam*.el can do that for me?
>
> Here is what I do.

I'm actually quite happy with the things that ifile does.  Note that
I do *all* my splitting with ifile (except for
nnmail-split-fancy-with-parent).  Here are my split rules:

(setq nnmail-split-methods 'nnmail-split-fancy)
(setq
 nnmail-split-fancy
 '(| (: nnmail-split-fancy-with-parent)
     (: ifile-recommend)
     "inbox"))

Thanks to ifile-gnus.el, ifile knows when I manually move a message
into another group.

So I think all I really need spam.el to do is to tell ifile that the
message is spam (perhaps by moving it into the spam group), and also
the message should be marked as expirable.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: spam*.el and ifile-gnus.el?

[David Aspinwall]

>>"Kai.Grossjohann" == Kai Grossjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

> Thanks to ifile-gnus.el, ifile knows when I manually move a message
> into another group.

I like ifile-gnus.el, but one problem I have is that I use
nnmail-expiry-target to expire articles into a group.  When
that happens, ifile gets run.  I don't really want things
split into the expired articles group though.  I'm using a
hack which seems to work:

(defadvice nnmail-expiry-target-group
  (around do-not-run-ifile-when-expiring activate)
  "Don't run ifile when doing expiry"
  (let ((ifile-program "/bin/true"))
    ad-do-it))

but that's pretty ugly.  Is there any better way to do it?

Re: spam*.el and ifile-gnus.el?

[Arnd Kohrs]

Hi,

can ifile-gnus be used in the context of nnimap?  As far as I understand it
is only applicable to nnml.

Regards,

Arnd.

Re: spam*.el and ifile-gnus.el?

[Jeremy H. Brown]

Arnd Kohrs <kohrs@castify.net> writes:
> can ifile-gnus be used in the context of nnimap?  As far as I understand it
> is only applicable to nnml.

Right now it only hooks into nnml; making it play with other backends
(at least nnfolder and nnimap) is on the TODO list.  It may be as
simple as applying the advice to
gnus-request-{move,replace,accept}-article instead of nnml-*, but I
haven't yet taken the time to grok what additional message-munging is
performed by those functions and the various encoding functions they
call, and what the effect would be of advising them instead of the
backend-specific functions they call.

Jeremy

Re: spam*.el and ifile-gnus.el?

[Andreas Fuchs]

Today, Jeremy H. Brown <jhbrown@ai.mit.edu> wrote:
> Right now it only hooks into nnml; making it play with other backends
> (at least nnfolder and nnimap) is on the TODO list.  It may be as
> simple as applying the advice to
> gnus-request-{move,replace,accept}-article instead of nnml-*, but I
> haven't yet taken the time to grok what additional message-munging is
> performed by those functions and the various encoding functions they
> call, and what the effect would be of advising them instead of the
> backend-specific functions they call.

Hm, I tried to do something similar (just rename the advised functions
in ifile-gnus.el) for imap at work, but that didn't work out as I
expected. It seems like not the entire article is known to nnimap when
it gets split, so ifile fails; I didn't investigate this further,
though.

If you need somebody to debug the nnimap backend, consider me a
volunteer (-:

Happy hacking,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
Was I helpful?  Let others know:
http://svcs.affero.net/rm.php?r=antifuchs

Re: spam*.el and ifile-gnus.el?

[Jeremy H. Brown]

Andreas Fuchs <asf@void.at> writes:
> Hm, I tried to do something similar (just rename the advised functions
> in ifile-gnus.el) for imap at work, but that didn't work out as I
> expected. It seems like not the entire article is known to nnimap when
> it gets split, so ifile fails; I didn't investigate this further,
> though.

Now that I think about this, this is probably a major problem with
using ifile and imap together: one of the main points using imap is
not downloading the entire message before you really want it.  I'll
bet gnus only downloads the headers for splitting.  To use client-side
ifile, gnus will have to download entire messages (or at least all the
text parts), which will probably have significant performance impact
when you're on a slow connection.  When I surface for air in a week or
two, I'll maybe look into implementing this as an option, and see how
much people grouse about performance impact.

Jeremy

PS For IMAP, server-side ifile-splitting would really be the right
   thing.  That is beyond the scope of my current mail-hacking
   ambitions.

Re: spam*.el and ifile-gnus.el?

[Arnd Kohrs]

    Jeremy> Now that I think about this, this is probably a major
    Jeremy> problem with using ifile and imap together: one of the main
    Jeremy> points using imap is not downloading the entire message
    Jeremy> before you really want it.  I'll bet gnus only downloads the
    Jeremy> headers for splitting.  To use client-side ifile, gnus will
    Jeremy> have to download entire messages (or at least all the text
    Jeremy> parts), which will probably have significant performance
    Jeremy> impact when you're on a slow connection.  When I surface for
    Jeremy> air in a week or two, I'll maybe look into implementing this
    Jeremy> as an option, and see how much people grouse about
    Jeremy> performance impact.


Well, this is the trade-off of using IMAP.  On the one hand it's a
relief that you store your mails somewhere on the net accessible and
save, and on the other hand you can't do nifty stuff such as nnir and
ifile.  I am about to re-migrate from nnml to imap.

However, before I start, I would like to raise two questions:

First, is it not so that during splitting for moving a mail from the imap-inbox
to the imap-destination-folder, the mail is in its text-entirety copied to
emacs and than stored on the imap-destination-folder?  So there would be
hardly any performance loss if all mails are downloaded in their
text-entirety assuming that most(all) mails get moved anyway to another
folder?  

Second, would ifile make sense on headers only, for spam decisions ?

Cheers,

Arnd.

Re: spam*.el and ifile-gnus.el?

[Simon Josefsson]

Arnd Kohrs <kohrs@castify.net> writes:

>     Jeremy> Now that I think about this, this is probably a major
>     Jeremy> problem with using ifile and imap together: one of the main
>     Jeremy> points using imap is not downloading the entire message
>     Jeremy> before you really want it.  I'll bet gnus only downloads the
>     Jeremy> headers for splitting.

Yes.

> First, is it not so that during splitting for moving a mail from the
> imap-inbox to the imap-destination-folder, the mail is in its
> text-entirety copied to emacs and than stored on the
> imap-destination-folder?  So there would be hardly any performance
> loss if all mails are downloaded in their text-entirety assuming
> that most(all) mails get moved anyway to another folder?

The nnimap splitter only downloads headers.  Moving the message is
done on the server.

It should be simple to use imap.el to iterate over all articles in a
mailbox and run some code on each message and then upload the result
to another mailbox.

Or use server-side filters like Sieve or Sendmail Milter.

Re: spam*.el and ifile-gnus.el?

[Jeremy H. Brown]

Arnd Kohrs <kohrs@castify.net> writes:
> Second, would ifile make sense on headers only, for spam decisions ?

I have no idea.  It would be an interesting experiment!

Jeremy

Re: spam*.el and ifile-gnus.el?

[Ted Zlatanov]

On Fri, 13 Sep 2002, kohrs@castify.net wrote:
> Second, would ifile make sense on headers only, for spam decisions ?

I think so, I am able to visually distinguish spam messages in about
95% of the cases based on the subject: header alone, in the summary
buffer.  Spam messages are generally about things you don't care about :)

Ted

Re: spam*.el and ifile-gnus.el?

[Clemens Fischer]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 13 Sep 2002, kohrs@castify.net wrote:
>> Second, would ifile make sense on headers only, for spam decisions ?
>
> I think so, I am able to visually distinguish spam messages in about
> 95% of the cases based on the subject: header alone, in the summary
> buffer.  Spam messages are generally about things you don't care about :)

letting ifile judge on the subject line alone if it usually gets
messages in toto may give wrong results.  or maybe if it is really
only the subject line, then you'll miss "only" the porn spams saying
"hello, darling" or "back from vacation".  chances rise if you feed
all of the headers, but full accuracy comes from full data, as always.

clemens

Re: spam*.el and ifile-gnus.el?

[Kai Großjohann]

jhbrown@ai.mit.edu (Jeremy H. Brown) writes:

> Right now it only hooks into nnml; making it play with other backends
> (at least nnfolder and nnimap) is on the TODO list.  It may be as
> simple as applying the advice to
> gnus-request-{move,replace,accept}-article instead of nnml-*, but I
> haven't yet taken the time to grok what additional message-munging is
> performed by those functions and the various encoding functions they
> call, and what the effect would be of advising them instead of the
> backend-specific functions they call.

Maybe the right solution is to change gnus-request-*-article to run
some hooks just before invoking the backend-specific functions.  Then
your code can add stuff to the hooks.  WDYT?

kai
-- 
~~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: spam*.el and ifile-gnus.el?

[Jeremy H. Brown]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> Maybe the right solution is to change gnus-request-*-article to run
> some hooks just before invoking the backend-specific functions.  Then
> your code can add stuff to the hooks.  WDYT?

That sounds perfect.  For full generality, you might even want hooks
before and after each of the backend-specific invocations.

Jeremy

Re: spam*.el and ifile-gnus.el?

[Jeremy H. Brown]

Interesting problem.  There isn't currently a better mechanism than
what you've done, but I'll add one.  I'd like suggestions on what it
should look like.  Some options:

(ifile-set-active bool), or
(ifile-activate) and (ifile-deactivate),  or
(with-ifile-inactive body) 

Preferences? Other ideas?

Thanks,
Jeremy



David Aspinwall <aspinwall@timesten.com> writes:
> I like ifile-gnus.el, but one problem I have is that I use
> nnmail-expiry-target to expire articles into a group.  When
> that happens, ifile gets run.  I don't really want things
> split into the expired articles group though.  I'm using a
> hack which seems to work:
> 
> (defadvice nnmail-expiry-target-group
>   (around do-not-run-ifile-when-expiring activate)
>   "Don't run ifile when doing expiry"
>   (let ((ifile-program "/bin/true"))
>     ad-do-it))
> 
> but that's pretty ugly.  Is there any better way to do it?

Re: spam*.el and ifile-gnus.el?

[Ted Zlatanov]

On Tue, 10 Sep 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
> Has anyone connected spam*.el and ifile-gnus.el?
> 
> Right now, I move messages into the spam group to tell ifile that
> they are spam, but maybe spam*.el can do that for me?

The new spam.el has actions on entering and exiting a spam group: mark
all unread articles with the spam-mark, and process all spam articles
with bogofilter, respectively.  This is (to quote the comment) a bit
raw, but the idea is that whatever is marked as spam when you exit a
spam group should be reported as spam.  Reporting may mean bogofilter
processing, ifile submission, or submission to a spam service.  The
ifile submission, specifically, would tell ifile that this mail was
split to the "spam" folder, but that should be a symbolic entity
rather than the static strings ifile uses right now, because the spam
group names should be customizable.

Ideas and patches to spam.el are welcome, of course.

Thanks
Ted

Re: spam*.el and ifile-gnus.el?

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This is (to quote the comment) a bit raw, but the idea is that
> whatever is marked as spam when you exit a spam group should be
> reported as spam.  Reporting may mean bogofilter processing, ifile
> submission, or submission to a spam service.

But the submission thing hasn't been written yet, I guess?

> Ideas and patches to spam.el are welcome, of course.

I want to use spam.el now, but I'm not quite sure what's the
recommended thing to use.  I think I want to let my mail be split as I
normally do now, but have the non-classified mail be passed to
something that will say whether it's spam or not.  Which is provided
nicely by spam.el, but I wonder whether bogofilter or ifile is
bestest?

(And gnus-ifile.el isn't in the Gnus distribution.  It probably
should be, right?)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: spam*.el and ifile-gnus.el?

[Ted Zlatanov]

On Sun, 29 Dec 2002, larsi@gnus.org wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> This is (to quote the comment) a bit raw, but the idea is that
>> whatever is marked as spam when you exit a spam group should be
>> reported as spam.  Reporting may mean bogofilter processing, ifile
>> submission, or submission to a spam service.
> 
> But the submission thing hasn't been written yet, I guess?

Not AFAIK.  I would allow the user to pipe it through an external
submitter, because the spam reporting logic is very complex if done
right.

Of course, these days reporting spam is becoming less useful since
spammers have started forging the message headers that let users trace
the message to the origin.

>> Ideas and patches to spam.el are welcome, of course.
> 
> I want to use spam.el now, but I'm not quite sure what's the
> recommended thing to use.  I think I want to let my mail be split as
> I normally do now, but have the non-classified mail be passed to
> something that will say whether it's spam or not.  Which is provided
> nicely by spam.el, but I wonder whether bogofilter or ifile is
> bestest?

That decision is up to you - spam.el is just intended to be a way for
users to choose their preferred way of sorting spam from non-spam.

Bogofilter is decent, and Kai and others swear by ifile.

<editorial>

I don't like statistical spam filters, personally, because they are
easily fooled.  Legitimate messages can look like spam (for instance,
a report of spam that crashes Gnus); spammers can easily insert
legitimate text in the message to offset word and sentence statistics.

I know statistical analysis is fashionable now, but there's no way
short of designing a full AI that spam can be identified by content
analysis.  We call a message spam when it's unsolicited, commercial,
and annoying.  Content can only determine the commercial nature of a
message, and even that is somewhat doubtful.  There's lots of messages
that meet only 2 of those three criteria.  I wouldn't want to classify
those messages as spam.

</editorial>

I think ORDB and such are the best way to detect spam.  Unfortunately
blackholes don't work in spam.el because of a bug in the DNS code I
reported a while ago; you may want to take a look at that.  I don't
think it was ever fixed, but Simon Josefsson may have looked at it.

> (And gnus-ifile.el isn't in the Gnus distribution.  It probably
> should be, right?)

That's up to the gnus-ifile.el author, I think he wanted to wait until
the package works with more backends.  I'm coming out of a long
Christmas break, sorry if I'm forgetting something...  My brain is not
up to non-holiday speed yet :)

Ted

Re: spam*.el and ifile-gnus.el?

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think ORDB and such are the best way to detect spam.  Unfortunately
> blackholes don't work in spam.el because of a bug in the DNS code I
> reported a while ago; you may want to take a look at that.

If it was posted on this list, I'll probably reach it, er, this
week.  Perhaps.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen