Fancy Mail Splitting documentation needs improved?

Fancy Mail Splitting documentation needs improved?

[Tommy Kelly]

I'm *really* struggling to understand "6.4.6 Fancy Mail
Splitting". In this situation there's always the danger that the reader
is at fault -- maybe there are some prerequisites they don't have. But I
don't believe that's the case here. I know *very* minimal lisp, but I
don't think that's what's holding me back. Besides, should someone
really need to be a lisp programmer to split their mail?

Here is the beginning of a rewrite. It'll be full of errors but that is,
I claim, the fault of the current doc. I could have asked many
individual questions to deal with the various uncertainties I have, but
maybe this'll catch a bunch of them in one go. If folk can help me
correct my errors, I think it'll help improve the whole thing.

Also, if I'm stepping on toes here, apologies. Happy to retire to my
cave if this is someone else's cookie.


##### START OF POSSIBLE RE-WRITE


6.4.6 Fancy Mail Splitting
==========================

If the rather simple, standard method for specifying how to split mail
doesn't allow you to do what you want, you can set
`nnmail-split-methods' to `nnmail-split-fancy'.  This allows more
sophisticated control, including calls to full-blown lisp functions of
your own, of how your incoming email gets split.

From a high level view, fancy splitting may be fancy, but it's not
really that complicated. Basically, we set the nnmail-split-fancy
variable to contain of a set of one or more email matching rules --
we'll call them "splits" from here on, 'cos we're using them to split
email, and 'cos it's shorter. Each split is, again basically, simply some
matching criteria and a corresponding Gnus group name. Each incoming
email is tested against the splits and is then placed in (or "split
into") the group, or groups, according to how it matches.

See, simple. But it is in the details that we'll find the power, and
devilish complexity, of fancy splitting. There are three scarily cool
things that may make it worth your while to have meandered here from the
world of simple splitting back in <ref to simple splitting>.

1. A fancy split is, in general, a recursive structure. You can have
splits within splits within splits. That's very useful. And
fancy. You'll see. 

2. Whether splits are invoked in parallel (e.g. to allow crossposting)
or in sequence is under your control

3. Splits can invoke lisp functions of your own construction

Those three things together mean you can build splitting systems of
immense complexity, and generally ruin your life. However, experience
has shown that unless you already know how fancy splitting works (in
which case move along, why are you here? -- you probably want to jump
straight to <ref to details below>), we should start off gently with a
simple example.  

6.4.6.1
-------
Perhaps the easiest to understand form of split is a lisp list
containing three components as follows:     

(FIELD VALUE GROUP)

For example:

(subject  "gnus"  "mail-about-gnus")

This split would take all mails having "gnus" in the subject and place
them into the group "mail-about-gnus". Although this is an example of a
valid split, it would not be appropriate to set your nnmail-split-fancy
variable to that alone. The reason is that it does not tell Gnus what to
do with mails that do *not* match that criterion. So, to set up that
split in your .gnus file you would do something like this:

(setq nnmail-split-fancy '(| (subject  "gnus"  "mail-about-gnus")
                             "INBOX")

Here, our simple split example is given as the second item in a more
complex split -- i.e. showing the recursion mentioned earlier. There are
three components to the more complex split: 

| - this says we will treat all of the remaining items in the list as
    splits but that we'll process them in order, and that we'll stop
    whenever we get a match. 

(subject  "gnus"  "mail-about-gnus") - our original simple split. As
    described, it takes mails with "gnus" in the subject line and puts
    them into the "mail-about-gnus" group

"INBOX" - a catchall, used if all previous splits fail to match. In
    other words, all mails that do not match the "mail-about-gnus" split
    will be placed in the group, "INBOX". 

So the overall effect of that split is to place emails with "gnus" in
the subject into the "mail-about-gnus" group, and everything else into
the "INBOX" group. Simple.

6.4.6.2
-------
As we've just seen, the "mail-about-gnus" split is a simple version of
a more general, possibly recursive, very fancy structure. In general, a
split is a lisp list taking one of four forms (note that the example
from 6.4.6.1 is a simple example of the first of the four forms). The
forms are:

(FIELD VALUE [- RESTRICT [...] ] SPLIT [INVERT-PARTIAL])

OR

(<operator> SPLIT1 [SPLIT2 [SPLIT3 [...]]])

   where <operator> is one of: |, & or :

OR

"<group-name>"

OR

junk

<There would then follow a description of each of those four types, with
examples of each ... before finally ending up with the big original
example, with more detailed commenting and commentary>

##### END OF POSSIBLE RE-WRITE

OK, I'm stopping here for comments from y'all. Note that I'm thinking
aloud in the above. For example, the idea that there are four 
forms of split could probably be simplified into a single syntax
covering all forms. But I haven't figure out what that looks like yet.

Comments *very* welcome.

Tommy

Re: Fancy Mail Splitting documentation needs improved?

[Kan-Ru Chen]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

Tommy Kelly <tommy.kelly@verilab.com> writes:

> OK, I'm stopping here for comments from y'all. Note that I'm thinking
> aloud in the above. For example, the idea that there are four 
> forms of split could probably be simplified into a single syntax
> covering all forms. But I haven't figure out what that looks like yet.

Great idea and rewrite!  I was confused at first when reading the
fancy split method, as I thought the | and & operations was logical
operations, but instead they are group operations. Each SPLIT will be
processed (&) or until one matched (|).

I am implementing the logical operators, so the split rule will look
like sieve rule:

--8<---------------cut here---------------start------------->8---
(setq nnmail-split-fancy
      `(| (or (("x-bogosity" "spam" t)
               ("x-spam-status" "yes" t)
               ("list-id" ".*yahoogroups.com" t)
               (to "undisclosed.*recipients" t))
              "mail.spam")
          (or (("list-id" "awesome" t)
               ("list-id" "notmuch" t)
               ("list-id" "madbutterfly" t))
              "mail.list")
          ,(format-time-string "mail.misc-%Y")))
--8<---------------cut here---------------end--------------->8---

The logical operator syntax is:

--8<---------------cut here---------------start------------->8---
(<OP> (<CLAUSE 1> <CLAUSE 2> ...)
      SPLIT)
--8<---------------cut here---------------end--------------->8---

Each clause is a field split that returns non-nil SPLIT, if the OP of
the clauses was true, then split to SPLIT.

The syntax is not finalized yet, but already fulfilled my needs.  I'd
like to drop the third argument of field split, so that the clauses
looks more clear.  I need your comments!

- Kanru

P.S. Draft patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fancysplit.patch --]
[-- Type: text/x-diff, Size: 1443 bytes --]

diff --git a/lisp/nnmail.el b/lisp/nnmail.el
index c86e96b..00de781 100644
--- a/lisp/nnmail.el
+++ b/lisp/nnmail.el
@@ -1389,6 +1389,10 @@ See the documentation for the variable `nnmail-split-fancy' for details."
      ((null split)
       nil)
 
+     ;; t split
+     ((eq split t)
+      (list t))
+
      ;; A group name.  Do the \& and \N subs into the string.
      ((stringp split)
       (when nnmail-split-tracing
@@ -1401,6 +1405,28 @@ See the documentation for the variable `nnmail-split-fancy' for details."
 	(push "junk" nnmail-split-trace))
       (list 'junk))
 
+     ;; Logical OR operation.
+     ((eq (car split) 'or)
+      (let (done
+            (clauses (cadr split))
+            (split (cddr split)))
+        (while (and (not done) (car clauses))
+          (setq done (car (nnmail-split-it (car clauses)))
+                clauses (cdr clauses)))
+        (when done
+          (nnmail-split-it (car split)))))
+
+     ;; Logical AND operation.
+     ((eq (car split) 'and)
+      (let ((done t)
+            (clauses (cadr split))
+            (split (cddr split)))
+        (while (and done (car clauses))
+          (setq done (car (nnmail-split-it (car clauses)))
+                clauses (cdr clauses)))
+        (when (and done (null clauses))
+          (nnmail-split-it (car split)))))
+
      ;; Builtin & operation.
      ((eq (car split) '&)
       (apply 'nconc (mapcar 'nnmail-split-it (cdr split))))

Re: Fancy Mail Splitting documentation needs improved?

[Lars Magne Ingebrigtsen]

Kan-Ru Chen <kanru@kanru.info> writes:

> Great idea and rewrite!  I was confused at first when reading the
> fancy split method, as I thought the | and & operations was logical
> operations, but instead they are group operations. Each SPLIT will be
> processed (&) or until one matched (|).

Well...  er...  what's the difference?  `|' and `&' are logical
operations, and they short-circuit as expected.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Fancy Mail Splitting documentation needs improved?

[Kan-Ru Chen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kan-Ru Chen <kanru@kanru.info> writes:
>
>> Great idea and rewrite!  I was confused at first when reading the
>> fancy split method, as I thought the | and & operations was logical
>> operations, but instead they are group operations. Each SPLIT will be
>> processed (&) or until one matched (|).
>
> Well...  er...  what's the difference?  `|' and `&' are logical
> operations, and they short-circuit as expected.

OK, the difference is that with `|' and `&' you have to specify the
target SPLIT for every rule.  What in my mind is something like
programmable rules.  Ideally it would look like:

--8<---------------cut here---------------start------------->8---
(if (or (field-match "list-id" "...")
        (field-match "list-id" "..."))
    (split "mail.list"))
--8<---------------cut here---------------end--------------->8---

Of course this could be done via the `:' operator, but will loose the
benefits from field cache.  What to do? I'm pondering...

- Kanru

Re: Fancy Mail Splitting documentation needs improved?

[Lars Magne Ingebrigtsen]

Kan-Ru Chen <kanru@kanru.info> writes:

>>> Great idea and rewrite!  I was confused at first when reading the
>>> fancy split method, as I thought the | and & operations was logical
>>> operations, but instead they are group operations. Each SPLIT will be
>>> processed (&) or until one matched (|).
>>
>> Well...  er...  what's the difference?  `|' and `&' are logical
>> operations, and they short-circuit as expected.
>
> OK, the difference is that with `|' and `&' you have to specify the
> target SPLIT for every rule.  What in my mind is something like
> programmable rules.  Ideally it would look like:
> (if (or (field-match "list-id" "...")
>         (field-match "list-id" "..."))
>     (split "mail.list"))
> Of course this could be done via the `:' operator, but will loose the
> benefits from field cache.  What to do? I'm pondering...

I'm probably misunderstanding something here, but `|' and `&' in the
splitting rules (should) work just the same as `or' and `and' in Lisp:

(or 'foo 'bar 'zot)
=> foo

(and nil nil 'foo)
=> nil

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Fancy Mail Splitting documentation needs improved?

[Kan-Ru Chen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kan-Ru Chen <kanru@kanru.info> writes:
>>
>> OK, the difference is that with `|' and `&' you have to specify the
>> target SPLIT for every rule.  What in my mind is something like
>> programmable rules.  Ideally it would look like:
>> (if (or (field-match "list-id" "...")
>>         (field-match "list-id" "..."))
>>     (split "mail.list"))
>> Of course this could be done via the `:' operator, but will loose the
>> benefits from field cache.  What to do? I'm pondering...
>
> I'm probably misunderstanding something here, but `|' and `&' in the
> splitting rules (should) work just the same as `or' and `and' in Lisp:
>
> (or 'foo 'bar 'zot)
> => foo
>
> (and nil nil 'foo)
> => nil

So something that analog to `cond' would be very useful.  Multiple
condition that leads to one destination.

- Kanru

Re: Fancy Mail Splitting documentation needs improved?

[Andreas Schwab]

Kan-Ru Chen <kanru@kanru.info> writes:

> So something that analog to `cond' would be very useful.  Multiple
> condition that leads to one destination.

That's what (| ...) does.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fancy Mail Splitting documentation needs improved?

[Kan-Ru Chen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Kan-Ru Chen <kanru@kanru.info> writes:
>
>> So something that analog to `cond' would be very useful.  Multiple
>> condition that leads to one destination.
>
> That's what (| ...) does.
>
> Andreas.

With current `&' and '|' operator you cannot split a mail to a
destination depends on multiple condition.

OK. Maybe we can, by specify the SPLIT target of a FIELD as nil:

 (& ("subject" "gnus" nil)
    (from "lars" nil)
    "guns.lars")

But how do I say I want to spilt the mail to "mail.spam" if its
"x-bogosity" contains "spam" *or* its "x-spam-status" contains "yes".

Of course I can write

 (| ("x-bogosity" "spam" "mail.spam")
    ("x-spam-status" "yes" "mail.spam")
    "mail.misc")

See, for every FIELD I have to duplicate the SPLIT, "mail.spam" in this
case.

- Kanru
-- 
A badly written book is only a blunder. A bad translation of a good
book is a crime.
                -- Gilbert Highet

Re: Fancy Mail Splitting documentation needs improved?

[Lars Magne Ingebrigtsen]

Kan-Ru Chen <kanru@kanru.info> writes:

> Of course I can write
>
>  (| ("x-bogosity" "spam" "mail.spam")
>     ("x-spam-status" "yes" "mail.spam")
>     "mail.misc")
>
> See, for every FIELD I have to duplicate the SPLIT, "mail.spam" in this
> case.

Yes, I think that's the case.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Fancy Mail Splitting documentation needs improved?

[Lars Magne Ingebrigtsen]

Tommy Kelly <tommy.kelly@verilab.com> writes:

> I'm *really* struggling to understand "6.4.6 Fancy Mail
> Splitting".

Yes, it's written kinda computer sciencey.  That is, incomprehensible. 

> ##### START OF POSSIBLE RE-WRITE

Could you send a patch?  And do you have FSF copyright assignment papers
on file?

> | - this says we will treat all of the remaining items in the list as
>     splits but that we'll process them in order, and that we'll stop
>     whenever we get a match. 

Yes...  it's the traditional `or' operator.

> OK, I'm stopping here for comments from y'all. Note that I'm thinking
> aloud in the above. For example, the idea that there are four 
> forms of split could probably be simplified into a single syntax
> covering all forms. But I haven't figure out what that looks like yet.
>
> Comments *very* welcome.

Looks good to me.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Fancy Mail Splitting documentation needs improved?

[Tommy Kelly]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Tommy Kelly <tommy.kelly@verilab.com> writes:
>
>> I'm *really* struggling to understand "6.4.6 Fancy Mail
>> Splitting".
>
> Yes, it's written kinda computer sciencey.  That is, incomprehensible. 

I have a <cough>PhD in CompSci. I don't think that's the problem ;-)

>> ##### START OF POSSIBLE RE-WRITE
>
> Could you send a patch?  And do you have FSF copyright assignment papers
> on file?

I'll do both once I've got a finished version, yes? 
(And so now I need to figure out how to do the patch thing).

Tommy

Re: Fancy Mail Splitting documentation needs improved?

[Lars Magne Ingebrigtsen]

Tommy Kelly <tommy.kelly@verilab.com> writes:

> I'll do both once I've got a finished version, yes? 

Sure.

> (And so now I need to figure out how to do the patch thing).

Assuming you're working from the git version, `C-x v ='.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen