checking several headers when splitting mail

checking several headers when splitting mail

[lee]

Hi,

how do you check several headers when splitting mail? With neither the
normal splitting, nor fancy splitting I could figure out how to do that.

For an example, let´s say I get an email with headers like


From: somesender
Subject: this subject


Now I want to go all incoming mail from "somesender" with the subject
"this subject" into the group "mail.somesender-subject".

As an additional question, with fancy splitting, what´s a regexp that
matches a phone number like "+491234567" in the subject? I tried
"\\+491234567", and it won´t match.

Re: checking several headers when splitting mail

[Tassilo Horn]

lee <lee@yun.yagibdah.de> writes:

Hi!

[no clue about splitting, cause I do that on the server side using
Sieve...]

> As an additional question, with fancy splitting, what´s a regexp that
> matches a phone number like "+491234567" in the subject? I tried
> "\\+491234567", and it won´t match.

It does!

(string-match "\\+491234567" "Re: Call my via +491234567 now!") ;; C-x C-e
==> 16

Bye,
Tassilo
-- 
Sent from my Emacs

Re: checking several headers when splitting mail

[Andreas Schwab]

Tassilo Horn <tassilo@member.fsf.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> As an additional question, with fancy splitting, what´s a regexp that
>> matches a phone number like "+491234567" in the subject? I tried
>> "\\+491234567", and it won´t match.
>
> It does!

It doesn't.

> (string-match "\\+491234567" "Re: Call my via +491234567 now!") ;; C-x C-e
> ==> 16

(string-match "\\<\\+491234567\\>" "Re: Call my via +491234567 now!")

You either need to make ?+ a word character in
nnmail-split-fancy-syntax-table, or use ".*\\+491234567".

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: checking several headers when splitting mail

[lee]

Tassilo Horn <tassilo@member.fsf.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
> Hi!
>
> [no clue about splitting, cause I do that on the server side using
> Sieve...]
>
>> As an additional question, with fancy splitting, what´s a regexp that
>> matches a phone number like "+491234567" in the subject? I tried
>> "\\+491234567", and it won´t match.
>
> It does!
>
> (string-match "\\+491234567" "Re: Call my via +491234567 now!") ;; C-x C-e
> ==> 16

Yeah, it works with normal splitting and doesn´t with fancy splitting.

Re: checking several headers when splitting mail

[Lars Magne Ingebrigtsen]

lee <lee@yun.yagibdah.de> writes:

> how do you check several headers when splitting mail? With neither the
> normal splitting, nor fancy splitting I could figure out how to do that.

Use an `&' rule.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: checking several headers when splitting mail

[lee]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> how do you check several headers when splitting mail? With neither the
>> normal splitting, nor fancy splitting I could figure out how to do that.
>
> Use an `&' rule.

Thanks! The funny thing is that I just figured it out a few minutes ago
and was going to post the solution here. On entering this group I'm
seeing your answer :)

As I already suspected, an (& ...) rule is needed. What I didn't expect
is that it's surprisingly simple. I thought a SPLIT always has to have
three parts and found that it doesn't. To give an example:


(setq nnmail-split-fancy '(|
			   ;; some splitting rules can go here

			   (& (from
			       "root@yun\\.yagibdah\\.de"
			       "mail.sys.yun")
			      ("Subject"
			       "/var/log/.*"
			       "mail.sys.yun"))

			   ;; some more splitting rules can go here
			   )


Above rule checks the from headers *and* the Subject: header and splits
mail into the mail.sys.yun group only when both the from headers and the
Subject: header matches.

What looks strange to me is that I need to specify the group the mail
should go into for both the "from" and the "Subject" test. Perhaps
there's another way to do it?

Re: checking several headers when splitting mail

[Andreas Schwab]

lee <lee@yun.yagibdah.de> writes:

> To give an example:
>
>
> (setq nnmail-split-fancy '(|
> 			   ;; some splitting rules can go here
>
> 			   (& (from
> 			       "root@yun\\.yagibdah\\.de"
> 			       "mail.sys.yun")
> 			      ("Subject"
> 			       "/var/log/.*"
> 			       "mail.sys.yun"))
>
> 			   ;; some more splitting rules can go here
> 			   )
>
>
> Above rule checks the from headers *and* the Subject: header and splits
> mail into the mail.sys.yun group only when both the from headers and the
> Subject: header matches.

Except that it doesn't work like that:

(& SPLIT...): Process each SPLIT expression.

Thus the mail is stored into mail.sys.yun if From matches "root@..."
*or* Subject matches "/var/log/...".  If you want the mail to be
delivered only when *both* conditions match, you need to nest the
splits:

                            (from "root@yun\\.yagibdah\\.de"
			          ("Subject" "/var/log/.*"
			           "mail.sys.yun"))

(The | and & symbols should not be interpreted as logical operators, but
rather as sequencing operators.)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: checking several headers when splitting mail

[Lars Magne Ingebrigtsen]

lee <lee@yun.yagibdah.de> writes:

> (setq nnmail-split-fancy '(|
> 			   ;; some splitting rules can go here
>
> 			   (& (from
> 			       "root@yun\\.yagibdah\\.de"
> 			       "mail.sys.yun")
> 			      ("Subject"
> 			       "/var/log/.*"
> 			       "mail.sys.yun"))
>
> 			   ;; some more splitting rules can go here
> 			   )

[...]

> What looks strange to me is that I need to specify the group the mail
> should go into for both the "from" and the "Subject" test. Perhaps
> there's another way to do it?

Does the following work?

(& (from "root@yun\\.yagibdah\\.de")
   ("Subject" "/var/log/.*")
   "mail.sys.yun")

I haven't tried tweaking the fancy splitting stuff in a while, but I
seem to remember something like that should work.   

Or:

(& (from "root@yun\\.yagibdah\\.de" "")
   ("Subject" "/var/log/.*" "")
   "mail.sys.yun")

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: checking several headers when splitting mail

[lee]

Andreas Schwab <schwab@linux-m68k.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> To give an example:
>>
>>
>> (setq nnmail-split-fancy '(|
>> 			   ;; some splitting rules can go here
>>
>> 			   (& (from
>> 			       "root@yun\\.yagibdah\\.de"
>> 			       "mail.sys.yun")
>> 			      ("Subject"
>> 			       "/var/log/.*"
>> 			       "mail.sys.yun"))
>>
>> 			   ;; some more splitting rules can go here
>> 			   )
>>
>>
>> Above rule checks the from headers *and* the Subject: header and splits
>> mail into the mail.sys.yun group only when both the from headers and the
>> Subject: header matches.
>
> Except that it doesn't work like that:

Hm, you're right. I tested it again and found that everything sent from
root is put into mail.sys.yun, regardless of the subject.

> If you want the mail to be delivered only when *both* conditions
> match, you need to nest the splits:

Yes, that's what I want :)


>                             (from "root@yun\\.yagibdah\\.de"
> 			          ("Subject" "/var/log/.*"
> 			           "mail.sys.yun"))

This doesn't work. I wonder that it doesn't produce an error message. I
spent quite some time trying to understand the documentation about fancy
splitting, and obviously I failed. What actually is a SPLIT?

Re: checking several headers when splitting mail

[lee]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> (setq nnmail-split-fancy '(|
>> 			   ;; some splitting rules can go here
>>
>> 			   (& (from
>> 			       "root@yun\\.yagibdah\\.de"
>> 			       "mail.sys.yun")
>> 			      ("Subject"
>> 			       "/var/log/.*"
>> 			       "mail.sys.yun"))
>>
>> 			   ;; some more splitting rules can go here
>> 			   )
>
> [...]
>
>> What looks strange to me is that I need to specify the group the mail
>> should go into for both the "from" and the "Subject" test. Perhaps
>> there's another way to do it?
>
> Does the following work?
>
> (& (from "root@yun\\.yagibdah\\.de")
>    ("Subject" "/var/log/.*")
>    "mail.sys.yun")
>
> I haven't tried tweaking the fancy splitting stuff in a while, but I
> seem to remember something like that should work.   
>
> Or:
>
> (& (from "root@yun\\.yagibdah\\.de" "")
>    ("Subject" "/var/log/.*" "")
>    "mail.sys.yun")

Both of them seem to make otherwise unmatched mail go to mail.sys.yun.

What is the effect of an empty string at the place where I would usually
put the name of the destination group?

I even had a version that used nil instead of a group name, and it
didn't work. The documentation says that when a SPLIT is nil, the SPLIT
is ignored. What exactly does that mean?

I understand the (| thing like:

  (| (either one condition in this list matches) or "mail goes here")

... and processing stops at the first match found.

 
And (& is like:

  (& (any of the conditions in this list may match) or "mail goes here")

... and processing does not stop at the first match found.

This makes the (& thing seem primarily useful for splitting the same
message into several different groups.


What I don't understand is how it's possible to use the (& thing as:

  (& (any of the conditions in this list may match))

... and processing does not stop at the first match found.

That seems very irregular to me because you cannot use:

 (from "sender")

... or can you? And where does the mail go when none of the conditions
in the list match when the "or" part is missing?

The "(from "sender")" doesn't seem to be a SPLIT?? A SPLIT always seems
to have three parts? If you can use it, what does it mean/do?


Is it even /possible/ to use any of the above or a combination thereof
to require several headers to match before a message is split into a
destination group?

Re: checking several headers when splitting mail

[Andreas Schwab]

lee <lee@yun.yagibdah.de> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>>                             (from "root@yun\\.yagibdah\\.de"
>> 			          ("Subject" "/var/log/.*"
>> 			           "mail.sys.yun"))
>
> This doesn't work.

In which way does it not work?

> What actually is a SPLIT?

Any of the forms described in the doc string.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: checking several headers when splitting mail

[lee]

Andreas Schwab <schwab@linux-m68k.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>>                             (from "root@yun\\.yagibdah\\.de"
>>> 			          ("Subject" "/var/log/.*"
>>> 			           "mail.sys.yun"))
>>
>> This doesn't work.
>
> In which way does it not work?

It doesn't seem to have any effects. I expected it to yield an error
message, and to my surprise it didn't. When I press B q on a message
that supposedly would match, like

,----
| From: root <root@yun.yagibdah.de>
| Subject: /var/log/exim4/rejectlog
`----

... gnus says it'll go to mail.0-incoming. That's the group into which
all mail goes that isn't put somewhere else by a splitting rule it
matches.

>> What actually is a SPLIT?
>
> Any of the forms described in the doc string.

The documentation has an example which I find very confusing and says:

,---- [ 6.3.6 Fancy Mail Splitting ]
| A split is a (possibly) recursive structure where each split may
| contain other splits.  Here are the possible split syntaxes:
| 
| `group'
|      If the split is a string, that will be taken as a group name.
|      Normal regexp match expansion will be done.  See below for
|      examples.
|[...]
`----

The definition of a SPLIT is recursive: It only tells me that a SPLIT is
a structure which can be recursive and can contain other structures
which can be recursive and can contain other structures which can be
recursive and can contain other structures which can be recursive ---
and so on, until there's a stack overflow.

It doesn't tell me what a SPLIT is, other than that it is a structure. A
group name like "mail.sys.yun" is not a structure, so that cannot be a
SPLIT. Hence the list of possible split syntaxes produces syntax errors
when I read it ...

When I started trying to figure it out, after quite a while I put the
following comment about nnmail-split-fancy into my ~/.gnus file to get
an idea about what a SPLIT might be:

,----
| ;; order DOES matter
| ;; (| split split split ... GRUPPE)
| ;; (& split split split ... GRUPPE)
| ;; split = (HEADER REGEXP GRUPPE)
| ;; If the split is `nil', it is ignored.
`----

GRUPPE means the group the mail is put into. Perhaps I'm totally
mistaken since I don't understand the documentation. If I understand it
correctly, I think there is no way to check on several headers other
than to write my own function for it and call it in a splitting
rule. Try it :)

Re: checking several headers when splitting mail

[Ted Zlatanov]

On Wed, 15 Jun 2011 22:42:33 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> You either need to make ?+ a word character in
AS> nnmail-split-fancy-syntax-table, or use ".*\\+491234567".

Could you show an example?  It could go in the manual...  I don't know
how to do it, for instance :)

Thanks
Ted

Re: checking several headers when splitting mail

[Andreas Schwab]

lee <lee@yun.yagibdah.de> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> lee <lee@yun.yagibdah.de> writes:
>>
>>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>>
>>>>                             (from "root@yun\\.yagibdah\\.de"
>>>> 			          ("Subject" "/var/log/.*"
>>>> 			           "mail.sys.yun"))
>>>
>>> This doesn't work.
>>
>> In which way does it not work?
>
> It doesn't seem to have any effects. I expected it to yield an error
> message, and to my surprise it didn't. When I press B q on a message
> that supposedly would match, like
>
> ,----
> | From: root <root@yun.yagibdah.de>
> | Subject: /var/log/exim4/rejectlog
> `----
>
> ... gnus says it'll go to mail.0-incoming.

That's because "\\</var/log/.*" doesn't match "/var/log/exim4/rejectlog".

> A group name like "mail.sys.yun" is not a structure, so that cannot be
> a SPLIT.

The format of this variable is SPLIT, where SPLIT can be one of
the following:

GROUP: Mail will be stored in GROUP (a string).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: checking several headers when splitting mail

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 15 Jun 2011 22:42:33 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>
> AS> You either need to make ?+ a word character in
> AS> nnmail-split-fancy-syntax-table, or use ".*\\+491234567".
>
> Could you show an example?

Read the doc string of modify-syntax-entry.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: checking several headers when splitting mail

[Ted Zlatanov]

On Tue, 28 Jun 2011 20:04:58 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Wed, 15 Jun 2011 22:42:33 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>> 
AS> You either need to make ?+ a word character in
AS> nnmail-split-fancy-syntax-table, or use ".*\\+491234567".
>> 
>> Could you show an example?

AS> Read the doc string of modify-syntax-entry.

So, for those who are interested, it's useful to test fancy splits like
this:

#+begin_src lisp
(let ((mytable (copy-syntax-table nnmail-split-fancy-syntax-table)))
  (when (with-syntax-table mytable
          (string-match "\\</var/log" "/var/log"))
    (message "Matches without modification"))
  (modify-syntax-entry ?/ "w" mytable)
  (when (with-syntax-table mytable
          (string-match "\\</var/log" "/var/log"))
    (message "Matches with modification")))

#+end_src

Would that be useful in the manual as a demonstration of what a word
character will do?

Now for the next questions: is there any downside to modifying
`nnmail-split-fancy-syntax-table'?  And why does ".*\\+491234567" work?
What exactly are we looking at to match against the regular expression,
in other words, and why does that match when the regular expression is
".*\\+491234567" but not otherwise?

Thanks for your help, as always.
Ted

Re: checking several headers when splitting mail

[lee]

Andreas Schwab <schwab@linux-m68k.org> writes:

> lee <lee@yun.yagibdah.de> writes:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> lee <lee@yun.yagibdah.de> writes:
>>>
>>>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>>>
>>>>>                             (from "root@yun\\.yagibdah\\.de"
>>>>> 			          ("Subject" "/var/log/.*"
>>>>> 			           "mail.sys.yun"))
>>>>
>>>> This doesn't work.
>>>
>>> In which way does it not work?
>>
>> It doesn't seem to have any effects. I expected it to yield an error
>> message, and to my surprise it didn't. When I press B q on a message
>> that supposedly would match, like
>>
>> ,----
>> | From: root <root@yun.yagibdah.de>
>> | Subject: /var/log/exim4/rejectlog
>> `----
>>
>> ... gnus says it'll go to mail.0-incoming.
>
> That's because "\\</var/log/.*" doesn't match "/var/log/exim4/rejectlog".

Huh?

(from "root@yun\\.yagibdah\\.de" ("Subject" "var/log/.*" "mail.sys.yun"))

... seems to work, thanks :)

Why does "var/log/.*" match "/var/log/exim4/rejectlog" and "/var/log/.*"
does not?  How is one supposed to figure out what matches what when this
is so weird?

>> A group name like "mail.sys.yun" is not a structure, so that cannot be
>> a SPLIT.
>
> The format of this variable is SPLIT, where SPLIT can be one of
> the following:
>
> GROUP: Mail will be stored in GROUP (a string).

Hm.  Even if the documentation would just say so, it's still hard to
understand.  Perhaps I can make a suggestion about how to improve it
once I understand it better.

Re: checking several headers when splitting mail

[lee]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 28 Jun 2011 20:04:58 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>
> AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Wed, 15 Jun 2011 22:42:33 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>>> 
> AS> You either need to make ?+ a word character in
> AS> nnmail-split-fancy-syntax-table, or use ".*\\+491234567".
>>> 
>>> Could you show an example?
>
> AS> Read the doc string of modify-syntax-entry.
>
> So, for those who are interested, it's useful to test fancy splits like
> this:
>
> #+begin_src lisp
> (let ((mytable (copy-syntax-table nnmail-split-fancy-syntax-table)))
>   (when (with-syntax-table mytable
>           (string-match "\\</var/log" "/var/log"))
>     (message "Matches without modification"))
>   (modify-syntax-entry ?/ "w" mytable)
>   (when (with-syntax-table mytable
>           (string-match "\\</var/log" "/var/log"))
>     (message "Matches with modification")))
>
> #+end_src
>
> Would that be useful in the manual as a demonstration of what a word
> character will do?

Hm, I guess so.  Adding a reference to some documentation about what a
syntax-table is would be useful, too.

> Now for the next questions: is there any downside to modifying
> `nnmail-split-fancy-syntax-table'?  And why does ".*\\+491234567" work?
> What exactly are we looking at to match against the regular expression,
> in other words, and why does that match when the regular expression is
> ".*\\+491234567" but not otherwise?

And another question: My very first attempt to modify an elisp program
looks like this:


#+begin_src lisp
(let (
      (mytable (copy-syntax-table nnmail-split-fancy-syntax-table))
      (mstrg "/var/log"))
  (when (with-syntax-table mytable
	  (string-match (concat "\\<" mstrg) mstrg))
    (message "'%s' matches '%s'without modification" (concat "\\<" mstrg) mstrg))
  (modify-syntax-entry ?/ "w" mytable)
  (when (with-syntax-table mytable
	  (string-match (concat "\\<" mstrg) mstrg))
    (message "'%s' matches '%s' without modification" (concat "\\<" mstrg) mstrg)))
#+end_src


... and it gives different results for the matching than your code does.
Why?

My idea was to eventually make it a function that can be bound to a
key.  That could be useful when working on splitting rules.


Isn't there a way to make these splitting rules easier so that people
who don't speak elisp can understand how to create them?  Something like


if $h_From: matches root@localhost then
  if $h_Subject: matches "/var/log/.*" then
    save "mail.localhost.sys"
  endif
endif


... won't even need (much) explanation.  Exim does it like this in its
.forward files[1].


[1]: http://www.exim.org/exim-html-current/doc/html/spec_html/filter_ch03.html

Re: checking several headers when splitting mail

[lee]

lee <lee@yun.yagibdah.de> writes:

> #+begin_src lisp
> (let (
>       (mytable (copy-syntax-table nnmail-split-fancy-syntax-table))
>       (mstrg "/var/log"))
>   (when (with-syntax-table mytable
> 	  (string-match (concat "\\<" mstrg) mstrg))
>     (message "'%s' matches '%s'without modification" (concat "\\<" mstrg) mstrg))
>   (modify-syntax-entry ?/ "w" mytable)
>   (when (with-syntax-table mytable
> 	  (string-match (concat "\\<" mstrg) mstrg))
>     (message "'%s' matches '%s' without modification" (concat "\\<" mstrg) mstrg)))
> #+end_src
>
>
> ... and it gives different results for the matching than your code does.

Sorry, never mind, it does give the same results.

Feel free to include this into the documentation:


(defun my-gnus-fancy-splitting-match-test (mstrg)
  "Test string matching for creating fancy splitting rules."

  (interactive "MString to test matching on: " mstrg)
  (let ( (mytable (copy-syntax-table nnmail-split-fancy-syntax-table) )
	 (xstrg (concat "\\<" mstrg) )  )

    (if (with-syntax-table mytable
	  (string-match xstrg mstrg) )
	(message "'%s' matches '%s' WITHOUT modification" xstrg mstrg)
      ;; else
      (modify-syntax-entry ?/ "w" mytable)
	(if (with-syntax-table mytable
			(string-match xstrg mstrg) )
	    (message "'%s' matches '%s' WITH modification" xstrg mstrg)
	  ;; else
	  (message "'%s' doesn't match '%s'" xstrg mstrg) )  )   )    )


... but please verify that it's ok first.