`nnmail-split-fancy' regexp

`nnmail-split-fancy' regexp

[Petersen Jens-Ulrik (NRC/Tokyo)]

I found the restriction that string VALUE regexp's should match to
whole words too restrictive...


1999-06-24  Jens-Ulrik Petersen <jens-ulrik.petersen@nokia.com>

	* nnmail.el (nnmail-split-fancy): Update docstring to reflect
	following change: ie that a VALUE need no longer match complete
	words.
	(nnmail-split-it): Change regexp so that VALUE doesn't have to
	match a whole word.  This makes it possible to match for example
	<user@some.domain>.


--- nnmail.el~	Thu Jun 24 17:15:03 1999
+++ nnmail.el	Thu Jun 24 17:15:03 1999
@@ -306,8 +306,8 @@
   the buffer containing the message headers.  The return value FUNCTION
   should be a split, which is then recursively processed.
 
-FIELD must match a complete field name.  VALUE must match a complete
-word according to the `nnmail-split-fancy-syntax-table' syntax table.
+FIELD must match a complete field name.  VALUE is matched
+according to the `nnmail-split-fancy-syntax-table' syntax table.
 You can use \".*\" in the regexps to match partial field names or words.
 
 FIELD and VALUE can also be lisp symbols, in that case they are expanded
@@ -1147,11 +1147,11 @@
 			     (if (symbolp field)
 				 (cdr (assq field nnmail-split-abbrev-alist))
 			       field)
-			     "\\):.*\\)\\<\\("
+			     "\\):.*\\)\\("
 			     (if (symbolp value)
 				 (cdr (assq value nnmail-split-abbrev-alist))
 			       value)
-			     "\\)\\>")))
+			     "\\)")))
 	(push (cons split regexp) nnmail-split-cache)
 	;; Now that it's in the cache, just call nnmail-split-it again
 	;; on the same split, which will find it immediately in the cache.

Re: `nnmail-split-fancy' regexp

[Per Abrahamsen]

"Petersen Jens-Ulrik (NRC/Tokyo)" <jens-ulrik.petersen@nokia.com> writes:

> I found the restriction that string VALUE regexp's should match to
> whole words too restrictive...

There are zillions of nnmail-split-fancy rules written using that
assumption, and many of them represent a huge work effort.  Mine has
approximately 300 rules.  

If you want the rules to be interpreted differently, invent a new
variable and a new split method for it.

Re: `nnmail-split-fancy' regexp

[Petersen Jens-Ulrik (NRC/Tokyo)]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

   > "Petersen Jens-Ulrik (NRC/Tokyo)" <jens-ulrik.petersen@nokia.com> writes:
   > 
   > > I found the restriction that string VALUE regexp's should match to
   > > whole words too restrictive...
   > 
   > There are zillions of nnmail-split-fancy rules written using that
   > assumption, and many of them represent a huge work effort.  Mine has
   > approximately 300 rules.  

Ok, I take your point, and I suppose I suspected as much...

   > If you want the rules to be interpreted differently, invent a new
   > variable and a new split method for it.

Since it is such a small change, that sounds like overkill to me.

Couldn't we then have a variable (say
`nnmail-split-fancy-match-words') defaulting to t, giving the current
bahaviour, to control this?  If this is acceptable I happy to send in
a patch to implement it.

Jens

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

On Jun 24, 1999, Per Abrahamsen <abraham@dina.kvl.dk> wrote:

> "Petersen Jens-Ulrik (NRC/Tokyo)" <jens-ulrik.petersen@nokia.com> writes:
>> I found the restriction that string VALUE regexp's should match to
>> whole words too restrictive...

> There are zillions of nnmail-split-fancy rules written using that
> assumption, and many of them represent a huge work effort.  Mine has
> approximately 300 rules.  

> If you want the rules to be interpreted differently, invent a new
> variable and a new split method for it.

How about using a magic meaningless prefix to select non-full-word
matches, say, `[partial]'.  I doubt anyone starts their VALUE splits
with `[partial]', particularly because of the duplicate `a', which
wouldn't be totally useless in terms of regular expressions.  I've
just implemented it; patch attached.

While I was at it, I took the time to introduce a new feature that I
had requested for some weeks ago.  As a memory refresher, the problem
was that I subscribed to the "libtool" and "bug-libtool" mailing
lists, and there was no way to avoid that messages posted only to
"bug-libtool" were cross-posted to "libtool" without preventing
cross-posting when the message was *really* posted to both lists.  The 
syntax I've come up with is:

(FIELD VALUE [- RESTRICT [- RESTRICT [...]]] SPLIT)

So that now I can write:

(| (& (any "libtool@gnu\\.org" - "bug-libtool" "libtool")
      (any "bug-libtool@gnu\\.org" "bug-libtool")
      ;; ...
      )
   "misc")

The construction above is not equivalent to:

(| (& (any "libtool@gnu\\.org" (| (any "bug-libtool@gnu\\.org" nil)
                                  "libtool"))
      (any "bug-libtool@gnu\\.org" "bug-libtool") ;; ...

because the latter would not cross-post in case a message is posted to
both mailing lists, whereas the former will.  A RESTRICT clause will
only be considered if there is a match that starts after the end of
the FIELD match and ends after the beginning of the VALUE match.


And since I was rewriting that code anyway, I took the time to fix the
searching mechanism so that it looks for multiple matches of FIELD
VALUE, so that SPLITs that involve \N substitutions are properly
handled, causing cross-posting without introducing duplicates.


-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

[-- Attachment #2: partial&restrict.patch --]
[-- Type: application/x-patch, Size: 10195 bytes --]

Re: `nnmail-split-fancy' regexp

[Kai.Grossjohann]

Petersen Jens-Ulrik (NRC/Tokyo) <jens-ulrik.petersen@nokia.com> writes:

  > Couldn't we then have a variable (say
  > `nnmail-split-fancy-match-words') defaulting to t, giving the current
  > bahaviour, to control this?  If this is acceptable I happy to send in
  > a patch to implement it.

Since it is so easy to include ".*" in the regexes where you don't
want word matching, I'm somewhat reluctant to agree to this.

How many of your split rules actually need ".*"?

The documentation explicitly mentions ".*", btw.

kai
-- 
Life is hard and then you die.

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

On Jun 25, 1999, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:

> Petersen Jens-Ulrik (NRC/Tokyo) <jens-ulrik.petersen@nokia.com> writes:

>> Couldn't we then have a variable (say
>> `nnmail-split-fancy-match-words') defaulting to t, giving the current
>> bahaviour, to control this?  If this is acceptable I happy to send in
>> a patch to implement it.

> Since it is so easy to include ".*" in the regexes where you don't
> want word matching, I'm somewhat reluctant to agree to this.

It doesn't work in all cases.  Assume the (bug-)?libtool example I was
trying to work around.  I ended up trying to match
".*[< ]libtool@gnu\\.org", but it wouldn't work because it wouldn't
match the beginning of a word if the header line was just:
To: libtool@gnu.org

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

Re: `nnmail-split-fancy' regexp

[Kai.Grossjohann]

Alexandre Oliva <oliva@dcc.unicamp.br> writes:

  > [...] Assume the (bug-)?libtool example I was
  > trying to work around.  I ended up trying to match
  > ".*[< ]libtool@gnu\\.org", but it wouldn't work because it wouldn't
  > match the beginning of a word if the header line was just:
  > To: libtool@gnu.org

Well, would this work?  "\\(^\\|.*[< ,]\\)libtool@gnus\\.org"

I think you're using the wrong approach, though.  Most mailing lists
contain a header which uniquely idenfies the mails.  Often, the header
is X-Loop, but the Sender header can also often be used.

Hm.  How does one match "Sender: foo-owner@frob.org" exactly with
fancy splitting?  Maybe it's not so easy after all :-(

Yet another approach would be to split on the first match only, to
check for bug-libtool first, but to give two groups if this matches.
Lessee...

(| ("Sender" "bug-libtool-owner" (& "bug-libtool group" "libtool group"))
   ("Sender" "libtool-owner" "libtool group"))

[time passes]

Oh!  The header is "owner-libtool" rather than "libtool-owner", so it
is trivial to match "owner-libtool" but not "owner-bug-libtool".

Just my two cents.
kai
-- 
Life is hard and then you die.

Re: `nnmail-split-fancy' regexp

[Per Abrahamsen]

Alexandre Oliva <oliva@dcc.unicamp.br> writes:

> How about using a magic meaningless prefix to select non-full-word
> matches, say, `[partial]'.

How about using ".*" as the magic prefix instead?

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

On Jun 25, 1999, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:

> Well, would this work?  "\\(^\\|.*[< ,]\\)libtool@gnus\\.org"

Nope, it's not a beginning-of-line (or of string) for `^' to match.

> I think you're using the wrong approach, though.  Most mailing lists
> contain a header which uniquely idenfies the mails.

But there are messages that are posted to multiple mailing lists, and
I'd receive multiple copies of the message with different
Sender/X-loop/whatever headers.  If I relied on them, I'd have to
disable discarding of duplicates, which I don't want to do, or I'd
have to live without cross-posting.

Anyway, my particular problem doesn't matter much; the fact is that
there's no way to make sure you're not getting the implicit .* to
match a `-' you don't want it to match.  Although it is nice to have
full words by default, `-' is a word separator, so it doesn't always
get it right.

> Hm.  How does one match "Sender: foo-owner@frob.org" exactly with
> fancy splitting?  Maybe it's not so easy after all :-(

Precisely.

> (| ("Sender" "bug-libtool-owner" (& "bug-libtool group" "libtool group"))
>    ("Sender" "libtool-owner" "libtool group"))

Nope, I don't want cross-posting where it doesn't belong, i.e., if a
message was only posted to bug-libtool-owner, it shouldn't appear in
the libtool group.

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

Re: `nnmail-split-fancy' regexp

[Lars Magne Ingebrigtsen]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> How about using ".*" as the magic prefix instead?

Sounds like a good idea.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `nnmail-split-fancy' regexp

[Aaron M. Ucko]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Per Abrahamsen <abraham@dina.kvl.dk> writes:
> 
> > How about using ".*" as the magic prefix instead?
> 
> Sounds like a good idea.

Uhm, no.  As Alexandre Oliva observed a couple of days ago in
<orpv2kvakl.fsf@saci.lsd.dcc.unicamp.br>, the current syntax isn't
powerful enough to determine whether a header includes bar@baz,
foo-bar@baz, both, or neither.  As it stands, mail sent only to
foo-bar@baz gets treated as if it were also sent to bar@baz because
`-' is classified as a word separator.

-- 
Aaron M. Ucko, KB1CJC <amu@mit.edu> (finger amu@monk.mit.edu)

Re: `nnmail-split-fancy' regexp

[Per Abrahamsen]

amu@MIT.EDU (Aaron M. Ucko) writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Per Abrahamsen <abraham@dina.kvl.dk> writes:
> > 
> > > How about using ".*" as the magic prefix instead?
> > 
> > Sounds like a good idea.
> 
> Uhm, no.  As Alexandre Oliva observed a couple of days ago in
> <orpv2kvakl.fsf@saci.lsd.dcc.unicamp.br>, the current syntax isn't
> powerful enough to determine whether a header includes bar@baz,
> foo-bar@baz, both, or neither.  As it stands, mail sent only to
> foo-bar@baz gets treated as if it were also sent to bar@baz because
> `-' is classified as a word separator.

And how is this relevant for what the magic prefix should be?

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

On Jun 28, 1999, Per Abrahamsen <abraham@dina.kvl.dk> wrote:

> amu@MIT.EDU (Aaron M. Ucko) writes:
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> > Per Abrahamsen <abraham@dina.kvl.dk> writes:

>> > > How about using ".*" as the magic prefix instead?

>> > Sounds like a good idea.

>> Uhm, no.  As Alexandre Oliva observed a couple of days ago in

> And how is this relevant for what the magic prefix should be?

Ah, now I get it.  You're not suggesting that the feature is
unnecessary, you're just suggesting a different magic prefix.

Just a nit: in the case of \& substitutions, it isn't clear whether
the .* prefix is included in the match.  With the `[partial]'
implementation, it is not, which sounds reasonable, and probably
matches the users' expectations.  With `.*', it would not be
immediately clear whether `.*' is just a magic prefix or if it's
actually part of the regex.  Moreover, it would match everything up to 
the begining of the line, including the field name, which would break
the multi-match mechanism introduced in my last patch.  Let me try to
clarify.  Given a header line such as:

To: foo, bar

If we have rules for cross-posting that match both `foo' and `bar',
say `.*\(foo\|bar\)', `foo' will not be matched if `.*' is considered
part of the regexp, because the next backward-search would start just
before the previous match, i.e., just before the `To'.

So, even though `.*' appears to be much more intuitive, it is not
really that intuitive if you don't happen to know what's going on
behind the scenes, so I'd rather have a prefix that is clearly not
part of the regexp.  It might have been better to use a (cons 'partial
"regex") to denote partial matches...

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

Re: `nnmail-split-fancy' regexp

[Lars Magne Ingebrigtsen]

Alexandre Oliva <oliva@dcc.unicamp.br> writes:

> So, even though `.*' appears to be much more intuitive, it is not
> really that intuitive if you don't happen to know what's going on
> behind the scenes, so I'd rather have a prefix that is clearly not
> part of the regexp.  It might have been better to use a (cons 'partial
> "regex") to denote partial matches...

Yes, that sounds like a better idea than a "magic" prefix.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

On Jul  4, 1999, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> Alexandre Oliva <oliva@dcc.unicamp.br> writes:
>> So, even though `.*' appears to be much more intuitive, it is not
>> really that intuitive if you don't happen to know what's going on
>> behind the scenes, so I'd rather have a prefix that is clearly not
>> part of the regexp.  It might have been better to use a (cons 'partial
>> "regex") to denote partial matches...

> Yes, that sounds like a better idea than a "magic" prefix.

OTOH, it might break existing code that relies on the existing format.
In this sense, `.*' would be better.  Even if we have to special-case
it in the splitting engine and in the docs, it would be backward
compatible.  What do you think is more important?

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On Jul  4, 1999, Alexandre Oliva <oliva@dcc.unicamp.br> wrote:

> On Jul  4, 1999, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>> Alexandre Oliva <oliva@dcc.unicamp.br> writes:
>>> So, even though `.*' appears to be much more intuitive, it is not
>>> really that intuitive if you don't happen to know what's going on
>>> behind the scenes, so I'd rather have a prefix that is clearly not
>>> part of the regexp.  It might have been better to use a (cons 'partial
>>> "regex") to denote partial matches...

>> Yes, that sounds like a better idea than a "magic" prefix.

> OTOH, it might break existing code that relies on the existing format.
> In this sense, `.*' would be better.  Even if we have to special-case
> it in the splitting engine and in the docs, it would be backward
> compatible.

It seems to me that the attached patch is fully backward-compatible,
and its outcome is not surprising, not even in terms of \N
substitutions.

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{gnu.org,kaffe.org,{egcs,sourceware}.cygnus.com,samba.org}
*** E-mail about software projects will be forwarded to mailing lists

[-- Attachment #2: split-regex.patch --]
[-- Type: application/x-patch, Size: 905 bytes --]

Re: `nnmail-split-fancy' regexp

[Toby Speight]

Alexandre> Alexandre Oliva <URL:mailto:oliva@dcc.unicamp.br>

0> In <URL:news:org134ru9z.fsf@cupuacu.lsd.dcc.unicamp.br>, Alexandre
0> wrote:

Alexandre> OTOH, it might break existing code that relies on the
Alexandre> existing format.  In this sense, `.*' would be better.

Perhaps '*' might be best - a real regexp won't begin with '*', so
the software can distinguish it from true regexps for the traditional
splitter[1], and users won't be confused whether it's part of the
regexp for \& purposes.

[1] I have rules that require word boundaries at one end (like
    ".*@digitivity.com" to split work-related mail). I don't
    expect users with similar rules to have to change them (to
    ".*@digitivity.com\\>" in my case) because the magic prefix
    has changed Gnus's interpretation.

Re: `nnmail-split-fancy' regexp

[Alexandre Oliva]

On Jul  7, 1999, Toby Speight <Toby.Speight@digitivity.com> wrote:

Alexandre> OTOH, it might break existing code that relies on the
Alexandre> existing format.  In this sense, `.*' would be better.

> Perhaps '*' might be best - a real regexp won't begin with '*', so
> the software can distinguish it from true regexps for the traditional
> splitter[1], and users won't be confused whether it's part of the
> regexp for \& purposes.

I had found a way to arrange that `.*' was not surprising at all, and
the implementation has already been installed.  In particular, a `.*'
prefix won't change whether a trailing `\>' is added or not.

-- 
Alexandre Oliva http://www.dcc.unicamp.br/~oliva IC-Unicamp, Bra[sz]il
{oliva,Alexandre.Oliva}@dcc.unicamp.br  aoliva@{acm.org,computer.org}
oliva@{guarana.com,{gnu,kaffe,samba}.org,{egcs,sourceware}.cygnus.com}
*** E-mail about software projects will be forwarded to mailing lists