fancy split trouble

fancy split trouble

[Eric Abrahamsen]

Hi,

I recently deleted most of my hard drive, and naturally stage one of
reconstruction is putting my gnus setup back together again. I've always
had trouble with fancy splitting, but now I can't get subject-line
regexp matching to work at all. It's very simple: I have two kinds of
messages I need to match. In one, the subject line looks like this:

[group name] Some subject line.

In this case, I've hoped that ("subject" "[group name" "mail.group")
would work, but it doesn't. I've also tried single escaping and double
escaping the square brackets, and adding ".*" on either side -- no luck.

The other is this:

Blah blah blah blah MARKED SPAM

Again, I thought ("subject" "MARKED SPAM" "mail.spam") would do it, or
maybe ".*MARKED SPAM", but neither does anything.

Just to make sure, re-evaluating nnmail-split-fancy on the fly should
work, right? It doesn't set the variable once at gnus startup, does it?

Thanks
Eri

Re: fancy split trouble

[Tassilo Horn]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

Hi Eric,

> [group name] Some subject line.
>
> In this case, I've hoped that ("subject" "[group name" "mail.group")
> would work, but it doesn't. I've also tried single escaping and double
> escaping the square brackets, and adding ".*" on either side -- no
> luck.

What's the complete value?  IMO, something like

  `(| ("subject" (rx "[group name]") "mail.group")
      ("subject" (rx "MARKED SPAM" eol) "mail.spam")
      ;; more splits...
      ;; ... and a default target
      "mail.misc")

should do the trick.

> Just to make sure, re-evaluating nnmail-split-fancy on the fly should
> work, right?

I think so.  And you split mail from `mail-sources', not nnimap, right?

Bye,
Tassilo

Re: fancy split trouble

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

Ups, too fast.  This

>   `(| ("subject" (rx "[group name]") "mail.group")
>       ("subject" (rx "MARKED SPAM" eol) "mail.spam")
>       ;; more splits...
>       ;; ... and a default target
>       "mail.misc")

should have been

  `(| ("subject" ,(rx "[group name]") "mail.group")
      ("subject" ,(rx "MARKED SPAM" eol) "mail.spam")
      ;; more splits...
      ;; ... and a default target
      "mail.misc")

i.e., the `rx' forms should be evaluated.  (`rx' is super-handy if you
aren't completely sure how things with special meaning in regexes have
to be quoted to match literally.)

Bye,
Tassilo

Re: fancy split trouble

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Tassilo Horn <tsdh@gnu.org> writes:
>
> Ups, too fast.  This
>
>>   `(| ("subject" (rx "[group name]") "mail.group")
>>       ("subject" (rx "MARKED SPAM" eol) "mail.spam")
>>       ;; more splits...
>>       ;; ... and a default target
>>       "mail.misc")
>
> should have been
>
>   `(| ("subject" ,(rx "[group name]") "mail.group")
>       ("subject" ,(rx "MARKED SPAM" eol) "mail.spam")
>       ;; more splits...
>       ;; ... and a default target
>       "mail.misc")
>
> i.e., the `rx' forms should be evaluated.  (`rx' is super-handy if you
> aren't completely sure how things with special meaning in regexes have
> to be quoted to match literally.)

Thanks! I didn't know about `rx', that was helpful.

I'm definitely using nnmail -- I use mpop to bring mail down into a
local maildir, and then a maildir source to pull into gnus.

Here's what I've got at present, it just doesn't work!

(setq nnmail-split-methods 'bbdb/gnus-split-method
      bbdb-split-gnus-nomatch-function 'nnmail-split-fancy)

(setq nnmail-split-fancy
      `(| (: gnus-registry-split-fancy-with-parent)
	  (| ("subject" ,(rx "MCLC") "mail.mclc")
	     ("subject" ,(rx "[literocracy]") "mail.literocracy")
	     ("subject" ,(rx "MARKED SPAM" eol) "mail.PRspam")
	     ("to" ,(rx "django-users") "mail.django"))
	  "mail.misc"))

Hope I'm doing something obviously wrong...

Re: fancy split trouble

[Andreas Schwab]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> (setq nnmail-split-fancy
>       `(| (: gnus-registry-split-fancy-with-parent)
> 	  (| ("subject" ,(rx "MCLC") "mail.mclc")
> 	     ("subject" ,(rx "[literocracy]") "mail.literocracy")
> 	     ("subject" ,(rx "MARKED SPAM" eol) "mail.PRspam")
> 	     ("to" ,(rx "django-users") "mail.django"))

Note that fancy splitting implicitly puts word boundaries around the
matches (see nnmail-split-fancy-match-partial-words).  Since ?\[ and ?\]
are not word characters by default, a word boundary cannot match around
them.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: fancy split trouble

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> (setq nnmail-split-fancy
>>       `(| (: gnus-registry-split-fancy-with-parent)
>> 	  (| ("subject" ,(rx "MCLC") "mail.mclc")
>> 	     ("subject" ,(rx "[literocracy]") "mail.literocracy")
>> 	     ("subject" ,(rx "MARKED SPAM" eol) "mail.PRspam")
>> 	     ("to" ,(rx "django-users") "mail.django"))
>
> Note that fancy splitting implicitly puts word boundaries around the
> matches (see nnmail-split-fancy-match-partial-words).  Since ?\[ and ?\]
> are not word characters by default, a word boundary cannot match around
> them.

Thanks to you both! For various reasons I ended up using the
invert-partial flag -- mostly it just seemed cleaner. Things are working
great now, apart from the fact that I still can't split on BBDB! Here's
what worked out (I had to evaluate the "junk" symbols):

(setq nnmail-split-fancy
      `(|
	("subject" ,(rx "MCLC") "mail.mclc")
	("subject" ,(rx "[literocracy]") "mail.literocracy" t)
	(any ,(rx "django-users") ,'junk)
	("subject" ,(rx "[sqlalchemy") ,'junk t)
	("subject" ,(rx "[Image-SIG") ,'junk t)
	("subject" ,(rx "[cherrypy-users") ,'junk t)
	("subject" ,(rx "[Javascript") ,'junk t)
	"mail.misc"))

Re: fancy split trouble

[Tassilo Horn]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Thanks to you both! For various reasons I ended up using the
> invert-partial flag -- mostly it just seemed cleaner.

Ah, yes, that's even better.  I didn't see it while glancing at the
docs.

> Things are working great now, apart from the fact that I still can't
> split on BBDB!

To debug that, I'd start with an entry

  (defun foo () (message "SPLITTING!") nil)

  `(| (: foo)
     ;; now the other splits
     )

just to verify that the BBDB splitting really delegates to your splits
if it couldn't find a match itself.  I guess it doesn't, so I'd start
edebugging bbdb/gnus-split-method.

Bye,
Tassilo

Re: fancy split trouble

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Thanks to you both! For various reasons I ended up using the
>> invert-partial flag -- mostly it just seemed cleaner.
>
> Ah, yes, that's even better.  I didn't see it while glancing at the
> docs.
>
>> Things are working great now, apart from the fact that I still can't
>> split on BBDB!
>
> To debug that, I'd start with an entry
>
>   (defun foo () (message "SPLITTING!") nil)
>
>   `(| (: foo)
>      ;; now the other splits
>      )
>
> just to verify that the BBDB splitting really delegates to your splits
> if it couldn't find a match itself.  I guess it doesn't, so I'd start
> edebugging bbdb/gnus-split-method.

Erm, I think I just hadn't gotten BBDB installed correctly. I've lost a
few years of customization (cruft!) and had forgotten how it had been.

But now I finally have a handle on fancy splitting -- thanks for that.

E

Re: fancy split trouble

[Andreas Schwab]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> (I had to evaluate the "junk" symbols):

No, you don't.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: fancy split trouble

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> (I had to evaluate the "junk" symbols):
>
> No, you don't.

Ah, you're right, I guess I was misinterpreted errors.

Re: fancy split trouble

[Ted Zlatanov]

On Tue, 16 Oct 2012 15:03:45 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> (setq nnmail-split-fancy
>> `(| (: gnus-registry-split-fancy-with-parent)
>> (| ("subject" ,(rx "MCLC") "mail.mclc")
>> ("subject" ,(rx "[literocracy]") "mail.literocracy")
>> ("subject" ,(rx "MARKED SPAM" eol) "mail.PRspam")
>> ("to" ,(rx "django-users") "mail.django"))

AS> Note that fancy splitting implicitly puts word boundaries around the
AS> matches (see nnmail-split-fancy-match-partial-words).  Since ?\[ and ?\]
AS> are not word characters by default, a word boundary cannot match around
AS> them.

Based on years of confusion, I think Gnus fancy splitting should use
simple regex matching without words and
`nnmail-split-fancy-match-partial-words' and so on.  If I say "x.*y" I
want it to match "xy" or "x and [y]".

I feel strongly the above should be the default.  It's what people
expect and it makes no sense to stop matching on word boundaries.

Can that be integrated into fancy splitting easily or do I need to set
global options?  And how many of you can I expect with pitchforks and
torches at my house if I start on a crusade to make this the Gnus
default?

Ted

Re: fancy split trouble

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I feel strongly the above should be the default.  It's what people
> expect and it makes no sense to stop matching on word boundaries.

Of course, this will break a lot of people's split patterns.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: fancy split trouble

[Ted Zlatanov]

On Sun, 23 Dec 2012 14:39:48 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I feel strongly the above should be the default.  It's what people
>> expect and it makes no sense to stop matching on word boundaries.

AS> Of course, this will break a lot of people's split patterns.

I'm willing to do this, especially if there's a global Gnus
`gnus-fancy-split-simplified' variable to control it, which is t in new
releases but missing in older releases.  Users simply set
`gnus-fancy-split-simplified' to nil and voila, they are back to the old
split behavior.

As I said, I'm basing my proposal on years of confusion about the
current behavior.  I think it will make Gnus better to bite the bullet
and make simple regexes the default.

Ted

Re: fancy split trouble

[Eric Abrahamsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 23 Dec 2012 14:39:48 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>
> AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> I feel strongly the above should be the default.  It's what people
>>> expect and it makes no sense to stop matching on word boundaries.
>
> AS> Of course, this will break a lot of people's split patterns.
>
> I'm willing to do this, especially if there's a global Gnus
> `gnus-fancy-split-simplified' variable to control it, which is t in new
> releases but missing in older releases.  Users simply set
> `gnus-fancy-split-simplified' to nil and voila, they are back to the old
> split behavior.
>
> As I said, I'm basing my proposal on years of confusion about the
> current behavior.  I think it will make Gnus better to bite the bullet
> and make simple regexes the default.

But I was so pleased to finally (after three years) figure out how it
works!

But yes, there's enough that's confusing about gnus, I vote for the
change. And as prominent a notification as possible.

Gnus should have a permanently-connected "self" nntp server that it can
use to send messages to the user. "Hey, you need to adjust your split
regexps!" Or, "Hey, there's a new XXX variable!" Or, "Hey, I deleted all
your archived messages!"

Mostly joking...

nnvolatile newsgroup (was: fancy split trouble)

[Ted Zlatanov]

On Mon, 24 Dec 2012 10:22:41 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> Gnus should have a permanently-connected "self" nntp server that it can
EA> use to send messages to the user. "Hey, you need to adjust your split
EA> regexps!" Or, "Hey, there's a new XXX variable!" Or, "Hey, I deleted all
EA> your archived messages!"

That's a really good idea.

We already have this with the nndraft backend.  Gnus could have a
built-in backend like nndraft that shows errors and warnings generated
on startup (so the contents disappear when Gnus is shut down, and
therefore the backend can store everything directly in memory).

Call it nnvolatile.  Does this exist already?

I realize this will annoy expert users, but I think it would be very
nice for everyone else, and it's the right place to put these messages
so they don't interfere with the user's Gnus experience.

Ted

Re: fancy split trouble

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Based on years of confusion, I think Gnus fancy splitting should use
> simple regex matching without words and
> `nnmail-split-fancy-match-partial-words' and so on.  If I say "x.*y" I
> want it to match "xy" or "x and [y]".

Yeah, probably.  But it'll break all existing fancy splitting users.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: fancy split trouble

[Ted Zlatanov]

On Tue, 25 Dec 2012 12:55:38 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Based on years of confusion, I think Gnus fancy splitting should use
>> simple regex matching without words and
>> `nnmail-split-fancy-match-partial-words' and so on.  If I say "x.*y" I
>> want it to match "xy" or "x and [y]".

LI> Yeah, probably.  But it'll break all existing fancy splitting users.

Oooh they think they're so fancy!  We'll show 'em!

Are you planning a release soon?  This seems like a change that should
not happen right before a major Gnus release.

Ted

Re: fancy split trouble

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LI> Yeah, probably.  But it'll break all existing fancy splitting users.
>
> Oooh they think they're so fancy!  We'll show 'em!

They should all get off our lawns!  That's what I think!
Whippersnappers!

> Are you planning a release soon?  This seems like a change that should
> not happen right before a major Gnus release.

The `unexist' nnimap problem has to be dealt with first.  It's been
kinda broken since well it was written, and I just can't reproduce the
problems, but it's a pretty major problem...

I thought I had a handle on it last week, but then it went away, and I
couldn't reproduce it any more.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: fancy split trouble

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Tassilo Horn <tsdh@gnu.org> writes:
>

Whoops, looks like it's BBDB that's done it. If I set
`nnmail-split-methods' directly to `nnmail-split-fancy', it works as
advertised. I'm using development versions of both Gnus and BBDB, let me
see if I've done something wrong here...

In the meantime, given the back-quoted form of `nnmail-split-fancy', do
I need to do something special to get mail junking (ie deletion) working
correctly? This leaves "[Javascript]" messages in the misc directory:

(setq nnmail-split-fancy
      `(|
	("subject" ,(rx "MCLC") "mail.mclc")
	("subject" ,(rx "[literocracy]") "mail.literocracy")
	("subject" "MARKED SPAM" "mail.PRspam")
	("to" ,(rx "django-users") "mail.django")
	("subject" ,(rx "[Javascript]") 'junk)
	"mail.misc"))

Re: fancy split trouble

[Tassilo Horn]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> In the meantime, given the back-quoted form of `nnmail-split-fancy',
> do I need to do something special to get mail junking (ie deletion)
> working correctly? This leaves "[Javascript]" messages in the misc
> directory:
>
> (setq nnmail-split-fancy
>       `(|
> 	("subject" ,(rx "MCLC") "mail.mclc")
> 	("subject" ,(rx "[literocracy]") "mail.literocracy")
> 	("subject" "MARKED SPAM" "mail.PRspam")
> 	("to" ,(rx "django-users") "mail.django")
> 	("subject" ,(rx "[Javascript]") 'junk)
> 	"mail.misc"))

See Andreas answer.  You need to set

  (setq nnmail-split-fancy-match-partial-words t)

to match for things like "[Javascript]".  Then, you'd probably want to
activate word boundary matches for the other splits manually, e.g., (rx
bow "MCLC" eow) instead of (rx "MCLC") which would also match
"fooMCLCbar" [ok, that's rather uncommon].

Bye,
Tassilo