splitting working now : some issues/questions

splitting working now : some issues/questions

[Richard Riley]

My splitting is now working. Before I ask Qs, heres the essense of my
setup now.

  
,----
|   (setq spam-use-bogofilter t)
|   (setq spam-use-blackholes t)
|   
|   (require 'spam)  ; ** (1)
|   (spam-initialize)
|   
|   (setq nnimap-inbox "INBOX")
|   (setq nnimap-split-methods 'default)
|   (setq nnimap-split-download-body t)
|   
|   (setq nnmail-split-fancy '(|(: spam-split)"mail")) ; ** (2)
|   (setq nnmail-split-methods 'nnmail-split-fancy)
|   
|   (setq spam-autodetect-recheck-messages nil)
|   (setq spam-mark-ham-unread-before-move-from-spam-group t)
|   (setq spam-mark-new-messages-in-spam-group-as-spam t)
|   (setq spam-split-group "Gnus-Spam")
|   
|   (setq spam-split-disabled nil)
|   
|   (setq gnus-select-method '(nnml ""
|                                   (nnmail-split-methods nil)
|                                   ))
|   
|   (setq gnus-secondary-select-methods nil)
|   
|   (add-to-list 'gnus-secondary-select-methods
|                '(nnimap "riley"
|                         (nnir-search-engine imap) ; ** (3)
|                         (nnimap-address "offlineimap")
|                         ))
`----

I have numbered the 3 points I have Qs and comments about. 

(1) : mea culpa. because of a reorg of my startup files gnus was being
initialized before these two lines. There was no obvious error however
and bogofilter filtering still occurred. But often fell over with no
obvious reasons. Possibly something could be added to warn the user
inside spam.el? Certainly loading gnus after this removed a lot of
problems.

(2) This was a major issue. I dont or didnt really want to move my mail
from INBOX (which maps to my gmail inbox). However setting the final
clause to "INBOX" saw ALL mail processed each and very time as if it was
unread. Possibly something else triggered this but I couldnt find the
setting. Now mail is processed and moved in to "mail" there are no
problems since INBOX only contains NEW mail now. This is different than
how it worked for me in normal Gnus,

(3) Its unclear to me (a novice elisper) when one must pass values as
part of the select. Using things like the nnimap variables as an example
I had assumed this line was unnecessary if I had earlier something like

(setq nnir-search-engine 'imap)

However nnir was not enabled for the select method. Could someone please
explain this.

But the bottom line is that it now works. I *think* something has
changed with regard to it always processing all messages in the INBOX -
it didnt do it before for me but I am happy enough with the stop gap
measure of placing all processed mails into "mail" for the time being
where it is not processed further.

Powerful but heady stuff.

Re: splitting working now : some issues/questions

[Lars Magne Ingebrigtsen]

Richard Riley <rileyrg@googlemail.com> writes:

> But the bottom line is that it now works. I *think* something has
> changed with regard to it always processing all messages in the INBOX -
> it didnt do it before for me but I am happy enough with the stop gap
> measure of placing all processed mails into "mail" for the time being
> where it is not processed further.

Yes, nnimap goes through all the mail in the INBOX now -- even the stuff
it's seen before.  I'm not sure whether this is a feature or a bug...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: splitting working now : some issues/questions

[Richard Riley]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Richard Riley <rileyrg@googlemail.com> writes:
>
>> But the bottom line is that it now works. I *think* something has
>> changed with regard to it always processing all messages in the INBOX -
>> it didnt do it before for me but I am happy enough with the stop gap
>> measure of placing all processed mails into "mail" for the time being
>> where it is not processed further.
>
> Yes, nnimap goes through all the mail in the INBOX now -- even the stuff
> it's seen before.  I'm not sure whether this is a feature or a bug...

Its nice in that when reading inbox mails in other clients such as k9 on
a mobile then gnus will still split it. Its not so nice in that its new
behaviour and many people will have thousands of emails in their INBOX
(and why I spent hours scratching my head wondering why it kept hiding
all my mail ;)

If possible I think it should have a setting and default to the old
behaviour.

Re: splitting working now : some issues/questions

[Ted Zlatanov]

On Fri, 08 Oct 2010 16:46:54 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

RR> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Richard Riley <rileyrg@googlemail.com> writes:
>> 
>>> But the bottom line is that it now works. I *think* something has
>>> changed with regard to it always processing all messages in the INBOX -
>>> it didnt do it before for me but I am happy enough with the stop gap
>>> measure of placing all processed mails into "mail" for the time being
>>> where it is not processed further.
>> 
>> Yes, nnimap goes through all the mail in the INBOX now -- even the stuff
>> it's seen before.  I'm not sure whether this is a feature or a bug...

RR> Its nice in that when reading inbox mails in other clients such as k9 on
RR> a mobile then gnus will still split it. Its not so nice in that its new
RR> behaviour and many people will have thousands of emails in their INBOX
RR> (and why I spent hours scratching my head wondering why it kept hiding
RR> all my mail ;)

RR> If possible I think it should have a setting and default to the old
RR> behaviour.

nnimap should default to 'unseen and allow 'unread as the selection
criteria for splitting.

Ted

Re: splitting working now : some issues/questions

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> nnimap should default to 'unseen and allow 'unread as the selection
> criteria for splitting.

Actually, looking over the code, it's trying to avoid all messages
marked with \Seen when splitting.  But it didn't actually work because
of a marks interface change.  This has now been fixed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: splitting working now : some issues/questions

[Richard Riley]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> nnimap should default to 'unseen and allow 'unread as the selection
>> criteria for splitting.
>
> Actually, looking over the code, it's trying to avoid all messages
> marked with \Seen when splitting.  But it didn't actually work because
> of a marks interface change.  This has now been fixed.

Oh super ;) I can move all my emails back into INBOX...

I'd assumed from your comments earlier when I raised this that this was
a feature and wasn't to be changed.

Re: splitting working now : some issues/questions

[Richard Riley]

Richard Riley <rileyrg@googlemail.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>
>>> nnimap should default to 'unseen and allow 'unread as the selection
>>> criteria for splitting.
>>
>> Actually, looking over the code, it's trying to avoid all messages
>> marked with \Seen when splitting.  But it didn't actually work because
>> of a marks interface change.  This has now been fixed.
>
> Oh super ;) I can move all my emails back into INBOX...
>
> I'd assumed from your comments earlier when I raised this that this was
> a feature and wasn't to be changed.

Could (or should) the group parameter spam destination please default to
the groups server? Currently if I set the spam dest as an unqualified
Gnus-Spam for my nnimap server then its going to nnml:Gnus-Spam. I need
to currently fully qualify it to e.g nnimap+riley:Gnus-Spam.

cheers

r.

Re: splitting working now : some issues/questions

[Ted Zlatanov]

On Sun, 10 Oct 2010 03:09:55 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

RR> Could (or should) the group parameter spam destination please default to
RR> the groups server? Currently if I set the spam dest as an unqualified
RR> Gnus-Spam for my nnimap server then its going to nnml:Gnus-Spam. I need
RR> to currently fully qualify it to e.g nnimap+riley:Gnus-Spam.

I think it should behave like a regular group name, which is the
behaviour you're noting.

Ted

Re: splitting working now : some issues/questions

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 10 Oct 2010 03:09:55 +0200 Richard Riley <rileyrg@googlemail.com> wrote:
>
> RR> Could (or should) the group parameter spam destination please default to
> RR> the groups server? Currently if I set the spam dest as an unqualified
> RR> Gnus-Spam for my nnimap server then its going to nnml:Gnus-Spam. I need
> RR> to currently fully qualify it to e.g nnimap+riley:Gnus-Spam.
>
> I think it should behave like a regular group name, which is the
> behaviour you're noting.
>
> Ted

I am wondering why this would be better since unqualified is more
"local" and is the way for things like spam-split and spam-inbox. If
working on a group that belongs to a server it seems logical that merely
setting a group destination without fully qualifier such as
"nnimap+riles" means on that current group's server on not on some other
such as nnml which may, or may not, be random depending on select order.

Re: splitting working now : some issues/questions

[Richard Riley]

Richard Riley <rileyrg@googlemail.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>
>>> nnimap should default to 'unseen and allow 'unread as the selection
>>> criteria for splitting.
>>
>> Actually, looking over the code, it's trying to avoid all messages
>> marked with \Seen when splitting.  But it didn't actually work because
>> of a marks interface change.  This has now been fixed.
>
> Oh super ;) I can move all my emails back into INBOX...
>
> I'd assumed from your comments earlier when I raised this that this was
> a feature and wasn't to be changed.
>


spam-split seems to be marking things as unread if I specify INBOX (the
current group) or nothing as the last clause. so I read all my mail, mark
a few as spam and quit the group. Next thing all the ones I read as
marked as unread again. Spam marked ones are moved correctly. How to
specify "dont move anywhere but leave as unread"?

Re: splitting working now : some issues/questions

[Ted Zlatanov]

On Sun, 10 Oct 2010 08:57:34 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

RR> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Sun, 10 Oct 2010 03:09:55 +0200 Richard Riley <rileyrg@googlemail.com> wrote:
>> 
RR> Could (or should) the group parameter spam destination please default to
RR> the groups server? Currently if I set the spam dest as an unqualified
RR> Gnus-Spam for my nnimap server then its going to nnml:Gnus-Spam. I need
RR> to currently fully qualify it to e.g nnimap+riley:Gnus-Spam.
>> 
>> I think it should behave like a regular group name, which is the
>> behaviour you're noting.

RR> I am wondering why this would be better since unqualified is more
RR> "local" and is the way for things like spam-split and spam-inbox. If
RR> working on a group that belongs to a server it seems logical that merely
RR> setting a group destination without fully qualifier such as
RR> "nnimap+riles" means on that current group's server on not on some other
RR> such as nnml which may, or may not, be random depending on select order.

For spam-split and general splitting, actually I have wanted for a long
time the group name to be qualified, but currently splitting only works
to the same server.  To keep with the Gnus conventions, I'd prefer to
keep group names qualified whenever poassible.

Ted

Re: splitting working now : some issues/questions

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 10 Oct 2010 08:57:34 +0200 Richard Riley <rileyrg@googlemail.com> wrote:
>
> RR> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Sun, 10 Oct 2010 03:09:55 +0200 Richard Riley <rileyrg@googlemail.com>
> wrote:
>>> 
> RR> Could (or should) the group parameter spam destination please default to
> RR> the groups server? Currently if I set the spam dest as an unqualified
> RR> Gnus-Spam for my nnimap server then its going to nnml:Gnus-Spam. I need
> RR> to currently fully qualify it to e.g nnimap+riley:Gnus-Spam.
>>> 
>>> I think it should behave like a regular group name, which is the
>>> behaviour you're noting.
>
> RR> I am wondering why this would be better since unqualified is more
> RR> "local" and is the way for things like spam-split and spam-inbox. If
> RR> working on a group that belongs to a server it seems logical that merely
> RR> setting a group destination without fully qualifier such as
> RR> "nnimap+riles" means on that current group's server on not on some other
> RR> such as nnml which may, or may not, be random depending on select order.
>
> For spam-split and general splitting, actually I have wanted for a long
> time the group name to be qualified, but currently splitting only works
> to the same server.  To keep with the Gnus conventions, I'd prefer to
> keep group names qualified whenever poassible.
>
> Ted
>

Yes, but why? If you are working a group in a certain server and
something operates on that server, unqualified makes a lot more sense
and is simply more convenient keeping in mind the manual talking about
not qualifying in other such settings. "spam" means "spam" folder on
this server, "nnml+otherserver:spam" means something else.

Re: splitting working now : some issues/questions

[Richard Riley]

Richard Riley <rileyrg@googlemail.com> writes:

> Richard Riley <rileyrg@googlemail.com> writes:
>
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>>
>>>> nnimap should default to 'unseen and allow 'unread as the selection
>>>> criteria for splitting.
>>>
>>> Actually, looking over the code, it's trying to avoid all messages
>>> marked with \Seen when splitting.  But it didn't actually work because
>>> of a marks interface change.  This has now been fixed.
>>
>> Oh super ;) I can move all my emails back into INBOX...
>>
>> I'd assumed from your comments earlier when I raised this that this was
>> a feature and wasn't to be changed.
>>
>
> spam-split seems to be marking things as unread if I specify INBOX (the
> current group) or nothing as the last clause. so I read all my mail, mark
> a few as spam and quit the group. Next thing all the ones I read as
> marked as unread again. Spam marked ones are moved correctly. How to
> specify "dont move anywhere but leave as unread"?

Hmm. Possibly a hickup with offlineimap and dovecot. This behaviour has
stopped happening for now. unread remain unread.

Re: splitting working now : some issues/questions

[Andreas Schwab]

Richard Riley <rileyrg@googlemail.com> writes:

> "spam" means "spam" folder on this server

Actually it means the "spam" group on the default server
(gnus-select-method).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: splitting working now : some issues/questions

[Richard Riley]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Richard Riley <rileyrg@googlemail.com> writes:
>
>> "spam" means "spam" folder on this server
>
> Actually it means the "spam" group on the default server
> (gnus-select-method).

I was being hypothetical  about what I feel it should be for consistency
with other settings.

Here, I'll replace what you snipped to make it clear again:

,----
| Yes, but why? If you are working a group in a certain server and
| something operates on that server, unqualified makes a lot more sense
| and is simply more convenient keeping in mind the manual talking about
| not qualifying in other such settings. "spam" means "spam" folder on
| this server, "nnml+otherserver:spam" means something else.
`----

I'm not saying this to be difficult - its what I assumed it would be
keeping in mind the caveats on other settings NOT being qualified so
that they do indeed default to the current server.

Re: splitting working now : some issues/questions

[Ted Zlatanov]

On Sun, 10 Oct 2010 12:17:25 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

RR> Ted Zlatanov <tzz@lifelogs.com> writes:
>> For spam-split and general splitting, actually I have wanted for a long
>> time the group name to be qualified, but currently splitting only works
>> to the same server.  To keep with the Gnus conventions, I'd prefer to
>> keep group names qualified whenever poassible.

RR> Yes, but why?

1) to keep Gnus consistent

2) to avoid inconveniencing the people who have been using spam.el for
many years now

3) because it uses the Gnus "move article" facility, which uses
qualified names

RR> If you are working a group in a certain server and something
RR> operates on that server, unqualified makes a lot more sense and is
RR> simply more convenient keeping in mind the manual talking about not
RR> qualifying in other such settings. "spam" means "spam" folder on
RR> this server, "nnml+otherserver:spam" means something else.

Well, spam.el has been around for a while now (since 2002).  You are the
first person to say they want the move destination to be unqualified.
So while it may be that all the other users have been annoyed by that
but kept quiet, it's more likely it's not a big deal.  So unless I hear
from more people supporting your view, I'd rather keep the status quo.

Thanks
Ted

Re: splitting working now : some issues/questions

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 10 Oct 2010 12:17:25 +0200 Richard Riley <rileyrg@googlemail.com> wrote:
>
> RR> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> For spam-split and general splitting, actually I have wanted for a long
>>> time the group name to be qualified, but currently splitting only works
>>> to the same server.  To keep with the Gnus conventions, I'd prefer to
>>> keep group names qualified whenever poassible.
>
> RR> Yes, but why?
>
> 1) to keep Gnus consistent

Thats my whole point : Its not consistent. Some places need it
qualifying others dont. And if you dont it defaults to the current
server. This is NOT how its done here. And that saw my mail suddenly
vanish and totally hog my network and cause my gmail bandwidth for a day
to run out. Admittedly that was in the context of the INBOX spam
handling bug causing ALL messages to be reprocessed.

>
> 2) to avoid inconveniencing the people who have been using spam.el for
> many years now

Since a lot is changing and certainly my entire spam set up had to be
rewired as a result of the way INBOX was dealt with. Which due to lack
of feedback I assumed was redesign as opposed to bugs. It seems to have
settled back now and was indeed bugs.

>
> 3) because it uses the Gnus "move article" facility, which uses
> qualified names

So it can qualify it before calling that? At least some other vars are
server local.

"because thats the way it is" hardly encourages debate. It makes sense
to me at least that a folder name UNQUALIFIED should be local to that
server. In fact it seems so obvious I wonder why we debate it ;)

>
> RR> If you are working a group in a certain server and something
> RR> operates on that server, unqualified makes a lot more sense and is
> RR> simply more convenient keeping in mind the manual talking about not
> RR> qualifying in other such settings. "spam" means "spam" folder on
> RR> this server, "nnml+otherserver:spam" means something else.
>
> Well, spam.el has been around for a while now (since 2002).  You are the
> first person to say they want the move destination to be unqualified.

Where did I say that? I said I would have *expected* it to be local
because of other such settings which are local. Why qualify something
when on a server each and every time? If you want it moved elsewhere on
another server then qualify it.  A huge hurdle with convincing people to
use Gnus I find in #emacs for example are such inconsistencies and the
general difficulty with customisation. You are aware such a mindset
exists I assume? It's not easy. I am only conveying my views here on how
it would make more sense to be done another way. Of course I understand
reluctance to change as well. The current changes are bordering on
miraculous and make Gnus a wonderful client.

> So while it may be that all the other users have been annoyed by that
> but kept quiet, it's more likely it's not a big deal.  So unless I hear
> from more people supporting your view, I'd rather keep the status quo.

Fine if thats the way you see it. The improvements done to Gnus suggest
that "because thats the way its always been" are not the only criteria
under consideration while development and improvements take place.

I'll wave the white flag now.

cheers for the feedback

r.

Re: splitting working now : some issues/questions

[Ted Zlatanov]

On Sun, 10 Oct 2010 18:28:10 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

>> 1) to keep Gnus consistent

RR> Thats my whole point : Its not consistent. Some places need it
RR> qualifying others dont. And if you dont it defaults to the current
RR> server. 

Sorry, can you tell me where, besides the splitting methods, does Gnus
take an unqualified group name to mean "in the current server"?  I may
very well be wrong but I don't know of such behavior.

>> 3) because it uses the Gnus "move article" facility, which uses
>> qualified names

RR> So it can qualify it before calling that? At least some other vars are
RR> server local.

But if I qualify it, there's no way to pass unqualified group names.
IOW if I really did want to move to the primary server's "foo" group, I
can't do it now because it means "the current server's 'foo' group".

This is the heart of the problem.  If you can solve it, I'm much more in
favor of the change, despite the documentation change and possible
breakage for other spam.el users, because it is sensible.

If I get a bunch more people saying "no, do it Richard's way, even if it
breaks things" then I'll make that change despite my concerns.  That's
the voting I was referring to.  I don't mean your proposal will be
ignored unless there's more votes for it.

>> So while it may be that all the other users have been annoyed by that
>> but kept quiet, it's more likely it's not a big deal.  So unless I hear
>> from more people supporting your view, I'd rather keep the status quo.

RR> Fine if thats the way you see it. The improvements done to Gnus suggest
RR> that "because thats the way its always been" are not the only criteria
RR> under consideration while development and improvements take place.

No, definitely not.  The nnimap.el fixes Lars is doing, for instance,
came about because he asked "what needs work" and IMAP support was at
the top of the list (even though imap.el was not terribly broken).  I
worked on the internal GnuTLS support for Emacs because the Gnus SSL/TLS
connectivity has been a continuous source of bug reports and complaints.
Julien's work was to scratch an itch.  So there's all kinds of reasons.

Ted

Re: splitting working now : some issues/questions

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 10 Oct 2010 18:28:10 +0200 Richard Riley <rileyrg@googlemail.com> wrote:
>
>>> 1) to keep Gnus consistent
>
> RR> Thats my whole point : Its not consistent. Some places need it
> RR> qualifying others dont. And if you dont it defaults to the current
> RR> server. 
>
> Sorry, can you tell me where, besides the splitting methods, does Gnus
> take an unqualified group name to mean "in the current server"?  I may
> very well be wrong but I don't know of such behavior.

The splitting methods ...

I wont argue the case anymore. it just seemed to me much more
natural. It still does. But its not me changing the SW and I do
appreciate the efforts Lars is going to so I wont make any more comments
after this post on it. You must understand that it not doing what I
expected (rightly or wrongly) caused me awful headaches with literally
thousands of emails being chucked around taking hours (but as I said
that was in conjunction with the bug with splitting everything).

Bottom line if it was new feature : if specifying a destination folder in
a group which belongs to a server then an unqualified group name for
that destination group should refer to a group local to that server. It
just makes sense. Want a specific OTHER server? Then qualify it.

Anyway, I think thats about it really ;)