message splitting gone haywire

message splitting gone haywire

[Eric Abrahamsen]

I've been keeping up to date with git gnus (I'm on Ubuntu's 23.2 emacs),
and a week or so ago, message splitting started going haywire. There's
not much of a discernible pattern: messages are just going in the wrong
groups.

I'm using the registry, with gnus-registry-split-fancy-with-parent. Also
BBDB 3.0, with bbdb/gnus-split method. Then some plain old
nnmail-split-fancy regexps. None of these seems obviously the culprit.
Some messages that should be split into their own groups by
nnmail-split-fancy go into those groups half the time, into mail.misc
the other half (they used to all go into the proper group, they're
automated messages and neither the message nor the regexp has changed).
A message I just sent from one of my email addresses to another of my
email addresses went into a group that's only referenced by a
gnus.private line in a BBDB record (not my own record, needless to say).

Like I said, no one thing seems to be the clear source of the problem.
Has anyone else seen anything like this over the past few weeks? I can
paste the relevant part of my config, if necessary.

Thanks,
Eric

Re: message splitting gone haywire

[Ted Zlatanov]

On Thu, 18 Aug 2011 14:38:48 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> I've been keeping up to date with git gnus (I'm on Ubuntu's 23.2 emacs),
EA> and a week or so ago, message splitting started going haywire. There's
EA> not much of a discernible pattern: messages are just going in the wrong
EA> groups.

EA> I'm using the registry, with gnus-registry-split-fancy-with-parent. Also
EA> BBDB 3.0, with bbdb/gnus-split method. Then some plain old
EA> nnmail-split-fancy regexps. None of these seems obviously the culprit.
EA> Some messages that should be split into their own groups by
EA> nnmail-split-fancy go into those groups half the time, into mail.misc
EA> the other half (they used to all go into the proper group, they're
EA> automated messages and neither the message nor the regexp has changed).
EA> A message I just sent from one of my email addresses to another of my
EA> email addresses went into a group that's only referenced by a
EA> gnus.private line in a BBDB record (not my own record, needless to say).

EA> Like I said, no one thing seems to be the clear source of the problem.
EA> Has anyone else seen anything like this over the past few weeks? I can
EA> paste the relevant part of my config, if necessary.

I didn't touch the registry around the time you mention.  But I would
simply bisect the issue by limiting splitting to one method at a time
and see if it causes the aberrant behavior.

Ted

Re: message splitting gone haywire

[Eric Abrahamsen]

On Thu, Sep 29 2011, Ted Zlatanov wrote:

> On Thu, 18 Aug 2011 14:38:48 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>
> EA> I've been keeping up to date with git gnus (I'm on Ubuntu's 23.2 emacs),
> EA> and a week or so ago, message splitting started going haywire. There's
> EA> not much of a discernible pattern: messages are just going in the wrong
> EA> groups.
>
> EA> I'm using the registry, with gnus-registry-split-fancy-with-parent. Also
> EA> BBDB 3.0, with bbdb/gnus-split method. Then some plain old
> EA> nnmail-split-fancy regexps. None of these seems obviously the culprit.
> EA> Some messages that should be split into their own groups by
> EA> nnmail-split-fancy go into those groups half the time, into mail.misc
> EA> the other half (they used to all go into the proper group, they're
> EA> automated messages and neither the message nor the regexp has changed).
> EA> A message I just sent from one of my email addresses to another of my
> EA> email addresses went into a group that's only referenced by a
> EA> gnus.private line in a BBDB record (not my own record, needless to say).
>
> EA> Like I said, no one thing seems to be the clear source of the problem.
> EA> Has anyone else seen anything like this over the past few weeks? I can
> EA> paste the relevant part of my config, if necessary.
>
> I didn't touch the registry around the time you mention.  But I would
> simply bisect the issue by limiting splitting to one method at a time
> and see if it causes the aberrant behavior.

Eventually I figured out what was happening -- in the case of a few
regular email threads (newsletters in particular), one earlier issue had
gone into the wrong group, and all subsequent issues went into the same
group. Standard registry behavior, but I had physically moved later
issues into the proper group, and new issues were still getting split
into the wrong group. Eventually I realized that the originally
offending message was still buried back in the wrong group, and once I
rooted out all of them, things went back to normal.

This happened with two or three separate threads (not sure how),
creating the impression that all hell had broken loose with splitting.

I'm not sure what proper behavior should be, but I did expect that
split-with-parent would send new messages to the location of the
newest/nearest parent, and not to where some distant ancestor had once
gone.

Thanks!
Eric

-- 
GNU Emacs 23.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.4)
 of 2011-04-04 on rothera, modified by DebiannNo Gnus v0.18

Re: message splitting gone haywire

[Ted Zlatanov]

On Thu, 29 Sep 2011 17:00:16 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> Eventually I figured out what was happening -- in the case of a few
EA> regular email threads (newsletters in particular), one earlier issue had
EA> gone into the wrong group, and all subsequent issues went into the same
EA> group. Standard registry behavior, but I had physically moved later
EA> issues into the proper group, and new issues were still getting split
EA> into the wrong group. Eventually I realized that the originally
EA> offending message was still buried back in the wrong group, and once I
EA> rooted out all of them, things went back to normal.

EA> This happened with two or three separate threads (not sure how),
EA> creating the impression that all hell had broken loose with splitting.

EA> I'm not sure what proper behavior should be, but I did expect that
EA> split-with-parent would send new messages to the location of the
EA> newest/nearest parent, and not to where some distant ancestor had once
EA> gone.

Good point.  Well, we build the full list of groups, then we call
`gnus-registry-post-process-groups' on it.  That looks at
`gnus-registry-split-strategy' which is nil by default; the defcustom
says:

  '(choice :tag "Splitting strategy"
           (const :tag "Only use single choices, discard multiple matches" nil)
           (const :tag "Majority of matches wins" majority)
           (const :tag "First found wins"  first))

Have you customized this?  My guess is you have 'first and the first
group in the list was from the oldest reference.

Maybe I should add 'oldest and 'newest strategies...  We have the
article notice dates in the registry.

Ted

Re: message splitting gone haywire

[Eric Abrahamsen]

On Thu, Sep 29 2011, Ted Zlatanov wrote:

> On Thu, 29 Sep 2011 17:00:16 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>
> EA> Eventually I figured out what was happening -- in the case of a few
> EA> regular email threads (newsletters in particular), one earlier issue had
> EA> gone into the wrong group, and all subsequent issues went into the same
> EA> group. Standard registry behavior, but I had physically moved later
> EA> issues into the proper group, and new issues were still getting split
> EA> into the wrong group. Eventually I realized that the originally
> EA> offending message was still buried back in the wrong group, and once I
> EA> rooted out all of them, things went back to normal.
>
> EA> This happened with two or three separate threads (not sure how),
> EA> creating the impression that all hell had broken loose with splitting.
>
> EA> I'm not sure what proper behavior should be, but I did expect that
> EA> split-with-parent would send new messages to the location of the
> EA> newest/nearest parent, and not to where some distant ancestor had once
> EA> gone.
>
> Good point.  Well, we build the full list of groups, then we call
> `gnus-registry-post-process-groups' on it.  That looks at
> `gnus-registry-split-strategy' which is nil by default; the defcustom
> says:
>
>   '(choice :tag "Splitting strategy"
>            (const :tag "Only use single choices, discard multiple matches" nil)
>            (const :tag "Majority of matches wins" majority)
>            (const :tag "First found wins"  first))
>
> Have you customized this?  My guess is you have 'first and the first
> group in the list was from the oldest reference.

Huh, I set this to 'majority, probably early on when I had no idea how
all this actually worked. That would indicate that I must have had more
messages in the "wrong" group than in the "right", yes? It sure didn't
seem that way at the time, though I could very well have been wrong.
Anyway, now that I know how this actually works, it does seem that a
'newest and 'oldest would be handy.

Thanks!

-- 
GNU Emacs 23.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.4)
 of 2011-04-04 on rothera, modified by DebiannNo Gnus v0.18

Re: message splitting gone haywire

[Ted Zlatanov]

On Thu, 29 Sep 2011 21:55:04 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> On Thu, Sep 29 2011, Ted Zlatanov wrote:
>> On Thu, 29 Sep 2011 17:00:16 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>> 
EA> I'm not sure what proper behavior should be, but I did expect that
EA> split-with-parent would send new messages to the location of the
EA> newest/nearest parent, and not to where some distant ancestor had once
EA> gone.
>> 
>> Good point.  Well, we build the full list of groups, then we call
>> `gnus-registry-post-process-groups' on it.  That looks at
>> `gnus-registry-split-strategy' which is nil by default; the defcustom
>> says:
>> 
>> '(choice :tag "Splitting strategy"
>> (const :tag "Only use single choices, discard multiple matches" nil)
>> (const :tag "Majority of matches wins" majority)
>> (const :tag "First found wins"  first))
>> 
>> Have you customized this?  My guess is you have 'first and the first
>> group in the list was from the oldest reference.

EA> Huh, I set this to 'majority, probably early on when I had no idea how
EA> all this actually worked. That would indicate that I must have had more
EA> messages in the "wrong" group than in the "right", yes? It sure didn't
EA> seem that way at the time, though I could very well have been wrong.
EA> Anyway, now that I know how this actually works, it does seem that a
EA> 'newest and 'oldest would be handy.

Yes, or there is a bug in my code ;)  I should improve the logging in
that area.

I get 'newest but I can't think of a reason for 'oldest.  Can you?

Either way they should be easy to implement, I just order the references
by age and then act as if the strategy was 'first.  But I just can't
think of a way to use 'oldest.

Ted

Re: message splitting gone haywire

[Eric Abrahamsen]

On Thu, Sep 29 2011, Ted Zlatanov wrote:

> On Thu, 29 Sep 2011 21:55:04 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>
> EA> On Thu, Sep 29 2011, Ted Zlatanov wrote:
>>> On Thu, 29 Sep 2011 17:00:16 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>>> 
> EA> I'm not sure what proper behavior should be, but I did expect that
> EA> split-with-parent would send new messages to the location of the
> EA> newest/nearest parent, and not to where some distant ancestor had once
> EA> gone.
>>> 
>>> Good point.  Well, we build the full list of groups, then we call
>>> `gnus-registry-post-process-groups' on it.  That looks at
>>> `gnus-registry-split-strategy' which is nil by default; the defcustom
>>> says:
>>> 
>>> '(choice :tag "Splitting strategy"
>>> (const :tag "Only use single choices, discard multiple matches" nil)
>>> (const :tag "Majority of matches wins" majority)
>>> (const :tag "First found wins"  first))
>>> 
>>> Have you customized this?  My guess is you have 'first and the first
>>> group in the list was from the oldest reference.
>
> EA> Huh, I set this to 'majority, probably early on when I had no idea how
> EA> all this actually worked. That would indicate that I must have had more
> EA> messages in the "wrong" group than in the "right", yes? It sure didn't
> EA> seem that way at the time, though I could very well have been wrong.
> EA> Anyway, now that I know how this actually works, it does seem that a
> EA> 'newest and 'oldest would be handy.
>
> Yes, or there is a bug in my code ;)  I should improve the logging in
> that area.
>
> I get 'newest but I can't think of a reason for 'oldest.  Can you?

No I can't. The only time I wouldn't want 'newest is if I wanted to put
one or two recent messages somewhere other than with their parents, but
still wanted future messages to continue to be split with parents. This
seems best served by 'majority (or a working version thereof :)).

-- 
GNU Emacs 23.2.1 (i686-pc-linux-gnu, GTK+ Version 2.24.4)
 of 2011-04-04 on rothera, modified by DebiannNo Gnus v0.18

Re: message splitting gone haywire

[Adam Sjøgren]

On Thu, 29 Sep 2011 07:47:22 -0500, Ted wrote:

>   '(choice :tag "Splitting strategy"
>            (const :tag "Only use single choices, discard multiple matches" nil)
>            (const :tag "Majority of matches wins" majority)
>            (const :tag "First found wins"  first))

> Have you customized this?  My guess is you have 'first and the first
> group in the list was from the oldest reference.

> Maybe I should add 'oldest and 'newest strategies...  We have the
> article notice dates in the registry.

Why not have Gnus prompt the user?

"This article doesn't have a well defined destination, go by first (this
time only), majority (now and forever), skip (the three next times),
last (for the next 20 minutes), nearest (from next time and then on),
random, or randomly select a method (for a random period of time)
(f/m/s/l/n/r/R)?"


  Best regards,

    Adam

-- 
 "Soon we'll have spent a whole month at sea,                 Adam Sjøgren
  splitting atoms for no apparent reason"                asjo@koldfront.dk

Re: message splitting gone haywire

[Ted Zlatanov]

On Thu, 29 Sep 2011 20:49:14 +0200 asjo@koldfront.dk (Adam Sjøgren) wrote: 

AS> On Thu, 29 Sep 2011 07:47:22 -0500, Ted wrote:
>> '(choice :tag "Splitting strategy"
>> (const :tag "Only use single choices, discard multiple matches" nil)
>> (const :tag "Majority of matches wins" majority)
>> (const :tag "First found wins"  first))

>> Have you customized this?  My guess is you have 'first and the first
>> group in the list was from the oldest reference.

>> Maybe I should add 'oldest and 'newest strategies...  We have the
>> article notice dates in the registry.

AS> Why not have Gnus prompt the user?

AS> "This article doesn't have a well defined destination, go by first (this
AS> time only), majority (now and forever), skip (the three next times),
AS> last (for the next 20 minutes), nearest (from next time and then on),
AS> random, or randomly select a method (for a random period of time)
AS> (f/m/s/l/n/r/R)?"

I could add 'ask, but we're talking potentially hundreds of these
prompts during splitting.  Also I don't think a temporary splitting
strategy is a good thing because you won't necessarily know where things
went if you split 20 minutes later.  Similarly the random choice can
cause confusion.

I think the only choices available from 'ask should be 'first,
'majority, and 'newest.  Once chosen they should persist until the end
of the session, using the Customize system.  The user can save the
preference at that point.

I need to add better logging to tell the user what happened.  Maybe a
registry-split-log buffer?

Thanks
Ted

Re: message splitting gone haywire

[Adam Sjøgren]

On Thu, 29 Sep 2011 15:21:37 -0500, Ted wrote:

> I could add 'ask, but we're talking potentially hundreds of these
> prompts during splitting.  Also I don't think a temporary splitting
> strategy is a good thing because you won't necessarily know where things
> went if you split 20 minutes later.  Similarly the random choice can
> cause confusion.

> I think the only choices available from 'ask should be 'first,
> 'majority, and 'newest.  Once chosen they should persist until the end
> of the session, using the Customize system.  The user can save the
> preference at that point.

Ok, taking my sarcasm voice off again: I think you are trying to make
the system too clever and too magic for the users and the programs own
good. Look at how confused Eric Abrahamsen got by the *intended and
chosen* behaviour!

I'd say make the sane choice the default, and don't offer any other
choices until someone a) argues that it is really, really important, and
b) wants to use it.

I think I am getting old and grumpy.


   Sorry,

    Adam

-- 
 "Americans loves sports in which they are the best,          Adam Sjøgren
  like basketball and diabetes"                          asjo@koldfront.dk

Re: message splitting gone haywire

[Ted Zlatanov]

On Thu, 29 Sep 2011 22:38:56 +0200 asjo@koldfront.dk (Adam Sjøgren) wrote: 

AS> On Thu, 29 Sep 2011 15:21:37 -0500, Ted wrote:
>> I could add 'ask, but we're talking potentially hundreds of these
>> prompts during splitting.  Also I don't think a temporary splitting
>> strategy is a good thing because you won't necessarily know where things
>> went if you split 20 minutes later.  Similarly the random choice can
>> cause confusion.

>> I think the only choices available from 'ask should be 'first,
>> 'majority, and 'newest.  Once chosen they should persist until the end
>> of the session, using the Customize system.  The user can save the
>> preference at that point.

AS> Ok, taking my sarcasm voice off again: I think you are trying to make
AS> the system too clever and too magic for the users and the programs own
AS> good. Look at how confused Eric Abrahamsen got by the *intended and
AS> chosen* behaviour!

Heh, scanning through hundreds of messages definitely shuts off my
sarcasm meter.  I think you're right, in any case.

AS> I'd say make the sane choice the default, and don't offer any other
AS> choices until someone a) argues that it is really, really important, and
AS> b) wants to use it.

I thought I wanted 'majority but I'm pretty sure 'newest will do the job
better.  Similarly 'first is not better than 'newest and usually worse.

So, unless someone has objections, I'll make 'newest the default and
only option.

Ted