Issue with nnml backend and gnus-group-get-new-news()

Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

Hello,

[ Sorry you receive this message several times but I'm really having
  hard time to post on ding :( ]

I'm using nnml backend for my emails which are stored in /var/spool directory.

When using gnus-group-get-new-news() with a prefix, Gnus moves the
emails from the spool directory to the temporary Incoming files but it
forgets to split the email after that. So IOW all emails fetched with
'C-u 2 g' never reached any groups, they seem to be just deleted !

I'm using Gnus shipped with emacs-cvs.

Is it really a bug, or something I'm missing ?

Re: Issue with nnml backend and gnus-group-get-new-news()

[Tassilo Horn]

Francis Moreau <francis.moro@gmail.com> writes:

Hi Francis,

> I'm using nnml backend for my emails which are stored in /var/spool
> directory.
>
> When using gnus-group-get-new-news() with a prefix, Gnus moves the
> emails from the spool directory to the temporary Incoming files but it
> forgets to split the email after that. So IOW all emails fetched with
> 'C-u 2 g' never reached any groups, they seem to be just deleted !

Hm, could you please provide the relevant parts of your config?  I think
that maybe your splitting rules don't match anything, and so all your
mails end up in some "misc" group you probably aren't subscribed to.  In
*Group* hit ^ to get to the server buffer, and there hit RET on your
nnml backend.  Check for unsubscribed groups there.

Bye,
Tassilo

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

Hi Tassilo,

On Mon, Nov 30, 2009 at 10:53 AM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
> Hm, could you please provide the relevant parts of your config?

Sure, it's quite simple:

(setq nnmail-split-methods
        ("save.wyplay.bugs" "^From: bugzilla-daemon@.*")
        ("Inbox" "")))


> I think
> that maybe your splitting rules don't match anything, and so all your
> mails end up in some "misc" group you probably aren't subscribed to.

Hm, I don't think so since they work as expected if I call
gnus-group-get-new-news() _without_ prefix.

Could you give it a try and see if it works as exepcted on your side ?

> In
> *Group* hit ^ to get to the server buffer, and there hit RET on your
> nnml backend.  Check for unsubscribed groups there.

There's no 'misc' group in the nnml server.

Thanks
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Tassilo Horn]

Francis Moreau <francis.moro@gmail.com> writes:

Hi Francis,

> On Mon, Nov 30, 2009 at 10:53 AM, Tassilo Horn <tassilo@member.fsf.org> wrote:
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>> Hm, could you please provide the relevant parts of your config?
>
> Sure, it's quite simple:
>
> (setq nnmail-split-methods
>         ("save.wyplay.bugs" "^From: bugzilla-daemon@.*")
>         ("Inbox" "")))

Do you have this in your .gnus.el?  The syntax is wrong, and normally
you should get an error with that.  This would be correct:

--8<---------------cut here---------------start------------->8---
(setq nnmail-split-methods
        '(("save.wyplay.bugs" "^From: bugzilla-daemon@.*")
          ("Inbox" "")))
--8<---------------cut here---------------end--------------->8---

>> I think that maybe your splitting rules don't match anything, and so
>> all your mails end up in some "misc" group you probably aren't
>> subscribed to.
>
> Hm, I don't think so since they work as expected if I call
> gnus-group-get-new-news() _without_ prefix.

Hm, I don't know how the prefix should interfer with the splitting.  Do
you provide a prefix value that includes or excludes those groups?

> Could you give it a try and see if it works as exepcted on your side ?

I use fancy mail splitting, and it works for me.  But I didn't test with
a prefix argument till now, because those two accounts I use nnml and
POP3 sources for are very low traffic.

--8<---------------cut here---------------start------------->8---
(setq nnmail-split-methods 'nnmail-split-fancy
      nnmail-split-fancy
      '(| (any "tassilo.horn@gmx.de"     "gmx")
          (any "tassilo.horn@freenet.de" "freenet")
          "misc"))
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Mon, Nov 30, 2009 at 1:26 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
>
> Do you have this in your .gnus.el?  The syntax is wrong, and normally
> you should get an error with that.  This would be correct:
>

you're right, I messed it up during the copy & paste operation.

>
>>> I think that maybe your splitting rules don't match anything, and so
>>> all your mails end up in some "misc" group you probably aren't
>>> subscribed to.
>>
>> Hm, I don't think so since they work as expected if I call
>> gnus-group-get-new-news() _without_ prefix.
>
> Hm, I don't know how the prefix should interfer with the splitting.

I have no idea, but it really looks like a bug so I'm reporting.

>  Do
> you provide a prefix value that includes or excludes those groups?

From the help documentation:

  Get newly arrived articles.
  If ARG is a number, it specifies which levels you are interested in
  re-scanning.  If ARG is non-nil and not a number, this will force
  "hard" re-reading of the active files from all servers.

>
>> Could you give it a try and see if it works as exepcted on your side ?
>
> I use fancy mail splitting, and it works for me.  But I didn't test with
> a prefix argument till now, because those two accounts I use nnml and
> POP3 sources for are very low traffic.

Well I think the issue is when testing _with_ a prefix argument.

thanks
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Tassilo Horn]

Francis Moreau <francis.moro@gmail.com> writes:

>> Do you provide a prefix value that includes or excludes those groups?
>
> From the help documentation:
>
>   Get newly arrived articles.
>   If ARG is a number, it specifies which levels you are interested in
>   re-scanning.  If ARG is non-nil and not a number, this will force
>   "hard" re-reading of the active files from all servers.

Yeah, I know the semantics.  What I wanted to know is the level of the
greups in question, and the prefix argument you provide. :-)

Bye,
Tassilo

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Mon, Nov 30, 2009 at 2:38 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>>> Do you provide a prefix value that includes or excludes those groups?
>>
>> From the help documentation:
>>
>>   Get newly arrived articles.
>>   If ARG is a number, it specifies which levels you are interested in
>>   re-scanning.  If ARG is non-nil and not a number, this will force
>>   "hard" re-reading of the active files from all servers.
>
> Yeah, I know the semantics.  What I wanted to know is the level of the
> greups in question, and the prefix argument you provide. :-)
>

oops. sorry.

So in my case I pass 2 as argument (C-u 2 g) and all mails from group
whose level is 1 or 2 are fetched but no splitted (they're stuck in
Incoming files).

-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Tassilo Horn]

Francis Moreau <francis.moro@gmail.com> writes:

Hi Francis,

> So in my case I pass 2 as argument (C-u 2 g) and all mails from group
> whose level is 1 or 2 are fetched but no splitted (they're stuck in
> Incoming files).

Hm, now I was able to receive mails from POP3 accounts into my nnml
groups.  Those groups are all level 5.  I fetched mail with `5 g', and 2
new mails were received, stored as Incoming files, split and delivered
into the right groups.

So at least with my Gnus version (Emacs CVS from yesterday), it seems to
work.  So either it's a strange misconfiguration on your side, or
"normal" mail splitting is broken (I use fancy mail splitting), or a bug
in an older emacs version that's fixed in the meantime.

Hm, what does M-x nnmail-split-history RET say?

You might also want to (setq nnmail-split-tracing t) and check the value
of `nnmail-split-trace' after fetching mail.

Bye,
Tassilo

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

Hello Tassilo

Thank you for your time.

On Tue, Dec 1, 2009 at 10:55 AM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
> Hi Francis,
>
>> So in my case I pass 2 as argument (C-u 2 g) and all mails from group
>> whose level is 1 or 2 are fetched but no splitted (they're stuck in
>> Incoming files).
>
> Hm, now I was able to receive mails from POP3 accounts into my nnml
> groups.  Those groups are all level 5.  I fetched mail with `5 g', and 2
> new mails were received, stored as Incoming files, split and delivered
> into the right groups.
>
> So at least with my Gnus version (Emacs CVS from yesterday), it seems to
> work.  So either it's a strange misconfiguration on your side, or
> "normal" mail splitting is broken (I use fancy mail splitting), or a bug
> in an older emacs version that's fixed in the meantime.

I'm using Gnus from Emacs CVS which I updated 2 months ago.

>
> Hm, what does M-x nnmail-split-history RET say?

So I tried different values for the prefix and here are the ouptut of
M-x nnmail-split-history for each values. For each values, a mail was
previously sent and should reach my 'Inbox' group.

'g'    Ok -> Inbox:2599
'1 g' Ok -> Inbox:2600

'2 g' Ko -> Inbox:17
'3 g' Ko -> Inbox:18
'4 g' Ko -> Inbox:19
'5 g' Ko -> Inbox:20

'g'    Ok -> Inbox:2601

Ok means that the mail finally was delivered to the right group
('Inbox'), otherwise Ko, the mail is stuck in the 'Incoming' stage.

As you can see the sequence number after "Inbox::" seems to have been
reset in the non working case.

Hope that helps.
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Tassilo Horn]

Francis Moreau <francis.moro@gmail.com> writes:

Hi Francis!

>> Hm, what does M-x nnmail-split-history RET say?
>
> So I tried different values for the prefix and here are the ouptut of
> M-x nnmail-split-history for each values. For each values, a mail was
> previously sent and should reach my 'Inbox' group.
>
> 'g'    Ok -> Inbox:2599
> '1 g' Ok -> Inbox:2600
>
> '2 g' Ko -> Inbox:17
> '3 g' Ko -> Inbox:18
> '4 g' Ko -> Inbox:19
> '5 g' Ko -> Inbox:20
>
> 'g'    Ok -> Inbox:2601
>
> Ok means that the mail finally was delivered to the right group
> ('Inbox'), otherwise Ko, the mail is stuck in the 'Incoming' stage.
>
> As you can see the sequence number after "Inbox::" seems to have been
> reset in the non working case.

Oh, that's interesting.

Please enter your Inbox and do `j 17', ..., `j 20', and check if those
messages are indeed the new ones, which are also there as Incoming
files.  If that's indeed the case, then it seems possible that old mails
are overwritten, which would be an extremely serious bug.  Another
possibility would be that in that case Gnus reuses article numbers of
already deleted messages, and the fetching somehow assumes that they
always increase, and so the new ones with old article numbers aren't
recognized as being new.

In any case, please write a bug report (M-x gnus-bug) with a very
detailed description.  The trace analysis you did above is good.

Bye,
Tassilo

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

Hello Tassilo,

On Tue, Dec 1, 2009 at 9:17 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
> Hi Francis!
>
>>> Hm, what does M-x nnmail-split-history RET say?
>>
>> So I tried different values for the prefix and here are the ouptut of
>> M-x nnmail-split-history for each values. For each values, a mail was
>> previously sent and should reach my 'Inbox' group.
>>
>> 'g'    Ok -> Inbox:2599
>> '1 g' Ok -> Inbox:2600
>>
>> '2 g' Ko -> Inbox:17
>> '3 g' Ko -> Inbox:18
>> '4 g' Ko -> Inbox:19
>> '5 g' Ko -> Inbox:20
>>
>> 'g'    Ok -> Inbox:2601
>>
>> Ok means that the mail finally was delivered to the right group
>> ('Inbox'), otherwise Ko, the mail is stuck in the 'Incoming' stage.
>>
>> As you can see the sequence number after "Inbox::" seems to have been
>> reset in the non working case.
>
> Oh, that's interesting.
>
> Please enter your Inbox and do `j 17', ..., `j 20', and check if those
> messages are indeed the new ones, which are also there as Incoming
> files.

Ok doing 'j 17, j 18..., j20' always gives the same result:

   Couldn't go to article 17
   Couldn't go to article 18
   ...
   Couldn't go to article 20

> If that's indeed the case, then it seems possible that old mails
> are overwritten, which would be an extremely serious bug.  Another
> possibility would be that in that case Gnus reuses article numbers of
> already deleted messages, and the fetching somehow assumes that they
> always increase, and so the new ones with old article numbers aren't
> recognized as being new.
>
> In any case, please write a bug report (M-x gnus-bug) with a very
> detailed description.  The trace analysis you did above is good.
>

Ok I'll do that.

Thanks !
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> So I tried different values for the prefix and here are the ouptut of
> M-x nnmail-split-history for each values. For each values, a mail was
> previously sent and should reach my 'Inbox' group.
>
> 'g'    Ok -> Inbox:2599
> '1 g' Ok -> Inbox:2600
>
> '2 g' Ko -> Inbox:17

[...]

> Ok means that the mail finally was delivered to the right group
> ('Inbox'), otherwise Ko, the mail is stuck in the 'Incoming' stage.

Are you sure you don't have two groups named "Inbox"?  Maybe check if
any of the other servers in the server buffer have a group named Inbox?
Or check your .newsrc.eld for other occurrences of the word "Inbox"?

Dan

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Thu, Dec 3, 2009 at 12:57 AM, Dan Christensen <jdc@uwo.ca> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>> So I tried different values for the prefix and here are the ouptut of
>> M-x nnmail-split-history for each values. For each values, a mail was
>> previously sent and should reach my 'Inbox' group.
>>
>> 'g'    Ok -> Inbox:2599
>> '1 g' Ok -> Inbox:2600
>>
>> '2 g' Ko -> Inbox:17
>
> [...]
>
>> Ok means that the mail finally was delivered to the right group
>> ('Inbox'), otherwise Ko, the mail is stuck in the 'Incoming' stage.
>
> Are you sure you don't have two groups named "Inbox"?  Maybe check if
> any of the other servers in the server buffer have a group named Inbox?

Yes there're two others servers having a group name Inbox:

a) The virtual server I setup to read the cached/persistent articles.
Here is how it's configured:

(nnml "cache"
      (nnml-directory "~/News/cache/")
      (nnml-active-file "~/News/cache/active"))

I checked there to see if it contains the missing mail but it doesn't have them.

b) The mairix server. But if I tried to look into the Inbox group of
that server, Gnus tells me:

"Couldn't request group nnml+mairix-ephemeral:Inbox: Invalid group (no
such directory)"

> Or check your .newsrc.eld for other occurrences of the word "Inbox"?

I have several occurences of "Inbox" in this file:

(setq gnus-newsrc-alist ... ("nnml:Inbox" 1 ((1 . 2616))...)
(setq gnus-killed-list ... "nnfolder:Inbox" ... "nnml+mairix:Inbox" ...)
(setq gnus-topic-topology '(.... (("Inbox" visible nil nil)) ...)
(setq gnus-topic-alist '((... ("Inbox" "nnml:Inbox"...)...)

-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> (setq gnus-killed-list ... "nnfolder:Inbox" ... "nnml+mairix:Inbox" ...)

That nnfolder:Inbox looks suspicious to me.  In the *Group* buffer,
can you find that group by typing either `9 L' or by typing
`j nnfolder:Inbox RET'?

And why do you have an Inbox with mairix?  If any mail gets split to
there, it will get deleted when mairix recreates the search results.

Dan

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Thu, Dec 3, 2009 at 3:09 PM, Dan Christensen <jdc@uwo.ca> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>> (setq gnus-killed-list ... "nnfolder:Inbox" ... "nnml+mairix:Inbox" ...)
>
> That nnfolder:Inbox looks suspicious to me.  In the *Group* buffer,
> can you find that group by typing either `9 L' or by typing
> `j nnfolder:Inbox RET'?

Doing `j nnfolder:Inbox RET', Gnus shows me:

 K      *: Inbox


> And why do you have an Inbox with mairix?  If any mail gets split to
> there, it will get deleted when mairix recreates the search results.

I've no idea

thanks
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> On Thu, Dec 3, 2009 at 3:09 PM, Dan Christensen <jdc@uwo.ca> wrote:
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>>> (setq gnus-killed-list ... "nnfolder:Inbox" ... "nnml+mairix:Inbox" ...)
>>
>> That nnfolder:Inbox looks suspicious to me.  In the *Group* buffer,
>> can you find that group by typing either `9 L' or by typing
>> `j nnfolder:Inbox RET'?
>
> Doing `j nnfolder:Inbox RET', Gnus shows me:
>
>  K      *: Inbox

What happens if you subscribe to it (with `u') and try to enter it 
or update it (`ESC g')?

>> And why do you have an Inbox with mairix?  If any mail gets split to
>> there, it will get deleted when mairix recreates the search results.
>
> I've no idea

I suspect that your mail is getting sent to the wrong Inbox, but I don't
know how to debug it further.  Maybe you should grep through all of your
files for a string that is in one of the missing messages, right after
the message arrives?

Maybe you could change your split rules so that mail goes to a different
group, say "newInbox", and see if you still have this problem.

Dan

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Thu, Dec 3, 2009 at 11:10 PM, Dan Christensen <jdc@uwo.ca> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>> On Thu, Dec 3, 2009 at 3:09 PM, Dan Christensen <jdc@uwo.ca> wrote:
>>> Francis Moreau <francis.moro@gmail.com> writes:
>>>
>>>> (setq gnus-killed-list ... "nnfolder:Inbox" ... "nnml+mairix:Inbox" ...)
>>>
>>> That nnfolder:Inbox looks suspicious to me.  In the *Group* buffer,
>>> can you find that group by typing either `9 L' or by typing
>>> `j nnfolder:Inbox RET'?
>>
>> Doing `j nnfolder:Inbox RET', Gnus shows me:
>>
>>  K      *: Inbox
>
> What happens if you subscribe to it (with `u') and try to enter it
> or update it (`ESC g')?

Couldn't activate group nnfolder:Inbox: /home/fbuihuu/Mail/Inbox/ is a directory

This dead group exists probably because I was using nnfolder backend
first and then I switched to nnmail.

>
>>> And why do you have an Inbox with mairix?  If any mail gets split to
>>> there, it will get deleted when mairix recreates the search results.
>>
>> I've no idea
>
> I suspect that your mail is getting sent to the wrong Inbox, but I don't
> know how to debug it further.  Maybe you should grep through all of your
> files for a string that is in one of the missing messages, right after
> the message arrives?

I already did that and only Incoming files were including the missing mails.

>
> Maybe you could change your split rules so that mail goes to a different
> group, say "newInbox", and see if you still have this problem.

I still have the problem with a new rule like you described.

BTW, can you reproduce the issue on your side ?

Thanks
-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> BTW, can you reproduce the issue on your side ?

No, I can't, and mail loss is a serious issue that would be highly
visible on the list if it happened in ordinary situations.  It must 
be a result of your configuration (including the history of groups you
created and deleted), but I don't know how to debug it further.  If
you've just started using Gnus, you could try starting over from
scratch: export your mail to some format, delete all Gnus files
(including .newsrc* and the tree of mail folders), and import your mail
into a fresh set-up of Gnus.  You'll lose your marks, but maybe it's
worth it.

Dan

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Tue, Dec 8, 2009 at 3:06 PM, Dan Christensen <jdc@uwo.ca> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>> BTW, can you reproduce the issue on your side ?
>
> No, I can't, and mail loss is a serious issue that would be highly
> visible on the list if it happened in ordinary situations.

Well, the problem come up only when using a prefix.

>  It must
> be a result of your configuration (including the history of groups you
> created and deleted), but I don't know how to debug it further.

Probably, but I'm a poor Gnus user so I don't know how to debug this either...

> If
> you've just started using Gnus, you could try starting over from
> scratch: export your mail to some format, delete all Gnus files
> (including .newsrc* and the tree of mail folders), and import your mail
> into a fresh set-up of Gnus.  You'll lose your marks, but maybe it's
> worth it.

Ouch, that would be the last thing I would do.

I always felt that Gnus is a bit fragile, simply kill and create
(wrong) groups can mess up the whole configuration. That's sad.

-- 
Francis

Re: Issue with nnml backend and gnus-group-get-new-news()

[Francis Moreau]

On Tue, Dec 8, 2009 at 3:06 PM, Dan Christensen <jdc@uwo.ca> wrote:
> Francis Moreau <francis.moro@gmail.com> writes:
>
>> BTW, can you reproduce the issue on your side ?
>
> No, I can't, and mail loss is a serious issue that would be highly
> visible on the list if it happened in ordinary situations.  It must
> be a result of your configuration (including the history of groups you
> created and deleted), but I don't know how to debug it further.

Ok this is probably due to the history of groups I created and deleted
because I tried my configuration on another computer where gnus wasn't
installed before and it works as expected.

So gnus doesn't like either my newsrc.eld or my active file or some
other internal files...

-- 
Francis