Problem with nnml group

Problem with nnml group

[Leonidas Tsampros]

Hi fellow Gnusers,

today I'm facing my first serious problem with nnml and Gnus.

I'm using gnus-version "Gnus v5.13" that comes with emacs-version
"24.0.50.1" compiled on Thursday the 10th of March using the sources
from the bzr emacs trunk branch.

All of my sent mails are saved in nnml:outbox. I can confirm that this
directory on the local filesystem contains 467 mail files (plus .marks
and .overview).

Entering the group does not prompt me how many mails to fetch but
instead it goes straight to the Summary buffer showing the last 5 mail
stored in there. Pressing / O, renders the message "No old news".

I did try to recompact the group with no luck. I rebooted Gnus, but
again no luck. G E output follows:

("nnml:outbox" 3
 ((1 . 5))
 ((seen
   (1 . 5)
   472))
 "nnml:")

The marks file contains the following:
$ cat .marks
((read (1 . 5) (6 . 469)))

The overview file under outbox/ contains the overview lines for messages
1 to 5 which are indeed the 5 last messages I sent today. Since these
are the last messages I sent, I would expect them to be 470-475
instead. That means, that Gnus got confused today and overwrote 5 old
sent mails of mine.

I'm wondering what the heck went wrong. Should I rename the mail files
from 1 to 5 to 470-475 and try to edit the group configuration manually?

I'm completely at a loss at what maybe the error here and I'm afraid
that everytimeI send new mail that gets saved in the outbox will
overwrite older sent mail.

Apart from not knowing the trigger of this behaviour, I'm very anxious
over this.

Is there anyway to 'reset' this group somehow?

Do you have any remedy for this situation ?

Thanks,
Leonidas

Re: Problem with nnml group

[Lars Magne Ingebrigtsen]

Leonidas Tsampros <ltsampros@upnet.gr> writes:

> The overview file under outbox/ contains the overview lines for messages
> 1 to 5 which are indeed the 5 last messages I sent today. Since these
> are the last messages I sent, I would expect them to be 470-475
> instead. That means, that Gnus got confused today and overwrote 5 old
> sent mails of mine.

Are you sure that articles 1-5 really did exist before?  I think there's
a protection in nnml somewhere against overwriting files.

Anyway, it sounds like nnml thinks that there's only five messages in
the group, for some reason.  What does ~/Mail/active say about the
group?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Problem with nnml group

[Leonidas Tsampros]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Leonidas Tsampros <ltsampros@upnet.gr> writes:
>
>> The overview file under outbox/ contains the overview lines for messages
>> 1 to 5 which are indeed the 5 last messages I sent today. Since these
>> are the last messages I sent, I would expect them to be 470-475
>> instead. That means, that Gnus got confused today and overwrote 5 old
>> sent mails of mine.
>
> Are you sure that articles 1-5 really did exist before?  I think there's
> a protection in nnml somewhere against overwriting files.
>
> Anyway, it sounds like nnml thinks that there's only five messages in
> the group, for some reason.  What does ~/Mail/active say about the
> group?

Hi Lars,

thanks for the answer. I've been meaning to take a look at the relevant
code but I have no time currently.

I think the trigger of the issue is a combination of the below:

1) (setq nnml-directory "~/Mail.Gnus")
   (setq mail-source-delete-incoming t)
   (setq mail-sources '((directory :path "~/Mail"
                                   :suffix "")))
   (setq gnus-message-archive-group "nnml:outbox")

2) fire up emacs/gnus from a directory other than the home
directory (that was the only different parameter in contrast with my
everyday Gnus-workflow).

3) Move files from the nnml:outbox to another group, thus creating holes
in the article ranges.

However, I've not been up to the task of trying to reproduce the bug
yet, so all the above is just speculation.

With regards to the active file, I remember that it was completely wrong
and I removed the relevant entry. Then I edited the group and marks by
hand to reflect the correct values, restarted Gnus and this fixed the
issue for me.

Now G E on nnml:outbox reports: 

("nnml:outbox" 3
 ((1 . 484))
 ((seen
   (1 . 484)))
 "nnml:")

The outbox/.marks file:
((read (1 . 484)))

My Mail.Gnus/active file:
outbox 484 1 y

and I can confirm that new mail gets saved successfully without
overwriting old mail.

However, going back to the original problem, I cannot guarantee that
mails 1-5 existed. I had done a mass mail move recently to nnimap but I
did not meddle with such old mails (I only moved specific mails after
01/09/2010).

Now my only remaining problem is that when I enter the group I get 227
emails whereas the outbox folder itself reports:

[ltsampros@laptop]outbox $ ls -1 | sort -n | wc -l
483

and the mode line writes:
nnml:outbox [470] Bot (227,0) (Summary Plugged)

I can notice that the only mails that are being displayed have sent
dates gt 01/01/2011.

Upon entering the group, I think the total number is also displayed
correctly:

How many articles from outbox (available 484, default 200) however
whatever number I enter there a completely arbitrary number of articles
appear on the summary buffer (e.g. for the default number 200, only the
last 15 mails appear).

I really cannot understand what is going on. Fortunately, I have kept a
backup since the problem appeared.

Best Regards,
Leonidas Tsampros

Re: Problem with nnml group

[Leonidas Tsampros]

Leonidas Tsampros <ltsampros@upnet.gr> writes:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Leonidas Tsampros <ltsampros@upnet.gr> writes:
>>
>>> The overview file under outbox/ contains the overview lines for messages
>>> 1 to 5 which are indeed the 5 last messages I sent today. Since these
>>> are the last messages I sent, I would expect them to be 470-475
>>> instead. That means, that Gnus got confused today and overwrote 5 old
>>> sent mails of mine.
>>
>> Are you sure that articles 1-5 really did exist before?  I think there's
>> a protection in nnml somewhere against overwriting files.
>>
>> Anyway, it sounds like nnml thinks that there's only five messages in
>> the group, for some reason.  What does ~/Mail/active say about the
>> group?
>
> Hi Lars,
>
> thanks for the answer. I've been meaning to take a look at the relevant
> code but I have no time currently.
>
> I think the trigger of the issue is a combination of the below:
>
> 1) (setq nnml-directory "~/Mail.Gnus")
>    (setq mail-source-delete-incoming t)
>    (setq mail-sources '((directory :path "~/Mail"
>                                    :suffix "")))
>    (setq gnus-message-archive-group "nnml:outbox")
>
> 2) fire up emacs/gnus from a directory other than the home
> directory (that was the only different parameter in contrast with my
> everyday Gnus-workflow).
>
> 3) Move files from the nnml:outbox to another group, thus creating holes
> in the article ranges.
>
> However, I've not been up to the task of trying to reproduce the bug
> yet, so all the above is just speculation.
>
> With regards to the active file, I remember that it was completely wrong
> and I removed the relevant entry. Then I edited the group and marks by
> hand to reflect the correct values, restarted Gnus and this fixed the
> issue for me.
>
> Now G E on nnml:outbox reports: 
>
> ("nnml:outbox" 3
>  ((1 . 484))
>  ((seen
>    (1 . 484)))
>  "nnml:")
>
> The outbox/.marks file:
> ((read (1 . 484)))
>
> My Mail.Gnus/active file:
> outbox 484 1 y
>
> and I can confirm that new mail gets saved successfully without
> overwriting old mail.
>
> However, going back to the original problem, I cannot guarantee that
> mails 1-5 existed. I had done a mass mail move recently to nnimap but I
> did not meddle with such old mails (I only moved specific mails after
> 01/09/2010).
>
> Now my only remaining problem is that when I enter the group I get 227
> emails whereas the outbox folder itself reports:
>
> [ltsampros@laptop]outbox $ ls -1 | sort -n | wc -l
> 483
>
> and the mode line writes:
> nnml:outbox [470] Bot (227,0) (Summary Plugged)
>
> I can notice that the only mails that are being displayed have sent
> dates gt 01/01/2011.
>
> Upon entering the group, I think the total number is also displayed
> correctly:
>
> How many articles from outbox (available 484, default 200) however
> whatever number I enter there a completely arbitrary number of articles
> appear on the summary buffer (e.g. for the default number 200, only the
> last 15 mails appear).
>
> I really cannot understand what is going on. Fortunately, I have kept a
> backup since the problem appeared.
>
> Best Regards,
> Leonidas Tsampros

Replying to my previous email with some more information:

[ltsampros@laptop]outbox $ ls -1 | grep -vi new | sort -n | awk -F" " '{sum = sum + $0} END {print sum}'
117121

The sum of all the article numbers in the outbox is 117121 while it
should have been 117855 (sum of all integers between 1 and 485).

Which means there is more than 1 mail missing from that folder. Do you
think that such article holes play a role in the above problems I'm
facing?

Thanks,
Leonidas Tsampros

Re: Problem with nnml group

[Lars Magne Ingebrigtsen]

Leonidas Tsampros <ltsampros@upnet.gr> writes:

> How many articles from outbox (available 484, default 200) however
> whatever number I enter there a completely arbitrary number of articles
> appear on the summary buffer (e.g. for the default number 200, only the
> last 15 mails appear).

That's normal.  Gnus doesn't know what messages actually exist, so it
will just request the 200 highest-numbered messages.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen