mismatching summary buffer lines and article content?

mismatching summary buffer lines and article content?

[Simon Josefsson]

Since I started to use the latest Gnus versions I've noticed that
sometimes the summary buffer author/subject/etc line is out of sync with
the article content.  Sometimes restarting Gnus has been sufficient to
solve the problem, but I now have one article that is incorrect every
time.  It is in a nnimap group, and I'm using the agent on it.  The
article _may_ have been downloaded through a 'J u' or 'J s' but I am not
certain.  Before I solved the problem by closing Gnus and 'rm -rf
~/News/agent/nnimap/foo/bar' and let Gnus rebuild the local cache, but
it doesn't feel like a good solution.

/Simon

Re: mismatching summary buffer lines and article content?

[Steinar Bang]

>>>>> Simon Josefsson <simon@josefsson.org>:

> Since I started to use the latest Gnus versions I've noticed that
> sometimes the summary buffer author/subject/etc line is out of sync
> with the article content.  Sometimes restarting Gnus has been
> sufficient to solve the problem, but I now have one article that is
> incorrect every time.  It is in a nnimap group, and I'm using the
> agent on it.

Right.

That's one of the two nnimap agent issues I see:
 1. sometimes the wrong article is cached
 2. sometimes the NOV file, or part of the NOV file, is cached in an
    article 

I've tried debugging it.  Unfortunately I don't have a reproducable
way of triggering the issue, so it's hard.  There is a thread in this
group where I've put what I've seen (or think I've seen) when debugging
this issue.  I'm not sure if there is any real value in the information
in that thread, though.

I have this setting for caching articles when reading them:
 ;; Store article in agent when reading it while online
 (add-hook 'gnus-select-article-hook 'gnus-agent-fetch-selected-article)

No idea if this has any effect or not.

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <simon@josefsson.org>:
>
>> Since I started to use the latest Gnus versions I've noticed that
>> sometimes the summary buffer author/subject/etc line is out of sync
>> with the article content.  Sometimes restarting Gnus has been
>> sufficient to solve the problem, but I now have one article that is
>> incorrect every time.  It is in a nnimap group, and I'm using the
>> agent on it.
>
> Right.
>
> That's one of the two nnimap agent issues I see:
>  1. sometimes the wrong article is cached
>  2. sometimes the NOV file, or part of the NOV file, is cached in an
>     article 
>
> I've tried debugging it.  Unfortunately I don't have a reproducable
> way of triggering the issue, so it's hard.  There is a thread in this
> group where I've put what I've seen (or think I've seen) when debugging
> this issue.  I'm not sure if there is any real value in the information
> in that thread, though.
>
> I have this setting for caching articles when reading them:
>  ;; Store article in agent when reading it while online
>  (add-hook 'gnus-select-article-hook 'gnus-agent-fetch-selected-article)
>
> No idea if this has any effect or not.

Yes, I'm using the same hook.  Sometimes I have the feeling that it is
also related to async pre-fetching, but I'm not sure -- sometimes the
article content has been garbage looking like NOV lines.  That's when I
restart Gnus and it is gone.

/Simon

Re: mismatching summary buffer lines and article content?

[Eric S Fraga]

Simon Josefsson <simon@josefsson.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>>>>>> Simon Josefsson <simon@josefsson.org>:
>>
>>> Since I started to use the latest Gnus versions I've noticed that
>>> sometimes the summary buffer author/subject/etc line is out of sync
>>> with the article content.  Sometimes restarting Gnus has been
>>> sufficient to solve the problem, but I now have one article that is
>>> incorrect every time.  It is in a nnimap group, and I'm using the
>>> agent on it.

[...]

> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
> also related to async pre-fetching, but I'm not sure -- sometimes the
> article content has been garbage looking like NOV lines.  That's when I
> restart Gnus and it is gone.
>
> /Simon
>

Just to add a data point: I also see this sporadically but not often
and, luckily, to date, I have been able to retrieve the original article
by re-starting gnus.  I also cannot reproduce it on demand
unfortunately.

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1 + No Gnus v0.14

Re: mismatching summary buffer lines and article content?

[Steinar Bang]

>>>>> Eric S Fraga <e.fraga@ucl.ac.uk>:

> Just to add a data point: I also see this sporadically but not often
> and, luckily, to date, I have been able to retrieve the original
> article by re-starting gnus.

I usually get the original article by deleting the wrong article in the
agent cache.

Re: mismatching summary buffer lines and article content?

[Lars Magne Ingebrigtsen]

Simon Josefsson <simon@josefsson.org> writes:

> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
> also related to async pre-fetching, but I'm not sure -- sometimes the
> article content has been garbage looking like NOV lines.  That's when I
> restart Gnus and it is gone.

Yeah, I've seen the "article not matching up with the summary buffer"
thing in NNTP groups, too.  But I've been totally unable to reproduce
it, and I haven't seen any obvious bugs in the code, either, so it's
really difficult to fix.

My main suspect is the async pre-fetch code, too, but there may also be
interactions between the Agent and the pre-fetch code.  Oy vey.

Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
should default to t...  Anyway, could you try setting that to nil and
see whether the problems disappear?  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>> also related to async pre-fetching, but I'm not sure -- sometimes the
>> article content has been garbage looking like NOV lines.  That's when I
>> restart Gnus and it is gone.
>
> Yeah, I've seen the "article not matching up with the summary buffer"
> thing in NNTP groups, too.  But I've been totally unable to reproduce
> it, and I haven't seen any obvious bugs in the code, either, so it's
> really difficult to fix.
>
> My main suspect is the async pre-fetch code, too, but there may also be
> interactions between the Agent and the pre-fetch code.  Oy vey.
>
> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
> should default to t...  Anyway, could you try setting that to nil and
> see whether the problems disappear?  

I can try -- however I have tried another thing for a while now that
seems to have made the problem go away: removing
gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
removed the hook on 28 February and I don't recall seeing the problem
since then.

I'll add the hook back now and see if I start to notice the problem
again.  When I have seen the problem a few times, I'll disable
gnus-asynchronous to see if that also independently fixes the problem.

/Simon

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Simon Josefsson <simon@josefsson.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Simon Josefsson <simon@josefsson.org> writes:
>>
>>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>>> also related to async pre-fetching, but I'm not sure -- sometimes the
>>> article content has been garbage looking like NOV lines.  That's when I
>>> restart Gnus and it is gone.
>>
>> Yeah, I've seen the "article not matching up with the summary buffer"
>> thing in NNTP groups, too.  But I've been totally unable to reproduce
>> it, and I haven't seen any obvious bugs in the code, either, so it's
>> really difficult to fix.
>>
>> My main suspect is the async pre-fetch code, too, but there may also be
>> interactions between the Agent and the pre-fetch code.  Oy vey.
>>
>> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
>> should default to t...  Anyway, could you try setting that to nil and
>> see whether the problems disappear?  
>
> I can try -- however I have tried another thing for a while now that
> seems to have made the problem go away: removing
> gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
> removed the hook on 28 February and I don't recall seeing the problem
> since then.
>
> I'll add the hook back now and see if I start to notice the problem
> again.  When I have seen the problem a few times, I'll disable
> gnus-asynchronous to see if that also independently fixes the problem.

Re-enabling gnus-agent-fetch-selected-article made the problem come
back: I have seen the problem several times since the email above on
March 7th.

I have now disabled gnus-asynchronous.

/Simon

Re: mismatching summary buffer lines and article content?

[Ludovic Courtès]

Hi,

Simon Josefsson <simon@josefsson.org> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Simon Josefsson <simon@josefsson.org> writes:
>>>
>>>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>>>> also related to async pre-fetching, but I'm not sure -- sometimes the
>>>> article content has been garbage looking like NOV lines.  That's when I
>>>> restart Gnus and it is gone.
>>>
>>> Yeah, I've seen the "article not matching up with the summary buffer"
>>> thing in NNTP groups, too.  But I've been totally unable to reproduce
>>> it, and I haven't seen any obvious bugs in the code, either, so it's
>>> really difficult to fix.
>>>
>>> My main suspect is the async pre-fetch code, too, but there may also be
>>> interactions between the Agent and the pre-fetch code.  Oy vey.
>>>
>>> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
>>> should default to t...  Anyway, could you try setting that to nil and
>>> see whether the problems disappear?  
>>
>> I can try -- however I have tried another thing for a while now that
>> seems to have made the problem go away: removing
>> gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
>> removed the hook on 28 February and I don't recall seeing the problem
>> since then.
>>
>> I'll add the hook back now and see if I start to notice the problem
>> again.  When I have seen the problem a few times, I'll disable
>> gnus-asynchronous to see if that also independently fixes the problem.
>
> Re-enabling gnus-agent-fetch-selected-article made the problem come
> back: I have seen the problem several times since the email above on
> March 7th.
>
> I have now disabled gnus-asynchronous.

I have ‘gnus-asynchronous’ to nil and I see the problem from time to
time (on an nnimap group with the agent enabled.)

Thanks,
Ludo’.

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi,
>
> Simon Josefsson <simon@josefsson.org> writes:
>
>> Simon Josefsson <simon@josefsson.org> writes:
>>
>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>
>>>> Simon Josefsson <simon@josefsson.org> writes:
>>>>
>>>>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>>>>> also related to async pre-fetching, but I'm not sure -- sometimes the
>>>>> article content has been garbage looking like NOV lines.  That's when I
>>>>> restart Gnus and it is gone.
>>>>
>>>> Yeah, I've seen the "article not matching up with the summary buffer"
>>>> thing in NNTP groups, too.  But I've been totally unable to reproduce
>>>> it, and I haven't seen any obvious bugs in the code, either, so it's
>>>> really difficult to fix.
>>>>
>>>> My main suspect is the async pre-fetch code, too, but there may also be
>>>> interactions between the Agent and the pre-fetch code.  Oy vey.
>>>>
>>>> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
>>>> should default to t...  Anyway, could you try setting that to nil and
>>>> see whether the problems disappear?  
>>>
>>> I can try -- however I have tried another thing for a while now that
>>> seems to have made the problem go away: removing
>>> gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
>>> removed the hook on 28 February and I don't recall seeing the problem
>>> since then.
>>>
>>> I'll add the hook back now and see if I start to notice the problem
>>> again.  When I have seen the problem a few times, I'll disable
>>> gnus-asynchronous to see if that also independently fixes the problem.
>>
>> Re-enabling gnus-agent-fetch-selected-article made the problem come
>> back: I have seen the problem several times since the email above on
>> March 7th.
>>
>> I have now disabled gnus-asynchronous.
>
> I have ‘gnus-asynchronous’ to nil and I see the problem from time to
> time (on an nnimap group with the agent enabled.)

Do you have the gnus-agent-fetch-selected-article hook enabled in
gnus-select-article-hook?

/Simon

Re: mismatching summary buffer lines and article content?

[Ludovic Courtès]

Simon Josefsson <simon@josefsson.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi,
>>
>> Simon Josefsson <simon@josefsson.org> writes:
>>
>>> Simon Josefsson <simon@josefsson.org> writes:
>>>
>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>
>>>>> Simon Josefsson <simon@josefsson.org> writes:
>>>>>
>>>>>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>>>>>> also related to async pre-fetching, but I'm not sure -- sometimes the
>>>>>> article content has been garbage looking like NOV lines.  That's when I
>>>>>> restart Gnus and it is gone.
>>>>>
>>>>> Yeah, I've seen the "article not matching up with the summary buffer"
>>>>> thing in NNTP groups, too.  But I've been totally unable to reproduce
>>>>> it, and I haven't seen any obvious bugs in the code, either, so it's
>>>>> really difficult to fix.
>>>>>
>>>>> My main suspect is the async pre-fetch code, too, but there may also be
>>>>> interactions between the Agent and the pre-fetch code.  Oy vey.
>>>>>
>>>>> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
>>>>> should default to t...  Anyway, could you try setting that to nil and
>>>>> see whether the problems disappear?  
>>>>
>>>> I can try -- however I have tried another thing for a while now that
>>>> seems to have made the problem go away: removing
>>>> gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
>>>> removed the hook on 28 February and I don't recall seeing the problem
>>>> since then.
>>>>
>>>> I'll add the hook back now and see if I start to notice the problem
>>>> again.  When I have seen the problem a few times, I'll disable
>>>> gnus-asynchronous to see if that also independently fixes the problem.
>>>
>>> Re-enabling gnus-agent-fetch-selected-article made the problem come
>>> back: I have seen the problem several times since the email above on
>>> March 7th.
>>>
>>> I have now disabled gnus-asynchronous.
>>
>> I have ‘gnus-asynchronous’ to nil and I see the problem from time to
>> time (on an nnimap group with the agent enabled.)
>
> Do you have the gnus-agent-fetch-selected-article hook enabled in
> gnus-select-article-hook?

Yes.

Ludo’.

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Now the problem has happened with gnus-asynchronous set to nil too, so I
think we can rule that out.  The problem starts when
gnus-select-article-hook contains gnus-agent-fetch-selected-article.

Now I'll disable it and hopefully won't see the problem again...

/Simon

Re: mismatching summary buffer lines and article content?

[Lars Magne Ingebrigtsen]

Simon Josefsson <simon@josefsson.org> writes:

> Now the problem has happened with gnus-asynchronous set to nil too, so I
> think we can rule that out.  The problem starts when
> gnus-select-article-hook contains gnus-agent-fetch-selected-article.
>
> Now I'll disable it and hopefully won't see the problem again...

Keep us posted.  :-)

I've had a peek at that function, and I can't see anything obviously
wrong in what it's doing, but it's a long and kinda complicated
function... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mismatching summary buffer lines and article content?

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> I've had a peek at that function, and I can't see anything obviously
> wrong in what it's doing, but it's a long and kinda complicated
> function...

I've tried debugging into it multiple times, but I haven't seen anything
conclusive, other than that it doesn't work right.

By "work right" I mean "pull down the article and save it, if it hasn't
been pulled down already".

Sometimes it does exactly that.  Sometimes it does nothing at all,
sometimes it pulls down and saves the wrong article... and finally:
sometimes it saves NOV information as the article.

What I've seen when stepping through it, are: sometimes it doesn't
download an article, and that sometimes it downloads an article.  I
don't think I've seen it pull down the wrong article while debugging,
and I definitely haven't seen it put NOV data into the file, while
debugging.

Re: mismatching summary buffer lines and article content?

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Sometimes it does exactly that.  Sometimes it does nothing at all,
> sometimes it pulls down and saves the wrong article... and finally:
> sometimes it saves NOV information as the article.

Hm.  It sounds like it might be a race condition of some kind, perhaps?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mismatching summary buffer lines and article content?

[Eric S Fraga]

Simon Josefsson <simon@josefsson.org> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Simon Josefsson <simon@josefsson.org> writes:
>>>
>>>> Yes, I'm using the same hook.  Sometimes I have the feeling that it is
>>>> also related to async pre-fetching, but I'm not sure -- sometimes the
>>>> article content has been garbage looking like NOV lines.  That's when I
>>>> restart Gnus and it is gone.
>>>
>>> Yeah, I've seen the "article not matching up with the summary buffer"
>>> thing in NNTP groups, too.  But I've been totally unable to reproduce
>>> it, and I haven't seen any obvious bugs in the code, either, so it's
>>> really difficult to fix.
>>>
>>> My main suspect is the async pre-fetch code, too, but there may also be
>>> interactions between the Agent and the pre-fetch code.  Oy vey.
>>>
>>> Oh, I see that `gnus-asynchronous' defaults to nil?  Hm, I think it
>>> should default to t...  Anyway, could you try setting that to nil and
>>> see whether the problems disappear?  
>>
>> I can try -- however I have tried another thing for a while now that
>> seems to have made the problem go away: removing
>> gnus-agent-fetch-selected-article from gnus-select-article-hook.  I
>> removed the hook on 28 February and I don't recall seeing the problem
>> since then.
>>
>> I'll add the hook back now and see if I start to notice the problem
>> again.  When I have seen the problem a few times, I'll disable
>> gnus-asynchronous to see if that also independently fixes the problem.
>
> Re-enabling gnus-agent-fetch-selected-article made the problem come
> back: I have seen the problem several times since the email above on
> March 7th.
>
> I have now disabled gnus-asynchronous.
>
> /Simon
>

I have an email now that I cannot read with similar symptoms to those
described in this thread.  I did have (until a few minutes ago) the
gnus-agent-fetch-selected-article in the gnus-select-article-hook;
gnus-asynchronous was nil and remains so.

The problem is: how can I get my email back?  Is there something I can
clear on my system or can I kick gnus somewhere to make it reload my
messages from the server?

Thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.55.g87c42.dirty)

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I have an email now that I cannot read with similar symptoms to those
> described in this thread.  I did have (until a few minutes ago) the
> gnus-agent-fetch-selected-article in the gnus-select-article-hook;
> gnus-asynchronous was nil and remains so.
>
> The problem is: how can I get my email back?  Is there something I can
> clear on my system or can I kick gnus somewhere to make it reload my
> messages from the server?

Yes, I typically just remove ~/News/agent/nnimap/SERVER/GROUP/ and let
Gnus re-create it.

/Simon

Re: mismatching summary buffer lines and article content?

[Eric S Fraga]

Simon Josefsson <simon@josefsson.org> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> I have an email now that I cannot read with similar symptoms to those
>> described in this thread.  I did have (until a few minutes ago) the
>> gnus-agent-fetch-selected-article in the gnus-select-article-hook;
>> gnus-asynchronous was nil and remains so.
>>
>> The problem is: how can I get my email back?  Is there something I can
>> clear on my system or can I kick gnus somewhere to make it reload my
>> messages from the server?
>
> Yes, I typically just remove ~/News/agent/nnimap/SERVER/GROUP/ and let
> Gnus re-create it.
>
> /Simon
>

Thanks.  I think I'll remove all the groups for one server as things
seem to have gone crazy in the past two days: a large percentage
(i.e. >1%) of my emails seem to be screwed up.  They are fine on the
server (verified using a webmail interface) so it must be the agent?

Is there any way to debug the agent system that I can help with?

The only change I have made recently to my configuration is to have GCC
messages automatically marked as read.  I used to have this set before
(several months ago now) but unset it because I believe it was causing
problems.  I think I am going to unset it again as it seems to have some
interaction with the agent system.

Thanks again,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1 + No Gnus v0.14

Re: mismatching summary buffer lines and article content?

[Lars Magne Ingebrigtsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Thanks.  I think I'll remove all the groups for one server as things
> seem to have gone crazy in the past two days: a large percentage
> (i.e. >1%) of my emails seem to be screwed up.  They are fine on the
> server (verified using a webmail interface) so it must be the agent?

Yes.

> Is there any way to debug the agent system that I can help with?

If you can come up with a recipe for how to reproduce the bug, then that
would be helpful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mismatching summary buffer lines and article content?

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> Thanks.  I think I'll remove all the groups for one server as things
>> seem to have gone crazy in the past two days: a large percentage
>> (i.e. >1%) of my emails seem to be screwed up.  They are fine on the
>> server (verified using a webmail interface) so it must be the agent?
>
> Yes.
>
>> Is there any way to debug the agent system that I can help with?
>
> If you can come up with a recipe for how to reproduce the bug, then that
> would be helpful.

Adding gnus-agent-fetch-selected-article to gnus-select-article-hook
will cause the problem to have happened a few times after a few days of
normal Gnus usage.

:-)

/Simon