Updated IMAP unread count correction

Updated IMAP unread count correction

[Dave Abrahams]

Hi All,

I recently had to rewrite the code at
http://www.emacswiki.org/emacs/GnusNiftyTricks#toc8 in order to remove
noise (huge unread counts) from my Group buffer.  If you follow the link
you'll see the rewrite.  I'm writing to request that this functionality
be incorporated into Gnus, so it won't break for users when Gnus is
updated.

Thanks,

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Updated IMAP unread count correction

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> I recently had to rewrite the code at
> http://www.emacswiki.org/emacs/GnusNiftyTricks#toc8 in order to remove
> noise (huge unread counts) from my Group buffer.  If you follow the link
> you'll see the rewrite.  I'm writing to request that this functionality
> be incorporated into Gnus, so it won't break for users when Gnus is
> updated.

I'm not quite sure what you mean here.  Ma Gnus keeps track of
unexisting messages in nnimap groups, so all the unread counts should be
correct...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Sun Jun 10 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> I recently had to rewrite the code at
>> http://www.emacswiki.org/emacs/GnusNiftyTricks#toc8 in order to remove
>> noise (huge unread counts) from my Group buffer.  If you follow the link
>> you'll see the rewrite.  I'm writing to request that this functionality
>> be incorporated into Gnus, so it won't break for users when Gnus is
>> updated.
>
> I'm not quite sure what you mean here.  Ma Gnus keeps track of
> unexisting messages in nnimap groups, so all the unread counts should be
> correct...

Well, for me, they weren't.  It may simply be a matter of Ma Gnus'
remembered state not corresponding to the state on the server; I don't
know.  I do use other mail clients, too, FWIW.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Updated IMAP unread count correction

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> Well, for me, they weren't.  It may simply be a matter of Ma Gnus'
> remembered state not corresponding to the state on the server; I don't
> know.  I do use other mail clients, too, FWIW.

Are you able to reproduce what gets `unexist' tracking to become faulty?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Sun Jun 10 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> Well, for me, they weren't.  It may simply be a matter of Ma Gnus'
>> remembered state not corresponding to the state on the server; I don't
>> know.  I do use other mail clients, too, FWIW.
>
> Are you able to reproduce what gets `unexist' tracking to become faulty?

I'm afraid I have no idea how to reproduce it, nor do I know how it
happened in the first place.  And since I am using the referenced code,
I never see an incorrect count anymore, so I wouldn't notice if it came
back.  I could turn that code off, and we'd surely see the wrong counts
again, but I don't know that it would help us find the cause.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Updated IMAP unread count correction

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> I'm afraid I have no idea how to reproduce it, nor do I know how it
> happened in the first place.  And since I am using the referenced code,
> I never see an incorrect count anymore, so I wouldn't notice if it came
> back.  I could turn that code off, and we'd surely see the wrong counts
> again, but I don't know that it would help us find the cause.

Well, without running the default code, we'll never find out.  I'm not
able to reproduce the problem.

`G E' on a group will show you what messages Gnus thinks don't exist on
the server.  (In the `unexist' list.)  If that gets out of sync,
something wrong happened.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Sun Jun 10 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> I'm afraid I have no idea how to reproduce it, nor do I know how it
>> happened in the first place.  And since I am using the referenced code,
>> I never see an incorrect count anymore, so I wouldn't notice if it came
>> back.  I could turn that code off, and we'd surely see the wrong counts
>> again, but I don't know that it would help us find the cause.
>
> Well, without running the default code, we'll never find out.  

The you mean the standard code?

It won't do any good at all to run that code unless I can get it to
report the right number, so that I'll notice when it reports the wrong
one.  If you can tell me how to get things back in sync, I can try
running the normal code again.  

[Incidentally, one thing I dearly wish for in Gnus is the ability to say
"throw out all the information you have that can be derived from the
server (and re-derive it on demand)."  Gnus caches all kinids of things
in all kinds of ways and when things get out-of-sync, it's always a
nightmare for me.]

I don't know if it helps, but I am using a local dovecot with
offlineimap and recently had to rebuild the local mailstore.  Thus the
UIDs of messages on the server could easily have been changed.

> I'm not able to reproduce the problem.
>
> `G E' on a group will show you what messages Gnus thinks don't exist on
> the server.  (In the `unexist' list.)  If that gets out of sync,
> something wrong happened.

`G E' on my INBOX currently shows more than 800 "seen" entries and about
3 in the unexist list.  There are only 3 messages actually existing in
that mailbox.  Not sure what that tells you, though.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Updated IMAP unread count correction

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> It won't do any good at all to run that code unless I can get it to
> report the right number, so that I'll notice when it reports the wrong
> one.  If you can tell me how to get things back in sync, I can try
> running the normal code again.  

(gnus-clean-old-newsrc t) and then `g'.

> [Incidentally, one thing I dearly wish for in Gnus is the ability to say
> "throw out all the information you have that can be derived from the
> server (and re-derive it on demand)."

`M-g' on each group will do that.

> I don't know if it helps, but I am using a local dovecot with
> offlineimap and recently had to rebuild the local mailstore.  Thus the
> UIDs of messages on the server could easily have been changed.

Gnus should detect that and do a full rescan.  Unless the IMAP server
doesn't change its UIDVALIDITY...

>> `G E' on a group will show you what messages Gnus thinks don't exist on
>> the server.  (In the `unexist' list.)  If that gets out of sync,
>> something wrong happened.
>
> `G E' on my INBOX currently shows more than 800 "seen" entries and about
> 3 in the unexist list.  There are only 3 messages actually existing in
> that mailbox.  Not sure what that tells you, though.

It tells me that your .newsrc.eld data is messed up.  :-) But not much
more.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Updated IMAP unread count correction

[Julien Cubizolles]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dave Abrahams <dave@boostpro.com> writes:

>> If you can tell me how to get things back in sync, I can try running
>> the normal code again.
>
> (gnus-clean-old-newsrc t) and then `g'.

I'm tempted to use it to clean some strange counts I have remaining on
some IMAP accounts but won't it clear the all the marks in all the
groups (newsgroup and IMAP alike) ?

>> [Incidentally, one thing I dearly wish for in Gnus is the ability to say
>> "throw out all the information you have that can be derived from the
>> server (and re-derive it on demand)."
>
> `M-g' on each group will do that.

Correct me if I'm wrong:

(gnus-clean-old-newsrc t) and then `g' will "refresh" all groups

whereas

(gnus-clean-old-newsrc t) and then `M-g' will "refresh" only the group
the pointer/topic is on ?

Julien.

Re: Updated IMAP unread count correction

[Lars Ingebrigtsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Correct me if I'm wrong:
>
> (gnus-clean-old-newsrc t) and then `g' will "refresh" all groups
>
> whereas
>
> (gnus-clean-old-newsrc t) and then `M-g' will "refresh" only the group
> the pointer/topic is on ?

Yes.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Sun Jun 10 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> It won't do any good at all to run that code unless I can get it to
>> report the right number, so that I'll notice when it reports the wrong
>> one.  If you can tell me how to get things back in sync, I can try
>> running the normal code again.  
>
> (gnus-clean-old-newsrc t) and then `g'.

OK, I'll try that out later today.

>> [Incidentally, one thing I dearly wish for in Gnus is the ability to say
>> "throw out all the information you have that can be derived from the
>> server (and re-derive it on demand)."
>
> `M-g' on each group will do that.

That couldn't possibly fix things like incorrect information in the
agent, the gnus cache, and the registry, could it?  What use would those
stores be if they disappeared the moment you hit `M-g'?

>> I don't know if it helps, but I am using a local dovecot with
>> offlineimap and recently had to rebuild the local mailstore.  Thus the
>> UIDs of messages on the server could easily have been changed.
>
> Gnus should detect that and do a full rescan.  Unless the IMAP server
> doesn't change its UIDVALIDITY...

I don't know what that is, but FWIW, I did a fresh rebuild / reinstall
of dovecot to get to this point.

>>> `G E' on a group will show you what messages Gnus thinks don't exist on
>>> the server.  (In the `unexist' list.)  If that gets out of sync,
>>> something wrong happened.
>>
>> `G E' on my INBOX currently shows more than 800 "seen" entries and about
>> 3 in the unexist list.  There are only 3 messages actually existing in
>> that mailbox.  Not sure what that tells you, though.
>
> It tells me that your .newsrc.eld data is messed up.  :-) But not much
> more.

Right.  It would be awesome if Gnus could implement some optional
internal checks for such data getting out-of-sync.  I'd be willing to
run with such checks on for a couple weeks, even if it slowed Gnus down,
if it would help us track down some of these issues.

BTW, the need to `C-g' to get out of apparently-hung nntp server
connections is still de rigeur for me.  If that is liable to mess up
Gnus' state, maybe we could do something to make that more reliable?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: Updated IMAP unread count correction

[Lars Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

>> `M-g' on each group will do that.
>
> That couldn't possibly fix things like incorrect information in the
> agent, the gnus cache, and the registry, could it?  What use would those
> stores be if they disappeared the moment you hit `M-g'?

No, `M-g' doesn't do anything with the other stores.

> BTW, the need to `C-g' to get out of apparently-hung nntp server
> connections is still de rigeur for me.  If that is liable to mess up
> Gnus' state, maybe we could do something to make that more reliable?

It's very unlikely to mess up anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Tue Jun 12 2012, Dave Abrahams <dave-AT-boostpro.com> wrote:


on Wed Sep 05 2012, Lars Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:

<restoring context>

>>> [Incidentally, one thing I dearly wish for in Gnus is the ability to say
>>> "throw out all the information you have that can be derived from the
>>> server (and re-derive it on demand)."
>>
>>> `M-g' on each group will do that.
>>
>> That couldn't possibly fix things like incorrect information in the
>> agent, the gnus cache, and the registry, could it?  What use would those
>> stores be if they disappeared the moment you hit `M-g'?
>
> No, `M-g' doesn't do anything with the other stores.

So, again, I dearly wish for a way to flush all that state so I can get
back to a "clean state."

>> BTW, the need to `C-g' to get out of apparently-hung nntp server
>> connections is still de rigeur for me.  If that is liable to mess up
>> Gnus' state, maybe we could do something to make that more reliable?
>
> It's very unlikely to mess up anything.

You said the opposite in
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9075#16

So, I don't mean to be rude, but... which is it?

BTW, I'm considered something of an expert in error recovery.  It's
nothing earth-shaking, but I might be able to help you get this under
control if it isn't already.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Updated IMAP unread count correction

[Lars Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> So, again, I dearly wish for a way to flush all that state so I can get
> back to a "clean state."

Yes, that would be nice.  But no such thing exists.

>> It's very unlikely to mess up anything.
>
> You said the opposite in
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9075#16
>
> So, I don't mean to be rude, but... which is it?

I thought you were talking about `C-g'-ing while Gnus was working.  When
it's waiting for network traffic, it's unlikely to be doing something
destructive that'll mess up anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Wed Sep 05 2012, Lars Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> So, again, I dearly wish for a way to flush all that state so I can get
>> back to a "clean state."
>
> Yes, that would be nice.  But no such thing exists.

Can you give any hints about how to get there?

>>> It's very unlikely to mess up anything.
>>
>> You said the opposite in
>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9075#16
>>
>> So, I don't mean to be rude, but... which is it?
>
> I thought you were talking about `C-g'-ing while Gnus was working.  When
> it's waiting for network traffic, it's unlikely to be doing something
> destructive that'll mess up anything.

Oh... well, under normal circumstances there's no way to tell whether
it's working or waiting for network traffic, right?

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Updated IMAP unread count correction

[Lars Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

>>> So, again, I dearly wish for a way to flush all that state so I can get
>>> back to a "clean state."
>>
>> Yes, that would be nice.  But no such thing exists.
>
> Can you give any hints about how to get there?

At this point, your guess is as good as mine.  There's the Agent, and
the cache, and possibly some of the sync stuff, and possibly some
non-server marks...  

> Oh... well, under normal circumstances there's no way to tell whether
> it's working or waiting for network traffic, right?

Usually you're able to make an educated guess based on what Gnus is
saying.  Like "opening server..." and stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Wed Sep 05 2012, Lars Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>>>> So, again, I dearly wish for a way to flush all that state so I can get
>>>> back to a "clean state."
>>>
>>> Yes, that would be nice.  But no such thing exists.
>>
>> Can you give any hints about how to get there?
>
> At this point, your guess is as good as mine.  There's the Agent, and
> the cache, and possibly some of the sync stuff, and possibly some
> non-server marks...  

This is why I avoid as many of those mechanisms as possible and try to
talk directly to a local server for everything.

>
>> Oh... well, under normal circumstances there's no way to tell whether
>> it's working or waiting for network traffic, right?
>
> Usually you're able to make an educated guess based on what Gnus is
> saying.  Like "opening server..." and stuff.

Sure, but then there's a race condition.  If you decide to hit `C-g' and
Gnus decides to finish its remote work before the `C-g' is processed,
you've just trashed Gnus' state.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Updated IMAP unread count correction

[Dave Abrahams]

on Wed Sep 05 2012, Dave Abrahams <dave-AT-boostpro.com> wrote:

> on Wed Sep 05 2012, Lars Ingebrigtsen <larsi-AT-gnus.org> wrote:
>
>> Dave Abrahams <dave@boostpro.com> writes:
>>
>>>>> So, again, I dearly wish for a way to flush all that state so I can get
>>>>> back to a "clean state."
>>>>
>>>> Yes, that would be nice.  But no such thing exists.
>>>
>>> Can you give any hints about how to get there?
>>
>> At this point, your guess is as good as mine.  There's the Agent, and
>> the cache, and possibly some of the sync stuff, and possibly some
>> non-server marks...  
>
> This is why I avoid as many of those mechanisms as possible and try to
> talk directly to a local server for everything.
>
>>
>>> Oh... well, under normal circumstances there's no way to tell whether
>>> it's working or waiting for network traffic, right?
>>
>> Usually you're able to make an educated guess based on what Gnus is
>> saying.  Like "opening server..." and stuff.
>
> Sure, but then there's a race condition.  If you decide to hit `C-g' and
> Gnus decides to finish its remote work before the `C-g' is processed,
> you've just trashed Gnus' state.

Soooo... is this just not a concern for you?  It just occurred to me
that I commonly hit `C-g' several times in a row, so when the 2nd one
gets processed Gnus is almost *guaranteed* not to be waiting for network
traffic, right?

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost

Re: Updated IMAP unread count correction

[Lars Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> Soooo... is this just not a concern for you? 

No, it's not.  I can't recall any time that hitting `C-g' actually had
an adverse permanent effect in practice.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: Updated IMAP unread count correction

[Myles]

Dave Abrahams <dave <at> boostpro.com> writes:

> 
> 
> on Sun Jun 10 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:
> 
> > Dave Abrahams <dave <at> boostpro.com> writes:
> >
> >> Well, for me, they weren't.  It may simply be a matter of Ma Gnus'
> >> remembered state not corresponding to the state on the server; I don't
> >> know.  I do use other mail clients, too, FWIW.
> >
> > Are you able to reproduce what gets `unexist' tracking to become faulty?
> 
> I'm afraid I have no idea how to reproduce it, nor do I know how it
> happened in the first place.  And since I am using the referenced code,
> I never see an incorrect count anymore, so I wouldn't notice if it came
> back.  I could turn that code off, and we'd surely see the wrong counts
> again, but I don't know that it would help us find the cause.
> 

Just saying, to help in the diagnosis, that I had what I *think* was a similar
problem but couldn't find the cause, or the fix:

https://groups.google.com/forum/?fromgroups#!topic/gnu.emacs.gnus/6Sgs70YU0hM

and may be I was asking for help in the wrong place.  I had to jump ship in the
end (to mu4e) but I'll keep an eye on gnus for sure.

Myles