gnus and imap

gnus and imap

[Vitaly Mayatskikh]

Hi there!

As you all may know, Gnus counts articles in newsgroup within one single
sequence (min . max). This prevents imap backend to display correct
numbers of read/unread articles and cause many other weird and ugly
things, because imap stores uids of articles in sequence set. What do
you think of extending active info in Gnus from sequence to the
sequence set? I did it experimentally with adding "unread" list to the
group info and changing all work with gnus-active to manipulations
with gnus-info-read and gnus-info-unread, and nnimap works just fine for
me now. However, all other backends are broken at the moment, but if you
think it is a good intention (to extend active info to sequence set), it
will be no problems to add this feature for all backends.

Thanks!
-- 
wbr, Vitaly

Re: gnus and imap

[Frank Schmitt]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:

> Hi there!
>
> As you all may know, Gnus counts articles in newsgroup within one single
> sequence (min . max). This prevents imap backend to display correct
> numbers of read/unread articles and cause many other weird and ugly
> things, because imap stores uids of articles in sequence set. What do
> you think of extending active info in Gnus from sequence to the
> sequence set? I did it experimentally with adding "unread" list to the
> group info and changing all work with gnus-active to manipulations
> with gnus-info-read and gnus-info-unread, and nnimap works just fine for
> me now. However, all other backends are broken at the moment, but if you
> think it is a good intention (to extend active info to sequence set), it
> will be no problems to add this feature for all backends.

I think this is a very good proposal. The problem which decays my Gnus
IMAP experience most is, that the unread count is often wrong after you
accessed your IMAP folder with an other instance of Gnus or an other
client. Does your change fix this?

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: gnus and imap

[Vitaly Mayatskikh]

Frank Schmitt <ich@frank-schmitt.net> writes:

>> me now. However, all other backends are broken at the moment, but if you
>> think it is a good intention (to extend active info to sequence set), it
>> will be no problems to add this feature for all backends.
>
> I think this is a very good proposal. The problem which decays my Gnus
> IMAP experience most is, that the unread count is often wrong after you
> accessed your IMAP folder with an other instance of Gnus or an other
> client. Does your change fix this?

Yes. At least, I hope so. You can take these patches here:

git clone git://repo.or.cz/more-gnus.git

Keep in mind: only nnimap backend will work.
-- 
wbr, Vitaly

Re: gnus and imap

[Vegard Vesterheim]

On Wed, 20 Aug 2008 00:24:30 +0200 Vitaly Mayatskikh <v.mayatskih@gmail.com> wrote:

> Frank Schmitt <ich@frank-schmitt.net> writes:
>
>>> me now. However, all other backends are broken at the moment, but if you
>>> think it is a good intention (to extend active info to sequence set), it
>>> will be no problems to add this feature for all backends.
>>
>> I think this is a very good proposal. The problem which decays my Gnus
>> IMAP experience most is, that the unread count is often wrong after you
>> accessed your IMAP folder with an other instance of Gnus or an other
>> client. Does your change fix this?
>
> Yes. At least, I hope so. 

Great, the problem with wrong unread count also bothers me.

> You can take these patches here:
>
> git clone git://repo.or.cz/more-gnus.git

Doesn't work for me:

  fatal: Unable to look up repo.or.cz (port 9418) (Temporary failure in name resolution)

 - Vegard V - 

Re: gnus and imap

[Vitaly Mayatskikh]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

>> git clone git://repo.or.cz/more-gnus.git
>
> Doesn't work for me:
>
>   fatal: Unable to look up repo.or.cz (port 9418) (Temporary failure in name resolution)

I was able to clone it right now, please, try again.
-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
> As you all may know, Gnus counts articles in newsgroup within one single
> sequence (min . max). This prevents imap backend to display correct
> numbers of read/unread articles and cause many other weird and ugly
> things, because imap stores uids of articles in sequence set. What do
> you think of extending active info in Gnus from sequence to the
> sequence set? I did it experimentally with adding "unread" list to the
> group info and changing all work with gnus-active to manipulations
> with gnus-info-read and gnus-info-unread, and nnimap works just fine for
> me now. However, all other backends are broken at the moment, but if you
> think it is a good intention (to extend active info to sequence set), it
> will be no problems to add this feature for all backends.

I think it's a good idea. All those problems with wrong article counts
due to "holes" in the article numbers have driven me nuts while doing
nnmairix...

However, saying that adding this feature to all back ends is "no
problem" seems a bit optimistic to me. Gnus has lots of back ends (close
to 30), and there are external back ends like nnshimbun which are not
part of Gnus. So in my opinion the old code should be kept and used as a
fall-back method for back ends which do not store the unread sequence in
the info list.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

> However, saying that adding this feature to all back ends is "no
> problem" seems a bit optimistic to me. 

All, what it needs, is to fill gnus-info-unread (possibly, with the same
value as in active info). Overall changes in backends are quite small,
see: http://repo.or.cz/w/more-gnus.git?a=commitdiff;h=5345887a980cdd001a7c48e68143106cbd94efae

> Gnus has lots of back ends (close
> to 30), and there are external back ends like nnshimbun which are not
> part of Gnus. So in my opinion the old code should be kept and used as a
> fall-back method for back ends which do not store the unread sequence in
> the info list.

Ok, sounds reasonable. I'll try to fix Gnus for using gnus-info-unread
when possible. If backend wants to deliver articles in sequence set, it
should fill gnus-info-unread, otherwise Gnus will take bits from
gnus-active.

As I found, a lot of old code in Gnus simply doesn't work (nnir + imap,
agent + imap). Is it a good idea to keep infrastructure for this broken
code at all?
-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> However, saying that adding this feature to all back ends is "no
>> problem" seems a bit optimistic to me. 
>
> All, what it needs, is to fill gnus-info-unread (possibly, with the same
> value as in active info). Overall changes in backends are quite small,
> see: http://repo.or.cz/w/more-gnus.git?a=commitdiff;h=5345887a980cdd001a7c48e68143106cbd94efae

Yes, the changes are small. I was just referring to the sheer number of
back ends which makes this difficult. Besides, there are back ends which
are really difficult to grasp, like nnmaildir (at least for me).

> Ok, sounds reasonable. I'll try to fix Gnus for using gnus-info-unread
> when possible. If backend wants to deliver articles in sequence set, it
> should fill gnus-info-unread, otherwise Gnus will take bits from
> gnus-active.

Sounds good to me. However, before you do too much work on this, and
since this is a change in the Gnus internals, maybe someone who is more
familiar with the code base (Reiner?) can say if this is desirable to
do.

> As I found, a lot of old code in Gnus simply doesn't work (nnir + imap,
> agent + imap).

Really? I don't use these features, but was under the impression that
nnir and the agent work with nnimap.

Regards,
David

Re: gnus and imap

[Tibor Simko]

On Thu, 21 Aug 2008, David Engster wrote:
> Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> As I found, a lot of old code in Gnus simply doesn't work (nnir +
>> imap, agent + imap).
>
> Really? I don't use these features, but was under the impression that
> nnir and the agent work with nnimap.

Yes, I do use nnir with nnimap.

Best regards
-- 
Tibor Simko

Re: gnus and imap

[Vitaly Mayatskikh]

Tibor Simko <tibor.simko@cern.ch> writes:

>> Really? I don't use these features, but was under the impression that
>> nnir and the agent work with nnimap.
>
> Yes, I do use nnir with nnimap.

I've tried once to setup nnir for imap, but wasn't lucky. Can you,
please, show nnir-related part of your config?

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

>Tibor Simko <tibor.simko@cern.ch> writes:
>> Yes, I do use nnir with nnimap.

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
> I've tried once to setup nnir for imap, but wasn't lucky. Can you,
> please, show nnir-related part of your config?

By the way, Sacha Chua has released a draft of her Gnus chapter for her
"Wicked cool Emacs" book, which covers setting up Gnus for mail,
including searching via nnir:

http://sachachua.com/notebook/wickedcool/wc-emacs-05-gnus.html

I guess this one could also be a good link for gnus.org, though I don't
know if this is still up when the book is released...

-David

Re: gnus and imap

[Tibor Simko]

On Fri, 22 Aug 2008, Vitaly Mayatskikh wrote:
> I've tried once to setup nnir for imap, but wasn't lucky. Can you,
> please, show nnir-related part of your config?

(require 'nnir)

(setq gnus-select-method '(nnnil "")
      mail-sources '((file :path "/foo/bar/blah"))
      gnus-secondary-select-methods '((nnml "")
                                      (nnimap "foo.bar.com"
                                              (nnimap-address "foo.bar.com")
                                              (nnimap-stream ssl)
                                              (nnimap-server-port 993)
                                              (nnimap-logout-timeout 10)
                                              (nnimap-list-pattern ("FOO" "BAR" "BAZ.*" ))
                                              (nnir-search-engine imap))
                                      (nntp "gmane" (nntp-address "news.gmane.org"))))

Best regards
-- 
Tibor Simko ** CERN Document Server ** <http://cds.cern.ch/>

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

> Yes, the changes are small. I was just referring to the sheer number of
> back ends which makes this difficult. Besides, there are back ends which
> are really difficult to grasp, like nnmaildir (at least for me).

I decided to teach Gnus how to handle sequence sets in active info.
There's a lot of work, but it lets all backends to work without any
changes (if they don't want to pass sequence set instead of old single
sequence, of course). Basically, it works (somehow) now, but there are
a lot of bugs.

> Sounds good to me. However, before you do too much work on this, and
> since this is a change in the Gnus internals, maybe someone who is more
> familiar with the code base (Reiner?) can say if this is desirable to
> do.

Well, it must be fixed anyway.I have switched to Zimbra imap server
recently, and Gnus shows me hundreds of thousands of unread articles in
groups, and it works very slowly and consumes tons of memory.

-- 
wbr, Vitaly

Re: gnus and imap

[Frank Schmitt]

David Engster <deng@randomsample.de> writes:

>> As I found, a lot of old code in Gnus simply doesn't work (nnir + imap,
>> agent + imap).
>
> Really? I don't use these features, but was under the impression that
> nnir and the agent work with nnimap.

Agent + imap gives all sorts of problems for me and several people in
#gnus on freenode, mainly moved or deleted articles still showing up in
Summary.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: gnus and imap

[Reiner Steib]

On Thu, Aug 21 2008, David Engster wrote:

> Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>>> However, saying that adding this feature to all back ends is "no
>>> problem" seems a bit optimistic to me. 
>>
>> All, what it needs, is to fill gnus-info-unread (possibly, with the same
>> value as in active info). Overall changes in backends are quite small,
>> see: http://repo.or.cz/w/more-gnus.git?a=commitdiff;h=5345887a980cdd001a7c48e68143106cbd94efae

I guess that the patches are too large (> 15 lines of new code) to be
installed without legal papers.  So we need a copyright assignment,
either for Gnus or for Emacs (which covers Gnus as well).  If you are
willing to sign, I'll send you the form off-list.  Please let me know.

If you are willing to assign the code to the FSF, could you please
post the complete patch here along with ChangeLog entries?

> Yes, the changes are small. I was just referring to the sheer number of
> back ends which makes this difficult. Besides, there are back ends which
> are really difficult to grasp, like nnmaildir (at least for me).
>
>> Ok, sounds reasonable. I'll try to fix Gnus for using gnus-info-unread
>> when possible. If backend wants to deliver articles in sequence set, it
>> should fill gnus-info-unread, otherwise Gnus will take bits from
>> gnus-active.
>
> Sounds good to me. However, before you do too much work on this, and
> since this is a change in the Gnus internals, maybe someone who is more
> familiar with the code base (Reiner?) can say if this is desirable to
> do.

Unfortunately I'm not too familiar with the backend code and I'm quite
short of free time ATM (and I'll be on holiday during most of
September).  I'll be glad if you and/or other developers find time to
review and test Vitaly's patches.

Additionally, we need to decide how to continue Gnus development, so
I'm not sure if we want to install back end changes at this time.  See
this article:

,----[ http://thread.gmane.org/87wsi91d8d.fsf%40marauder.physik.uni-ulm.de ]
| From: Reiner Steib
| Subject: Gnus development - Emacs 23 feature freeze, Emacs 22.3 pretest
| Newsgroups: gmane.emacs.gnus.general
| To: Lars Magne Ingebrigtsen
| Cc: Miles Bader, ding@gnus.org
| Date: Fri, 22 Aug 2008 13:49:22 +0200
| Message-ID: <87wsi91d8d.fsf@marauder.physik.uni-ulm.de>
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: gnus and imap

[Vitaly Mayatskikh]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> Sounds good to me. However, before you do too much work on this, and
>> since this is a change in the Gnus internals, maybe someone who is more
>> familiar with the code base (Reiner?) can say if this is desirable to
>> do.
>
> Unfortunately I'm not too familiar with the backend code and I'm quite
> short of free time ATM (and I'll be on holiday during most of
> September).  I'll be glad if you and/or other developers find time to
> review and test Vitaly's patches.
>
> Additionally, we need to decide how to continue Gnus development, so
> I'm not sure if we want to install back end changes at this time.

Well, these changes will simply break everything except nnimap, so it
can't be committed at the moment. Another idea is to fix active info in
Gnus itself and to make it able to handle ranges of article numbers, not
only one range. It doesn't require any changes in backends (unless
some of them want to pass ranges of articles, like nnimap). I'm working
on it currently. I hope, it will solve many problems with articles
numbers.

-- 
wbr, Vitaly

Re: gnus and imap

[Ted Zlatanov]

On Fri, 22 Aug 2008 14:30:14 +0200 Vitaly Mayatskikh <v.mayatskih@gmail.com> wrote: 

VM> Well, these changes will simply break everything except nnimap, so it
VM> can't be committed at the moment. Another idea is to fix active info in
VM> Gnus itself and to make it able to handle ranges of article numbers, not
VM> only one range. It doesn't require any changes in backends (unless
VM> some of them want to pass ranges of articles, like nnimap). I'm working
VM> on it currently. I hope, it will solve many problems with articles
VM> numbers.

I am grateful for your work on this.  It will make Gnus IMAP support
better, for sure.

Could you let us know when your code is ready for integration, and if
you need any assistance with getting to that point?  I'll do whatever I
can to help you.

Thanks
Ted

Re: gnus and imap

[Vitaly Mayatskikh]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Could you let us know when your code is ready for integration, and if
> you need any assistance with getting to that point?  I'll do whatever I
> can to help you.

Basically, it works now. At least, I'm reading and answering all my mail
with patched Gnus. However, there are a lot of bugs around, and I highly
appreciate any help with testing it and finding problem places. You can
take patches from git repo:

git clone git://repo.or.cz/more-gnus.git
cd more-gnus
git checkout origin/active-info

And you'll have patched files.

Or even simpler: http://repo.or.cz/w/more-gnus.git?a=snapshot;h=6f516e017ea4421ce4c23c478710e5b9fe756e2e;sf=tgz

Thanks!
-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Reiner Steib <reinersteib+gmane@imap.cc> writes:
> Unfortunately I'm not too familiar with the backend code and I'm quite
> short of free time ATM (and I'll be on holiday during most of
> September).  I'll be glad if you and/or other developers find time to
> review and test Vitaly's patches.

I'm also pretty busy right now, but at least testing is always
possible.

Vitaly, if you're willing to assign copyright to the FSF, please post
patches (preferably against current Gnus CVS) as soon as you think it's
ready for testing.

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>I decided to teach Gnus how to handle sequence sets in active info.

I'm not sure what that implies. Does that mean you're extending active
to contain the whole sequence of available article numbers, not just the
highest and lowest one?

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

> Vitaly, if you're willing to assign copyright to the FSF, please post
> patches (preferably against current Gnus CVS) as soon as you think it's
> ready for testing.

That's my intentions. I didn't cover all Gnus sources yet, so is it a
good idea to post patches?

>>I decided to teach Gnus how to handle sequence sets in active info.
>
> I'm not sure what that implies. Does that mean you're extending active
> to contain the whole sequence of available article numbers, not just the
> highest and lowest one?

Yes, exactly.

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
> David Engster <deng@randomsample.de> writes:
>
>> Vitaly, if you're willing to assign copyright to the FSF, please post
>> patches (preferably against current Gnus CVS) as soon as you think it's
>> ready for testing.
>
> That's my intentions. I didn't cover all Gnus sources yet, so is it a
> good idea to post patches?

Yes. Release early, release often. :-)

>>>I decided to teach Gnus how to handle sequence sets in active info.
>>
>> I'm not sure what that implies. Does that mean you're extending active
>> to contain the whole sequence of available article numbers, not just the
>> highest and lowest one?
>
> Yes, exactly.

OK, I'm currently running Gnus with your patches. Nothing exploded. :-)
Group checking seems a bit slower, but I guess that's unavoidable for
better article counts. There are surely some bugs - I sometimes get
wrong article counts when *entering* the group, which is kinda funny,
since it usually was vice versa. Also, groups sometimes seem to be
updated twice when checking for new mail.

I only skimmed through your patches. I see you extended the
nntp-server-buffer to also include a range of all article numbers upon
updating the group. This should work. I still wonder though, why did you
abandon your previous effort to transmit a range of read articles in
gnus-info? It seems easier to me.

Thanks for your work, it's really appreciated. I'll be checking the code
further over the weekend and maybe also get to debugging if I find the
time.

By the way, it'd be good if you could update to CVS Gnus on your
side.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

> OK, I'm currently running Gnus with your patches. Nothing exploded. :-)

Good to know ;)

> Group checking seems a bit slower, but I guess that's unavoidable for
> better article counts.

Now it executes 2 separate queries, for "seen" and "unseen" articles. It
is possible to avoid executing of both queries for the case of unchanged
group (no new articles), because IMAP server returns total count of
articles in group within answer to SELECT command. Some other speedups
are possible also.

> There are surely some bugs - I sometimes get
> wrong article counts when *entering* the group, which is kinda funny,
> since it usually was vice versa. 

That's interesting. Can you tell me how do you customize Gnus in your
~/.gnus (agent, cache, etc)?

> Also, groups sometimes seem to be updated twice when checking for new
> mail.

AFAIK, it was always so. May be I'm wrong, but I'm seeing the same thing
in Gnus w/o any regards to patches. It was so at least in Emacs pre-22.

> I only skimmed through your patches. I see you extended the
> nntp-server-buffer to also include a range of all article numbers upon
> updating the group. This should work. I still wonder though, why did you
> abandon your previous effort to transmit a range of read articles in
> gnus-info? It seems easier to me.

Well, it needs some special flag to detect old vs new backend. It looks
like ugly solution for me :)

> Thanks for your work, it's really appreciated. I'll be checking the code
> further over the weekend and maybe also get to debugging if I find the
> time.

Cool! Thanks!

> By the way, it'd be good if you could update to CVS Gnus on your
> side.

Ok, will do.

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> There are surely some bugs - I sometimes get
>> wrong article counts when *entering* the group, which is kinda funny,
>> since it usually was vice versa. 
>
> That's interesting. Can you tell me how do you customize Gnus in your
> ~/.gnus (agent, cache, etc)?

The problem lies in gnus-read-active-file-2, which is called by
gnus-get-unread-articles. This uses nnimap-retrieve-groups to get a
partial active file of the checked groups, which is of the "regular"
(max . min) style and overwrites the previous "new" active sequence in
the hash table.

I'm not sure how to solve this. nnimap-retrieve-groups could return the
whole sequence, just like nnimap-request-group. Or we could change the
whole check-for-new-news sequence which is currently done by Gnus, so
that it does not use a partial active file.

Anyway, I have the feeling this gets messy. I'm not sure if this is the
right way to do this. The main problem I see is that this will break any
code which depends on gnus-active returning the cons (max . min). For
example, this currently breaks nnmairix, which isn't that much of a
problem since I can change this. But I wonder how much external code
exists which uses gnus-active.

We could solve this by extending the gnus-active macro, but this is
somewhat ugly, IMO. It may also be that there exists code (also in Gnus)
which directly accesses the hash table.

In my opinion, we should re-think your first effort, using group
info. You said:

> Well, it needs some special flag to detect old vs new backend. It looks
> like ugly solution for me :)

What about not extending the group info, but putting the unread sequence
in the existing marks section. Either the back end does this, then it
can be included, or we do it the old way. Just a suggestion, I don't
know if this works. As I said, I'm not that familiar with the back end
internals.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

>> That's interesting. Can you tell me how do you customize Gnus in your
>> ~/.gnus (agent, cache, etc)?
>
> The problem lies in gnus-read-active-file-2, which is called by
> gnus-get-unread-articles. This uses nnimap-retrieve-groups to get a
> partial active file of the checked groups, which is of the "regular"
> (max . min) style and overwrites the previous "new" active sequence in
> the hash table.

I can't get this scenario (probably, due to different settings). Will dig
around more.

> I'm not sure how to solve this. nnimap-retrieve-groups could return the
> whole sequence, just like nnimap-request-group. Or we could change the
> whole check-for-new-news sequence which is currently done by Gnus, so
> that it does not use a partial active file.
>
> Anyway, I have the feeling this gets messy. I'm not sure if this is the
> right way to do this. The main problem I see is that this will break any
> code which depends on gnus-active returning the cons (max . min). For
> example, this currently breaks nnmairix, which isn't that much of a
> problem since I can change this. But I wonder how much external code
> exists which uses gnus-active.

As I understand, this change affects only "middle level" back ends like
yours nnmairix, which lays in between of Gnus and real transport back
ends. Of course, it is hard to say what exactly can be broken.

> We could solve this by extending the gnus-active macro, but this is
> somewhat ugly, IMO. It may also be that there exists code (also in Gnus)
> which directly accesses the hash table.

We have sources and can fix this code ;) The only problem is a large
amount of regressions.

> In my opinion, we should re-think your first effort, using group
> info. You said:
>
>> Well, it needs some special flag to detect old vs new backend. It looks
>> like ugly solution for me :)
>
> What about not extending the group info, but putting the unread sequence
> in the existing marks section. Either the back end does this, then it
> can be included, or we do it the old way. Just a suggestion, I don't
> know if this works. As I said, I'm not that familiar with the back end
> internals.

I'm only afraid of spaghetti code. For me it's better to have one clean
and consistent solution.

Thanks for your help!

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> The problem lies in gnus-read-active-file-2, which is called by
>> gnus-get-unread-articles. This uses nnimap-retrieve-groups to get a
>> partial active file of the checked groups, which is of the "regular"
>> (max . min) style and overwrites the previous "new" active sequence in
>> the hash table.
>
> I can't get this scenario (probably, due to different settings). Will dig
> around more.

Try playing with gnus-read-active-file and call gnus-group-get-new-news
with a level argument. nnimap is my primary select method, maybe this
also comes into play.

I tested this with

(save-excursion
  (set-buffer gnus-group-buffer)
  (gnus-group-jump-to-group "sent")
  (gnus-group-get-new-news-this-group)
  (princ (gnus-active "sent") t);; will return full range
  (gnus-group-get-new-news 3)
  (princ (gnus-active "sent") t) ;; returns one range, e.g. (1 . 2500)
)

>> Anyway, I have the feeling this gets messy. I'm not sure if this is the
>> right way to do this. The main problem I see is that this will break any
>> code which depends on gnus-active returning the cons (max . min). For
>> example, this currently breaks nnmairix, which isn't that much of a
>> problem since I can change this. But I wonder how much external code
>> exists which uses gnus-active.
>
> As I understand, this change affects only "middle level" back ends like
> yours nnmairix, which lays in between of Gnus and real transport back
> ends. Of course, it is hard to say what exactly can be broken.

That's right (both statements).

>> What about not extending the group info, but putting the unread sequence
>> in the existing marks section. Either the back end does this, then it
>> can be included, or we do it the old way. Just a suggestion, I don't
>> know if this works. As I said, I'm not that familiar with the back end
>> internals.
>
> I'm only afraid of spaghetti code. For me it's better to have one clean
> and consistent solution.

Absolutely. The problem is that active information is used throughout
Gnus, and the code is in parts already pretty convoluted, especially the
ones dealing with updating groups and checking for unread messages. I
guess that putting unread information in gnus-info would affect fewer
parts of the code.

Another suggestion: maybe all this work should be strictly kept in the
back end itself?  We could implement a new back end function, like
'nnimap-unread-articles GROUP SERVER', which just returns the number of
unread articles. We can check for the existence of specific back end
functions with gnus-check-backend-function, so no need for additional
flags.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

>> I can't get this scenario (probably, due to different settings). Will dig
>> around more.
>
> Try playing with gnus-read-active-file and call gnus-group-get-new-news
> with a level argument. nnimap is my primary select method, maybe this
> also comes into play.

May be so, because your test code doesn't show me the bug in my
configuration (nnnil is primary, two nnimaps are secondary).

>> I'm only afraid of spaghetti code. For me it's better to have one clean
>> and consistent solution.
>
> Absolutely. The problem is that active information is used throughout
> Gnus, and the code is in parts already pretty convoluted, especially the
> ones dealing with updating groups and checking for unread messages. I
> guess that putting unread information in gnus-info would affect fewer
> parts of the code.

Yes, but we still have to fix all this convoluted code (for
gnus-unread-info or whatever). Unread articles calculations (count of
articles, list of articles) are totally wrong for the case of IMAP.

> Another suggestion: maybe all this work should be strictly kept in the
> back end itself?  We could implement a new back end function, like
> 'nnimap-unread-articles GROUP SERVER', which just returns the number of
> unread articles. We can check for the existence of specific back end
> functions with gnus-check-backend-function, so no need for additional
> flags.

It might be a good idea to extend api of back ends. By the way, have Gnus
developers any plans of doing large changes in Gnus, like refactoring,
global cleanup, etc?

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> Absolutely. The problem is that active information is used throughout
>> Gnus, and the code is in parts already pretty convoluted, especially the
>> ones dealing with updating groups and checking for unread messages. I
>> guess that putting unread information in gnus-info would affect fewer
>> parts of the code.
>
> Yes, but we still have to fix all this convoluted code (for
> gnus-unread-info or whatever). Unread articles calculations (count of
> articles, list of articles) are totally wrong for the case of IMAP.

I wonder why it's that bad with the Zimbra imapd. Doesn't it produce
contiguous UIDs? I mean, I also get wrong article numbers with dovecot,
but only in cases when I delete or move articles, but these are not
"totally" wrong. As the documentation says, the number is an estimate,
after all.

By the way, the reason for generating a partial active file for nnimap
is simply speed. Scanning every single group is slow (this is what
gnus-group-get-new-news-this-group does - it's much slower than just
gnus-group-get-new-news). See this thread for an explanation:

http://thread.gmane.org/gmane.emacs.gnus.general/26128/focus=26762

> It might be a good idea to extend api of back ends.

I also further searched through gmane for the problem of wrong article
counts, since this topic came up frequently on the ding list. And lo and
behold, there's stuff already in Gnus for this, it's just not documented
in the manual.

,----
| gnus-request-group-articles is a compiled Lisp function in `gnus-int.el'.
| (gnus-request-group-articles group)
| 
| Request a list of existing articles in group.
`----

This function calls a back end function
*-request-group-articles. However, this is currently only implemented
for nntp.

We could implement this for nnimap and use it for the calculation of the
unread count, and also for the calculation of the total number of
articles in a group, which is also usually wrong in case of gaps in the
article numbers. I think this is much easier to handle than extending
gnus-active.

However, we should make this optional, since it will make IMAP access
slower, which may be a problem for people accessing IMAP servers over
slow lines. We should create a new server variable, something like
'exact-count' or something.

BTW, there was also a patch for this, though for nnml, but it was not
integrated into Gnus:

http://thread.gmane.org/gmane.emacs.gnus.general/56714

The comments on this are interesting, especially the ones from Kevin
Greiner, which led me to the above request-group-articles function.

>  By the way, have Gnus developers any plans of doing large changes in
> Gnus, like refactoring, global cleanup, etc?

Not that I know of.

-David

Re: gnus and imap

[Reiner Steib]

On Sun, Aug 24 2008, David Engster wrote:

> Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> Yes, but we still have to fix all this convoluted code (for
>> gnus-unread-info or whatever). Unread articles calculations (count of
>> articles, list of articles) are totally wrong for the case of IMAP.
>
> I wonder why it's that bad with the Zimbra imapd. Doesn't it produce
> contiguous UIDs? I mean, I also get wrong article numbers with dovecot,
> but only in cases when I delete or move articles, but these are not
> "totally" wrong. As the documentation says, the number is an estimate,
> after all.

In my nnimap usage, I don't see wrong article counts.  There's also
code introduced in Oort that's supposed to give correct unread counts:

,----[ (info "(gnus)Oort Gnus") ]
|         * Unread count correct in nnimap groups.
| 
|           The estimated number of unread articles in the group buffer
|           should now be correct for nnimap groups.  This is achieved
|           by calling `nnimap-fixup-unread-after-getting-new-news' from
|           the `gnus-setup-news-hook' (called on startup) and
|           `gnus-after-getting-new-news-hook'. (called after getting
|           new mail).  If you have modified those variables from the
|           default, you may want to add
|           `nnimap-fixup-unread-after-getting-new-news' again.  If you
|           were happy with the estimate and want to save some (minimal)
|           time when getting new mail, remove the function.
`----

BTW, the functions is
`gnus-fixup-nnimap-unread-after-getting-new-news' not
`nnimap-fixup-unread-after-getting-new-news':

,----[ ChangeLog.2 ]
| 2002-09-27  Simon Josefsson  <jas@extundo.com>
| 
| 	* gnus-start.el (gnus-fixup-nnimap-unread-after-getting-new-news): New.
| 	(gnus-setup-news-hook): Use it.
| 	(gnus-after-getting-new-news-hook): Ditto.
| 
| 	* nnimap.el (nnimap-fixup-unread-after-getting-new-news): Remove.
`----

> BTW, there was also a patch for this, though for nnml, but it was not
> integrated into Gnus:
>
> http://thread.gmane.org/gmane.emacs.gnus.general/56714

We have papers on file for David Hanak.  So if his code is useful, we
could use it.

> The comments on this are interesting, especially the ones from Kevin
> Greiner, which led me to the above request-group-articles function.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: gnus and imap

[David Engster]

Reiner Steib <reinersteib+gmane@imap.cc> writes:
> In my nnimap usage, I don't see wrong article counts.  There's also
> code introduced in Oort that's supposed to give correct unread counts:
[...]
> BTW, the functions is
> `gnus-fixup-nnimap-unread-after-getting-new-news' not
> `nnimap-fixup-unread-after-getting-new-news':

Yes. This does indeed solve most problems, but not all. I guess the main
problem is that nnimap-update-unseen, on which the above fixup function
depends, is only called upon closing a group. I remember that I still
had situations where I got wrong unread article counts, but I can't
currently reproduce this with dovecot.

I still wonder why there are so many problems with the Zimbra imapd
server. It must have to do with how Zimbra generate UIDs for the
articles. Dovecot and most other imap servers at least try to keep them
contiguous.

Anyway, the fixup function only deals with unread articles, but I think
we should also fix the problem of the total article count. Many people
(including me) have the problem with Gnus thinking that a group contains
much more articles than there actually are. This can currently only be
solved by respooling.

>> http://thread.gmane.org/gmane.emacs.gnus.general/56714
>
> We have papers on file for David Hanak.  So if his code is useful, we
> could use it.

I wouldn't use it in the current form, but I think we should follow the
general idea, i.e. using gnus-request-group-articles (which is already
there).

-David

Re: gnus and imap

[James Cloos]

>>>>> "David" == David Engster <deng@randomsample.de> writes:

David> I remember that I still had situations where I got wrong unread
David> article counts, but I can't currently reproduce this with dovecot.

Try cacheing an article or two.  The count in the *Group* buffer will be
the largest article number on the server less one of the article numbers
in the cache (I cannot remember whether it is the oldest or most recent;
either way you'll likely have gaps and therefore get inaccurate info in
*Group*.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: gnus and imap

[Daniel Pittman]

David Engster <deng@randomsample.de> writes:
> Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>>> Absolutely. The problem is that active information is used throughout
>>> Gnus, and the code is in parts already pretty convoluted, especially the
>>> ones dealing with updating groups and checking for unread messages. I
>>> guess that putting unread information in gnus-info would affect fewer
>>> parts of the code.
>>
>> Yes, but we still have to fix all this convoluted code (for
>> gnus-unread-info or whatever). Unread articles calculations (count of
>> articles, list of articles) are totally wrong for the case of IMAP.
>
> I wonder why it's that bad with the Zimbra imapd. Doesn't it produce
> contiguous UIDs? 

No, it doesn't.  They increment, I strongly suspect, on a partially time
based counter, or possibly an account-wide number of deliveries based
counter, since one email in a group overnight gets around the same total
UID increment as the Linux kernel list in the same period, with some
hundred.

I have not yet taken the time to see if they available source code
explains why this is, or if it can be fixed, but the UID space is
*definitely* non-contiguous in Zimbra.

(Which, yes, does rather suck.)

Regards,
        Daniel

Re: gnus and imap

[David Engster]

Daniel Pittman <daniel@rimspace.net> writes:
> I have not yet taken the time to see if they available source code
> explains why this is, or if it can be fixed, but the UID space is
> *definitely* non-contiguous in Zimbra.

Thanks for your explanation. While this behavior of Zimbra might be
peculiar, it's in accordance with the IMAP RFC, which only demands UIDs
to be unique and ascending. So I'm afraid it is Gnus which will have to
be fixed here, which - being originally only a nntp client - still
assumes that article numbers are more or less contiguous.

-David

Re: gnus and imap

[Ted Zlatanov]

On Mon, 25 Aug 2008 11:46:05 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Daniel Pittman <daniel@rimspace.net> writes:
>> I have not yet taken the time to see if they available source code
>> explains why this is, or if it can be fixed, but the UID space is
>> *definitely* non-contiguous in Zimbra.

DE> Thanks for your explanation. While this behavior of Zimbra might be
DE> peculiar, it's in accordance with the IMAP RFC, which only demands UIDs
DE> to be unique and ascending. So I'm afraid it is Gnus which will have to
DE> be fixed here, which - being originally only a nntp client - still
DE> assumes that article numbers are more or less contiguous.

I would appreciate a quick primer:

What elements of the Gnus data formats enforce this assumption currently?

What backend functions make this assumption?  Which functions outside
the backends have knowledge of the data format internals?

Thanks
Ted

Re: gnus and imap

[David Engster]

Ted Zlatanov <tzz@lifelogs.com> writes:
> DE> Thanks for your explanation. While this behavior of Zimbra might be
> DE> peculiar, it's in accordance with the IMAP RFC, which only demands UIDs
> DE> to be unique and ascending. So I'm afraid it is Gnus which will have to
> DE> be fixed here, which - being originally only a nntp client - still
> DE> assumes that article numbers are more or less contiguous.
>
> I would appreciate a quick primer:
>
> What elements of the Gnus data formats enforce this assumption currently?
>
> What backend functions make this assumption?  Which functions outside
> the backends have knowledge of the data format internals?

As far as I see, Gnus almost always tries to determine article counts
from the active data and the stuff that is saved in the group's
info[1]. However, all you can get from this data are upper bounds. It
will only be exact in the case of strictly contiguous article numbers.

For example:

In `gnus-articles-to-read', the total number of articles in a group is
simply the length of

(setq articles (gnus-uncompress-range (gnus-active group)))

With 'select' being the number of articles you want to retrieve from a
group, Gnus queries the backend with the following article list:

;; Select the N most recent articles.
(setq articles (nthcdr (- number select) articles))))

(where 'number' is the length of 'articles').

Similarly, unread articles are determined in
`gnus-list-of-unread-articles' by doing the following:

;; This function returns a list of article numbers based on the
;; difference between the ranges of read articles in this group and
;; the range of active articles.

In the case of nnimap, this estimate is currently "fixed" (in fact
overridden) through an imap-search for UNSEEN, which is done when
closing a nnimap group and saved in a separate hash table.

Article counts are done in the backend interface. AFAIK, the backends
itself don't assume much about article numbers. They just have to be
unique there.

I agree with Vitaly that the best way to solve this is to give the
backend interface complete knowledge of the available article numbers in
a group. We could use an additional backend function for this,
nnchoke-request-group-articles, which already exists for nntp, but
somehow isn't used further in Gnus. Based on what
nntp-request-group-articles returns, the function for nnimap would
roughly look like this

(deffoo nnimap-request-group-articles (group &optional server)
  (nnimap-possibly-change-group group server)
  (let ((articles (with-current-buffer nnimap-server-buffer
		    (imap-search "ALL"))))
    (if articles
      (with-current-buffer nntp-server-buffer
	(erase-buffer)
	(insert 
	 (format "211 %d %d %d %s\n"
		 (length articles)
		 (car articles)
		 (car (last articles))
		 group))
	(insert (mapconcat
		 'number-to-string
		 articles
		 "\n"))
	(insert "\n.\n")
	t)
      nil)))

Using this function will make backend access slower, so it should be
optional, maybe configurable through an additional server variable.

As Vitaly mentioned, this information should be cached so that we don't
have to get it every time a group is accessed.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

>> Yes, but we still have to fix all this convoluted code (for
>> gnus-unread-info or whatever). Unread articles calculations (count of
>> articles, list of articles) are totally wrong for the case of IMAP.
>
> I wonder why it's that bad with the Zimbra imapd. Doesn't it produce
> contiguous UIDs? I mean, I also get wrong article numbers with dovecot,
> but only in cases when I delete or move articles, but these are not
> "totally" wrong. As the documentation says, the number is an estimate,
> after all.

I have a lot of mail imported from another IMAP server, and, thus, have
two uid's spaces. Now my groups uids look like ((4000 . 30000) (100000
. 110000)).

>> It might be a good idea to extend api of back ends.
>
> I also further searched through gmane for the problem of wrong article
> counts, since this topic came up frequently on the ding list. And lo and
> behold, there's stuff already in Gnus for this, it's just not documented
> in the manual.

Wrong articles count is only one of problems. Getting old articles in
nnimap group is close to impossible: I can't just tell "give me last 5
old articles", because Gnus calculates wrong uids for these
articles. Instead of it, I have to ask for last 1000 (or 10000, or
100000) articles :(

Also, I don't like any fixup hooks. Usually that means bad design of
software, and usually they don't fix the problem at all.

> We could implement this for nnimap and use it for the calculation of the
> unread count, and also for the calculation of the total number of
> articles in a group, which is also usually wrong in case of gaps in the
> article numbers. I think this is much easier to handle than extending
> gnus-active.

Yes, we need to delegate calculation of read/unread articles lists to
back ends (or implement these calculations correctly in Gnus for every
case).

> However, we should make this optional, since it will make IMAP access
> slower, which may be a problem for people accessing IMAP servers over
> slow lines. We should create a new server variable, something like
> 'exact-count' or something.

I wish Gnus to cache all needed meta data, like other mail clients do. It
is really not necessary to ask for all seen/unseen uids again and
again, but rather to search uids since last update. Currently, I simply
can't use Gnus with slow and unstable connection (gprs), because it
dies before Gnus finishes to update all my groups :) It is even hard to
update linux-kernel mailing list on 20mb link.

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> I also further searched through gmane for the problem of wrong article
>> counts, since this topic came up frequently on the ding list. And lo and
>> behold, there's stuff already in Gnus for this, it's just not documented
>> in the manual.
>
> Wrong articles count is only one of problems. Getting old articles in
> nnimap group is close to impossible: I can't just tell "give me last 5
> old articles", because Gnus calculates wrong uids for these
> articles. Instead of it, I have to ask for last 1000 (or 10000, or
> 100000) articles :(

Well, this is a result of the wrong article count. Gnus simply uses

(- (1+ (cdr active)) (car active))

i.e. highest-lowest as the number of articles in a group. Since Zimbra
doesn't generate contiguous UIDs, this number is far too large. When you
tell Gnus to get the latest 5 articles, it simply queries the back end
with the UIDs from 'highest-4' to 'highest' - again assuming contiguous
article numbers. This is why you have to query Gnus with insane numbers
of articles, and of course this is slow, since the back end gets queried
with the fully expanded sequence.

> Also, I don't like any fixup hooks. Usually that means bad design of
> software, and usually they don't fix the problem at all.

At the moment, I see two options to get rid of this problem:

1) Make nnimap always return contiguous article numbers. Currently,
nnimap simply uses the UIDs provided by the IMAP server as article
numbers. We could change nnimap to use its own set of contiguous article
numbers and map those to the UIDs of the IMAP server.

The advantage would be that changes would be more or less confined to
nnimap, but this also implies that other back ends wouldn't profit from
this.

2) Make Gnus correctly deal with non-contiguous article numbers.

Obviously, all back ends would profit from this, but this would mean
(probably large) changes throughout the back end interface. We could use
gnus-request-group-articles to get a list of all article numbers in a
group (if the back end supports it). Gnus could calculate article counts
from this (total as well as unread) and also query the back end with the
correct sequence of article numbers.

Other options?

> I wish Gnus to cache all needed meta data, like other mail clients do. It
> is really not necessary to ask for all seen/unseen uids again and
> again, but rather to search uids since last update. 

Yes, caching would have to be done as well. We could use a new hash
table for this. There's one for the nnimap-fixup function anyway...

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

>> articles. Instead of it, I have to ask for last 1000 (or 10000, or
>> 100000) articles :(
>
> Well, this is a result of the wrong article count. Gnus simply uses
>
> (- (1+ (cdr active)) (car active))
>
> i.e. highest-lowest as the number of articles in a group. Since Zimbra
> doesn't generate contiguous UIDs, this number is far too large. When you
> tell Gnus to get the latest 5 articles, it simply queries the back end
> with the UIDs from 'highest-4' to 'highest' - again assuming contiguous
> article numbers. This is why you have to query Gnus with insane numbers
> of articles, and of course this is slow, since the back end gets queried
> with the fully expanded sequence.

Yes, I know that.

>> Also, I don't like any fixup hooks. Usually that means bad design of
>> software, and usually they don't fix the problem at all.
>
> At the moment, I see two options to get rid of this problem:
>
> 1) Make nnimap always return contiguous article numbers. Currently,
> nnimap simply uses the UIDs provided by the IMAP server as article
> numbers. We could change nnimap to use its own set of contiguous article
> numbers and map those to the UIDs of the IMAP server.

IMAP has this feature. When client executtes SEARCH (FETCH, etc) query
without UID keyword, it returns contiguous numbers starting from 1.

> The advantage would be that changes would be more or less confined to
> nnimap, but this also implies that other back ends wouldn't profit from
> this.

> 2) Make Gnus correctly deal with non-contiguous article numbers.
>
> Obviously, all back ends would profit from this, but this would mean
> (probably large) changes throughout the back end interface. We could use
> gnus-request-group-articles to get a list of all article numbers in a
> group (if the back end supports it). Gnus could calculate article counts
> from this (total as well as unread) and also query the back end with the
> correct sequence of article numbers.
>
> Other options?

I vote for the 2nd solution. It is the most honest way to deal with
different types of mail sources. I think, representation of articles
numbers is an internal business of back end.

>> I wish Gnus to cache all needed meta data, like other mail clients do. It
>> is really not necessary to ask for all seen/unseen uids again and
>> again, but rather to search uids since last update. 
>
> Yes, caching would have to be done as well. We could use a new hash
> table for this. There's one for the nnimap-fixup function anyway...

Other things like asynchronous work, multiple connections to one server
will be fine :)

-- 
wbr, Vitaly

Re: gnus and imap

[Ted Zlatanov]

On Sun, 24 Aug 2008 10:47:52 +0200 Vitaly Mayatskikh <v.mayatskih@gmail.com> wrote: 

VM> It might be a good idea to extend api of back ends. 

Agreed.  Can you propose specific changes?

VM> By the way, have Gnus developers any plans of doing large changes in
VM> Gnus, like refactoring, global cleanup, etc?

I don't know of such plans.  Considering the code works well for many
people, it's been in use for a long time, and that it's one of the
largest Emacs Lisp packages, I think such work has to have some clearly
defined goals and benefits.  I know there's issues with direct access to
the data structures in many places (which you've found while makings
your IMAP fixes) and with handling of multibyte strings.  The code is
tightly coupled so refactoring and cleanup are a lot of work.

A complete rewrite might be a better idea.  It is expensive and
dangerous, but also it's a chance to take advantage of many years of
experience with Gnus usability and features.  Maybe this should be the
next version of Gnus (6.x).

Ted

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

>> I can't get this scenario (probably, due to different settings). Will dig
>> around more.
>
> Try playing with gnus-read-active-file and call gnus-group-get-new-news
> with a level argument. nnimap is my primary select method, maybe this
> also comes into play.
>
> I tested this with
>
> (save-excursion
>   (set-buffer gnus-group-buffer)
>   (gnus-group-jump-to-group "sent")
>   (gnus-group-get-new-news-this-group)
>   (princ (gnus-active "sent") t);; will return full range
>   (gnus-group-get-new-news 3)
>   (princ (gnus-active "sent") t) ;; returns one range, e.g. (1 . 2500)
> )

Ok, what's going on in gnus-get-unread-articles: gnus-read-active-file-2
is called for every group in retrieve-groups list. However, imap's group
has no chance to be in retrieve-groups, because it falls into cond's
branch '(and method (eq method-type 'foreign))'. I've tried it with
nnimap as primary method also.

-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
> Ok, what's going on in gnus-get-unread-articles: gnus-read-active-file-2
> is called for every group in retrieve-groups list. However, imap's group
> has no chance to be in retrieve-groups, because it falls into cond's
> branch '(and method (eq method-type 'foreign))'. I've tried it with
> nnimap as primary method also.

If you look at how method-type is set, you'll see that foreign servers
are those which are neither in gnus-select-method nor in
gnus-secondary-select-methods. So, I wouldn't know why this
happens.

You see that Gnus iterates through newsrc, which is
gnus-newsrc-alist. You can try it manually by first evaluating

(setq newsrc gnus-newsrc-alist)

and then do repeatedly

(progn
  (setq group (gnus-info-group (setq info (pop newsrc))))
  (setq method (gnus-info-method info))
  (cond 
    ((or (null method) (gnus-server-equal method gnus-select-method))
      (message "%s is primary" group))
    ((gnus-secondary-method-p method)
      (message "%s is secondary" group))
    (t (message "%s is foreign" group))))

which will show you in what category each group falls into. For any
method in gnus-select-method or gnus-secondary-select-methods you should
get primary or secondary, resp. If not, something strange happened to
your setup. Foreign groups are usually those to which you subscribed to
using gnus-group-browse-foreign-server.

-David

Re: gnus and imap

[Vitaly Mayatskikh]

David Engster <deng@randomsample.de> writes:

> which will show you in what category each group falls into. For any
> method in gnus-select-method or gnus-secondary-select-methods you should
> get primary or secondary, resp. If not, something strange happened to
> your setup. Foreign groups are usually those to which you subscribed to
> using gnus-group-browse-foreign-server.

Right, I have subscribed to all groups through
gnus-group-browse-foreign-server. I've fixed nnimap-retrieve-groups and
gnus-active-to-gnus-format (it was called by gnus-read-active-file-2).

Doh, so many ugly quirks... :(
-- 
wbr, Vitaly

Re: gnus and imap

[David Engster]

> David Engster <deng@randomsample.de> writes:
>> which will show you in what category each group falls into. For any
>> method in gnus-select-method or gnus-secondary-select-methods you should
>> get primary or secondary, resp. If not, something strange happened to
>> your setup. Foreign groups are usually those to which you subscribed to
>> using gnus-group-browse-foreign-server.

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
> Right, I have subscribed to all groups through
> gnus-group-browse-foreign-server. I've fixed nnimap-retrieve-groups and
> gnus-active-to-gnus-format (it was called by gnus-read-active-file-2).

I just wanted to mention that I checked out your new patches and will
test them. I'm currently pretty busy with another project (and work, of
course), but I'm hoping I can get to this over the coming weekend.

-David

Re: gnus and imap

[David Engster]

> Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
>> Right, I have subscribed to all groups through
>> gnus-group-browse-foreign-server. I've fixed nnimap-retrieve-groups and
>> gnus-active-to-gnus-format (it was called by gnus-read-active-file-2).

I've finally come to test your patches. Sorry for the delay.

I'm still sometimes encountering wrong article counts upon entering a
group. This happens when I delete articles in a group and enter it
without a rescan. The group will then show an unread count which is
usually equal to the number of deleted articles. This gets reset to zero
when you exit the group, so you can only see that with a three-pane
view, where the group buffer is always visible.

However, this is only indirectly a consequence of your patches. The
problem seems to be that when you delete some articles in a group, the
active information does not get updated in the hash table, so that
(gnus-range-difference active range) will return a range containing the
deleted articles.

This is all fixable, of course. However, before continuing with this
approach, I'd still like to suggest using a new backend function instead
of storing the number of available articles numbers in the
active-hashtb. The latter implies that we would have to check the whole
Gnus trunk for regressions regarding usage of the active-hashtb, which
will be a lot of work. I think that using something like
nnimap-request-group-articles and leaving the active-hashtb as it is
would be easier to do in the long term. If you (or others) don't object,
I could try to change your patches using that new backend function.

Best,
David

Re: gnus and imap

[Dan Christensen]

In 2008, regarding Vitaly Mayatskikh's patches, David Engster writes:

> I've finally come to test your patches. Sorry for the delay.
>
> I'm still sometimes encountering wrong article counts upon entering a
> group. This happens when I delete articles in a group and enter it
> without a rescan. The group will then show an unread count which is
> usually equal to the number of deleted articles. This gets reset to zero
> when you exit the group, so you can only see that with a three-pane
> view, where the group buffer is always visible.
>
> However, this is only indirectly a consequence of your patches. The
> problem seems to be that when you delete some articles in a group, the
> active information does not get updated in the hash table, so that
> (gnus-range-difference active range) will return a range containing the
> deleted articles.
>
> This is all fixable, of course. However, before continuing with this
> approach, I'd still like to suggest using a new backend function instead
> of storing the number of available articles numbers in the
> active-hashtb. The latter implies that we would have to check the whole
> Gnus trunk for regressions regarding usage of the active-hashtb, which
> will be a lot of work. I think that using something like
> nnimap-request-group-articles and leaving the active-hashtb as it is
> would be easier to do in the long term. If you (or others) don't object,
> I could try to change your patches using that new backend function.
>
> Best,
> David

I'm wondering if Vitaly or David have made any progress on the imap
unread count.  I'm using dovecot, and I regularly get wrong unread
counts in my nnimap groups, even with
gnus-fixup-nnimap-unread-after-getting-new-news in
gnus-after-getting-new-news-hook.

I think David's idea of adding an optional backend function which
returns more detailed information is the way to go.  The core of Gnus
can check for this function and use it if it exists, falling back to the
old way if not.  This allows for a gradual transition without needing
massive changes all at once.

Dan

Re: gnus and imap

[Dan Christensen]

Dan Christensen <jdc@uwo.ca> writes:

> I'm wondering if Vitaly or David have made any progress on the imap
> unread count.  I'm using dovecot, and I regularly get wrong unread
> counts in my nnimap groups, even with
> gnus-fixup-nnimap-unread-after-getting-new-news in
> gnus-after-getting-new-news-hook.
>
> I think David's idea of adding an optional backend function which
> returns more detailed information is the way to go.  The core of Gnus
> can check for this function and use it if it exists, falling back to the
> old way if not.  This allows for a gradual transition without needing
> massive changes all at once.

I found another thread with promising code for the nnml backend, written
by David Hanak.  It seemed like only minor details needed to be worked
out:

  http://thread.gmane.org/gmane.emacs.gnus.general/56714

The thread I was referring to above discussed code written by Vitaly
Mayatskikh and included comments by a different David, David Engster:

  http://thread.gmane.org/gmane.emacs.gnus.general/67238

Both seem like they were within reach of a solution to this problem that
has plagued Gnus for a long time!  Maybe we can reach a consensus about
the best way to proceed, and finally get it done?

Dan

PS: There are also two solutions posted at

  http://www.emacswiki.org/emacs/GnusNiftyTricks  

Re: gnus and imap

[David Engster]

Dan Christensen <jdc@uwo.ca> writes:
> Dan Christensen <jdc@uwo.ca> writes:
>> I'm wondering if Vitaly or David have made any progress on the imap
>> unread count.

No, not me, at least.

> The thread I was referring to above discussed code written by Vitaly
> Mayatskikh and included comments by a different David, David Engster:
>
>   http://thread.gmane.org/gmane.emacs.gnus.general/67238
>
> Both seem like they were within reach of a solution to this problem that
> has plagued Gnus for a long time!  Maybe we can reach a consensus about
> the best way to proceed, and finally get it done?

I think Vitaly was closest since he had actual patches. How close those
patches are to an actual solution - I'm not sure. The Gnus back end code
is very convoluted, and there are differences in how
primary/secondary/foreign groups are handled, which makes all of this
stuff rather unpleasant.

Regards,
David

gnus and IMAP

[me]

Hi all,

This might be a repeating question, if so please dont hurt me =). I wish
to learn and use gnus as my main mail reader and my company run this thing
called Exchange as server. Because I need that "thing" to access
adresslists I have to run both gnus and that "outlook" thing. I have tried
some way of accessing the IMAP server and Im stuck.

When i run imap from mail-sources it will not sync the imap folder, just
get new once. You can set the predicate to nil but then I get all the
messages in the inbox every time it gets them.
I also tried running nnimap backend but I cant get that working. I will
login and all but I cant subscribe to the inbox folder.
What I want to do is sync the imap folders onto gnus but still have it in
"outlook".

Any help/configure example/shoulder to cry on/slap in the face telling me
I cant to this/anything in this matter would be greatfully accepted.

Please reply to my email adress since im not on the list.

Regards
Jerry

Re: gnus and IMAP

[Niklas Morberg]

me@vger.org writes:

> When i run imap from mail-sources it will not sync the
> imap folder, just get new once.

I don't quite understand the problem. Gnus does not
automatically retrieve (I assume this is what you mean
by `sync') new mail. If you press `g' in the *Group*
buffer, Gnus will get new mail and news items.

But you really should use nnimap.

> I will login and all but I cant subscribe to the inbox
> folder.

Why not? What is the error message you get when you try to
subscribe?

My settings for running IMAP against an Exchange server are:

(setq gnus-nntp-server nil)
(setq gnus-select-method '(nnimap "your.mailserver.here" (nnimap-stream ssl))))
(setq nnmail-expiry-target "nnimap\+your.mailserver.here:Deleted Items")
(setq gnus-message-archive-group "\"nnimap\+your.mailserver.here:Sent Items\"")

I don't think ssl comes with emacs, but you can find it in
the gnus/contrib directory.

> What I want to do is sync the imap folders onto
> gnus but still have it in "outlook".

Are you sure that this is what you want? I want to have all
my messages on the Exchange server and then read them with
either Gnus or Outlook (or any other software for that
matter).

Niklas

Re: gnus and IMAP

[me]

On Tue, 9 Jul 2002, Niklas Morberg wrote:

> me@vger.org writes:
> 
> > I will login and all but I cant subscribe to the inbox
> > folder.
> 
> Why not? What is the error message you get when you try to
> subscribe?
> 
> My settings for running IMAP against an Exchange server are:
> 
> (setq gnus-nntp-server nil)
> (setq gnus-select-method '(nnimap "your.mailserver.here" (nnimap-stream ssl))))
> (setq nnmail-expiry-target "nnimap\+your.mailserver.here:Deleted Items")
> (setq gnus-message-archive-group "\"nnimap\+your.mailserver.here:Sent Items\"")
> 
> I don't think ssl comes with emacs, but you can find it in
> the gnus/contrib directory.
> 

This is my config now:

(setq gnus-nntp-server nil)

(setq gnus-select-method
	  '(nnml ""))

(setq gnus-secondary-select-methods
      '((nnimap "my.mail.server"
                (nnimap-server-port 143)
                (nnimap-address "my.mail.server")
				(nnimap-expunge-on-close never))))

(setq mail-sources
	  '((file :path "/my/spool/file")))

(setq nnmail-split-methods
	  '(("inbox" "")))

(setq mm-discouraged-alternatives
	  '("text/html" "text/richtext"))

(setq nnimap-split-inbox
	  '("inbox"))

(setq nnimap-split-methods
	  '(("imap" "")))

(setq nnimap-split-predicate
	  '("UNDELETED"))


I can subscribe to inbox but I cant get imap. Is this even the correct way
of doing things? Im not sure.


> > What I want to do is sync the imap folders onto
> > gnus but still have it in "outlook".
> 
> Are you sure that this is what you want? I want to have all
> my messages on the Exchange server and then read them with
> either Gnus or Outlook (or any other software for that
> matter).
> 

No, im not sure at all =). I want to run it as you have but have more mail
sources.

Im running gnus 5.8.8 right now, is that the lastest version btw?

Regards
Jerry

Re: gnus and IMAP

[me]

Okey, running nnimap as gnus-select-method works. Question is why i cant
run it as secondary-method or am I just confused.

I want to be able to get pop mail from some other server, store that
locally and read the mail from the imap server as well.

Re: gnus and IMAP

[me]

Got it working now, was just to enter the server buffer and subscribe from
there.

config now:

(setq gnus-nntp-server nil)

(setq gnus-select-method '(nnnil))

(setq gnus-secondary-select-methods
      '((nnimap "my.mail.server"
                (nnimap-server-port 143)
                (nnimap-address "my.mail.server")
				(nnimap-expunge-on-close never))
		(nnml "")))

(setq mail-sources
	  '((file :path "/my/spool/file")))

(setq nnmail-split-methods
	  '(("localhost" "")))

(setq mm-discouraged-alternatives
	  '("text/html" "text/richtext"))

(setq nnimap-split-predicate
	  '("UNDELETED"))



Thanks for the help =)