Re: gnus and imap

[David Engster <deng@randomsample.de>]

Ted Zlatanov <tzz@lifelogs.com> writes:
> DE> Thanks for your explanation. While this behavior of Zimbra might be
> DE> peculiar, it's in accordance with the IMAP RFC, which only demands UIDs
> DE> to be unique and ascending. So I'm afraid it is Gnus which will have to
> DE> be fixed here, which - being originally only a nntp client - still
> DE> assumes that article numbers are more or less contiguous.
>
> I would appreciate a quick primer:
>
> What elements of the Gnus data formats enforce this assumption currently?
>
> What backend functions make this assumption?  Which functions outside
> the backends have knowledge of the data format internals?

As far as I see, Gnus almost always tries to determine article counts
from the active data and the stuff that is saved in the group's
info[1]. However, all you can get from this data are upper bounds. It
will only be exact in the case of strictly contiguous article numbers.

For example:

In `gnus-articles-to-read', the total number of articles in a group is
simply the length of

(setq articles (gnus-uncompress-range (gnus-active group)))

With 'select' being the number of articles you want to retrieve from a
group, Gnus queries the backend with the following article list:

;; Select the N most recent articles.
(setq articles (nthcdr (- number select) articles))))

(where 'number' is the length of 'articles').

Similarly, unread articles are determined in
`gnus-list-of-unread-articles' by doing the following:

;; This function returns a list of article numbers based on the
;; difference between the ranges of read articles in this group and
;; the range of active articles.

In the case of nnimap, this estimate is currently "fixed" (in fact
overridden) through an imap-search for UNSEEN, which is done when
closing a nnimap group and saved in a separate hash table.

Article counts are done in the backend interface. AFAIK, the backends
itself don't assume much about article numbers. They just have to be
unique there.

I agree with Vitaly that the best way to solve this is to give the
backend interface complete knowledge of the available article numbers in
a group. We could use an additional backend function for this,
nnchoke-request-group-articles, which already exists for nntp, but
somehow isn't used further in Gnus. Based on what
nntp-request-group-articles returns, the function for nnimap would
roughly look like this

(deffoo nnimap-request-group-articles (group &optional server)
  (nnimap-possibly-change-group group server)
  (let ((articles (with-current-buffer nnimap-server-buffer
		    (imap-search "ALL"))))
    (if articles
      (with-current-buffer nntp-server-buffer
	(erase-buffer)
	(insert 
	 (format "211 %d %d %d %s\n"
		 (length articles)
		 (car articles)
		 (car (last articles))
		 group))
	(insert (mapconcat
		 'number-to-string
		 articles
		 "\n"))
	(insert "\n.\n")
	t)
      nil)))

Using this function will make backend access slower, so it should be
optional, maybe configurable through an additional server variable.

As Vitaly mentioned, this information should be cached so that we don't
have to get it every time a group is accessed.

-David