Re: nnimap problem solved by removing .agentview and .overview

[Kevin Greiner <kgreiner@xpediantsolutions.com>]

Simon Josefsson <jas@extundo.com> writes:

> "Steven E. Harris" <seh@panix.com> writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> The problem is that, when the agent is turned on, your backend
>>> requests don't go directly to the backend.  They instead go to the
>>> agent which tries to respond using what it has locally cached.  If,
>>> and only if, you request something that is not in the cache, will the
>>> agent request the MISSING content from the real backend.
>>
>> How is one ever able to see something that something exists that is
>> not locally cached? How does Gnus ever know that there is something
>> new on the IMAP server that it has not seen?

Some requests are only intercepted with running unplugged.  For
example the request to get the active range is passed through to the
backend with running plugged.  Let's work an example for those reading
along.

  Yesterday, we'll asume that your active range consisted of eight
  messages (numbered 1 through 8) and that you used gnus to view the
  folder's summary so all eight are known to the agent.

  Today, we'll assume that two new messages have arrived so we know
  that your active range is now 1-10.  When gnus starts it queries the
  server for the current active range so now gnus knows that your
  active range is 1-10.  Now, you open the folder to display the
  summary buffer.  The summary functions make a request for all
  article headers in the range 1-10.  That request is redirected to
  the agent by gnus internals.  The agent checks its .overview file,
  sees that it has headers for articles 1-8 (from yesterday) and
  forwards a modified request for the range 9-10 to the server.  The
  agent gets the headers for articles 9-10 from the server, merges
  them into its .overview so that it never needs to ask the server for
  them, then constructs a response that satisfies the summary buffer's
  original request for article headers in the range 1-10.

>> Also, I thought this behind-the-back-change is what the IMAP
>> UIDVALIDITY response is supposed to help discern. I thought that with
>> UIDVALIDITY Gnus could detect that it wasn't the last client to have
>> altered the mailbox. Reading RFC 3501, I see that's not the case;
>> change in UIDVALIDITY only warns that the old message IDs may have
>> changed.

>> What if, upon contacting the IMAP server and noting agreeable
>> UIDVALIDITY, Gnus does a SEARCH or UID query from its last known
>> message ID, checking to see if any new IDs are present that would not
>> have appeared in a check for what is UNSEEN? Isn't there some way like
>> this to figure out that the server has messages that the client has
>> never heard of?
>
> Gnus is supposed the invoke these SEARCH requests every time, upon
> entering the group, to sync all flags, and its list of which articles
> exists.  However, if you only press 'g' and then ask the agent to
> fetch articles, the group might not have been entered, so nnimap
> haven't had any chance of updating the information.
>
> However, if entering and exiting a group when plugged in doesn't
> notice all articles, there is a bug.  I suspect the problem lies in
> the agent rather than in nnimap, or (perhaps even more likely) in
> interaction between nnimap and the agent.
>
> Is there a simple recipe for reproducing this?
>
>>> Frankly, I don't see how to make the agent work better with nnimap
>>> without introducing nnimap specific code into the agent or changing
>>> the backend interface.  Neither is something to take lightly so I
>>> don't believe that this situation will be changing anytime soon.
>>
>> Are these interfaces documented somewhere? It's hard to comment on
>> potential solutions without knowing the current constraints.
>
> I'd even take another step back: Exactly what problem is in the design
> of agent make it incompatible with nnimap?  I believe the designs are
> pretty compatible.

The agent assumes that the backend will never change the state of
anything already known to the agent.  The fact that UIDVALIDITY exists
is proof that that isn't the case with imap.

We could propagate the uid concept into the agent then have the agent
verify its uid against the backend's uid before trusting its own
cache.  It would probably work but I'm worried about the performance
impact (would nnimap have to query the server each time the uid is
checked?) and having to dummy up uid values for all of the other
backends.

>> Again, it's surprising that even after four or five years of Gnus
>> supporting IMAP, we still haven't done better than other IMAP
>> clients.
>
> Not many people are working on nnimap.el, nor improving the generic
> backend interface, these days, it seems...

I agree.

Kevin