Round two of requests for new optional methods.

Round two of requests for new optional methods.

[visigoth]

[ I expect this to be "theoretical for qgnus" talk, but figured I'd throw
  it on the table since it's been much on my mind lately.  Flame away.  :) ]

So--the other year I asked for a few new backend features to make IMAP
easier to implement.  I wasn't quite on-the-ball enough to get it
implemented in a timely manner, but the things are, nonetheless,
there.

Now that I've got an implementation going, I understand better what
needs to be done to make this thing work well with Gnus and IMAP being
the beasts they are.  (And, well, having delved into the code, I'm
better equipped to make patches if it comes to that.  Scary code,
though.  ;>)

The main thing that's slowing the nnimap backend down now is that IMAP
and NNTP are capable of providing quite fundamentally different
information.

Our first example:

In NNTP, the server knows nothing about you.  All the server can tell
the client is what messages exist.  The client (Gnus) uses these
numbers to figure out how many new messages you have by comparing them
with what messages you've read.

In IMAP, the server knows about your messages, and while Gnus knows
some stuff, it doesn't know what messages you may've read with other
(heretical) readers, say, on a Mac or something, because that's all
you had to use.  IMAP can tell you how many new messages you have--and
that's fast, because it's just a count.  It can also tell you what
messages exist--but that's slow, because IMAP actually remembers
message numbers as "UID"s, which aren't cached as well.  (In addition,
you have to select a mailbox in order to get the info about the first
and last UIDs in the group.)  On top of that, UIDs aren't necessarily
allocated sequentially (although they're guaranteed to increase), so
message counts aren't always right.

In my mind, this calls for a backend method that would allow Gnus to
say, instead of "What messages exist" "How many new messages?" and
"How many total messages?" and the like.  THis info is very quickly
provided by IMAP.

The group-info updating code is also a point that could be improved
upon.  This takes a LOOONG time when you're subbed to 50-200 groups.
You have to select each group, do a query to find out which flags are
on which messages, and then move on to the next group.  (Group
switching takes 1-2 seconds before the folder's well cached n the
server.)  I don't know how long the query takes--I haven't tried
writing that yet, because I expect it to be really slow.

Alongside the "how many new messages do you have", which takes a small
amount of time, we could take the update-group-info method and call it
only upon group entry.  This way, we only have the cost of the query
when the group is entered.  It might take a few seconds (probably not
more than two or so, since this is, again, fast info) to enter a
group, but it will not, at least, take 400 seconds to update 200
groups all at once.

With these changes:

* add nnchoke-request-new-message-count
* call nnchoke-request-update-info at group entry, not active-time

I think nnimap can be quite efficient.


I must admit that I have not looked at how the group-buffer generating
code works, so I don't know how easy it would be to make it use a
simple number instead of the newsrc and active stuff.  The update-info
stuff, I think, could easily be moved, or a different version of it
could be created with a new name to be called in different places or
something.

Does anyone use this feature other than me?

In any case--let me know what you think.

Thanks,
John.

Re: Round two of requests for new optional methods.

[Paul Franklin]

While John's proposing changing what happens at active-time, I'd like
to point out that it would be helpful to some of us if less happened
at active-time for nnml groups as well.  There's a lot of stat() calls
happening per group, and this kills performance if your nnml files are
across NFS.

It appears that there's a lot more going on than just reading the nnml
server's active file (which would then get cached by NFS while
activating all groups on that server), and that these things could be
moved to group entry time (or maybe in some cases, under
nnmail-get-new-mail).

One of these days I'll understand the Gnus code well enough that I
could just do this on my own...  Sigh.

--Paul

Re: Round two of requests for new optional methods.

[Lars Magne Ingebrigtsen]

Paul Franklin <paul@cs.washington.edu> writes:

> It appears that there's a lot more going on than just reading the nnml
> server's active file (which would then get cached by NFS while
> activating all groups on that server), and that these things could be
> moved to group entry time (or maybe in some cases, under
> nnmail-get-new-mail).

No -- all Gnus does when activating nnml groups is read the active
file.  That is, if you have the nnml select method in
`gnus-secondary-select-methods'.  If not, Gnus will have to request
info on each separate nnml group at startup.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Round two of requests for new optional methods.

[Lars Magne Ingebrigtsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> Alongside the "how many new messages do you have", which takes a small
> amount of time, we could take the update-group-info method and call it
> only upon group entry.  This way, we only have the cost of the query
> when the group is entered.  It might take a few seconds (probably not
> more than two or so, since this is, again, fast info) to enter a
> group, but it will not, at least, take 400 seconds to update 200
> groups all at once.
> 
> With these changes:
> 
> * add nnchoke-request-new-message-count

I can add that, I guess-

> * call nnchoke-request-update-info at group entry, not active-time

Or nnimap could just ignore that call, and we could add a new function
to be called at group entry.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Round two of requests for new optional methods.

[visigoth]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

> > * add nnchoke-request-new-message-count
> I can add that, I guess-

Ahh.  Thanks much!  That single method will make it -sooo- much easier
to work with IMAP.  In fact--it might even let me throw out UIDs.
Hmm.  I'll have to think about that one.

If it'd be useful, I can try to work up a patch for you.

> Or nnimap could just ignore that call, and we could add a new function
> to be called at group entry.

Either would be fine for me.  Is anything using the current method?
(I was actually rather surprised to find out when it was called--I
don't know if that issue is documented.)

John.

Re: Round two of requests for new optional methods.

[Lars Magne Ingebrigtsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> 
> > > * add nnchoke-request-new-message-count
> > I can add that, I guess-
> 
> Ahh.  Thanks much!  That single method will make it -sooo- much easier
> to work with IMAP.  

It'll require some tippy-footing in the group buffer generation code,
since it'll have to bypass the normal calculation methods.

> > Or nnimap could just ignore that call, and we could add a new function
> > to be called at group entry.
> 
> Either would be fine for me.  Is anything using the current method?

nnvirtual does.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen