Re: Gnus' speed

[Ted Zlatanov <tzz@lifelogs.com>]

On Thu, 30 Jul 2009 15:58:37 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think the backend rewrite would be fairly minor, but there are many
>> internal and user-visible functions in Gnus that also will need to be
>> modified.  It's a lot of work.

DP> I think that saying "fairly minor" is misleading here: adding the capability
DP> to the IMAP backend isn't enormously hard, but defining a sound (and portable)
DP> API, and implementing it, is quite non-trivial.

I'd do it with the IMAP convention of "1.2.3" to mean part 3 of part 2
of part 1 as an optional parameter for the backend command to fetch the
body, and an extra backend command to get a message's structure (list of
strings as above with the associated data).  I'd move all the rest of
the code into a separate async-bodypart-fetch library that, given a list
of body parts, can interact with the existing Gnus user-facing
functionality that builds and interacts with the article buffer.  The
async-bodypart-fetch library should be usable by other ELisp-basedmail
clients, so it should not depend on any Gnus code.

DP> As a guide, if someone does want to do this, from when I investigated this
DP> some time back I found the best strategy was likely to be:

DP> 1. Define an API for "obtain the MIME structure" in the back-ends.
DP> 2. Implement it in the "front-end" based on fetching the whole article.
DP>    You need this later anyhow.
DP> 3. Replace all the bits of code scattered all over the place that take the
DP>    whole article and look for structure with use of the API above.
DP>    Just keep fetching the whole article and all at this point, when it comes
DP>    to getting the data out.

DP> 4. Replace the front-end MIME structure code with a back-end API, and use the
DP>    same code for the default implementation for back-ends that don't provide
DP>    an optimized version.
DP> 5. Maybe, add an IMAP version at this point.

DP> 6. Define an API for fetching a MIME part based on the structure.
DP> 7. Provide a default implementation, then move down code that parses the
DP>    article into parts into the content.

DP> 8. Replace all the code that assumes it sits in a full article buffer with
DP>    something like '(gnes-in-article-buffer &body)' that will build a suitable
DP>    buffer, then run the old code.
DP> 9. Gradually replace instances of that with content-aware versions of the
DP>    same.

DP> That way you can release something useful along the way, building useful code
DP> at each stage, without having to get everything done at once.  You can look
DP> forward to person-months if not person-years of work in the transition,
DP> though.

Your itemized list seems very thorough and you've obviously thought
about this more than I.  Are you interested in coordinating this effort?
If we present it appropriately (especially that it benefits all ELisp
mail clients, not just Gnus) we may get more help from the core Emacs
developers.  I could work on the backend fetch and the
async-bodypart-fetch library, but I doubt I would be helpful on the
front end, especially the code that generates the article buffer from a
message body.

Ted