Gnus' speed

Gnus' speed

[Daniel Clemente]

Hi,

  I think operations in Gnus are not only slower than many other clients, but also too slow to keep the user happy. I suffer the consequences since I use nnimap in Gnus with Gmail (note that the Gmail web interface is much faster).

  To discuss how to improve Gnus' speed I started this page on EmacsWiki
http://www.emacswiki.org/emacs/GnusSpeed

  There we can discuss possible solutions, workarounds, … and share experiences.


-- Daniel

Re: Gnus' speed

[Leo]

On 2009-07-28 19:34 +0100, Daniel Clemente wrote:
> Hi,
>
>   I think operations in Gnus are not only slower than many other
> clients, but also too slow to keep the user happy. I suffer the
> consequences since I use nnimap in Gnus with Gmail (note that the
> Gmail web interface is much faster).
>
>   To discuss how to improve Gnus' speed I started this page on
> EmacsWiki
> http://www.emacswiki.org/emacs/GnusSpeed

I disabled gnus agent and adaptive scoring (the adaptive score files can
go up to tens of megabytes) and Gnus has been pretty fast except when
downloading attachments. I think it is due to the external program used
(gnutls?). I suspected by using openssl it will be faster but I have
never got it to work with gnus. So now I use gmail for emails and Gnus
for news.

>   There we can discuss possible solutions, workarounds, … and share
> experiences.
>
>
> -- Daniel

-- 
Leo's Emacs uptime: 48 days, 7 hours, 13 minutes, 25 seconds

Re: Gnus' speed

[CHENG Gao]

*On Tue, 28 Jul 2009 20:34:48 +0200
* Also sprach Daniel Clemente <dcl441-bugs@yahoo.com>:

> Hi,
>
>   I think operations in Gnus are not only slower than many other clients, but also too slow to keep the user happy. I suffer the consequences since I use nnimap in Gnus with Gmail (note that the Gmail web interface is much faster).
>
>   To discuss how to improve Gnus' speed I started this page on EmacsWiki
> http://www.emacswiki.org/emacs/GnusSpeed
>
>   There we can discuss possible solutions, workarounds, … and share experiences.
>
>
> -- Daniel

I wont say Gnus it TOO slow, and I'm a happy Gnus user. It's true Gnus
could be fairly slow for some jobs.

To make my life a little better, I tried to use Gnus in full offline
mode.

I use fetchmail+maildrop to get mails and deliver to local Maildir, then
use nnmaildir to read mails. Good enough for me.

nnrss is really a headache, esp. when I have many subscriptions. Gnus is
slow in comparison with other dedicated RSS readers. I use NetNewsWire
and can see the speed gap. I am trying to find a solution to make RSS
reading offline. 

I found RSS2Leafnode
(http://user42.tuxfamily.org/rss2leafnode/index.html), but it needs
Leafnode 2, and Macports only has Leafnode 1. I need find some time to
install Leafnode 2 from source and check if RSS2Leafnode is good enough.

Another candidate is nntp//rss (http://www.methodize.org/nntprss/). But
its download link is b0rked.

Yet another candidate is rssdrop
(http://search.cpan.org/~acg/rssdrop-0.2/), which could deliver RSS
feeds to local Maildir.

Re: Gnus' speed

[Daniel Clemente]

El dt, jul 28 2009 a les 23:03, Leo va escriure:
>> http://www.emacswiki.org/emacs/GnusSpeed
>
> I disabled gnus agent and adaptive scoring (the adaptive score files can
> go up to tens of megabytes) 

  I didn't know about adaptive scoring, but it seems it is already disabled by default.
  I added this tip to the wiki; thanks.

> and Gnus has been pretty fast except when
> downloading attachments.
  I experience this too. When I press ENTER, Gnus downloads all attachments by default, even if they are >10 Mb and I don't want them.
  Probably there's a variable which says „only download attachments when I ask you to“, but I don't know which one. gnus-.*attachment didn't match any.

> I think it is due to the external program used
> (gnutls?). I suspected by using openssl it will be faster but I have
> never got it to work with gnus. So now I use gmail for emails and Gnus
> for news.

  That's sad; integration of mail with Emacs is worth it.



-- Daniel

Re: Gnus' speed

[David Engster]

Daniel Clemente <dcl441-bugs@yahoo.com> writes:
> El dt, jul 28 2009 a les 23:03, Leo va escriure:
>>> http://www.emacswiki.org/emacs/GnusSpeed
>>
>> I disabled gnus agent and adaptive scoring (the adaptive score files can
>> go up to tens of megabytes) 
>
>   I didn't know about adaptive scoring, but it seems it is already disabled by default.
>   I added this tip to the wiki; thanks.

Don't have time to edit the Wiki at the moment, so some unsorted general
hints:

* Scoring in general is slow. For maximum speed, one should omit scoring
  completely.

* Set the 'large-newsgroup-initial' group parameter to a small value
  (e.g. 50), so that you get smaller summary buffers.

* Sort articles/threads by number.

* If you use nnimap, put an appropriate server definition in
  gnus-select-method or gnus-secondary-select-methods. Don't use it as a
  foreign server.

* Use group levels to make checking for new mails faster. Use high
  levels (4 or 5) for groups like spam/ham and less important mailing lists
  etc. which you do not need to check regularly. Use
  'gnus-activate-level' to specify which groups you'd like to be
  activated on startup. Put your important mail groups on level 1 and
  use prefix arguments like '1 g' to specify which groups you'd like to
  check for new mail.

* Use shell scripts to retrieve RSS feeds asynchronously (e.g. via
  cron). Set nnrss-use-local to 't' and use
  'nnrss-generate-download-script' to generate the shell script for
  retrieving the feeds. If you use shimbuns, there's 'shimbun-use-local'
  and 'nnshimbun-generate-download-script' which do the same.

* Depending on the IMAP server and the back end it uses, it might be
  wise to keep your groups small. Use expiry to automatically create
  archive groups (see variable nnmail-fancy-expiry-targets). Use
  searching facilities like nnir/nnmairix for an efficient search in
  those archives, so that you don't have to build huge summary buffers
  with thousands of mails.

-David

Re: Gnus' speed

[Leo]

On 2009-07-29 09:03 +0100, Daniel Clemente wrote:
>> and Gnus has been pretty fast except when downloading attachments.
>   I experience this too. When I press ENTER, Gnus downloads all
>   attachments by default, even if they are >10 Mb and I don't want
>   them. Probably there's a variable which says „only download
>   attachments when I ask you to“, but I don't know which one.
>   gnus-.*attachment didn't match any.

I just remembered that I did read somewhere that it downloads the whole
attachment while IMAP support some kind of preview feature, which is not
available in Gnus.

-- 
Leo's Emacs uptime: 48 days, 20 hours, 4 minutes, 51 seconds

Re: Gnus' speed

[Karl Kleinpaste]

David Engster <deng@randomsample.de> writes:
> * Scoring in general is slow. For maximum speed, one should omit
>   scoring completely.

I am surprised that anyone would want to do away with scoring; its
utility is so high in terms of determining what is/isn't worth reading
that, even if slow, it is too valuable to ignore.

That said, I don't find it slow.  I even have an "all" scorefile along
with the usual per-group scorefiles, and I don't perceive any special
penalty from using scoring.

Scorefile processing is immensely faster than the older killfile
processing (which capability still exists), by a factor of 3 or 4, as I
recall from tests we ran about 10 years ago.  I don't believe the
fundamental structure of either scoring or killing has changed in a
geological epoch.  Scoring was a humongous step forward in both
capability (fine granularity; display enhancements) over killing (blind,
deaf, and dumb as to any nuances).

Re: Gnus' speed

[David Engster]

Karl Kleinpaste <karl@kleinpaste.org> writes:
> David Engster <deng@randomsample.de> writes:
>> * Scoring in general is slow. For maximum speed, one should omit
>>   scoring completely.
>
> I am surprised that anyone would want to do away with scoring; its
> utility is so high in terms of determining what is/isn't worth reading
> that, even if slow, it is too valuable to ignore.
>
> That said, I don't find it slow.  I even have an "all" scorefile along
> with the usual per-group scorefiles, and I don't perceive any special
> penalty from using scoring.

It depends on the scoring you do. If you score against the whole head or
even the body, scoring becomes incredibly slow since Gnus has to request
the head/body, resp.

For example, I have a group with a score file like

(("head"
  ("somestring" nil nil s)))

and reading its ~940 articles takes ages:

gnus-score-headers                                                            1           40.348502     40.348502
gnus-score-body                                                               1           40.346746     40.346746
gnus-request-head                                                             938         40.214296000  0.0428723837

Scoring just against the basic headers like from/to/subject probably
does not have this problem.

-David

Re: Gnus' speed

[Karl Kleinpaste]

David Engster <deng@randomsample.de> writes:
> It depends on the scoring you do. If you score against the whole head or
> even the body, scoring becomes incredibly slow since Gnus has to request
> the head/body, resp.

For those who have reasonable access to your news admins, and if your
server runs INN of any recent vintage, you can do away with most needs
for 'body" search using keywords.

Two aspects:
- The admin can tell INN to include Keywords in overviews simply by
adding it to overview.fmt.  Trivial, harmless, cost-free.
- The admin can enable INN's auto-generated Keywords support.  This eats
some CPU time as articles pass into the system, but not nearly as much
as was once feared.  Basically, the enabling of this causes innd to
analyze articles for most-used words.  Having found the top N of these,
it cobbles up a fake Keywords for the overview without actually changing
the article content.

In either case, for actual original keywords of INN's own keyword
analysis, the Keywords header can be scored by Gnus.  Notice when you do
scoring that there is an 'e' option, for "extra" headers.  There are a
couple variables to set:
- gnus-extra-headers for basic scoring capability, and 
- nnmail-extra-headers, if you want this sort of support for incoming
  mail as well.
E.g.
(setq gnus-extra-headers
      '(Keywords Newsgroups NNTP-Posting-Host To X-Moodwatch))

Other headers besides Keywords can be added to overviews.  My
overview.fmt contains:

Keywords:full
Newsgroups:full
NNTP-Posting-Host:full
Content-Type:full

- Newsgroups is useful, distinct from Xref, because local newsgroup
support may not equal groups to which an article was actually posted.
- NNTP-Posting-Host is useful for "all" scoring-into-oblivion when mass
spam attacks occur.
- Content-Type is useful because I score users based on whether they
bother me with (notably) text/html.

Re: Gnus' speed

[David Engster]

Karl Kleinpaste <karl@kleinpaste.org> writes:
> David Engster <deng@randomsample.de> writes:
>> It depends on the scoring you do. If you score against the whole head or
>> even the body, scoring becomes incredibly slow since Gnus has to request
>> the head/body, resp.
>
> For those who have reasonable access to your news admins, and if your
> server runs INN of any recent vintage, you can do away with most needs
> for 'body" search using keywords.

You are right.

My general assessment "scoring is slow" is not correct. It's not the
actual scoring that is slow, but only the retrieval of additional
header/body information from the article, which however can often be
avoided as you describe.

-David

nnrss through Google Reader (was: Gnus' speed)

[Ted Zlatanov]

On Wed, 29 Jul 2009 15:07:00 +0800 CHENG Gao <chenggao@gmail.com> wrote: 

CG> nnrss is really a headache, esp. when I have many subscriptions. Gnus is
CG> slow in comparison with other dedicated RSS readers. I use NetNewsWire
CG> and can see the speed gap. I am trying to find a solution to make RSS
CG> reading offline. 
...
CG> Another candidate is nntp//rss (http://www.methodize.org/nntprss/). But
CG> its download link is b0rked.

I use it and have the JAR file.  It's OK, but has trouble with some
feeds.

I'd really like to use Google Reader.  Its protocol has been
reverse-engineered here:

http://code.google.com/p/pyrfeed/wiki/GoogleReaderAPI 

in case anyone is interested.  The API may change before release so any
code written may have to be redone.  Looks like the easy way to get
all the current headlines is:

https://www.google.com/reader/atom/user/-/state/com.google/reading-list?xt=user/-/state/com.google/read

according to one comment.

This would be a nice mix of nnkiboze and nnrss, but all the work is done
at the Google server and you're not killing the host of the RSS feed
with repeated requests.  You can get the headlines for a specific feed
only, too.

Ted

Re: Gnus' speed

[Ted Zlatanov]

On Wed, 29 Jul 2009 10:03:05 +0200 Daniel Clemente <dcl441-bugs@yahoo.com> wrote: 

DC> El dt, jul 28 2009 a les 23:03, Leo va escriure:
>> and Gnus has been pretty fast except when
>> downloading attachments.

DC>   I experience this too. When I press ENTER, Gnus downloads all
DC>   attachments by default, even if they are >10 Mb and I don't want
DC>   them.  Probably there's a variable which says „only download
DC>   attachments when I ask you to“, but I don't know which
DC>   one. gnus-.*attachment didn't match any.

IMAP, at least, supports downloading any part of a MIME message
individually.  I don't believe Gnus uses that functionality but it
could.  I think the backend rewrite would be fairly minor, but there are
many internal and user-visible functions in Gnus that also will need to
be modified.  It's a lot of work.

Ted

Re: Gnus' speed

[Ted Zlatanov]

On Wed, 29 Jul 2009 10:44:36 +0200 David Engster <deng@randomsample.de> wrote: 

DE> * If you use nnimap, put an appropriate server definition in
DE>   gnus-select-method or gnus-secondary-select-methods. Don't use it as a
DE>   foreign server.

It's always annoyed me that this is the case.  Why are foreign servers
slower to check mail?  It looks like they re-run the hook to get new
news on every group, is that it?

Ted

Re: Gnus' speed

[Ted Zlatanov]

On Wed, 29 Jul 2009 14:44:53 +0200 David Engster <deng@randomsample.de> wrote: 

DE> My general assessment "scoring is slow" is not correct. It's not the
DE> actual scoring that is slow, but only the retrieval of additional
DE> header/body information from the article, which however can often be
DE> avoided as you describe.

This connects with my other comment on IMAP retrieval.  But there's a
bigger problem: Gnus is by design synchronous.  You enter a group, then
wait for the summary buffer to be built.  Gnus doesn't have the concept
of "enter a buffer and let the articles come in asynchronously" and I
doubt it's possible without some multithreading support in Emacs Lisp,
which has been discussed many times but is probably far in the future
(at least a year, judging by threads in emacs-devel).  Gnus blocks on
many other operations too.

Ted

Re: Gnus' speed

[Reiner Steib]

On Wed, Jul 29 2009, David Engster wrote:

> For example, I have a group with a score file like
>
> (("head"
>   ("somestring" nil nil s)))
>
> and reading its ~940 articles takes ages:
[...]
> Scoring just against the basic headers like from/to/subject probably
> does not have this problem.

,----[ <f1> v gnus-inhibit-slow-scoring RET ]
| gnus-inhibit-slow-scoring is a variable defined in `gnus-score.el'.
| Its value is "^nntp[+:]"
| 
| Documentation:
| Inhibit slow scoring, e.g. scoring on headers or body.
| 
| If a regexp, scoring on headers or body is inhibited if the group
| matches the regexp.  If it is t, scoring on headers or body is
| inhibited for all groups.
| 
| You can customize this variable.
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus' speed

[Reiner Steib]

On Tue, Jul 28 2009, Daniel Clemente wrote:

>   To discuss how to improve Gnus' speed I started this page on EmacsWiki
> http://www.emacswiki.org/emacs/GnusSpeed
>
>   There we can discuss possible solutions, workarounds, … and share
>   experiences.

I think such tips should be in the official manual [1] *if* [2] they
are useful and correct.

[1] We already have (info "(gnus)Slow Machine"), we could add a more
    general node under (info "(gnus)Customization") or even (info
    "(gnus)Appendices").

[2] I didn't read this page completely and I don't have time to check
    every edit of some wiki page.  More likely I (and other
    developers) will review patches and additions for the manual.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus' speed

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> But there's a bigger problem: Gnus is by design synchronous.  You
Ted> enter a group, then wait for the summary buffer to be built.  Gnus
Ted> doesn't have the concept of "enter a buffer and let the articles
Ted> come in asynchronously" and I doubt it's possible without some
Ted> multithreading support in Emacs Lisp, which has been discussed many
Ted> times but is probably far in the future (at least a year, judging
Ted> by threads in emacs-devel).  Gnus blocks on many other operations
Ted> too.

Do you mean, this can't be done without threads because it means too
much rewriting of Gnus?

It seems to me that in theory Gnus could work asynchronously by doing
work in process filters.

Tom

Re: Gnus' speed

[Kevin Ryde]

Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
> (gnus)Slow Machine

I never understood why startup takes quite so long (with those
newsgroups check bits set to nil, and only a dozen or so local nntp and
nnml subscribed, but lots unsubscribed).  Maybe it's just the quantity
of code loaded.  Did someone profile it a few years ago?

Re: Gnus' speed

[Kevin Ryde]

CHENG Gao <chenggao@gmail.com> writes:
>
> RSS2Leafnode ... needs Leafnode 2, and Macports only has Leafnode 1.

Any nntp with local (ie. non-propagated) groups you can post to is ok.
I always meant to check with maybe cnews or inn, but was too afraid to
bork my spool and/or newsrc.  It doesn't have to be on your exact local
machine, if something nearby can serve.

To stay vaguely on-topic, pre-rendering of html helps gnus by having
plain text in the messages in the spool, which of course can be faster
to go through than w3m or whatnot from within gnus.

Re: Gnus' speed

[Daniel Pittman]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Wed, 29 Jul 2009 10:03:05 +0200 Daniel Clemente <dcl441-bugs@yahoo.com> wrote: 
>
> DC> El dt, jul 28 2009 a les 23:03, Leo va escriure:
>>> and Gnus has been pretty fast except when
>>> downloading attachments.
>
> DC>   I experience this too. When I press ENTER, Gnus downloads all
> DC>   attachments by default, even if they are >10 Mb and I don't want
> DC>   them.  Probably there's a variable which says „only download
> DC>   attachments when I ask you to“, but I don't know which
> DC>   one. gnus-.*attachment didn't match any.
>
> IMAP, at least, supports downloading any part of a MIME message
> individually.  I don't believe Gnus uses that functionality but it
> could.

It does, indeed.

> I think the backend rewrite would be fairly minor, but there are many
> internal and user-visible functions in Gnus that also will need to be
> modified.  It's a lot of work.

I think that saying "fairly minor" is misleading here: adding the capability
to the IMAP backend isn't enormously hard, but defining a sound (and portable)
API, and implementing it, is quite non-trivial.

As a guide, if someone does want to do this, from when I investigated this
some time back I found the best strategy was likely to be:

1. Define an API for "obtain the MIME structure" in the back-ends.
2. Implement it in the "front-end" based on fetching the whole article.
   You need this later anyhow.
3. Replace all the bits of code scattered all over the place that take the
   whole article and look for structure with use of the API above.
   Just keep fetching the whole article and all at this point, when it comes
   to getting the data out.

4. Replace the front-end MIME structure code with a back-end API, and use the
   same code for the default implementation for back-ends that don't provide
   an optimized version.
5. Maybe, add an IMAP version at this point.

6. Define an API for fetching a MIME part based on the structure.
7. Provide a default implementation, then move down code that parses the
   article into parts into the content.

8. Replace all the code that assumes it sits in a full article buffer with
   something like '(gnes-in-article-buffer &body)' that will build a suitable
   buffer, then run the old code.
9. Gradually replace instances of that with content-aware versions of the
   same.

That way you can release something useful along the way, building useful code
at each stage, without having to get everything done at once.  You can look
forward to person-months if not person-years of work in the transition,
though.

Regards,
        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Gnus' speed

[Daniel Pittman]

Daniel Clemente <dcl441-bugs@yahoo.com> writes:
> El dt, jul 28 2009 a les 23:03, Leo va escriure:

[...]

>> and Gnus has been pretty fast except when
>> downloading attachments.
>   I experience this too. When I press ENTER, Gnus downloads all attachments
>   by default, even if they are >10 Mb and I don't want them.
>
>   Probably there's a variable which says „only download attachments when I
>   ask you to“, but I don't know which one. gnus-.*attachment didn't match
>   any.

Nope.  This is quite non-trivial, because IMAP is the only storage format that
provides an API to the MIME structure, and IMAP is a late arrival in the world
of Gnus.

Regards,
        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Gnus' speed

[Daniel Pittman]

Tom Tromey <tromey@redhat.com> writes:
>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Ted> But there's a bigger problem: Gnus is by design synchronous.  You
> Ted> enter a group, then wait for the summary buffer to be built.  Gnus
> Ted> doesn't have the concept of "enter a buffer and let the articles
> Ted> come in asynchronously" and I doubt it's possible without some
> Ted> multithreading support in Emacs Lisp, which has been discussed many
> Ted> times but is probably far in the future (at least a year, judging
> Ted> by threads in emacs-devel).  Gnus blocks on many other operations
> Ted> too.
>
> Do you mean, this can't be done without threads because it means too
> much rewriting of Gnus?

No, because it requires rewriting of *Emacs* to add support for threads.

That, in turn, would require rebuilding the Emacs Lisp engine to support
concurrency, which in turn will drive you completely insane because a
completely dynamically scoped language doesn't play nicely with threads.

However, if you are seriously interested there is currently some little effort
in either implementing ELisp in Guile, or adding threads, discussed in the
Emacs development groups over the last few months.

> It seems to me that in theory Gnus could work asynchronously by doing work
> in process filters.

Actually, you could also do it with idle timers, and/or redisplay stuff, but
this is enormously non-trivial to achieve and requires rewriting most of the
core of Gnus.

Regards,
        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Gnus' speed

[Adam Sjøgren]

On Thu, 30 Jul 2009 10:29:11 +1000, Kevin wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> (gnus)Slow Machine

> I never understood why startup takes quite so long (with those
> newsgroups check bits set to nil, and only a dozen or so local nntp and
> nnml subscribed, but lots unsubscribed).  Maybe it's just the quantity
> of code loaded.  Did someone profile it a few years ago?

Could some of you quantify this 'slow startup'?

Maybe I've just gotten used to the speed, or lack thereof, of Gnus, so
it doesn't feel so slow to me - maybe the GHz-machines has helped, or
maybe I use Gnus differently - I don't know.

On my work-machine (Intel Core2 1.86GHz), where I only read email (with
nnml) it takes ~4s.

At home (Athlon Dual Core 4850e 2.5GHz) where I connect to three
news-servers (a local, gmane and an external - but only 112 groups in
total), read email (with nnml) and have 18 nnrss groups (fetched via
cron), it takes ~14s.

(I measured simply by stop-watching from M-x gnus until the group buffer
shows up and is ready.)


  Best regards,

    Adam

-- 
 "Angels can fly because they take themselves lightly."       Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Gnus' speed

[Ted Zlatanov]

On Thu, 30 Jul 2009 15:58:37 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think the backend rewrite would be fairly minor, but there are many
>> internal and user-visible functions in Gnus that also will need to be
>> modified.  It's a lot of work.

DP> I think that saying "fairly minor" is misleading here: adding the capability
DP> to the IMAP backend isn't enormously hard, but defining a sound (and portable)
DP> API, and implementing it, is quite non-trivial.

I'd do it with the IMAP convention of "1.2.3" to mean part 3 of part 2
of part 1 as an optional parameter for the backend command to fetch the
body, and an extra backend command to get a message's structure (list of
strings as above with the associated data).  I'd move all the rest of
the code into a separate async-bodypart-fetch library that, given a list
of body parts, can interact with the existing Gnus user-facing
functionality that builds and interacts with the article buffer.  The
async-bodypart-fetch library should be usable by other ELisp-basedmail
clients, so it should not depend on any Gnus code.

DP> As a guide, if someone does want to do this, from when I investigated this
DP> some time back I found the best strategy was likely to be:

DP> 1. Define an API for "obtain the MIME structure" in the back-ends.
DP> 2. Implement it in the "front-end" based on fetching the whole article.
DP>    You need this later anyhow.
DP> 3. Replace all the bits of code scattered all over the place that take the
DP>    whole article and look for structure with use of the API above.
DP>    Just keep fetching the whole article and all at this point, when it comes
DP>    to getting the data out.

DP> 4. Replace the front-end MIME structure code with a back-end API, and use the
DP>    same code for the default implementation for back-ends that don't provide
DP>    an optimized version.
DP> 5. Maybe, add an IMAP version at this point.

DP> 6. Define an API for fetching a MIME part based on the structure.
DP> 7. Provide a default implementation, then move down code that parses the
DP>    article into parts into the content.

DP> 8. Replace all the code that assumes it sits in a full article buffer with
DP>    something like '(gnes-in-article-buffer &body)' that will build a suitable
DP>    buffer, then run the old code.
DP> 9. Gradually replace instances of that with content-aware versions of the
DP>    same.

DP> That way you can release something useful along the way, building useful code
DP> at each stage, without having to get everything done at once.  You can look
DP> forward to person-months if not person-years of work in the transition,
DP> though.

Your itemized list seems very thorough and you've obviously thought
about this more than I.  Are you interested in coordinating this effort?
If we present it appropriately (especially that it benefits all ELisp
mail clients, not just Gnus) we may get more help from the core Emacs
developers.  I could work on the backend fetch and the
async-bodypart-fetch library, but I doubt I would be helpful on the
front end, especially the code that generates the article buffer from a
message body.

Ted

Re: Gnus' speed

[Daniel Pittman]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Thu, 30 Jul 2009 15:58:37 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 
>
> DP> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> I think the backend rewrite would be fairly minor, but there are many
>>> internal and user-visible functions in Gnus that also will need to be
>>> modified.  It's a lot of work.
>
> DP> I think that saying "fairly minor" is misleading here: adding the capability
> DP> to the IMAP backend isn't enormously hard, but defining a sound (and portable)
> DP> API, and implementing it, is quite non-trivial.
>
> I'd do it with the IMAP convention of "1.2.3" to mean part 3 of part 2 of
> part 1 as an optional parameter for the backend command to fetch the body,
> and an extra backend command to get a message's structure (list of strings
> as above with the associated data).

Given that the IMAP data structure is more or less an sexpr, using that fairly
directly made sense to me.  Given it is really the only model, too, with any
widespread use ...

[...]

> Your itemized list seems very thorough and you've obviously thought
> about this more than I.  Are you interested in coordinating this effort?

At the moment, no, otherwise I would probably be writing the code instead of
just talking about it.  As it is, I looked at this a couple of years back, so
my knowledge might well be out-of-date.

Sorry.  I hate to just offer advice and not actual code, but I don't think
volunteering for the later will help anyone out.

Regards,
        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Gnus' speed

[Bill White]

On Wed Jul 29 2009 at 13:30, Ted Zlatanov <tzz@lifelogs.com> wrote:

> On Wed, 29 Jul 2009 14:44:53 +0200 David Engster <deng@randomsample.de> wrote: 
>
> DE> My general assessment "scoring is slow" is not correct. It's not
> DE> the actual scoring that is slow, but only the retrieval of
> DE> additional header/body information from the article, which however
> DE> can often be avoided as you describe.
>
> This connects with my other comment on IMAP retrieval.  But there's a
> bigger problem: Gnus is by design synchronous.  You enter a group,
> then wait for the summary buffer to be built.  Gnus doesn't have the
> concept of "enter a buffer and let the articles come in
> asynchronously" and I doubt it's possible without some multithreading
> support in Emacs Lisp, which has been discussed many times but is
> probably far in the future (at least a year, judging by threads in
> emacs-devel).  Gnus blocks on many other operations too.

To get around that problem I run gnus in a separate instance of emacs,
and I leave it running there for as long as that emacs is alive -
usually many days.

Cheers -

bw
-- 
Bill White . billw@wolfram.com . http://members.wolfram.com/billw
"No ma'am, we're musicians."

Re: nnrss through Google Reader

[Ted Zlatanov]

On Wed, 29 Jul 2009 13:20:09 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I'd really like to use Google Reader.  Its protocol has been
TZ> reverse-engineered here:

TZ> http://code.google.com/p/pyrfeed/wiki/GoogleReaderAPI 

TZ> in case anyone is interested.  The API may change before release so any
TZ> code written may have to be redone.  Looks like the easy way to get
TZ> all the current headlines is:

TZ> https://www.google.com/reader/atom/user/-/state/com.google/reading-list?xt=user/-/state/com.google/read

TZ> according to one comment.

TZ> This would be a nice mix of nnkiboze and nnrss, but all the work is done
TZ> at the Google server and you're not killing the host of the RSS feed
TZ> with repeated requests.  You can get the headlines for a specific feed
TZ> only, too.

Funnily enough, my other RSS reader (NetNewsWire) now synchronizes with
Google Reader, so if Gnus could do it too, I'd be quite happy.

I looked into it, and Reader is an OK web interface on top of a pretty
good feed database.  Gnus can probably do what NetNewsWire and others
have done to integrate.  Now, NNW had help from Google directly, so I
assume they are using an "approved" API.  We'd have to use the
reverse-engineered knowledge, which is risky.  

In addition, I'm not sure if supporting a proprietary product is the
right thing, or if I should think about a more generic framework that
supports Google Reader but doesn't require it.  Tighter integration is
surely better in the short term.

Ted

Re: Gnus' speed

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Wed, 29 Jul 2009 10:03:05 +0200 Daniel Clemente <dcl441-bugs@yahoo.com> wrote: 
DC> El dt, jul 28 2009 a les 23:03, Leo va escriure:
>>> and Gnus has been pretty fast except when
>>> downloading attachments.

DC> I experience this too. When I press ENTER, Gnus downloads all
DC> attachments by default, even if they are >10 Mb and I don't want
DC> them.  Probably there's a variable which says „only download
DC> attachments when I ask you to“, but I don't know which
DC> one. gnus-.*attachment didn't match any.

> IMAP, at least, supports downloading any part of a MIME message
> individually.  I don't believe Gnus uses that functionality but it
> could.  I think the backend rewrite would be fairly minor, but there
> are many internal and user-visible functions in Gnus that also will
> need to be modified.  It's a lot of work.

Partial IMAP downloads have been discussed many times on this list.
Heh... you've been part of such discussions yourself...:-)
	http://thread.gmane.org/gmane.emacs.gnus.general/62566/focus=62942

Here's my idea for a solution:
	http://article.gmane.org/gmane.emacs.gnus.general/45714
	nntp://news.gmane.org/gmane.emacs.gnus.general/45714

Re: Gnus' speed

[Ted Zlatanov]

On Sun, 02 Aug 2009 16:20:14 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Partial IMAP downloads have been discussed many times on this list.
SB> Heh... you've been part of such discussions yourself...:-)
SB> 	http://thread.gmane.org/gmane.emacs.gnus.general/62566/focus=62942

3+ years ago...  How time flies!

SB> Here's my idea for a solution:
SB> 	http://article.gmane.org/gmane.emacs.gnus.general/45714
SB> 	nntp://news.gmane.org/gmane.emacs.gnus.general/45714

Steinar originally wrote:
> A quick revisit of the idea:
>  - in the agent cache, replace the delayed parts with a
>    message/external-body URL part
> 	<http://www.faqs.org/rfcs/rfc2017.html>
>    where the URL is an IMAP URL:
> 	<http://www.faqs.org/rfcs/rfc2192.html>
>  - when the user wishes to view or save the delayed part, it is
>    downloaded from the server, and the message/external-body is
>    replaced with the actual part

I think that's a reasonable plan.  I don't know the agent, so I can't do
the agent cache piece, and I don't know much about generating the
article buffer, but I can probably help with the IMAP interaction and
the backend modifications.

Are you interested in working on this?  Is anyone else?

Ted

Re: Gnus' speed

[Kevin Ryde]

asjo@koldfront.dk (Adam Sjøgren) writes:
>
> On my work-machine (Intel Core2 1.86GHz), where I only read email (with
> nnml) it takes ~4s.

About 15 secs, with 1/6 the raw mhz :).  Looks like it's the code
loading taking the time, rather than initializations.

Re: Gnus' speed

[Daniel Clemente]

  Many thanks for the comments in this thread. I have updated the wiki page with all these tips, some of them were copied unchanged (in case you don't want them there, please tell me or edit directly the page).

http://www.emacswiki.org/emacs/GnusSpeed

Re: Gnus' speed

[Daniel Clemente]

Hi,

El dc, jul 29 2009 a les 20:55, Reiner Steib va escriure:
>>
>> http://www.emacswiki.org/emacs/GnusSpeed
>>
> I think such tips should be in the official manual [1] *if* [2] they
> are useful and correct.
>

  the target user for this information are all Gnus users (also Emacs beginners with few months' experience), and I think that the wiki is more or less easy to edit.
  Writing into the official manual sets too high entry barriers for collaboration: find latest version, read it a bit, learn CVS, look for the correct section, write well, write 100% correct stuff, sign legal papers, wait for approval, wait more, … The wiki is instantaneous and allows both complete and unfinished content.

  Therefore I suggest writing more in EmacsWiki, and when there is more information and more consolidated and validated, integrate it into the manual.

  The wiki can also be useful to coordinate other development tasks.

-- Daniel

Re: Gnus' speed

[Steinar Bang]

>>>>> Daniel Clemente <dcl441-bugs@yahoo.com>:

>   Many thanks for the comments in this thread. I have updated the wiki
>   page with all these tips, some of them were copied unchanged (in
>   case you don't want them there, please tell me or edit directly the
>   page).

> http://www.emacswiki.org/emacs/GnusSpeed

One title there is misleading:
  http://www.emacswiki.org/emacs/GnusSpeed#toc5

"Gnus doesn't support MIME decoding".

Gnus does indeed support MIME decoding.

What the Gnus IMAP backend does not support, is download of individual
MIME separated message parts, even though that is functionality
supported by the IMAP spec.

Don't know how to put that more compactly, but the current title should
be changed.

Re: Gnus' speed

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> One title there is misleading:
>   http://www.emacswiki.org/emacs/GnusSpeed#toc5

> "Gnus doesn't support MIME decoding".

> Gnus does indeed support MIME decoding.

> What the Gnus IMAP backend does not support, is download of individual
> MIME separated message parts, even though that is functionality
> supported by the IMAP spec.

> Don't know how to put that more compactly, but the current title should
> be changed.

"Gnus nnimap doesn't do partial download of messages", perhaps?

Re: Gnus' speed

[Daniel Clemente]

El dc, ago 05 2009 a les 07:55, Steinar Bang va escriure:
>> Don't know how to put that more compactly, but the current title should
>> be changed.
>
> "Gnus nnimap doesn't do partial download of messages", perhaps?

  Ok, changed to something better. But you may change directly what you want, it's *our* wiki.

  Thanks

Re: Gnus' speed

[Steinar Bang]

>>>>> Daniel Clemente <dcl441-bugs@yahoo.com>:

>   Ok, changed to something better. But you may change directly what
>   you want, it's *our* wiki.

I know.  And I say similar things to others suggesting changes to stuff
I've written on wikis.

But it still feels rude to change someone else's text.

Re: Gnus' speed

[Miles Bader]

David Engster <deng@randomsample.de> writes:
>> I am surprised that anyone would want to do away with scoring; its
>> utility is so high in terms of determining what is/isn't worth reading
>> that, even if slow, it is too valuable to ignore.
>>
>> That said, I don't find it slow.  I even have an "all" scorefile along
>> with the usual per-group scorefiles, and I don't perceive any special
>> penalty from using scoring.
>
> It depends on the scoring you do. If you score against the whole head or
> even the body, scoring becomes incredibly slow since Gnus has to request
> the head/body, resp.

Indeed, but of course the solution is to not add score rules using
all-headers/body, not to disable scoring entirely!

The "normal" scoring rules seem quite fast to me, essentially in the
noise for my usage (mostly netnews though).  I suppose if you're on a
very old and slow machine, you might not want to use them.

And the as the GP says, the utility of scoring is quite high.

-Miles

-- 
Opposition, n. In politics the party that prevents the Goverment from running
amok by hamstringing it.

Re: Gnus' speed

[Miles Bader]

Kevin Ryde <user42@zip.com.au> writes:
> I never understood why startup takes quite so long (with those
> newsgroups check bits set to nil, and only a dozen or so local nntp and
> nnml subscribed, but lots unsubscribed).  Maybe it's just the quantity
> of code loaded.  Did someone profile it a few years ago?

On my work machine, Gnus startup is slow because my home directory and
mail directories are on NFS, and NFS is an insane dog (operations that
don't require much I/O to my homedir are all quite fast).

-Miles

-- 
I'm beginning to think that life is just one long Yoko Ono album; no rhyme
or reason, just a lot of incoherent shrieks and then it's over.  --Ian Wolff

Re: Gnus' speed

[Daniel Pittman]

Miles Bader <miles@gnu.org> writes:
> David Engster <deng@randomsample.de> writes:
>>> I am surprised that anyone would want to do away with scoring; its
>>> utility is so high in terms of determining what is/isn't worth reading
>>> that, even if slow, it is too valuable to ignore.
>>>
>>> That said, I don't find it slow.  I even have an "all" scorefile along
>>> with the usual per-group scorefiles, and I don't perceive any special
>>> penalty from using scoring.
>>
>> It depends on the scoring you do. If you score against the whole head or
>> even the body, scoring becomes incredibly slow since Gnus has to request
>> the head/body, resp.
>
> Indeed, but of course the solution is to not add score rules using
> all-headers/body, not to disable scoring entirely!

...or do something else to address the problem; body rules were quite
reasonable on a six year old machine using a local leafnode NNTP "cache" that
meant no non-local network activity to fetch the body and apply the score.

Regards,
        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons
   Looking for work?  Love Perl?  In Melbourne, Australia?  We are hiring.

Re: Gnus' speed

[Steinar Bang]

>>>>> Miles Bader <miles@gnu.org>:

> Kevin Ryde <user42@zip.com.au> writes:

>> I never understood why startup takes quite so long (with those
>> newsgroups check bits set to nil, and only a dozen or so local nntp
>> and nnml subscribed, but lots unsubscribed).  Maybe it's just the
>> quantity of code loaded.  Did someone profile it a few years ago?

> On my work machine, Gnus startup is slow because my home directory and
> mail directories are on NFS, and NFS is an insane dog (operations that
> don't require much I/O to my homedir are all quite fast).

Heh... I had a 1996/1997 vintage DEC HiNote Ultra, with 24MB of RAM and
1GB of HD, running debian, as my personal machine.

Around 2001 it had become cumbersome to use for Gnus.  The XEmacs
process with Gnus quickly used too much memory for the little machine,
and I had to switch back to using GNU Emacs for Gnusing.  XEmacs+Gnus
became too slow.  And even GNU Emacs+Gnus became slow (when the machine
started paging),... and web pages started using too much huge JPEGs and
GIFs and flash animations... so the machine that had been sufficient
until then, became too slow for web browsing, and ended up in a drawer,
where it still is. 

But right now I'm running Gnus in GNU Emacs (which has all of the
multimedia capabilities, and more, as the old XEmacs), on an Acer Aspire
One 110.  Physically as small as my old HiNote Ultra, but performance
wise a lot faster, with its 1GB of RAM and 8GB of SSD.  The processor is
a lot faster also of course, but the 1GB of RAM is probably the most
important thing.

But heh... the screen size and resolution, is about the same as the old
HiNote Ultra (it was a small and *sweet* machine).  The keyboard is
better, and the AA1's touchpad is better than the HiNote's rollerball.

But I digress... Gnus performance, right... acceptable.

There used to be a problem with sparse IMAP groups, or was it just the
size of the group?  Anyway, my main inbox is pretty large, and
performance hasn't become enough of an issue yet, that I've expired its
content to the archive group.

Re: Gnus' speed

[Miles Bader]

Steinar Bang <sb@dod.no> writes:
> But heh... the screen size and resolution, is about the same as the old
> HiNote Ultra (it was a small and *sweet* machine).  The keyboard is
> better, and the AA1's touchpad is better than the HiNote's rollerball.

Dunno abouto the hinote, but I _loved_ the little trackballs on
Panasonic notebooks -- way, way, more precise than a trackpad.

Sadly, Panasonic succumbed to the groupthink that's almost eliminated
non-trackpad pointer controls on laptops, and those little trackballs
are no more.

Stupid consumers... (but I repeat myself)

-Miles

-- 
Insurrection, n. An unsuccessful revolution.