Faster, lolcat. Faster!

Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Now that Gnus is bug-free (because I haven't hacked much on Gnus the
past years) and has HTML support (because I have this week), I think
it's time to have a long good look at why `g' is so slow.

I mean, it's just querying some mail servers and some nntp servers.  I
though take less than a second, total, if you're not contacting half the
servers in the world.

Just to see where we were, I did a `g' and straced Emacs.

[larsi@quimbies ~/pgnus]$ grep stat /tmp/s2 | wc -l
5832
[larsi@quimbies ~/pgnus]$ grep open /tmp/s2 | wc -l
305

Right.  It stat()-et 5832 files.  5832 files!!1!  And opened and read
and wrote 305 files.  Geez.

So I think there should be room for improvement.

My plan is to do the following:

0) Remove almost all of my Gnus settings, so that I get the "default"
user experience.

1) General optimisations.  Instrument `file-exists-p' and friends and
see what's accessing all these files, and try to clamp down on that, as
well as doing general performance analysis and fixing up functions/areas
that are particularly slow.

2) Feature reduction.  If there are features that take a lot of time
when you do a `g', but don't give compelling results, then they should
default to "off".

3) Streaming.  I think many of the things that talk to servers over the
net can be more streaming than they are now.  I had a peek at pop3, just
to take an example, and I think I gleaned from it that it requests one
message, and then waits for it to arrive, and then requests the next,
etc.  If you just request all the articles, and then wait for them all
to arrive, it'll be oodles faster.  I think there's probably some
opportunities in that area, both for pop3, nntp and imap, as the obvious
candidates (although I haven't actually looked at the code for any of
these in depth).

4) Multi-threading.  This is the most difficult bit, but could yield
good results.  If you're requesting active files from two different nntp
servers, and getting mail, there's really no reason why those three
activities shouldn't happen at the same time.  Implementing asynchronous
things in Emacs is a pain, because you have to use process filters and
the like, but it seems to me (at 4:15, and I couldn't sleep) that it
should be possible to gather up the actions that can be async, launch
them all, wait for them all to finish, and then continue on with the
data that we've now gotten.  But this is a way, way bigger project than
the first three points here, so I'll tackle it last...  if ever...

So there you go.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 04:22:44 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Now that Gnus is bug-free (because I haven't hacked much on Gnus the
LMI> past years) and has HTML support (because I have this week), I think
LMI> it's time to have a long good look at why `g' is so slow.

LMI> I mean, it's just querying some mail servers and some nntp servers.  I
LMI> though take less than a second, total, if you're not contacting half the
LMI> servers in the world.

LMI> Just to see where we were, I did a `g' and straced Emacs.

LMI> [larsi@quimbies ~/pgnus]$ grep stat /tmp/s2 | wc -l
LMI> 5832
LMI> [larsi@quimbies ~/pgnus]$ grep open /tmp/s2 | wc -l
LMI> 305

LMI> Right.  It stat()-et 5832 files.  5832 files!!1!  And opened and read
LMI> and wrote 305 files.  Geez.

In case you have time:

Spooling with nnmaildir is painful with lots of messages.  See the
thread on emacs-devel that has some proposed patches and a long
explanation from Paul Jarc and comments from me.

Ted

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Spooling with nnmaildir is painful with lots of messages.  See the
> thread on emacs-devel that has some proposed patches and a long
> explanation from Paul Jarc and comments from me.

I haven't actually used nnmaildir, so I think I may be the wrong person
to do fixups there.  But I'll have a look at the thread.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Now that Gnus is bug-free (because I haven't hacked much on Gnus the past
> years) and has HTML support (because I have this week), I think it's time to
> have a long good look at why `g' is so slow.
>
> I mean, it's just querying some mail servers and some nntp servers.  I
> though take less than a second, total, if you're not contacting half the
> servers in the world.

[...]

> 3) Streaming.  I think many of the things that talk to servers over the
>    net can be more streaming than they are now.

Probably.  Watch out for nasty pipelining issues with obscure servers.
Not that I have touched anything doing this for years, so can't remember what
they were.  IIRC some NNTP / NNRP servers discard input after some errors,
among other things.

(Not, I suspect, that you are not already all over that.)

[...]

> 4) Multi-threading.  This is the most difficult bit, but could yield
>    good results.  If you're requesting active files from two different nntp
>    servers, and getting mail, there's really no reason why those three
>    activities shouldn't happen at the same time.  Implementing asynchronous
>    things in Emacs is a pain

...but some people have gone ahead and built nicer models over the top of it.
My personal favorite is the heart of `jabber.el'[1], which is a finite state
machine implementation designed to do asynchronous stuff.[2]

(It works very nicely for my chatting with Emacs, anyhow. ;)

        Daniel

Footnotes: 
[1]  http://emacs-jabber.sourceforge.net/

[2]  http://emacs-jabber.git.sourceforge.net/git/gitweb.cgi?p=emacs-jabber/emacs-jabber;a=blob;f=fsm.el;h=512cef0e803faef8d05a666a901a3b6af41b0f4f;hb=HEAD

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> Probably.  Watch out for nasty pipelining issues with obscure servers.
> Not that I have touched anything doing this for years, so can't remember what
> they were.  IIRC some NNTP / NNRP servers discard input after some errors,
> among other things.
>
> (Not, I suspect, that you are not already all over that.)

Yeah, it can be tricky, but I think it should be doable for pop3, at
least.  But I say that before I've tried.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Russ Allbery]

Daniel Pittman <daniel@rimspace.net> writes:

> Probably.  Watch out for nasty pipelining issues with obscure servers.
> Not that I have touched anything doing this for years, so can't remember
> what they were.  IIRC some NNTP / NNRP servers discard input after some
> errors, among other things.

You should be fine with NNTP servers provided that you follow section 3.5
of RFC 3977.  (NNTP calls this pipelining since streaming was already used
for something different in the NNTP protocol.)  We considered requiring a
capability be advertised to indicate that the server supported pipelining,
but found that support was already so widespread that it wasn't necessary.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Faster, lolcat. Faster!

[David Engster]

Lars Magne Ingebrigtsen writes:
> 4) Multi-threading.  This is the most difficult bit, but could yield
> good results.  If you're requesting active files from two different nntp
> servers, and getting mail, there's really no reason why those three
> activities shouldn't happen at the same time.  Implementing asynchronous
> things in Emacs is a pain, because you have to use process filters and
> the like,

There is work on getting support for multi-threading into Emacs
proper. In fact, they already succeeded in running Gnus in its own
thread in an Emacs session.

http://thread.gmane.org/gmane.emacs.devel/114603/focus=115908

-David

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> There is work on getting support for multi-threading into Emacs
> proper. In fact, they already succeeded in running Gnus in its own
> thread in an Emacs session.
>
> http://thread.gmane.org/gmane.emacs.devel/114603/focus=115908

Awesome.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Tom Tromey]

>>>>> "David" == David Engster <deng@randomsample.de> writes:

Replying to an old-ish note.

Lars> Lars Magne Ingebrigtsen writes:
Lars> 4) Multi-threading.  This is the most difficult bit, but could yield
Lars> good results.  If you're requesting active files from two different nntp
Lars> servers, and getting mail, there's really no reason why those three
Lars> activities shouldn't happen at the same time.  Implementing asynchronous
Lars> things in Emacs is a pain, because you have to use process filters and
Lars> the like,

David> There is work on getting support for multi-threading into Emacs
David> proper. In fact, they already succeeded in running Gnus in its own
David> thread in an Emacs session.

If it is not too hard, I think it would probably be preferable to fix
this in Gnus.  The multi-threading work is still quite some ways away.

Tom

Re: Faster, lolcat. Faster!

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Right.  It stat()-et 5832 files.  5832 files!!1!  And opened and read
> and wrote 305 files.  Geez.

If it is that many, it sounds like it has opened a lot of agent
files...? 

Could also be score files for the groups...?

But that is just guessing.  Your instrumentation will of course tell
what is touched in due time.

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 04:22:44 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Now that Gnus is bug-free (because I haven't hacked much on Gnus the
LMI> past years) and has HTML support (because I have this week), I think
LMI> it's time to have a long good look at why `g' is so slow.

Another performance issue I've noticed is that N foreign groups on the
same server (subscribed through the server buffer and without an
explicitly configured server entry) seem to always be slower than N
groups on an explicitly configured (primary or secondary) server.  This
is definitely true for IMAP; I don't know if it's just a nnimap issue.

In the "too slow to be done properly" category are the unread/unseen
nnimap counts which as you know are usually inaccurate.  We've had
patches over the years but I don't think anyone has actually solved the
problem.

Ted

Re: Faster, lolcat. Faster!

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Thu, Sep 02 2010, Ted Zlatanov wrote: 

> In the "too slow to be done properly" category are the 
> unread/unseen nnimap counts which as you know are usually 
> inaccurate.  We've had patches over the years but I don't think 
> anyone has actually solved the problem. 

+1

The way I fix it is by running M-g over all the group.
Which is even slower.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Faster, lolcat. Faster!

[Frank Schmitt]

Julien Danjou <julien@danjou.info> writes:

> On Thu, Sep 02 2010, Ted Zlatanov wrote: 
>
>> In the "too slow to be done properly" category are the unread/unseen
>> nnimap counts which as you know are usually inaccurate.  We've had
>> patches over the years but I don't think anyone has actually solved
>> the problem. 
>
> +1
>
> The way I fix it is by running M-g over all the group.
> Which is even slower.

me2. This really, really sucks.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: Faster, lolcat. Faster!

[David Engster]

Frank Schmitt writes:
> Julien Danjou <julien@danjou.info> writes:
>
>> On Thu, Sep 02 2010, Ted Zlatanov wrote: 
>>
>>> In the "too slow to be done properly" category are the unread/unseen
>>> nnimap counts which as you know are usually inaccurate.  We've had
>>> patches over the years but I don't think anyone has actually solved
>>> the problem. 
>>
>> +1
>>
>> The way I fix it is by running M-g over all the group.
>> Which is even slower.
>
> me2. This really, really sucks.

This was last discussed here:

http://article.gmane.org/gmane.emacs.gnus.general/67238

The idea was to make Gnus able to see the whole list of available
article numbers on the server. Currently, due to its roots in NNTP, it
only uses the active information (lowest . highest). 

Vitaly had some patches which extended gnus-active to either be a cons
or a list, containing the old active information or the full list of
article numbers, resp. My opinion was that it would be better to leave
gnus-active alone and introduce a new function in the back end API for
this. In fact, there already is an (more or less undocumented) function
`gnus-request-group-articles', which is implemented for NNTP. However,
it's straightforward to do for nnimap, for example:

(deffoo nnimap-request-group-articles (group &optional server)
  (nnimap-possibly-change-group group server)
  (let ((articles (with-current-buffer nnimap-server-buffer
		    (imap-search "ALL"))))
    (if articles
      (with-current-buffer nntp-server-buffer
	(erase-buffer)
	(insert 
	 (format "211 %d %d %d %s\n"
		 (length articles)
		 (car articles)
		 (car (last articles))
		 group))
	(insert (mapconcat
		 'number-to-string
		 articles
		 "\n"))
	(insert "\n.\n")
	t)
      nil)))

The complicated part is to use this function in the general Gnus back
end. It's pretty scary down there.

-David

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 15:00:06 +0200 David Engster <deng@randomsample.de> wrote: 

DE> This was last discussed here:

DE> http://article.gmane.org/gmane.emacs.gnus.general/67238
...
DE> The complicated part is to use this function in the general Gnus back
DE> end. It's pretty scary down there.

Yeah, that's why I brought it up.  If Lars can't figure it out I may
have to look at that code, and then dark gods will drag me down to the
bottom of the ocean (as seen in Disney's "Little Mermaid 7: Call of Cthulhu").

Ted

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> DE> http://article.gmane.org/gmane.emacs.gnus.general/67238
> ...
> DE> The complicated part is to use this function in the general Gnus back
> DE> end. It's pretty scary down there.
>
> Yeah, that's why I brought it up.  If Lars can't figure it out I may
> have to look at that code, and then dark gods will drag me down to the
> bottom of the ocean (as seen in Disney's "Little Mermaid 7: Call of Cthulhu").

Heh heh.

I think `*-request-group-articles' would work for all backends where it
makes sense, and imap is probably one of them.  I think.  Or does it?

I mean, the main problem is that Gnus thinks that there are, say 55
unread articles, and the article range is 200-300.  Even if you have the
-group-articles command return something that says "the range is
200-300, but there are only 75 actual articles in that range", then it
doesn't really help.  You don't know which of the 55 articles Gnus
thinks are unread are still on the server.

So the server would have to return

((200 . 210) 215 (245 . 276))

or something, and then Gnus could figure it all out.

So that would be a new `*-request-group-ranges' function, which is
certainly possible.  If the backend supports it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[David Engster]

Lars Magne Ingebrigtsen writes:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> DE> http://article.gmane.org/gmane.emacs.gnus.general/67238
>> ...
>> DE> The complicated part is to use this function in the general Gnus back
>> DE> end. It's pretty scary down there.
>>
>> Yeah, that's why I brought it up.  If Lars can't figure it out I may
>> have to look at that code, and then dark gods will drag me down to the
>> bottom of the ocean (as seen in Disney's "Little Mermaid 7: Call of Cthulhu").
>
> Heh heh.
>
> I think `*-request-group-articles' would work for all backends where it
> makes sense, and imap is probably one of them.  I think.  Or does it?
>
> I mean, the main problem is that Gnus thinks that there are, say 55
> unread articles, and the article range is 200-300.  Even if you have the
> -group-articles command return something that says "the range is
> 200-300, but there are only 75 actual articles in that range", then it
> doesn't really help.  You don't know which of the 55 articles Gnus
> thinks are unread are still on the server.
>
> So the server would have to return
>
> ((200 . 210) 215 (245 . 276))
>
> or something, and then Gnus could figure it all out.

Yes, exactly. The nnimap-request-group-articles I posted simply returns
the list of available article numbers in that group (which could be
further compressed into the above range notation). Requesting the range
of articles can take a second for very large groups, so some kind of
caching should be done.

Anyway, this would be an optional back end function, and Gnus can query
the back end for availability and use it (if the user wants it). I for
example don't have much problems with the article count, but I don't
delete much and use Dovecot. But there are IMAP servers out there
(Zimbra...) which use absolutely insane values for article numbers;
maybe it's using those for Mersenne prime searching or something.

-David

request-group-articles for nnimap (was: Faster, lolcat. Faster!)

[Ted Zlatanov]

On Thu, 02 Sep 2010 16:39:11 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Yes, exactly. The nnimap-request-group-articles I posted simply returns
DE> the list of available article numbers in that group (which could be
DE> further compressed into the above range notation). Requesting the range
DE> of articles can take a second for very large groups, so some kind of
DE> caching should be done.

Let the backend decide if caching is possible since it has the best
knowledge of when the data is old.  It also knows best where to cache
(on the server, in memory, on disk, etc.)  The caching policy can be set
globally, but the rest should be on the backend IMO.  Otherwise this
sounds great.

DE> Anyway, this would be an optional back end function, and Gnus can query
DE> the back end for availability and use it (if the user wants it). I for
DE> example don't have much problems with the article count, but I don't
DE> delete much and use Dovecot. But there are IMAP servers out there
DE> (Zimbra...) which use absolutely insane values for article numbers;
DE> maybe it's using those for Mersenne prime searching or something.

So there's two pieces: the general facility and support for it in
nnimap.  Let's get the nnimap piece done first and verify it works since
we're close.  Can you modify your function to do compressed ranges?  Do
you know of any alternate implementations or patches (I haven't kept track)?

Thanks
Ted

Re: request-group-articles for nnimap

[David Engster]

Ted Zlatanov writes:
> On Thu, 02 Sep 2010 16:39:11 +0200 David Engster <deng@randomsample.de> wrote: 
>
> DE> Yes, exactly. The nnimap-request-group-articles I posted simply returns
> DE> the list of available article numbers in that group (which could be
> DE> further compressed into the above range notation). Requesting the range
> DE> of articles can take a second for very large groups, so some kind of
> DE> caching should be done.
>
> Let the backend decide if caching is possible since it has the best
> knowledge of when the data is old.  It also knows best where to cache
> (on the server, in memory, on disk, etc.)  The caching policy can be set
> globally, but the rest should be on the backend IMO. 

Yes, it's probably easier when the backend does it. But we can worry
about that later. :-)

> DE> Anyway, this would be an optional back end function, and Gnus can query
> DE> the back end for availability and use it (if the user wants it). I for
> DE> example don't have much problems with the article count, but I don't
> DE> delete much and use Dovecot. But there are IMAP servers out there
> DE> (Zimbra...) which use absolutely insane values for article numbers;
> DE> maybe it's using those for Mersenne prime searching or something.
>
> So there's two pieces: the general facility and support for it in
> nnimap.  Let's get the nnimap piece done first and verify it works since
> we're close.  Can you modify your function to do compressed ranges?

It's a simple matter of calling gnus-compress-sequence.

>  Do you know of any alternate implementations or patches (I haven't
> kept track)?

As I wrote, Vitaly already had patches, also for the general backend,
but he extended the active information, without the usage of a new API
function. I think his patches can still be seen here:

http://repo.or.cz/w/more-gnus.git

My last test of those patches still showed some problems (see
http://thread.gmane.org/gmane.emacs.gnus.general/67238/focus=67407), but
they already show which parts of Gnus are affected.

We should first decide if we should build upon Vitaly's work with the
extended active, or if we use 'nnchoke-request-group-articles'. I have
difficulties following Lars' programming spree; maybe he has already
committed something which has decided this. ;-)

-David

Re: request-group-articles for nnimap

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> We should first decide if we should build upon Vitaly's work with the
> extended active, or if we use 'nnchoke-request-group-articles'. I have
> difficulties following Lars' programming spree; maybe he has already
> committed something which has decided this. ;-)

Nope.  :-)

I think `nnchoke-request-group-articles' is the way to go, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: request-group-articles for nnimap

[Ted Zlatanov]

On Fri, 03 Sep 2010 20:10:41 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov writes:
>> So there's two pieces: the general facility and support for it in
>> nnimap.  Let's get the nnimap piece done first and verify it works since
>> we're close.  Can you modify your function to do compressed ranges?

DE> It's a simple matter of calling gnus-compress-sequence.

OK.  Can you add that and send the patch as you have it?  It would be
really helpful.

DE> We should first decide if we should build upon Vitaly's work with the
DE> extended active, or if we use 'nnchoke-request-group-articles'.

I agree with Lars we should do the latter.

Ted

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Another performance issue I've noticed is that N foreign groups on the
> same server (subscribed through the server buffer and without an
> explicitly configured server entry) seem to always be slower than N
> groups on an explicitly configured (primary or secondary) server.

Gnus should gather all groups from the different servers, whether
they're foreign or not, first, and then do the right thing on them.
That would make the foreign/secondary distinction almost disappear.

> In the "too slow to be done properly" category are the unread/unseen
> nnimap counts which as you know are usually inaccurate.  We've had
> patches over the years but I don't think anyone has actually solved the
> problem.

And I've never used imap.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 16:17:28 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Another performance issue I've noticed is that N foreign groups on the
>> same server (subscribed through the server buffer and without an
>> explicitly configured server entry) seem to always be slower than N
>> groups on an explicitly configured (primary or secondary) server.

LMI> Gnus should gather all groups from the different servers, whether
LMI> they're foreign or not, first, and then do the right thing on them.
LMI> That would make the foreign/secondary distinction almost disappear.

Does "should" refer to how things work (as in, "AFAIK") or how they
should work (as in, "so fix it, you lazy bastard")?

>> In the "too slow to be done properly" category are the unread/unseen
>> nnimap counts which as you know are usually inaccurate.  We've had
>> patches over the years but I don't think anyone has actually solved the
>> problem.

LMI> And I've never used imap.  :-)

You can get started easily.  Just write your own IMAP server with a NNTP
backend ;)

Ted

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Does "should" refer to how things work (as in, "AFAIK") or how they
> should work (as in, "so fix it, you lazy bastard")?

The latter.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Thu, Sep 02 2010, Lars Magne Ingebrigtsen wrote: 

> My plan is to do the following: 

Another thing I'd like to have, is standard Maildir support.
I mean, with a standard handling of flags, and not things stored 
aside from the message.

So that you can access your Maildir from Gnus properly, but also 
From any other mail clients.

This would be probably faster than the way nnmaildir handle things 
right now, I think. No more millions of files to create.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 13:13:06 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Thu, Sep 02 2010, Lars Magne Ingebrigtsen wrote: 
>> My plan is to do the following: 

JD> Another thing I'd like to have, is standard Maildir support.
JD> I mean, with a standard handling of flags, and not things stored aside
JD> from the message.

nnmaildir already does that.  The problem is that Gnus flags are a
superset of what Maildir variants can handle.

JD> So that you can access your Maildir from Gnus properly, but also From
JD> any other mail clients.

JD> This would be probably faster than the way nnmaildir handle things
JD> right now, I think. No more millions of files to create.

The nnmaildir slowness comes from its desire to be safe for access from
multiple machines to a shared spool.  If it kept a message database it
would be much faster.  The patches I mentioned were discussed on
emacs-devel move some of the message count tracking to a single file,
but it really needs to be a single file for flags and counts.

I asked Paul Jarc for comments on my proposal.  See
http://thread.gmane.org/gmane.emacs.devel/125311/focus=126444 for
details.  I think nnmaildir can be ridiculously fast for single-client
access if we allow it.

Ted

Re: Faster, lolcat. Faster!

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On Thu, Sep 02 2010, Ted Zlatanov wrote: 

> nnmaildir already does that.  The problem is that Gnus flags are 
> a superset of what Maildir variants can handle. 

This is not a good reason to store *all* flags in a non-standard 
way.
 
> The nnmaildir slowness comes from its desire to be safe for 
> access from multiple machines to a shared spool.  If it kept a 
> message database it would be much faster.  The patches I 
> mentioned were discussed on emacs-devel move some of the message 
> count tracking to a single file, but it really needs to be a 
> single file for flags and counts.   I asked Paul Jarc for 
> comments on my proposal.  See 
> http://thread.gmane.org/gmane.emacs.devel/125311/focus=126444 
> for details.  I think nnmaildir can be ridiculously fast for 
> single-client access if we allow it. 

That would be great, because I'm not sure that there's a point 
allowing multiple machine access from Gnus on Maildirs.
Allowing multiple machine access to Maildir is a thing IMAP is 
good at.  On the other hand, I'm forced to use IMAP on *localhost* 
to access my Maildirs from Gnus, because Gnus does not cope 
correctly with Maildir.

Man, this is all upside down!

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Thu, 02 Sep 2010 15:04:29 +0200, Julien wrote:

> That would be great, because I'm not sure that there's a point
> allowing multiple machine access from Gnus on Maildirs.

If you don't want multiple client access to the emails, why use maildir?


  Best regards,

    Adam

-- 
 "(Phew, it's hard to explain with my poor English.)"         Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Ted Zlatanov]

On Thu, 02 Sep 2010 15:04:29 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Thu, Sep 02 2010, Ted Zlatanov wrote: 
>> nnmaildir already does that.  The problem is that Gnus flags are a
>> superset of what Maildir variants can handle. 

JD> This is not a good reason to store *all* flags in a non-standard way.

Yes, of course.  I didn't think I implied it was.

>> The nnmaildir slowness comes from its desire to be safe for access
>> from multiple machines to a shared spool.  If it kept a message
>> database it would be much faster.  The patches I mentioned were
>> discussed on emacs-devel move some of the message count tracking to
>> a single file, but it really needs to be a single file for flags and
>> counts.   I asked Paul Jarc for comments on my proposal.  See
>> http://thread.gmane.org/gmane.emacs.devel/125311/focus=126444 for
>> details.  I think nnmaildir can be ridiculously fast for
>> single-client access if we allow it. 

JD> That would be great, because I'm not sure that there's a point
JD> allowing multiple machine access from Gnus on Maildirs.

For shared filesystems like NFS it's extremely useful.  I think that's a
small and shrinking proportion of Gnus users but we can't ignore them.

JD> Allowing multiple machine access to Maildir is a thing IMAP is good
JD> at.

s/IMAP is/IMAP servers are/ to be clear.

JD> On the other hand, I'm forced to use IMAP on *localhost* to access
JD> my Maildirs from Gnus, because Gnus does not cope correctly with
JD> Maildir.

Yeah.  Whereas nnmaildir's problem is that it's slow, nnimap stores
flags in a server-specific way (no sharing of the spool), requires a
server to be running, has some issues with message counts, usually
requires authentication, etc.  So we can't ask Gnus users to use it when
nnmaildir makes more sense.

Ted

Re: Faster, lolcat. Faster!

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> On Thu, Sep 02 2010, Ted Zlatanov wrote:
>> nnmaildir already does that.  The problem is that Gnus flags are a
>> superset of what Maildir variants can handle.

> This is not a good reason to store *all* flags in a non-standard way.

Dovecot adds custom flags to the filename
 http://wiki.dovecot.org/MailboxFormat/Maildir#Maildir_filename_extensions

But that may not be compatible with all maildir clients...?

Re: Faster, lolcat. Faster!

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Now that Gnus is bug-free (because I haven't hacked much on Gnus the
LMI> past years) and has HTML support (because I have this week), I think
LMI> it's time to have a long good look at why `g' is so slow.

One of the big issues I've seen is that it can be repetative.

When I used nnmh (I think) gnus would read the whole thing three times
at startup.  Once was for finding new groups.  I forget why the third.

A single read through all of the spool would be enough....

Similarly, with nnimap it gets the details on each known group, then
does an lsub to get the full list of subscribed groups (as the imap
server understands the term) and gets the details on each group
returned by the lsub.  So every already-known group is detailed twice.

Again, it would be better to start with the lsub and iterate through all
of the groups once, noting for each whether gnus already knows about
that group.

Iterating only once would cut the startup time in half.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> When I used nnmh (I think) gnus would read the whole thing three times
> at startup.  Once was for finding new groups.  I forget why the third.
>
> A single read through all of the spool would be enough....

Definitely.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Now that Gnus is bug-free (because I haven't hacked much on Gnus the
> past years) and has HTML support (because I have this week), I think
> it's time to have a long good look at why `g' is so slow.

One thing that needs optimization is nnvirtual with nntp groups: it
always re-activates component groups one by one, which does a GROUP
request to the server for each group... which is absurdly slow.
(Especially if you have lots of component groups, the server is
Gmane and you don't live in Norway.)

It could instead do a few batches of pipelined LIST ACTIVE commands,
like the nntp backend usually does.

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> It could instead do a few batches of pipelined LIST ACTIVE commands,
> like the nntp backend usually does.

Hm.  Does nnvirtual need to do that at all?  Can't it just rely on the
data that Gnus already has on the groups?  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ok, I've now pushed out the major, major changes to the way `g' works
(part 1).  It now gathers all groups by select method, sorts them, and
issues `*-request-list' and friends where possible.

But WARNING WARNING: The code is incredibly crufted, and has 25 years of
change history.  There are lots of subtleties that I just didn't get why
were there, so I basically ripped out a lot of code, and inserted new
code instead.

So things probably don't work very well right now.

It works for me, though.

Where a `g' would stat 5800 files before this change, it now stats 270
files.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It works for me, though.

It does, it does.  So give it a whirl, and mail me all the backtraces.
:-) 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 19:57:04 +0200, Lars wrote:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> It works for me, though.

> It does, it does.  So give it a whirl, and mail me all the backtraces.
> :-) 

First observation: One of my newsgroups (from my primary server) now
shows up with 52 unread articles. When I enter the group, all articles
are read. Exit, the count is 0. Fetch news with g, the count is up to 52
again.

          52: dk.edb.internet.software.apache 

What information do you need? I have toggled-debug-on-error, but no
errors.

That is the only odd thing I've noticed from 3.2 minutes of use.


  Best regards,

    Adam

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

> First observation: One of my newsgroups (from my primary server) now
> shows up with 52 unread articles. When I enter the group, all articles
> are read. Exit, the count is 0. Fetch news with g, the count is up to 52
> again.
>
>           52: dk.edb.internet.software.apache 

Hm...  could it be an interaction the with agent or the cache or
something?  If you `M-g' the group, what happens?  Is there anything
special about the group in any way?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 20:34:21 +0200, Lars wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> First observation: One of my newsgroups (from my primary server) now
>> shows up with 52 unread articles. When I enter the group, all articles
>> are read. Exit, the count is 0. Fetch news with g, the count is up to 52
>> again.
>> 
>> 52: dk.edb.internet.software.apache 

> Hm...  could it be an interaction the with agent or the cache or
> something?

I don't use the agent; my *Server* buffer looks like this:

     {nnml:archive} (opened)
     {nntp:news.sslug.dk} (opened)
     {nntp:news.gwene.org} (opened)
     {nntp:news.gmane.org} (opened)
     {nnml:} (opened)
     {nntp:news.koldfront.local} (opened)

I don't think I have any cached articles in the group - none are listed
in the *Summary* buffer.

Let me check in ~/News/ ...

  $ find ~/News | grep dk.edb.internet.software.apache
  /home/asjo/News/marks/news.koldfront.local/dk/edb/internet/software/apache
  /home/asjo/News/marks/news.koldfront.local/dk/edb/internet/software/apache/.marks
  $ cat /home/asjo/News/marks/news.koldfront.local/dk/edb/internet/software/apache/.marks
  ((read (1 . 463)) (reply 369 416 422 431 435 440 444))
  $ 

Hm, but this looks a little suspicious:

  $ grep apache ~/.newsrc
  dk.edb.internet.software.apache: 1-411
  dk.edb.internet.software.apache: 1-411
  dk.edb.internet.software.apache: 1-463
  $ 

...?

Hm. It is listed three times in ~/.newsrc.eld as well?! (Harder to show,
as it is one long line. Hm, ok, maybe like this:)

  $ perl -pe 's/\) \(/\)\n\(/g;' .newsrc.eld | grep software.apache
  ("dk.edb.internet.software.apache" 3 ((1 . 411))
  ("dk.edb.internet.software.apache" 3 ((1 . 411))
  ("dk.edb.internet.software.apache" 3 ((1 . 463))
  ("dk" "dk.edb.sikkerhed" "dk.edb.tekst" "dk.admin.opslag" "dk.admin.netikette" "dk.admin" "dk.edb.internet.software.apache" "dk.edb.programmering" "dk.edb.programmering.perl" "dk.edb.sysadmin" "dk.edb.system.unix" "dk.edb.system.unix.hardware" "dk.helbred")
  $ 

>  If you `M-g' the group, what happens?

The count goes down to 0 if I do that.

> Is there anything special about the group in any way?

Not that I know of.


  Best regards,

    Adam

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 20:45:35 +0200, Adam wrote:

>   $ perl -pe 's/\) \(/\)\n\(/g;' .newsrc.eld | grep software.apache

Or even:

  $ perl -pe 's/\) \(/\)\n\(/g;' .newsrc.eld | grep -A 2 software.apache | head -12
  ("dk.edb.internet.software.apache" 3 ((1 . 411))
  ((reply 369)
  (seen (350 . 411))))
  --
  ("dk.edb.internet.software.apache" 3 ((1 . 411))
  ((reply 369)
  (seen (350 . 411))))
  --
  ("dk.edb.internet.software.apache" 3 ((1 . 463))
  ((seen (335 . 345) 347 (349 . 463))
  (reply 369 416 422 431 435 440 444)))
  --
  $ 


  Best regards,

     Adam

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

>   $ perl -pe 's/\) \(/\)\n\(/g;' .newsrc.eld | grep -A 2 software.apache | head -12
>   ("dk.edb.internet.software.apache" 3 ((1 . 411))
>   ((reply 369)
>   (seen (350 . 411))))
>   --
>   ("dk.edb.internet.software.apache" 3 ((1 . 411))
>   ((reply 369)
>   (seen (350 . 411))))
>   --
>   ("dk.edb.internet.software.apache" 3 ((1 . 463))
>   ((seen (335 . 345) 347 (349 . 463))
>   (reply 369 416 422 431 435 440 444)))

Ok, then I would guess that your .newsrc.eld file is slightly corrupted,
and you should delete the extra instances.

But I'm not quite sure why they should show up using the first data in
your group buffer now, while they were using the last instances before.
It could be a subtle change in the order things are happening...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

I've now implemented streaming pop3 retrieval.  It's a bit scary playing
with the mail, since if you're doing something wrong, then, er, the mail
goes missing.

But it sure is faster.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But it sure is faster.

And it now compiles and stuff, so I've pushed it out.

Handle with care.  Remember to have

(setq mail-source-delete-incoming 10)

or the like, because the mail be need respooled if I did something that
made the email go away.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Ok, now the overwhelming time spent is reading and writing the .overview
files for nnml when getting new email.  I wonder why I do that.  I mean,
can't I just append the new NOV lines to the end?  Anybody remember?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ok, now the overwhelming time spent is reading and writing the .overview
> files for nnml when getting new email.  I wonder why I do that.  I mean,
> can't I just append the new NOV lines to the end?  Anybody remember?

I sure couldn't, so I've now rewritten this.  Whoosh.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Ok, now the overwhelming time spent is reading and writing the .overview
>> files for nnml when getting new email.  I wonder why I do that.  I mean,
>> can't I just append the new NOV lines to the end?  Anybody remember?
>
> I sure couldn't, so I've now rewritten this.  Whoosh.

So now there are scary sweeping mail changes with both nnml, pop3 and
mail-sources at the same time.  Eek.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sun, 05 Sep 2010 00:50:51 +0200, Lars wrote:

>> I sure couldn't, so I've now rewritten this.  Whoosh.

> So now there are scary sweeping mail changes with both nnml, pop3 and
> mail-sources at the same time.  Eek.

Finally it feels a little dangerous to run No Gnus again!


  :-),

   Adam

-- 
 "I'm not sure it never causes any harm to all users."        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

> Finally it feels a little dangerous to run No Gnus again!

:-)

I've tried it at work, too, with Emacs 22, and it seems to work OK.
Didn't lose any mail that I could see...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sun, 05 Sep 2010 02:19:35 +0200, Lars wrote:

> Didn't lose any mail that I could see...

Haha!


  :-D,

   Adam

-- 
 "I'm not sure it never causes any harm to all users."        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So now there are scary sweeping mail changes with both nnml, pop3
> and mail-sources at the same time. Eek.

Maybe we should have let those changes simmer a bit in Git before
synching them to the Emacs trunk... On the other hand, if there's a
mail-eating bug in there, we now have a much higher chance of
finding it. :-)

Re: Faster, lolcat. Faster!

[Steinar Bang]

>>>>> Romain Francoise <romain@orebokech.com>:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> So now there are scary sweeping mail changes with both nnml, pop3 and
>> mail-sources at the same time. Eek.

> Maybe we should have let those changes simmer a bit in Git before
> synching them to the Emacs trunk... On the other hand, if there's a
> mail-eating bug in there, we now have a much higher chance of finding
> it. :-)

Well... there's also a reported nnimap slowdown which worries me a lot.

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've now implemented streaming pop3 retrieval.

Hmm, I don't use pop3.el but from a cursory look, it doesn't check
if the server supports the pipelining capability (with CAPA). Isn't
that a problem?

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> Hmm, I don't use pop3.el but from a cursory look, it doesn't check
> if the server supports the pipelining capability (with CAPA). Isn't
> that a problem?

Are there any pop3 servers that check whether they've received more
commands before they process the current command?  I mean, to not
support streaming you have to (kind of) go to a lot of trouble by
peeking at the stream and stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Are there any pop3 servers that check whether they've received more
> commands before they process the current command?

A naive server could just drain the socket when it becomes ready and
assume that it will read only one command, since the protocol
doesn't allow pipelining natively...

But I don't known whether any real-world POP3 server does that.

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> A naive server could just drain the socket when it becomes ready and
> assume that it will read only one command, since the protocol
> doesn't allow pipelining natively...

If I were writing a pop3 server I'd have a loop like:

while (1) {
      line = read_line();
      execute_command(line);
}

Which would just work.

> But I don't known whether any real-world POP3 server does that.

I guess we'll find out.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 20:45:35 +0200, Adam wrote:

>   $ grep apache ~/.newsrc
>   dk.edb.internet.software.apache: 1-411
>   dk.edb.internet.software.apache: 1-411
>   dk.edb.internet.software.apache: 1-463
>   $ 

It might be an "old wound"; I think I'll just clean up .newsrc* and see
what happens.


  Best regards,

    Adam

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 20:50:15 +0200, Adam wrote:

> It might be an "old wound"; I think I'll just clean up .newsrc* and see
> what happens.

Did that; it went away.

So far, so good.


  Best regards,

    Adam

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Where a `g' would stat 5800 files before this change, it now stats
> 270 files.

It does 1417 calls here, compared to 3961 before (I use nnml).
Hard to tell the difference with a hot cache, though...

Everything seems to work, but like Adam, one of my groups now has a
totally wrong unread count, and when I try to enter it there's no
unread articles... there seems to be two versions of it in
gnus-newsrc-alist, one of which has totally wonky values. Not sure
how that can happen.

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

>> Where a `g' would stat 5800 files before this change, it now stats
>> 270 files.
>
> It does 1417 calls here, compared to 3961 before (I use nnml).
> Hard to tell the difference with a hot cache, though...

Hm.  nnml shouldn't really stat much at all except its active file.
What does strace is being statted?  

> Everything seems to work, but like Adam, one of my groups now has a
> totally wrong unread count, and when I try to enter it there's no
> unread articles... there seems to be two versions of it in
> gnus-newsrc-alist, one of which has totally wonky values. Not sure
> how that can happen.

I seem to vaguely recall that back in ... er...  sometime...  there was
a bug in the topic/group code when killing/yanking groups that could
lead to doubling of entries.

I'll try to reproduce by editing the .newsrc.eld file to get duplicates
and try to get the wrong behaviour. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'll try to reproduce by editing the .newsrc.eld file to get duplicates
> and try to get the wrong behaviour. 

Yes, I'm able to reproduce it perfectly.

But I think the right solution might just be to write a function that
goes through the newsrc alist and remove duplicates.  Or rather, check
for duplicates while building the newsrc hash table, and then remove
them there.  I'll do that.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But I think the right solution might just be to write a function that
> goes through the newsrc alist and remove duplicates.  Or rather, check
> for duplicates while building the newsrc hash table, and then remove
> them there.  I'll do that.

I've now done this.  It's just one obarray lookup per group extra, so it
should be no major deal.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[James Cloos]

[Replying to a random article in the thread]

With imap startup now takes about twice as long as it used to.

And emacs must be sending the imap commands slower, too, since there is
more idle time.

The backend psql process used to max out a core (as seen in top(1); it
is really waiting on ram for most of that time) and take about 4/9
seconds per group between the initial run thorugh, the lsub, and the
second run through.

Now it only hits about 85% in top(1), and it still ran through the
groups twice.  (The mean was 5.5 imap commands per group.  The startup
runs an EXAMIME and a STATUS on each group and, if there are messages,
a FETCH to get the list of message UIDs.  If it went through twice,
that suggests that about half of the groups were empty.  That sounds
reasonable.)

One speedup would be to do the EXAMINE/STATUS/FETCH all on one packet,
instead of one packet per command.

But it still needs to learn to do the lsub first, when collecting the
list of groups, and then do a single EXAMINE/STATUS/FETCH tuple for
each group.  And then subscribe the new ones if that option is set.

I've looked at the code in the past, but although some parts of gnus are
easy to follow, some are very much not.  I never saw the proper solution,
code-wise, when I looked at it....

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> With imap startup now takes about twice as long as it used to.
>
> And emacs must be sending the imap commands slower, too, since there is
> more idle time.

I haven't touched any of the nnimap code, but I may be using other
commands than before.  I now default to `nnimap-retrieve-groups' instead
of `nnimap-request-list'.  Could you try edebugging
`gnus-read-active-for-groups' in gnus-start.el and see what it's doing,
and whether calling the other function is faster?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[James Cloos]

Ran into another bug.

I'm still at commit 0df157421bb63b5b6978a36912c39c91bf550b82.

g no longer honours its arg to limit to just groups of that level
or lower.

I have my more important groups at lower level in part so that I can
skip the less important groups when running (gnus-group-get-new-news).

Now I have to List all groups, determine the cutoff point, and pass
that value as an arg to M-g.

M-g over imap groups is much slower than g, since it does:

,----
| 31172 SELECT "ctan-ann@dante.de"
| 31173 UID SEARCH UNSEEN UNDELETED
| 31174 UID SEARCH SEEN
| 31175 UID SEARCH FLAGGED
| 31176 UID SEARCH ANSWERED
| 31177 UID SEARCH RECENT
| 31178 FETCH 1,* UID
| 31179 NOOP
| 31180 STATUS "ctan-ann@dante.de" (UNSEEN)
| 31181 EXPUNGE
| 31182 NOOP
`----

with each line in a separate write(2) call.  Doing more is not a bug,
but the effective inability to use (gnus-group-get-new-news) at all is.

(I use a tty between emacs and my imapd, so one command per write(2) is
not as much of a problem as it is over tcp(7).  Using a tcp socket --
even over the loopback interfact -- has less than half the throughput as
using a tty.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> g no longer honours its arg to limit to just groups of that level
> or lower.
>
> I have my more important groups at lower level in part so that I can
> skip the less important groups when running (gnus-group-get-new-news).
>
> Now I have to List all groups, determine the cutoff point, and pass
> that value as an arg to M-g.

I'm not sure I understand what happens.  You have groups on high levels
that you don't want activated when you do a `g'?  But now `g' activates
them anyway, and this takes a long time?

And these are nnimap groups?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> I'm not sure I understand what happens.  You have groups on high levels
LMI> that you don't want activated when you do a `g'?  But now `g' activates
LMI> them anyway, and this takes a long time?

I didn't let it continue to see whether it activates them, but it i try
to retrieve the info for them, unlike before.

,----< help for gnus-group-get-new-news
| Get newly arrived articles.
| If ARG is a number, it specifies which levels you are interested in
| re-scanning.  If ARG is non-nil and not a number, this will force
| "hard" re-reading of the active files from all servers.
`----

I use a numeric ARG to gnus-group-get-new-news all the time to do just
as the docs say: only re-scan that level and lower.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> ,----< help for gnus-group-get-new-news
> | Get newly arrived articles.
> | If ARG is a number, it specifies which levels you are interested in
> | re-scanning.  If ARG is non-nil and not a number, this will force
> | "hard" re-reading of the active files from all servers.
> `----
>
> I use a numeric ARG to gnus-group-get-new-news all the time to do just
> as the docs say: only re-scan that level and lower.

But that's `g', not `M-g':

> Now I have to List all groups, determine the cutoff point, and pass
> that value as an arg to M-g.

Which is what I didn't understand.

But I think the behavior you describe has been fixed in git earlier
today.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> g no longer honours its arg to limit to just groups of that level
> or lower.

I think I understand what's happening, and it's only relevant for the
backends that have -retrieve-groups interface, and that are slow.  I.e.,
nnimap.  :-)

I've now pushed a fix out that will probably fix this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've now pushed a fix out that will probably fix this.

I mean.  Now, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Steinar Bang]

>>>>> James Cloos <cloos@jhcloos.com>:

> [Replying to a random article in the thread]
> With imap startup now takes about twice as long as it used to.

Hm... I'll stay on the fence a bit, then.

I'm dependent on the nnimap server, and don't feel capable of fixing
something like that.

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> But I think the right solution might just be to write a function that
>> goes through the newsrc alist and remove duplicates.  Or rather, check
>> for duplicates while building the newsrc hash table, and then remove
>> them there.  I'll do that.

> I've now done this.  It's just one obarray lookup per group extra, so it
> should be no major deal.

Works for me. It nuked the good entry and kept the broken one, but I
guess there's no way around that...

While we're on the subject of newsrc entries: I've had the following
in my configuration for ages, and (if I remember correctly), it
saves quite a bit of startup/save time:

(setq gnus-save-newsrc-file nil
      gnus-read-newsrc-file nil)

Maybe we should flip the defaults?

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> (setq gnus-save-newsrc-file nil
>       gnus-read-newsrc-file nil)
>
> Maybe we should flip the defaults?

Hm...  that's a bit scary.  The .newsrc file won't be updated, but then
if something does update it, and you're using an older Gnus, then that
will be used instead of your .newsrc.eld, so it'll kinda nuke your new
info.

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Adam Sjøgren]

On Sat, 04 Sep 2010 21:57:49 +0200, Lars wrote:

>> It does 1417 calls here, compared to 3961 before (I use nnml).
>> Hard to tell the difference with a hot cache, though...

> Hm.  nnml shouldn't really stat much at all except its active file.
> What does strace is being statted?  

In my setup (nnml, no agent + nntp), I get 24 stat()s when going g:

      6 /home/asjo/News/drafts/delayed
      3 /home/asjo/Mail/archive
      3 /home/asjo/.emacs-mail-crash-box
      2 /home/asjo/Mail/
      1 /var/mail/asjo
      1 /home/asjo/News/drafts/queue
      1 /home/asjo/News/drafts/drafts
      1 /home/asjo/News/drafts/delayed/
      1 /home/asjo/News/drafts/
      1 /home/asjo/News/drafts
      1 /home/asjo/Mail/archive/active
      1 /home/asjo/Mail/active
      1 /home/asjo/Mail
      1 /home/asjo/


   Best regards,

     Adsm

-- 
 "The laws of perspective have been repealed!                 Adam Sjøgren
  Objects no longer diminish in size with distance!"     asjo@koldfront.dk

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Hm.  nnml shouldn't really stat much at all except its active file.
> What does strace is being statted?

My nnfolder mail archives, and it's because of nnvirtual again. I
have one archive group per month, and collect them into one virtual
archive per year and a global one. Activating each group does a few
stat calls.

Moving the virtual groups to a higher level to get them out of the
way brings the stat count down to 302.

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> My nnfolder mail archives, and it's because of nnvirtual again. I
> have one archive group per month, and collect them into one virtual
> archive per year and a global one. Activating each group does a few
> stat calls.

Yeah.  This patch makes the problem go away, but is it the right thing
to do?

diff --git a/lisp/nnvirtual.el b/lisp/nnvirtual.el
index 94f4321..2032335 100644
--- a/lisp/nnvirtual.el
+++ b/lisp/nnvirtual.el
@@ -689,7 +689,8 @@ based on the marks on the component groups."
     ;; Into all-marks we put (g marks).
     ;; We also increment cnt and tot here, and compute M (max of sizes).
     (mapc (lambda (g)
-	    (setq active (gnus-activate-group g)
+	    (setq active (or (gnus-active g)
+			     (gnus-activate-group g))
 		  min (car active)
 		  max (cdr active))
 	    (when (and active (>= max min) (not (zerop max)))

If you `M-g' a virtual group, you probably want to trigger a rescan from
all the component groups, too, I think?

So I think I have to add some thing to make the `g' thing when doing
virtual groups do the thing in the patch, which the `M-g' behaviour is
kept as is.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Romain Francoise]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> If you `M-g' a virtual group, you probably want to trigger a rescan from
> all the component groups, too, I think?

Probably, yes.

> So I think I have to add some thing to make the `g' thing when doing
> virtual groups do the thing in the patch, which the `M-g' behaviour is
> kept as is.

Yes, but it doesn't feel right to make `g' and `M-g' do something
different. Do we already have backends which do something like that?

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Romain Francoise <romain@orebokech.com> writes:

> Yes, but it doesn't feel right to make `g' and `M-g' do something
> different. Do we already have backends which do something like that?

If the nnvirtual groups are always activated last in a `g' run, then the
end result would be identical to `M-g' -- if the component groups to the
nnvirtual group are on a level that means they are activated...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Faster, lolcat. Faster!

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Where a `g' would stat 5800 files before this change, it now stats 270
> files.

Which files did it stat before and no longer does?  Or: "what kind of files"?

Re: Faster, lolcat. Faster!

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Which files did it stat before and no longer does?  Or: "what kind of
> files"?

nnmh was quite stat-happy, but it was mainly just, well, everything.
Now it issues `-request-list' instead of inspecting each foreign group
individually.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen