Re: Slow Access from Gnus

Re: Slow Access from Gnus

[Reiner Steib]

On Fri, Apr 04 2008, Jake Colman wrote:

>>>>>> "SB" == Steinar Bang <sb@dod.no> writes:
>    SB> Have you tried sniffing the group opening with eg. wireshark to
>    SB> see if the delay is simply network related?
>
> When I enter a traditional nntp-based newsgroup (e.g., comp.emacs.help)
> I see some quick 'nntp read' messages and then I'm in.  Depending on the
> number of messages that have been posted since the last time I entered
> the newsgroup, I will see more or less 'nntp read' messages.
>
> But when I entered gmane.discuss just now, with only two messages posted
> since I last checked it yesterday, I still saw a huge amount of 'nntp
> read' traffic.  It's almost as if Gnus is downloading the entire group
> and doesn't know how much I've already seen.

Set nntp-record-commands to t and check if the commands in the
*nntp-log* buffer are significantly different e.g. for gnu.emacs.help
and gmane.emacs.help.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow Access from Gnus

[Jake Colman]

>>>>> "RS" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

   RS> On Fri, Apr 04 2008, Jake Colman wrote:
   >>>>>>> "SB" == Steinar Bang <sb@dod.no> writes:
   SB> Have you tried sniffing the group opening with eg. wireshark to
   SB> see if the delay is simply network related?
   >> 
   >> When I enter a traditional nntp-based newsgroup (e.g., comp.emacs.help)
   >> I see some quick 'nntp read' messages and then I'm in.  Depending on the
   >> number of messages that have been posted since the last time I entered
   >> the newsgroup, I will see more or less 'nntp read' messages.
   >> 
   >> But when I entered gmane.discuss just now, with only two messages posted
   >> since I last checked it yesterday, I still saw a huge amount of 'nntp
   >> read' traffic.  It's almost as if Gnus is downloading the entire group
   >> and doesn't know how much I've already seen.

   RS> Set nntp-record-commands to t and check if the commands in the
   RS> *nntp-log* buffer are significantly different e.g. for
   RS> gnu.emacs.help and gmane.emacs.help.

Very, very odd.  The commands are identical.

For example:

20080404T131502.278 news.gmane.org GROUP gmane.emacs.gnus.general
20080404T131502.400 news.gmane.org XOVER 1-66679
20080404T131611.177 free.teranews.com GROUP comp.emacs
20080404T131611.296 free.teranews.com XOVER 1-69787

Yes, it appears that for every gmane group it actually rereads the
entire "active file" (do I know what I'm talkig about?) or something.
Even if I do a catch-up in my gmane group it rereads everything.  If I
enter comp.emacs twice in a row, the second is almost instantaneous.
Entering the same gmane group twice will take the same amount of time
each time as it does all those kilobytes of nntp reads.

-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: Slow Access from Gnus

[Reiner Steib]

On Fri, Apr 04 2008, Jake Colman wrote:

> 20080404T131502.278 news.gmane.org GROUP gmane.emacs.gnus.general
> 20080404T131502.400 news.gmane.org XOVER 1-66679
> 20080404T131611.177 free.teranews.com GROUP comp.emacs
> 20080404T131611.296 free.teranews.com XOVER 1-69787
>
> Yes, it appears that for every gmane group it actually rereads the
> entire "active file" (do I know what I'm talkig about?) or something.

Gnus fetches the whole NOV (overview) data if `gnus-fetch-old-headers'
is t.  Try setting it to some number, say 100.  Does it help?

Then you should see something like this in the *nntp-log* buffer:

[DATE...] news.gmane.org XOVER 66578-66679

(See the function `nntp-retrieve-headers-with-xover')

In contrast to "normal" news servers, Gmane doesn't expire articles so
the overview might become really large.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow Access from Gnus

[Jake Colman]

>>>>> "RS" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

   RS> On Fri, Apr 04 2008, Jake Colman wrote:
   >> 20080404T131502.278 news.gmane.org GROUP gmane.emacs.gnus.general
   >> 20080404T131502.400 news.gmane.org XOVER 1-66679
   >> 20080404T131611.177 free.teranews.com GROUP comp.emacs
   >> 20080404T131611.296 free.teranews.com XOVER 1-69787
   >> 
   >> Yes, it appears that for every gmane group it actually rereads the
   >> entire "active file" (do I know what I'm talkig about?) or something.

   RS> Gnus fetches the whole NOV (overview) data if
   RS> `gnus-fetch-old-headers' is t.  Try setting it to some number,
   RS> say 100.  Does it help?

   RS> Then you should see something like this in the *nntp-log* buffer:

   RS> [DATE...] news.gmane.org XOVER 66578-66679

   RS> (See the function `nntp-retrieve-headers-with-xover')

   RS> In contrast to "normal" news servers, Gmane doesn't expire
   RS> articles so the overview might become really large.


WOW!  What a difference!  My prior setting for 'gnus-fetch-old-headers'
was t.  I set it to 500 and I now enter the group virtually
instantaneously.  Is there any practical downside to specifying a number
instead of 't'?  What setting do _you_ use?


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: Slow Access from Gnus

[Reiner Steib]

On Fri, Apr 04 2008, Jake Colman wrote:

> WOW!  What a difference!  My prior setting for
> 'gnus-fetch-old-headers' was t.  I set it to 500 and I now enter the
> group virtually instantaneously.  Is there any practical downside to
> specifying a number instead of 't'?

You might not see old posting in the threads...

,----[ (info "(gnus)Filling In Threads") ]
| `gnus-fetch-old-headers'
|      If non-`nil', Gnus will attempt to build old threads by fetching
|      more old headers--headers to articles marked as read.  If you would
|      like to display as few summary lines as possible, but still
|      connect as many loose threads as possible, you should set this
|      variable to `some' or a number.  If you set it to a number, no
|      more than that number of extra old headers will be fetched.  In
|      either case, fetching old headers only works if the back end you
|      are using carries overview files--this would normally be `nntp',
|      `nnspool', `nnml', and `nnmaildir'.  Also remember that if the
|      root of the thread has been expired by the server, there's not
|      much Gnus can do about that.
| 
|      This variable can also be set to `invisible'.  This won't have any
|      visible effects, but is useful if you use the `A T' command a lot
|      (*note Finding the Parent::).
`----

> What setting do _you_ use?

I've set it to t, but I fetch all newsgroups groups from my local
leafnode server, which does expire Gmane groups, so I don't experience
the slowness.  Maybe we should warn about this setting when using
news.gmane.org.  Do others see the slow access on Gmane with as well
with `gnus-fetch-old-headers' set to t?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow Access from Gnus

[Sven Joachim]

On 2008-04-04 21:54 +0200, Reiner Steib wrote:

> On Fri, Apr 04 2008, Jake Colman wrote:
>
>> WOW!  What a difference!  My prior setting for
>> 'gnus-fetch-old-headers' was t.  I set it to 500 and I now enter the
>> group virtually instantaneously.  Is there any practical downside to
>> specifying a number instead of 't'?
>
> You might not see old posting in the threads...
>
> ,----[ (info "(gnus)Filling In Threads") ]
> | `gnus-fetch-old-headers'
> |      If non-`nil', Gnus will attempt to build old threads by fetching
> |      more old headers--headers to articles marked as read.  If you would
> |      like to display as few summary lines as possible, but still
> |      connect as many loose threads as possible, you should set this
> |      variable to `some' or a number.  If you set it to a number, no
> |      more than that number of extra old headers will be fetched.  In
> |      either case, fetching old headers only works if the back end you
> |      are using carries overview files--this would normally be `nntp',
> |      `nnspool', `nnml', and `nnmaildir'.  Also remember that if the
> |      root of the thread has been expired by the server, there's not
> |      much Gnus can do about that.
> | 
> |      This variable can also be set to `invisible'.  This won't have any
> |      visible effects, but is useful if you use the `A T' command a lot
> |      (*note Finding the Parent::).
> `----
>
>> What setting do _you_ use?
>
> I've set it to t, but I fetch all newsgroups groups from my local
> leafnode server, which does expire Gmane groups, so I don't experience
> the slowness.  Maybe we should warn about this setting when using
> news.gmane.org.

Seems like a good idea, see below.

>  Do others see the slow access on Gmane with as well
> with `gnus-fetch-old-headers' set to t?

I just tried with gmane.linux.debian.devel.bugs.general (HUGE!), where I
have two dozen ticked articles.  After ~1 minute Emacs still was
displaying something like "nntp read: 92167k" (yes, ca. 90 Megabyte!),
then I got tired and typed C-g.  Fortunately, memory footprint dropped
to a reasonable level afterwards.

Sven

Re: Slow Access from Gnus

[Jake Colman]

>>>>> "RS" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

   RS> On Fri, Apr 04 2008, Jake Colman wrote:

   >> WOW!  What a difference!  My prior setting for
   >> 'gnus-fetch-old-headers' was t.  I set it to 500 and I now enter
   >> the group virtually instantaneously.  Is there any practical
   >> downside to specifying a number instead of 't'?

   RS> You might not see old posting in the threads...


Is there a way to set the gnus-fetch-old-headers variable one way for
gmane groups and another way for standard nntp groups. (those which
expire their articles)?  Maybe some sort of pre- and post-hook?


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: Slow Access from Gnus

[Reiner Steib]

[ Stripping gmane.dicuss ]

On Sun, Apr 06 2008, Jake Colman wrote:

> Is there a way to set the gnus-fetch-old-headers variable one way for
> gmane groups and another way for standard nntp groups.

Probably like this:

(add-to-list 'gnus-parameters '("^gmane\\." (gnus-fetch-old-headers 500)))

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow Access from Gnus

[Reiner Steib]

On Fri, Apr 04 2008, Sven Joachim wrote:

> On 2008-04-04 21:54 +0200, Reiner Steib wrote:
>> On Fri, Apr 04 2008, Jake Colman wrote:
>>> WOW!  What a difference!  My prior setting for
>>> 'gnus-fetch-old-headers' was t.  I set it to 500 and I now enter the
>>> group virtually instantaneously.  
[...]
>> Maybe we should warn about this setting when using news.gmane.org.
>
> Seems like a good idea, see below.

I added a warning in the documentation.  Maybe we could also warn when
entering the group.

>>  Do others see the slow access on Gmane with as well
>> with `gnus-fetch-old-headers' set to t?
>
> I just tried with gmane.linux.debian.devel.bugs.general (HUGE!), where I
> have two dozen ticked articles.  After ~1 minute Emacs still was
> displaying something like "nntp read: 92167k" (yes, ca. 90 Megabyte!),
> then I got tired and typed C-g.  Fortunately, memory footprint dropped
> to a reasonable level afterwards.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/