Gnus slow?

Gnus slow?

[Katsumi Yamaoka]

Hi,

First of all, I don't have any difficulty in using Gnus.  Today
I'm using both No Gnus v0.2 and latest Gnus CVS for a while, but
I was not able to find out a remarkable difference of speed
between them so far.  However, a friend of mine wrote to me that
the recent Gnus is much slower than No Gnus v0.2 for fetching
mails.  IIRC, I also have heard similar utterances by several
people in groups and mailing lists.

Exceptionally, since Gnus became slowdown obviously when the
nntp marks feature was introduced, I set `nntp-marks-is-evil' as
t and removed all nntp marks files.  I haven't changed the value
of `nntp-marks-is-evil' since then.

Didn't anyone experience the slowness of Gnus like them?  Was it
already solved?  Any informations are appreciated and will help
them.

Best regards,

Re: Gnus slow?

[Romain Francoise]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Exceptionally, since Gnus became slowdown obviously when the
> nntp marks feature was introduced, I set `nntp-marks-is-evil' as
> t and removed all nntp marks files.  I haven't changed the value
> of `nntp-marks-is-evil' since then.

In my case it is very slow to fetch new articles even if I disable NNTP
marks via `nntp-marks-is-evil'.  I have reverted to No Gnus from May
12th and it is much faster (by several orders of magnitude) to fetch
articles.

-- 
Romain Francoise <romain@orebokech.com> | I like the streets when
it's a miracle -- http://orebokech.com/ | they're empty, I can make the
                                        | rest up.

Re: Gnus slow?

[Gabor Z.Papp]

Romain Francoise <romain <at> orebokech.com> writes:

> In my case it is very slow to fetch new articles even if I disable NNTP
> marks via `nntp-marks-is-evil'.  I have reverted to No Gnus from May
> 12th and it is much faster (by several orders of magnitude) to fetch
> articles.

I have noticed the same slowdown effect with cvs co on 20041109,
and I'm using Gnus only with nnmaildir and imap, not nntp at all.
Scanning through the nnmaildir is much slower than with earlier
cvs snapshots (like 200409xx) and forwarding mails also often
lock up gnus (feeding the spam filter) for several seconds.

But I'm not a developer, so dunno about details. :-)

Re: Gnus slow?

[Jesper Harder]

Gabor Z.Papp <gzp@papp.hu> writes:

> Romain Francoise <romain <at> orebokech.com> writes:
>
>> In my case it is very slow to fetch new articles even if I disable
>> NNTP marks via `nntp-marks-is-evil'.  I have reverted to No Gnus
>> from May 12th and it is much faster (by several orders of
>> magnitude) to fetch articles.
>
> I have noticed the same slowdown effect with cvs co on 20041109, and
> I'm using Gnus only with nnmaildir and imap, not nntp at all.
> Scanning through the nnmaildir is much slower than with earlier cvs
> snapshots (like 200409xx)

Try to play around with `M-x elp-instrument-package gnus'[1], and
compare the results from `M-x elp-results' between the different Gnus
version.  Posts the results if you can reproduce some significant
differences.

[1] Or maybe 'nnml', 'nntp' etc depending on which operations are
slow.

-- 
Jesper Harder                                <http://purl.org/harder/>

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <m3pt2lim2r.fsf@defun.localdomain> Jesper Harder wrote:

> Try to play around with `M-x elp-instrument-package gnus'[1], and
> compare the results from `M-x elp-results' between the different Gnus
> version.  Posts the results if you can reproduce some significant
> differences.

> [1] Or maybe 'nnml', 'nntp' etc depending on which operations are
> slow.

That's a good idea.  Though it is accompanied by pains, you will
surely be able to find a difference.

Re: Gnus slow?

[Adam Sjøgren]

On Wed, 10 Nov 2004 21:47:11 +0900, Katsumi wrote:

> Hi, First of all, I don't have any difficulty in using Gnus. Today
> I'm using both No Gnus v0.2 and latest Gnus CVS for a while, but I
> was not able to find out a remarkable difference of speed between
> them so far. However, a friend of mine wrote to me that the recent
> Gnus is much slower than No Gnus v0.2 for fetching mails. IIRC, I
> also have heard similar utterances by several people in groups and
> mailing lists.

I don't know when this started, it may not be related at all, but I'll
mention it just in case:

Recently I got annoyed by hearing my harddisk make a lot of noise
every time I pressed 'g' in Gnus - when checking for new news on the
three news-servers I use - so I tried strace'ing XEmacs/Gnus.

(I was expecting it to be rather silent when checking news-servers
over the net...)

Gnus was trying to opening a lot of files every time I pressed 'g' -
from the names I gathered that it could be something to do with the
Agent (.agentview?), so I went to the server-buffer and unagentized
everything. (I don't use any functionality of the Agent, that I know
of, but back when it became on by default I didn't turn it off).

That change has made the harddisk-noise while checking for new news go
away. Now I only hear noise when mail is being split and stored, and
after that none of the previous 'hitting the harddisk' occurs.

I don't know if this is significant, to be expected, or something else
entirely, but there you go.


This observation is based on a the cvs-gnus I'm posting this article
with, last updated, uh, this summer; No Gnus v0.3, XEmacs 21.4 (patch
15).

I should probably update.


  Best regards,

    Adam

-- 
 "More than anything, I won't try"                            Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Gnus slow?

[Romain Francoise]

Here are the ELP results.  In all cases I started a new Emacs instance,
started Gnus, started profiling, hit `g', then stopped profiling at the
"Checking new news...done" message.  The two Gnus versions tested are a
No Gnus checkout as of May 12th (which I will name version A) and the
latest CVS from this morning (version B).  Four tests are performed: for
the `gnus' and the `nntp' package, for versions A and B.  I have one
nnml server (but no new articles for it), and three nntp servers:
news.free.fr, news.gmane.org and news.gnus.org.

Package gnus, version A (first 10 lines):

Function Name                     Call Count  Elapsed Time  Average Time
================================  ==========  ============  ============
gnus-group-get-new-news           1           1.465511      1.465511
gnus-get-unread-articles          1           1.277827      1.277827
gnus-activate-group               405         0.6969249999  0.0017208024
gnus-request-scan                 200         0.5125570000  0.0025627850
gnus-topic-prepare-topic          44          0.509935      0.0115894318
gnus-read-active-file-2           2           0.469293      0.2346465
gnus-retrieve-groups              2           0.464826      0.232413
gnus-group-list-groups            1           0.1848060000  0.1848060000

Package gnus, version B (first 10 lines):

Function Name                     Call Count  Elapsed Time  Average Time
================================  ==========  ============  ============
gnus-group-get-new-news           1           29.122044     29.122044
gnus-get-unread-articles          1           28.808084     28.808084
gnus-activate-group               808         28.273399000  0.0349918304
gnus-topic-prepare-topic          44          0.7627989999  0.0173363409
gnus-request-scan                 200         0.5203380000  0.0026016900
gnus-read-active-file-2           2           0.402818      0.201409
gnus-retrieve-groups              2           0.398634      0.199317
gnus-group-list-groups            1           0.310868      0.310868

Package nntp, version A:

Function Name                     Call Count  Elapsed Time  Average Time
================================  ==========  ============  ============
nntp-accept-process-output        1592        1.5516599999  0.0009746608
nntp-retrieve-groups              2           1.5244879999  0.7622439999
nntp-accept-response              1525        1.502601      0.0009853121
nntp-request-group                1           0.065724      0.065724
nntp-find-connection-buffer       1598        0.0067499999  4.224...e-06
nntp-erase-buffer                 205         0.0009970000  4.863...e-06
nntp-open-server                  3           0.0002960000  9.866...e-05
nntp-server-opened                11          6.2e-05       5.636...e-06
nntp-possibly-change-group        3           5.3e-05       1.766...e-05
nntp-decode-text                  1           1.5e-05       1.5e-05
nntp-close-group                  1           4e-06         4e-06

Package nntp, version B:

Function Name                     Call Count  Elapsed Time  Average Time
================================  ==========  ============  ============
nntp-accept-process-output        40433       39.736943999  0.0009827849
nntp-request-group                404         39.094312999  0.0967681014
nntp-retrieve-groups              2           1.069817      0.5349085
nntp-accept-response              1075        1.0523970000  0.0009789739
nntp-find-connection-buffer       40439       0.1772040000  4.382...e-06
nntp-erase-buffer                 1011        0.0073319999  7.252...e-06
nntp-possibly-change-group        406         0.0068940000  1.698...e-05
nntp-server-opened                820         0.0052629999  6.418...e-06
nntp-decode-text                  404         0.0048309999  1.195...e-05
nntp-request-update-info          404         0.0015660000  3.876...e-06
nntp-open-server                  6           0.000618      0.000103
nntp-close-group                  1           4e-06         4e-06

My observations:
- with version B, fetching articles takes 29s in the first run, more
  than 40s in the second where version A takes 1.5s.
- version B has a lot more of NNTP activity: it call nntp-request-group
  for each group.
- version B activates twice as many groups as version A.

Cheers,

-- 
Romain Francoise <romain@orebokech.com> | Sometimes I don't know where
it's a miracle -- http://orebokech.com/ | this dirty road is taking me,
                                        | sometimes I can't even see
                                        | the reason why.

Re: Gnus slow?

[Katsumi Yamaoka]

I also tried ELP on No Gnus v0.2 and the latest CVS.  The result
is that the latest CVS is faster than No Gnus v0.2 10 to 20%.
It will be meaningless to compare it with your result since I
subscribe to two foreign servers using nntp-open-via-(ssh and
telnet), the local news server, many shimbun groups and the nnml
groups, though.

>>>>> In <87r7n0zjx9.fsf@orebokech.com> Romain Francoise wrote:

(No Gnus checkout as of May 12th)
> Package gnus, version A (first 10 lines):
> Function Name                     Call Count  Elapsed Time  Average Time
> ================================  ==========  ============  ============
> gnus-activate-group               405         0.6969249999  0.0017208024

(the latest CVS)
> Package gnus, version B (first 10 lines):
> Function Name                     Call Count  Elapsed Time  Average Time
> ================================  ==========  ============  ============
> gnus-activate-group               808         28.273399000  0.0349918304

> - version B activates twice as many groups as version A.

I don't know why gnus-activate-group spends so many time and
seems to be called twice in B, but it is dominant.

Some gnus-* functions are inlined in it and they cannot be
profiled by ELP individually.  So, you need to plunge a scalpel
into those adhesion.  To do that, replace all the
`(inline (gnus-* ...))' forms into `(gnus-* ...)' in the
gnus-activate-group function and recompile gnus-start.el.
Otherwise, it may be enough to only evaluate the function
definition of gnus-activate-group using `M-x eval-region' or
`M-x eval-last-sexp' (`C-x C-e').

And then, perform ELP to the `gnus' package again.

Re: Gnus slow?

[Romain Francoise]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I don't know why gnus-activate-group spends so many time and
> seems to be called twice in B, but it is dominant.

It is slow because in version B, Gnus issues GROUP commands to the NNTP
server and it is synchronous so it has to wait for the answer from the
server.  To make it worse, it does it twice since groups are activated
twice.

You will find captures of the NNTP conversation between Gnus and
news.gmane.org here: <URL: http://orebokech.com/tmp/gnus/>.  Version A
issues LIST ACTIVE commands only, asynchronously.  Version B first
issues GROUP commands for each of my groups, synchronously, then issues
LIST ACTIVE commands like version A, then does GROUP commands again.  I
see two separate issues:
- why does version B send GROUP commands where version A doesn't?  They
  seem unnecessary.
- even so, why does version B send them twice?

> Some gnus-* functions are inlined in it and they cannot be
> profiled by ELP individually.  So, you need to plunge a scalpel
> into those adhesion.  To do that, replace all the
> `(inline (gnus-* ...))' forms into `(gnus-* ...)' in the
> gnus-activate-group function and recompile gnus-start.el.
> Otherwise, it may be enough to only evaluate the function
> definition of gnus-activate-group using `M-x eval-region' or
> `M-x eval-last-sexp' (`C-x C-e').

> And then, perform ELP to the `gnus' package again.

I will perform more ELP tests later today, thanks for investigating this
issue.

-- 
Romain Francoise <romain@orebokech.com> | There's no stronger wind than
it's a miracle -- http://orebokech.com/ | the one that blows down a
                                        | lonesome railroad line.

Re: Gnus slow?

[Katsumi Yamaoka]

You might not need to perform ELP again.

>>>>> In <87u0rv8on2.fsf@orebokech.com> Romain Francoise wrote:

>> I don't know why gnus-activate-group spends so many time and
>> seems to be called twice in B, but it is dominant.

It seems to be caused by calling nntp-request-group by way of
gnus-request-group.  It should have been clear if I looked at
the nntp profile which you got.  Probably, we should clarify the
reason why it is called frequently.

> It is slow because in version B, Gnus issues GROUP commands to the NNTP
> server and it is synchronous so it has to wait for the answer from the
> server.  To make it worse, it does it twice since groups are activated
> twice.

> You will find captures of the NNTP conversation between Gnus and
> news.gmane.org here: <URL: http://orebokech.com/tmp/gnus/>.  Version A
> issues LIST ACTIVE commands only, asynchronously.  Version B first
> issues GROUP commands for each of my groups, synchronously, then issues
> LIST ACTIVE commands like version A, then does GROUP commands again.  I
> see two separate issues:
> - why does version B send GROUP commands where version A doesn't?  They
>   seem unnecessary.
> - even so, why does version B send them twice?

I saw those captures.  Thanks.

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <b9ysm7fl9m9.fsf@jpl.org> Katsumi Yamaoka wrote:

> You might not need to perform ELP again.

Romain, you do not need to perform ELP again. ;-)

> Probably, we should clarify the reason why it is called frequently.

That is indeed caused by the nntp marks feature even if
nntp-marks-is-evil is t.  The nntp-request-group function which
issues the GROUP command is called by the
gnus-get-unread-articles-in-group function when the `g' command
is executed.  It is because the nntp marks feature provide the
nntp-request-update-info function.  It did not exist before May
13.

Although I don't have a concrete plan yet, the problem will be
solved during next week.

Re: Gnus slow?

[Romain Francoise]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Romain, you do not need to perform ELP again. ;-)

Good, it gets tedious after a while.  :-)

> That is indeed caused by the nntp marks feature even if
> nntp-marks-is-evil is t.  The nntp-request-group function which
> issues the GROUP command is called by the
> gnus-get-unread-articles-in-group function when the `g' command
> is executed.  It is because the nntp marks feature provide the
> nntp-request-update-info function.  It did not exist before May
> 13.

NNTP marks are evil, indeed!

> Although I don't have a concrete plan yet, the problem will be
> solved during next week.

Excellent.  Did you find out why groups are activated twice?

-- 
Romain Francoise <romain@orebokech.com> | Pray the day ain't poison,
it's a miracle -- http://orebokech.com/ | stand among the ones that
                                        | live in lonely indecision.

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <873bzf6umt.fsf@orebokech.com> Romain Francoise wrote:

> Did you find out why groups are activated twice?

Hmm, the last message I wrote doesn't indeed make sense.  I must
return to an English writing course.  Let me exercise again. :-p

The reason the recent Gnus uselessly issues the GROUP command
for all the subscribed nntp groups is that the nntp marks feature
provides the `nntp-request-update-info' function.  It always
returns a non-nil value.  When executing the `g' command, it is
called by the `gnus-get-unread-articles-in-group' function by
way of the `gnus-request-update-info' function.  If it returns a
non-nil value, the `gnus-activate-group' function is called with
the third argument `t' and the GROUP command is issued.  Before
May 13, `gnus-request-update-info' returned nil because the
`nntp-request-update-info' function didn't exist then.

Anyway, I've installed the change which makes
`nntp-request-update-info' return nil if `nntp-marks-is-evil' is
non-nil.  Please test!

Note that you have to rebuild Gnus rather than replacing nntp.el
with the new one since the function in question is inlined in
some places.

Re: Gnus slow?

[Romain Francoise]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Anyway, I've installed the change which makes
> `nntp-request-update-info' return nil if `nntp-marks-is-evil' is
> non-nil.  Please test!

This fixes my performance problems, thank you very much!

-- 
Romain Francoise <romain@orebokech.com> | Wait, stop, wait just hold on
it's a miracle -- http://orebokech.com/ | a minute.

Re: Gnus slow?

[Dan Christensen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Anyway, I've installed the change which makes
> `nntp-request-update-info' return nil if `nntp-marks-is-evil' is
> non-nil.  Please test!

I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.
It does have a doc string, and this doc string says

  *If non-nil, Gnus will never generate and use marks file for nntp groups.
  See `nnml-marks-is-evil' for more information.

But I can't get emacs to show me the doc string for nnml-marks-is-evil
(although this variable does show up in the Gnus manual).

Dan

Re: Gnus slow?

[Reiner Steib]

On Mon, Nov 15 2004, Dan Christensen wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>> Anyway, I've installed the change which makes
>> `nntp-request-update-info' return nil if `nntp-marks-is-evil' is
>> non-nil.  Please test!
>
> I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.
> It does have a doc string, and this doc string says
>
>   *If non-nil, Gnus will never generate and use marks file for nntp groups.
>   See `nnml-marks-is-evil' for more information.
>
> But I can't get emacs to show me the doc string for nnml-marks-is-evil

Probably you haven't loaded `nnml.el'.  After `(require 'nnml)' the
variable should be known.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus slow?

[Dan Christensen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Mon, Nov 15 2004, Dan Christensen wrote:
>
>> But I can't get emacs to show me the doc string for nnml-marks-is-evil
>
> Probably you haven't loaded `nnml.el'.  After `(require 'nnml)' the
> variable should be known.

Whoops, that's right.  But my other comment is probably correct,
as I grepped the source of the manual:

>> I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.

Dan

Re: Gnus slow?

[Romain Francoise]

Dan Christensen <jdc@uwo.ca> writes:

> I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.

--- gnus.texi	15 Nov 2004 20:46:10 +0100	7.69
+++ gnus.texi	16 Nov 2004 22:17:29 +0100	
@@ -12803,6 +12803,15 @@
 don't like to see Emacs eat your available CPU power), you might set
 this to, say, 1.
 
+@item nntp-marks-is-evil
+@vindex nntp-marks-is-evil
+If non-@code{nil}, this back end will ignore any @sc{marks} files.  The
+default is @code{nil}.
+
+@item nntp-marks-directory
+@vindex nntp-marks-directory
+The directory where @sc{marks} for nntp groups will be stored.
+
 @end table
 
 @menu

-- 
Romain Francoise <romain@orebokech.com> | It was fourteen degrees below
it's a miracle -- http://orebokech.com/ | on a screeching march 23.

Re: Gnus slow?

[Simon Josefsson]

Romain Francoise <romain@orebokech.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.
>
> --- gnus.texi	15 Nov 2004 20:46:10 +0100	7.69
> +++ gnus.texi	16 Nov 2004 22:17:29 +0100	

Installed, thanks.

Re: Gnus slow?

[Frank Schmitt]

Simon Josefsson <jas@extundo.com> writes:

> Romain Francoise <romain@orebokech.com> writes:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> I noticed that nntp-marks-is-evil isn't documented in the Gnus manual.
>>
>> --- gnus.texi	15 Nov 2004 20:46:10 +0100	7.69
>> +++ gnus.texi	16 Nov 2004 22:17:29 +0100	
>
> Installed, thanks.

Perhaps a note which benefits nntp marks offer would be good. I for
instance have not the faintest clue if I want or need them.

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: Gnus slow?

[Romain Francoise]

Frank Schmitt <ich@Frank-Schmitt.net> writes:

> Perhaps a note which benefits nntp marks offer would be good. I for
> instance have not the faintest clue if I want or need them.

See GNUS-NEWS:

 ** The nntp back end store article marks in `~/News/marks'.

 The directory can be changed using the (customizable) variable
 `nntp-marks-directory', and marks can be disabled using the (backend)
 variable `nntp-marks-is-evil'.  The advantage of this is that you can
 copy `~/News/marks' (using rsync, scp or whatever) to another Gnus
 installation, and it will realize what articles you have read and
 marked.  The data in `~/News/marks' has priority over the same data
 in `~/.newsrc.eld'.

-- 
Romain Francoise <romain@orebokech.com> | Last night the whole of hell
it's a miracle -- http://orebokech.com/ | came down to bathe and dine
                                        | with us.

Re: Gnus slow?

[Frank Schmitt]

Romain Francoise <romain@orebokech.com> writes:

> Frank Schmitt <ich@Frank-Schmitt.net> writes:
>
>> Perhaps a note which benefits nntp marks offer would be good. I for
>> instance have not the faintest clue if I want or need them.
>
> See GNUS-NEWS:
>
>  ** The nntp back end store article marks in `~/News/marks'.
>
>  The directory can be changed using the (customizable) variable
>  `nntp-marks-directory', and marks can be disabled using the (backend)
>  variable `nntp-marks-is-evil'.  The advantage of this is that you can
>  copy `~/News/marks' (using rsync, scp or whatever) to another Gnus
>  installation, and it will realize what articles you have read and
>  marked.  The data in `~/News/marks' has priority over the same data
>  in `~/.newsrc.eld'.

Does this still work, if I use different servers on the tow machines?

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: Gnus slow?

[Romain Francoise]

Frank Schmitt <ich@Frank-Schmitt.net> writes:

> Does this still work, if I use different servers on the tow machines?

I don't know.  This whole thread started because I *don't* use the nntp
marks feature.  Kai could probably tell you more about it.

-- 
Romain Francoise <romain@orebokech.com> | I am the walrus. I am the
it's a miracle -- http://orebokech.com/ | walrus. I am the walrus.

Re: Gnus slow?

[Kai Grossjohann]

Frank Schmitt <ich@Frank-Schmitt.net> writes:

> Does this still work, if I use different servers on the tow machines?

Gnus remembers the article numbers, so if the two servers use the same
article numbers, then things might work.

Most servers do NOT use the same article numbers as any other server.

Kai

Re: Gnus slow?

[Frank Schmitt]

Kai Grossjohann <kai@emptydomain.de> writes:

> Frank Schmitt <ich@Frank-Schmitt.net> writes:
>
>> Does this still work, if I use different servers on the tow machines?
>
> Gnus remembers the article numbers, so if the two servers use the same
> article numbers, then things might work.
>
> Most servers do NOT use the same article numbers as any other server.

Then I still don't understand the benefits of those NNTP marks. Now I can
copy marks, but before I could copy newsrc and newsrc.eld, couldn't I?

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: Gnus slow?

[Kai Grossjohann]

Frank Schmitt <ich@Frank-Schmitt.net> writes:

> Then I still don't understand the benefits of those NNTP marks. Now I can
> copy marks, but before I could copy newsrc and newsrc.eld, couldn't I?

The difference is that .newsrc* contain all servers in one file, but
the .marks files are specific to each server (to each group, even).

This means, you can read servers A, B, C at work and servers A, D, E
at home and sync the marks files for A and then you'll get
synchronization for that server between home and work.

In the long run, I envision that the .newsrc* files whither away, to
be replaced with server-specific or group-specific files.

Kai

nnimap syncing (was: Gnus slow?)

[Steven E. Harris]

Kai Grossjohann <kai@emptydomain.de> writes:

> This means, you can read servers A, B, C at work and servers A, D, E
> at home and sync the marks files for A and then you'll get
> synchronization for that server between home and work.

I wish I could get this to work for my nnimap groups. I finally
convinced my ISP to enable user-defined flags, so "gnus-expire" or
whatever the expiry flag is shows up, but my expiry marks don't seem
to be recognized across different machines. That is, I see an nnimap
article at work, mark it as expirable, come home and connect to the
same server, and see that same article with no expiry mark.

We discussed this a few months ago¹, and at that time the advice was
twofold: stop syncing .newsrc.eld, and get user-defined flags enabled
on the IMAP server. I've done both, and still find the whole syncing
system to be a mess. After each sync, the agent remains confused as
ever.


Footnotes: 
¹ http://article.gmane.org/gmane.emacs.gnus.general/58285

-- 
Steven E. Harris

Re: nnimap syncing

[Simon Josefsson]

"Steven E. Harris" <seh@panix.com> writes:

> Kai Grossjohann <kai@emptydomain.de> writes:
>
>> This means, you can read servers A, B, C at work and servers A, D, E
>> at home and sync the marks files for A and then you'll get
>> synchronization for that server between home and work.
>
> I wish I could get this to work for my nnimap groups. I finally
> convinced my ISP to enable user-defined flags, so "gnus-expire" or
> whatever the expiry flag is shows up, but my expiry marks don't seem
> to be recognized across different machines. That is, I see an nnimap
> article at work, mark it as expirable, come home and connect to the
> same server, and see that same article with no expiry mark.
>
> We discussed this a few months ago¹, and at that time the advice was
> twofold: stop syncing .newsrc.eld, and get user-defined flags enabled
> on the IMAP server. I've done both, and still find the whole syncing
> system to be a mess. After each sync, the agent remains confused as
> ever.

Do you have imap-log buffer output for when you set the expire-mark in
one Gnus, and the log from when entering the group in the other Gnus?

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <861xeowjm0.fsf@ketchup.de.uu.net> Kai Grossjohann wrote:

> Frank Schmitt <ich@Frank-Schmitt.net> writes:

>> Then I still don't understand the benefits of those NNTP marks. Now I can
>> copy marks, but before I could copy newsrc and newsrc.eld, couldn't I?

> The difference is that .newsrc* contain all servers in one file, but
> the .marks files are specific to each server (to each group, even).

> This means, you can read servers A, B, C at work and servers A, D, E
> at home and sync the marks files for A and then you'll get
> synchronization for that server between home and work.

> In the long run, I envision that the .newsrc* files whither away, to
> be replaced with server-specific or group-specific files.

I understand how the nntp marks feature is useful and why it is
activated for all users by default.  However, in fact, it
obviously slows down Gnus.  It is not useful for _all_ users and
it will let those who don't know it simply say "No Gnus is slow".
Isn't it necessary to write clearly the case where it is useful
and the way to deactivate it?  For instance, in the ``Finding
the News'' section.

In addition, I don't think the nntp marks feature is _evil_ even
though I set `nntp-marks-is-evil' as t.  How about the following?

(defvoo nntp-disable-marks (if (boundp 'nntp-marks-is-evil)
			       (symbol-value 'nntp-marks-is-evil))
  "*docstring.")

Re: Gnus slow?

[Matt Armstrong]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <861xeowjm0.fsf@ketchup.de.uu.net> Kai Grossjohann wrote:

[...]

>> In the long run, I envision that the .newsrc* files whither away,
>> to be replaced with server-specific or group-specific files.
>
> I understand how the nntp marks feature is useful and why it is
> activated for all users by default.  However, in fact, it obviously
> slows down Gnus.

[...]

I don't think we should be debating about whether nntp-marks-is-evil
should be t or nil by default.  Ideally, we'd just find a way to make
this fast:

    (setq nntp-marks-is-evil nil)

Disclaimer: I'm just a long time Gnus user who can't actually
*contribute* to this because I seem to have some block when it comes
to writing code in emacs+elisp.  :-)


-- 
matt

Re: Gnus slow?

[Kai Grossjohann]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I understand how the nntp marks feature is useful and why it is
> activated for all users by default.  However, in fact, it
> obviously slows down Gnus.

Right.  That's a bug which is my fault, I guess :-(

(It isn't slow for me, but I should have woken up on the second or
third report about the slowness.)

> It is not useful for _all_ users and it will let those who don't
> know it simply say "No Gnus is slow".  Isn't it necessary to write
> clearly the case where it is useful and the way to deactivate it?
> For instance, in the ``Finding the News'' section.

Yes, it should clearly be said how to turn it off.

Kai

Re: Gnus slow?

[Romain Francoise]

Kai Grossjohann <kai@emptydomain.de> writes:

> The difference is that .newsrc* contain all servers in one file, but
> the .marks files are specific to each server (to each group, even).

> This means, you can read servers A, B, C at work and servers A, D, E
> at home and sync the marks files for A and then you'll get
> synchronization for that server between home and work.

Thanks for the insight, Kai!

I summarized this useful information in a node in the manual, I'd be
grateful to anyone with commit access to proofread and install it:

(I got rid of @sc{} for "marks", it made the text look goofy.)


2004-11-23  Romain Francoise  <romain@orebokech.com>

	* gnus.texi (NNTP marks): New node.
	(NNTP): Move NNTP marks variables to the new node.


--- gnus.texi	22 Nov 2004 22:58:24 +0100	7.73
+++ gnus.texi	23 Nov 2004 21:41:30 +0100	
@@ -646,6 +646,7 @@
 * Direct Functions::            Connecting directly to the server.
 * Indirect Functions::          Connecting indirectly to the server.
 * Common Variables::            Understood by several connection functions.
+* NNTP marks::                  Storing marks for @acronym{NNTP} servers.
 
 Getting Mail
 
@@ -12804,21 +12805,13 @@
 don't like to see Emacs eat your available CPU power), you might set
 this to, say, 1.
 
-@item nntp-marks-is-evil
-@vindex nntp-marks-is-evil
-If non-@code{nil}, this back end will ignore any @sc{marks} files.  The
-default is @code{nil}.
-
-@item nntp-marks-directory
-@vindex nntp-marks-directory
-The directory where @sc{marks} for nntp groups will be stored.
-
 @end table
 
 @menu
 * Direct Functions::            Connecting directly to the server.
 * Indirect Functions::          Connecting indirectly to the server.
 * Common Variables::            Understood by several connection functions.
+* NNTP marks::                  Storing marks for @acronym{NNTP} servers.
 @end menu
 
 
@@ -13068,6 +13061,52 @@
 
 @end table
 
+@node NNTP marks
+@subsubsection NNTP marks
+@cindex storing NNTP marks
+
+Gnus stores marks (@pxref{Marking Articles}) for @acronym{NNTP}
+servers in marks files.  A marks file records what marks you have set
+in a group and each file is specific to the corresponding server.
+Marks files are stored in @file{~/News/marks}
+(@code{nntp-marks-directory}) under a classic hierarchy resembling
+that of a news server, for example marks for the group
+@samp{gmane.discuss} on the news.gmane.org server will be stored in
+the file @file{~/News/marks/news.gmane.org/gmane/discuss/.marks}.
+
+Marks files are useful because you can copy the @file{~/News/marks}
+directory (using rsync, scp or whatever) to another Gnus installation,
+and it will realize what articles you have read and marked.  The data
+in @file{~/News/marks} has priority over the same data in
+@file{~/.newsrc.eld}.
+
+Note that marks files are very much server-specific: Gnus remembers
+the article numbers so if you don't use the same servers on both
+installations things are most likely to break (most @acronym{NNTP}
+servers do not use the same article numbers as any other server).
+However, if you use servers A, B, C on one installation and servers A,
+D, E on the other, you can sync the marks files for A and then you'll
+get synchronization for that server between the two installations.
+
+Using @acronym{NNTP} marks can possibly incur a performance penalty so
+if Gnus feels sluggish, try setting the @code{nntp-marks-is-evil}
+variable to @code{t}.  Marks will then be stored in @file{~/.newsrc.eld}.
+
+Related variables:
+
+@table @code
+
+@item nntp-marks-is-evil
+@vindex nntp-marks-is-evil
+If non-@code{nil}, this back end will ignore any marks files.  The
+default is @code{nil}.
+
+@item nntp-marks-directory
+@vindex nntp-marks-directory
+The directory where marks for nntp groups will be stored.
+
+@end table
+
 
 @node News Spool
 @subsection News Spool

-- 
Romain Francoise <romain@orebokech.com> | Shine the headlight, straight
it's a miracle -- http://orebokech.com/ | into my eyes.

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <87wtwcz47z.fsf@orebokech.com> Romain Francoise wrote:

> I summarized this useful information in a node in the manual, I'd be
> grateful to anyone with commit access to proofread and install it:

> (I got rid of @sc{} for "marks", it made the text look goofy.)

> 2004-11-23  Romain Francoise  <romain@orebokech.com>

> 	* gnus.texi (NNTP marks): New node.
> 	(NNTP): Move NNTP marks variables to the new node.

Thanks.  It looks complete to me.  I'll install the patch in No
Gnus.  Romain, have you already signed a paper? (If so, you'd
better have a write access to CVS, I think. ;-)

In addition to this, I'd like to add a note to the end of the
`Finding the News' section as follows:

Note: For the @acronym{NNTP} groups, what is called the @acronym{NNTP}
marks feature is activated by default.  Although it is activated by
default since it is very useful, getting new news may be made slow in
some cases.  @xref{NNTP marks} for more information.

Unfortunately, I'm not a good English writer and don't have a
good Japanese-English translating software.  So, I greatly
appreciate someone correcting it.  Thanks in advance.

Re: Gnus slow?

[Romain Francoise]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> It looks complete to me.  I'll install the patch in No Gnus.  Romain,
> have you already signed a paper?

Yes, I have copyright assignment papers on file for Gnus, and Emacs.

> In addition to this, I'd like to add a note to the end of the
> `Finding the News' section as follows:

> Note: For the @acronym{NNTP} groups, what is called the @acronym{NNTP}
> marks feature is activated by default.  Although it is activated by
> default since it is very useful, getting new news may be made slow in
> some cases.  @xref{NNTP marks} for more information.

How about:

Note: the @acronym{NNTP} back end stores marks in marks files
(@pxref{NNTP marks}).  This feature makes it easy to share marks between
several Gnus installations, but may slow down things a bit when fetching
new articles.  @xref{NNTP marks} for more information.

-- 
Romain Francoise <romain@orebokech.com> | There are doors that open by
it's a miracle -- http://orebokech.com/ | themselves.

Re: Gnus slow?

[Katsumi Yamaoka]

>>>>> In <87act6xtmo.fsf@orebokech.com> Romain Francoise wrote:

> Yes, I have copyright assignment papers on file for Gnus, and Emacs.

That's good.

>> In addition to this, I'd like to add a note to the end of the
>> `Finding the News' section as follows:

> How about:

> Note: the @acronym{NNTP} back end stores marks in marks files
> (@pxref{NNTP marks}).  This feature makes it easy to share marks between
> several Gnus installations, but may slow down things a bit when fetching
> new articles.  @xref{NNTP marks} for more information.

Thank you.  It is better than mine because it specifies how that
feature is useful.  I'll install it later.