[gmane.emacs.devel] gnus-read-mark not preserved

[gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

This has not yet garnered a response over at emacs-devel, and I'm
still seeing the problem.  Is this list appropriate?  I'm not sure if
it's an Emacs or Gnus bug or a server or user problem.  I don't know
if it's relevant, but I always start Gnus with gnus-unplugged.

Steve Berman


[-- Attachment #2: Type: message/rfc822, Size: 4878 bytes --]

From: Stephen Berman <Stephen.Berman@gmx.net>
To: emacs-devel@gnu.org
Subject: gnus-read-mark not preserved
Date: Mon, 04 Jun 2007 11:23:53 +0200
Message-ID: <87lkf0f4ye.fsf@escher.local.home>

When I read newsgroup articles with Gnus they get marked with `R'
(gnus-read-mark) and are not displayed the next time I enter the
group.  However, I have been noticing that, when I

1. enter a group that satisfies gnus-large-newsgroup,
2. have Gnus fetch less than all the articles, 
3. read articles, exit the group, and at some point exit Emacs, 

then the next time I enter the group in a new Emacs session, the
articles I read the last time are displayed, i.e., not marked read.

I am not certain this happens every time and with every group, and I
am not certain, but I think, this does not happen when I fetch all
unread articles.  I have noticed this repeatedly with
gmane.emacs.pretest.bugs and gmane.emacs.devel (I have
gnus-select-method set to '(nntp "news.gmane.org"); those are the
groups for which conditions 1 and 2 above most often hold according to
my Gnus usage.

I am currently using GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version
2.10.6) of 2007-06-02 on escher, but I have noticed this for some time
on CVS builds, though I cannot say how long (but I'm pretty sure I
haven't experienced this for a very long time, otherwise I would have
reported it earlier, though possibly my usage wrt condition 2 has
changed, making this appear more frequently for me).

Is this a Gnus bug, or a Gmane bug, or am I overlooking some setting
that triggers this behavior?

Steve Berman

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

It seems that no one else is having this problem, so I assume I must
have a bad setting somewhere.  I would be grateful for any advice
about how to solve this problem.[1]  It is making it very unpleasant to
use Gnus to read certain newsgroups.

Steve Berman

Footnotes: 
[1]  I know I can comment out all my Gnus setting and try them one at
a time, but given that I only see the problem when I restart Emacs,
that would be very time-consuming.  I hope there's some other way of
looking for the cause.

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Katsumi Yamaoka]

>>>>> In <87r6o471vv.fsf@escher.local.home> Stephen Berman wrote:

> It seems that no one else is having this problem, so I assume I must
> have a bad setting somewhere.  I would be grateful for any advice
> about how to solve this problem.[1]  It is making it very unpleasant to
> use Gnus to read certain newsgroups.

I have never used Gnus in the unplugged condition, so I am not
skilled in the J-* commands and was not successful in reproducing
the problem so far.  We will probably need the concrete way to
necessarily be able to reproduce it...

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

On Fri, 22 Jun 2007 19:45:13 +0900 Katsumi Yamaoka <yamaoka@jpl.org> wrote:

>>>>>> In <87r6o471vv.fsf@escher.local.home> Stephen Berman wrote:
>
>> It seems that no one else is having this problem, so I assume I must
>> have a bad setting somewhere.  I would be grateful for any advice
>> about how to solve this problem.[1]  It is making it very unpleasant to
>> use Gnus to read certain newsgroups.
>
> I have never used Gnus in the unplugged condition, so I am not
> skilled in the J-* commands and was not successful in reproducing
> the problem so far.  We will probably need the concrete way to
> necessarily be able to reproduce it...

If only I knew a concrete, and reliable, way to reproduce it!  The
recipe in my OP was the best I could come up with, but as I noted in a
followup a few minutes ago, I am also seeing the problem in other
circumstances.  I suppose I can stop using gnus-unplugged for a while;
if I don't experience the problem then, that would at least strongly
implicate gnus-unplugged, and if I still do, then it would exonerate
gnus-unplugged.

Steve Berman

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

On Tue, 19 Jun 2007 08:52:16 +0200 Stephen Berman <Stephen.Berman@gmx.net> wrote:

> [...] I have been noticing that, when I
>
> 1. enter a group that satisfies gnus-large-newsgroup,
> 2. have Gnus fetch less than all the articles, 
> 3. read articles, exit the group, and at some point exit Emacs, 
>
> then the next time I enter the group in a new Emacs session, the
> articles I read the last time are displayed, i.e., not marked read.

It appears that all marks are lost: they are not marked new (.) and,
what's more distressing, ticks (!) are also undone.

Steve Berman

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

On Fri, 22 Jun 2007 11:58:18 +0200 Stephen Berman <Stephen.Berman@gmx.net> wrote:

> On Tue, 19 Jun 2007 08:52:16 +0200 Stephen Berman <Stephen.Berman@gmx.net> wrote:
>
>> [...] I have been noticing that, when I
>>
>> 1. enter a group that satisfies gnus-large-newsgroup,
>> 2. have Gnus fetch less than all the articles, 
>> 3. read articles, exit the group, and at some point exit Emacs, 
>>
>> then the next time I enter the group in a new Emacs session, the
>> articles I read the last time are displayed, i.e., not marked read.

Now I'm also seeing this in groups that do not satisfy conditions 1
and 2 above.  Could it be that my .newsrc.eld file is (sometimes,
under certain cirmcumstances) not being (fully) updated.  Is there
some way to force this?

Steve Berman

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Florent Rougon]

Hi,

Stephen Berman <Stephen.Berman@gmx.net> wrote:

> It appears that all marks are lost: they are not marked new (.) and,
> what's more distressing, ticks (!) are also undone.

I believe I have the same problem. A few years ago, I used Gnus for both
mail and news[1] without any problem. Then, by the end of 2001, I
stopped reading the news and kept Gnus only for the mail. Lately (in
April or so), I wanted to read some newsgroups again in Unplugged mode
but encountered this problem: every now and then, Gnus didn't remember a
bunch of articles I had read and was displaying them as unread again (it
seems to be "all articles later than $some_date").

This happened every two days, or something like that. If I read a group
and entered it a few minutes later, no problem. But when I started to
read it again the day after, or two days later, then for some reason,
Gnus had forgotten a lot of the articles I had read and was displaying
them as unread again. This made news-reading almost impossible, so I
stopped reading news, again. :-/

I tried killing the groups, removing them from and readding them to the
agent, deleting the files under ~/News/agent/nntp/$server and checking
that no information whatsoever from the newsgroup I was trying to read
was present in ~/.newsrc.eld before subscribing to it again, but nothing
helped. Every time I recreated the group and started to read it again,
Gnus would forget a whole bunch of the read messages one or two days
later.

If someone can help us solve this issue, I'd be very grateful. Thanks.

  [1] with gnus-unplugged

-- 
Florent

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1662 bytes --]

>>>>> In <87r6nwm40f.fsf@florent.maison> Florent Rougon wrote:

> Stephen Berman <Stephen.Berman@gmx.net> wrote:

>> It appears that all marks are lost: they are not marked new (.) and,
>> what's more distressing, ticks (!) are also undone.

> I believe I have the same problem.

I found the way to reproduce it and a solution for No Gnus, not
for Gnus v5.11.

The point is that the nntp marks are not updated when unplugged,
and those marks are preferred to the ones that are saved in the
~/.newsrc.eld file next time you start Gnus.  This will arise
not only with nntp but also with nnimap, nnmaildir, etc.  So,
*I wish to ask people who use those backends together with the*
*agent to test the attached patch*, since I use only nntp and nnml.

Gnus v5.11 does not use the nntp marks, so there will probably
be another cause.  I am going to look into it further.

Here is the way to reproduce the problem for people who don't
use `unplugged' usually.  You need to have the agent enabled of
course.

1. Make sure Gnus is plugged.

2. Enter to a certain group in which there are a few unread
   articles using the RET key.

3. Mark unread articles with the # marks, and type `J s' to
   download those articles.

4. Exit the summary buffer using `q'.

5. Type `J j' to make Gnus unplugged.

6. Eenter to the group that you entered to before, read unread
   articles, and exit the summary buffer using `q'.

7. Make sure there is no unread article in that group, and exit
   Gnus using `q'.

8. Restart Emacs and Gnus.  And you will see those articles that
   are not marked as read.

The patch attached can be used with the trunk and the released
No Gnus v0.6.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 587 bytes --]

--- nnagent.el~	2007-01-24 07:15:37 +0000
+++ nnagent.el	2007-06-29 10:46:02 +0000
@@ -132,6 +132,12 @@
     (insert ")\n")
     (write-region (point-min) (point-max) (gnus-agent-lib-file "flags")
 		  t 'silent))
+  ;; Also set mark for the original back end.
+  (let ((gnus-agent nil)
+	(method (gnus-find-method-for-group group)))
+    (when (gnus-check-backend-function 'request-set-mark (car method))
+      (funcall (gnus-get-function method 'request-set-mark)
+	       group action server)))
   nil)
 
 (deffoo nnagent-retrieve-headers (articles &optional group server fetch-old)

[-- Attachment #3: Type: text/plain, Size: 20 bytes --]


See you next week.

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Florent Rougon]

Yes, it works! :-)

Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> The patch attached can be used with the trunk and the released
> No Gnus v0.6.

With your recipe and without the patch, I can reproduce the bug. Again
with the same recipe but with the patch applied, the bug doesn't happen.
I'll now start reading again some nntp newsgroups and tell you if
anything wrong happens, but I think you did spot and fix the problem I
was experiencing.

Thank you very much. :-)

-- 
Florent

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

>>>>> In <87ejjukg6s.fsf@florent.maison> Florent Rougon wrote:

> With your recipe and without the patch, I can reproduce the bug. Again
> with the same recipe but with the patch applied, the bug doesn't happen.
> I'll now start reading again some nntp newsgroups and tell you if
> anything wrong happens, but I think you did spot and fix the problem I
> was experiencing.

Thanks for verifying the patch.  However, what I did might be
a kludge.  That is, making nnagent handle marks of the specific
back ends looks ill-mannered, since to abstract the actual back
ends seems to be one of the aims of nnagent.  The reason I changed
my idea is because I found the variable:

gnus-agent-synchronize-flags

The default value is nil.  See (info "(gnus)Agent and flags").
Does setting it to t solve the problem that you all are troubled
with?  Could someone, who uses Gnus v5.11, try it?  (I couldn't
reproduce the problem with Gnus v5.11 so far, though.)

But, for all that, updating at least the nntp marks when unplugged
is absolutely necessary.  Because the nntp marks can be copied to
another nntp client that connects to the same nntp server.

The patch I posted is probably harmful to some back end(s) that
keeps marks in the server, not the local system, and is unable
to set marks when unplugged.  I'm not quite sure of it but at
least the nnimap back end seems to be applied to.  Therefore, I
installed the code that I modified so that it may work with only
the nntp back end, in the Gnus CVS trunk.

Here is a revised patch for the released No Gnus v0.6:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 681 bytes --]

--- nnagent.el~	2007-01-24 07:15:37 +0000
+++ nnagent.el	2007-07-02 12:10:33 +0000
@@ -132,6 +132,14 @@
     (insert ")\n")
     (write-region (point-min) (point-max) (gnus-agent-lib-file "flags")
 		  t 'silent))
+  ;; Also set the marks for the original back end that keeps marks in
+  ;; the local system.
+  (let ((gnus-agent nil))
+    (when (and (memq (car gnus-command-method) '(nntp))
+	       (gnus-check-backend-function 'request-set-mark
+					    (car gnus-command-method)))
+      (funcall (gnus-get-function gnus-command-method 'request-set-mark)
+	       group action server)))
   nil)
 
 (deffoo nnagent-retrieve-headers (articles &optional group server fetch-old)

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

On Mon, 02 Jul 2007 21:15:35 +0900 Katsumi Yamaoka <yamaoka@jpl.org> wrote:

>>>>>> In <87ejjukg6s.fsf@florent.maison> Florent Rougon wrote:
>
>> With your recipe and without the patch, I can reproduce the bug. Again
>> with the same recipe but with the patch applied, the bug doesn't happen.
>> I'll now start reading again some nntp newsgroups and tell you if
>> anything wrong happens, but I think you did spot and fix the problem I
>> was experiencing.
>
> Thanks for verifying the patch.  However, what I did might be
> a kludge.  That is, making nnagent handle marks of the specific
> back ends looks ill-mannered, since to abstract the actual back
> ends seems to be one of the aims of nnagent.  The reason I changed
> my idea is because I found the variable:
>
> gnus-agent-synchronize-flags
>
> The default value is nil.  See (info "(gnus)Agent and flags").
> Does setting it to t solve the problem that you all are troubled
> with?  Could someone, who uses Gnus v5.11, try it?  (I couldn't
> reproduce the problem with Gnus v5.11 so far, though.)

I am using Gnus 5.11 as included in the Emacs CVS trunk (currently GNU
Emacs 22.1.50.6 (i686-pc-linux-gnu, GTK+ Version 2.10.6) of 2007-06-30
on escher).  Here gnus-agent-synchronize-flags has the default value
t[1] (and the Gnus info lacks a node "Agents and flags").  I am still
experiencing this problem and I haven't changed the value of this
variable, so setting it to t is not the solution, at least in 5.11.
Unfortunately, I still have found no reliable recipe for reproducing
the problem.  For whatever reason, I have sometimes gone through
several Emacs sessions without seeing it, then it shows up again in
some group.  Throughout, I've been consistently using gnus-unplugged.

Steve Berman

Footnotes: 
[1]  But the code has this rather puzzling comment [sic]:
  ;; If the default switches to something else than nil, then the function
  ;; should be fixed not be exceedingly slow.  See 2005-09-20 ChangeLog entry.

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 2245 bytes --]

>>>>> In <87ir921r5w.fsf@escher.local.home> Stephen Berman wrote:
> On Mon, 02 Jul 2007 21:15:35 +0900 Katsumi Yamaoka <yamaoka@jpl.org> wrote:

>> gnus-agent-synchronize-flags
>>
>> The default value is nil.  See (info "(gnus)Agent and flags").

> I am using Gnus 5.11 as included in the Emacs CVS trunk (currently GNU
> Emacs 22.1.50.6 (i686-pc-linux-gnu, GTK+ Version 2.10.6) of 2007-06-30
> on escher).  Here gnus-agent-synchronize-flags has the default value
> t[1] (and the Gnus info lacks a node "Agents and flags").

Sorry, I overlooked it, since I usually use No Gnus and see the
No Gnus sources.  I wonder why the default value differs between
Gnus v5.11 and No Gnus.  If anything, I think it should always be
t, or be hard coded.  There's a comment in gnus-agent.el:

> Footnotes:
> [1]  But the code has this rather puzzling comment [sic]:
>   ;; If the default switches to something else than nil, then the function
>   ;; should be fixed not be exceedingly slow.  See 2005-09-20 ChangeLog entry.

However, so far I don't see Gnus to be exceedingly slow.

Apart from this, I found an obvious bug that concerns the Agent,
though it might be different from Stephen Berman's problem.

cf. http://news.gmane.org/group/gmane.emacs.bugs/thread=16061

The prerequisites to cause this bug are:

1. You have persistent (i.e., cached, marked with `*') articles
   that you have read (or marked as ticked) in a group.

2. Some or all of those articles have been expired in the server.

3. You have ever performed `gnus-agent-expire' or
   `gnus-agent-regenerate', or have ever simply deleted the
   ~/News/agent/ directory.  (In the last case, the directory
   and files are regenerated automatically.)

In that case, some or all of persistent articles that have been
expired in the server are recognized as unfetched articles,
because the Agent cannot fetch those headers from the server.
And then, they are all marked as unread when you run the `c'
command (`gnus-summary-catchup-and-exit'), that recognizes
unfetched articles as unread articles.

I've fixed the `gnus-summary-catchup' function so that it might
not recognize cached articles as unfetched articles, in both the
v5-10 branch and the trunk.

The patch for Gnus v5.11 is here:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 561 bytes --]

--- gnus-sum.el~	2007-06-06 15:39:11 +0000
+++ gnus-sum.el	2007-07-04 09:02:52 +0000
@@ -10514,7 +10514,8 @@
 		      (gnus-sorted-nunion
                        (gnus-sorted-intersection gnus-newsgroup-unreads
 						 gnus-newsgroup-downloadable)
-                       gnus-newsgroup-unfetched)))
+		       (gnus-sorted-difference gnus-newsgroup-unfetched
+					       gnus-newsgroup-cached))))
 	    ;; We actually mark all articles as canceled, which we
 	    ;; have to do when using auto-expiry or adaptive scoring.
 	    (gnus-summary-show-all-threads)

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1102 bytes --]

Hi,

Doesn't your problem apply to this case?  I tried using unpatched
Gnus v5.11 for a couple of these days to read and post mail and news
articles as usual, however I didn't encounter any other trouble.

>>>>> Katsumi Yamaoka wrote:

> The prerequisites to cause this bug are:

> 1. You have persistent (i.e., cached, marked with `*') articles
>    that you have read (or marked as ticked) in a group.

> 2. Some or all of those articles have been expired in the server.

> 3. You have ever performed `gnus-agent-expire' or
>    `gnus-agent-regenerate', or have ever simply deleted the
>    ~/News/agent/ directory.  (In the last case, the directory
>    and files are regenerated automatically.)

> In that case, some or all of persistent articles that have been
> expired in the server are recognized as unfetched articles,
> because the Agent cannot fetch those headers from the server.
> And then, they are all marked as unread when you run the `c'
> command (`gnus-summary-catchup-and-exit'), that recognizes
> unfetched articles as unread articles.

[...]

> The patch for Gnus v5.11 is here:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 561 bytes --]

--- gnus-sum.el~	2007-06-06 15:39:11 +0000
+++ gnus-sum.el	2007-07-04 09:02:52 +0000
@@ -10514,7 +10514,8 @@
 		      (gnus-sorted-nunion
                        (gnus-sorted-intersection gnus-newsgroup-unreads
 						 gnus-newsgroup-downloadable)
-                       gnus-newsgroup-unfetched)))
+		       (gnus-sorted-difference gnus-newsgroup-unfetched
+					       gnus-newsgroup-cached))))
 	    ;; We actually mark all articles as canceled, which we
 	    ;; have to do when using auto-expiry or adaptive scoring.
 	    (gnus-summary-show-all-threads)

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Stephen Berman]

On Fri, 06 Jul 2007 10:08:49 +0900 Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> Hi,
>
> Doesn't your problem apply to this case?  I tried using unpatched
> Gnus v5.11 for a couple of these days to read and post mail and news
> articles as usual, however I didn't encounter any other trouble.

I haven't tried the patch yet, because you wrote:
>
>> The prerequisites to cause this bug are:
>
>> 1. You have persistent (i.e., cached, marked with `*') articles
>>    that you have read (or marked as ticked) in a group.

I don't have cached articles, all the articles remain on the server.
So I assumed you were addressing a different problem.  I will apply
your patch and report back.  But, as I've noted, I don't experience
this problem regularly (and in the last week or so only a couple of
times), so even if I don't see it again for a while, that doesn't
necessarily mean this fixes it (unless I never see it again).

Steve Berman

Re: [gmane.emacs.devel] gnus-read-mark not preserved

[Florent Rougon]

Hi Katsumi,

A few fays after you posted

  http://article.gmane.org/gmane.emacs.gnus.general/64858

I made as you suggested, i.e.:

  1. Set gnus-agent-synchronize-flags to t (previously, I had it to nil,
     by default);

  2. Apply the patch in that message.

Since then, I didn't have much time to read NNTP news, but everytime I
did, things behaved perfectly fine. Therefore, I suppose that either one
or both of these actions are responsible for fixing my problem.

This was with:

  GNU Emacs 21.4.1 (i486-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of
  2007-06-19 on ninsei, modified by Debian

and

  No Gnus v0.6

I think I'm going to switch to Emacs 22 and a more recent No Gnus soon,
so this is probably my last report in this configuration.

Again, thanks for your help.

-- 
Florent