Re: Gnus lies about downloaded state

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin Greiner writes:

  Kevin> Well, the good news is that you and Danny are reporting consistent
  Kevin> results.  The bad news is that we've narrowed the problem to
  Kevin> gnus-sum!

  Kevin> What value do you have for gnus-summary-mark-positions?  (It's
  Kevin> buffer local to the summary buffer.)

gnus-summary-mark-positions's value is 
((download . 8)
 (score . 2)
 (replied . 1)
 (unread . 0))

  Kevin> You should see (download . #) where # is the column number where
  Kevin> you expect to see the +/- download mark.

  Kevin> If that looks right, try adding this statement to the beginning of
  Kevin> gnus-summary-insert-line

  Kevin> (princ (format "%s %s\n" gnus-tmp-current undownloaded (get-buffer
  Kevin> "**scratch**")))

nil t
nil t

This is strange.  gnus-summary-insert-line only called twice?  I have ~250
unread articles.  and Why article number is nil?  hmmm...

  Kevin> This will print the article number and download status (downloaded
  Kevin> ==> nil / undownloaded ==> t) to the scratch buffer.

  Kevin> After displaying the summary, check scratch to see if the download
  Kevin> flag matches what is displayed.  By your descriptions it should not
  Kevin> match.

  Kevin> Pls send me the value of your gnus-format-specs.

((summary-dummy "                %(|---------------------------|%) %S\n"
                (progn
                  (insert "                ")
                  (gnus-put-text-property
                   (point)
                   (progn
                     (insert "|---------------------------|")
                     (point))
                   gnus-mouse-face-prop gnus-mouse-face)
                  (insert " " gnus-tmp-subject "\n")))
 (summary-mode "Gnus: %g [%A] %Z"
               (concat
                (format "Gnus: %s [%d] %s"
                        (gnus-short-group-name gnus-tmp-group-name)
                        gnus-tmp-article-number gnus-tmp-unread-and-unselected)))
 (summary "%U%R%z%5N%O %(%8&user-date; %~(pad 2)~(max 2)~(ignore \"1\")t%[%5k:%* %-20,20n%] %B%s%)\n"
          (let
              (gnus-position)
            (insert
             (format "%c%c%c%5d%c " gnus-tmp-unread gnus-tmp-replied gnus-tmp-score-char
                     (aref gnus-tmp-header 0)
                     gnus-tmp-downloaded))
            (gnus-put-text-property
             (point)
             (progn
               (insert
                (format "%8s %2s%c%5s:"
                        (gnus-user-date
                         (aref gnus-tmp-header 3))
                        (let*
                            ((val
                              (eval
                               (let
                                   ((val
                                     (eval
                                      (let
                                          ((val
                                            (eval
                                             (int-to-string
                                              (gnus-summary-number-of-articles-in-thread
                                               (and
                                                (boundp 'thread)
                                                (car thread))
                                               gnus-tmp-level)))))
                                        (if
                                            (equal val "1")
                                            "" val)))))
                                 (if
                                     (>
                                      (string-width val)
                                      2)
                                     (gnus-correct-substring val 0 2)
                                   val))))
                             (need
                              (- 2
                                 (string-width val))))
                          (if
                              (> need 0)
                              (concat
                               (make-string need 32)
                               val nil)
                            val))
                        gnus-tmp-opening-bracket
                        (gnus-summary-line-message-size gnus-tmp-header)))
               (setq gnus-position
                     (point))
               (insert
                (format " %-20s%c %s%s"
                        (let*
                            ((val
                              (eval
                               (if
                                   (>
                                    (string-width gnus-tmp-name)
                                    20)
                                   (gnus-correct-substring gnus-tmp-name 0 20)
                                 gnus-tmp-name)))
                             (need
                              (- 20
                                 (string-width val))))
                          (if
                              (> need 0)
                              (concat nil val
                                      (make-string need 32))
                            val))
                        gnus-tmp-closing-bracket gnus-tmp-thread-tree-header-string gnus-tmp-subject-or-nil))
               (point))
             gnus-mouse-face-prop gnus-mouse-face)
            (insert "\n")
            (if gnus-position
                (gnus-put-text-property gnus-position
                                        (1+ gnus-position)
                                        'gnus-position t))))
 (group-mode "Gnus: %%b {%M%:%S}"
             (concat
              (format "Gnus: %%b {%s%s%s}" gnus-tmp-news-method gnus-tmp-colon gnus-tmp-news-server)))
 (group "%m%M%S%P%2L [.%~(ignore 0)-3U !%~(ignore 0)-3T] %4y: %(%-20uG%) %20ue\n"
        (progn
          (insert
           (format "%c%c%c%s%2d [.%-3s !%-3s] %4s: "
                   (gnus-group-new-mail gnus-tmp-group)
                   gnus-tmp-marked-mark gnus-tmp-subscribed gnus-group-indentation gnus-tmp-level
                   (let*
                       ((val
                         (eval
                          (let
                              ((val
                                (eval
                                 (int-to-string
                                  (gnus-number-of-unseen-articles-in-group gnus-tmp-group)))))
                            (if
                                (equal val "0")
                                "" val))))
                        (need
                         (- 3
                            (string-width val))))
                     (if
                         (> need 0)
                         (concat nil val
                                 (make-string need 32))
                       val))
                   (let*
                       ((val
                         (eval
                          (let
                              ((val
                                (eval
                                 (int-to-string
                                  (gnus-range-length
                                   (cdr
                                    (assq 'tick gnus-tmp-marked)))))))
                            (if
                                (equal val "0")
                                "" val))))
                        (need
                         (- 3
                            (string-width val))))
                     (if
                         (> need 0)
                         (concat nil val
                                 (make-string need 32))
                       val))
                   gnus-tmp-number-of-unread))
          (gnus-put-text-property
           (point)
           (progn
             (insert
              (format "%-20s"
                      (gnus-user-format-function-G gnus-tmp-header)))
             (point))
           gnus-mouse-face-prop gnus-mouse-face)
          (insert
           (format " %20s\n"
                   (gnus-user-format-function-e gnus-tmp-header)))))
 (version . "21.2.1")
 (article-mode "Gnus: %g [%w] %S%m"
               (concat
                (format "Gnus: %s [%s] %s%s"
                        (gnus-short-group-name gnus-tmp-group-name)
                        (gnus-article-wash-status)
                        gnus-tmp-subject
                        (gnus-article-mime-part-status)))))

  Kevin> Kevin


-- 
Danny Siu

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin Greiner writes:

  Kevin> Danny, We've been on this thread for a week now.  Can you summarize
  Kevin> the error that you are currently seeing?  We've had so many posts
  Kevin> that I'm not certain that I understand the current situation.

I think I am close to nailing down the cause of the problem.  Will need to
debug more to confirm my result.  Here are the high level bits.

The problem arises in gnus-summary-prepare-unthreaded and
gnus-summary-prepare-threads where gnus-newsgroup-undownloaded contains
incorrect article numbers of undownloadeds when those articles are actually
downloaded.  i tracked it down to gnus-select-newsgroup() which uses
gnus-agent-get-undownloaded-list to update gnus-newsgroup-undownloaded.

In gnus-agent-get-undownloaded-list(), sometimes 'undownloaded' is
calculated incorrectly and hence affacts gnus-newsgroup-undownloaded which
used by gnus-sum.  undownloaded contains a list of article number which are
in fact downloaded.  That is confirmed by breaking and looking at (alist
(gnus-agent-load-alist gnus-newsgroup-name)) at the very beginning.

One more observation: entering a group with and without prefix (C-u) MADE a
different on how the summary buffer is highlighted.  C-u enter a group
almost always give me a correctly highlighted summary versus entering a
group without prefix will show the problem of incorrectly highlighted
download states.  In debugging gnus-agent-get-undownloaded-list(), I think
the two cases differs only in the content of (headers
gnus-newsgroup-headers).

Any additional info is welcome.  I will dig in more later today.

  Kevin> If the problem is in the selection of the face, you should check
  Kevin> the value of gnus-summary-highlight.  It's value is expanded into a
  Kevin> cond so the first case that matches determines the face.

-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> For article A, this is correct (if ugly) behavior.  The agent and
> cache are two distinct services so an article can be cached yet
> undownloaded :(.

FWIW, I just checked on article 44159 in gnus.ding on quimby.  It is
cached and marked as not downloaded.  But I have a file
News/agent/nntp/quimby/gnus/ding/44159, and it has the right contents.

And gnus-agent-article-alist has the entry (44159 . 731213) for this
article.

So I suspect that ticked articles suffer from the same bug as M-u'd
ones.  (Probably most articles are downloaded anyway -- including the
ones I later tick -- because I have the habit of doing `J s' after
getting new news, and also I have gnus-agent-fetch-selected-article in
gnus-select-article-hook.)
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> We've been on this thread for a week now.  Can you summarize the
>> error that you are currently seeing?  We've had so many posts that
>> I'm not certain that I understand the current situation.
>
> Same here :-)  I just know what I'm seeing.
>
> Again:
>
> Let A be a ticked (and thus, cached) article.  Let B be an unread
> article (that I have read before but then hit M-u on it).  Let C be
> an unseen article (displayed with `.').  Let these be the only
> articles that are shown after RET on the group.
>
> A is shown in gnus-summary-high-undownloaded-face, and the %O marker
> is -.
> B is shown in gnus-summary-normal-undownloaded-face, and the %O
> marker is -.
> C is shown in gnus-summary-normal-unread-face, and the %O marker is +.
>
> I'm not sure whether the %O marker for A is correct or not: OT1H, it
> *should* be downloaded, but OTOH, it's also cached.

For article A, this is correct (if ugly) behavior.  The agent and
cache are two distinct services so an article can be cached yet
undownloaded :(.

As for the rest, I'll keep looking.

Kevin

Re: Gnus lies about downloaded state

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> We've been on this thread for a week now.  Can you summarize the
> error that you are currently seeing?  We've had so many posts that
> I'm not certain that I understand the current situation.

Same here :-)  I just know what I'm seeing.

Again:

Let A be a ticked (and thus, cached) article.  Let B be an unread
article (that I have read before but then hit M-u on it).  Let C be
an unseen article (displayed with `.').  Let these be the only
articles that are shown after RET on the group.

A is shown in gnus-summary-high-undownloaded-face, and the %O marker
is -.
B is shown in gnus-summary-normal-undownloaded-face, and the %O
marker is -.
C is shown in gnus-summary-normal-unread-face, and the %O marker is +.

I'm not sure whether the %O marker for A is correct or not: OT1H, it
*should* be downloaded, but OTOH, it's also cached.

But I'm certain that the %O marker for B is wrong: I can `g' on the
article while unplugged.

The %O marker for C seems to be correct.

Now I read article C and then hit M-g.

A is shown as before, in gnus-summary-high-undownloaded-face with %O
being -.
B has changed.  It is shown in gnus-summary-normal-unread-face, and
the %O marker is now +.
Now C is not shown anymore, of course.

This behavior is reproducable.

> If the problem is in the selection of the face, you should check the
> value of gnus-summary-highlight.  It's value is expanded into a cond
> so the first case that matches determines the face.

I'm enclosing my value below, but I doubt that it is very relevant
because the +/- marker is also wrong when the face is wrong.

/----
| (((and
|    (< score default-low)
|    (= mark gnus-ancient-mark))
|   . gnus-summary-low-ancient-face)
|  ((and
|    (> score default-high)
|    (= mark gnus-ancient-mark))
|   . gnus-summary-high-ancient-face)
|  ((= mark gnus-ancient-mark)
|   . gnus-summary-normal-ancient-face)
|  ((= mark gnus-dormant-mark)
|   . kai-gnus-summary-dormant-face)
|  ((= mark gnus-expirable-mark)
|   . kai-gnus-summary-expirable-face)
|  ((eq mark gnus-canceled-mark)
|   . gnus-summary-cancelled-face)
|  ((and uncached
|        (> score default-high))
|   . gnus-summary-high-undownloaded-face)
|  ((and uncached
|        (< score default-low))
|   . gnus-summary-low-undownloaded-face)
|  (uncached . gnus-summary-normal-undownloaded-face)
|  ((and
|    (> score default-high)
|    (or
|     (eq mark gnus-dormant-mark)
|     (eq mark gnus-ticked-mark)))
|   . gnus-summary-high-ticked-face)
|  ((and
|    (< score default-low)
|    (or
|     (eq mark gnus-dormant-mark)
|     (eq mark gnus-ticked-mark)))
|   . gnus-summary-low-ticked-face)
|  ((or
|    (eq mark gnus-dormant-mark)
|    (eq mark gnus-ticked-mark))
|   . gnus-summary-normal-ticked-face)
|  ((and
|    (> score default-high)
|    (eq mark gnus-ancient-mark))
|   . gnus-summary-high-ancient-face)
|  ((and
|    (< score default-low)
|    (eq mark gnus-ancient-mark))
|   . gnus-summary-low-ancient-face)
|  ((eq mark gnus-ancient-mark)
|   . gnus-summary-normal-ancient-face)
|  ((and
|    (> score default-high)
|    (eq mark gnus-unread-mark))
|   . gnus-summary-high-unread-face)
|  ((and
|    (< score default-low)
|    (eq mark gnus-unread-mark))
|   . gnus-summary-low-unread-face)
|  ((eq mark gnus-unread-mark)
|   . gnus-summary-normal-unread-face)
|  ((> score default-high)
|   . gnus-summary-high-read-face)
|  ((< score default-low)
|   . gnus-summary-low-read-face)
|  (t . gnus-summary-normal-read-face))
\----


-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kai Großjohann]

Danny Siu <dsiu@adobe.com> writes:

> I still see the problem even with your patch which is already in CVS.  Kai,
> do you still experience the problem with the patch?

I'm running a version which has this entry first in lisp/ChangeLog:

2003-01-22  Teodor Zlatanov  <tzz@lifelogs.com>

	* gnus-sum.el (gnus-spam-mark): set to `$'

Just after that comes a change by Kevin -- is that the patch you're
talking about?

Anyhow, I still see the behavior I described on Saturday: if there
are new messages in a group, old but M-u'd articles show up as if
undownloaded (but I can look at them while unplugged, so they're
obviously downloaded), and after reading all the new articles, the
old M-u'd articles show up as downloaded again.
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kevin Greiner]

Danny Siu <dsiu@adobe.com> writes:

> Kevin Greiner writes:
>
>   Kevin> OK, have you applied my patch to gnus-sum?  That patch will not fix
>   Kevin> the face being displayed.  However, it WILL fix a major problem
>   Kevin> with updating the downloaded mark.  I'm reluctant to check-in this
>   Kevin> patch because it is in such a basic function.  My concern is that
>   Kevin> other routines may actually depend on the bug's existance :(.
>
> Kevin,
>
> I still see the problem even with your patch which is already in CVS.  Kai,
> do you still experience the problem with the patch?
>
> I am about do some debugging cuz it is really annoying to see my
> ticked/dormant articles show up as in undownloaded face.

Danny,
We've been on this thread for a week now.  Can you summarize the error
that you are currently seeing?  We've had so many posts that I'm not certain
that I understand the current situation.

If the problem is in the selection of the face, you should check the
value of gnus-summary-highlight.  It's value is expanded into a cond
so the first case that matches determines the face.

Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin Greiner writes:

  Kevin> OK, have you applied my patch to gnus-sum?  That patch will not fix
  Kevin> the face being displayed.  However, it WILL fix a major problem
  Kevin> with updating the downloaded mark.  I'm reluctant to check-in this
  Kevin> patch because it is in such a basic function.  My concern is that
  Kevin> other routines may actually depend on the bug's existance :(.

Kevin,

I still see the problem even with your patch which is already in CVS.  Kai,
do you still experience the problem with the patch?

I am about do some debugging cuz it is really annoying to see my
ticked/dormant articles show up as in undownloaded face.

-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I'm afraid I kinda lost track about the situation now.  I still see
> problems with the indication of the downloaded state, but I have more
> data.  In this group, I have some ticked (and thus, cached) articles,
> and there are some old articles that I have marked as unread with
> M-u.  And finally, there are unseen articles (marked with ".").  They
> are marked as downloaded because I did `J s' after `g'.
>
> In this case, the M-u'd articles are marked as not downloaded.
> Now I read the new message and then hit M-g in the group.
> Then the M-u'd messages are correctly marked as downloaded: I see
> the + for %O and the color is the default color.
>
> I get the feeling that this is not about corruption of the state that
> the agent keeps.  Instead, it seems that the summary buffer display
> interprets the data wrongly.  But it is somewhat strange that the
> presence of "." (unseen?) articles has effect on that interpretation.
>
> FWIW, ticked (and thus, cached) articles are also always shown in the
> "undownloaded" face and with %O=-.


OK, have you applied my patch to gnus-sum?  That patch will not fix
the face being displayed.  However, it WILL fix a major problem with
updating the downloaded mark.  I'm reluctant to check-in this patch
because it is in such a basic function.  My concern is that other
routines may actually depend on the bug's existance :(.

Kevin

Re: Gnus lies about downloaded state

[Kai Großjohann]

I'm afraid I kinda lost track about the situation now.  I still see
problems with the indication of the downloaded state, but I have more
data.  In this group, I have some ticked (and thus, cached) articles,
and there are some old articles that I have marked as unread with
M-u.  And finally, there are unseen articles (marked with ".").  They
are marked as downloaded because I did `J s' after `g'.

In this case, the M-u'd articles are marked as not downloaded.
Now I read the new message and then hit M-g in the group.
Then the M-u'd messages are correctly marked as downloaded: I see
the + for %O and the color is the default color.

I get the feeling that this is not about corruption of the state that
the agent keeps.  Instead, it seems that the summary buffer display
interprets the data wrongly.  But it is somewhat strange that the
presence of "." (unseen?) articles has effect on that interpretation.

FWIW, ticked (and thus, cached) articles are also always shown in the
"undownloaded" face and with %O=-.
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kevin Greiner]

Danny Siu <dsiu@adobe.com> writes:

> Kevin Greiner writes:
>
>   Kevin> Your screen shot shows that articles 17663 though 17718 have a (-)
>   Kevin> undownloaded flag AND are highlighted with the
>   Kevin> gnus-summary-normal-undownloaded-face.  Your format places the
>   Kevin> download mark (+/-) AFTER the article number (the (+) before the
>   Kevin> number is its score).
>
>   Kevin> Please verify that these articles have been fetched into the agent
>   Kevin> by excuting
>
>   Kevin> M-: (assq 17663 gnus-agent-article-alist)
>
> (17663 . 731230) 
>
>   Kevin> If it returns (17663), everything is working fine.  If it returns
>   Kevin> (17663 . #) then I can't explain this.
>
> Well, I look into the .agentview file and found that 17663 lies (btw 17354 .
> 17720) in the list which starts with 731230
>
> ( (731231 17721 . 17748) 
>   (nil 15 (39 . 90) (94 . 108) ...)
>   (731230 (6 . 14) .... (17354 . 17720)) )
>
> i don't know enough the rev 2 of .agentview format to understant what this
> means.  also, the file 17663 is in the agent dir.
>
> does the .agentview seem ok?

Absolutely.

That first number is the day on which the article was fetched.  The
remaining is a compressed sequence of article numbers sharing the same
day.

Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin Greiner writes:

  Kevin> Your screen shot shows that articles 17663 though 17718 have a (-)
  Kevin> undownloaded flag AND are highlighted with the
  Kevin> gnus-summary-normal-undownloaded-face.  Your format places the
  Kevin> download mark (+/-) AFTER the article number (the (+) before the
  Kevin> number is its score).

  Kevin> Please verify that these articles have been fetched into the agent
  Kevin> by excuting

  Kevin> M-: (assq 17663 gnus-agent-article-alist)

(17663 . 731230) 

  Kevin> If it returns (17663), everything is working fine.  If it returns
  Kevin> (17663 . #) then I can't explain this.

Well, I look into the .agentview file and found that 17663 lies (btw 17354 .
17720) in the list which starts with 731230

( (731231 17721 . 17748) 
  (nil 15 (39 . 90) (94 . 108) ...)
  (731230 (6 . 14) .... (17354 . 17720)) )

i don't know enough the rev 2 of .agentview format to understant what this
means.  also, the file 17663 is in the agent dir.

does the .agentview seem ok?

  Kevin> As for debugging, I'd debug in gnus-summary-insert-line.

  Kevin> You should be able to trace though it's setting gnus-tmp-downloaded
  Kevin> to either 32 (no-mark), 43 (undownloaded), or 45 (downloaded).

  Kevin> I do have a patch for gnus-sum.  You should be able to use
  Kevin> gnus-agent-fetch-selected-article after installing it.

Thanks!
-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kevin Greiner]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

Danny Siu <dsiu@adobe.com> writes:

> I have removed gnus-agent-fetch-selected-article from
> gnus-select-article-hook but the summary still highlights incorrectly the
> downloaded ones with gnus-summary-normal-undownloaded-face in few of my
> nnimap groups.  I tried removing the agent cache and gnus-agent-regenerate
> but that didn't help. 
>
> Attached screen shot.  here is my gnus-summary-line-format.
>
> (setq gnus-summary-line-format "%U%R%z%5N%O %(%8&user-date; %~(pad 2)~(max
> 2)~(ignore \"1\")t%[%5k:%* %-20,20n%] %B%s%)\n")
>
> where should i start debugging this problem?

Your screen shot shows that articles 17663 though 17718 have a (-)
undownloaded flag AND are highlighted with the
gnus-summary-normal-undownloaded-face.  Your format places the
download mark (+/-) AFTER the article number (the (+) before the
number is its score).

Please verify that these articles have been fetched into the agent by
excuting

M-: (assq 17663 gnus-agent-article-alist)

If it returns (17663), everything is working fine.  If it returns
(17663 . #) then I can't explain this.

As for debugging, I'd debug in gnus-summary-insert-line.

You should be able to trace though it's setting gnus-tmp-downloaded to
either 32 (no-mark), 43 (undownloaded), or 45 (downloaded).

I do have a patch for gnus-sum.  You should be able to use
gnus-agent-fetch-selected-article after installing it.


[-- Attachment #2: Patch to fix line position calculation. --]
[-- Type: text/plain, Size: 3597 bytes --]

--- ../lisp.cvs_ref/gnus-sum.el	Mon Jan 13 23:06:13 2003
+++ gnus-sum.el	Thu Jan 16 16:11:41 2003
@@ -4022,40 +4022,41 @@
     (gnus-summary-goto-subject article)
     (let* ((datal (gnus-data-find-list article))
 	   (data (car datal))
-	   (length (when (cdr datal)
-		     (- (gnus-data-pos data)
-			(gnus-data-pos (cadr datal)))))
 	   (buffer-read-only nil)
 	   (level (gnus-summary-thread-level)))
       (gnus-delete-line)
-      (gnus-summary-insert-line
-       header level nil 
-       (memq article gnus-newsgroup-undownloaded)
-       (gnus-article-mark article)
-       (memq article gnus-newsgroup-replied)
-       (memq article gnus-newsgroup-expirable)
-       ;; Only insert the Subject string when it's different
-       ;; from the previous Subject string.
-       (if (and
-	    gnus-show-threads
-	    (gnus-subject-equal
-	     (condition-case ()
-		 (mail-header-subject
-		  (gnus-data-header
-		   (cadr
-		    (gnus-data-find-list
-		     article
-		     (gnus-data-list t)))))
-	       ;; Error on the side of excessive subjects.
-	       (error ""))
-	     (mail-header-subject header)))
-	   ""
-	 (mail-header-subject header))
-       nil (cdr (assq article gnus-newsgroup-scored))
-       (memq article gnus-newsgroup-processable))
-      (when length
-	(gnus-data-update-list
-	 (cdr datal) (- length (- (gnus-data-pos data) (point))))))))
+      (let ((inserted (- (point)
+                         (progn
+                           (gnus-summary-insert-line
+                            header level nil 
+                            (memq article gnus-newsgroup-undownloaded)
+                            (gnus-article-mark article)
+                            (memq article gnus-newsgroup-replied)
+                            (memq article gnus-newsgroup-expirable)
+                            ;; Only insert the Subject string when it's different
+                            ;; from the previous Subject string.
+                            (if (and
+                                 gnus-show-threads
+                                 (gnus-subject-equal
+                                  (condition-case ()
+                                      (mail-header-subject
+                                       (gnus-data-header
+                                        (cadr
+                                         (gnus-data-find-list
+                                          article
+                                          (gnus-data-list t)))))
+                                    ;; Error on the side of excessive subjects.
+                                    (error ""))
+                                  (mail-header-subject header)))
+                                ""
+                              (mail-header-subject header))
+                            nil (cdr (assq article gnus-newsgroup-scored))
+                            (memq article gnus-newsgroup-processable))
+                           (point)))))
+        (when (cdr datal)
+          (gnus-data-update-list
+           (cdr datal) 
+           (- (gnus-data-pos data) (gnus-data-pos (cadr datal)) inserted)))))))
 
 (defun gnus-summary-update-article (article &optional iheader)
   "Update ARTICLE in the summary buffer."
@@ -6776,7 +6777,7 @@
       (gnus-summary-goto-subject article t)))
   (gnus-summary-limit (append articles gnus-newsgroup-limit))
   (gnus-summary-position-point))
-  
+ 
 (defun gnus-summary-goto-subject (article &optional force silent)
   "Go the subject line of ARTICLE.
 If FORCE, also allow jumping to articles not currently shown."

[-- Attachment #3: Type: text/plain, Size: 9 bytes --]




Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

Kevin Greiner writes:

  Kevin> Let's go with that assumption for now.  Try removing
  Kevin> gnus-agent-fetch-selected-article from the hook then see if you run
  Kevin> into any problems.  You can use either 'J s' to fetch by process
  Kevin> mark or 'J S' to fetch by downloadable mark.

I have removed gnus-agent-fetch-selected-article from
gnus-select-article-hook but the summary still highlights incorrectly the
downloaded ones with gnus-summary-normal-undownloaded-face in few of my
nnimap groups.  I tried removing the agent cache and gnus-agent-regenerate
but that didn't help. 

Attached screen shot.  here is my gnus-summary-line-format.

(setq gnus-summary-line-format "%U%R%z%5N%O %(%8&user-date; %~(pad 2)~(max
2)~(ignore \"1\")t%[%5k:%* %-20,20n%] %B%s%)\n")

where should i start debugging this problem?
-- 
Danny Siu

[-- Attachment #2: undownload.png --]
[-- Type: image/png, Size: 81199 bytes --]

Re: Gnus lies about downloaded state

[Kevin Greiner]

Danny Siu <dsiu@adobe.com> writes:

> Kevin,
>
> I have installed your patch and it correctly signaled ("Format Spec doesn't
> match argument type") when the summary buffer lies about the downloaded
> state.  So is gnus-agent-fetch-selected-article the curpit?
>

Let's go with that assumption for now.  Try removing
gnus-agent-fetch-selected-article from the hook then see if you run
into any problems.  You can use either 'J s' to fetch by process mark
or 'J S' to fetch by downloadable mark.

Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin,

I have installed your patch and it correctly signaled ("Format Spec doesn't
match argument type") when the summary buffer lies about the downloaded
state.  So is gnus-agent-fetch-selected-article the curpit?

I am using the latest CVS as of "01/15/2002 11:49pm -0800"

Kevin Greiner writes:

  Kevin> Kai, You're going to be particularly interested in this one.  I
  Kevin> just proved, to my own satisfaction as least, that the
  Kevin> gnus-summary-update-article-line call in
  Kevin> gnus-agent-fetch-selected-article is corrupting the summary headers
  Kevin> that are used to layout the summary.

  Kevin> I've attached a patch to introduce my diagnostic code.  You'll
  Kevin> probably have to customize it (there is a constant one in the code
  Kevin> that should be a parameter) to say that all is OK when the summary
  Kevin> first opens.  I've bound the test to 'z' so it is simple to
  Kevin> execute.

  Kevin> I also call the test code from gnus-agent-fetch-selected-article.

  Kevin> On my installation, the first call to
  Kevin> gnus-summary-update-article-line in
  Kevin> gnus-agent-fetch-selected-article shifted the pos of each following
  Kevin> article by one character.  If each subsequent call has a similar
  Kevin> effect, ...

  Kevin> Kevin


-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kevin Greiner]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> 1) Open one of the problem groups.
>
> nntp+quimby:gnus.ding
>
>> 2) Identify the article number of an article that is marked as
>>    undownloaded yet is stored in the agent.
>
> 45110
>
>> 3) Do this: m-: (memq <art-num> gnus-newsgroup-undownloaded)
>>    Do you get nil or a list?
>
> (memq 45110 gnus-newsgroup-undownloaded)
> => (45110)
>
>> 4) Now do c-h v gnus-agent-article-alist.
>> 5) Find <art-num> in the display.
>> 6) Is <art-num> in a cons that looks like (<art-num>) or (<art-num> . #)?
>
> (assoc 45110 gnus-agent-article-alist)
> => (45110 . 731230)
>
> So the character is wrong even though the data is right.  That goes
> well with my experience: these articles behave as if downloaded, even
> though %O says they're not.
>
> FWIW, the message 45110 is the grandparent of your posting that I'm
> replying to (ie, it's the great-grandparent of this message I'm
> typing).  I've read that message before, it was marked R or O in the
> summary buffer.


Kai, 
You're going to be particularly interested in this one.  I just
proved, to my own satisfaction as least, that the
gnus-summary-update-article-line call in
gnus-agent-fetch-selected-article is corrupting the summary headers
that are used to layout the summary.

I've attached a patch to introduce my diagnostic code.  You'll
probably have to customize it (there is a constant one in the code
that should be a parameter) to say that all is OK when the summary
first opens.  I've bound the test to 'z' so it is simple to execute.

I also call the test code from gnus-agent-fetch-selected-article.

On my installation, the first call to gnus-summary-update-article-line
in gnus-agent-fetch-selected-article shifted the pos of each following
article by one character.  If each subsequent call has a similar
effect, ...

Kevin

[-- Attachment #2: A patch to debug gnus-sum and gnus-agent. --]
[-- Type: text/plain, Size: 2979 bytes --]

--- ../lisp.cvs_ref/gnus-agent.el	Wed Jan 15 21:41:57 2003
+++ gnus-agent.el	Wed Jan 15 22:18:17 2003
@@ -847,16 +847,19 @@
   "Fetch the current article as it is selected.
 This can be added to `gnus-select-article-hook' or
 `gnus-mark-article-hook'."
+  (kjg-gnus-summary-test "entering gnus-agent-fetch-selected-article")
   (let ((gnus-command-method gnus-current-select-method))
     (when (and gnus-plugged (gnus-agent-method-p gnus-command-method))
       (when (gnus-agent-fetch-articles
              gnus-newsgroup-name
 	     (list gnus-current-article))
+        (kjg-gnus-summary-test "gnus-agent-fetch-selected-article (after fetch)")
 	(setq gnus-newsgroup-undownloaded
 	      (delq gnus-current-article gnus-newsgroup-undownloaded))
 	(gnus-summary-update-article-line
 	 gnus-current-article
-	 (gnus-summary-article-header gnus-current-article))))))
+	 (gnus-summary-article-header gnus-current-article))
+        (kjg-gnus-summary-test "gnus-agent-fetch-selected-article (after update)")))))
 
 ;;;
 ;;; Internal functions
@@ -2091,7 +2094,6 @@
                                         day
                                       (let (found
                                             (days gnus-agent-expire-days))
-                                        (debug)
                                         (catch 'found
                                           (while (and (not found)
                                                       days)
*** ../lisp.cvs_ref/gnus-sum.el	Mon Jan 13 23:06:13 2003
--- gnus-sum.el	Wed Jan 15 22:21:49 2003
***************
*** 1526,1531 ****
--- 1526,1532 ----
  ;; Non-orthogonal keys
  
  (gnus-define-keys gnus-summary-mode-map
+   "z" kjg-gnus-summary-test
    " " gnus-summary-next-page
    "\177" gnus-summary-prev-page
    [delete] gnus-summary-prev-page
***************
*** 6776,6781 ****
--- 6777,6804 ----
        (gnus-summary-goto-subject article t)))
    (gnus-summary-limit (append articles gnus-newsgroup-limit))
    (gnus-summary-position-point))
+ 
+ (defun kjg-gnus-summary-test (&optional prefix)
+   (interactive)
+   (let ((bug nil))
+     (save-excursion
+       (goto-char (point-min))
+       (let ((b (get-buffer "*scratch*")))
+         (while (looking-at ".....\\([0-9]+\\)")
+           (let* ((art (string-to-int (match-string 1)))
+                  (data (gnus-data-find art))
+                  (pt (gnus-data-pos data))
+                  (pnt (1+ (point))))
+             (unless (eq pt pnt)
+               (let ((msg (format "Art#: %d pt: %d point: %d" art pt pnt)))
+                 (when (and prefix (not bug))
+                   (princ prefix b)
+                   (princ "\n" b))
+                 (setq bug t)
+                 (princ msg b)
+                 (princ "\n" b)))
+             (forward-line 1)))))
+     (gnus-message 1 (if bug "Problem" "All OK"))))
    
  (defun gnus-summary-goto-subject (article &optional force silent)
    "Go the subject line of ARTICLE.

Re: Gnus lies about downloaded state

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> 1) Open one of the problem groups.
>
> nntp+quimby:gnus.ding
>
>> 2) Identify the article number of an article that is marked as
>>    undownloaded yet is stored in the agent.
>
> 45110
>
>> 3) Do this: m-: (memq <art-num> gnus-newsgroup-undownloaded)
>>    Do you get nil or a list?
>
> (memq 45110 gnus-newsgroup-undownloaded)
> => (45110)
>
>> 4) Now do c-h v gnus-agent-article-alist.
>> 5) Find <art-num> in the display.
>> 6) Is <art-num> in a cons that looks like (<art-num>) or (<art-num> . #)?
>
> (assoc 45110 gnus-agent-article-alist)
> => (45110 . 731230)
>
> So the character is wrong even though the data is right.  That goes
> well with my experience: these articles behave as if downloaded, even
> though %O says they're not.
>
> FWIW, the message 45110 is the grandparent of your posting that I'm
> replying to (ie, it's the great-grandparent of this message I'm
> typing).  I've read that message before, it was marked R or O in the
> summary buffer.


Well, the good news is that you and Danny are reporting consistent
results.  The bad news is that we've narrowed the problem to gnus-sum!

What value do you have for gnus-summary-mark-positions?  (It's buffer
local to the summary buffer.)

You should see (download . #) where # is the column number where you
expect to see the +/- download mark.

If that looks right, try adding this statement to the beginning of
gnus-summary-insert-line

(princ (format "%s  %s\n" gnus-tmp-current undownloaded (get-buffer "**scratch**")))

This will print the article number and download status (downloaded ==>
nil / undownloaded ==> t) to the scratch buffer.

After displaying the summary, check scratch to see if the download
flag matches what is displayed.  By your descriptions it should not
match.

Pls send me the value of your gnus-format-specs.

Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

Kevin Greiner writes:

  Kevin> 1) Open one of the problem groups.

  Kevin> 2) Identify the article number of an article that is marked as
  Kevin>    undownloaded yet is stored in the agent.

  Kevin> 3) Do this: m-: (memq <art-num> gnus-newsgroup-undownloaded)
  Kevin>    Do you get nil or a list?

a list.

  Kevin> 4) Now do c-h v gnus-agent-article-alist.

  Kevin> 5) Find <art-num> in the display.

  Kevin> 6) Is <art-num> in a cons that looks like (<art-num>) or (<art-num>
  Kevin>    . #)?

the latter. (17582 . 731230)

  Kevin> Kevin

-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> 1) Open one of the problem groups.

nntp+quimby:gnus.ding

> 2) Identify the article number of an article that is marked as
>    undownloaded yet is stored in the agent.

45110

> 3) Do this: m-: (memq <art-num> gnus-newsgroup-undownloaded)
>    Do you get nil or a list?

(memq 45110 gnus-newsgroup-undownloaded)
=> (45110)

> 4) Now do c-h v gnus-agent-article-alist.
> 5) Find <art-num> in the display.
> 6) Is <art-num> in a cons that looks like (<art-num>) or (<art-num> . #)?

(assoc 45110 gnus-agent-article-alist)
=> (45110 . 731230)

So the character is wrong even though the data is right.  That goes
well with my experience: these articles behave as if downloaded, even
though %O says they're not.

FWIW, the message 45110 is the grandparent of your posting that I'm
replying to (ie, it's the great-grandparent of this message I'm
typing).  I've read that message before, it was marked R or O in the
summary buffer.
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I think I've noticed that selecting an article, then M-u on it,
> causes that article to be displayed with %O = "-" when selecting the
> group later on.
>
> Note: I have gnus-use-cache = t, and ticked (and thus, cached)
> articles appear with %O = "-", as well.
>
> Does this help?

Actually, I'm a little confused.  Let's see if we can isolate this
problem a little.

1) Open one of the problem groups.
2) Identify the article number of an article that is marked as
   undownloaded yet is stored in the agent.
3) Do this: m-: (memq <art-num> gnus-newsgroup-undownloaded)
   Do you get nil or a list?
4) Now do c-h v gnus-agent-article-alist.
5) Find <art-num> in the display.
6) Is <art-num> in a cons that looks like (<art-num>) or (<art-num> . #)?

Kevin

Re: Gnus lies about downloaded state

[Kai Großjohann]

I think I've noticed that selecting an article, then M-u on it,
causes that article to be displayed with %O = "-" when selecting the
group later on.

Note: I have gnus-use-cache = t, and ticked (and thus, cached)
articles appear with %O = "-", as well.

Does this help?
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Danny Siu]

More info: the problem doesn't occur in all the groups; i only have 2 or 3
groups that i found the download state inconsistent.  those group tends to
have large number of unread articles but i can't find a pattern of which the
problem would exhibit.  

Also, I have tried gnus-agent-regenerate and then 'J s' but still doesn't
help.  Furthermore, I removed the agent directory for the group all together
and did 'J s'.  No luck either.

I start to think whether it is the agent problem OR the summary
hightlighting bug?

Kevin Greiner writes:

  Kevin> OK OK.  I can be hard headed at times but this isn't one of them.

  Kevin> I believe that I found this myself just the other night.  In any
  Kevin> case, I've been unable to reproduce this problem with the 6.131
  Kevin> version of gnus-agent. Pls download it then repeat your tests.

  Kevin> VERY IMPORTANT: Run gnus-agent-regenerate to repair your groups
  Kevin> first!!  You've already downloaded a number of articles, no need to
  Kevin> download them again.

  Kevin> Here's what I tried.
  Kevin> 1) Opened gnus.ding
  Kevin> 2) Set the downloadable mark on 100 articles (A random mix of
  Kevin>    headers with about 70 already having their article's fetched
  Kevin>    into the agent).
  Kevin> 3) Closed gnus.ding
  Kevin> 4) Executed fetch session ('J s') in the group buffer.
  Kevin> 5) Opened gnus.ding; At this point, all of the downloadable marks
  Kevin>    were gone and all articles displayed with a downloaded(+) mark.

  Kevin> Kevin


-- 
Danny Siu

Re: Gnus lies about downloaded state

[Kai Großjohann]

Yes, I just opened a group which had a new, fresh, unread, article
that was marked "-" (and also in a greenish color).
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kai Großjohann]

Now I checked again: I'm sitting in the train at the moment, and Gnus
was started in the train.  So there was no way for me to fetch
anything into the agent.

Yet, some messages are marked - and are downloaded.  (They're not
cached.)  I this group I only see old (O) articles that are like
this, I'd have to check whether I can find new ones that are marked
wrongly.
-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kai Großjohann]

kgreiner@xpediantsolutions.com writes:

> The 'J S' command in the summary buffer was not refreshing the buffer.
> You can only execute 'J S' when plugged so I'm not sure if it has any
> bearing on your problem.

I'm almost sure I used `J s' from the group buffer.  But I'm not 100%
sure, so I'll check again.

-- 
Ambibibentists unite!

Re: Gnus lies about downloaded state

[Kevin Greiner]

Danny Siu <dsiu@adobe.com> writes:

> i am seeing the same thing.  i just did 'J s' and am sure all articles are
> downloaded but the summary buffer still highlights those as undownloaded.
>
> it didn't happen yesterday.  i am using the latest from cvs.
>
> screen shot attached.
>
> Kai Großjohann writes:
>
>   Kai> Look at this screenshot.  The "-" in the leftmost column means the
>   Kai> article isn't downloaded, but Gnus is unplugged, and I can see the
>   Kai> message!
>
>   Kai> What's the story?
>
>   Kai> I think this happened only recently.  -- Ambibibentists unite!
>

OK OK.  I can be hard headed at times but this isn't one of them.

I believe that I found this myself just the other night.  In any case,
I've been unable to reproduce this problem with the 6.131 version of
gnus-agent. Pls download it then repeat your tests.

VERY IMPORTANT: 
Run gnus-agent-regenerate to repair your groups first!!  You've
already downloaded a number of articles, no need to download them
again.

Here's what I tried.
1) Opened gnus.ding
2) Set the downloadable mark on 100 articles (A random mix of headers
   with about 70 already having their article's fetched into the
   agent).
3) Closed gnus.ding
4) Executed fetch session ('J s') in the group buffer.
5) Opened gnus.ding; At this point, all of the downloadable marks were
   gone and all articles displayed with a downloaded(+) mark.

Kevin

Re: Gnus lies about downloaded state

[Danny Siu]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]


i am seeing the same thing.  i just did 'J s' and am sure all articles are
downloaded but the summary buffer still highlights those as undownloaded.

it didn't happen yesterday.  i am using the latest from cvs.

screen shot attached.

Kai Großjohann writes:

  Kai> Look at this screenshot.  The "-" in the leftmost column means the
  Kai> article isn't downloaded, but Gnus is unplugged, and I can see the
  Kai> message!

  Kai> What's the story?

  Kai> I think this happened only recently.  -- Ambibibentists unite!

-- 
Danny Siu

[-- Attachment #2: undownload.png --]
[-- Type: image/png, Size: 85409 bytes --]

Re: Gnus lies about downloaded state

[kgreiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Look at this screenshot.  The "-" in the leftmost column means the
> article isn't downloaded, but Gnus is unplugged, and I can see the
> message!
>
> What's the story?
>
> I think this happened only recently.

The 'J S' command in the summary buffer was not refreshing the buffer.
You can only execute 'J S' when plugged so I'm not sure if it has any
bearing on your problem.

Kevin

Gnus lies about downloaded state

[Kai Großjohann]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Look at this screenshot.  The "-" in the leftmost column means the
article isn't downloaded, but Gnus is unplugged, and I can see the
message!

What's the story?

I think this happened only recently.
-- 
Ambibibentists unite!


[-- Attachment #2: screenshot --]
[-- Type: image/png, Size: 33285 bytes --]