gnus-agent showing wrong data: some investigations

gnus-agent showing wrong data: some investigations

[Kai Großjohann]

[-- Attachment #1: Type: text/plain, Size: 65 bytes --]

I entered gnus.ding and saw the summary buffer you can see here:

[-- Attachment #2: ss.png --]
[-- Type: image/png, Size: 16014 bytes --]

[-- Attachment #3: Type: text/plain, Size: 4122 bytes --]


Look at the message from Karl Pflästerer where point is.  It is
message 46119.  It is marked with "-" in the first column, ie it is
marked as undownloaded.  I guess this is because
gnus-agent-get-undownloaded-list returns it as undownloaded:

Debugger entered--returning value: (46058 46067 46073 46107 46116 42510 42511 42517 42523 42527 42528 42529 42530 42531 42533 42535 42540 42579 42586 42601 42602 42608 42610 42614 42616 42618 42627 42639 42647 42648 42652 42653 42654 42745 42749 42774 42848 42853 42854 42856 42859 42860 42861 42862 42866 42868 42869 42870 42878 42885 42886 42899 42914 42915 42971 43013 43017 43111 43114 43167 43169 43211 43224 43225 43272 43273 43644 43696 43718 43726 43752 43754 43755 43761 43762 43857 44057 44125 44126 44139 44144 44147 44154 44156 44159 45715 46111 46119)
  gnus-agent-get-undownloaded-list()
  (progn (gnus-agent-get-undownloaded-list))
  (if gnus-agent (progn (gnus-agent-get-undownloaded-list)))
  (when gnus-agent (gnus-agent-get-undownloaded-list))
  (cond ((null articles) (quote quit)) ((eq articles 0) nil) (t (setq gnus-newsgroup-dependencies ...) (gnus-set-global-variables) (setq gnus-newsgroup-headers ...) (when cached ...) (when gnus-suppress-duplicates ...) (setq gnus-newsgroup-limit ...) (setq fetched-articles ...) (setq gnus-newsgroup-articles fetched-articles) (setq gnus-newsgroup-unreads ...) (gnus-compute-unseen-list) (gnus-update-missing-marks ...) (when ... ...) (when gnus-agent ...) (when gnus-list-identifiers ...) (setq gnus-newsgroup-auto-expire ...) (unless gnus-single-article-buffer ...) (when gnus-newsgroup-headers ...) (or gnus-newsgroup-headers t)))
  (let* ((entry ...) (gnus-summary-ignore-duplicates ...) (info ...) articles fetched-articles cached) (unless (gnus-check-server ...) (error "Couldn't open server")) (or (and entry ...) (gnus-activate-group group) (progn ... ...)) (unless (gnus-request-group group t) (when ... ...) (error "Couldn't request group %s: %s" group ...)) (setq gnus-newsgroup-name group gnus-newsgroup-unselected nil gnus-newsgroup-unreads (gnus-list-of-unread-articles group)) (let (...) (setq gnus-newsgroup-display ...)) (gnus-summary-setup-default-charset) (when (gnus-virtual-group-p group) (setq cached gnus-newsgroup-cached)) (setq gnus-newsgroup-unreads (gnus-sorted-ndifference ... gnus-newsgroup-dormant)) (setq gnus-newsgroup-processable nil) (gnus-update-read-articles group gnus-newsgroup-unreads) (when info (gnus-adjust-marked-articles info)) (if (setq articles select-articles) (setq gnus-newsgroup-unselected ...) (setq articles ...)) (cond (... ...) (... nil) (t ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...)))
  gnus-select-newsgroup("nntp+quimby:gnus.ding" nil nil)
  gnus-summary-read-group-1("nntp+quimby:gnus.ding" nil t nil nil nil)
  gnus-summary-read-group("nntp+quimby:gnus.ding" nil t nil nil nil nil)
  gnus-group-read-group(nil t)
  gnus-group-select-group(nil)
  gnus-topic-select-group(nil)
  call-interactively(gnus-topic-select-group)

The .agentview file in that directory starts out like this:

((731268 46118 . 46123) (731267 46090 . 46117) [...deletia...]

So I think that the .agentview file thinks it has been downloaded.

The .overview file has an entry for that article, which looks okay:

46119	Re: Proposed patch	sigurd@12move.de (Karl =?iso-8859-1?q?Pfl=E4sterer?=)	Fri, 21 Feb 2003 22:58:07 +0100	<m3el61tiw2.fsf@hamster.pflaesterer.de>	<m3znoskl7d.fsf@hamster.pflaesterer.de> <84isvfe05f.fsf@lucy.is.informatik.uni-duisburg.de> <m3smujums4.fsf@hamster.pflaesterer.de> <m3of55tt40.fsf@hamster.pflaesterer.de> <84d6lls5gk.fsf@lucy.is.informatik.uni-duisburg.de>	4124	38	Xref: quimby.gnus.org gnus.ding:46119

I also ran (gnus-agent-check-overview-buffer) on that buffer, and it
appears to be sorted.

I *think* that this means that gnus-agent-get-undownloaded-list has a
bug.  But I'm not sure.  What do you think about the above evidence?

(gnus-agent-get-undownloaded-list has complex logic, I'm having
trouble to grok it.)
-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

Additional data point: gnus-agent-read-agentview appears to return
the right data.

ELISP> (assoc 46119 (let ((gnus-agent-read-agentview "nntp+quimby:gnus.ding"))
         (gnus-agent-read-agentview "~/News/agent/nntp/quimby/gnus/ding/.agentview")))
(46119 . 731268)

-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I *think* that this means that gnus-agent-get-undownloaded-list has a
> bug.  But I'm not sure.  What do you think about the above evidence?

It turns out that gnus-newsgroup-headers is not sorted when
gnus-agent-get-undownloaded-list is called: the first entry is for
46120 (which is larger than 46119), and after that come the lower
entries.

What can I do to prevent gnus-newsgroup-headers from being unsorted?
-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> I *think* that this means that gnus-agent-get-undownloaded-list has a
>> bug.  But I'm not sure.  What do you think about the above evidence?
>
> It turns out that gnus-newsgroup-headers is not sorted when
> gnus-agent-get-undownloaded-list is called: the first entry is for
> 46120 (which is larger than 46119), and after that come the lower
> entries.
>
> What can I do to prevent gnus-newsgroup-headers from being unsorted?

I've now changed gnus-agent-get-undownloaded-list to sort
gnus-newsgroup-headers, and now it works!
-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>>
>>> I *think* that this means that gnus-agent-get-undownloaded-list has a
>>> bug.  But I'm not sure.  What do you think about the above evidence?
>>
>> It turns out that gnus-newsgroup-headers is not sorted when
>> gnus-agent-get-undownloaded-list is called: the first entry is for
>> 46120 (which is larger than 46119), and after that come the lower
>> entries.
>>
>> What can I do to prevent gnus-newsgroup-headers from being unsorted?
>
> I've now changed gnus-agent-get-undownloaded-list to sort
> gnus-newsgroup-headers, and now it works!

Fabulous news.

I've added a sort to my copy.  I should have it checked-in this
weekend unless you beat me to it.

Kevin

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> I've added a sort to my copy.  I should have it checked-in this
> weekend unless you beat me to it.

I didn't commit it because I think it isn't right.  It would be
doctoring the symptoms instead of curing the disease.  At least
that's the way I understand it.

OTOH, I've been looking at where gnus-newsgroup-headers comes from,
and it could indeed come from a non-sorted input.  Hm.  (For example,
if gnus-select-newsgroup is called with some articles to select, then
that list of articles could be non-sorted.)  Hmmm...  An uneasy
feeling remains.  I'm not sure what to do, or what to think.

Anyone?
-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> I've added a sort to my copy.  I should have it checked-in this
>> weekend unless you beat me to it.
>
> I didn't commit it because I think it isn't right.  It would be
> doctoring the symptoms instead of curing the disease.  At least
> that's the way I understand it.

The problem is that gnus-newsgroup-headers IS SORTED, just not in
ascending order.  If I remember right, it matches the order of the
articles in the summary buffer.

> OTOH, I've been looking at where gnus-newsgroup-headers comes from,
> and it could indeed come from a non-sorted input.  Hm.  (For example,
> if gnus-select-newsgroup is called with some articles to select, then
> that list of articles could be non-sorted.)  Hmmm...  An uneasy
> feeling remains.  I'm not sure what to do, or what to think.
>

Here's what I'm currently trying.

(defun gnus-agent-get-undownloaded-list ()
  "Construct list of articles that have not been downloaded."
  (let ((gnus-command-method (gnus-find-method-for-group gnus-newsgroup-name)))
    (when (set (make-local-variable 'gnus-newsgroup-agentized) (gnus-agent-method-p gnus-command-method))
      (let* ((alist (gnus-agent-load-alist gnus-newsgroup-name))
             (headers (sort (mapcar (lambda (h) (mail-header-number h)) gnus-newsgroup-headers) '<))
             (undownloaded (list nil))
             (tail-undownloaded undownloaded)
             (unfetched (list nil))
             (tail-unfetched unfetched))
        (while (and alist headers)
          (let ((a (caar alist))
                (h (car headers)))
            (cond ((< a h)
                   ;; Ignore IDs in the alist that are not being
                   ;; displayed in the summary.
                   (pop alist))
                  ((> a h)
                   ;; Headers that are not in the alist should be
                   ;; fictious (see nnagent-retrieve-headers); they
                   ;; imply that this article isn't in the agent.
                   (gnus-agent-append-to-list tail-undownloaded h)
                   (gnus-agent-append-to-list tail-unfetched    h)
                   (pop headers)) 
                  ((cdar alist)
                   (pop alist)
                   (pop headers)
                   nil                  ; ignore already downloaded
                   )
                  (t
                   (pop alist)
                   (pop headers)
                   (gnus-agent-append-to-list tail-undownloaded a)))))

        (while headers
          (let ((num (pop headers)))
            (gnus-agent-append-to-list tail-undownloaded num)
            (gnus-agent-append-to-list tail-unfetched    num)))

        (setq gnus-newsgroup-undownloaded (cdr undownloaded)
              gnus-newsgroup-unfetched    (cdr unfetched))))))

Kevin

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> I've added a sort to my copy.  I should have it checked-in this
>>> weekend unless you beat me to it.
>>
>> I didn't commit it because I think it isn't right.  It would be
>> doctoring the symptoms instead of curing the disease.  At least
>> that's the way I understand it.
>
> The problem is that gnus-newsgroup-headers IS SORTED, just not in
> ascending order.  If I remember right, it matches the order of the
> articles in the summary buffer.

Yes, that's what I thought after looking and thinking some more.

>> OTOH, I've been looking at where gnus-newsgroup-headers comes from,
>> and it could indeed come from a non-sorted input.  Hm.  (For example,
>> if gnus-select-newsgroup is called with some articles to select, then
>> that list of articles could be non-sorted.)  Hmmm...  An uneasy
>> feeling remains.  I'm not sure what to do, or what to think.
>
> Here's what I'm currently trying.

It's better than what I just committed, I think.  Sorry for the bad
commit.  It was me getting too impatient again.  I think it would be
good if you committed yours.
-- 
A preposition is not a good thing to end a sentence with.

Re: gnus-agent showing wrong data: some investigations

[Kevin Greiner]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>>
>>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>>
>>>> I've added a sort to my copy.  I should have it checked-in this
>>>> weekend unless you beat me to it.
>>>
>>> I didn't commit it because I think it isn't right.  It would be
>>> doctoring the symptoms instead of curing the disease.  At least
>>> that's the way I understand it.
>>
>> The problem is that gnus-newsgroup-headers IS SORTED, just not in
>> ascending order.  If I remember right, it matches the order of the
>> articles in the summary buffer.
>
> Yes, that's what I thought after looking and thinking some more.
>
>>> OTOH, I've been looking at where gnus-newsgroup-headers comes from,
>>> and it could indeed come from a non-sorted input.  Hm.  (For example,
>>> if gnus-select-newsgroup is called with some articles to select, then
>>> that list of articles could be non-sorted.)  Hmmm...  An uneasy
>>> feeling remains.  I'm not sure what to do, or what to think.
>>
>> Here's what I'm currently trying.
>
> It's better than what I just committed, I think.  Sorry for the bad
> commit.  It was me getting too impatient again.  I think it would be
> good if you committed yours.

Hey, no problem at all. You've been reporting a problem for weeks.
I'm hoping that this fixes it.

I've checked-in a new copy of gnus-agent.  Let me know if you run into
any problems.  This copy had minimal testing as I built it just for
the commit.

Kevin

Re: gnus-agent showing wrong data: some investigations

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> I've checked-in a new copy of gnus-agent.  Let me know if you run into
> any problems.  This copy had minimal testing as I built it just for
> the commit.

Seems to be fine.  Thanks!
-- 
A preposition is not a good thing to end a sentence with.