two agent nits

two agent nits

[Simon Josefsson]

* I can't seem to get the agent to fetch read articles.
  gnus-agent-consider-all-articles's value is t
  The default agent category predicate is 'true'.
  `J s' and `J u' just download unread or ticked articles.

* The custom type of gnus-agent-expire-days is integer.  What if I
  don't want articles to be expired?  Nil?  0?  Either way, it should
  be documented and the custom type improved.

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> * I can't seem to get the agent to fetch read articles.
>   gnus-agent-consider-all-articles's value is t
>   The default agent category predicate is 'true'.
>   `J s' and `J u' just download unread or ticked articles.

One partial solution to above was posted here recently to similar
query of mine.  It forces gnus to download anything you read.

Could have repercussions in groups where you don't want that but I
suspect it could be put into Group Parameters or agent-parameters.
(I don't know how myself)

This is a partial repost of Kevin G's  (hope he doesn't mind) from
message-id:
    Message-ID: <u7k1nvdzb.fsf@xpediantsolutions.com>
======================================================================
>> HP Harry Putnam <reader@newsguy.com> writes:

>> HP Whats the trick way to sort a large summary buffer to display only
>> HP undownloaded messages?  I'm set up to display + for downloaded and -
>> HP for undownloaded.  No faces confusion.  Seemed at least a chance that
>> HP `/ m - <RET>' would collect those marked with minus sign but it doesn't
>> HP work that way.

> KG People used to complain that the download mark overrode the other
> KG marks so it was separated out.  I suppose that a
> KG gnus-summary-limit-to-undownloaded could be written, there's just not
> KG been any demand for it.

> KG This isn't meant to be a solution, just an explanation.

>> HP I find that I often have a few undownloaded messages mixed in with
>> HP many downloaded ones.  Due to my reading habits no doubt.  Even
>> HP though the category predicate is `true'.  I rarely go unplugged so
>> HP sometimes read messages before they are downloaded which I think is
>> HP what causes the random undownloaded messages.

> KG I may have a solution for this one.  Add
> KG gnus-agent-fetch-selected-article to the gnus-select-article-hook.
> KG This will result in undownloaded messages being downloaded to the
> KG agent as you read them.

>> HP  Anyway, how can I sort a few dozen undownloaded into a display from a
>> HP  summary of buffer of several thousand?

> KG How about an alternative solution?

> KG Use 'M-^ 1 0 0 0 0 J #' to mark all articles for fetching then use
> KG 'J S' to fetch them.  The nice part is that the agent will only fetch
> KG those articles that have not already been fetched.
> KG 
> KG Kevin
========================================================================

Re: two agent nits

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> * I can't seem to get the agent to fetch read articles.
>>   gnus-agent-consider-all-articles's value is t
>>   The default agent category predicate is 'true'.
>>   `J s' and `J u' just download unread or ticked articles.
>
> One partial solution to above was posted here recently to similar
> query of mine.  It forces gnus to download anything you read.
>
> Could have repercussions in groups where you don't want that but I
> suspect it could be put into Group Parameters or agent-parameters.
> (I don't know how myself)

Yes, I already use this, but re-reading all articles in all groups
doesn't seem like the best way to get articles into the agent.  I
think my current setting should work.

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:


>>>   gnus-agent-consider-all-articles's value is t
[...]
> Yes, I already use this, but re-reading all articles in all groups
> doesn't seem like the best way to get articles into the agent.  I
> think my current setting should work.

Looking at that variable, it does seem to qualify as a bug that it
doesn't have the effect you expect.

Re: two agent nits

[Kevin Greiner]

Simon Josefsson <jas@extundo.com> writes:

> * I can't seem to get the agent to fetch read articles.
>   gnus-agent-consider-all-articles's value is t
>   The default agent category predicate is 'true'.
>   `J s' and `J u' just download unread or ticked articles.

Have to tried setting gnus-agent-consider-all-articles to nil?  That
seems to work for me.

> * The custom type of gnus-agent-expire-days is integer.  What if I
>   don't want articles to be expired?  Nil?  0?  Either way, it should
>   be documented and the custom type improved.

gnus-agent-expire-days is archiac (perhaps the documentation should
say that) as it provides a global value.  It used to provide a list of
regular expressions matching group names but that didn't provide a
visual interface for maintaining it.  This use was deleted in 5.10 and
replaced by agent parameters in categories, groups, and topics (See
Agent Basics in the manual).

If you want to globally turn expiration OFF, see
gnus-agent-enable-expiration.

If you want to selectively control expiration, try editing your
category.  That is, open the category buffer, select the appropriate
category, then type 'e'.

Kevin

Re: two agent nits

[Simon Josefsson]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> * I can't seem to get the agent to fetch read articles.
>>   gnus-agent-consider-all-articles's value is t
>>   The default agent category predicate is 'true'.
>>   `J s' and `J u' just download unread or ticked articles.
>
> Have to tried setting gnus-agent-consider-all-articles to nil?  That
> seems to work for me.

I tried, but still the same.  Pressing `J u' on groups just say it is
finished, but only the unread/ticked articles are in my local cache.

Hm.  The manual and the docstring for the variable doesn't seem to be
in sync.  I thought the variable did what the docstring said, but the
manual just discuss missing headers.  Which one is correct?

gnus-agent-consider-all-articles's value is t
If non-nil, consider also the read articles for downloading.

`gnus-agent-consider-all-articles'
     If `gnus-agent-consider-all-articles' is non-`nil', the agent will
     fetch all missing headers.  When `nil', the agent will fetch only
     new headers.  The default is `nil'.

>> * The custom type of gnus-agent-expire-days is integer.  What if I
>>   don't want articles to be expired?  Nil?  0?  Either way, it should
>>   be documented and the custom type improved.
>
> gnus-agent-expire-days is archiac (perhaps the documentation should
> say that) as it provides a global value.  It used to provide a list of
> regular expressions matching group names but that didn't provide a
> visual interface for maintaining it.  This use was deleted in 5.10 and
> replaced by agent parameters in categories, groups, and topics (See
> Agent Basics in the manual).
>
> If you want to globally turn expiration OFF, see
> gnus-agent-enable-expiration.

Ah, that variable is what I want.  Thanks.  I added a link in the
docstring in case others make the same mistake.

Re: two agent nits

[Harry Putnam]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

>> * I can't seem to get the agent to fetch read articles.
>>   gnus-agent-consider-all-articles's value is t
>>   The default agent category predicate is 'true'.
>>   `J s' and `J u' just download unread or ticked articles.
>
> Have to tried setting gnus-agent-consider-all-articles to nil?  That
> seems to work for me.

That doesn't work here

Re: two agent nits

[Kevin Greiner]

Simon Josefsson <jas@extundo.com> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> * I can't seem to get the agent to fetch read articles.
>>>   gnus-agent-consider-all-articles's value is t
>>>   The default agent category predicate is 'true'.
>>>   `J s' and `J u' just download unread or ticked articles.
>>
>> Have to tried setting gnus-agent-consider-all-articles to nil?  That
>> seems to work for me.
>
> I tried, but still the same.  Pressing `J u' on groups just say it is
> finished, but only the unread/ticked articles are in my local cache.
>
> Hm.  The manual and the docstring for the variable doesn't seem to be
> in sync.  I thought the variable did what the docstring said, but the
> manual just discuss missing headers.  Which one is correct?

The variable gnus-agent-consider-all-articles appears in
gnus-agent-fetch-headers but NOT gnus-agent-fetch-articles.  However,
gnus-agent-fetch-group-1 calls gnus-agent-fetch-headers to get the
list of new articles so gnus-agent-consider-all-articles may, by
modifying the return value of gnus-agent-fetch-headers, effect the
list of articles being fetched by gnus-agent-fetch-articles.

In fact, that is almost certainly what is going wrong in your case.
For you to have read the article, you had to open the summary buffer.
That implies that gnus-agent-fetch-headers already fetched the headers
of each read article.  Now, it's later and you want
gnus-agent-fetch-group-1 to fetch the article.  The only problem is
that it uses gnus-agent-fetch-header to get the article list and
gnus-agent-fetch-header skips over those articles because their
headers have already been fetched.

I'll look into this further to see if I can reproduce it here.  If I
can, I'll work up a patch.  It will be a few days as I'm currently
busy on other projects.

Kevin

Re: two agent nits

[Simon Josefsson]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>>> * I can't seem to get the agent to fetch read articles.
>>>>   gnus-agent-consider-all-articles's value is t
>>>>   The default agent category predicate is 'true'.
>>>>   `J s' and `J u' just download unread or ticked articles.
>>>
>>> Have to tried setting gnus-agent-consider-all-articles to nil?  That
>>> seems to work for me.
>>
>> I tried, but still the same.  Pressing `J u' on groups just say it is
>> finished, but only the unread/ticked articles are in my local cache.
>>
>> Hm.  The manual and the docstring for the variable doesn't seem to be
>> in sync.  I thought the variable did what the docstring said, but the
>> manual just discuss missing headers.  Which one is correct?
>
> The variable gnus-agent-consider-all-articles appears in
> gnus-agent-fetch-headers but NOT gnus-agent-fetch-articles.  However,
> gnus-agent-fetch-group-1 calls gnus-agent-fetch-headers to get the
> list of new articles so gnus-agent-consider-all-articles may, by
> modifying the return value of gnus-agent-fetch-headers, effect the
> list of articles being fetched by gnus-agent-fetch-articles.

Thanks for the pointers, I think I isolated the problem, from g-a-f-a:

  (let* ((fetch-all (and gnus-agent-consider-all-articles
                         ;; Do not fetch all headers if the predicate
                         ;; implies that we only consider unread articles.
                         (not (gnus-predicate-implies-unread
                               (gnus-agent-find-parameter group
                                                          'agent-predicate)))))

Fetch-all evaluate to nil for me, causing the function to only return
a list of unread articles.  Since g-a-c-a-a is t, the reason fetch-al
is nil is because of the second statement.

(gnus-predicate-implies-unread 'true)
 => ignore

(not (gnus-predicate-implies-unread 'true))
 => nil

Reading the docstring:

(gnus-predicate-implies-unread PREDICATE)
  Say whether PREDICATE implies unread articles only.
  It is okay to miss some cases, but there must be no false positives.
  That is, if this function returns true, then indeed the predicate must
  return only unread articles.

The 'ignore return value is not documented, and because it is non-nil
it is treated as true by the caller in this case.

If I apply this change, everything works, but I can't tell if it is
the right thing.

--- gnus-agent.el.~6.180.~	2003-12-01 12:35:54.000000000 +0100
+++ gnus-agent.el	2003-12-01 17:39:06.000000000 +0100
@@ -2455,7 +2455,7 @@
         ((not function)
          nil)
         ((functionp function)
-         'ignore)
+	 nil)
         ((memq (car function) '(or and not))
          (apply (car function)
                 (mapcar 'gnus-function-implies-unread-1 (cdr function))))

Now, continuing, I found that most of my groups had .agentviews
looking like:

((731544 66) (731540 64 . 65) ...
2

I.e., they have large integers in them, which look like corruption.
If I remove the .agentview, and then J u, it works completely.

I'm removing all .agentview's and restarting Gnus, I'll see if I can
reproduce getting the large integers.

Re: two agent nits

[Kevin Greiner]

Simon Josefsson <jas@extundo.com> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>>
>>>> Simon Josefsson <jas@extundo.com> writes:
>>>>
>>>>> * I can't seem to get the agent to fetch read articles.
>>>>>   gnus-agent-consider-all-articles's value is t
>>>>>   The default agent category predicate is 'true'.
>>>>>   `J s' and `J u' just download unread or ticked articles.
>>>>
>>>> Have to tried setting gnus-agent-consider-all-articles to nil?  That
>>>> seems to work for me.
>>>
>>> I tried, but still the same.  Pressing `J u' on groups just say it is
>>> finished, but only the unread/ticked articles are in my local cache.
>>>
>>> Hm.  The manual and the docstring for the variable doesn't seem to be
>>> in sync.  I thought the variable did what the docstring said, but the
>>> manual just discuss missing headers.  Which one is correct?
>>
>> The variable gnus-agent-consider-all-articles appears in
>> gnus-agent-fetch-headers but NOT gnus-agent-fetch-articles.  However,
>> gnus-agent-fetch-group-1 calls gnus-agent-fetch-headers to get the
>> list of new articles so gnus-agent-consider-all-articles may, by
>> modifying the return value of gnus-agent-fetch-headers, effect the
>> list of articles being fetched by gnus-agent-fetch-articles.
>
> Thanks for the pointers, I think I isolated the problem, from g-a-f-a:
>
>   (let* ((fetch-all (and gnus-agent-consider-all-articles
>                          ;; Do not fetch all headers if the predicate
>                          ;; implies that we only consider unread articles.
>                          (not (gnus-predicate-implies-unread
>                                (gnus-agent-find-parameter group
>                                                           'agent-predicate)))))
>
> Fetch-all evaluate to nil for me, causing the function to only return
> a list of unread articles.  Since g-a-c-a-a is t, the reason fetch-al
> is nil is because of the second statement.
>
> (gnus-predicate-implies-unread 'true)
>  => ignore
>
> (not (gnus-predicate-implies-unread 'true))
>  => nil
>
> Reading the docstring:
>
> (gnus-predicate-implies-unread PREDICATE)
>   Say whether PREDICATE implies unread articles only.
>   It is okay to miss some cases, but there must be no false positives.
>   That is, if this function returns true, then indeed the predicate must
>   return only unread articles.
>
> The 'ignore return value is not documented, and because it is non-nil
> it is treated as true by the caller in this case.
>
> If I apply this change, everything works, but I can't tell if it is
> the right thing.

Thanks for isolating the problem.  I'm also not certain that this is
the right thing to do.  It is certainly the correct answer if the
function is gnus-agent-true but what if the function is a compiled
function returned by gnus-category-make-function?

> --- gnus-agent.el.~6.180.~	2003-12-01 12:35:54.000000000 +0100
> +++ gnus-agent.el	2003-12-01 17:39:06.000000000 +0100
> @@ -2455,7 +2455,7 @@
>          ((not function)
>           nil)
>          ((functionp function)
> -         'ignore)
> +	 nil)
>          ((memq (car function) '(or and not))
>           (apply (car function)
>                  (mapcar 'gnus-function-implies-unread-1 (cdr function))))


> Now, continuing, I found that most of my groups had .agentviews
> looking like:
>
> ((731544 66) (731540 64 . 65) ...
> 2
>
> I.e., they have large integers in them, which look like corruption.

That isn't corruption.  The large numbers are the timestamps when the
articles were fetched.  The original format looked like
((66 . 731544) (64 . 731540) (65 . 731540)) but it was slow to load
for large groups as it repeated the large numbers endlessly.  The
gnus-agent-load-alist function now loads either format and converts
internally to the alist form.

Kevin

Re: two agent nits

[Simon Josefsson]

> Simon Josefsson <jas@extundo.com> writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>>>
>>>>> Simon Josefsson <jas@extundo.com> writes:
>>>>>
>>>>>> * I can't seem to get the agent to fetch read articles.
>>>>>>   gnus-agent-consider-all-articles's value is t
>>>>>>   The default agent category predicate is 'true'.
>>>>>>   `J s' and `J u' just download unread or ticked articles.
>>>>>
>>>>> Have to tried setting gnus-agent-consider-all-articles to nil?  That
>>>>> seems to work for me.
>>>>
>>>> I tried, but still the same.  Pressing `J u' on groups just say it is
>>>> finished, but only the unread/ticked articles are in my local cache.
>>>>
>>>> Hm.  The manual and the docstring for the variable doesn't seem to be
>>>> in sync.  I thought the variable did what the docstring said, but the
>>>> manual just discuss missing headers.  Which one is correct?
>>>
>>> The variable gnus-agent-consider-all-articles appears in
>>> gnus-agent-fetch-headers but NOT gnus-agent-fetch-articles.  However,
>>> gnus-agent-fetch-group-1 calls gnus-agent-fetch-headers to get the
>>> list of new articles so gnus-agent-consider-all-articles may, by
>>> modifying the return value of gnus-agent-fetch-headers, effect the
>>> list of articles being fetched by gnus-agent-fetch-articles.
>>
>> Thanks for the pointers, I think I isolated the problem, from g-a-f-a:
>>
>>   (let* ((fetch-all (and gnus-agent-consider-all-articles
>>                          ;; Do not fetch all headers if the predicate
>>                          ;; implies that we only consider unread
>> articles.
>>                          (not (gnus-predicate-implies-unread
>>                                (gnus-agent-find-parameter group
>>                                                           'agent-predicate)))))
>>
>> Fetch-all evaluate to nil for me, causing the function to only return
>> a list of unread articles.  Since g-a-c-a-a is t, the reason fetch-al
>> is nil is because of the second statement.
>>
>> (gnus-predicate-implies-unread 'true)
>>  => ignore
>>
>> (not (gnus-predicate-implies-unread 'true))
>>  => nil
>>
>> Reading the docstring:
>>
>> (gnus-predicate-implies-unread PREDICATE)
>>   Say whether PREDICATE implies unread articles only.
>>   It is okay to miss some cases, but there must be no false positives.
>>   That is, if this function returns true, then indeed the predicate must
>>   return only unread articles.
>>
>> The 'ignore return value is not documented, and because it is non-nil
>> it is treated as true by the caller in this case.
>>
>> If I apply this change, everything works, but I can't tell if it is
>> the right thing.
>
> Thanks for isolating the problem.  I'm also not certain that this is
> the right thing to do.  It is certainly the correct answer if the
> function is gnus-agent-true but what if the function is a compiled
> function returned by gnus-category-make-function?

Returning nil may be a false negative, in that case.  But the docstring
says that's fine, just as long as there aren't any false positives.  I
guess the only bad thing that happens is that the agent downloads all
headers?

>> Now, continuing, I found that most of my groups had .agentviews
>> looking like:
>>
>> ((731544 66) (731540 64 . 65) ...
>> 2
>>
>> I.e., they have large integers in them, which look like corruption.
>
> That isn't corruption.  The large numbers are the timestamps when the
> articles were fetched.  The original format looked like
> ((66 . 731544) (64 . 731540) (65 . 731540)) but it was slow to load
> for large groups as it repeated the large numbers endlessly.  The
> gnus-agent-load-alist function now loads either format and converts
> internally to the alist form.

Ah, right, thanks.  Weird I had to remove the .agentview file to trigger
re-downloading articles then.  Too late to debug now.  Gnus has filled my
disk with 300MB worth of data, and counting, so I think it works now...

Re: two agent nits

[Harry Putnam]

"Simon Josefsson" <jas@extundo.com> writes:

> Ah, right, thanks.  Weird I had to remove the .agentview file to trigger
> re-downloading articles then.  Too late to debug now.  Gnus has filled my
> disk with 300MB worth of data, and counting, so I think it works now...

Good going Simon.  That is something I've struggled
(non-productively) with for a while.  Let us know if it seems to have
really solved the non-download read syndrome.  Beats the snot out of
full reads and the like.

Re: two agent nits

[Wes Hardaker]

>>>>> On Mon, 01 Dec 2003 14:35:51 -0600, Harry Putnam <reader@newsguy.com> said:

Harry> Good going Simon.  That is something I've struggled
Harry> (non-productively) with for a while.  Let us know if it seems
Harry> to have really solved the non-download read syndrome.  Beats
Harry> the snot out of full reads and the like.

Definitely!  I've been begging for this for ages.  I keep an
incredible amount of read mail around that need access frequently.
Basically, I don't use the agent because it's impossible for me do.
This may change that.

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

Re: two agent nits

[Kevin Greiner]

[-- Attachment #1: Type: text/plain, Size: 3961 bytes --]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>>> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>>>>
>>>>> Simon Josefsson <jas@extundo.com> writes:
>>>>>
>>>>>> * I can't seem to get the agent to fetch read articles.
>>>>>>   gnus-agent-consider-all-articles's value is t
>>>>>>   The default agent category predicate is 'true'.
>>>>>>   `J s' and `J u' just download unread or ticked articles.
>>>>>
>>>>> Have to tried setting gnus-agent-consider-all-articles to nil?  That
>>>>> seems to work for me.
>>>>
>>>> I tried, but still the same.  Pressing `J u' on groups just say it is
>>>> finished, but only the unread/ticked articles are in my local cache.
>>>>
>>>> Hm.  The manual and the docstring for the variable doesn't seem to be
>>>> in sync.  I thought the variable did what the docstring said, but the
>>>> manual just discuss missing headers.  Which one is correct?
>>>
>>> The variable gnus-agent-consider-all-articles appears in
>>> gnus-agent-fetch-headers but NOT gnus-agent-fetch-articles.  However,
>>> gnus-agent-fetch-group-1 calls gnus-agent-fetch-headers to get the
>>> list of new articles so gnus-agent-consider-all-articles may, by
>>> modifying the return value of gnus-agent-fetch-headers, effect the
>>> list of articles being fetched by gnus-agent-fetch-articles.
>>
>> Thanks for the pointers, I think I isolated the problem, from g-a-f-a:
>>
>>   (let* ((fetch-all (and gnus-agent-consider-all-articles
>>                          ;; Do not fetch all headers if the predicate
>>                          ;; implies that we only consider unread articles.
>>                          (not (gnus-predicate-implies-unread
>>                                (gnus-agent-find-parameter group
>>                                                           'agent-predicate)))))
>>
>> Fetch-all evaluate to nil for me, causing the function to only return
>> a list of unread articles.  Since g-a-c-a-a is t, the reason fetch-al
>> is nil is because of the second statement.
>>
>> (gnus-predicate-implies-unread 'true)
>>  => ignore
>>
>> (not (gnus-predicate-implies-unread 'true))
>>  => nil
>>
>> Reading the docstring:
>>
>> (gnus-predicate-implies-unread PREDICATE)
>>   Say whether PREDICATE implies unread articles only.
>>   It is okay to miss some cases, but there must be no false positives.
>>   That is, if this function returns true, then indeed the predicate must
>>   return only unread articles.
>>
>> The 'ignore return value is not documented, and because it is non-nil
>> it is treated as true by the caller in this case.
>>
>> If I apply this change, everything works, but I can't tell if it is
>> the right thing.
>
> Thanks for isolating the problem.  I'm also not certain that this is
> the right thing to do.  It is certainly the correct answer if the
> function is gnus-agent-true but what if the function is a compiled
> function returned by gnus-category-make-function?
>
>> --- gnus-agent.el.~6.180.~	2003-12-01 12:35:54.000000000 +0100
>> +++ gnus-agent.el	2003-12-01 17:39:06.000000000 +0100
>> @@ -2455,7 +2455,7 @@
>>          ((not function)
>>           nil)
>>          ((functionp function)
>> -         'ignore)
>> +	 nil)
>>          ((memq (car function) '(or and not))
>>           (apply (car function)
>>                  (mapcar 'gnus-function-implies-unread-1 (cdr function))))
>
>

Simon,

I finally realized what bothered me with this patch.  If you look
closely, the resulting code has a cond statement with four cases.  In
three cases, the result is simply nil.  In the fourth case, the
function recurses.  Taken together that means that the function simply
degenerates into nil.

So here's my version of the patch.  It is unfortunately a good deal
more complex.  I'd appreciate some feedback before I check it in.


[-- Attachment #2: Patch to fetch read articles. --]
[-- Type: text/plain, Size: 3597 bytes --]

--- lisp.cvs_ref/gnus-agent.el	Mon Dec  1 23:05:10 2003
+++ lisp/gnus-agent.el	Tue Dec  2 00:00:26 2003
@@ -2445,22 +2445,58 @@
 (defun gnus-predicate-implies-unread (predicate)
   "Say whether PREDICATE implies unread articles only.
 It is okay to miss some cases, but there must be no false positives.
-That is, if this function returns true, then indeed the predicate must
+That is, if this predicate returns true, then indeed the predicate must
 return only unread articles."
-  (gnus-function-implies-unread-1 (gnus-category-make-function predicate)))
+  (eq t (gnus-function-implies-unread-1 
+         (gnus-category-make-function-1 predicate))))
 
 (defun gnus-function-implies-unread-1 (function)
-  (cond ((eq function (symbol-function 'gnus-agent-read-p))
-         nil)
-        ((not function)
-         nil)
-        ((functionp function)
-         'ignore)
-        ((memq (car function) '(or and not))
-         (apply (car function)
-                (mapcar 'gnus-function-implies-unread-1 (cdr function))))
-        (t
-         (error "Unknown function: %s" function))))
+  "Recursively evaluate a predicate function to determine whether it can select
+any read articles.  Returns t if the function is known to never
+return read articles, nil when it is known to always return read
+articles, and t_nil when the function may return both read and unread
+articles."
+  (let ((func (car function))
+        (args (mapcar 'gnus-function-implies-unread-1 (cdr function))))
+    (cond ((eq func 'and)
+           (cond ((memq t args) ; if any argument returns only unread articles
+                  ;; then that argument constrains the result to only unread articles.
+                  t)
+                 ((memq 't_nil args) ; if any argument is indeterminate
+                  ;; then the result is indeterminate
+                  't_nil)))
+          ((eq func 'or)
+           (cond ((memq nil args) ; if any argument returns read articles
+                  ;; then that argument ensures that the results includes read articles.
+                  nil)
+                 ((memq 't_nil args) ; if any argument is indeterminate
+                  ;; then that argument ensures that the results are indeterminate
+                  't_nil)
+                 (t ; if all arguments return only unread articles
+                  ;; then the result returns only unread articles
+                  t)))
+          ((eq func 'not)
+           (cond ((eq (car args) 't_nil) ; if the argument is indeterminate
+                  ; then the result is indeterminate
+                  (car args))
+                 (t ; otherwise
+                  ; toggle the result to be the opposite of the argument
+                  (not (car args)))))
+          ((eq func 'gnus-agent-read-p)
+           nil) ; The read predicate NEVER returns unread articles
+          ((eq func 'gnus-agent-false)
+           t) ; The false predicate returns t as the empty set excludes all read articles
+          ((eq func 'gnus-agent-true)
+           nil) ; The true predicate ALWAYS returns read articles
+          ((catch 'found-match
+             (let ((alist gnus-category-predicate-alist))
+               (while alist
+                 (if (eq func (cdar alist))
+                     (throw 'found-match t)
+                   (setq alist (cdr alist))))))
+           't_nil) ; All other predicates return read and unread articles
+          (t
+           (error "Unknown predicate function: %s" function)))))
 
 (defun gnus-group-category (group)
   "Return the category GROUP belongs to."

[-- Attachment #3: Type: text/plain, Size: 7 bytes --]


Kevin

Re: two agent nits

[Simon Josefsson]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> I finally realized what bothered me with this patch.  If you look
> closely, the resulting code has a cond statement with four cases.  In
> three cases, the result is simply nil.  In the fourth case, the
> function recurses.  Taken together that means that the function simply
> degenerates into nil.

Oops.

> So here's my version of the patch.  It is unfortunately a good deal
> more complex.  I'd appreciate some feedback before I check it in.

I did not read the patch, but I applied it and invoked `J s' and it
has been churning along for a few hours.  It made me realize 1 GB
isn't enough memory in a desktop machine these days, but other than
that it seems to work.

Once I have everything downloaded I'll start to use it for real.

Thanks.

Re: two agent nits

[Simon Josefsson]

Simon Josefsson <jas@extundo.com> writes:

> Hm.  The manual and the docstring for the variable doesn't seem to be
> in sync.  I thought the variable did what the docstring said, but the
> manual just discuss missing headers.  Which one is correct?
>
> gnus-agent-consider-all-articles's value is t
> If non-nil, consider also the read articles for downloading.
>
> `gnus-agent-consider-all-articles'
>      If `gnus-agent-consider-all-articles' is non-`nil', the agent will
>      fetch all missing headers.  When `nil', the agent will fetch only
>      new headers.  The default is `nil'.

I changed the manual, it now says:

`gnus-agent-consider-all-articles'
     If `gnus-agent-consider-all-articles' is non-`nil', the agent will
     let the agent predicate decide whether articles need to be
     downloaded or not, for all articles.  When `nil', the default, the
     agent will only let the predicate decide whether unread articles
     are downloaded or not.  If you enable this, you may also want to
     look into the agent expiry settings (see *note Category Variables::),
     so that the agent doesn't download articles which the agent will
     later expire, over and over again.

Re: two agent nits

[Kevin Greiner]

Simon Josefsson <jas@extundo.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Hm.  The manual and the docstring for the variable doesn't seem to be
>> in sync.  I thought the variable did what the docstring said, but the
>> manual just discuss missing headers.  Which one is correct?
>>
>> gnus-agent-consider-all-articles's value is t
>> If non-nil, consider also the read articles for downloading.
>>
>> `gnus-agent-consider-all-articles'
>>      If `gnus-agent-consider-all-articles' is non-`nil', the agent will
>>      fetch all missing headers.  When `nil', the agent will fetch only
>>      new headers.  The default is `nil'.
>
> I changed the manual, it now says:
>
> `gnus-agent-consider-all-articles'
>      If `gnus-agent-consider-all-articles' is non-`nil', the agent will
>      let the agent predicate decide whether articles need to be
>      downloaded or not, for all articles.  When `nil', the default, the
>      agent will only let the predicate decide whether unread articles
>      are downloaded or not.  If you enable this, you may also want to
>      look into the agent expiry settings (see *note Category Variables::),
>      so that the agent doesn't download articles which the agent will
>      later expire, over and over again.

Sounds good.  I'll update the docstring to something similar.

Kevin

Re: two agent nits

[Harry Putnam]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> So here's my version of the patch.  It is unfortunately a good deal
> more complex.  I'd appreciate some feedback before I check it in.

So Kevin/Simon,
Does  cvs gnus now have this patch.  Is it on by default?

Re: two agent nits

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> So here's my version of the patch.  It is unfortunately a good deal
>> more complex.  I'd appreciate some feedback before I check it in.
>
> So Kevin/Simon,
> Does  cvs gnus now have this patch.  Is it on by default?

The patch was installed.  The agent now downloads everything for me,
and things are quite snappy when data is stored locally, as expected.

A minor bug: C-u G DEL on a group doesn't seem to remove its entry
from the agent active file.  M-x gnus-agent-expire RET do discover it
and allow me to delete it though.

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

>> So Kevin/Simon,
>> Does  cvs gnus now have this patch.  Is it on by default?
>
> The patch was installed.  The agent now downloads everything for me,
> and things are quite snappy when data is stored locally, as expected.

No joy here.  I updated cvs gnus and running either Ju on a specific
group or a general Js  I still see read articles that are not being
downloaded.  My predicate is true and the groups I'm testing are
listed in that category.

Is it necessary to delete all .agentview still.  Are there other
repercussions to doing that?
 

Re: two agent nits

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>>> So Kevin/Simon,
>>> Does  cvs gnus now have this patch.  Is it on by default?
>>
>> The patch was installed.  The agent now downloads everything for me,
>> and things are quite snappy when data is stored locally, as expected.
>
> No joy here.  I updated cvs gnus and running either Ju on a specific
> group or a general Js  I still see read articles that are not being
> downloaded.  My predicate is true and the groups I'm testing are
> listed in that category.

And your gnus-agent-consider-all-articles's value is t?

> Is it necessary to delete all .agentview still.

Perhaps, I did it on all groups.  Perhaps I should have only done so
for a few and debug why it didn't work for others...

> Are there other repercussions to doing that?

Not as far as I know.  Wiping anything under ~/News/agent should be
safe.  Gnus have to download it again, of course, but nothing beyond
that.

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>>> So Kevin/Simon,
>>>> Does  cvs gnus now have this patch.  Is it on by default?
>>>
>>> The patch was installed.  The agent now downloads everything for me,
>>> and things are quite snappy when data is stored locally, as expected.
>>
>> No joy here.  I updated cvs gnus and running either Ju on a specific
>> group or a general Js  I still see read articles that are not being
>> downloaded.  My predicate is true and the groups I'm testing are
>> listed in that category.
>
> And your gnus-agent-consider-all-articles's value is t?

OOPs.. I got the notion somewhere it would be on by default.

>> Is it necessary to delete all .agentview still.
>
> Perhaps, I did it on all groups.  Perhaps I should have only done so
> for a few and debug why it didn't work for others...

I can report that it seems to be working as expected without having
done anything to .agentview files.  Once I set
gnus-agent-consider-all-articles to t and ran Js... well gnus is
still chewing up the inodes as I write.

Re: two agent nits

[Harry Putnam]

Harry Putnam <reader@newsguy.com> writes:

> OOPs.. I got the notion somewhere it would be on by default.
>
>>> Is it necessary to delete all .agentview still.
>>
>> Perhaps, I did it on all groups.  Perhaps I should have only done so
>> for a few and debug why it didn't work for others...
>
> I can report that it seems to be working as expected without having
> done anything to .agentview files.  Once I set
> gnus-agent-consider-all-articles to t and ran Js... well gnus is
> still chewing up the inodes as I write.

Simon, I'm seeing lots of time chewed up when running Js.  Even
though I'm fairly close to up with newsfeed.  That is, I may have
100-200 messages counting all groups to download.  But I see massive
amounts of time go by when running Js.  Judging by the lack of
activity on my asdsl modem lights it doesn't seem to be conversing
with nntp server.

Are you seeing any delays.

Re: two agent nits

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> OOPs.. I got the notion somewhere it would be on by default.
>>
>>>> Is it necessary to delete all .agentview still.
>>>
>>> Perhaps, I did it on all groups.  Perhaps I should have only done so
>>> for a few and debug why it didn't work for others...
>>
>> I can report that it seems to be working as expected without having
>> done anything to .agentview files.  Once I set
>> gnus-agent-consider-all-articles to t and ran Js... well gnus is
>> still chewing up the inodes as I write.
>
> Simon, I'm seeing lots of time chewed up when running Js.  Even
> though I'm fairly close to up with newsfeed.  That is, I may have
> 100-200 messages counting all groups to download.  But I see massive
> amounts of time go by when running Js.  Judging by the lack of
> activity on my asdsl modem lights it doesn't seem to be conversing
> with nntp server.
>
> Are you seeing any delays.

Yup.  It delays while saying "Fetching headers" but no headers are
transfered over the network.  I think it is simply processing the
headers internally.  I'm sure it can be optimized, profiling it with
elp would be the first towards that.  My results below, but doing M-x
load RET gnus-agent.el RET first would probably have generated more
information.

Function Name                                                                 Call Count  Elapsed Time  Average Time
============================================================================  ==========  ============  ============
gnus-agent-fetch-session                                                      1           191.742998    191.742998
gnus-agent-fetch-group-1                                                      325         191.72513400  0.5899234892
gnus-get-newsgroup-headers-xover                                              325         158.08053799  0.4864016553
gnus-agent-fetch-headers                                                      325         28.212026000  0.0868062338
gnus-retrieve-headers                                                         362         10.029830999  0.0277067154
gnus-agent-load-alist                                                         1159        7.9795109999  0.0068848239
gnus-cache-file-contents                                                      1159        7.7396720000  0.0066778878
gnus-agent-read-agentview                                                     503         6.585462      0.0130923697
gnus-cache-retrieve-headers                                                   180         4.8972399999  0.0272068888
gnus-agent-retrieve-headers                                                   180         4.8745889999  0.0270810499
gnus-point-at-eol                                                             500918      3.1774099999  6.343...e-06
gnus-agent-braid-nov                                                          182         2.2013949999  0.0120955769
gnus-agent-save-alist                                                         184         2.0454090000  0.0111163532
gnus-agent-check-overview-buffer                                              362         1.5467199999  0.0042727071
gnus-uncompress-range                                                         4561        1.0758729999  0.0002358853

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> Harry Putnam <reader@newsguy.com> writes:
>>
>>> OOPs.. I got the notion somewhere it would be on by default.
>>>
>>>>> Is it necessary to delete all .agentview still.
>>>>
>>>> Perhaps, I did it on all groups.  Perhaps I should have only done so
>>>> for a few and debug why it didn't work for others...
>>>
>>> I can report that it seems to be working as expected without having
>>> done anything to .agentview files.  Once I set
>>> gnus-agent-consider-all-articles to t and ran Js... well gnus is
>>> still chewing up the inodes as I write.
>>
>> Simon, I'm seeing lots of time chewed up when running Js.  Even
>> though I'm fairly close to up with newsfeed.  That is, I may have
>> 100-200 messages counting all groups to download.  But I see massive
>> amounts of time go by when running Js.  Judging by the lack of
>> activity on my asdsl modem lights it doesn't seem to be conversing
>> with nntp server.
>>
>> Are you seeing any delays.
>
> Yup.  It delays while saying "Fetching headers" but no headers are
> transfered over the network.  I think it is simply processing the
> headers internally.  I'm sure it can be optimized, profiling it with
> elp would be the first towards that.  My results below, but doing M-x
> load RET gnus-agent.el RET first would probably have generated more
> information.

This looks pretty wild...

> gnus-point-at-eol      500918      3.1774099999  6.343...e-06

Simon, what is the syntax for running that profile?  I couldn't get
even a remote clue from C-h f elp<spc><spc>.  And was discouraged to
find that an `i' (index) search on `elp' in emacs info pages just
brings up lots of hits help `help'.

`s' search gives no hits on `[^h]elp-'

Re: two agent nits

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> This looks pretty wild...
>
>> gnus-point-at-eol      500918      3.1774099999  6.343...e-06
>
> Simon, what is the syntax for running that profile?  I couldn't get
> even a remote clue from C-h f elp<spc><spc>.  And was discouraged to
> find that an `i' (index) search on `elp' in emacs info pages just
> brings up lots of hits help `help'.
>
> `s' search gives no hits on `[^h]elp-'

See the Troubleshooting section of the Gnus manual:

   A fancier approach is to use the elisp profiler, ELP.  The profiler
is (or should be) fully documented elsewhere, but to get you started
there are a few steps that need to be followed.  First, instrument the
part of Gnus you are interested in for profiling, e.g. `M-x
elp-instrument-package RET gnus' or `M-x elp-instrument-package RET
message'.  Then perform the operation that is slow and press `M-x
elp-results'.  You will then see which operations that takes time, and
can debug them further.  If the entire operation takes much longer than
the time spent in the slowest function in the profiler output, you
probably profiled the wrong part of Gnus.  To reset profiling
statistics, use `M-x elp-reset-all'.  `M-x elp-restore-all' is supposed
to remove profiling, but given the complexities and dynamic code
generation in Gnus, it might not always work perfectly.

Re: two agent nits

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> See the Troubleshooting section of the Gnus manual:

Nice... got it working... thanks