[patch] Agent and download scoring etc.

[patch] Agent and download scoring etc.

[Mike McEwan]

I've been tinkering again (I wonder what it's like outside?). I tried
to get the Agent to download articles based on their scores only to
discover (or perhaps rediscover) the Agent way of scoring things is
completely separate from normal score files. 

Lars, I know you're mime, mime, miming at the moment, but since we're
alpha again, I wonder if you might whack on the following if you think
it looks OK :-). It's the ol' `why can't Agent use my normal score files'
patch.
 
The patch allows the Agent to decide what to download with the
permitted subset of the normal score files when the download score
rule of a category, or locally, the `agent-score' parameter of a
group, is `file'. The old stuff still takes precedence.

I wasn't sure how category download score rule or group parameter
`agent-score' are supposed to be specified and couldn't find any doc
so I've made some assumptions:

Category rule

 `file'  -  use normal score files.

 `("filename" ["filename"])' - use the specified file(s) (e.g. `("~/News/agent.SCORE")').

 `(("from" ("Lars Magne Ingebrigtsen" 1000 nil s))' - use specified rules.

 Group Parameters

 `(agent-score . file)'      - use normal score files.

 `(agent-score "filename" ["filename"])' - use the specified file(s).

 `(agent-score ("from" ("Lars Magne Ingebrigtsen" 1000 nil s)))' - use specified rules.


I hope the above doesn't clobber anyone's current usage too
much. Group parameters take precedence over category rules. There's a
couple of other little mods to try and speed things up a bit.

See what you think.

-- 
Mike.

--- ChangeLog.orig	1998/09/03 21:23:53
+++ ChangeLog	1998/09/05 16:28:46
@@ -1,3 +1,14 @@
+1998-09-05  Mike McEwan  <mike@lotusland.demon.co.uk>
+
+	* gnus-agent.el (gnus-agent-scoreable-headers): New variable.
+	(gnus-agent-fetch-group-1): Score article headers using normal
+ 	group score files if the download score rule of a category/group
+ 	is `file'.
+	(gnus-agent-fetch-group-1): Don't parse the entire .overview when
+ 	deciding what articles to download.
+	(gnus-agent-fetch-group-1): Don't push headers through scoring and
+ 	predicate processing if predicate is `true' or `false'.
+
 Thu Sep  3 15:23:22 1998  Lars Magne Ingebrigtsen  <larsi@menja.ifi.uio.no>
 
 	* gnus.el: Pterodactyl Gnus v0.14 is released.

--- gnus-agent.el.orig	Tue Sep  1 20:22:11 1998
+++ gnus-agent.el	Sat Sep  5 17:51:29 1998
@@ -27,7 +27,8 @@
 (require 'gnus-cache)
 (require 'nnvirtual)
 (require 'gnus-sum)
-(eval-when-compile (require 'cl))
+(eval-when-compile (require 'cl)
+		   (require 'gnus-score))
 
 (defcustom gnus-agent-directory (nnheader-concat gnus-directory "agent/")
   "Where the Gnus agent will store its files."
@@ -94,6 +95,13 @@
 (defvar gnus-agent-send-mail-function nil)
 (defvar gnus-agent-article-file-coding-system 'no-conversion)
 
+(defconst gnus-agent-scoreable-headers
+  (list
+   "subject" "from" "date" "message-id" 
+   "references" "chars" "lines" "xref")
+  "Headers that are considered when scoring articles
+for download via the Agent.")
+
 ;; Dynamic variables
 (defvar gnus-headers)
 (defvar gnus-score)
@@ -894,27 +902,63 @@
       ;; Parse them and see which articles we want to fetch.
       (setq gnus-newsgroup-dependencies
 	    (make-vector (length articles) 0))
-      (setq gnus-newsgroup-headers
-	    (gnus-get-newsgroup-headers-xover articles nil nil group))
+      ;; No need to call `gnus-get-newsgroup-headers-xover' with 
+      ;; the entire .overview for group as we still have the just
+      ;; downloaded headers in `gnus-agent-overview-buffer'.
+      (let ((nntp-server-buffer gnus-agent-overview-buffer))
+	(setq gnus-newsgroup-headers
+	      (gnus-get-newsgroup-headers-xover articles nil nil group)))
       (setq category (gnus-group-category group))
       (setq predicate
 	    (gnus-get-predicate 
 	     (or (gnus-group-get-parameter group 'agent-predicate)
 		 (cadr category))))
-      (setq score-param
-	    (or (gnus-group-get-parameter group 'agent-score)
-		(caddr category)))
-      (when score-param
-	(gnus-score-headers (list (list score-param))))
-      (setq arts nil)
-      (while (setq gnus-headers (pop gnus-newsgroup-headers))
-	(setq gnus-score
-	      (or (cdr (assq (mail-header-number gnus-headers)
-			     gnus-newsgroup-scored))
-		  gnus-summary-default-score))
-	(when (funcall predicate)
-	  (push (mail-header-number gnus-headers)
-		arts)))
+      ;; Do we want to download everything, or nothing?
+      (if (or (eq (caaddr predicate) 'gnus-agent-true)
+	      (eq (caaddr predicate) 'gnus-agent-false))
+	  ;; Yes.
+	  (setq arts (symbol-value 
+		      (cadr (assoc (caaddr predicate) 
+				   '((gnus-agent-true articles)
+				     (gnus-agent-false nil))))))
+	;; No, we need to decide what we want.
+	(setq score-param
+	      (let ((score-method (or 
+				   (gnus-group-get-parameter group 'agent-score t)
+				   (caddr category))))
+		(when score-method
+		  (require 'gnus-score)
+		  (if (eq score-method 'file)
+		      (let ((entries
+			     (gnus-score-load-files
+			      (gnus-all-score-files group)))
+			    list score-file)
+			(while (setq list (car entries))
+			  (push (car list) score-file)
+			  (setq list (cdr list))
+			  (while list
+			    (when (member (caar list)
+					  gnus-agent-scoreable-headers)
+			      (push (car list) score-file))
+			    (setq list (cdr list)))
+			  (setq score-param 
+				(append score-param (list (nreverse score-file)))
+				score-file nil entries (cdr entries)))
+			(list score-param))
+		    (if (stringp (car score-method))
+			score-method
+		      (list (list score-method)))))))
+	(when score-param
+	  (gnus-score-headers score-param))
+	(setq arts nil)
+	(while (setq gnus-headers (pop gnus-newsgroup-headers))
+	  (setq gnus-score
+		(or (cdr (assq (mail-header-number gnus-headers)
+			       gnus-newsgroup-scored))
+		    gnus-summary-default-score))
+	  (when (funcall predicate)
+	    (push (mail-header-number gnus-headers)
+		  arts))))
       ;; Fetch the articles.
       (when arts
 	(gnus-agent-fetch-articles group arts)))

Re: [patch] Agent and download scoring etc.

[Lars Magne Ingebrigtsen]

Mike McEwan <mike@lotusland.demon.co.uk> writes:

> It's the ol' `why can't Agent use my normal score files' patch.

Thanks for the patch; I've applied it to Pterodactyl Gnus v0.17.

Could you also send me a patch for the manual to document this
much-wished-for feature?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: [patch] Agent and download scoring etc.

[Mike McEwan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Could you also send me a patch for the manual to document this
> much-wished-for feature?

Your wish...

I've also included another small update to `gnus-agent.el' to allow
the use of lists in a group's parameters when specifying `agent-predicate'.

-- 
Mike.

--- ChangeLog.orig	Sun Sep  6 20:01:14 1998
+++ ChangeLog	Sun Sep  6 20:07:09 1998
@@ -1,3 +1,8 @@
+1998-09-06  Mike McEwan  <mike@lotusland.demon.co.uk>
+
+	* gnus-agent.el (gnus-agent-fetch-group-1): Allow lists when
+	specifying `agent-predicate' in a group's parameters.
+
 Sat Sep  5 21:55:01 1998  Lars Magne Ingebrigtsen  <larsi@menja.ifi.uio.no>
 
 	* gnus.el: Pterodactyl Gnus v0.16 is released.
--- gnus-agent.el.orig	Sun Sep  6 20:01:14 1998
+++ gnus-agent.el	Sun Sep  6 20:03:24 1998
@@ -911,7 +911,7 @@
       (setq category (gnus-group-category group))
       (setq predicate
 	    (gnus-get-predicate 
-	     (or (gnus-group-get-parameter group 'agent-predicate)
+	     (or (gnus-group-get-parameter group 'agent-predicate t)
 		 (cadr category))))
       ;; Do we want to download everything, or nothing?
       (if (or (eq (caaddr predicate) 'gnus-agent-true)
--- ChangeLog.orig	1998/09/05 23:27:13
+++ ChangeLog	1998/09/06 19:12:16
@@ -1,3 +1,8 @@
+1998-09-06  Mike McEwan  <mike@lotusland.demon.co.uk>
+
+	* gnus.texi (Category Syntax): Added doc about agent categories
+	and download scoring.
+
 1998-09-05 17:36:14  Lars Magne Ingebrigtsen  <larsi@gnus.org>
 
 	* gnus.texi (Sorting Groups): Change.

--- gnus.texi.orig	1998/09/05 23:27:14
+++ gnus.texi	1998/09/06 19:40:41
@@ -11562,11 +11562,21 @@
 @item
 a score rule which (generally) gives you a finer granularity when
 deciding what articles to download.  (Note that this @dfn{download
-score} is wholly unrelated to normal scores.)
+score} is not necessarily related to normal scores.)
 @end enumerate
 
-A predicate consists of predicates with logical operators sprinkled in
-between.
+A predicate in its simplest form can be a single predicate such as
+@code{true} or @code{false}. These two will download every available
+article or nothing respectively. In the case of these two special
+predicates an additional score rule is superfluous.
+
+Predicates of @code{high} or @code{low} download articles in respect of
+their scores in relationship to @code{gnus-agent-high-score} and
+@code{gnus-agent-low-score} as descibed below.
+
+To gain even finer control of what is to be regarded eligible for
+download a predicate can consist of a number of predicates with logical
+operators sprinkled in between.
 
 Perhaps some examples are in order.
 
@@ -11602,7 +11612,7 @@
 
 @table @code
 @item short
-True iff the article is shorter than @code{gnus-agent-short-article}
+True if the article is shorter than @code{gnus-agent-short-article}
 lines; default 100.
 
 @item long
@@ -11610,15 +11620,15 @@
 lines; default 200.
 
 @item low
-True iff the article has a download score less than
+True if the article has a download score less than
 @code{gnus-agent-low-score}; default 0.
 
 @item high
-True iff the article has a download score greater than
+True if the article has a download score greater than
 @code{gnus-agent-high-score}; default 0.
 
 @item spam
-True iff the Gnus Agent guesses that the article is spam.  The
+True if the Gnus Agent guesses that the article is spam.  The
 heuristics may change over time, but at present it just computes a
 checksum and sees whether articles match.
 
@@ -11634,14 +11644,186 @@
 @code{gnus-headers} and @code{gnus-score} dynamic variables are bound to
 useful values.
 
+For example, you could decide that you don't want to download articles
+that were posted more than a certain number of days ago (e.g. posted
+more than @code{gnus-agent-expire-days} ago) you might write a function
+something along the lines of the following:
+
+@lisp
+(defun my-article-old-p ()
+  "Say whether an article is old."
+  (< (time-to-day (date-to-time (mail-header-date gnus-headers)))
+     (- (time-to-day (current-time)) gnus-agent-expire-days)))
+@end lisp
+
+with the predicate then defined as:
+
+@lisp
+(not my-article-old-p)
+@end lisp
+
+or you could append your predicate to the predefined
+@code{gnus-category-predicate-alist} in your @file{~/.gnus.el} or
+wherever.  (Note: this would have to be at a point *after*
+@code{gnus-agent} has been loaded via @code{(gnus-agentize)})   
+
+@lisp
+(defvar  gnus-category-predicate-alist
+  (append gnus-category-predicate-alist
+	 '((old . my-article-old-p))))
+@end lisp
+
+and simply specify your predicate as:
+
+@lisp
+(not old)
+@end lisp
+
+If/when using something like the above, be aware that there are many
+misconfigured systems/mailers out there and so an article's date is not
+always a reliable indication of when it was posted. Hell, some people
+just don't give a damm.
+
+
+The above predicates apply to *all* the groups which belong to the
+category. However, if you wish to have a specific predicate for an
+individual group within a category, or you're just too lazy to set up a
+new category, you can enter a group's individual predicate in it's group 
+parameters like so:
+
+@lisp
+(agent-predicate . short)
+@end lisp
+
+This is the group parameter equivalent of the agent category
+default. Note that when specifying a single word predicate like this,
+the @code{agent-predicate} specification must be in dotted pair
+notation.
+
+The equivalent of the longer example from above would be:
+
+@lisp
+(agent-predicate or high (and (not low) (not long)))
+@end lisp
+
+The outer parenthesis required in the category specification are not
+entered here as, not being in dotted pair notation, the value of the
+predicate is assumed to be a list. 
+ 
+
 Now, the syntax of the download score is the same as the syntax of
 normal score files, except that all elements that require actually
 seeing the article itself are verboten.  This means that only the
-following headers can be scored on: @code{From}, @code{Subject},
-@code{Date}, @code{Xref}, @code{Lines}, @code{Chars}, @code{Message-ID},
-and @code{References}.
+following headers can be scored on: @code{Subject}, @code{From},
+@code{Date}, @code{Message-ID}, @code{References}, @code{Chars},
+@code{Lines}, and @code{Xref}.
+
+As with predicates, the specification of the @code{download score rule}
+to use in respect of a group can be in either the category definition if
+it's to be applicable to all groups in therein, or a group's parameters
+if it's to be specific to that group.
+
+In both of these places the @code{download score rule} can take one of
+three forms:
+
+@table @code
+@enumerate
+@item 
+Score rule
+
+This has the same syntax as a normal gnus score file except only a
+subset of scoring keywords are available as mentioned above.
+
+example:
+
+@itemize @bullet
+@item 
+Category specification
 
+@lisp
+(("from"        
+       ("Lars Ingebrigtsen" 1000000 nil s))
+("lines"
+       (500 -100 nil <)))
+@end lisp
+
+@item 
+Group Parameter specification
+
+@lisp
+(agent-score ("from"        
+                   ("Lars Ingebrigtsen" 1000000 nil s))
+             ("lines"
+                   (500 -100 nil <)))
+@end lisp
+
+Again, note the omission of the outermost parenthesis here.
+@end itemize
+
+@item 
+Agent score file
 
+These score files must *only* contain the permitted scoring keywords
+stated above.
+
+example:
+
+@itemize @bullet
+@item 
+Category specification
+
+@lisp
+("~/News/agent.SCORE")
+@end lisp
+
+or perhaps
+
+@lisp
+("~/News/agent.SCORE" "~/News/agent.group.SCORE")
+@end lisp
+
+@item 
+Group Parameter specification
+
+@lisp
+(agent-score "~/News/agent.SCORE")
+@end lisp
+
+Additional score files can be specified as above. Need I say anything
+about parenthesis.
+@end itemize
+
+@item 
+Use @code{normal} score files
+
+If you dont want to maintain two sets of scoring rules for a group, and
+your desired @code{downloading} criteria for a group are the same as your
+@code{reading} criteria then you can tell the agent to refer to your
+@code{normal} score files when deciding what to download.
+
+These directives in either the category definition or a group's
+parameters will cause the agent to read in all the applicable score
+files for a group, *filtering out* those those sections that do not
+relate to one of the permitted subset of scoring keywords.
+
+@itemize @bullet
+@item 
+Category Specification
+
+@lisp
+file
+@end lisp
+
+@item 
+Group Parameter specification
+
+@lisp
+(agent-score . file)
+@end lisp
+@end itemize
+@end enumerate
+@end table
+ 
 @node The Category Buffer
 @subsubsection The Category Buffer
 

Re: [patch] Agent and download scoring etc.

[Lars Magne Ingebrigtsen]

Mike McEwan <mike@lotusland.demon.co.uk> writes:

> > Could you also send me a patch for the manual to document this
> > much-wished-for feature?
> 
> Your wish...

Thanks; I've applied the bits other than

> -True iff the article has a download score greater than
> +True if the article has a download score greater than

When I say "iff", I mean "iff".  :-)  ("If and only if", that is.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen