From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/50095 Path: main.gmane.org!not-for-mail From: Kevin Greiner Newsgroups: gmane.emacs.gnus.general Subject: Re: The agent shows ghost messages Date: Thu, 13 Feb 2003 07:50:11 -0600 Sender: owner-ding@hpc.uh.edu Message-ID: References: <4nwukeiy3c.fsf@lockgroove.bwh.harvard.edu> <4nr8amfwem.fsf@lockgroove.bwh.harvard.edu> <4nfzqur9al.fsf@lockgroove.bwh.harvard.edu> <4nk7g6tvdf.fsf@lockgroove.bwh.harvard.edu> <4nadh1k3wy.fsf@lockgroove.bwh.harvard.edu> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: main.gmane.org 1045144188 19476 80.91.224.249 (13 Feb 2003 13:49:48 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Thu, 13 Feb 2003 13:49:48 +0000 (UTC) Return-path: Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by main.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 18jJk6-00053Z-00 for ; Thu, 13 Feb 2003 14:49:34 +0100 Original-Received: from sina.hpc.uh.edu ([129.7.128.10] ident=lists) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 18jJlH-00012s-00; Thu, 13 Feb 2003 07:50:47 -0600 Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Thu, 13 Feb 2003 07:51:45 -0600 (CST) Original-Received: from sclp3.sclp.com (sclp3.sclp.com [66.230.238.2]) by sina.hpc.uh.edu (8.9.3/8.9.3) with SMTP id HAA07742 for ; Thu, 13 Feb 2003 07:51:24 -0600 (CST) Original-Received: (qmail 87964 invoked by alias); 13 Feb 2003 13:50:22 -0000 Original-Received: (qmail 87959 invoked from network); 13 Feb 2003 13:50:21 -0000 Original-Received: from quimby.gnus.org (80.91.224.244) by 66.230.238.6 with SMTP; 13 Feb 2003 13:50:21 -0000 Original-Received: from news by quimby.gnus.org with local (Exim 3.12 #1 (Debian)) id 18jJv8-0004sc-00 for ; Thu, 13 Feb 2003 15:00:58 +0100 Original-To: ding@gnus.org Original-Path: not-for-mail Original-Newsgroups: gnus.ding Original-Lines: 423 Original-NNTP-Posting-Host: p140.n-dapop03.stsn.com Original-X-Trace: quimby.gnus.org 1045144858 18761 63.241.218.140 (13 Feb 2003 14:00:58 GMT) Original-X-Complaints-To: usenet@quimby.gnus.org Original-NNTP-Posting-Date: 13 Feb 2003 14:00:58 GMT User-Agent: Gnus/5.090016 (Oort Gnus v0.16) Emacs/21.2 Cancel-Lock: sha1:EHd+BX/rB+Nwp9OhJ5UBIDOPFMw= Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:50095 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:50095 --=-=-= Niklas Morberg writes: > Kevin Greiner writes: > >>> 1. Start with two unread and undownloaded messages in a >>> group. >>> >>> 2. Read the first message. >>> >> >> Niklas, pls do the following command at this point: >> M-: (debug-on-entry 'gnus-agent-expire) >> M-: (setq gnus-verbose 7) >> >>> 3. Move the second message to another group with `B m' >>> without reading it. >> >> You should have been thrown into the debugger. Did that happen? > > Yes. > >> Was the first argument to gnus-agent-expire a list of one >> integer; the article # of the article being moved? > > Yes. Complete output: > > * gnus-agent-expire((8673) "incoming" force) > gnus-request-move-article(8673 "incoming" "mailse01.axis.se" (gnus-request-accept-article "INBOX" (quote nil) t t) t) > gnus-summary-move-article(nil) > * call-interactively(gnus-summary-move-article) > >> Type 'c' to continue. > > Once again, I end up in the debugger. I hit 'c' again. > >> Take a look at the message buffer. gnus-agent-expire should have >> printed messages telling you that it removed the NOV entry for the >> article in question. Did it? > > No. The contents of the message buffer (after having exited > and re-entered the "incoming" group says): > > Moving to INBOX: (8673)... > Loading gnus-dup...done > Entering debugger... > [2 times] > Mark set [2 times] > iswitchb-read-buffer: Quit > Continuing. > Expiring articles in incoming > gnus-agent-expire: Loading overview... Done > gnus-agent-expire: Sorting entries... Done > gnus-agent-expire: Merging entries... Done > Expiry...done > Entering debugger... > [2 times] > Mark set [2 times] > Continuing. > nnimap: Setting marks in INBOX...done > nnimap: Updating info for INBOX...done > nnimap: Setting marks in incoming...done > Retrieving newsgroup: incoming... > nnimap: Updating info for incoming...done > Fetching headers for incoming...done > Generating summary...done > No more unread articles > >>> 4. Exit the group. >>> >>> 5. Re-enter the same group. The second message shows up in >>> the summary buffer, but it not possible to look at it. >>> It has not been downloaded and it is not on the server. >>> >>> This only happens if gnus-agent-mark-unread-after-downloaded >>> is nil. Setting g-a-m-u-a-d to non-nil makes the problem go >>> away. >> >> You should be using revision 6.142, or later, of >> gnus-agent.el. Are you? > > Yup: > > $ cvs status gnus-agent.el > =================================================================== > File: gnus-agent.el Status: Up-to-date > > Working revision: 6.142 > Repository revision: 6.142 /usr/local/cvsroot/gnus/lisp/gnus-agent.el,v > Sticky Tag: (none) > Sticky Date: (none) > Sticky Options: (none) > > Before doing the steps above I disabled all spam > functionality, exited emacs, downloaded a fresh gnus from > CVS, built it, installed it and then started emacs. > > Niklas Niklas, The problem is that gnus-agent-expire failed to expire the article. That shouldn't happen when the third parameter is force. It implies that either g-a-e is broken or that 8673 was never fetched into the agent. I've attached a copy of just g-a-e from revision 6.136 as I've been concerned with some recent changes to it. Either append this file to your gnus-agent.el or load it AFTER opening a group. Once you've loaded it, set gnus-verbose to 7 and repeat the 'B m' test. --=-=-= Content-Type: application/emacs-lisp Content-Disposition: attachment; filename=gnus-agent-expire.el Content-Transfer-Encoding: 8bit Content-Description: Old version of gnus-agent-expire (defun gnus-agent-expire (&optional articles group force) "Expire all old articles. If you want to force expiring of certain articles, this function can take ARTICLES, GROUP and FORCE parameters as well. The articles on which the expiration process runs are selected as follows: if ARTICLES is null, all read and unmarked articles. if ARTICLES is t, all articles. if ARTICLES is a list, just those articles. Setting GROUP will limit expiration to that group. FORCE is equivalent to setting gnus-agent-expire-days to zero(0)." (interactive) (if (or (not (eq articles t)) (yes-or-no-p (concat "Are you sure that you want to expire all articles in " (if group group "every agentized group") "."))) (let ((methods (if group (list (gnus-find-method-for-group group)) gnus-agent-covered-methods)) (day (if (numberp gnus-agent-expire-days) (- (time-to-days (current-time)) gnus-agent-expire-days) nil)) gnus-command-method sym arts pos history overview file histories elem art nov-file low info unreads marked article orig lowest highest found days) (save-excursion (setq overview (gnus-get-buffer-create " *expire overview*")) (unwind-protect (while (setq gnus-command-method (pop methods)) (when (file-exists-p (gnus-agent-lib-file "active")) (with-temp-buffer (nnheader-insert-file-contents (gnus-agent-lib-file "active")) (gnus-active-to-gnus-format gnus-command-method (setq orig (gnus-make-hashtable (count-lines (point-min) (point-max)))))) (dolist (expiring-group (gnus-groups-from-server gnus-command-method)) (if (or (not group) (equal group expiring-group)) (let* ((dir (concat (gnus-agent-directory) (gnus-agent-group-path expiring-group) "/")) (active (gnus-gethash-safe expiring-group orig)) (day (if (numberp day) day (let (found (days gnus-agent-expire-days)) (catch 'found (while (and (not found) days) (when (eq 0 (string-match (caar days) expiring-group)) (throw 'found (- (time-to-days (current-time)) (cadar days)))) (pop days)) ;; No regexp matched so set ;; a limit that will block ;; expiration in this group. 0))))) (when active (gnus-agent-load-alist expiring-group) (gnus-message 5 "Expiring articles in %s" expiring-group) (let* ((info (gnus-get-info expiring-group)) (alist gnus-agent-article-alist) (specials (if alist (list (caar (last alist))))) (unreads ;; Articles that are excluded from the expiration process (cond (gnus-agent-expire-all ;; All articles are marked read by global decree nil) ((eq articles t) ;; All articles are marked read by function parameter nil) ((not articles) ;; Unread articles are marked protected from expiration ;; Don't call gnus-list-of-unread-articles as it returns articles that have not been fetched into the agent. (ignore-errors (gnus-agent-unread-articles expiring-group))) (t ;; All articles EXCEPT those named by the caller are protected from expiration (gnus-sorted-difference (gnus-uncompress-range (cons (caar alist) (caar (last alist)))) (sort articles '<))))) (marked ;; More articles that are exluded from the expiration process (cond (gnus-agent-expire-all ;; All articles are unmarked by global decree nil) ((eq articles t) ;; All articles are unmarked by function parameter nil) (articles ;; All articles may as well be unmarked as the unreads list already names the articles we are going to keep nil) (t ;; Ticked and/or dormant articles are excluded from expiration (nconc (gnus-uncompress-range (cdr (assq 'tick (gnus-info-marks info)))) (gnus-uncompress-range (cdr (assq 'dormant (gnus-info-marks info)))))))) (nov-file (concat dir ".overview")) (cnt 0) (completed -1) dlist type) ;; The normal article alist contains ;; elements that look like (article# . ;; fetch_date) I need to combine other ;; information with this list. For ;; example, a flag indicating that a ;; particular article MUST BE KEPT. To ;; do this, I'm going to transform the ;; elements to look like (article# ;; fetch_date keep_flag ;; NOV_entry_marker) Later, I'll reverse ;; the process to generate the expired ;; article alist. ;; Convert the alist elements to ;; (article# fetch_date nil nil). (setq dlist (mapcar (lambda (e) (list (car e) (cdr e) nil nil)) alist)) ;; Convert the keep lists to elements ;; that look like (article# nil ;; keep_flag nil) then append it to the ;; expanded dlist These statements are ;; sorted by ascending precidence of the ;; keep_flag. (setq dlist (nconc dlist (mapcar (lambda (e) (list e nil 'unread nil)) unreads))) (setq dlist (nconc dlist (mapcar (lambda (e) (list e nil 'marked nil)) marked))) (setq dlist (nconc dlist (mapcar (lambda (e) (list e nil 'special nil)) specials))) (set-buffer overview) (erase-buffer) (when (file-exists-p nov-file) (gnus-message 7 "gnus-agent-expire: Loading overview...") (nnheader-insert-file-contents nov-file) (goto-char (point-min)) (let (p) (while (< (setq p (point)) (point-max)) (condition-case nil ;; If I successfully read an ;; integer (the plus zero ;; ensures a numeric type), ;; prepend a marker entry to ;; the list (push (list (+ 0 (read (current-buffer))) nil nil (set-marker (make-marker) p)) dlist) (error (gnus-message 1 "gnus-agent-expire: read error occurred when reading expression at %s in %s. Skipping to next line." (point) nov-file))) ;; Whether I succeeded, or failed, ;; it doesn't matter. Move to the ;; next line then try again. (forward-line 1))) (gnus-message 7 "gnus-agent-expire: Loading overview... Done")) (set-buffer-modified-p nil) ;; At this point, all of the information ;; is in dlist. The only problem is ;; that much of it is spread across ;; multiple entries. Sort then MERGE!! (gnus-message 7 "gnus-agent-expire: Sorting entries... ") ;; If two entries have the same ;; article-number then sort by ascending ;; keep_flag. (let ((special 0) (marked 1) (unread 2)) (setq dlist (sort dlist (lambda (a b) (cond ((< (nth 0 a) (nth 0 b)) t) ((> (nth 0 a) (nth 0 b)) nil) (t (let ((a (or (symbol-value (nth 2 a)) 3)) (b (or (symbol-value (nth 2 b)) 3))) (<= a b)))))))) (gnus-message 7 "gnus-agent-expire: Sorting entries... Done") (gnus-message 7 "gnus-agent-expire: Merging entries... ") (let ((dlist dlist)) (while (cdr dlist) ; I'm not at the end-of-list (if (eq (caar dlist) (caadr dlist)) (let ((first (cdr (car dlist))) (secnd (cdr (cadr dlist)))) (setcar first (or (car first) (car secnd))) ; fetch_date (setq first (cdr first) secnd (cdr secnd)) (setcar first (or (car first) (car secnd))) ; Keep_flag (setq first (cdr first) secnd (cdr secnd)) (setcar first (or (car first) (car secnd))) ; NOV_entry_marker (setcdr dlist (cddr dlist))) (setq dlist (cdr dlist))))) (gnus-message 7 "gnus-agent-expire: Merging entries... Done") (let* ((len (float (length dlist))) (alist (list nil)) (tail-alist alist)) (while dlist (let ((new-completed (truncate (* 100.0 (/ (setq cnt (1+ cnt)) len))))) (when (> new-completed completed) (setq completed new-completed) (gnus-message 9 "%3d%% completed..." completed))) (let* ((entry (car dlist)) (article-number (nth 0 entry)) (fetch-date (nth 1 entry)) (keep (nth 2 entry)) (marker (nth 3 entry))) (cond ;; Kept articles are unread, marked, or special. (keep (when fetch-date (unless (file-exists-p (concat dir (number-to-string article-number))) (setf (nth 1 entry) nil) (gnus-message 3 "gnus-agent-expire cleared download flag on article %d as the cached article file is missing." (caar dlist))) (unless marker (gnus-message 1 "gnus-agent-expire detected a missing NOV entry. Run gnus-agent-regenerate-group to restore it."))) (gnus-agent-append-to-list tail-alist (cons article-number fetch-date))) ;; The following articles are READ, UNMARKED, and ORDINARY. ;; See if they can be EXPIRED!!! ((setq type (cond ((not (integerp fetch-date)) 'read) ;; never fetched article (may expire right now) ((not (file-exists-p (concat dir (number-to-string article-number)))) (setf (nth 1 entry) nil) 'externally-expired) ;; Can't find the cached article. Handle case as though this article was never fetched. ;; We now have the arrival day, so we see ;; whether it's old enough to be expired. ((< fetch-date day) 'expired) (force 'forced))) ;; I found some reason to expire this entry. (let ((actions nil)) (when (memq type '(forced expired)) (ignore-errors ; Just being paranoid. (delete-file (concat dir (number-to-string article-number))) (push "expired cached article" actions)) (setf (nth 1 entry) nil) ) (when marker (push "NOV entry removed" actions) (goto-char marker) (gnus-delete-line)) ;; If considering all articles is set, I can only expire article IDs that are no longer in the active range. (if (and gnus-agent-consider-all-articles (>= article-number (car active))) ;; I have to keep this ID in the alist (gnus-agent-append-to-list tail-alist (cons article-number fetch-date)) (push (format "Removed %s article number from article alist" type) actions)) (gnus-message 7 "gnus-agent-expire: Article %d: %s" article-number (mapconcat 'identity actions ", ")))) (t (gnus-agent-append-to-list tail-alist (cons article-number fetch-date))) ) ;; Clean up markers as I want to recycle this buffer over several groups. (when marker (set-marker marker nil)) (setq dlist (cdr dlist)))) (setq alist (cdr alist)) (let ((inhibit-quit t)) (unless (equal alist gnus-agent-article-alist) (setq gnus-agent-article-alist alist) (gnus-agent-save-alist expiring-group)) (when (buffer-modified-p) (let ((coding-system-for-write gnus-agent-file-coding-system)) (gnus-make-directory dir) (write-region (point-min) (point-max) nov-file nil 'silent) ;; clear the modified flag as that I'm not confused by its status on the next pass through this routine. (set-buffer-modified-p nil))) (when (eq articles t) (gnus-summary-update-info))))))))))) (kill-buffer overview))))) (gnus-message 4 "Expiry...done")) --=-=-= Thanks, Kevin --=-=-=--