From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/24223 Path: main.gmane.org!not-for-mail From: Jim Pick Newsgroups: gmane.emacs.gnus.general Subject: [PATCH] Mail fetching on memory-poor machines Date: 13 Jul 1999 22:13:12 -0700 Sender: owner-ding@hpc.uh.edu Message-ID: <87hfn7stjr.fsf@pepper.jimpick.com> NNTP-Posting-Host: coloc-standby.netfonds.no Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: main.gmane.org 1035161826 7535 80.91.224.250 (21 Oct 2002 00:57:06 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Mon, 21 Oct 2002 00:57:06 +0000 (UTC) Return-Path: Original-Received: from farabi.math.uh.edu (farabi.math.uh.edu [129.7.128.57]) by sclp3.sclp.com (8.8.5/8.8.5) with ESMTP id BAA23732 for ; Wed, 14 Jul 1999 01:14:40 -0400 (EDT) Original-Received: from sina.hpc.uh.edu (lists@Sina.HPC.UH.EDU [129.7.3.5]) by farabi.math.uh.edu (8.9.1/8.9.1) with ESMTP id AAB28371; Wed, 14 Jul 1999 00:14:23 -0500 (CDT) Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Wed, 14 Jul 1999 00:14:56 -0500 (CDT) Original-Received: from sclp3.sclp.com (root@sclp3.sclp.com [204.252.123.139]) by sina.hpc.uh.edu (8.9.3/8.9.3) with ESMTP id AAA12248 for ; Wed, 14 Jul 1999 00:14:45 -0500 (CDT) Original-Received: from tia.jimpick.com (tia.jimpick.com [204.209.212.111]) by sclp3.sclp.com (8.8.5/8.8.5) with ESMTP id BAA23706 for ; Wed, 14 Jul 1999 01:13:41 -0400 (EDT) Original-Received: from (pepper.jimpick.com) [204.209.212.121] by tia.jimpick.com with esmtp (Exim 2.05 #1) id 114HMG-0000Ql-00 (Debian); Tue, 13 Jul 1999 22:13:29 -0700 Original-Received: from jim by pepper.jimpick.com with local (Exim 3.02 #1 (Debian)) id 114HM0-00068K-00; Tue, 13 Jul 1999 22:13:12 -0700 Original-To: ding@gnus.org X-Url: http://www.jimpick.com/ Original-Lines: 132 User-Agent: Gnus/5.070095 (Pterodactyl Gnus v0.95) Emacs/20.3 Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:24223 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:24223 Hi, I subscribe to a lot of mailing lists. I get about 6MB of new email on a slow day, which I sort using mailagent into mbox files in a spool directory. I then pull it all into Gnus with the following: (setq mail-sources '((directory :path "~/var/spool/mail" :suffix ""))) I'm using an nnfolder backend. The _big_ problem with what I was doing was that it was starting to chew up a huge amount of memory as my nnfolder's grew. I'm running this on a laptop with only 64MB (laptop memory is pricey). For every file in the incoming spool directory, the mail-source-fetch-directory method creates a new buffer in which it appends the new messages to the old nnfolder. Unfortunately, these buffers don't get saved and killed until after _all_ the mail has been fetched. Unfortunately, my nnfolders have grown to the point where they are several times larger than my physical memory. Ugh. I'm in big trouble if new messages arrive for more than a few of them at once. Over the last few days, I've had a number of instances where I tried to scan in new mail, causing emacs swapped to death - and I lost mail. This probably isn't as big a problem with the nnml backend, because the size of the files in the working set there is much smaller. I want to use nnfolders as my backend though (quicker backups). Since I don't use gnus to sort my mail, I don't really see the point in reading in multiple nnfolders into memory, and then only saving them at the end of the process. Rather, it makes more sense to save the buffers after each file in the spool directory has been read in. There doesn't appear to be a way to tell the mail-source-fetch-directory method to do this. So I created a little patch to achieve what I want. The patch probably isn't general enough for everybody. I'm just sending it in to show how I solved my problem. With this patch, Gnus is almost perfect for me. :-) Cheers, - Jim diff -urN pgnus-0.95.backup/lisp/mail-source.el pgnus-0.95/lisp/mail-source.el --- pgnus-0.95.backup/lisp/mail-source.el Fri Jul 9 15:22:57 1999 +++ pgnus-0.95/lisp/mail-source.el Tue Jul 13 21:15:53 1999 @@ -163,7 +163,7 @@ (t value))) -(defun mail-source-fetch (source callback) +(defun mail-source-fetch (source callback exit-func) "Fetch mail from SOURCE and call CALLBACK zero or more times. CALLBACK will be called with the name of the file where (some of) the mail from SOURCE is put. @@ -180,7 +180,7 @@ callback mail-source-crash-box))) (+ found (condition-case err - (funcall function source callback) + (funcall function source callback exit-func) (error (unless (yes-or-no-p (format "Mail source error (%s). Continue? " err)) @@ -325,7 +325,7 @@ ;;; Different fetchers ;;; -(defun mail-source-fetch-file (source callback) +(defun mail-source-fetch-file (source callback exit-func) "Fetcher for single-file sources." (mail-source-bind (file source) (mail-source-run-script @@ -339,7 +339,7 @@ postscript (format-spec-make ?t mail-source-crash-box))) 0)))) -(defun mail-source-fetch-directory (source callback) +(defun mail-source-fetch-directory (source callback exit-func) "Fetcher for directory sources." (mail-source-bind (directory source) (let ((found 0) @@ -349,10 +349,12 @@ (when (and (file-regular-p file) (funcall predicate file) (mail-source-movemail file mail-source-crash-box)) - (incf found (mail-source-callback callback file)))) + (incf found (mail-source-callback callback file)) + (when exit-func + (funcall exit-func)))) found))) -(defun mail-source-fetch-pop (source callback) +(defun mail-source-fetch-pop (source callback exit-func) "Fetcher for single-file sources." (mail-source-bind (pop source) (mail-source-run-script @@ -406,7 +408,7 @@ mail-source-password-cache)) 0)))) -(defun mail-source-fetch-maildir (source callback) +(defun mail-source-fetch-maildir (source callback exit-func) "Fetcher for maildir sources." (mail-source-bind (maildir source) (let ((found 0) diff -urN pgnus-0.95.backup/lisp/nnmail.el pgnus-0.95/lisp/nnmail.el --- pgnus-0.95.backup/lisp/nnmail.el Wed Jul 7 05:58:15 1999 +++ pgnus-0.95/lisp/nnmail.el Tue Jul 13 21:17:29 1999 @@ -1443,7 +1443,8 @@ file ',(intern (format "%s-save-mail" method)) ',spool-func (nnmail-get-split-group orig-file source) - ',(intern (format "%s-active-number" method)))))) + ',(intern (format "%s-active-number" method)))) + exit-func)) (incf total new) (incf i)))) ;; If we did indeed read any incoming spools, we save all info.