[PATCH] Mail fetching on memory-poor machines

[Jim Pick <jim@jimpick.com>]

Hi,

I subscribe to a lot of mailing lists.  I get about 6MB of new email
on a slow day, which I sort using mailagent into mbox files in a spool
directory.  I then pull it all into Gnus with the following:

(setq mail-sources
          '((directory :path "~/var/spool/mail"
                     :suffix "")))

I'm using an nnfolder backend.

The _big_ problem with what I was doing was that it was starting to
chew up a huge amount of memory as my nnfolder's grew.  I'm running
this on a laptop with only 64MB (laptop memory is pricey).

For every file in the incoming spool directory, the
mail-source-fetch-directory method creates a new buffer in which it
appends the new messages to the old nnfolder.  Unfortunately, these
buffers don't get saved and killed until after _all_ the mail has been
fetched.

Unfortunately, my nnfolders have grown to the point where they are
several times larger than my physical memory.  Ugh.  I'm in big
trouble if new messages arrive for more than a few of them at once.
Over the last few days, I've had a number of instances where I tried
to scan in new mail, causing emacs swapped to death - and I lost mail.

This probably isn't as big a problem with the nnml backend, because
the size of the files in the working set there is much smaller.  I
want to use nnfolders as my backend though (quicker backups).

Since I don't use gnus to sort my mail, I don't really see the point
in reading in multiple nnfolders into memory, and then only saving
them at the end of the process.  Rather, it makes more sense to save
the buffers after each file in the spool directory has been read in.

There doesn't appear to be a way to tell the
mail-source-fetch-directory method to do this.  So I created a little
patch to achieve what I want.

The patch probably isn't general enough for everybody.  I'm just
sending it in to show how I solved my problem.  With this patch, Gnus
is almost perfect for me.  :-)

Cheers,

 - Jim


diff -urN pgnus-0.95.backup/lisp/mail-source.el pgnus-0.95/lisp/mail-source.el
--- pgnus-0.95.backup/lisp/mail-source.el	Fri Jul  9 15:22:57 1999
+++ pgnus-0.95/lisp/mail-source.el	Tue Jul 13 21:15:53 1999
@@ -163,7 +163,7 @@
    (t
     value)))
 
-(defun mail-source-fetch (source callback)
+(defun mail-source-fetch (source callback exit-func)
   "Fetch mail from SOURCE and call CALLBACK zero or more times.
 CALLBACK will be called with the name of the file where (some of)
 the mail from SOURCE is put.
@@ -180,7 +180,7 @@
                      callback mail-source-crash-box)))
       (+ found
          (condition-case err
-             (funcall function source callback)
+             (funcall function source callback exit-func)
            (error
             (unless (yes-or-no-p
 		     (format "Mail source error (%s).  Continue? " err))
@@ -325,7 +325,7 @@
 ;;; Different fetchers
 ;;;
 
-(defun mail-source-fetch-file (source callback)
+(defun mail-source-fetch-file (source callback exit-func)
   "Fetcher for single-file sources."
   (mail-source-bind (file source)
     (mail-source-run-script
@@ -339,7 +339,7 @@
 	     postscript (format-spec-make ?t mail-source-crash-box)))
 	0))))
 
-(defun mail-source-fetch-directory (source callback)
+(defun mail-source-fetch-directory (source callback exit-func)
   "Fetcher for directory sources."
   (mail-source-bind (directory source)
     (let ((found 0)
@@ -349,10 +349,12 @@
 	(when (and (file-regular-p file)
 		   (funcall predicate file)
 		   (mail-source-movemail file mail-source-crash-box))
-	  (incf found (mail-source-callback callback file))))
+	  (incf found (mail-source-callback callback file))
+	  (when exit-func
+	    (funcall exit-func))))
       found)))
 
-(defun mail-source-fetch-pop (source callback)
+(defun mail-source-fetch-pop (source callback exit-func)
   "Fetcher for single-file sources."
   (mail-source-bind (pop source)
     (mail-source-run-script
@@ -406,7 +408,7 @@
 		    mail-source-password-cache))
 	0))))
 
-(defun mail-source-fetch-maildir (source callback)
+(defun mail-source-fetch-maildir (source callback exit-func)
   "Fetcher for maildir sources."
   (mail-source-bind (maildir source)
     (let ((found 0)
diff -urN pgnus-0.95.backup/lisp/nnmail.el pgnus-0.95/lisp/nnmail.el
--- pgnus-0.95.backup/lisp/nnmail.el	Wed Jul  7 05:58:15 1999
+++ pgnus-0.95/lisp/nnmail.el	Tue Jul 13 21:17:29 1999
@@ -1443,7 +1443,8 @@
 			   file ',(intern (format "%s-save-mail" method))
 			   ',spool-func
 			   (nnmail-get-split-group orig-file source)
-			   ',(intern (format "%s-active-number" method))))))
+			   ',(intern (format "%s-active-number" method))))
+		       exit-func))
 	    (incf total new)
 	    (incf i))))
       ;; If we did indeed read any incoming spools, we save all info.