[PATCH] Mail fetching on memory-poor machines

[PATCH] Mail fetching on memory-poor machines

[Jim Pick]

Hi,

I subscribe to a lot of mailing lists.  I get about 6MB of new email
on a slow day, which I sort using mailagent into mbox files in a spool
directory.  I then pull it all into Gnus with the following:

(setq mail-sources
          '((directory :path "~/var/spool/mail"
                     :suffix "")))

I'm using an nnfolder backend.

The _big_ problem with what I was doing was that it was starting to
chew up a huge amount of memory as my nnfolder's grew.  I'm running
this on a laptop with only 64MB (laptop memory is pricey).

For every file in the incoming spool directory, the
mail-source-fetch-directory method creates a new buffer in which it
appends the new messages to the old nnfolder.  Unfortunately, these
buffers don't get saved and killed until after _all_ the mail has been
fetched.

Unfortunately, my nnfolders have grown to the point where they are
several times larger than my physical memory.  Ugh.  I'm in big
trouble if new messages arrive for more than a few of them at once.
Over the last few days, I've had a number of instances where I tried
to scan in new mail, causing emacs swapped to death - and I lost mail.

This probably isn't as big a problem with the nnml backend, because
the size of the files in the working set there is much smaller.  I
want to use nnfolders as my backend though (quicker backups).

Since I don't use gnus to sort my mail, I don't really see the point
in reading in multiple nnfolders into memory, and then only saving
them at the end of the process.  Rather, it makes more sense to save
the buffers after each file in the spool directory has been read in.

There doesn't appear to be a way to tell the
mail-source-fetch-directory method to do this.  So I created a little
patch to achieve what I want.

The patch probably isn't general enough for everybody.  I'm just
sending it in to show how I solved my problem.  With this patch, Gnus
is almost perfect for me.  :-)

Cheers,

 - Jim


diff -urN pgnus-0.95.backup/lisp/mail-source.el pgnus-0.95/lisp/mail-source.el
--- pgnus-0.95.backup/lisp/mail-source.el	Fri Jul  9 15:22:57 1999
+++ pgnus-0.95/lisp/mail-source.el	Tue Jul 13 21:15:53 1999
@@ -163,7 +163,7 @@
    (t
     value)))
 
-(defun mail-source-fetch (source callback)
+(defun mail-source-fetch (source callback exit-func)
   "Fetch mail from SOURCE and call CALLBACK zero or more times.
 CALLBACK will be called with the name of the file where (some of)
 the mail from SOURCE is put.
@@ -180,7 +180,7 @@
                      callback mail-source-crash-box)))
       (+ found
          (condition-case err
-             (funcall function source callback)
+             (funcall function source callback exit-func)
            (error
             (unless (yes-or-no-p
 		     (format "Mail source error (%s).  Continue? " err))
@@ -325,7 +325,7 @@
 ;;; Different fetchers
 ;;;
 
-(defun mail-source-fetch-file (source callback)
+(defun mail-source-fetch-file (source callback exit-func)
   "Fetcher for single-file sources."
   (mail-source-bind (file source)
     (mail-source-run-script
@@ -339,7 +339,7 @@
 	     postscript (format-spec-make ?t mail-source-crash-box)))
 	0))))
 
-(defun mail-source-fetch-directory (source callback)
+(defun mail-source-fetch-directory (source callback exit-func)
   "Fetcher for directory sources."
   (mail-source-bind (directory source)
     (let ((found 0)
@@ -349,10 +349,12 @@
 	(when (and (file-regular-p file)
 		   (funcall predicate file)
 		   (mail-source-movemail file mail-source-crash-box))
-	  (incf found (mail-source-callback callback file))))
+	  (incf found (mail-source-callback callback file))
+	  (when exit-func
+	    (funcall exit-func))))
       found)))
 
-(defun mail-source-fetch-pop (source callback)
+(defun mail-source-fetch-pop (source callback exit-func)
   "Fetcher for single-file sources."
   (mail-source-bind (pop source)
     (mail-source-run-script
@@ -406,7 +408,7 @@
 		    mail-source-password-cache))
 	0))))
 
-(defun mail-source-fetch-maildir (source callback)
+(defun mail-source-fetch-maildir (source callback exit-func)
   "Fetcher for maildir sources."
   (mail-source-bind (maildir source)
     (let ((found 0)
diff -urN pgnus-0.95.backup/lisp/nnmail.el pgnus-0.95/lisp/nnmail.el
--- pgnus-0.95.backup/lisp/nnmail.el	Wed Jul  7 05:58:15 1999
+++ pgnus-0.95/lisp/nnmail.el	Tue Jul 13 21:17:29 1999
@@ -1443,7 +1443,8 @@
 			   file ',(intern (format "%s-save-mail" method))
 			   ',spool-func
 			   (nnmail-get-split-group orig-file source)
-			   ',(intern (format "%s-active-number" method))))))
+			   ',(intern (format "%s-active-number" method))))
+		       exit-func))
 	    (incf total new)
 	    (incf i))))
       ;; If we did indeed read any incoming spools, we save all info.

Re: [PATCH] Mail fetching on memory-poor machines

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Jim Pick <jim@jimpick.com>  on Wed, 14 Jul 1999
| I'm using an nnfolder backend.

Don't use nnfolder.  Use nnml instead.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.8 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3jN7Rgl+vIlSVSNkRAgbIAKCoysam/okn/yjbsP/4uWQoJM7s9gCfY7rI
AR0E+PTm3GMl/pFluXKJlyg=
=qDns
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: [PATCH] Mail fetching on memory-poor machines

[Jim Pick]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Jim Pick <jim@jimpick.com>  on Wed, 14 Jul 1999
> | I'm using an nnfolder backend.
> 
> Don't use nnfolder.  Use nnml instead.

Bzzt.  Wrong answer.

I like nnfolder.

I'm not going to waste that many inodes.  It takes forever to back it
up using rsync.  I know, I've been using nnml for the past year, and
just recently switched to nnfolder.

My patch fixes the problem for me.  Spectacularily.  I presented it to
the list so that others might benefit.

Why not fix the problem?  The results of the problem are so bad that
it's basically a bug.  Fixing it seems like a more logical thing to
do, rather than force everybody to use the program in a certain way.

(granted, my fix might not be optimal - if I got some feedback on what
was wrong or right with the approach, I might develop a patch suitable
for inclusion)

Cheers,

 - Jim

Re: [PATCH] Mail fetching on memory-poor machines

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Jim Pick <jim@jimpick.com>  on Wed, 14 Jul 1999
| Why not fix the problem?  The results of the problem are so bad that
| it's basically a bug.  Fixing it seems like a more logical thing to
| do, rather than force everybody to use the program in a certain way.

Because, if I understand what your code does (which might not be the case),
it only works for people who use Gnus "your" way while slowing it down
unnecessarilly for those who are not on memory-starved systems, have large
nnfolders, and do use Gnus to split mail sources.

Personally, I am all for lean and mean, which is one of the reasons why I
don't use nnfolder.  I tried it, briefly, back when I was on a 12MB system.
That was a mistake :).

For what its worth, inode starvation is not as likely as you might think.
The average mail message is ~3-4kB; the average inode size on a modern
filesystem is 4kB.  One message, one inode.  On an ~1GB filesystem you will
have around 256k inodes (more, actually).  In practical terms, you will run
out of space before you run out of inodes, unless your filesystem is tuned
with unusually large inodes or you are suffering under a woefully
restrictive inode quota.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.8 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3jTd8gl+vIlSVSNkRAq8ZAJ9qy2roxy7JgfnhUDNZrU3sFnnlbwCgz/KX
tFqSmLkn2MR74Xjc8PIa/uw=
=Bsku
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: [PATCH] Mail fetching on memory-poor machines

[Jim Pick]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Jim Pick <jim@jimpick.com>  on Wed, 14 Jul 1999
> | Why not fix the problem?  The results of the problem are so bad that
> | it's basically a bug.  Fixing it seems like a more logical thing to
> | do, rather than force everybody to use the program in a certain way.
> 
> Because, if I understand what your code does (which might not be the case),
> it only works for people who use Gnus "your" way while slowing it down
> unnecessarilly for those who are not on memory-starved systems, have large
> nnfolders, and do use Gnus to split mail sources.

I know.  I stated that the patch wasn't for everybody when I posted
it!

It wouldn't be too hard to put in a customize option to enable/disable
it - but I haven't gotten any feedback on whether this is wanted.  The
patch was ugly enough that I'm not sure that if it should be
integrated as is.

I can think of several different ways of achieving a similar effect.
For example, it might make sense for Gnus to track the size of the
buffers currently in use, and only save the least-recently-used ones
to disk to free up some space.  That's overkill for my case though.

I didn't put a lot of effort into it because Lars might want to do it
some other way.  The patch was just posted to illuminate the problem
(hopefully).

I'll live if the problem isn't fixed upstream - but I fixed it here,
so I thought I'd share.
 
> Personally, I am all for lean and mean, which is one of the reasons why I
> don't use nnfolder.  I tried it, briefly, back when I was on a 12MB system.
> That was a mistake :).

I still like nnfolder.  I don't find it very objectionable at all
(with my patch).

> For what its worth, inode starvation is not as likely as you might think.
> The average mail message is ~3-4kB; the average inode size on a modern
> filesystem is 4kB.  One message, one inode.  On an ~1GB filesystem you will
> have around 256k inodes (more, actually).  In practical terms, you will run
> out of space before you run out of inodes, unless your filesystem is tuned
> with unusually large inodes or you are suffering under a woefully
> restrictive inode quota.

My primary objection to nnml is primarily that it is painful to do
anything that scans the disk.  I've got a very large mail archive.

As an aside - I did manage to run out of inodes once when I was using
nnml before.  I compressed my mail files, and was using jka-compr.
When the files are compressed, they average out to be smaller than
than default 4kB per inode, and it suddenly becomes possible to run
out of inodes.  In that case, I just backed everything up, and
reformatted the partition with more inodes though.  No big deal.

Cheers,

 - Jim

Re: [PATCH] Mail fetching on memory-poor machines

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Jim Pick <jim@jimpick.com>  on Wed, 14 Jul 1999
| I can think of several different ways of achieving a similar effect.
| For example, it might make sense for Gnus to track the size of the
| buffers currently in use, and only save the least-recently-used ones
| to disk to free up some space.  That's overkill for my case though.

Perhaps... but if you were going to go this route, it might be better to be 
a little more heuristic about it.  That is, count how many messages are
added to each folder, and when it comes time to do some pruning, kill the
buffer with the fewest appends.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.8 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3jfPVgl+vIlSVSNkRAoRkAJ9Pt6Axw46aXWDRKJG1gd2nnlYfEQCeO3Xv
en8IF4sUdfIrL0lUHVURvGw=
=tU8D
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.