nnfolder slow to append to large groups

nnfolder slow to append to large groups

[Dan Christensen]

I use nnfolder to archive messages, and I have one group in particular
that I regularly move messages to whose file is almost 13MB in size.
Each time I move a message there, it takes 2 or 3 seconds until I can
use emacs again, so I'm wondering whether this can be sped up at all.

To be precise, if I move a message from an otherwise empty group, it
takes about 1.5 to 2 seconds.  If I move from a group which is 1MB in
size, it take 2.5 to 3 seconds.  This is with everything in cache.

Just editing the big file directly in emacs and saving a changed
version takes about 1 second.

Copying the file with "cp bigfile bigfile2; sync" takes 1.04s elapsed
time, roughly in line with saving the file in emacs.  Without the sync,
it is 0.07s.

As another data point, appending to a file this large with, e.g., 
"echo foo >> bigfile; sync" takes 0.018s elapsed time.

All of these are faster than the move using Gnus.  First, my question:
why should the move take any longer than about 1.04s when I am moving
from an otherwise empty group?  Resaving the file for the empty group
should be instantaneous, so it seems like Gnus is adding some overhead
somewhere.  Any ideas on how to fix this?  ...  Hmm, it looks to me like
Gnus kills the buffer for the file when not using it, and so the file
needs to be reread between operations.  Is this true??  If so, is it
intentional?

Other ideas for improvement:

1) Telling emacs not to call fsync by setting write-region-inhibit-fsync
to t helps a lot, but it still takes about 1 second.  And while I would
be comfortable having this set to t for mail files, I'm not sure about
setting it generally.  Would an option like nnfolder-inhibit-fsync which
tells nnfolder-save-buffer to temporarily set write-region-inhibit-fsync
be a horrible idea?

2) Even better would be if emacs could signal to the OS to sync the
file, but to do so asynchronously.  Is this possible?

3) Another minor factor is that since gnus seems to kill the buffer
between uses, a backup file is created each time.  But I don't think
this causes much overhead, since in my tests emacs saves a file in
about the same amount of time whether or not a backup is being updated.

4) The operation that is happening here is an append for the destination
group (which is the large one).  In my tests, append-to-file is
extremely fast.  Could nnfolder be taught to use this for the
destination group??  Backups wouldn't be made in this case, but maybe
that's ok, or maybe nnfolder could do a backup if it's the first change
to the file during this session.

This last idea would be a huge speed-up.  Thoughts?  Which function is
it that is doing the insertion of the new article?

Dan