Re: [9fans] Scaleable mail repositories.

[C H Forsyth <forsyth@vitanuova.com>]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

i don't understand why, now that it's easy to write file servers
(compared to unix days), it's necessary to store the mail messages
as actual separate files or directories.  the main problem with
upas/fs i find is that it rewrites the file instead of treating it
as append-only, and it reads the whole thing into memory (in a moderately
bulky format); rather than maintaining a separate index file or files,
and loading as needed.   both the storage and index structure can
then be made suitable for the task.

[-- Attachment #2: Type: message/rfc822, Size: 3962 bytes --]

From: Lyndon Nerenberg <lyndon@orthanc.ca>
To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu>
Subject: [9fans] Scaleable mail repositories.
Date: Sun, 30 Oct 2005 20:06:37 -0800
Message-ID: <D99CBD3A-2049-4C0C-913F-93151D7D56EE@orthanc.ca>


On Oct 29, 2005, at 6:18 PM, Paul Lalonde wrote:

> I used to keep all my mail this way using MH; it worked well up to  
> the point where directories got so full that directory operations  
> were taking too long (I remember a nasty n^2 sort in the early days  
> of linux).  If Plan9's file handling is up to it, I'm in favour of  
> this approach.  Just remember that it has to work well with 10's of  
> thousands of emails in a directory - search is making organization  
> obsolete.  Heck, some of us didn't manage organization before  
> search made it obsolete.

The trick to making this work well is to do what the Cyrus IMAP  
server does: use the MH style one-message-per-file layout, and keep  
an index cache of the commonly accessed items (from, to, date, mime  
structure).  It's fast, and it scales very well.  When I was at  
Messaging Direct we sold a commercial version of the Cyrus server.   
We later designed our own IMAP server, but we kept the Cyrus file and  
cache layout as it was still the fastest and most scalable solution  
to the problem.

--lyndon