From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: From: C H Forsyth Date: Mon, 31 Oct 2005 10:55:10 +0000 To: 9fans@cse.psu.edu Subject: Re: [9fans] Scaleable mail repositories. In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="upas-vddxtudmbktypqdrugeyxvxyrl" Topicbox-Message-UUID: a3153f48-ead0-11e9-9d60-3106f5b1d025 This is a multi-part message in MIME format. --upas-vddxtudmbktypqdrugeyxvxyrl Content-Disposition: inline Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit i don't understand why, now that it's easy to write file servers (compared to unix days), it's necessary to store the mail messages as actual separate files or directories. the main problem with upas/fs i find is that it rewrites the file instead of treating it as append-only, and it reads the whole thing into memory (in a moderately bulky format); rather than maintaining a separate index file or files, and loading as needed. both the storage and index structure can then be made suitable for the task. --upas-vddxtudmbktypqdrugeyxvxyrl Content-Type: message/rfc822 Content-Disposition: inline Received: from mail.cse.psu.edu ([130.203.4.6]) by doppio; Mon Oct 31 04:06:23 GMT 2005 Received: from psuvax1.cse.psu.edu (localhost [127.0.0.1]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id DD97363FB7 for ; Sun, 30 Oct 2005 23:07:17 -0500 (EST) X-Original-To: 9fans@cse.psu.edu Delivered-To: 9fans@cse.psu.edu Received: from localhost (localhost [127.0.0.1]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id 3286F63EBB for <9fans@cse.psu.edu>; Sun, 30 Oct 2005 23:06:47 -0500 (EST) Received: from mail.cse.psu.edu ([127.0.0.1]) by localhost (psuvax1 [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 24523-02-56 for <9fans@cse.psu.edu>; Sun, 30 Oct 2005 23:06:43 -0500 (EST) Received: from orthanc.ca (orthanc.ca [209.89.70.53]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id DBEBC63B57 for <9fans@cse.psu.edu>; Sun, 30 Oct 2005 23:06:42 -0500 (EST) Received: from peregrin.orthanc.ca (d64-180-189-53.bchsia.telus.net [64.180.189.53]) (authenticated bits=0) by orthanc.ca (8.13.3/8.13.3) with ESMTP id j9V46dmd061983 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <9fans@cse.psu.edu>; Sun, 30 Oct 2005 21:06:40 -0700 (MST) (envelope-from lyndon@orthanc.ca) Received: from [127.0.0.1] (localhost [127.0.0.1]) by peregrin.orthanc.ca (8.13.5.Beta0/8.13.5.Beta0) with ESMTP id j9V46c3a006306 for <9fans@cse.psu.edu>; Sun, 30 Oct 2005 20:06:38 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v734) In-Reply-To: References: <7552e285cac28e40596d8d18ddb660ce@collyer.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Lyndon Nerenberg Date: Sun, 30 Oct 2005 20:06:37 -0800 To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu> X-Mailer: Apple Mail (2.734) X-Virus-Scanned: by amavisd-new at cse.psu.edu Subject: [9fans] Scaleable mail repositories. X-BeenThere: 9fans@cse.psu.edu X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu> List-Id: Fans of the OS Plan 9 from Bell Labs <9fans.cse.psu.edu> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: 9fans-bounces+forsyth=vitanuova.com@cse.psu.edu Errors-To: 9fans-bounces+forsyth=vitanuova.com@cse.psu.edu On Oct 29, 2005, at 6:18 PM, Paul Lalonde wrote: > I used to keep all my mail this way using MH; it worked well up to > the point where directories got so full that directory operations > were taking too long (I remember a nasty n^2 sort in the early days > of linux). If Plan9's file handling is up to it, I'm in favour of > this approach. Just remember that it has to work well with 10's of > thousands of emails in a directory - search is making organization > obsolete. Heck, some of us didn't manage organization before > search made it obsolete. The trick to making this work well is to do what the Cyrus IMAP server does: use the MH style one-message-per-file layout, and keep an index cache of the commonly accessed items (from, to, date, mime structure). It's fast, and it scales very well. When I was at Messaging Direct we sold a commercial version of the Cyrus server. We later designed our own IMAP server, but we kept the Cyrus file and cache layout as it was still the fastest and most scalable solution to the problem. --lyndon --upas-vddxtudmbktypqdrugeyxvxyrl--