Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom <quanstro@quanstro.net>]

> at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

i have a couple of reservations about mbox format.  first, a majority
of users that i need to support have inboxes >10mb and some
have mailboxes >100mb.  thus deleting an older message will require
rewriting the whole file, and if it's not the last message, will take
up quite a bit of storage, even on venti.

second, it's very difficult to cache.  for example, suppose i have two
instances of my mail fs interpreting the same mbox at the same time.
suppose that i delete the 5th out of 500 messages with the first.  the
second will then have to reread the entire mbox to get its pointers
right.  even if i write an index with the first, the entire index
needs to be reread.

third, since large messages tend to be really stinking huge, it is not
efficient to read entire messages to deduce the mime structure.  (ned
reads the mime structure of each doc on startup.) that last mime part
may be tens of mb into the

you'll notice that if one uses email in the way it was used
traditionally in unix environments, that a 500 message, inbox or 15mb
message is unreasonable.  for traditional uses, mbox format and
loading the whole stinking thing into memory is a great idea.

unfortunately, that's now how people use email these days.
our two machines running upas/fs burn up to 5gb of ram.
old rewritten mail adds several hundred mb/day to the dump.

> also file lengths is going to be a problem if i'm going to decode files within the fs.
>
> i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs

i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
mailbox types that support caching (mdir and imap4) keep their cache
below Maxcache or Σ of (actively) referenced messages, whichever is bigger.

i've also adapted the rest of upas, including deliver and marshal to
understand mdir format.  imap4d should be done soon.

this upas/fs keeps an index, so in most cases, emails are not reread unless they are
viewed.  the index contains some information -- like flags -- not recorded
in the mail.

i'd be happy to share a working copy to those interested.  but it's
not ready to be inflicted on the world yet.  the "From " line date is
unresolved and imap4d still thinks it knows how to append a message
to a mailbox.

> about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?

why not?  this seems an arbitrary distinction between the inbox and
every other mailbox.

- erik