[9fans] store 9p session-only values using lib9p

[9fans] store 9p session-only values using lib9p

[gdiaz]

Hello

I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue)

If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time.

could that value be saved for each session?

i would love to hear from other's experiences,

thanks,

gabi

Re: [9fans] store 9p session-only values using lib9p

[Russ Cox]

> If i save the offset adjustment in f->aux or simmilar,
> i can calculate the real file offset on the next T-reads,
> but that will only work if one client reads it at one time.

No it won't.  Even if just one client is reading, that client might seek.

You could assume that most reads will pick up where the
last one left off, and use f->aux to point at a single hint
that you can check.  Maybe you're willing to do more work
if you do get a seek.  Otherwise you'll need to build a more
complex data structure to help you map between the two.

Also, if f is a Fid* (not a File*), then you get a different f for each
different Topen of the file (and thus also for each client), so the
hint-per-Fid usually works pretty well in practice for this kind
of thing.

Russ

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

hello

you're right, i will think a bit more about it.

thank you very much,

gabi

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> Hello
>
> I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue)
>
> If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time.
>
> could that value be saved for each session?
>
> i would love to hear from other's experiences,

i've been spending a little time with upas/fs.  i don't think that in
general one can get by parsing part of an email, unless that part
happens to be top-level headers.  there are more transforms than
you've mentioned.  apple mail loves to put nulls in html email, for
example.

traditional mbox format is problematic, too.

if you wish to control memory usage (which is a serious problem for
upas/fs), i think the way to go is a cache with reference counting.

the stuff i have isn't ready for prime time, but it's coming along
nicely.

by the way, does anyone know the rational for the date on the
unix "From " line?  upas sets it to the date the message is originally
delivered to the inbox.  moving it from the inbox to another folder
does not change the date.

- erik

Re: [9fans] upas/fs

[Russ Cox]

> by the way, does anyone know the rational for the date on the
> unix "From " line?  upas sets it to the date the message is originally
> delivered to the inbox.  moving it from the inbox to another folder
> does not change the date.

the date is the date it was delivered.
it's a receiver-side postmark.
but you knew that.  what's your question?

russ

Re: [9fans] upas/fs

[erik quanstrom]

>> by the way, does anyone know the rationale for the date on the
>> unix "From " line?  upas sets it to the date the message is originally
>> delivered to the inbox.  moving it from the inbox to another folder
>> does not change the date.
>
> the date is the date it was delivered.
> it's a receiver-side postmark.
> but you knew that.  what's your question?

right.  since the date is attached when delivered to a mailbox,
why doesn't this date change when it's delivered to a secondary
mailbox?  why is the assignment a magical property of the inbox?

- erik

Re: [9fans] upas/fs

[Lyndon Nerenberg]

On 2008-Jun-11, at 19:31 , erik quanstrom wrote:

> right.  since the date is attached when delivered to a mailbox,
> why doesn't this date change when it's delivered to a secondary
> mailbox?  why is the assignment a magical property of the inbox?

Most likely it's just an artifact of the original UNIX mail
implementation. The \n^From<sp> separator line got generated at
initial delivery time, and the mail client used that as the display
time in the message summaries (e.g. Date: not spoken here). Therefore
it makes sense to preserve the initial separator line with it's date
intact to ensure consistency for display purposes. Think of it as the
UNIX ctime of the message.

--lyndon

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

hello

at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

also file lengths is going to be a problem if i'm going to decode files within the fs.

i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs

about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?

thanks,

gabi

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

i have a couple of reservations about mbox format.  first, a majority
of users that i need to support have inboxes >10mb and some
have mailboxes >100mb.  thus deleting an older message will require
rewriting the whole file, and if it's not the last message, will take
up quite a bit of storage, even on venti.

second, it's very difficult to cache.  for example, suppose i have two
instances of my mail fs interpreting the same mbox at the same time.
suppose that i delete the 5th out of 500 messages with the first.  the
second will then have to reread the entire mbox to get its pointers
right.  even if i write an index with the first, the entire index
needs to be reread.

third, since large messages tend to be really stinking huge, it is not
efficient to read entire messages to deduce the mime structure.  (ned
reads the mime structure of each doc on startup.) that last mime part
may be tens of mb into the

you'll notice that if one uses email in the way it was used
traditionally in unix environments, that a 500 message, inbox or 15mb
message is unreasonable.  for traditional uses, mbox format and
loading the whole stinking thing into memory is a great idea.

unfortunately, that's now how people use email these days.
our two machines running upas/fs burn up to 5gb of ram.
old rewritten mail adds several hundred mb/day to the dump.

> also file lengths is going to be a problem if i'm going to decode files within the fs.
>
> i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs

i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
mailbox types that support caching (mdir and imap4) keep their cache
below Maxcache or Σ of (actively) referenced messages, whichever is bigger.

i've also adapted the rest of upas, including deliver and marshal to
understand mdir format.  imap4d should be done soon.

this upas/fs keeps an index, so in most cases, emails are not reread unless they are
viewed.  the index contains some information -- like flags -- not recorded
in the mail.

i'd be happy to share a working copy to those interested.  but it's
not ready to be inflicted on the world yet.  the "From " line date is
unresolved and imap4d still thinks it knows how to append a message
to a mailbox.

> about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?

why not?  this seems an arbitrary distinction between the inbox and
every other mailbox.

- erik

Re: [9fans] store 9p session-only values using lib9p

[Fco. J. Ballesteros]

Funny, I've done the same in a different way.
see mail2fs in contrib/nemo.
Also, I have some proposal, skip to the end of the mail and let me know
what you think :)

In any case, I'd love to see/try your version of upas/fs et al.

Instead of adapting upas/fs, I use a mail2fs program that uses
upas to convert mail into an "unpacked" form. Each mail is a directory.
A "text" file contains the message text right as you would see it in a mail
reader (including relative paths for attachments). Each attach is decoded
and kept in the mail's directory ready to be copied, printed, etc.; if possible,
using the same file name reported by the attachment.

I use the very same approach for sending mail. A directory /mail/box/nemo/out
is spooled to send whichever file is placed on it as a mail. The format for the
outgoing messages is again similar to what you'd type in a mail writer.

The whole point is that now you editor (plus couple of scripts) becomes a mail
reader/writer without understanding anything about mail.

My proposal:
Why don't we reach an agreement and start using the very
same format. I suggest keeping my mbox format but adapting upas/fs to
understand it, which is a good idea. But I'm open to suggestions.

thanks

>  From: quanstro@quanstro.net
>  To: 9fans@9fans.net
>  Reply-To: 9fans@9fans.net
>  Date: Thu Jun 12 12:10:32 CET 2008
>  Subject: Re: [9fans] store 9p session-only values using lib9p
>
>  > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.
>
>  i have a couple of reservations about mbox format. first, a majority
>  of users that i need to support have inboxes >10mb and some
>  have mailboxes >100mb. thus deleting an older message will require
>  rewriting the whole file, and if it's not the last message, will take
>  up quite a bit of storage, even on venti.
>
>  second, it's very difficult to cache. for example, suppose i have two
>  instances of my mail fs interpreting the same mbox at the same time.
>  suppose that i delete the 5th out of 500 messages with the first. the
>  second will then have to reread the entire mbox to get its pointers
>  right. even if i write an index with the first, the entire index
>  needs to be reread.
>
>  third, since large messages tend to be really stinking huge, it is not
>  efficient to read entire messages to deduce the mime structure. (ned
>  reads the mime structure of each doc on startup.) that last mime part
>  may be tens of mb into the
>
>  you'll notice that if one uses email in the way it was used
>  traditionally in unix environments, that a 500 message, inbox or 15mb
>  message is unreasonable. for traditional uses, mbox format and
>  loading the whole stinking thing into memory is a great idea.
>
>  unfortunately, that's now how people use email these days.
>  our two machines running upas/fs burn up to 5gb of ram.
>  old rewritten mail adds several hundred mb/day to the dump.
>
>  > also file lengths is going to be a problem if i'm going to decode files within the fs.
>  >
>  > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs
>
>  i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
>  mailbox types that support caching (mdir and imap4) keep their cache
>  below Maxcache or Σ of (actively) referenced messages, whichever is bigger.
>
>  i've also adapted the rest of upas, including deliver and marshal to
>  understand mdir format. imap4d should be done soon.
>
>  this upas/fs keeps an index, so in most cases, emails are not reread unless they are
>  viewed. the index contains some information -- like flags -- not recorded
>  in the mail.
>
>  i'd be happy to share a working copy to those interested. but it's
>  not ready to be inflicted on the world yet. the "From " line date is
>  unresolved and imap4d still thinks it knows how to append a message
>  to a mailbox.
>
>  > about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?
>
>  why not? this seems an arbitrary distinction between the inbox and
>  every other mailbox.
>
>  - erik
>
>

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

hello


great news, i just used mbox to avoid reworking the whole thing at the same time, but you already done it. I was already aware of the mbox-pain.

i'll be pleased to help on this, test, bug report, try it on a public server or whatever it's needed to finish this up.

About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message. I'm not sure i understand the problem here.

thanks

gabi

Re: [9fans] store 9p session-only values using lib9p

[Fco. J. Ballesteros]

FTS, http://lsub.org/magic/man2html/1/mails
is the manual page for the mail program I refer to in the previous mail.

>  From: nemo@lsub.org
>  To: 9fans@9fans.net
>  Reply-To: 9fans@9fans.net
>  Date: Thu Jun 12 12:24:45 CET 2008
>  Subject: Re: [9fans] store 9p session-only values using lib9p
>
>  Funny, I've done the same in a different way.
>  see mail2fs in contrib/nemo.
>  Also, I have some proposal, skip to the end of the mail and let me know
>  what you think :)
>
>  In any case, I'd love to see/try your version of upas/fs et al.
>
>  Instead of adapting upas/fs, I use a mail2fs program that uses
>  upas to convert mail into an "unpacked" form. Each mail is a directory.
>  A "text" file contains the message text right as you would see it in a mail
>  reader (including relative paths for attachments). Each attach is decoded
>  and kept in the mail's directory ready to be copied, printed, etc.; if possible,
>  using the same file name reported by the attachment.
>
>  I use the very same approach for sending mail. A directory /mail/box/nemo/out
>  is spooled to send whichever file is placed on it as a mail. The format for the
>  outgoing messages is again similar to what you'd type in a mail writer.
>
>  The whole point is that now you editor (plus couple of scripts) becomes a mail
>  reader/writer without understanding anything about mail.
>
>  My proposal:
>  Why don't we reach an agreement and start using the very
>  same format. I suggest keeping my mbox format but adapting upas/fs to
>  understand it, which is a good idea. But I'm open to suggestions.
>
>  thanks
>
>  > From: quanstro@quanstro.net
>  > To: 9fans@9fans.net
>  > Reply-To: 9fans@9fans.net
>  > Date: Thu Jun 12 12:10:32 CET 2008
>  > Subject: Re: [9fans] store 9p session-only values using lib9p
>  >
>  > > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.
>  >
>  > i have a couple of reservations about mbox format. first, a majority
>  > of users that i need to support have inboxes >10mb and some
>  > have mailboxes >100mb. thus deleting an older message will require
>  > rewriting the whole file, and if it's not the last message, will take
>  > up quite a bit of storage, even on venti.
>  >
>  > second, it's very difficult to cache. for example, suppose i have two
>  > instances of my mail fs interpreting the same mbox at the same time.
>  > suppose that i delete the 5th out of 500 messages with the first. the
>  > second will then have to reread the entire mbox to get its pointers
>  > right. even if i write an index with the first, the entire index
>  > needs to be reread.
>  >
>  > third, since large messages tend to be really stinking huge, it is not
>  > efficient to read entire messages to deduce the mime structure. (ned
>  > reads the mime structure of each doc on startup.) that last mime part
>  > may be tens of mb into the
>  >
>  > you'll notice that if one uses email in the way it was used
>  > traditionally in unix environments, that a 500 message, inbox or 15mb
>  > message is unreasonable. for traditional uses, mbox format and
>  > loading the whole stinking thing into memory is a great idea.
>  >
>  > unfortunately, that's now how people use email these days.
>  > our two machines running upas/fs burn up to 5gb of ram.
>  > old rewritten mail adds several hundred mb/day to the dump.
>  >
>  > > also file lengths is going to be a problem if i'm going to decode files within the fs.
>  > >
>  > > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs
>  >
>  > i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
>  > mailbox types that support caching (mdir and imap4) keep their cache
>  > below Maxcache or Σ of (actively) referenced messages, whichever is bigger.
>  >
>  > i've also adapted the rest of upas, including deliver and marshal to
>  > understand mdir format. imap4d should be done soon.
>  >
>  > this upas/fs keeps an index, so in most cases, emails are not reread unless they are
>  > viewed. the index contains some information -- like flags -- not recorded
>  > in the mail.
>  >
>  > i'd be happy to share a working copy to those interested. but it's
>  > not ready to be inflicted on the world yet. the "From " line date is
>  > unresolved and imap4d still thinks it knows how to append a message
>  > to a mailbox.
>  >
>  > > about the From line, qmail man page about mbox format says it is composedof From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?
>  >
>  > why not? this seems an arbitrary distinction between the inbox and
>  > every other mailbox.
>  >
>  > - erik
>  >
>  >
>

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

Hello

I used your mail2fs to store my mail archives for a couple of months (and still use it ☺), but the main problem, which is parse attachments using upas/fs, is not solved.

what erik has done is great, i would vote to have it finished soon ☺

slds.

gabi

Re: [9fans] store 9p session-only values using lib9p

[Francisco J Ballesteros]

I don't understand.
mail2fs parses attachments. It uses upas/fs, but it parses attachs.


On Thu, Jun 12, 2008 at 12:39 PM,  <gdiaz@9grid.es> wrote:
> Hello
>
> I used your mail2fs to store my mail archives for a couple of months (and still use it ☺), but the main problem, which is parse attachments using upas/fs, is not solved.
>
> what erik has done is great, i would vote to have it finished soon ☺
>
> slds.
>
> gabi
>
>

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

Hello

iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation.

but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti :)

slds.

gabi

Re: [9fans] store 9p session-only values using lib9p

[Francisco J Ballesteros]

It copies *and* parses the attachment, but I admit that you have to
be able to keep at least the single mail in memory while converting it
to the new format.

Should this be a problem, perhaps doing what Russ suggested
(fixing upas/fs to let it handle big files) would be the right thing.

In any way, I'd still be converting my mail to the new format, because
of the convenience of having everything unpacked and ready to read,
and because of the space it's likely to be saving in venti.

On Thu, Jun 12, 2008 at 12:55 PM,  <gdiaz@9grid.es> wrote:
> Hello
>
> iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation.
>
> but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti :)
>
> slds.
>
> gabi
>
>

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message.

why not?  the "From " line is written when the message is delivered to a
mb.  but it's not part of the message proper.  so copying a message
really copies the rfc/822 part.  the from line is an awkward appendage.
it is strange to carry it forward in the special case of copying a mail
from one box to another.

- erik

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

Hello

right, that's just one of the problems erik solved and i was still solving ☺, the memory usage of upas/fs.

slds.

gabi

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> Funny, I've done the same in a different way.
> see mail2fs in contrib/nemo.
> Also, I have some proposal, skip to the end of the mail and let me know
> what you think :)
>
> In any case, I'd love to see/try your version of upas/fs et al.

/n/sources/contrib/quanstro/src/nupas.  cavet emptor.

nupas/fs is fully compatable with upas/*.

i have been using the mdir format for about 18 months.  but i have
just recently added the bits to reduce memory footprint.

i'm currently spending quite a bit of time on this so details may change.

i would appreciate any feedback.

> Instead of adapting upas/fs, I use a mail2fs program that uses
> upas to convert mail into an "unpacked" form. Each mail is a directory.
> A "text" file contains the message text right as you would see it in a mail
> reader (including relative paths for attachments). Each attach is decoded
> and kept in the mail's directory ready to be copied, printed, etc.; if possible,
> using the same file name reported by the attachment.

it's hard denying that this is some allure to this idea and i definately
considered it.  however, we get quite a bit of three-part email containing
the mime parent a 150 byte message and the 400 (with headers) byte
replyed-to message.  depending on the details of your format, this
could require 3 directories + 3 files or 6 blocks of storage for a
smaller-than 1 block message.  (assuming 8kb blocks.)  it also could
result in quite a bit more seeking when scanning a number of messages.

in addition, this format is not compatable with any of the existing
tools.  in mdir format, each file looks like a mailbox of exactly one
message.  in addition, in mdir format each message is self-contained.
i can just cp it.

finally, a number of email security standards (so-called spf2.0 and
s/mime) require the original email.  it's not clear to me that one can
reconstruct an original mime message from processed parts without
leaving some breadcrumbs behind.  there's no requirement not to
base64-encode us-ascii, for example.

- erik

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

stupid hacks:

	echo 1,10p id from|nupas/nedmail $*>[2=]
	echo 1,10p id flags|nupas/nedmail $*>[2=]

- erik

Re: [9fans] store 9p session-only values using lib9p

[gdiaz]

hello


hehe, i don't know why it is copied, probably because if you use mbox format you need the From line as a message delimeter, but if you're using other format, the From line could be avoided. As we are talking about email, may be it has a hidden meaning nobody remembers now ;).

greetings,

gabi

Re: [9fans] store 9p session-only values using lib9p

[Francisco J Ballesteros]

This mail with a reply inlined would be
msgdir/
msgdir/text
msgdir/raw

If, the reply was inlined. Anyone wanting to use the original can go use raw.
But the nice thing is that you can edit text, copy attachments in/out,
remove them, etc. etc.

Compatibility in my case is achieved by leaving upas et al. as they
are; I convert
just my mail once it entered the system.

On Thu, Jun 12, 2008 at 2:25 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> Funny, I've done the same in a different way.
>> see mail2fs in contrib/nemo.
>> Also, I have some proposal, skip to the end of the mail and let me know
>> what you think :)
>>
>> In any case, I'd love to see/try your version of upas/fs et al.
>
> /n/sources/contrib/quanstro/src/nupas.  cavet emptor.
>
> nupas/fs is fully compatable with upas/*.
>
> i have been using the mdir format for about 18 months.  but i have
> just recently added the bits to reduce memory footprint.
>
> i'm currently spending quite a bit of time on this so details may change.
>
> i would appreciate any feedback.
>
>> Instead of adapting upas/fs, I use a mail2fs program that uses
>> upas to convert mail into an "unpacked" form. Each mail is a directory.
>> A "text" file contains the message text right as you would see it in a mail
>> reader (including relative paths for attachments). Each attach is decoded
>> and kept in the mail's directory ready to be copied, printed, etc.; if possible,
>> using the same file name reported by the attachment.
>
> it's hard denying that this is some allure to this idea and i definately
> considered it.  however, we get quite a bit of three-part email containing
> the mime parent a 150 byte message and the 400 (with headers) byte
> replyed-to message.  depending on the details of your format, this
> could require 3 directories + 3 files or 6 blocks of storage for a
> smaller-than 1 block message.  (assuming 8kb blocks.)  it also could
> result in quite a bit more seeking when scanning a number of messages.
>
> in addition, this format is not compatable with any of the existing
> tools.  in mdir format, each file looks like a mailbox of exactly one
> message.  in addition, in mdir format each message is self-contained.
> i can just cp it.
>
> finally, a number of email security standards (so-called spf2.0 and
> s/mime) require the original email.  it's not clear to me that one can
> reconstruct an original mime message from processed parts without
> leaving some breadcrumbs behind.  there's no requirement not to
> base64-encode us-ascii, for example.
>
> - erik
>
>
>

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> if you view the date that way, as an integral part
> of the delivered message, it would sure be strange
> if saving the message to a different folder altered
> the delivery date.  receiving a message and filing
> a message are two very different things.
>
>> so copying a message
>> really copies the rfc/822 part.  the from line is an awkward appendage.
>> it is strange to carry it forward in the special case of copying a mail
>> from one box to another.
>
> the rfc822 part is the awkward appendage.
> mail used to be simple.

a compelling argument.  i agree.

- erik

Re: [9fans] store 9p session-only values using lib9p

[Russ Cox]

> why not?  the "From " line is written when the message is delivered to a
> mb.  but it's not part of the message proper.

that may be your mental model, but it's not mine.
to me, the From line is as much part of the message
as the rest of the mail.  like i said before, it is a
postmark.  it indicates the time the mail was originally
delivered along with the mail system's idea of the sender.

there was a time, in the calm pre-rfc822 days
when the from line was all the header you had.

for example, here's an entire message locally
delivered on plan 9 in 1992:

	From ken Tue Sep  8 03:42:43 EDT 1992
	i finally got my copy.

personally, when i've used mail systems without from lines,
i've always been annoyed, because then the dates
on the message are relative to someone else's clock,
not my local one.  i don't mean just time-zone
variations; plenty of people have their clocks set wrong.
i like that nedmail shows me the time the message
arrived, not the claimed date in the header.

if you view the date that way, as an integral part
of the delivered message, it would sure be strange
if saving the message to a different folder altered
the delivery date.  receiving a message and filing
a message are two very different things.

> so copying a message
> really copies the rfc/822 part.  the from line is an awkward appendage.
> it is strange to carry it forward in the special case of copying a mail
> from one box to another.

the rfc822 part is the awkward appendage.
mail used to be simple.

russ

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> If, the reply was inlined. Anyone wanting to use the original can go use raw.
> But the nice thing is that you can edit text, copy attachments in/out,
> remove them, etc. etc.

this is already possible with upas/fs, isn't it?  also, i get the benefit
of not being tied into message formats.  for example, i can use imap.

in addition, doesn't storing mail your way increase the storage requirements?
also, what do you do about header parsing without upas/fs's help?

(storing small mailboxes as a mdir even consumes more resources than
a mbox.  unfortunately, i have to account for massive mail boxes.)

> Compatibility in my case is achieved by leaving upas et al. as they
> are; I convert
> just my mail once it entered the system.

mboxes are the single biggest resource pig we have.  we comfortably
did development on a single pIII fs with a single ide disk and 256mb
of memory before email was treated as the great repository of all
human knowledge.

so my primary goal is to have mail use as few resources as
reasonablly possible.

- erik

Re: [9fans] store 9p session-only values using lib9p

[Fco. J. Ballesteros]

>  > But the nice thing is that you can edit text, copy attachments in/out,
>  > remove them, etc. etc.
>  this is already possible with upas/fs, isn't it? also, i get the benefit
>  of not being tied into message formats. for example, i can use imap.

Nope. If you try to remove an attach you remove the mail.
What I like of your way  is that making upas/fs understand this mbox format makes
all other tools available.

>  in addition, doesn't storing mail your way increase the storage requirements?

No. Same file sent multiple times with different mails (which I do get a lot) only
consumes a single place on venti.

>  also, what do you do about header parsing without upas/fs's help?

I run upas/fs on the raw file (e.g., in the Send script that sends this reply)

>  (storing small mailboxes as a mdir even consumes more resources than
>  a mbox. unfortunately, i have to account for massive mail boxes.)

(are there small mboxes anymore? ;)

I know, but it's so covenient to do a "looktags msgs quanstro mbox" and get the
paths/lines for messages relevant; etc.

Re: [9fans] store 9p session-only values using lib9p

[Fco. J. Ballesteros]

Sorry. I mean, "*Yes*, but ...".
I mean, you are right (unless you remove the raw files for non-spam mails
after some time; which I do not).

>  > in addition, doesn't storing mail your way increase the storage requirements?
>
>  No. Same file sent multiple times with different mails (which I do get a lot) only
>  consumes a single place on venti.

Re: [9fans] store 9p session-only values using lib9p

[erik quanstrom]

> (are there small mboxes anymore? ;)
>

du -a /mail/box/quanstro/mbox
4	/mail/box/quanstro/mbox/1212376173.00
2	/mail/box/quanstro/mbox/1212378184.00
4	/mail/box/quanstro/mbox/1213282217.00
4	/mail/box/quanstro/mbox/1193334801.00
2	/mail/box/quanstro/mbox/1211556429.00
1	/mail/box/quanstro/mbox/1193342921.00
4	/mail/box/quanstro/mbox/1206119475.00
3	/mail/box/quanstro/mbox/1186367197.00
7	/mail/box/quanstro/mbox/1185927350.00
3	/mail/box/quanstro/mbox/1212677409.00
2	/mail/box/quanstro/mbox/1193340043.00
5	/mail/box/quanstro/mbox/1197914075.00
4	/mail/box/quanstro/mbox/1166486503.00
2	/mail/box/quanstro/mbox/1168364456.00
1	/mail/box/quanstro/mbox/1212437662.00
5	/mail/box/quanstro/mbox/1204383150.00
5	/mail/box/quanstro/mbox/1192018998.00
35	/mail/box/quanstro/mbox/1170245420.00
2	/mail/box/quanstro/mbox/1170245727.00
4	/mail/box/quanstro/mbox/1187814435.00
1	/mail/box/quanstro/mbox/1198083270.00
6	/mail/box/quanstro/mbox/1176757449.00
106	/mail/box/quanstro/mbox

> I know, but it's so covenient to do a "looktags msgs quanstro mbox" and get the
> paths/lines for messages relevant; etc.

why not:

	bind /bin/nupas /bin/upas

	grep -l nemo /mail/fs/mbox/*/from |
		awk -F/ '{print $(NF-1) "p id subject" }' |
		upas/nedmail -n>[2=]
	22 Re: [9fans] store 9p session-only values using lib9p
or
	grep -l nemo /mail/fs/mbox/*/from
?  (actual tagging is yet to come.)

- erik

Re: [9fans] store 9p session-only values using lib9p

[lucio]

> Why don't we reach an agreement and start using the very
> same format. I suggest keeping my mbox format but adapting upas/fs to
> understand it, which is a good idea. But I'm open to suggestions.

...  and use base64 encoding of attachment names so they can be
displayed correctly (by a clever function) and yet not interfere with
the directory structure.

Count me in, I'm extremely interesting in giving Plan 9 the hottest
mail handling on the planet.  I'll deal with the privacy encoding and
decoding, specially if others keep me on track.

++L

Re: [9fans] store 9p session-only values using lib9p

[lucio]

> In any way, I'd still be converting my mail to the new format, because
> of the convenience of having everything unpacked and ready to read,
> and because of the space it's likely to be saving in venti.

It is unavoidable that once you have a permanent filesystem for mail
you'll also need the ability to present its index by different,
arbitrary sorting criteria.  I haven't really found the magic bullet
for that, but I suggest we don't forget to look for it.

++L