* [9fans] store 9p session-only values using lib9p @ 2008-06-11 20:06 gdiaz 2008-06-11 20:48 ` Russ Cox 2008-06-11 23:44 ` erik quanstrom 0 siblings, 2 replies; 30+ messages in thread From: gdiaz @ 2008-06-11 20:06 UTC (permalink / raw) To: 9fans Hello I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue) If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time. could that value be saved for each session? i would love to hear from other's experiences, thanks, gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-11 20:06 [9fans] store 9p session-only values using lib9p gdiaz @ 2008-06-11 20:48 ` Russ Cox 2008-06-11 20:58 ` gdiaz 2008-06-11 23:44 ` erik quanstrom 1 sibling, 1 reply; 30+ messages in thread From: Russ Cox @ 2008-06-11 20:48 UTC (permalink / raw) To: 9fans > If i save the offset adjustment in f->aux or simmilar, > i can calculate the real file offset on the next T-reads, > but that will only work if one client reads it at one time. No it won't. Even if just one client is reading, that client might seek. You could assume that most reads will pick up where the last one left off, and use f->aux to point at a single hint that you can check. Maybe you're willing to do more work if you do get a seek. Otherwise you'll need to build a more complex data structure to help you map between the two. Also, if f is a Fid* (not a File*), then you get a different f for each different Topen of the file (and thus also for each client), so the hint-per-Fid usually works pretty well in practice for this kind of thing. Russ ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-11 20:48 ` Russ Cox @ 2008-06-11 20:58 ` gdiaz 0 siblings, 0 replies; 30+ messages in thread From: gdiaz @ 2008-06-11 20:58 UTC (permalink / raw) To: 9fans hello you're right, i will think a bit more about it. thank you very much, gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-11 20:06 [9fans] store 9p session-only values using lib9p gdiaz 2008-06-11 20:48 ` Russ Cox @ 2008-06-11 23:44 ` erik quanstrom 2008-06-12 0:01 ` [9fans] upas/fs Russ Cox 2008-06-12 8:34 ` [9fans] store 9p session-only values using lib9p gdiaz 1 sibling, 2 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-11 23:44 UTC (permalink / raw) To: 9fans > Hello > > I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue) > > If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time. > > could that value be saved for each session? > > i would love to hear from other's experiences, i've been spending a little time with upas/fs. i don't think that in general one can get by parsing part of an email, unless that part happens to be top-level headers. there are more transforms than you've mentioned. apple mail loves to put nulls in html email, for example. traditional mbox format is problematic, too. if you wish to control memory usage (which is a serious problem for upas/fs), i think the way to go is a cache with reference counting. the stuff i have isn't ready for prime time, but it's coming along nicely. by the way, does anyone know the rational for the date on the unix "From " line? upas sets it to the date the message is originally delivered to the inbox. moving it from the inbox to another folder does not change the date. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] upas/fs 2008-06-11 23:44 ` erik quanstrom @ 2008-06-12 0:01 ` Russ Cox 2008-06-12 2:31 ` erik quanstrom 2008-06-12 8:34 ` [9fans] store 9p session-only values using lib9p gdiaz 1 sibling, 1 reply; 30+ messages in thread From: Russ Cox @ 2008-06-12 0:01 UTC (permalink / raw) To: 9fans > by the way, does anyone know the rational for the date on the > unix "From " line? upas sets it to the date the message is originally > delivered to the inbox. moving it from the inbox to another folder > does not change the date. the date is the date it was delivered. it's a receiver-side postmark. but you knew that. what's your question? russ ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] upas/fs 2008-06-12 0:01 ` [9fans] upas/fs Russ Cox @ 2008-06-12 2:31 ` erik quanstrom 2008-06-12 2:59 ` Lyndon Nerenberg 0 siblings, 1 reply; 30+ messages in thread From: erik quanstrom @ 2008-06-12 2:31 UTC (permalink / raw) To: 9fans >> by the way, does anyone know the rationale for the date on the >> unix "From " line? upas sets it to the date the message is originally >> delivered to the inbox. moving it from the inbox to another folder >> does not change the date. > > the date is the date it was delivered. > it's a receiver-side postmark. > but you knew that. what's your question? right. since the date is attached when delivered to a mailbox, why doesn't this date change when it's delivered to a secondary mailbox? why is the assignment a magical property of the inbox? - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] upas/fs 2008-06-12 2:31 ` erik quanstrom @ 2008-06-12 2:59 ` Lyndon Nerenberg 0 siblings, 0 replies; 30+ messages in thread From: Lyndon Nerenberg @ 2008-06-12 2:59 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 2008-Jun-11, at 19:31 , erik quanstrom wrote: > right. since the date is attached when delivered to a mailbox, > why doesn't this date change when it's delivered to a secondary > mailbox? why is the assignment a magical property of the inbox? Most likely it's just an artifact of the original UNIX mail implementation. The \n^From<sp> separator line got generated at initial delivery time, and the mail client used that as the display time in the message summaries (e.g. Date: not spoken here). Therefore it makes sense to preserve the initial separator line with it's date intact to ensure consistency for display purposes. Think of it as the UNIX ctime of the message. --lyndon ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-11 23:44 ` erik quanstrom 2008-06-12 0:01 ` [9fans] upas/fs Russ Cox @ 2008-06-12 8:34 ` gdiaz 2008-06-12 10:04 ` erik quanstrom 1 sibling, 1 reply; 30+ messages in thread From: gdiaz @ 2008-06-12 8:34 UTC (permalink / raw) To: 9fans hello at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded. also file lengths is going to be a problem if i'm going to decode files within the fs. i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-? thanks, gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 8:34 ` [9fans] store 9p session-only values using lib9p gdiaz @ 2008-06-12 10:04 ` erik quanstrom 2008-06-12 10:24 ` Fco. J. Ballesteros 2008-06-12 10:26 ` gdiaz 0 siblings, 2 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 10:04 UTC (permalink / raw) To: 9fans > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded. i have a couple of reservations about mbox format. first, a majority of users that i need to support have inboxes >10mb and some have mailboxes >100mb. thus deleting an older message will require rewriting the whole file, and if it's not the last message, will take up quite a bit of storage, even on venti. second, it's very difficult to cache. for example, suppose i have two instances of my mail fs interpreting the same mbox at the same time. suppose that i delete the 5th out of 500 messages with the first. the second will then have to reread the entire mbox to get its pointers right. even if i write an index with the first, the entire index needs to be reread. third, since large messages tend to be really stinking huge, it is not efficient to read entire messages to deduce the mime structure. (ned reads the mime structure of each doc on startup.) that last mime part may be tens of mb into the you'll notice that if one uses email in the way it was used traditionally in unix environments, that a 500 message, inbox or 15mb message is unreasonable. for traditional uses, mbox format and loading the whole stinking thing into memory is a great idea. unfortunately, that's now how people use email these days. our two machines running upas/fs burn up to 5gb of ram. old rewritten mail adds several hundred mb/day to the dump. > also file lengths is going to be a problem if i'm going to decode files within the fs. > > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs i've adapted upas/fs to use a directory for mailboxes (mdir) by default. mailbox types that support caching (mdir and imap4) keep their cache below Maxcache or Σ of (actively) referenced messages, whichever is bigger. i've also adapted the rest of upas, including deliver and marshal to understand mdir format. imap4d should be done soon. this upas/fs keeps an index, so in most cases, emails are not reread unless they are viewed. the index contains some information -- like flags -- not recorded in the mail. i'd be happy to share a working copy to those interested. but it's not ready to be inflicted on the world yet. the "From " line date is unresolved and imap4d still thinks it knows how to append a message to a mailbox. > about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-? why not? this seems an arbitrary distinction between the inbox and every other mailbox. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:04 ` erik quanstrom @ 2008-06-12 10:24 ` Fco. J. Ballesteros 2008-06-12 10:28 ` Fco. J. Ballesteros ` (3 more replies) 2008-06-12 10:26 ` gdiaz 1 sibling, 4 replies; 30+ messages in thread From: Fco. J. Ballesteros @ 2008-06-12 10:24 UTC (permalink / raw) To: 9fans Funny, I've done the same in a different way. see mail2fs in contrib/nemo. Also, I have some proposal, skip to the end of the mail and let me know what you think :) In any case, I'd love to see/try your version of upas/fs et al. Instead of adapting upas/fs, I use a mail2fs program that uses upas to convert mail into an "unpacked" form. Each mail is a directory. A "text" file contains the message text right as you would see it in a mail reader (including relative paths for attachments). Each attach is decoded and kept in the mail's directory ready to be copied, printed, etc.; if possible, using the same file name reported by the attachment. I use the very same approach for sending mail. A directory /mail/box/nemo/out is spooled to send whichever file is placed on it as a mail. The format for the outgoing messages is again similar to what you'd type in a mail writer. The whole point is that now you editor (plus couple of scripts) becomes a mail reader/writer without understanding anything about mail. My proposal: Why don't we reach an agreement and start using the very same format. I suggest keeping my mbox format but adapting upas/fs to understand it, which is a good idea. But I'm open to suggestions. thanks > From: quanstro@quanstro.net > To: 9fans@9fans.net > Reply-To: 9fans@9fans.net > Date: Thu Jun 12 12:10:32 CET 2008 > Subject: Re: [9fans] store 9p session-only values using lib9p > > > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded. > > i have a couple of reservations about mbox format. first, a majority > of users that i need to support have inboxes >10mb and some > have mailboxes >100mb. thus deleting an older message will require > rewriting the whole file, and if it's not the last message, will take > up quite a bit of storage, even on venti. > > second, it's very difficult to cache. for example, suppose i have two > instances of my mail fs interpreting the same mbox at the same time. > suppose that i delete the 5th out of 500 messages with the first. the > second will then have to reread the entire mbox to get its pointers > right. even if i write an index with the first, the entire index > needs to be reread. > > third, since large messages tend to be really stinking huge, it is not > efficient to read entire messages to deduce the mime structure. (ned > reads the mime structure of each doc on startup.) that last mime part > may be tens of mb into the > > you'll notice that if one uses email in the way it was used > traditionally in unix environments, that a 500 message, inbox or 15mb > message is unreasonable. for traditional uses, mbox format and > loading the whole stinking thing into memory is a great idea. > > unfortunately, that's now how people use email these days. > our two machines running upas/fs burn up to 5gb of ram. > old rewritten mail adds several hundred mb/day to the dump. > > > also file lengths is going to be a problem if i'm going to decode files within the fs. > > > > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs > > i've adapted upas/fs to use a directory for mailboxes (mdir) by default. > mailbox types that support caching (mdir and imap4) keep their cache > below Maxcache or Σ of (actively) referenced messages, whichever is bigger. > > i've also adapted the rest of upas, including deliver and marshal to > understand mdir format. imap4d should be done soon. > > this upas/fs keeps an index, so in most cases, emails are not reread unless they are > viewed. the index contains some information -- like flags -- not recorded > in the mail. > > i'd be happy to share a working copy to those interested. but it's > not ready to be inflicted on the world yet. the "From " line date is > unresolved and imap4d still thinks it knows how to append a message > to a mailbox. > > > about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-? > > why not? this seems an arbitrary distinction between the inbox and > every other mailbox. > > - erik > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:24 ` Fco. J. Ballesteros @ 2008-06-12 10:28 ` Fco. J. Ballesteros 2008-06-12 10:39 ` gdiaz ` (2 subsequent siblings) 3 siblings, 0 replies; 30+ messages in thread From: Fco. J. Ballesteros @ 2008-06-12 10:28 UTC (permalink / raw) To: 9fans FTS, http://lsub.org/magic/man2html/1/mails is the manual page for the mail program I refer to in the previous mail. > From: nemo@lsub.org > To: 9fans@9fans.net > Reply-To: 9fans@9fans.net > Date: Thu Jun 12 12:24:45 CET 2008 > Subject: Re: [9fans] store 9p session-only values using lib9p > > Funny, I've done the same in a different way. > see mail2fs in contrib/nemo. > Also, I have some proposal, skip to the end of the mail and let me know > what you think :) > > In any case, I'd love to see/try your version of upas/fs et al. > > Instead of adapting upas/fs, I use a mail2fs program that uses > upas to convert mail into an "unpacked" form. Each mail is a directory. > A "text" file contains the message text right as you would see it in a mail > reader (including relative paths for attachments). Each attach is decoded > and kept in the mail's directory ready to be copied, printed, etc.; if possible, > using the same file name reported by the attachment. > > I use the very same approach for sending mail. A directory /mail/box/nemo/out > is spooled to send whichever file is placed on it as a mail. The format for the > outgoing messages is again similar to what you'd type in a mail writer. > > The whole point is that now you editor (plus couple of scripts) becomes a mail > reader/writer without understanding anything about mail. > > My proposal: > Why don't we reach an agreement and start using the very > same format. I suggest keeping my mbox format but adapting upas/fs to > understand it, which is a good idea. But I'm open to suggestions. > > thanks > > > From: quanstro@quanstro.net > > To: 9fans@9fans.net > > Reply-To: 9fans@9fans.net > > Date: Thu Jun 12 12:10:32 CET 2008 > > Subject: Re: [9fans] store 9p session-only values using lib9p > > > > > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded. > > > > i have a couple of reservations about mbox format. first, a majority > > of users that i need to support have inboxes >10mb and some > > have mailboxes >100mb. thus deleting an older message will require > > rewriting the whole file, and if it's not the last message, will take > > up quite a bit of storage, even on venti. > > > > second, it's very difficult to cache. for example, suppose i have two > > instances of my mail fs interpreting the same mbox at the same time. > > suppose that i delete the 5th out of 500 messages with the first. the > > second will then have to reread the entire mbox to get its pointers > > right. even if i write an index with the first, the entire index > > needs to be reread. > > > > third, since large messages tend to be really stinking huge, it is not > > efficient to read entire messages to deduce the mime structure. (ned > > reads the mime structure of each doc on startup.) that last mime part > > may be tens of mb into the > > > > you'll notice that if one uses email in the way it was used > > traditionally in unix environments, that a 500 message, inbox or 15mb > > message is unreasonable. for traditional uses, mbox format and > > loading the whole stinking thing into memory is a great idea. > > > > unfortunately, that's now how people use email these days. > > our two machines running upas/fs burn up to 5gb of ram. > > old rewritten mail adds several hundred mb/day to the dump. > > > > > also file lengths is going to be a problem if i'm going to decode files within the fs. > > > > > > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs > > > > i've adapted upas/fs to use a directory for mailboxes (mdir) by default. > > mailbox types that support caching (mdir and imap4) keep their cache > > below Maxcache or Σ of (actively) referenced messages, whichever is bigger. > > > > i've also adapted the rest of upas, including deliver and marshal to > > understand mdir format. imap4d should be done soon. > > > > this upas/fs keeps an index, so in most cases, emails are not reread unless they are > > viewed. the index contains some information -- like flags -- not recorded > > in the mail. > > > > i'd be happy to share a working copy to those interested. but it's > > not ready to be inflicted on the world yet. the "From " line date is > > unresolved and imap4d still thinks it knows how to append a message > > to a mailbox. > > > > > about the From line, qmail man page about mbox format says it is composedof From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-? > > > > why not? this seems an arbitrary distinction between the inbox and > > every other mailbox. > > > > - erik > > > > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:24 ` Fco. J. Ballesteros 2008-06-12 10:28 ` Fco. J. Ballesteros @ 2008-06-12 10:39 ` gdiaz 2008-06-12 10:43 ` Francisco J Ballesteros 2008-06-12 12:25 ` erik quanstrom 2008-06-12 17:12 ` lucio 3 siblings, 1 reply; 30+ messages in thread From: gdiaz @ 2008-06-12 10:39 UTC (permalink / raw) To: 9fans Hello I used your mail2fs to store my mail archives for a couple of months (and still use it ☺), but the main problem, which is parse attachments using upas/fs, is not solved. what erik has done is great, i would vote to have it finished soon ☺ slds. gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:39 ` gdiaz @ 2008-06-12 10:43 ` Francisco J Ballesteros 2008-06-12 10:55 ` gdiaz 0 siblings, 1 reply; 30+ messages in thread From: Francisco J Ballesteros @ 2008-06-12 10:43 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs I don't understand. mail2fs parses attachments. It uses upas/fs, but it parses attachs. On Thu, Jun 12, 2008 at 12:39 PM, <gdiaz@9grid.es> wrote: > Hello > > I used your mail2fs to store my mail archives for a couple of months (and still use it ☺), but the main problem, which is parse attachments using upas/fs, is not solved. > > what erik has done is great, i would vote to have it finished soon ☺ > > slds. > > gabi > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:43 ` Francisco J Ballesteros @ 2008-06-12 10:55 ` gdiaz 2008-06-12 11:04 ` Francisco J Ballesteros 0 siblings, 1 reply; 30+ messages in thread From: gdiaz @ 2008-06-12 10:55 UTC (permalink / raw) To: 9fans Hello iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation. but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti :) slds. gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:55 ` gdiaz @ 2008-06-12 11:04 ` Francisco J Ballesteros 2008-06-12 11:11 ` gdiaz 2008-06-12 17:16 ` lucio 0 siblings, 2 replies; 30+ messages in thread From: Francisco J Ballesteros @ 2008-06-12 11:04 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs It copies *and* parses the attachment, but I admit that you have to be able to keep at least the single mail in memory while converting it to the new format. Should this be a problem, perhaps doing what Russ suggested (fixing upas/fs to let it handle big files) would be the right thing. In any way, I'd still be converting my mail to the new format, because of the convenience of having everything unpacked and ready to read, and because of the space it's likely to be saving in venti. On Thu, Jun 12, 2008 at 12:55 PM, <gdiaz@9grid.es> wrote: > Hello > > iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation. > > but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti :) > > slds. > > gabi > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 11:04 ` Francisco J Ballesteros @ 2008-06-12 11:11 ` gdiaz 2008-06-12 17:16 ` lucio 1 sibling, 0 replies; 30+ messages in thread From: gdiaz @ 2008-06-12 11:11 UTC (permalink / raw) To: 9fans Hello right, that's just one of the problems erik solved and i was still solving ☺, the memory usage of upas/fs. slds. gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 11:04 ` Francisco J Ballesteros 2008-06-12 11:11 ` gdiaz @ 2008-06-12 17:16 ` lucio 1 sibling, 0 replies; 30+ messages in thread From: lucio @ 2008-06-12 17:16 UTC (permalink / raw) To: 9fans > In any way, I'd still be converting my mail to the new format, because > of the convenience of having everything unpacked and ready to read, > and because of the space it's likely to be saving in venti. It is unavoidable that once you have a permanent filesystem for mail you'll also need the ability to present its index by different, arbitrary sorting criteria. I haven't really found the magic bullet for that, but I suggest we don't forget to look for it. ++L ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:24 ` Fco. J. Ballesteros 2008-06-12 10:28 ` Fco. J. Ballesteros 2008-06-12 10:39 ` gdiaz @ 2008-06-12 12:25 ` erik quanstrom 2008-06-12 12:32 ` erik quanstrom 2008-06-12 14:03 ` Francisco J Ballesteros 2008-06-12 17:12 ` lucio 3 siblings, 2 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 12:25 UTC (permalink / raw) To: 9fans > Funny, I've done the same in a different way. > see mail2fs in contrib/nemo. > Also, I have some proposal, skip to the end of the mail and let me know > what you think :) > > In any case, I'd love to see/try your version of upas/fs et al. /n/sources/contrib/quanstro/src/nupas. cavet emptor. nupas/fs is fully compatable with upas/*. i have been using the mdir format for about 18 months. but i have just recently added the bits to reduce memory footprint. i'm currently spending quite a bit of time on this so details may change. i would appreciate any feedback. > Instead of adapting upas/fs, I use a mail2fs program that uses > upas to convert mail into an "unpacked" form. Each mail is a directory. > A "text" file contains the message text right as you would see it in a mail > reader (including relative paths for attachments). Each attach is decoded > and kept in the mail's directory ready to be copied, printed, etc.; if possible, > using the same file name reported by the attachment. it's hard denying that this is some allure to this idea and i definately considered it. however, we get quite a bit of three-part email containing the mime parent a 150 byte message and the 400 (with headers) byte replyed-to message. depending on the details of your format, this could require 3 directories + 3 files or 6 blocks of storage for a smaller-than 1 block message. (assuming 8kb blocks.) it also could result in quite a bit more seeking when scanning a number of messages. in addition, this format is not compatable with any of the existing tools. in mdir format, each file looks like a mailbox of exactly one message. in addition, in mdir format each message is self-contained. i can just cp it. finally, a number of email security standards (so-called spf2.0 and s/mime) require the original email. it's not clear to me that one can reconstruct an original mime message from processed parts without leaving some breadcrumbs behind. there's no requirement not to base64-encode us-ascii, for example. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 12:25 ` erik quanstrom @ 2008-06-12 12:32 ` erik quanstrom 2008-06-12 14:03 ` Francisco J Ballesteros 1 sibling, 0 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 12:32 UTC (permalink / raw) To: 9fans stupid hacks: echo 1,10p id from|nupas/nedmail $*>[2=] echo 1,10p id flags|nupas/nedmail $*>[2=] - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 12:25 ` erik quanstrom 2008-06-12 12:32 ` erik quanstrom @ 2008-06-12 14:03 ` Francisco J Ballesteros 2008-06-12 14:25 ` erik quanstrom 1 sibling, 1 reply; 30+ messages in thread From: Francisco J Ballesteros @ 2008-06-12 14:03 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs This mail with a reply inlined would be msgdir/ msgdir/text msgdir/raw If, the reply was inlined. Anyone wanting to use the original can go use raw. But the nice thing is that you can edit text, copy attachments in/out, remove them, etc. etc. Compatibility in my case is achieved by leaving upas et al. as they are; I convert just my mail once it entered the system. On Thu, Jun 12, 2008 at 2:25 PM, erik quanstrom <quanstro@quanstro.net> wrote: >> Funny, I've done the same in a different way. >> see mail2fs in contrib/nemo. >> Also, I have some proposal, skip to the end of the mail and let me know >> what you think :) >> >> In any case, I'd love to see/try your version of upas/fs et al. > > /n/sources/contrib/quanstro/src/nupas. cavet emptor. > > nupas/fs is fully compatable with upas/*. > > i have been using the mdir format for about 18 months. but i have > just recently added the bits to reduce memory footprint. > > i'm currently spending quite a bit of time on this so details may change. > > i would appreciate any feedback. > >> Instead of adapting upas/fs, I use a mail2fs program that uses >> upas to convert mail into an "unpacked" form. Each mail is a directory. >> A "text" file contains the message text right as you would see it in a mail >> reader (including relative paths for attachments). Each attach is decoded >> and kept in the mail's directory ready to be copied, printed, etc.; if possible, >> using the same file name reported by the attachment. > > it's hard denying that this is some allure to this idea and i definately > considered it. however, we get quite a bit of three-part email containing > the mime parent a 150 byte message and the 400 (with headers) byte > replyed-to message. depending on the details of your format, this > could require 3 directories + 3 files or 6 blocks of storage for a > smaller-than 1 block message. (assuming 8kb blocks.) it also could > result in quite a bit more seeking when scanning a number of messages. > > in addition, this format is not compatable with any of the existing > tools. in mdir format, each file looks like a mailbox of exactly one > message. in addition, in mdir format each message is self-contained. > i can just cp it. > > finally, a number of email security standards (so-called spf2.0 and > s/mime) require the original email. it's not clear to me that one can > reconstruct an original mime message from processed parts without > leaving some breadcrumbs behind. there's no requirement not to > base64-encode us-ascii, for example. > > - erik > > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 14:03 ` Francisco J Ballesteros @ 2008-06-12 14:25 ` erik quanstrom 2008-06-12 14:49 ` Fco. J. Ballesteros 0 siblings, 1 reply; 30+ messages in thread From: erik quanstrom @ 2008-06-12 14:25 UTC (permalink / raw) To: 9fans > If, the reply was inlined. Anyone wanting to use the original can go use raw. > But the nice thing is that you can edit text, copy attachments in/out, > remove them, etc. etc. this is already possible with upas/fs, isn't it? also, i get the benefit of not being tied into message formats. for example, i can use imap. in addition, doesn't storing mail your way increase the storage requirements? also, what do you do about header parsing without upas/fs's help? (storing small mailboxes as a mdir even consumes more resources than a mbox. unfortunately, i have to account for massive mail boxes.) > Compatibility in my case is achieved by leaving upas et al. as they > are; I convert > just my mail once it entered the system. mboxes are the single biggest resource pig we have. we comfortably did development on a single pIII fs with a single ide disk and 256mb of memory before email was treated as the great repository of all human knowledge. so my primary goal is to have mail use as few resources as reasonablly possible. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 14:25 ` erik quanstrom @ 2008-06-12 14:49 ` Fco. J. Ballesteros 2008-06-12 14:55 ` Fco. J. Ballesteros 2008-06-12 15:06 ` erik quanstrom 0 siblings, 2 replies; 30+ messages in thread From: Fco. J. Ballesteros @ 2008-06-12 14:49 UTC (permalink / raw) To: 9fans > > But the nice thing is that you can edit text, copy attachments in/out, > > remove them, etc. etc. > this is already possible with upas/fs, isn't it? also, i get the benefit > of not being tied into message formats. for example, i can use imap. Nope. If you try to remove an attach you remove the mail. What I like of your way is that making upas/fs understand this mbox format makes all other tools available. > in addition, doesn't storing mail your way increase the storage requirements? No. Same file sent multiple times with different mails (which I do get a lot) only consumes a single place on venti. > also, what do you do about header parsing without upas/fs's help? I run upas/fs on the raw file (e.g., in the Send script that sends this reply) > (storing small mailboxes as a mdir even consumes more resources than > a mbox. unfortunately, i have to account for massive mail boxes.) (are there small mboxes anymore? ;) I know, but it's so covenient to do a "looktags msgs quanstro mbox" and get the paths/lines for messages relevant; etc. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 14:49 ` Fco. J. Ballesteros @ 2008-06-12 14:55 ` Fco. J. Ballesteros 2008-06-12 15:06 ` erik quanstrom 1 sibling, 0 replies; 30+ messages in thread From: Fco. J. Ballesteros @ 2008-06-12 14:55 UTC (permalink / raw) To: 9fans Sorry. I mean, "*Yes*, but ...". I mean, you are right (unless you remove the raw files for non-spam mails after some time; which I do not). > > in addition, doesn't storing mail your way increase the storage requirements? > > No. Same file sent multiple times with different mails (which I do get a lot) only > consumes a single place on venti. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 14:49 ` Fco. J. Ballesteros 2008-06-12 14:55 ` Fco. J. Ballesteros @ 2008-06-12 15:06 ` erik quanstrom 1 sibling, 0 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 15:06 UTC (permalink / raw) To: 9fans > (are there small mboxes anymore? ;) > du -a /mail/box/quanstro/mbox 4 /mail/box/quanstro/mbox/1212376173.00 2 /mail/box/quanstro/mbox/1212378184.00 4 /mail/box/quanstro/mbox/1213282217.00 4 /mail/box/quanstro/mbox/1193334801.00 2 /mail/box/quanstro/mbox/1211556429.00 1 /mail/box/quanstro/mbox/1193342921.00 4 /mail/box/quanstro/mbox/1206119475.00 3 /mail/box/quanstro/mbox/1186367197.00 7 /mail/box/quanstro/mbox/1185927350.00 3 /mail/box/quanstro/mbox/1212677409.00 2 /mail/box/quanstro/mbox/1193340043.00 5 /mail/box/quanstro/mbox/1197914075.00 4 /mail/box/quanstro/mbox/1166486503.00 2 /mail/box/quanstro/mbox/1168364456.00 1 /mail/box/quanstro/mbox/1212437662.00 5 /mail/box/quanstro/mbox/1204383150.00 5 /mail/box/quanstro/mbox/1192018998.00 35 /mail/box/quanstro/mbox/1170245420.00 2 /mail/box/quanstro/mbox/1170245727.00 4 /mail/box/quanstro/mbox/1187814435.00 1 /mail/box/quanstro/mbox/1198083270.00 6 /mail/box/quanstro/mbox/1176757449.00 106 /mail/box/quanstro/mbox > I know, but it's so covenient to do a "looktags msgs quanstro mbox" and get the > paths/lines for messages relevant; etc. why not: bind /bin/nupas /bin/upas grep -l nemo /mail/fs/mbox/*/from | awk -F/ '{print $(NF-1) "p id subject" }' | upas/nedmail -n>[2=] 22 Re: [9fans] store 9p session-only values using lib9p or grep -l nemo /mail/fs/mbox/*/from ? (actual tagging is yet to come.) - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:24 ` Fco. J. Ballesteros ` (2 preceding siblings ...) 2008-06-12 12:25 ` erik quanstrom @ 2008-06-12 17:12 ` lucio 3 siblings, 0 replies; 30+ messages in thread From: lucio @ 2008-06-12 17:12 UTC (permalink / raw) To: 9fans > Why don't we reach an agreement and start using the very > same format. I suggest keeping my mbox format but adapting upas/fs to > understand it, which is a good idea. But I'm open to suggestions. ... and use base64 encoding of attachment names so they can be displayed correctly (by a clever function) and yet not interfere with the directory structure. Count me in, I'm extremely interesting in giving Plan 9 the hottest mail handling on the planet. I'll deal with the privacy encoding and decoding, specially if others keep me on track. ++L ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:04 ` erik quanstrom 2008-06-12 10:24 ` Fco. J. Ballesteros @ 2008-06-12 10:26 ` gdiaz 2008-06-12 11:09 ` erik quanstrom 1 sibling, 1 reply; 30+ messages in thread From: gdiaz @ 2008-06-12 10:26 UTC (permalink / raw) To: 9fans hello great news, i just used mbox to avoid reworking the whole thing at the same time, but you already done it. I was already aware of the mbox-pain. i'll be pleased to help on this, test, bug report, try it on a public server or whatever it's needed to finish this up. About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message. I'm not sure i understand the problem here. thanks gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 10:26 ` gdiaz @ 2008-06-12 11:09 ` erik quanstrom 2008-06-12 13:29 ` gdiaz 2008-06-12 14:10 ` Russ Cox 0 siblings, 2 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 11:09 UTC (permalink / raw) To: 9fans > About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message. why not? the "From " line is written when the message is delivered to a mb. but it's not part of the message proper. so copying a message really copies the rfc/822 part. the from line is an awkward appendage. it is strange to carry it forward in the special case of copying a mail from one box to another. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 11:09 ` erik quanstrom @ 2008-06-12 13:29 ` gdiaz 2008-06-12 14:10 ` Russ Cox 1 sibling, 0 replies; 30+ messages in thread From: gdiaz @ 2008-06-12 13:29 UTC (permalink / raw) To: 9fans hello hehe, i don't know why it is copied, probably because if you use mbox format you need the From line as a message delimeter, but if you're using other format, the From line could be avoided. As we are talking about email, may be it has a hidden meaning nobody remembers now ;). greetings, gabi ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 11:09 ` erik quanstrom 2008-06-12 13:29 ` gdiaz @ 2008-06-12 14:10 ` Russ Cox 2008-06-12 14:10 ` erik quanstrom 1 sibling, 1 reply; 30+ messages in thread From: Russ Cox @ 2008-06-12 14:10 UTC (permalink / raw) To: 9fans > why not? the "From " line is written when the message is delivered to a > mb. but it's not part of the message proper. that may be your mental model, but it's not mine. to me, the From line is as much part of the message as the rest of the mail. like i said before, it is a postmark. it indicates the time the mail was originally delivered along with the mail system's idea of the sender. there was a time, in the calm pre-rfc822 days when the from line was all the header you had. for example, here's an entire message locally delivered on plan 9 in 1992: From ken Tue Sep 8 03:42:43 EDT 1992 i finally got my copy. personally, when i've used mail systems without from lines, i've always been annoyed, because then the dates on the message are relative to someone else's clock, not my local one. i don't mean just time-zone variations; plenty of people have their clocks set wrong. i like that nedmail shows me the time the message arrived, not the claimed date in the header. if you view the date that way, as an integral part of the delivered message, it would sure be strange if saving the message to a different folder altered the delivery date. receiving a message and filing a message are two very different things. > so copying a message > really copies the rfc/822 part. the from line is an awkward appendage. > it is strange to carry it forward in the special case of copying a mail > from one box to another. the rfc822 part is the awkward appendage. mail used to be simple. russ ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [9fans] store 9p session-only values using lib9p 2008-06-12 14:10 ` Russ Cox @ 2008-06-12 14:10 ` erik quanstrom 0 siblings, 0 replies; 30+ messages in thread From: erik quanstrom @ 2008-06-12 14:10 UTC (permalink / raw) To: 9fans > if you view the date that way, as an integral part > of the delivered message, it would sure be strange > if saving the message to a different folder altered > the delivery date. receiving a message and filing > a message are two very different things. > >> so copying a message >> really copies the rfc/822 part. the from line is an awkward appendage. >> it is strange to carry it forward in the special case of copying a mail >> from one box to another. > > the rfc822 part is the awkward appendage. > mail used to be simple. a compelling argument. i agree. - erik ^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2008-06-12 17:16 UTC | newest] Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2008-06-11 20:06 [9fans] store 9p session-only values using lib9p gdiaz 2008-06-11 20:48 ` Russ Cox 2008-06-11 20:58 ` gdiaz 2008-06-11 23:44 ` erik quanstrom 2008-06-12 0:01 ` [9fans] upas/fs Russ Cox 2008-06-12 2:31 ` erik quanstrom 2008-06-12 2:59 ` Lyndon Nerenberg 2008-06-12 8:34 ` [9fans] store 9p session-only values using lib9p gdiaz 2008-06-12 10:04 ` erik quanstrom 2008-06-12 10:24 ` Fco. J. Ballesteros 2008-06-12 10:28 ` Fco. J. Ballesteros 2008-06-12 10:39 ` gdiaz 2008-06-12 10:43 ` Francisco J Ballesteros 2008-06-12 10:55 ` gdiaz 2008-06-12 11:04 ` Francisco J Ballesteros 2008-06-12 11:11 ` gdiaz 2008-06-12 17:16 ` lucio 2008-06-12 12:25 ` erik quanstrom 2008-06-12 12:32 ` erik quanstrom 2008-06-12 14:03 ` Francisco J Ballesteros 2008-06-12 14:25 ` erik quanstrom 2008-06-12 14:49 ` Fco. J. Ballesteros 2008-06-12 14:55 ` Fco. J. Ballesteros 2008-06-12 15:06 ` erik quanstrom 2008-06-12 17:12 ` lucio 2008-06-12 10:26 ` gdiaz 2008-06-12 11:09 ` erik quanstrom 2008-06-12 13:29 ` gdiaz 2008-06-12 14:10 ` Russ Cox 2008-06-12 14:10 ` erik quanstrom
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).