9fans - fans of the OS Plan 9 from Bell Labs
 help / color / mirror / Atom feed
* Re: [9fans] Scaleable mail repositories.
@ 2005-10-31 15:33 Fco. J. Ballesteros
  2005-10-31 18:38 ` William Josephson
  0 siblings, 1 reply; 27+ messages in thread
From: Fco. J. Ballesteros @ 2005-10-31 15:33 UTC (permalink / raw)
  To: 9fans

:  5 seconds per 1000 seems a tad slow to me. But I'm picky. It still might 
:  be worth it since you get all the tools. It's certainly faster than mh 
:  ever was.

Yep. I agree. I think that the problem is that you need something else
(appart from grep) to search on big file trees. Any search tool that lets
you lookup file paths by content would work with your mail as well.
We have a home grown program that does that,
but it does not work well enough and
does not deserve distribution.

I have placed a copy of our local /sys/src/cmd/mail2fs at
/n/sources/contrib/nemo/mail2fs, in case anyone wants to experiment before
we clean it up.



^ permalink raw reply	[flat|nested] 27+ messages in thread
* Re: [9fans] Scaleable mail repositories.
@ 2005-11-09  9:45 Fco. J. Ballesteros
  2005-11-09 10:24 ` Charles Forsyth
  0 siblings, 1 reply; 27+ messages in thread
From: Fco. J. Ballesteros @ 2005-11-09  9:45 UTC (permalink / raw)
  To: 9fans

:  that would be nice, but i think it's a bit ambitious for what i'm
:  looking at currently.  the search engine would have to be quite
:  intelligent:
:  
:  1) it would have to be triggered on the arrival of new mail (otherwise
:  newly arrived messages would not be held in the index)

It would have to be triggered on the changing of files in the
file system. With some help from the fs, this becomes cheap.

:  2) it would have to know which parts of the file system contained
:  mail messages and MIME parse them (assuming the mail files
:  were stored in raw format, which seems necessary for digital
:  signature verification, not to mention efficiency of delivery
:  and storage).

Don't agree. I store the messages in cooked format. That makes it easy to
understand mime :-)
If you want the raw message for whatever purposes,
you might also keep that thing. 



^ permalink raw reply	[flat|nested] 27+ messages in thread
* Re: [9fans] Scaleable mail repositories.
@ 2005-10-31 15:19 Fco. J. Ballesteros
  0 siblings, 0 replies; 27+ messages in thread
From: Fco. J. Ballesteros @ 2005-10-31 15:19 UTC (permalink / raw)
  To: 9fans

:  And remember, whatever new scheme wins, it still has to be
:  able to read my 20+ years of mbox format messages too.

It can ;-), it's just a matter of converting them...

for (m in yourmboxes){
	mail2fs -d $home/mail/$m $m
}

I mean, I still use omail (~ acme's Mail) to read mboxes I did not
convert, and convert some of them to the dir hier. fmt. as I use them.
But IMHO, there is no problem with that.



^ permalink raw reply	[flat|nested] 27+ messages in thread
* Re: [9fans] Scaleable mail repositories.
@ 2005-10-31 15:14 Fco. J. Ballesteros
  2005-10-31 16:22 ` Ronald G Minnich
  0 siblings, 1 reply; 27+ messages in thread
From: Fco. J. Ballesteros @ 2005-10-31 15:14 UTC (permalink / raw)
  To: 9fans

It's implemented. I'm using it :-)
Not a rocket, but this is fast enough I'd say (thanks to fossil/venti):

; cd /usr/nemo/mail/mbox
; ls | wc
   1294    1294    9050	# 1294 mails, one dir per mail
; time grep 'no need to argue' */text 
3376/text:no need to argue.
3377/text:> no need to argue.
0.03u 0.10s 5.07r 	 grep no need to argue 3101/text 3373/text 3376/text ...

:  
:  just run mh 'scan' on 1000 files and make it as fast as the old 'msg' 
:  utility (which I went to from mh) and I'll buy it. MH got so painfully 
:  slow for me that I couldn't take it.
:  
:  But, hey, implement it and let's see .



^ permalink raw reply	[flat|nested] 27+ messages in thread
* Re: [9fans] Scaleable mail repositories.
@ 2005-10-31 11:32 Fco. J. Ballesteros
  2005-10-31 16:01 ` Ronald G Minnich
  0 siblings, 1 reply; 27+ messages in thread
From: Fco. J. Ballesteros @ 2005-10-31 11:32 UTC (permalink / raw)
  To: 9fans

It's easy to write file servers, but that does not mean that it's the
right way to proceed. IMHO, if you want to see your mail as files, and
you have a file server, it's easier to store the mail in that format. All
the code necessary to handle your storage and index structure becomes
fossil/venti, and all that has to be done is to convert from the mbox format
into your preferred archival format, and to feed upas with input messages
for sending. Isn't this more simple and powerful? Or are you thinking of
something else that is best done using the existing format?

:  i don't understand why, now that it's easy to write file servers
:  (compared to unix days), it's necessary to store the mail messages
:  as actual separate files or directories.  the main problem with
:  upas/fs i find is that it rewrites the file instead of treating it
:  as append-only, and it reads the whole thing into memory (in a moderately
:  bulky format); rather than maintaining a separate index file or files,
:  and loading as needed.   both the storage and index structure can
:  then be made suitable for the task.



^ permalink raw reply	[flat|nested] 27+ messages in thread
* Re: [9fans] rfork(RFPROC) and ffork()
@ 2005-10-30  1:10 geoff
  2005-10-30  1:18 ` Paul Lalonde
  0 siblings, 1 reply; 27+ messages in thread
From: geoff @ 2005-10-30  1:10 UTC (permalink / raw)
  To: quanstro, 9fans

I don't know either.  Creating a new file for each incoming message
seemed like an obvious thing to do in the mid-1980s, though the
concern then might have been i-node consumption on Unixes.

I co-wrote a message store in Inferno while at the labs that decoded
MIME content-transfer-encodings as it read each message off the
network and decoded each part into a separate file in a directory tree
that reflected the hierarchical structure of the MIME message.
upas/fs came later and does the MIME decoding, but not the breaking up
into separate files upon reception.



^ permalink raw reply	[flat|nested] 27+ messages in thread

end of thread, other threads:[~2005-11-10 11:55 UTC | newest]

Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-10-31 15:33 [9fans] Scaleable mail repositories Fco. J. Ballesteros
2005-10-31 18:38 ` William Josephson
  -- strict thread matches above, loose matches on Subject: below --
2005-11-09  9:45 Fco. J. Ballesteros
2005-11-09 10:24 ` Charles Forsyth
2005-11-09 14:19   ` Sam
2005-11-10  1:24     ` erik quanstrom
2005-11-10  2:30       ` Russ Cox
2005-11-10  6:33         ` Scott Schwartz
2005-11-10 11:55         ` erik quanstrom
2005-10-31 15:19 Fco. J. Ballesteros
2005-10-31 15:14 Fco. J. Ballesteros
2005-10-31 16:22 ` Ronald G Minnich
2005-10-31 18:37   ` William Josephson
2005-10-31 11:32 Fco. J. Ballesteros
2005-10-31 16:01 ` Ronald G Minnich
2005-10-31 15:06   ` jmk
2005-10-30  1:10 [9fans] rfork(RFPROC) and ffork() geoff
2005-10-30  1:18 ` Paul Lalonde
2005-10-31  4:06   ` [9fans] Scaleable mail repositories Lyndon Nerenberg
2005-10-31 10:55     ` C H Forsyth
2005-10-31 12:32       ` erik quanstrom
2005-11-01 19:56         ` rog
2005-11-01 22:29           ` Francisco J Ballesteros
2005-11-08 19:56             ` rog
2005-11-08 23:22               ` Joel Salomon
2005-11-09  0:51               ` Caerwyn Jones
2005-11-09  0:55                 ` Russ Cox
2005-11-09  3:32               ` erik quanstrom
2005-10-31 15:30       ` jmk

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).