Re: Huge groups

[Jason L Tibbitts III <tibbs@hpc.uh.edu>]

>>>>> "SB" == Steinar Bang <sb@metis.no> writes:

SB> A faster backend? Are there some formats that could be used? Isn't the
SB> qmail native format supposed to be fast? (at least according to Dan
SB> Bernstein...)

Well, I think it would be hard to get much faster than nnml.  You could
have something like nnfolder but keep an overview file that stores byte
offsets within the folder for each message, so it's one open and one seek
to grab a message.  (I use this format in an archiver I'm working with.)

But the question here is how to handle tens of thousands of messages, when
even the overhead of loading the overview file is too much.  The solution
is, of course, a database, but a simpler solution would be to just split
the messages up into smaller collections and optimize the common operations
to only deal with one piece of the collection.

I suppose this is easy enough to do manually in the cases where it is
really needed.  The only place where it can't be done is with the list
archive (which is why I brought the idea up ages ago).  The solution to
that was the ding-list-recent archive, accomplished through procmail
trickery.

 - J<