Re: db-backed mail back end

[Daniel Pittman <daniel@rimspace.net>]

On Wed, 23 Jan 2002, Lars Magne Ingebrigtsen wrote:
> Daniel Pittman <daniel@rimspace.net> writes:
> 
>> Gain? For me, the biggest pain with Gnus right now is that there are
>> some scalability issues. I got sick of losing information that had
>> been on mailing lists, so I stopped expiring mail a couple of years
>> ago.
> 
> A db doesn't necessarily give you any performance gain.  

No, it doesn't. 

> If you're not careful with how you set things up, you can get
> arbitrarily awful db performance (and I should know, since I've been
> Oracling a lot these past years. :-))

*grin*  We recently moved from storing our database information, for the
product I work on, from badly designed XML files (ick) to badly designed
SQL databases, resulting in a net performance loss... Heh.

Ahem. Anyway, adding a database is no assurance of better performance,
no. I suspect that you might gain more from using GDBM to hash some of
the overview and marks stuff than using SQL for data storage.

Also, it might be worth hashing the actual article storage in an NNML
group, from one directory to one-plus-n-by-m directories. :)


...but I can't say that I know any of this will work because I am still
too lazy to build the prototype, performance test it, then build the
real thing. 

>> Disk space is sufficiently cheap, even for a laptop, that I don't
>> have space issues with this. Gnus starts to feel a little pain,
>> sometimes, though when dealing with groups in the > 75,000 messages
>> range (with NNML.)
> 
> Perhaps the question should be restated as such:
> 
> If we were to design a mail back end that's supposed to scale to
> several hundred thousands of messages in thousands of groups -- how
> would we do that?  Perhaps somebody has pondered this question
> before.  :-)

That's a better question, I think. I never resolved it to my
satisfaction.

        Daniel

-- 
I am, as I said, inspired by the biological phenomena in which chemical
forces are used in repetitious fashion to produce all kinds of weird
effects (one of which is the author).
        -- Richard Feynman, _There's Plenty of Room at the Bottom_