Re: fast local-file storage for Gnus

[Ted Zlatanov <tzz@lifelogs.com>]

On Tue, 03 Jun 2003, d.love@dl.ac.uk wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> it seems like a very simple feature to add BerkeleyDB/DBM
>> interfaces that can be configured at compile-time like a lot of
>> other GNU packages do (at least cfengine, that I know of),
> 
> [Ah, the cfengine libdb configuration nightmare...  You'd certainly
> have to do better than that!]

I'm sure the implementation will not be hard for people who know
autoconf well.  Emacs detects other optional libraries just fine.

> I don't know for sure, but I doubt rms would want to insist on libdb
> being available to build Emacs.  Otherwise, instances of Emacs
> wouldn't be able to run the Gnus distributed with it if you insist
> on such support.

Well, there are two options then: distribute Emacs with its own file
storage mechanism (not a bad thing, if it's optimized for Lisp data
structures), or bundle something with Emacs.  But I only want this
functionality optionally available; if Emacs doesn't find libdb for
instance maybe it can switch to something slower but universally
available such as plain text storage.  I really just want an API like
the one in XEmacs, and what happens behind the curtain is not my
concern.  That way, even a relational database could be accessed
transparently through the same mechanism; I'm definitely not
suggesting that as a goal though.

You mentioned dynamic loading, that would be nice too but could be a
problem if libdb is modified or removed.

>> Gnus could probably use fast file databases in aspects other than
>> the registry.
> 
> You haven't explained why this is necessary.  If just speed is the
> issue, how are Emacs hash tables too slow?

Hash tables are not persistent.  The Gnus newsrc.eld file, for
instance, converts a hashtable into an alist and saves that.  The
gnus-registry does the same thing with the registry hashtable.  It's
not a bad approach, but it can be slow with large alists.  I'm sure
there are other applications (BBDB, for instance) that could use this
functionality.

Here's what I want, in an ideal world: a new data structure called a
persistent-hashtable.  It should be configurable to save on every
modification or only when requested.  It can use the alist-to-file
conversion right now, there's no external linking needed.  Then,
optionally, it can load another backend, for instance BerkeleyDB or
(some day) a RDBMS gateway to a table with key-value columns.  Is this
possible?

Ted