fast local-file storage for Gnus

fast local-file storage for Gnus

[Ted Zlatanov]

This may be more of an Emacs question, but what can we use to do fast
file-based storage in Gnus?  The registry and many others would
benefit from this.

If anyone knows of a simple way to tie a text key to a text value in a
single file, that's all I need (with the usual add, delete, change,
iterate functions).  If not, I could write a generic DB format in
Emacs Lisp that stores the data in a text file, encoded to escape
binary data, and indexed in some way.  I'm hoping to avoid that.

Thanks
Ted

Re: fast local-file storage for Gnus

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This may be more of an Emacs question, but what can we use to do fast
> file-based storage in Gnus?  The registry and many others would
> benefit from this.

A GDBM interface would be good.

> If anyone knows of a simple way to tie a text key to a text value in a
> single file, that's all I need (with the usual add, delete, change,
> iterate functions).  If not, I could write a generic DB format in
> Emacs Lisp that stores the data in a text file, encoded to escape
> binary data, and indexed in some way.  I'm hoping to avoid that.

BBDB uses print and read, and I guess that's the fastest you can do
in Emacs.  (Hm.  Maybe create Lisp and byte-compile it.)

-- 
This line is not blank.

Re: fast local-file storage for Gnus

[Matthias Andree]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> This may be more of an Emacs question, but what can we use to do fast
>> file-based storage in Gnus?  The registry and many others would
>> benefit from this.
>
> A GDBM interface would be good.

Ugh. SCNR.

-- 
Matthias Andree

Re: fast local-file storage for Gnus

[Ted Zlatanov]

On Sat, 24 May 2003, ma@dt.e-technik.uni-dortmund.de wrote:
> kai.grossjohann@gmx.net (Kai Großjohann) writes:

>> A GDBM interface would be good.
> 
> Ugh. SCNR.

Can you explain what you mean, please?

And how SCNR would integrate with Emacs?

Ideally, the functionality would not require patching or recompiling
Emacs, and would be entirely in Lisp, without installing external
tools.

Ted

Re: fast local-file storage for Gnus

[Kai Großjohann]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> A GDBM interface would be good.
>
> Ugh. 

Why?  May I ask?
-- 
This line is not blank.

Re: fast local-file storage for Gnus

[Matthias Andree]

On Sat, 24 May 2003, Ted Zlatanov wrote:

> On Sat, 24 May 2003, ma@dt.e-technik.uni-dortmund.de wrote:
> > kai.grossjohann@gmx.net (Kai Großjohann) writes:
> 
> >> A GDBM interface would be good.
> > 
> > Ugh. SCNR.
> 
> Can you explain what you mean, please?

Well, the GDBM history around fall 2002 wasn't very uhm convincing to me.

SuSE at that time shipped their SuSE Linux 8.1, with gdbm 1.8.0
(libgdbm.so.2) which used flock, and was thus unable to work across NFS.
Former versions of gdbm, gdbm.so.1, shipped with SuSE, had used fcntl,
and they worked across NFS. Why they chose to default to flock, I don't
know, and I haven't asked.

Then, someone decided to fix a bug and change the ABI (not the API) and
between gdbm 1.8.0 and 1.8.3 bumped the library version to gdbm.so.3,
thus, you couldn't upgrade gdbm to fix a bug without recompiling all
applications that used it. The maintainer justified himself with "it's
only an ABI change not an API change, so we needn't bump the major
revision of the software". Such idiocy makes gdbm "unstable" in my
perception.

> And how SCNR would integrate with Emacs?

Not at all, it's the acronym for "Sorry, could not resist".

> Ideally, the functionality would not require patching or recompiling
> Emacs, and would be entirely in Lisp, without installing external
> tools.

Sorry.

-- 
Matthias Andree

Re: fast local-file storage for Gnus

[Dave Love]

If there's really a reason (size?) you can't read data into Emacs,
your only option is to run something in a sub-process (à la Ispell).

Re: fast local-file storage for Gnus

[Josh Huber]

Dave Love <d.love@dl.ac.uk> writes:

> If there's really a reason (size?) you can't read data into Emacs,
> your only option is to run something in a sub-process (à la Ispell).

I take it this is not a standard feature across emacs/Xemacs:

http://www.xemacs.org/Documentation/21.5/html/lispref_52.html

I haven't used it, but wouldn't it be nice...hmm...

-- 
Josh Huber

Re: fast local-file storage for Gnus

[Ted Zlatanov]

On Fri, 30 May 2003, huber@alum.wpi.edu wrote:
> Dave Love <d.love@dl.ac.uk> writes:
> 
>> If there's really a reason (size?) you can't read data into Emacs,
>> your only option is to run something in a sub-process (à la
>> Ispell).
> 
> I take it this is not a standard feature across emacs/Xemacs:
> 
> http://www.xemacs.org/Documentation/21.5/html/lispref_52.html
> 
> I haven't used it, but wouldn't it be nice...hmm...

I don't know why it's not in Emacs yet, it seems like a very simple
feature to add BerkeleyDB/DBM interfaces that can be configured at
compile-time like a lot of other GNU packages do (at least cfengine,
that I know of), and interfaced similarly to the way XEmacs does it.
Does anyone know why it's not done yet, before I go asking
emacs-devel?

Gnus could probably use fast file databases in aspects other than the
registry.

Ted

Re: fast local-file storage for Gnus

[Dave Love]

Josh Huber <huber@alum.wpi.edu> writes:

> I take it this is not a standard feature across emacs/Xemacs:

No.  Is it even consistently present in XEmacs?

> http://www.xemacs.org/Documentation/21.5/html/lispref_52.html

Obviously, adding such a thing to Emacs wouldn't be difficult.  I did
it using my dynamic loading changes.  However, I think you'd at least
have to distribute the db library with Emacs to make sure it was
available.

Re: fast local-file storage for Gnus

[Dave Love]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't know why it's not in Emacs yet,

Partly because no-one has had a need, and perhaps partly because of
configuration issues.

> it seems like a very simple feature to add BerkeleyDB/DBM interfaces
> that can be configured at compile-time like a lot of other GNU
> packages do (at least cfengine, that I know of),

[Ah, the cfengine libdb configuration nightmare...  You'd certainly
have to do better than that!]

I don't know for sure, but I doubt rms would want to insist on libdb
being available to build Emacs.  Otherwise, instances of Emacs
wouldn't be able to run the Gnus distributed with it if you insist on
such support.

> Gnus could probably use fast file databases in aspects other than the
> registry.

You haven't explained why this is necessary.  If just speed is the
issue, how are Emacs hash tables too slow?

Re: fast local-file storage for Gnus

[Kai Großjohann]

Dave Love <d.love@dl.ac.uk> writes:

> Obviously, adding such a thing to Emacs wouldn't be difficult.  I did
> it using my dynamic loading changes.  However, I think you'd at least
> have to distribute the db library with Emacs to make sure it was
> available.

Wouldn't gdbm be sufficient?  Or maybe sdbm, the thing that comes
with Perl?
-- 
This line is not blank.

Re: fast local-file storage for Gnus

[Ted Zlatanov]

On Tue, 03 Jun 2003, d.love@dl.ac.uk wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> it seems like a very simple feature to add BerkeleyDB/DBM
>> interfaces that can be configured at compile-time like a lot of
>> other GNU packages do (at least cfengine, that I know of),
> 
> [Ah, the cfengine libdb configuration nightmare...  You'd certainly
> have to do better than that!]

I'm sure the implementation will not be hard for people who know
autoconf well.  Emacs detects other optional libraries just fine.

> I don't know for sure, but I doubt rms would want to insist on libdb
> being available to build Emacs.  Otherwise, instances of Emacs
> wouldn't be able to run the Gnus distributed with it if you insist
> on such support.

Well, there are two options then: distribute Emacs with its own file
storage mechanism (not a bad thing, if it's optimized for Lisp data
structures), or bundle something with Emacs.  But I only want this
functionality optionally available; if Emacs doesn't find libdb for
instance maybe it can switch to something slower but universally
available such as plain text storage.  I really just want an API like
the one in XEmacs, and what happens behind the curtain is not my
concern.  That way, even a relational database could be accessed
transparently through the same mechanism; I'm definitely not
suggesting that as a goal though.

You mentioned dynamic loading, that would be nice too but could be a
problem if libdb is modified or removed.

>> Gnus could probably use fast file databases in aspects other than
>> the registry.
> 
> You haven't explained why this is necessary.  If just speed is the
> issue, how are Emacs hash tables too slow?

Hash tables are not persistent.  The Gnus newsrc.eld file, for
instance, converts a hashtable into an alist and saves that.  The
gnus-registry does the same thing with the registry hashtable.  It's
not a bad approach, but it can be slow with large alists.  I'm sure
there are other applications (BBDB, for instance) that could use this
functionality.

Here's what I want, in an ideal world: a new data structure called a
persistent-hashtable.  It should be configurable to save on every
modification or only when requested.  It can use the alist-to-file
conversion right now, there's no external linking needed.  Then,
optionally, it can load another backend, for instance BerkeleyDB or
(some day) a RDBMS gateway to a table with key-value columns.  Is this
possible?

Ted

Re: fast local-file storage for Gnus

[Dave Love]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I'm sure the implementation will not be hard for people who know
> autoconf well.

[That might be me, which is why I made the comment!  The problem is in
guessing where on earth the library is, and checking versions.]

> You mentioned dynamic loading, that would be nice too but could be a
> problem if libdb is modified or removed.

It's pretty much the same as with Lisp libraries.  (It was
intentionally not done like the XEmacs implementation.)

> It's not a bad approach, but it can be slow with large alists.

You don't actually have to cons an alist to transfer data from a file
to a hash table.  (You can read data directly into a table, which is
probably faster.  If necessary, you could make a primitive to dump and
restore hash tables fast.)

If you want `fast' access to the data, you presumably have to read
them into memory anyway, so I think it needs measurements, unless
you've already made them.  I've had surprising results from measuring
this sort of thing in the past.

> Here's what I want, in an ideal world: a new data structure called a
> persistent-hashtable.

[...]

> Is this possible?

I don't see why not, if it's well designed, but you'd want to talk to
rms about it and be prepared to do the implementation.  I suspect it's
more difficult to get right than you might guess, but that's not a
reason for not doing it.

Re: fast local-file storage for Gnus

[Ted Zlatanov]

On Thu, 12 Jun 2003, d.love@dl.ac.uk wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> I'm sure the implementation will not be hard for people who know
>> autoconf well.
> 
> [That might be me, which is why I made the comment!  The problem is
> in guessing where on earth the library is, and checking versions.]

Unfortunately there's no libdb-config like for instance imlib2-config,
that sort of wrapper is very useful.  I would imagine a --with-libdb
switch, optional and left empty (no libdb) by default, would do the
job.  Anyhow, what I want is a generic interface, libdb is just a
container library and I don't want to write specifically for it.

>> It's not a bad approach, but it can be slow with large alists.
> 
> You don't actually have to cons an alist to transfer data from a
> file to a hash table.  (You can read data directly into a table,
> which is probably faster.  If necessary, you could make a primitive
> to dump and restore hash tables fast.)

Right, but I don't know how to write that primitive in C, and in Lisp
it would probably be too slow.  If we're going to write that
primitive, why not make it a generic interface with
get/set/delete/commit functions with a plain text backend?  

> If you want `fast' access to the data, you presumably have to read
> them into memory anyway, so I think it needs measurements, unless
> you've already made them.  I've had surprising results from
> measuring this sort of thing in the past.

I really don't know what to measure, since there's only the
alist-to-plain-file mechanism right now :)

>> Here's what I want, in an ideal world: a new data structure called
>> a persistent-hashtable.
> 
> [...]
> 
>> Is this possible?
> 
> I don't see why not, if it's well designed, but you'd want to talk
> to rms about it and be prepared to do the implementation.  I suspect
> it's more difficult to get right than you might guess, but that's
> not a reason for not doing it.

So emacs-devel would be the right place?

Thanks
Ted

Re: fast local-file storage for Gnus

[Dave Love]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Right, but I don't know how to write that primitive in C, and in Lisp
> it would probably be too slow.

One doesn't go writing things as primitives without measuring the Lisp
version (if writing it in Lisp is possible), at least without a lot of
experience.

> If we're going to write that
> primitive, why not make it a generic interface with
> get/set/delete/commit functions with a plain text backend?  

If you don't need to do any more than dump and restore a table, you
don't need to introduce a new Lisp type, for instance.

> I really don't know what to measure, since there's only the
> alist-to-plain-file mechanism right now :)

I don't understand why you think you need to build an alist:

(let ((table (make-hash-table)))	; should be given realistic size
  (with-temp-buffer
    (insert-file-contents "fred")
    (condition-case ()
	(let ((standard-input (current-buffer)))
	  (while t
	    (puthash (read) (read) table))) ; assumes LR arg evaluation
      (error nil)))
  table)

-- 
We should forget about small efficiencies, say about 97% of the
time: premature optimization is the root of all evil.  -- Donald Knuth