* fast local-file storage for Gnus @ 2003-05-23 20:31 Ted Zlatanov 2003-05-24 11:08 ` Kai Großjohann 2003-05-30 13:28 ` Dave Love 0 siblings, 2 replies; 16+ messages in thread From: Ted Zlatanov @ 2003-05-23 20:31 UTC (permalink / raw) This may be more of an Emacs question, but what can we use to do fast file-based storage in Gnus? The registry and many others would benefit from this. If anyone knows of a simple way to tie a text key to a text value in a single file, that's all I need (with the usual add, delete, change, iterate functions). If not, I could write a generic DB format in Emacs Lisp that stores the data in a text file, encoded to escape binary data, and indexed in some way. I'm hoping to avoid that. Thanks Ted ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-23 20:31 fast local-file storage for Gnus Ted Zlatanov @ 2003-05-24 11:08 ` Kai Großjohann [not found] ` <84wuggy548.fsf@lucy.is.informatik.uni-duis?= =?iso-8859-15?q?burg.de> 2003-05-30 13:28 ` Dave Love 1 sibling, 1 reply; 16+ messages in thread From: Kai Großjohann @ 2003-05-24 11:08 UTC (permalink / raw) Ted Zlatanov <tzz@lifelogs.com> writes: > This may be more of an Emacs question, but what can we use to do fast > file-based storage in Gnus? The registry and many others would > benefit from this. A GDBM interface would be good. > If anyone knows of a simple way to tie a text key to a text value in a > single file, that's all I need (with the usual add, delete, change, > iterate functions). If not, I could write a generic DB format in > Emacs Lisp that stores the data in a text file, encoded to escape > binary data, and indexed in some way. I'm hoping to avoid that. BBDB uses print and read, and I guess that's the fastest you can do in Emacs. (Hm. Maybe create Lisp and byte-compile it.) -- This line is not blank. ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <84wuggy548.fsf@lucy.is.informatik.uni-duis?= =?iso-8859-15?q?burg.de>]
* Re: fast local-file storage for Gnus [not found] ` <84wuggy548.fsf@lucy.is.informatik.uni-duis?= =?iso-8859-15?q?burg.de> @ 2003-05-24 12:03 ` Matthias Andree 2003-05-24 13:22 ` Ted Zlatanov 2003-05-24 13:36 ` Kai Großjohann 0 siblings, 2 replies; 16+ messages in thread From: Matthias Andree @ 2003-05-24 12:03 UTC (permalink / raw) kai.grossjohann@gmx.net (Kai Großjohann) writes: > Ted Zlatanov <tzz@lifelogs.com> writes: > >> This may be more of an Emacs question, but what can we use to do fast >> file-based storage in Gnus? The registry and many others would >> benefit from this. > > A GDBM interface would be good. Ugh. SCNR. -- Matthias Andree ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-24 12:03 ` Matthias Andree @ 2003-05-24 13:22 ` Ted Zlatanov 2003-05-24 13:47 ` Matthias Andree 2003-05-24 13:36 ` Kai Großjohann 1 sibling, 1 reply; 16+ messages in thread From: Ted Zlatanov @ 2003-05-24 13:22 UTC (permalink / raw) Cc: ding On Sat, 24 May 2003, ma@dt.e-technik.uni-dortmund.de wrote: > kai.grossjohann@gmx.net (Kai Großjohann) writes: >> A GDBM interface would be good. > > Ugh. SCNR. Can you explain what you mean, please? And how SCNR would integrate with Emacs? Ideally, the functionality would not require patching or recompiling Emacs, and would be entirely in Lisp, without installing external tools. Ted ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-24 13:22 ` Ted Zlatanov @ 2003-05-24 13:47 ` Matthias Andree 0 siblings, 0 replies; 16+ messages in thread From: Matthias Andree @ 2003-05-24 13:47 UTC (permalink / raw) Cc: Matthias Andree On Sat, 24 May 2003, Ted Zlatanov wrote: > On Sat, 24 May 2003, ma@dt.e-technik.uni-dortmund.de wrote: > > kai.grossjohann@gmx.net (Kai Großjohann) writes: > > >> A GDBM interface would be good. > > > > Ugh. SCNR. > > Can you explain what you mean, please? Well, the GDBM history around fall 2002 wasn't very uhm convincing to me. SuSE at that time shipped their SuSE Linux 8.1, with gdbm 1.8.0 (libgdbm.so.2) which used flock, and was thus unable to work across NFS. Former versions of gdbm, gdbm.so.1, shipped with SuSE, had used fcntl, and they worked across NFS. Why they chose to default to flock, I don't know, and I haven't asked. Then, someone decided to fix a bug and change the ABI (not the API) and between gdbm 1.8.0 and 1.8.3 bumped the library version to gdbm.so.3, thus, you couldn't upgrade gdbm to fix a bug without recompiling all applications that used it. The maintainer justified himself with "it's only an ABI change not an API change, so we needn't bump the major revision of the software". Such idiocy makes gdbm "unstable" in my perception. > And how SCNR would integrate with Emacs? Not at all, it's the acronym for "Sorry, could not resist". > Ideally, the functionality would not require patching or recompiling > Emacs, and would be entirely in Lisp, without installing external > tools. Sorry. -- Matthias Andree ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-24 12:03 ` Matthias Andree 2003-05-24 13:22 ` Ted Zlatanov @ 2003-05-24 13:36 ` Kai Großjohann 1 sibling, 0 replies; 16+ messages in thread From: Kai Großjohann @ 2003-05-24 13:36 UTC (permalink / raw) Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes: > kai.grossjohann@gmx.net (Kai Großjohann) writes: > >> A GDBM interface would be good. > > Ugh. Why? May I ask? -- This line is not blank. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-23 20:31 fast local-file storage for Gnus Ted Zlatanov 2003-05-24 11:08 ` Kai Großjohann @ 2003-05-30 13:28 ` Dave Love 2003-05-30 13:57 ` Josh Huber 1 sibling, 1 reply; 16+ messages in thread From: Dave Love @ 2003-05-30 13:28 UTC (permalink / raw) If there's really a reason (size?) you can't read data into Emacs, your only option is to run something in a sub-process (à la Ispell). ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-30 13:28 ` Dave Love @ 2003-05-30 13:57 ` Josh Huber 2003-05-30 15:23 ` Ted Zlatanov 2003-06-03 11:00 ` Dave Love 0 siblings, 2 replies; 16+ messages in thread From: Josh Huber @ 2003-05-30 13:57 UTC (permalink / raw) Dave Love <d.love@dl.ac.uk> writes: > If there's really a reason (size?) you can't read data into Emacs, > your only option is to run something in a sub-process (à la Ispell). I take it this is not a standard feature across emacs/Xemacs: http://www.xemacs.org/Documentation/21.5/html/lispref_52.html I haven't used it, but wouldn't it be nice...hmm... -- Josh Huber ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-30 13:57 ` Josh Huber @ 2003-05-30 15:23 ` Ted Zlatanov 2003-06-03 11:02 ` Dave Love 2003-06-03 11:00 ` Dave Love 1 sibling, 1 reply; 16+ messages in thread From: Ted Zlatanov @ 2003-05-30 15:23 UTC (permalink / raw) On Fri, 30 May 2003, huber@alum.wpi.edu wrote: > Dave Love <d.love@dl.ac.uk> writes: > >> If there's really a reason (size?) you can't read data into Emacs, >> your only option is to run something in a sub-process (à la >> Ispell). > > I take it this is not a standard feature across emacs/Xemacs: > > http://www.xemacs.org/Documentation/21.5/html/lispref_52.html > > I haven't used it, but wouldn't it be nice...hmm... I don't know why it's not in Emacs yet, it seems like a very simple feature to add BerkeleyDB/DBM interfaces that can be configured at compile-time like a lot of other GNU packages do (at least cfengine, that I know of), and interfaced similarly to the way XEmacs does it. Does anyone know why it's not done yet, before I go asking emacs-devel? Gnus could probably use fast file databases in aspects other than the registry. Ted ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-30 15:23 ` Ted Zlatanov @ 2003-06-03 11:02 ` Dave Love 2003-06-03 15:37 ` Ted Zlatanov 0 siblings, 1 reply; 16+ messages in thread From: Dave Love @ 2003-06-03 11:02 UTC (permalink / raw) Ted Zlatanov <tzz@lifelogs.com> writes: > I don't know why it's not in Emacs yet, Partly because no-one has had a need, and perhaps partly because of configuration issues. > it seems like a very simple feature to add BerkeleyDB/DBM interfaces > that can be configured at compile-time like a lot of other GNU > packages do (at least cfengine, that I know of), [Ah, the cfengine libdb configuration nightmare... You'd certainly have to do better than that!] I don't know for sure, but I doubt rms would want to insist on libdb being available to build Emacs. Otherwise, instances of Emacs wouldn't be able to run the Gnus distributed with it if you insist on such support. > Gnus could probably use fast file databases in aspects other than the > registry. You haven't explained why this is necessary. If just speed is the issue, how are Emacs hash tables too slow? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-06-03 11:02 ` Dave Love @ 2003-06-03 15:37 ` Ted Zlatanov 2003-06-12 22:25 ` Dave Love 0 siblings, 1 reply; 16+ messages in thread From: Ted Zlatanov @ 2003-06-03 15:37 UTC (permalink / raw) Cc: ding On Tue, 03 Jun 2003, d.love@dl.ac.uk wrote: > Ted Zlatanov <tzz@lifelogs.com> writes: > >> it seems like a very simple feature to add BerkeleyDB/DBM >> interfaces that can be configured at compile-time like a lot of >> other GNU packages do (at least cfengine, that I know of), > > [Ah, the cfengine libdb configuration nightmare... You'd certainly > have to do better than that!] I'm sure the implementation will not be hard for people who know autoconf well. Emacs detects other optional libraries just fine. > I don't know for sure, but I doubt rms would want to insist on libdb > being available to build Emacs. Otherwise, instances of Emacs > wouldn't be able to run the Gnus distributed with it if you insist > on such support. Well, there are two options then: distribute Emacs with its own file storage mechanism (not a bad thing, if it's optimized for Lisp data structures), or bundle something with Emacs. But I only want this functionality optionally available; if Emacs doesn't find libdb for instance maybe it can switch to something slower but universally available such as plain text storage. I really just want an API like the one in XEmacs, and what happens behind the curtain is not my concern. That way, even a relational database could be accessed transparently through the same mechanism; I'm definitely not suggesting that as a goal though. You mentioned dynamic loading, that would be nice too but could be a problem if libdb is modified or removed. >> Gnus could probably use fast file databases in aspects other than >> the registry. > > You haven't explained why this is necessary. If just speed is the > issue, how are Emacs hash tables too slow? Hash tables are not persistent. The Gnus newsrc.eld file, for instance, converts a hashtable into an alist and saves that. The gnus-registry does the same thing with the registry hashtable. It's not a bad approach, but it can be slow with large alists. I'm sure there are other applications (BBDB, for instance) that could use this functionality. Here's what I want, in an ideal world: a new data structure called a persistent-hashtable. It should be configurable to save on every modification or only when requested. It can use the alist-to-file conversion right now, there's no external linking needed. Then, optionally, it can load another backend, for instance BerkeleyDB or (some day) a RDBMS gateway to a table with key-value columns. Is this possible? Ted ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-06-03 15:37 ` Ted Zlatanov @ 2003-06-12 22:25 ` Dave Love 2003-06-13 14:33 ` Ted Zlatanov 0 siblings, 1 reply; 16+ messages in thread From: Dave Love @ 2003-06-12 22:25 UTC (permalink / raw) Ted Zlatanov <tzz@lifelogs.com> writes: > I'm sure the implementation will not be hard for people who know > autoconf well. [That might be me, which is why I made the comment! The problem is in guessing where on earth the library is, and checking versions.] > You mentioned dynamic loading, that would be nice too but could be a > problem if libdb is modified or removed. It's pretty much the same as with Lisp libraries. (It was intentionally not done like the XEmacs implementation.) > It's not a bad approach, but it can be slow with large alists. You don't actually have to cons an alist to transfer data from a file to a hash table. (You can read data directly into a table, which is probably faster. If necessary, you could make a primitive to dump and restore hash tables fast.) If you want `fast' access to the data, you presumably have to read them into memory anyway, so I think it needs measurements, unless you've already made them. I've had surprising results from measuring this sort of thing in the past. > Here's what I want, in an ideal world: a new data structure called a > persistent-hashtable. [...] > Is this possible? I don't see why not, if it's well designed, but you'd want to talk to rms about it and be prepared to do the implementation. I suspect it's more difficult to get right than you might guess, but that's not a reason for not doing it. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-06-12 22:25 ` Dave Love @ 2003-06-13 14:33 ` Ted Zlatanov 2003-06-17 22:06 ` Dave Love 0 siblings, 1 reply; 16+ messages in thread From: Ted Zlatanov @ 2003-06-13 14:33 UTC (permalink / raw) Cc: ding On Thu, 12 Jun 2003, d.love@dl.ac.uk wrote: > Ted Zlatanov <tzz@lifelogs.com> writes: > >> I'm sure the implementation will not be hard for people who know >> autoconf well. > > [That might be me, which is why I made the comment! The problem is > in guessing where on earth the library is, and checking versions.] Unfortunately there's no libdb-config like for instance imlib2-config, that sort of wrapper is very useful. I would imagine a --with-libdb switch, optional and left empty (no libdb) by default, would do the job. Anyhow, what I want is a generic interface, libdb is just a container library and I don't want to write specifically for it. >> It's not a bad approach, but it can be slow with large alists. > > You don't actually have to cons an alist to transfer data from a > file to a hash table. (You can read data directly into a table, > which is probably faster. If necessary, you could make a primitive > to dump and restore hash tables fast.) Right, but I don't know how to write that primitive in C, and in Lisp it would probably be too slow. If we're going to write that primitive, why not make it a generic interface with get/set/delete/commit functions with a plain text backend? > If you want `fast' access to the data, you presumably have to read > them into memory anyway, so I think it needs measurements, unless > you've already made them. I've had surprising results from > measuring this sort of thing in the past. I really don't know what to measure, since there's only the alist-to-plain-file mechanism right now :) >> Here's what I want, in an ideal world: a new data structure called >> a persistent-hashtable. > > [...] > >> Is this possible? > > I don't see why not, if it's well designed, but you'd want to talk > to rms about it and be prepared to do the implementation. I suspect > it's more difficult to get right than you might guess, but that's > not a reason for not doing it. So emacs-devel would be the right place? Thanks Ted ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-06-13 14:33 ` Ted Zlatanov @ 2003-06-17 22:06 ` Dave Love 0 siblings, 0 replies; 16+ messages in thread From: Dave Love @ 2003-06-17 22:06 UTC (permalink / raw) Ted Zlatanov <tzz@lifelogs.com> writes: > Right, but I don't know how to write that primitive in C, and in Lisp > it would probably be too slow. One doesn't go writing things as primitives without measuring the Lisp version (if writing it in Lisp is possible), at least without a lot of experience. > If we're going to write that > primitive, why not make it a generic interface with > get/set/delete/commit functions with a plain text backend? If you don't need to do any more than dump and restore a table, you don't need to introduce a new Lisp type, for instance. > I really don't know what to measure, since there's only the > alist-to-plain-file mechanism right now :) I don't understand why you think you need to build an alist: (let ((table (make-hash-table))) ; should be given realistic size (with-temp-buffer (insert-file-contents "fred") (condition-case () (let ((standard-input (current-buffer))) (while t (puthash (read) (read) table))) ; assumes LR arg evaluation (error nil))) table) -- We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. -- Donald Knuth ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-05-30 13:57 ` Josh Huber 2003-05-30 15:23 ` Ted Zlatanov @ 2003-06-03 11:00 ` Dave Love 2003-06-03 13:35 ` Kai Großjohann 1 sibling, 1 reply; 16+ messages in thread From: Dave Love @ 2003-06-03 11:00 UTC (permalink / raw) Josh Huber <huber@alum.wpi.edu> writes: > I take it this is not a standard feature across emacs/Xemacs: No. Is it even consistently present in XEmacs? > http://www.xemacs.org/Documentation/21.5/html/lispref_52.html Obviously, adding such a thing to Emacs wouldn't be difficult. I did it using my dynamic loading changes. However, I think you'd at least have to distribute the db library with Emacs to make sure it was available. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: fast local-file storage for Gnus 2003-06-03 11:00 ` Dave Love @ 2003-06-03 13:35 ` Kai Großjohann 0 siblings, 0 replies; 16+ messages in thread From: Kai Großjohann @ 2003-06-03 13:35 UTC (permalink / raw) Dave Love <d.love@dl.ac.uk> writes: > Obviously, adding such a thing to Emacs wouldn't be difficult. I did > it using my dynamic loading changes. However, I think you'd at least > have to distribute the db library with Emacs to make sure it was > available. Wouldn't gdbm be sufficient? Or maybe sdbm, the thing that comes with Perl? -- This line is not blank. ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2003-06-17 22:06 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-05-23 20:31 fast local-file storage for Gnus Ted Zlatanov 2003-05-24 11:08 ` Kai Großjohann [not found] ` <84wuggy548.fsf@lucy.is.informatik.uni-duis?= =?iso-8859-15?q?burg.de> 2003-05-24 12:03 ` Matthias Andree 2003-05-24 13:22 ` Ted Zlatanov 2003-05-24 13:47 ` Matthias Andree 2003-05-24 13:36 ` Kai Großjohann 2003-05-30 13:28 ` Dave Love 2003-05-30 13:57 ` Josh Huber 2003-05-30 15:23 ` Ted Zlatanov 2003-06-03 11:02 ` Dave Love 2003-06-03 15:37 ` Ted Zlatanov 2003-06-12 22:25 ` Dave Love 2003-06-13 14:33 ` Ted Zlatanov 2003-06-17 22:06 ` Dave Love 2003-06-03 11:00 ` Dave Love 2003-06-03 13:35 ` Kai Großjohann
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).