Re: Status of the nndb backend

[David Blacka <davidb@internic.net>]

>>>>> "joe" == joe hildebrand <joe.hildebrand@twcable.com> writes:

I wish I had time to work on (or even use) nndb.  I'm currently so
swamped right now, I question whether I have time to sleep.

 Kai> I wish I had time to do any work on it.  Back when I had time, I 
 Kai> tried to run nndb which dumped core while indexing messages.  As 
 Kai> I know next to nothing about Perl I was unable to find the 
 Kai> error.

 joe> I don't think we were ever able to reproduce this.  Perhaps it
 joe> was a low memory or disk space condition?  I was able to index
 joe> several thousand articles without a problem, but I had a
 joe> Sparc20, with about 8G of disk and ~128M of main memory.

No.  In fact, I've only rarely seen Perl core dump.  I was running
nndb on a HP something or other which wasn't especially endowed with
memory, but it certainly wasn't a low disk/low memory situation.
Perhaps perl 5.002 or berkeley db isn't so stable on your platform?

 Kai> I have tried to use nndb-0.14 and to issue the UPDATE command, 
 Kai> which updated a few groups then barfed.  This is with the 
 Kai> Berkeley DB backend on Perl 5.002.  Has anybody got this 
 Kai> working?  Maybe I just ought to try gdbm?

 joe> I think we decided that the Berkeley DBs were too big, and
 joe> changed the default to gdbm.  But I don't remember.  Do you,
 joe> Dave?  We did a bunch of performance tests (for a work-related
 joe> project) on the relative sizes and speeds of gdbm vs. berkeley.
 joe> I remember the verdict being that the gdbm databases were
 joe> smaller, and faster as long as you weren't accessing them via
 joe> NFS.  Over NFS, gdbm was *dog* slow.  Like 8-10 times as slow.
 joe> So if you use gdbm, put your databases on a local disk.

No, looking in the code, nndb-0.14 still has the default set as
berkeley db.  If you can store your database on a local disk, gdbm is
superior.  It is also a lot better for low memory situations.
However, if you are running over NFS (or any other situation where you
don't get OS disk caching improvements), the berkeley db hash is far
faster.

It should be fairly easy to switch.  I am reaching here, since I
haven't been using nndb for months now, but I think all you have to do
is change the configuration option and then re-index (which I think is
what happens with the UPDATE command).

 joe> Despite all of that, I would suggest mostly using the defaults,
 joe> if you can, since that is the most tested case.


-- 
David Blacka                                    Network Solutions, Inc.
Software Engineer                               Rwhois Development Team
davidb@internic.net                             Voice: (703) 742-4897