Re: Status of the nndb backend

Re: Status of the nndb backend

[David Blacka]

>>>>> "joe" == joe hildebrand <joe.hildebrand@twcable.com> writes:

I wish I had time to work on (or even use) nndb.  I'm currently so
swamped right now, I question whether I have time to sleep.

 Kai> I wish I had time to do any work on it.  Back when I had time, I 
 Kai> tried to run nndb which dumped core while indexing messages.  As 
 Kai> I know next to nothing about Perl I was unable to find the 
 Kai> error.

 joe> I don't think we were ever able to reproduce this.  Perhaps it
 joe> was a low memory or disk space condition?  I was able to index
 joe> several thousand articles without a problem, but I had a
 joe> Sparc20, with about 8G of disk and ~128M of main memory.

No.  In fact, I've only rarely seen Perl core dump.  I was running
nndb on a HP something or other which wasn't especially endowed with
memory, but it certainly wasn't a low disk/low memory situation.
Perhaps perl 5.002 or berkeley db isn't so stable on your platform?

 Kai> I have tried to use nndb-0.14 and to issue the UPDATE command, 
 Kai> which updated a few groups then barfed.  This is with the 
 Kai> Berkeley DB backend on Perl 5.002.  Has anybody got this 
 Kai> working?  Maybe I just ought to try gdbm?

 joe> I think we decided that the Berkeley DBs were too big, and
 joe> changed the default to gdbm.  But I don't remember.  Do you,
 joe> Dave?  We did a bunch of performance tests (for a work-related
 joe> project) on the relative sizes and speeds of gdbm vs. berkeley.
 joe> I remember the verdict being that the gdbm databases were
 joe> smaller, and faster as long as you weren't accessing them via
 joe> NFS.  Over NFS, gdbm was *dog* slow.  Like 8-10 times as slow.
 joe> So if you use gdbm, put your databases on a local disk.

No, looking in the code, nndb-0.14 still has the default set as
berkeley db.  If you can store your database on a local disk, gdbm is
superior.  It is also a lot better for low memory situations.
However, if you are running over NFS (or any other situation where you
don't get OS disk caching improvements), the berkeley db hash is far
faster.

It should be fairly easy to switch.  I am reaching here, since I
haven't been using nndb for months now, but I think all you have to do
is change the configuration option and then re-index (which I think is
what happens with the UPDATE command).

 joe> Despite all of that, I would suggest mostly using the defaults,
 joe> if you can, since that is the most tested case.


-- 
David Blacka                                    Network Solutions, Inc.
Software Engineer                               Rwhois Development Team
davidb@internic.net                             Voice: (703) 742-4897

Re: Status of the nndb backend

[joe.hildebrand]

>>>>> Kees de Bruin writes:

  KdB> Hello, I followed the discussion about the nndb backend, but 
  KdB> recently nothing has been said about it anymore. Is it still 
  KdB> under development, can it already be used, or what?

The reason you haven't seen anything is that both Dave Blacka and myself
moved to new jobs a few months ago.  My new job uses cc:Mail (which really,
truly sucks rocks) on Windows (I have a windows box just to read mail.  cost
effective, no?), so I wasn't able to do any of my own mail processing.
Dave is now at Network Solutions, doing rwhois development, so he has a lot
less time than he did.

I am moving to a new, new, job next week.  Perhaps there I'll have the
infrastructure in place to do more work on nndb.

0.14 should be usable; both Dave and I were using it as our primary mail
spooling tool.  We had both stopped using procmail, even.

  Kai> I wish I had time to do any work on it.  Back when I had time, I 
  Kai> tried to run nndb which dumped core while indexing messages.  As 
  Kai> I know next to nothing about Perl I was unable to find the 
  Kai> error.

I don't think we were ever able to reproduce this.  Perhaps it was a low
memory or disk space condition?  I was able to index several thousand articles 
without a problem, but I had a Sparc20, with about 8G of disk and ~128M of
main memory.

  Kai> I have tried to use nndb-0.14 and to issue the UPDATE command, 
  Kai> which updated a few groups then barfed.  This is with the 
  Kai> Berkeley DB backend on Perl 5.002.  Has anybody got this 
  Kai> working?  Maybe I just ought to try gdbm?

I think we decided that the Berkeley DBs were too big, and changed the default
to gdbm.  But I don't remember.  Do you, Dave?  We did a bunch of performance
tests (for a work-related project) on the relative sizes and speeds of gdbm
vs. berkeley.  I remember the verdict being that the gdbm databases were 
smaller, and faster as long as you weren't accessing them via NFS.  Over NFS,
gdbm was *dog* slow.  Like 8-10 times as slow.  So if you use gdbm, put your
databases on a local disk.

Despite all of that, I would suggest mostly using the defaults, if you can,
since that is the most tested case.

Re: Status of the nndb backend

[Kai Grossjohann]

>>>>> Kees de Bruin writes:

  KdB> Hello, I followed the discussion about the nndb backend, but
  KdB> recently nothing has been said about it anymore. Is it still
  KdB> under development, can it already be used, or what?

I wish I had time to do any work on it.  Back when I had time, I tried
to run nndb which dumped core while indexing messages.  As I know next
to nothing about Perl I was unable to find the error.

I have tried to use nndb-0.14 and to issue the UPDATE command, which
updated a few groups then barfed.  This is with the Berkeley DB
backend on Perl 5.002.  Has anybody got this working?  Maybe I just
ought to try gdbm?

Also, I'll mostly be gone till then end of September :-(

kai
-- 
What's a signature?

Status of the nndb backend

[Kees de Bruin]

Hello,

I followed the discussion about the nndb backend, but recently nothing has
been said about it anymore. Is it still under development, can it already
be used, or what?

Kind regards,

Kees de Bruin.

-- 
 _      _ _
| | ___| | |_   Kees de Bruin              mailto: kees_de_bruin@tasking.nl
| |/ / _ | _ \
|_|\_\___|___/  Tasking Software BV, The Netherlands, tel. +31-33-455 85 84

Life is painting a picture, not doing a sum     -Oliver Wendell Holmes, Jr.