nndb mailing list?

nndb mailing list?

[Joe Hildebrand]

Does anyone think an nndb mailing list would be useful?  I've been
getting some mail privately that I think more people would be
interested in.  I have a mailbot set up, but want to know whether the
discussion should take place here, or on a private list.

comments?

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              Lead Software Engineer
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: nndb mailing list?

[Greg Thompson]

>"Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:

  Joe> Does anyone think an nndb mailing list would be useful?  I've
  Joe> been getting some mail privately that I think more people would
  Joe> be interested in.  I have a mailbot set up, but want to know
  Joe> whether the discussion should take place here, or on a private
  Joe> list.

i'm not as much of an early adopter as some people around here, so i
haven't looked into nndb yet.  it sounds like it might be something
i'd switch to after i'm more familiar with it, and am satisfied that
it's stable enough to be easily installable and sufficiantly reliable.
with that in mind, i'd like to have the discussions take place here on
the ding list so i can keep up with what's going on in the nndb world,
learn a bit about the status of nndb and the problems people are or
aren't having with it, and get the info i need to make the decision to
switch over.
-- 
		-greg		       she opens her box of tricks and
				       begs me to pick from one of the
					  billion ways to feel no pain
								  -hum

Re: nndb mailing list?

[Colin Rafferty]

Joe Hildebrand writes:

> Does anyone think an nndb mailing list would be useful?  I've been
> getting some mail privately that I think more people would be
> interested in.  I have a mailbot set up, but want to know whether the
> discussion should take place here, or on a private list.

I think that it should go here, but that everybody should make sure that
they have "nndb" in the subject.  I think that at the level of mail,
just scoring on "nndb" should be enough.

-- 
Colin Rafferty
Violate the Communications Decency Act.

  "Thou shalt betroth a wife, and another man shall lie with her."
  (Punishment for anyone who violates the commandments, Deuteronomy 28:30)

Re: nndb mailing list?

[Dave Blacka]

>>>>> "gt" == Greg Thompson <gregt@visix.com> writes:

 gt> i'm not as much of an early adopter as some people around here, so i
 gt> haven't looked into nndb yet.  it sounds like it might be something
 gt> i'd switch to after i'm more familiar with it, and am satisfied that
 gt> it's stable enough to be easily installable and sufficiantly reliable.
 gt> with that in mind, i'd like to have the discussions take place here on
 gt> the ding list so i can keep up with what's going on in the nndb world,
 gt> learn a bit about the status of nndb and the problems people are or
 gt> aren't having with it, and get the info i need to make the decision to
 gt> switch over.

We can definately keep announcements and stuff for it on this list
regardless of whether or not we keep the nitty-gritty discussion of
nndb on the ding mailling list or not.

The question really is:  do people mind discussing the details of nndb
on this list, or should we move the discussions to a different list?

Dave

-- 
David Blacka                    |dblacka@fuentez.com
Software Engineer               |Fuentez Systems Concepts

Re: nndb mailing list?

[David Kågedal]

I would like to know what nndb is. I have yet only seen hints to what
its function really is.

-- 
David Kågedal     Lysator Academic Computer Society       davidk@lysator.liu.se
http://www.lysator.liu.se/~davidk/                              +46-13 17 65 89

Re: nndb mailing list?

[Joe Hildebrand]

> "David" == David K\egedal <davidk@lysator.liu.se> writes:

   David> I would like to know what nndb is. I have yet only seen
   David> hints to what its function really is.

That's because there is a massive conspiracy to keep you in the dark :)

nndb is a personal nntp server.  You put your mail into it.  The mail
is kept in a gdbm or berkeley db database, so access is about as fast
as we're going to get.  No groveling through .overview files.  No
gratuitous inode sucking.  It includes filtering rules ala procmail
(although they don't always work right), and indexed searching.  There
is the beginning of (ding) support for it.

The mail database can take a lot of disk space.

There is *no* documentation.

There is *no* guarantee of safety.

Get the current version (0.7) at:

 <URL:ftp://ftp.fuentez.com/pub/nndb.tar.gz>

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              Lead Software Engineer
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: nndb mailing list?

[Thomas Neumann]

Joe Hildebrand <hildjj@fuentez.com> writes:

> [...]  It includes filtering rules ala procmail
> (although they don't always work right), [...]

One can of course always use procmail to do the actual rule based mail
dispatching and have procmail pipe the message to nndel. That's what
I'm doing right now, and so far I'm very satisfied, having both the
power of the full procmail rule system and nndb's speedy message
access is a nice solution. I think it's time to thank the authors for
developing nndb. Nice job so far.

-t

Re: nndb mailing list?

[Lars Magne Ingebrigtsen]

Joe Hildebrand <hildjj@fuentez.com> writes:

> Does anyone think an nndb mailing list would be useful?  I've been
> getting some mail privately that I think more people would be
> interested in.  I have a mailbot set up, but want to know whether the
> discussion should take place here, or on a private list.

It's fine be me if nndb discussions take place on this list...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: nndb mailing list?

[Jason L Tibbitts III]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

LMI> It's fine be me if nndb discussions take place on this list...

It's also find by me if nndb announcements go out on ding-announce.  Just
be sure explain what it is so as to not bewilder the unwary.

BTW, I hope to have the entire ding mailing list and maybe all of my
mailing lists (a total of over 20000 articles) up on an nndb server
sometime soon (perhaps over the weekend).  I have a few issues with
authentication to deal with first but I'm not terribly concerned with
security for a first run.  This should exercise things a bit.

 - J<

Re: nndb mailing list?

[Joe Hildebrand]

> "Jason" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:

   Jason> BTW, I hope to have the entire ding mailing list and maybe
   Jason> all of my mailing lists (a total of over 20000 articles) up
   Jason> on an nndb server sometime soon (perhaps over the weekend).
   Jason> I have a few issues with authentication to deal with first
   Jason> but I'm not terribly concerned with security for a first
   Jason> run.  This should exercise things a bit.

Also BTW, we did some more testing on the differences between gdbm and
berkeley db.  Gdbm wins *big* on disk space (like one less digit in
the bytes column), and is not noticeably slower.  It will probably be
the default in 0.8.  We'll also include a script to convert one to the
other.

Does anyone know how to reliably tell what database format a file is
in?  Are there constant magic numbers for gdbm and berkeley db?  Would
it be a good thing for nndb to automatically translate formats for
you, if you start up with a different database type?

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              Lead Software Engineer
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: nndb mailing list?

[Dave Blacka]

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:

 Joe> Also BTW, we did some more testing on the differences between gdbm and
 Joe> berkeley db.  Gdbm wins *big* on disk space (like one less digit in
 Joe> the bytes column), and is not noticeably slower.  It will probably be
 Joe> the default in 0.8.  We'll also include a script to convert one to the
 Joe> other.

Ok.  Joe lied here a bit.  GDBM does actually win big on disk space:
Berkeley DB isn't really very space efficient when the data you are
storing has widely different sizes (like mail bodies), so the Berkely
DB file grows to enormous size.  For instance, I probably have at most
1 Meg of mail stored in a 7 MB mail.db file.  GDBM gets you much
closer to the actual size of the data.  However, GDBM and Berkeley DB
only have comparable speeds when you are hitting a local disk.  In my
testing so far, GDBM suffers from incredible performance problems over
NFS, while Berkeley DB behaves about the same.  Thus, if you are
storing your mail.db file on a local disk, switching over to GDBM is
probably a real good idea, otherwise it would probably be best to wait
until Joe and I rework the whole backend of nndb.

Assuming we can get all this to work, Joe and I will put all of the
message bodies in a separate database, and then write Tie:: modules to
implement different storage schemes.  The overview databases will
always be stored in a (separate) DB hash of some sort.

I'm guessing that the MH style storage scheme will probably be the
best and become the default.

Dave

-- 
David Blacka                    |dblacka@fuentez.com
Software Engineer               |Fuentez Systems Concepts

Re: nndb mailing list?

[Brad Miller]

>>>>> "Dave" == Dave Blacka <dblacka@fuentez.com> writes:

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:
Joe> Also BTW, we did some more testing on the differences between gdbm and
Joe> berkeley db.  Gdbm wins *big* on disk space (like one less digit in
Joe> the bytes column), and is not noticeably slower.  It will probably be
Joe> the default in 0.8.  We'll also include a script to convert one to the
Joe> other.

Dave> Ok.  Joe lied here a bit.  GDBM does actually win big on disk space:
Dave> Berkeley DB isn't really very space efficient when the data you are
Dave> storing has widely different sizes (like mail bodies), so the Berkely
Dave> DB file grows to enormous size.  For instance, I probably have at most
Dave> 1 Meg of mail stored in a 7 MB mail.db file.  GDBM gets you much
Dave> closer to the actual size of the data.  However, GDBM and Berkeley DB
Dave> only have comparable speeds when you are hitting a local disk.  In my
Dave> testing so far, GDBM suffers from incredible performance problems over
Dave> NFS, while Berkeley DB behaves about the same.  Thus, if you are
Dave> storing your mail.db file on a local disk, switching over to GDBM is
Dave> probably a real good idea, otherwise it would probably be best to wait
Dave> until Joe and I rework the whole backend of nndb.

The GroupLens Better Bit Bureau uses gdbm as its database, and I can
vouch for the fact that performance over NFS is really really bad.
Also, if you happen to get into a situation where you are frequently
updating the data associated with one key, the database will start to
grow all out of proportion.  The only solution to this seems to be to
call gdbm_reorganize, but that is terribly slow.  I've had some database
files go from 46Meg back down to 8Meg with one call to reorg!

\Brad
----------------------------------------------------------------------------
Brad Miller                    | e-mail: bmiller@cs.umn.edu
University of Minnesota        | phone:  (612) 626-8396     
Department of Computer Science | www:    http://www.cs.umn.edu/~bmiller
EE/CS 5-244                    | 
----------------------------------------------------------------------------
**  Save USENET Try GroupLens: http://www.cs.umn.edu/Research/GroupLens   **
----------------------------------------------------------------------------

Re: nndb mailing list?

[David Blacka]

>>>>> "Brad" == Brad Miller <bmiller@cs.umn.edu> writes:

 Brad> The GroupLens Better Bit Bureau uses gdbm as its database, and I can
 Brad> vouch for the fact that performance over NFS is really really bad.
 Brad> Also, if you happen to get into a situation where you are frequently
 Brad> updating the data associated with one key, the database will start to
 Brad> grow all out of proportion.  The only solution to this seems to be to
 Brad> call gdbm_reorganize, but that is terribly slow.  I've had some database
 Brad> files go from 46Meg back down to 8Meg with one call to reorg!
 Brad> \Brad

All gdbm_reorganize actually does is just pull everthing out of the
current database, and write it into a new one that replaces it.  So it
isn't a real speedy process, but it needs to be done to prevent the
database from growing to infinite size.

As far as nndb is concerned, we've found that the MH style backend is
probably the best solution, both space and speed wise.  While a
berkely db article database is lightning fast, it is *huge*.  The MH
style article db is not a whole lot slower, and only sucks up inodes.
The xover db (which is still some sort of db file) still can get
large, though.

I'm definately going to make the GDBM files reorganize occasionally.
Unfortunately, there isn't an equivalent berkeley db command.

-- 
David Blacka                    |dblacka@fuentez.com
Software Engineer               |Fuentez Systems Concepts