[Caml-list] Cross-platform DBM equivalent?

[Caml-list] Cross-platform DBM equivalent?

[Matt Gushee]

Hello, all--

I am developing an application that needs fast access to persistent
configuration data, and I thought that DBM might be a good way to
provide that functionality ... but I see that DBM isn't available on
Windows. Is there something similar that works on all platforms? Or an
alternative approach?

My project is a web server application, so it might be okay not to
support Windows, but I'd like it to be cross-platform if that can be
done without an unreasonable effort.

By the way, I'm at an early prototyping stage, so I can't be much more
specific about my needs. What I can say at this point is that speed is 
important; I think I need a key-value data structure, and it's probably
okay for the types of both keys and values to be limited to strings, as
with DBM.

Thanks in advance for your suggestions.

-- 
Matt Gushee                 When a nation follows the Way,
Englewood, Colorado, USA    Horses bear manure through
mgushee@havenrock.com           its fields;
http://www.havenrock.com/   When a nation ignores the Way,
                            Horses bear soldiers through
                                its streets.
                                
                            --Lao Tzu (Peter Merel, trans.)
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Alessandro Baretta]

Matt Gushee wrote:
> Hello, all--
> 
> I am developing an application that needs fast access to persistent
> configuration data, and I thought that DBM might be a good way to
> provide that functionality ...

Are you sure you need DBM to reed in a config file? Would 
the Scanf module not serve your purpose? If you need 
something really bizarre, go for PXP, the polymorphic XML 
parser, and use XML for configuration.

Alex

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Matt Gushee]

On Thu, Dec 26, 2002 at 09:39:33AM +0100, Alessandro Baretta wrote:
> 
> >I am developing an application that needs fast access to persistent
> >configuration data, and I thought that DBM might be a good way to
> >provide that functionality ...
> 
> Are you sure you need DBM to reed in a config file? Would 
> the Scanf module not serve your purpose?

Am I sure? No, I'm not sure of much of anything at the moment; as I
said, the project is at a very early stage. But I'm concerned about not
so much reading a config file as *accessing configuration data at
runtime*.

Let me try to clarify this a little. As I said, this is a web
application; its purpose is to generate on-the-fly graphs (bar graphs,
line graphs, pie charts, etc.) of numerical data (which will probably
come from a relational database in most cases). And I intend it 
eventually to be usable by not-very-technical people.

The way this is supposed to work is that when a client requests a
certain URL representing a generated image, my application obtains a 
mapping from the requested URL to the process that generates the 
image. Generally speaking, the process will consist of querying a data
source and passing the results to a library function that generates a
graphic.

Anyway, the configuration data my question referred to is the mapping of
URLs to processes. The URLs are defined by the user, and there is
potentially a large set of them. But for each request, the HTTP handler
needs to discover how to process one of those many URLs. So it's a
matter of fast random access to the configuration data, and DBM seems to
me a pretty good way to achieve that.

(I should also note that at this prototype stage, I am planning to
deploy the handler as a simple CGI program. I expect that for production
releases it would be preferable to have a long-running process that
talks to the web server through the JServ protocol or some such thing;
in that case it would probably be best to read in the URL map at startup
and just keep it in memory. But I'm trying to achieve visible results as
soon as possible, so I'd rather keep the handler component simple for
the moment).

-- 
Matt Gushee                 When a nation follows the Way,
Englewood, Colorado, USA    Horses bear manure through
mgushee@havenrock.com           its fields;
http://www.havenrock.com/   When a nation ignores the Way,
                            Horses bear soldiers through
                                its streets.
                                
                            --Lao Tzu (Peter Merel, trans.)
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Pierre Weis]

> On Thu, Dec 26, 2002 at 09:39:33AM +0100, Alessandro Baretta wrote:
> > 
> > >I am developing an application that needs fast access to persistent
> > >configuration data, and I thought that DBM might be a good way to
> > >provide that functionality ...

As far as I know the best (and simpler) way to do this for reasonable
number of URLs bindings (say thousands but not millions) is to create
a Hashtlbl.t or Map.t and dump it to file using output_value (then
read it back with input_value). In any case, I would start with this
solution, since it provides cross-platform persistency with 4 lines of
Caml code. A fast and easy way to obtain fast and good results!

Merry Christmas :)

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/

PS: Remember that input_value and output_value are not type safe. Be
careful to use some cautious programming methodology, such as the
trick I described once in this list, to define your own input and
output functions in the same phrase, in order to ensure their provably
correlated behaviour.


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Joshua D. Guttman]

Pierre Weis <pierre.weis@inria.fr> writes:

>   As far as I know the best (and simpler) way to do this for reasonable
>   number of URLs bindings (say thousands but not millions) is to create
>   a Hashtlbl.t or Map.t and dump it to file using output_value (then
>   read it back with input_value).

Is there a recommended data structure in case one needs tables for
reasonably fast access to millions or tens of millions of values?
Probably hash tables are no longer providing nearly-constant access
time at those sizes.  Is there something better in the standard
library?   

Thanks --

        Joshua 


-- 
	Joshua D. Guttman		<guttman@mitre.org> 
	MITRE, Mail Stop S119		Office:	+1 781 271 2654
	202 Burlington Rd.		Cell:	+1 781 526 5713
	Bedford, MA 01730-1420 USA	Fax:	+1 781 271 8953


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Pierre Weis]

> Pierre Weis <pierre.weis@inria.fr> writes:
> 
> >   As far as I know the best (and simpler) way to do this for reasonable
> >   number of URLs bindings (say thousands but not millions) is to create
> >   a Hashtlbl.t or Map.t and dump it to file using output_value (then
> >   read it back with input_value).
> 
> Is there a recommended data structure in case one needs tables for
> reasonably fast access to millions or tens of millions of values?
> Probably hash tables are no longer providing nearly-constant access
> time at those sizes.  Is there something better in the standard
> library?   
> 
> Thanks --
> 
>         Joshua 

You need to try :) I think that hash table and maps can handle
tens of millions of values if you have enough memory available. If
your hash tables are big enough and if your keys are reasonably
different strings, hash table will still give you nearly-constant
access time.

If you have not enough memory, you should consider mmap facilities
from the Bigarray module (memory mapping of files, accessed from the
disk by need when a given page is indeed accessed by the
application).

All the best for the next year!

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[David Brown]

On Thu, Dec 26, 2002 at 05:50:26PM +0100, Pierre Weis wrote:
> > On Thu, Dec 26, 2002 at 09:39:33AM +0100, Alessandro Baretta wrote:
> > > 
> > > >I am developing an application that needs fast access to persistent
> > > >configuration data, and I thought that DBM might be a good way to
> > > >provide that functionality ...
> 
> As far as I know the best (and simpler) way to do this for reasonable
> number of URLs bindings (say thousands but not millions) is to create
> a Hashtlbl.t or Map.t and dump it to file using output_value (then
> read it back with input_value). In any case, I would start with this
> solution, since it provides cross-platform persistency with 4 lines of
> Caml code. A fast and easy way to obtain fast and good results!

And, with another few lines of code, you can write the new data to a
second backup file, and then Sys.rename it onto the first.  Is
Sys.rename reliable on Windows, though?

Dave Brown
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Stefano Zacchiroli]

On Thu, Dec 26, 2002 at 05:50:26PM +0100, Pierre Weis wrote:
> As far as I know the best (and simpler) way to do this for reasonable
> number of URLs bindings (say thousands but not millions) is to create
> a Hashtlbl.t or Map.t and dump it to file using output_value (then
> read it back with input_value). In any case, I would start with this

Whath about memory consumption?

I know nothing about dbm internals but from my experience dbm doesn't
keep all the data in memory while Hashtbl and Map do.
Am I wrong?

Cheers.

-- 
Stefano Zacchiroli  -  Undergraduate Student of CS @ Uni. Bologna, Italy
 zack@{cs.unibo.it,debian.org,bononia.it} - http://www.bononia.it/zack/
 "I know you believe you understood what you think I said, but I am not
 sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Pierre Weis]

> On Thu, Dec 26, 2002 at 05:50:26PM +0100, Pierre Weis wrote:
> > As far as I know the best (and simpler) way to do this for reasonable
> > number of URLs bindings (say thousands but not millions) is to create
> > a Hashtlbl.t or Map.t and dump it to file using output_value (then
> > read it back with input_value). In any case, I would start with this
> 
> Whath about memory consumption?
> 
> I know nothing about dbm internals but from my experience dbm doesn't
> keep all the data in memory while Hashtbl and Map do.
> Am I wrong?
> 
> Cheers.
> 
> -- 
> Stefano Zacchiroli  -  Undergraduate Student of CS @ Uni. Bologna, Italy

You're right. If memory foot prints are of concern, you should try to
define your own data structures using big arrays and memory
mapping. However, this is not yet available in the current
implementation; we will be glad to have your contribution on this
topic.

All the best for the next year,

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Sven Luther]

On Fri, Dec 27, 2002 at 02:11:45PM +0100, Pierre Weis wrote:
> > On Thu, Dec 26, 2002 at 05:50:26PM +0100, Pierre Weis wrote:
> > > As far as I know the best (and simpler) way to do this for reasonable
> > > number of URLs bindings (say thousands but not millions) is to create
> > > a Hashtlbl.t or Map.t and dump it to file using output_value (then
> > > read it back with input_value). In any case, I would start with this
> > 
> > Whath about memory consumption?
> > 
> > I know nothing about dbm internals but from my experience dbm doesn't
> > keep all the data in memory while Hashtbl and Map do.
> > Am I wrong?
> > 
> > Cheers.
> > 
> > -- 
> > Stefano Zacchiroli  -  Undergraduate Student of CS @ Uni. Bologna, Italy
> 
> You're right. If memory foot prints are of concern, you should try to
> define your own data structures using big arrays and memory
> mapping. However, this is not yet available in the current
> implementation; we will be glad to have your contribution on this
> topic.

BTW, Does ocaml optimize two successive writes to the same bigarray
field, or is it possible to use a big array on volatile memory, like for
example the MMIO command register of some piece of hardware ?

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Dmitry Bely]

Pierre Weis <pierre.weis@inria.fr> writes:

>> > >I am developing an application that needs fast access to persistent
>> > >configuration data, and I thought that DBM might be a good way to
>> > >provide that functionality ...
>
> As far as I know the best (and simpler) way to do this for reasonable
> number of URLs bindings (say thousands but not millions) is to create
> a Hashtlbl.t or Map.t and dump it to file using output_value (then
> read it back with input_value). In any case, I would start with this
> solution, since it provides cross-platform persistency with 4 lines of
> Caml code. A fast and easy way to obtain fast and good results!

Can marshalling functions output be different for different versions of
Ocaml? If I try to feed incompatible data (e.g. form the previous version)
to input_value, what I will have then - segfault? If so, they can hardly be
used for saving configuration data.

- Dmitry Bely


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Pierre Weis]

> Pierre Weis <pierre.weis@inria.fr> writes:
> 
> >> > >I am developing an application that needs fast access to persistent
> >> > >configuration data, and I thought that DBM might be a good way to
> >> > >provide that functionality ...
> >
> > As far as I know the best (and simpler) way to do this for reasonable
> > number of URLs bindings (say thousands but not millions) is to create
> > a Hashtlbl.t or Map.t and dump it to file using output_value (then
> > read it back with input_value). In any case, I would start with this
> > solution, since it provides cross-platform persistency with 4 lines of
> > Caml code. A fast and easy way to obtain fast and good results!
> 
> Can marshalling functions output be different for different versions of
> Ocaml? If I try to feed incompatible data (e.g. form the previous version)
> to input_value, what I will have then - segfault? If so, they can hardly be
> used for saving configuration data.
> 
> - Dmitry Bely

There is is no warranty from the language definition, nor from the
implementor team, that the marshalling functions will never be
modified: imagine we find a new way to get data files much more
compact, then yes, sure, we will implement this new scheme as soon as
possible!

However, the modification of the output of marshalling functions is a
dramatic event, and in case of dramatic events the Caml team very
often provide translation programs fro the users. In the unlikely
event that the implementors do not provide such a tool, I'm pretty
sure that somebody in this list will rapidly contribute something
useful to translate from the old to the new format :)

All the best for the next year,

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[brogoff]

On Fri, 27 Dec 2002, Pierre Weis wrote:
> > Can marshalling functions output be different for different versions of
> > Ocaml? If I try to feed incompatible data (e.g. form the previous version)
> > to input_value, what I will have then - segfault? If so, they can hardly be
> > used for saving configuration data.
> > 
> > - Dmitry Bely
> 
> There is is no warranty from the language definition, nor from the
> implementor team, that the marshalling functions will never be
> modified: imagine we find a new way to get data files much more
> compact, then yes, sure, we will implement this new scheme as soon as
> possible!

One can also write IO routines which tag the file with a header string 
containing Sys.ocaml_version, and raise an exception if that header is 
incompatible with the version of OCaml you're using. That avoids the 
crash. 

Wishing for type safe value IO in OCaml for 2003!

-- Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Matt Gushee]

On Thu, Dec 26, 2002 at 05:50:26PM +0100, Pierre Weis wrote:
> > On Thu, Dec 26, 2002 at 09:39:33AM +0100, Alessandro Baretta wrote:
> > > 
> > > >I am developing an application that needs fast access to persistent
> > > >configuration data, and I thought that DBM might be a good way to
> > > >provide that functionality ...
> 
> As far as I know the best (and simpler) way to do this for reasonable
> number of URLs bindings (say thousands but not millions) is to create
> a Hashtlbl.t or Map.t and dump it to file using output_value (then
> read it back with input_value). In any case, I would start with this
> solution, since it provides cross-platform persistency with 4 lines of
> Caml code. A fast and easy way to obtain fast and good results!

That sounds good. I'll try it. Thank you very much for the suggestion!

-- 
Matt Gushee                 When a nation follows the Way,
Englewood, Colorado, USA    Horses bear manure through
mgushee@havenrock.com           its fields;
http://www.havenrock.com/   When a nation ignores the Way,
                            Horses bear soldiers through
                                its streets.
                                
                            --Lao Tzu (Peter Merel, trans.)
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Yaron M. Minsky]

Out of curiosity, why doesn't DBM work on Windows?  Berkeley DB is
certainly ported to Windows, so it seems like DBM should port over just
fine as well.  Anyone know what the issue is?

y

On Thu, 2002-12-26 at 02:17, Matt Gushee wrote:
> Hello, all--
> 
> I am developing an application that needs fast access to persistent
> configuration data, and I thought that DBM might be a good way to
> provide that functionality ... but I see that DBM isn't available on
> Windows. Is there something similar that works on all platforms? Or an
> alternative approach?
> 
> My project is a web server application, so it might be okay not to
> support Windows, but I'd like it to be cross-platform if that can be
> done without an unreasonable effort.
> 
> By the way, I'm at an early prototyping stage, so I can't be much more
> specific about my needs. What I can say at this point is that speed is 
> important; I think I need a key-value data structure, and it's probably
> okay for the types of both keys and values to be limited to strings, as
> with DBM.
> 
> Thanks in advance for your suggestions.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Cross-platform DBM equivalent?

[Xavier Leroy]

> I am developing an application that needs fast access to persistent
> configuration data, and I thought that DBM might be a good way to
> provide that functionality ... but I see that DBM isn't available on
> Windows.

As others mentioned, you could compile the C sources for a DBM
implementation on Windows, then build the Caml/DBM binding.

> Is there something similar that works on all platforms? Or an
> alternative approach?
> By the way, I'm at an early prototyping stage, so I can't be much more
> specific about my needs. What I can say at this point is that speed is 
> important; I think I need a key-value data structure, and it's probably
> okay for the types of both keys and values to be limited to strings, as
> with DBM.

I guess it all depends on the number of key-value pairs:

- For a few hundred entries, just parsing a plain text file is very
fast, e.g. using an ocamllex-generated lexer to do the parsing.  Plain
text files also make debugging easier.

- Up to 10^5 entries, input_value of a hash table or a "map" is quite
fast, e.g. less than 0.1 seconds.  This is what I use in the
SpamOracle mail filter to read the database of word frequencies.

It is true that the binary format used by input_value may change in
future OCaml releases, but so does the format of DBM databases, and
the input_value format hasn't changed in incompatible ways since 1996
or so :-)  Still, it is prudent to support conversions between
input_value form and some textual, portable form.

- Beyond 10^5 entries, either DBM or a real database (such as MySQL)
seems best.  Remember that DBM files are quite large, so an SQL
approach might be more effective.

Hope this helps,

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners