[Caml-list] ANN - Plasma-0.6 (Map/Reduce & distributed filesystem)

[Caml-list] ANN - Plasma-0.6 (Map/Reduce & distributed filesystem)

[Gerd Stolpmann]

Hi,

I've just released Plasma-0.6, a major update of Plasma with numerous
changes. The most important:

 * For checking map/reduce out, there is now a mode that works without
   distributed filesystem, and that does not require any deployment.
   The files are just stored in the normal filesystem. Of course, the
   job can then only run on a single computer, but at least it can
   take advantage of multiple cores.

 * Added the Mapred_toolkit layer. This layer allows purely functional
   map/reduce programs. Running a job is only a matter of calling
   a function like

   val mapl_sort_fold :
        mapl:(mapred_info -> 'a -> 'b list) rfun ->
        hash:(mapred_info -> 'b -> int) rfun ->
        cmp:(mapred_info -> 'b -> 'b -> int) rfun ->
        initfold:(mapred_info -> int -> 'c) rfun ->
        fold:(mapred_info -> 'c -> 'b -> 'c * 'd list) rfun ->
        ?finfold:(mapred_info -> 'c -> 'd list) rfun ->
        partition_of:(mapred_info -> 'b -> int) rfun ->
        ?initcombine:(mapred_info -> 'e) rfun ->
        ?combine:(mapred_info -> 'e -> 'b -> 'e * 'b list) rfun ->
        ?fincombine:(mapred_info -> 'e -> 'b list) rfun ->
        'a Place.t ->
        'd Place.t ->
        config ->
        'b Place.codec ->
          ('d,[`W]) Seq.seq list result

 * Support for binary file formats

 * The sorting criterion can now be freely defined

 * Support for combiners

 * Map-only jobs are now possible

 * The RPC channels for controlling the job execution are now also
   privacy-protected. Plasma can now be run in a highly secure mode,
   where all network traffic is authenticated, encrypted and integrity-
   protected.

 * Support for counters

 * Enormous performance improvements

 * A number of bug fixes and improvements of the robustness

 * Last but not least, there is now a lot more documentation including
   a quickstart guide, the toolkit howto, various session
   transcripts.


General information about Plasma:

Plasma consists now of three parts, namely PlasmaFS, PlasmaKV, and Plasma
Map/Reduce:

      * PlasmaFS is a distributed replicating filesystem. Unlike other
        such filesystems, it is transactional and exhibits transactions
        to the user. Also, it implements almost all of what is known as
        POSIX semantics, and it is mountable.
      * PlasmaKV is a key/value database on top of PlasmaFS. It is
        designed for ultra-high read workloads, and offers interesting
        properties borrowed from PlasmaFS (e.g. replication and ACID
        transactions).
      * Plasma Map/reduce implements a variant of the popular
        data processing scheme.

All pieces of software are bundled together in one download. The
project page with further links is

http://projects.camlcity.org/projects/plasma.html

There is now also a homepage at

http://plasma.camlcity.org

THIS IS NOW A BETA RELEASE! I'm searching for testers. Whoever has
access to a cluster please check Plasma out!

Plasma is installable via GODI for Ocaml 3.12.

For discussions on specifics of Plasma there is a separate mailing list:

https://godirepo.camlcity.org/mailman/listinfo/plasma-list

Gerd
-- 
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.