Core has landed

Core has landed

[Yaron Minsky]

We are proud to announce the first public release of core, Jane
Street's own alternative to OCaml's standard library.  We use this
library as the base for our own development, and we hope people on the
outside will find some use for it as well.

People should be warned that core is still in flux: there are
interfaces that we have plans to change, so if you're not willing to
come along for the ride, you shouldn't use it.  Also, be warned that
conformance with the OCaml standard library is not a goal, and we have
already deviated from it in a number of ways.  It's also worth noting
that we have only used and tested this library on x86 and x86-64 on
Linux, and we make no claims about other platforms.

You can find the library here:

   http://ocaml.janestcapital.com/?q=node/13

along with three other libraries that you will need to use along with
it: type-conv, sexplib and bin-prot.  These three libraries
provide macros for generating functions for serializing and
deserializing types. sexplib uses a human-readable s-expression
format, and bin-prot uses a high-performance binary protocol, and
type-conv is the common base of the other two libraries.  This is
also the first public release of bin-prot, and like sexplib, that
library can be used independently of core.

If you have any comments or patches, we'd love to hear about it.  Our
blog is a great place for comments:

   http://ocaml.janestcapital.com/?q=node/27

and patches should be sent to opensource@janestcapital.com.

All of the released libraries are licensed under the
LGPL-plus-linking-exception that is used by the OCaml standard
library.

y

-- 
Yaron Minsky <yminsky@janestcapital.com>

Re: [Caml-list] Core has landed

[Shivkumar Chandrasekaran]

On Fri, 2008-05-02 at 17:39 -0400, Yaron Minsky wrote:
> You can find the library here:
> 
>    http://ocaml.janestcapital.com/?q=node/13
> 
> along with three other libraries that you will need to use along with
> it: type-conv, sexplib and bin-prot.

Will these be released as GODI packages too?

Thanks for the great work (and making it freely available).

--shiv--

Re: [Caml-list] Core has landed

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Fri, May 2, 2008 at 7:40 PM, Shivkumar Chandrasekaran <shiv@ece.ucsb.edu>
wrote:

>
> On Fri, 2008-05-02 at 17:39 -0400, Yaron Minsky wrote:
> > You can find the library here:
> >
> >    http://ocaml.janestcapital.com/?q=node/13
> >
> > along with three other libraries that you will need to use along with
> > it: type-conv, sexplib and bin-prot.
>
> Will these be released as GODI packages too?
>

Yes, we plan to do that eventually.


> Thanks for the great work (and making it freely available).
>

Glad to do it.

y


>
> --shiv--
>
>
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 1932 bytes --]

Re: [Caml-list] Core has landed

[Berke Durak]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

That's wonderful!  An impressive extended core library.  Thanks a lot for
releasing that.  It's like receiving a new toy.

Now a few quick comments:

- Bigstring: I guess those are more for I/O.  The 16MB limit is not a
problem on 64-bit arhictectures.

- POLL and NODELAY in Linux_ext: that's very welcome.

- I guess JS isn't using Unicode.  After all, EBCDIC should be good enough
for stock tickers.  A Unicode library would have been nice - otherwise we
should complete the one in ExtLib.

- In tuple.ml, I find this worrying comment:

  The raison d'être for Space_safe_tuple<N> is that OCaml doesn't properly
  free variables matched in tuple patterns.

I didn't expect that, and that's quite annoying!  Shouldn't that be fixed in
the
Ocaml compiler?

- piecewise_linear.ml: I wonder what these are for.  Maybe for video games?

A very nice contribution to a "batteries included" standard library, indeed.
-- 
Berke Durak

[-- Attachment #2: Type: text/html, Size: 1183 bytes --]

Re: [Caml-list] Core has landed

[Richard Jones]

Thanks - I think this is an impressive contribution to OCaml & open
source.  For those not aware of the full extent of this library, I
went through and noted down this non-exhaustive list of features:

 - Bag (set type with duplicates)
 - Bigbuffer (unlimited Buffer type)
 - Bigstring (unlimited String type)
 - 8/16/32/64-bit signed/unsigned binary packing functions
 - Bool type
 - Safe finalization for reading/writing files
 - Function composition operators
 - Enhanced versions of stdlib modules such as Arg, Array, Printf, etc.
 - Mutexes
 - CRC functions
 - Dequeue type
 - Doubly-linked list type
 - Enhanced exception module
 - Fast hash table
 - Force once (thunk that can be forced only once)
 - Functional queue type
 - Min-heap type
 - Enhanced input/output channels
 - Closed interval type
 - Interval set type
 - Read files as lines
 - Linux-specific syscalls such as sendfile, get/set TCP options, epoll,
   splice
 - Memoization
 - Piece-wise linear interpolation of floats
 - Polymorphic map and set
 - Find size of OCaml structures
 - Space-efficient tuples
 - Synchronized queues
 - Thread-safe queues
 - Convenience functions for Unix times
 - Timed events
 - Tuple convenience functions
 - Extended Unix module filling in some missing syscalls such as
   sync, getrusage, inigroups, etc.

There's a Fedora package here:

  https://bugzilla.redhat.com/show_bug.cgi?id=445074

but note you will also need these dependencies:

  https://bugzilla.redhat.com/showdependencytree.cgi?id=445074

It would be nice to have bin-prot working on all architectures ...

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Richard Jones]

On Fri, May 02, 2008 at 05:39:41PM -0400, Yaron Minsky wrote:
> along with three other libraries that you will need to use along with
> it: type-conv, sexplib and bin-prot.

I should add that the full list of deps includes also RES and oUnit.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Richard Jones]

On Sat, May 03, 2008 at 10:07:27AM +0200, Berke Durak wrote:
> - I guess JS isn't using Unicode.  After all, EBCDIC should be good enough
> for stock tickers.  A Unicode library would have been nice - otherwise we
> should complete the one in ExtLib.

There's a perfectly good Unicode library for OCaml, called Camomile.

http://camomile.sourceforge.net/

It's been around for ages & works just fine.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Berke Durak]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

On Sat, May 3, 2008 at 10:22 AM, Richard Jones <rich@annexia.org> wrote:

>
> There's a perfectly good Unicode library for OCaml, called Camomile.
>
> http://camomile.sourceforge.net/
>
> It's been around for ages & works just fine.
>

I know, that's the one whose UTF8 module is in ExtLib.  For manipulating
Unicode
chars it is very complete and nice, but for UTF8 it lacks a few basic
functions.
I contributed two, substring extraction and appending a uChar as UTF8 to an
output_channel without creating a buffer, which should come out with the
next ExtLib,
but there are still missing basic functions (such as conversion functions
from buffer to uchar
array, etc.).
-- 
Berke

[-- Attachment #2: Type: text/html, Size: 1094 bytes --]

Re: [Caml-list] Core has landed

[Richard Jones]

On Sat, May 03, 2008 at 10:54:24AM +0200, Berke Durak wrote:
> I know, that's the one whose UTF8 module is in ExtLib.  For manipulating
> Unicode
> chars it is very complete and nice, but for UTF8 it lacks a few basic
> functions.
> I contributed two, substring extraction and appending a uChar as UTF8 to an
> output_channel without creating a buffer, which should come out with the
> next ExtLib,
> but there are still missing basic functions (such as conversion functions
> from buffer to uchar
> array, etc.).

Adding stuff to the extlib UTF8 module without having the
corresponding changes in Camomile is dangerous - the two cannot then
be used at the same time.  This is really a problem with extlib, it
just shouldn't be distributing this module.

In any case, why don't you just use Camomile?

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[David Teller]

We're integrating both Camomile and ExtLib (with shared UTF8 and UChar)
in Batteries Included, if you're interested.

Cheers,
 David


On Sat, 2008-05-03 at 10:08 +0100, Richard Jones wrote:
> Adding stuff to the extlib UTF8 module without having the
> corresponding changes in Camomile is dangerous - the two cannot then
> be used at the same time.  This is really a problem with extlib, it
> just shouldn't be distributing this module.
> 
> In any case, why don't you just use Camomile?
> 
> Rich.
> 
-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [Caml-list] Core has landed

[Richard Jones]

On Sat, May 03, 2008 at 11:14:12AM +0200, Berke Durak wrote:
> On Sat, May 3, 2008 at 11:08 AM, Richard Jones <rich@annexia.org> wrote:
>>  In any case, why don't you just use Camomile?
> I needed UTF and I was already using ExtLib.  But it's the same module
> anyway - so
> substring was missing from both.

Seriously, just what distribution are you using where installing both
isn't a single command away?  Surely you can run GODI at least, if not
a distro-specific command?  Last time I looked GODI ran on just about
anything except a ZX81 ...

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Stefano Zacchiroli]

On Sat, May 03, 2008 at 11:18:49AM +0200, David Teller wrote:
> We're integrating both Camomile and ExtLib (with shared UTF8 and UChar)
> in Batteries Included, if you're interested.

Can you comment on the possibility of integrating Core?

Thanks in advance,
Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Core has landed

[Berke Durak]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On Sat, May 3, 2008 at 11:47 AM, Richard Jones <rich@annexia.org> wrote:

> On Sat, May 03, 2008 at 11:14:12AM +0200, Berke Durak wrote:
> > On Sat, May 3, 2008 at 11:08 AM, Richard Jones <rich@annexia.org> wrote:
> >>  In any case, why don't you just use Camomile?
> > I needed UTF and I was already using ExtLib.  But it's the same module
> > anyway - so
> > substring was missing from both.
>
> Seriously, just what distribution are you using where installing both
> isn't a single command away?  Surely you can run GODI at least, if not
> a distro-specific command?  Last time I looked GODI ran on just about
> anything except a ZX81 ...


I'm using (Debian|Ubuntu)+GODI.

ExtLib uses Camomile's UTF8 module by Yamagata Yoriyuki which was missing a
UTF8.substring function I needed (and I wanted a fast one since I was
processing
200GB of Wikipedia stuff).

So I could have installed both Camomile and ExtLib but that wouldn't have
solved
my problem.
-- 
Berke

[-- Attachment #2: Type: text/html, Size: 1381 bytes --]

Re: [Caml-list] Core has landed

[Daniel Bünzli]

Le 3 mai 08 à 11:50, Stefano Zacchiroli a écrit :

> On Sat, May 03, 2008 at 11:18:49AM +0200, David Teller wrote:
>> We're integrating both Camomile and ExtLib (with shared UTF8 and  
>> UChar)
>> in Batteries Included, if you're interested.
>
> Can you comment on the possibility of integrating Core?

That's precisely the problem of these monolithic batteries included  
things, what gets in, what doesn't, what is redundant with other  
packages, what are my exact needs, what if suddenly a developer offers  
a better module.

Again, creating a developer-oriented module distribution/installation  
mechanism would allow to easily get all these modules separately and  
cherry-pick them. With all due respect for the work done I think the  
monolithic batteries included thing is solving the wrong problem --  
perl has no batteries included library, the batteries are cpan.

Best,

Daniel

Re: [Caml-list] Core has landed

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Sat, May 3, 2008 at 4:07 AM, Berke Durak <berke.durak@gmail.com> wrote:

> That's wonderful!  An impressive extended core library.  Thanks a lot for
> releasing that.  It's like receiving a new toy.
>
> Now a few quick comments:
>
> - Bigstring: I guess those are more for I/O.  The 16MB limit is not a
> problem on 64-bit arhictectures.


I/O, memory mapping, a few other things.


> - POLL and NODELAY in Linux_ext: that's very welcome.
>
> - I guess JS isn't using Unicode.  After all, EBCDIC should be good enough
> for stock tickers.  A Unicode library would have been nice - otherwise we
> should complete the one in ExtLib.


It's true, we don't really use Unicode at all for the stuff we do.


> - In tuple.ml, I find this worrying comment:
>
>   The raison d'être for Space_safe_tuple<N> is that OCaml doesn't properly
>   free variables matched in tuple patterns.
>
> I didn't expect that, and that's quite annoying!  Shouldn't that be fixed
> in the
> Ocaml compiler?


We would prefer that too, but we haven't been able to convince anyone that
it's worth doing.  There are apparently some reasons that it's painful to
fix in the bytecode compiler, and there's some reluctance to make the GC
behavior be significantly different between native and bytecodes.


> - piecewise_linear.ml: I wonder what these are for.  Maybe for video
> games?
>
> A very nice contribution to a "batteries included" standard library,
> indeed.



> --
> Berke Durak
>
>

[-- Attachment #2: Type: text/html, Size: 2677 bytes --]

Re: [Caml-list] Core has landed

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 186 bytes --]

On Sat, May 3, 2008 at 4:19 AM, Richard Jones <rich@annexia.org> wrote:

>
> It would be nice to have bin-prot working on all architectures ...
>

We're happy to receive bug reports....

[-- Attachment #2: Type: text/html, Size: 436 bytes --]

Re: [Caml-list] Core has landed

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

When doing "make", I get the error below. I think I have sexplib installed
correctly (through godi), and have tried on both Mac OS and Linux. Anyone
else get this error?

File "core_sexp.mli", line 2, characters 8-27:
Unbound module type Sexplib.Sexp_intf.S


2008/5/3 Yaron Minsky <yminsky@gmail.com>:

> On Sat, May 3, 2008 at 4:19 AM, Richard Jones <rich@annexia.org> wrote:
>
> >
> > It would be nice to have bin-prot working on all architectures ...
> >
>
> We're happy to receive bug reports....
>
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 1696 bytes --]

Re: [Caml-list] Core has landed

[Stefano Zacchiroli]

On Sat, May 03, 2008 at 12:27:19PM +0200, Daniel Bünzli wrote:
> Again, creating a developer-oriented module distribution/installation  
> mechanism would allow to easily get all these modules separately and  

Then go and create on.  That's the basic foundation of a do-ocracy.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Core has landed

[Ralph Douglass]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

The version of type-conv and sexplib in godi are behind.  You need to
uninstall them from godi, and install from the tarball on our website until
someone upgrades the godi packages.

2008/5/3 Ashish Agarwal <agarwal1975@gmail.com>:

> When doing "make", I get the error below. I think I have sexplib installed
> correctly (through godi), and have tried on both Mac OS and Linux. Anyone
> else get this error?
>
> File "core_sexp.mli", line 2, characters 8-27:
> Unbound module type Sexplib.Sexp_intf.S
>
>
> 2008/5/3 Yaron Minsky <yminsky@gmail.com>:
>
> > On Sat, May 3, 2008 at 4:19 AM, Richard Jones <rich@annexia.org> wrote:
> >
> > >
> > > It would be nice to have bin-prot working on all architectures ...
> > >
> >
> > We're happy to receive bug reports....
> >
> >
> > _______________________________________________
> > Caml-list mailing list. Subscription management:
> > http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> > Archives: http://caml.inria.fr
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>


-- 
Ralph

[-- Attachment #2: Type: text/html, Size: 2886 bytes --]

Re: Core has landed

[Sylvain Le Gall]

On 03-05-2008, Stefano Zacchiroli <zack@upsilon.cc> wrote:
> On Sat, May 03, 2008 at 12:27:19PM +0200, Daniel Bünzli wrote:
>> Again, creating a developer-oriented module distribution/installation  
>> mechanism would allow to easily get all these modules separately and  
>
> Then go and create on.  That's the basic foundation of a do-ocracy.
>

And you can do it on http://forge.ocamlcore.org  ;-)

Regards,
Sylvain Le Gall

Re: [Caml-list] Core has landed

[Daniel Bünzli]

Le 3 mai 08 à 17:42, Stefano Zacchiroli a écrit :

> On Sat, May 03, 2008 at 12:27:19PM +0200, Daniel Bünzli wrote:
>> Again, creating a developer-oriented module distribution/installation
>> mechanism would allow to easily get all these modules separately and
>
> Then go and create on.  That's the basic foundation of a do-ocracy.

Look, I have already stated on this list that I will not create such a  
system - I will certainly contribute again to the community but not  
under this form. And hey, there are some here with a lot more  
expertise in package management systems than I have...

Besides, if some are interested in improving the situation (for which  
I'm very grateful) why shouldn't I raise criticism on what they do ?  
Especially given that the batteries included path was already followed  
once and failed.

I think your initial question intimately shows the limitations of  
batteries included, what remains outside batteries included will still  
be "hard" (in the sense harder than it should be) to publish and use  
and batteries cannot include everything. Hence my comments about  
solving the problem at another level.

Best,

Daniel

Re: [Caml-list] Core has landed

[Dmitry Bely]

On Sat, May 3, 2008 at 1:39 AM, Yaron Minsky <yminsky@janestcapital.com> wrote:

> We are proud to announce the first public release of core, Jane
>  Street's own alternative to OCaml's standard library.  We use this
>  library as the base for our own development, and we hope people on the
>  outside will find some use for it as well.

As usual, Windows is not supported?

- Dmitry Bely

Re: [Caml-list] Core has landed

[Richard Jones]

On Sun, May 04, 2008 at 01:02:30AM +0400, Dmitry Bely wrote:
> On Sat, May 3, 2008 at 1:39 AM, Yaron Minsky <yminsky@janestcapital.com> wrote:
> > We are proud to announce the first public release of core, Jane
> >  Street's own alternative to OCaml's standard library.  We use this
> >  library as the base for our own development, and we hope people on the
> >  outside will find some use for it as well.
> 
> As usual, Windows is not supported?

Jane St. seem to be using CentOS, if some comments in the source are
correct.

There's also lots of #ifdef __linux__ in the code and it's obvious
there's a lot of Linux-specific work being done -- eg. bindings for
non-standard Linux system calls.

So there you go - it's open source, you can port it to Windows!

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Agreed.  We have a limited amount of energy to devote to a Windows port at
this time, but we would be happy for core to work well on Windows.  Most of
the library depends only on the version of OCaml, so it should be pretty
easy to port.  We've done a bit of ifdef work to make it a tad more portable
(it's known to compile so far on FC5, Centos/RHE 4 and 5, and OS X.)

y

On Sat, May 3, 2008 at 5:37 PM, Richard Jones <rich@annexia.org> wrote:

> On Sun, May 04, 2008 at 01:02:30AM +0400, Dmitry Bely wrote:
> > On Sat, May 3, 2008 at 1:39 AM, Yaron Minsky <yminsky@janestcapital.com>
> wrote:
> > > We are proud to announce the first public release of core, Jane
> > >  Street's own alternative to OCaml's standard library.  We use this
> > >  library as the base for our own development, and we hope people on
> the
> > >  outside will find some use for it as well.
> >
> > As usual, Windows is not supported?
>
> Jane St. seem to be using CentOS, if some comments in the source are
> correct.
>
> There's also lots of #ifdef __linux__ in the code and it's obvious
> there's a lot of Linux-specific work being done -- eg. bindings for
> non-standard Linux system calls.
>
> So there you go - it's open source, you can port it to Windows!
>
> Rich.
>
> --
> Richard Jones
> Red Hat
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2431 bytes --]

Re: [Caml-list] Core has landed

[Markus Mottl]

On Fri, May 2, 2008 at 7:40 PM, Shivkumar Chandrasekaran
<shiv@ece.ucsb.edu> wrote:
>  > along with three other libraries that you will need to use along with
>  > it: type-conv, sexplib and bin-prot.
>
>  Will these be released as GODI packages too?

I have just added bin-prot to GODI, the other two packages are already
available.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Markus Mottl]

2008/5/3 Berke Durak <berke.durak@gmail.com>:
> - Bigstring: I guess those are more for I/O.  The 16MB limit is not a
> problem on 64-bit arhictectures.

Right.  With Bigstrings it is not necessary to blit buffers
before/after performing I/O-system calls.  They also do not have to
(cannot) be moved around during heap compaction.  They also play
nicely with OCaml standard I/O-channels, using vector I/O to send data
buffered by the channel and in the bigstring in one system call.

> - POLL and NODELAY in Linux_ext: that's very welcome.

Note that we support epoll in the Linux extension library, i.e. this
allows you to scale your applications to thousands of descriptors.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Markus Mottl]

On Sat, May 3, 2008 at 4:19 AM, Richard Jones <rich@annexia.org> wrote:
>  It would be nice to have bin-prot working on all architectures ...

The biggest obstacle to this is that the current implementation
assumes little endian machines.  Both 64bit and 32bit are fully
supported on these platforms.  It would certainly be possible to
support other architectures, but good portability and performance are
not easily combined here.  If performance does not matter, it would be
easiest to just convert all data to the protocol already used by
little endian machines.  This would noticably slow down communication
with/among machines that are not little endians.  Since the vast
majority of computers are little endians, this won't matter for most
people.  OCaml takes a similar approach.

Note that communication _among_ machines with different endianness,
assuming that they all have the same byte layout, should work, too,
with the current binary protocol.  At least if you do not mix 32/64bit
machines there...

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Markus Mottl]

2008/5/3 Ashish Agarwal <agarwal1975@gmail.com>:
> When doing "make", I get the error below. I think I have sexplib installed
> correctly (through godi), and have tried on both Mac OS and Linux. Anyone
> else get this error?

Just update the sexplib package in Godi.  The new version should work fine now.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Stefano Zacchiroli]

On Sun, May 04, 2008 at 10:14:26PM -0400, Markus Mottl wrote:
> If performance does not matter, it would be easiest to just convert
> all data to the protocol already used by little endian machines.  This
> would noticably slow down communication with/among machines that are
> not little endians.  Since the vast majority of computers are little
> endians, this won't matter for most people.  OCaml takes a similar
> approach.

Sounds like a reasonable solution indeed. Way better than not having
bin-prot on some archs (this is particularly annoying in Debian, where
we support several big endian machines; the status quo would mean no
Core on them and in turn no application using Core on them. Currently we
patched Core on that architecture to remove the bin-prot dependency, but
is a rather hackish solution I would like to get rid off).

Do you plan to implement such a solution in forthcoming releases?

Thanks,
Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Core has landed

[Markus Mottl]

On Mon, May 5, 2008 at 2:42 AM, Stefano Zacchiroli <zack@upsilon.cc> wrote:
>  Sounds like a reasonable solution indeed. Way better than not having
>  bin-prot on some archs (this is particularly annoying in Debian, where
>  we support several big endian machines; the status quo would mean no
>  Core on them and in turn no application using Core on them. Currently we
>  patched Core on that architecture to remove the bin-prot dependency, but
>  is a rather hackish solution I would like to get rid off).
>
>  Do you plan to implement such a solution in forthcoming releases?

We currently do not have any immediate need and man power to fully
support big-endian machines (we also don't have access to any), but
we'll gladly accept patches.  This could be implemented using
platform-specific macros as is the case with 32/64bit.

This is what works / doesn't work as of now:

  *) Big-endian and little-endian machines cannot communicate with
each other for anything but very specific cases.  Don't use the binary
protocol in such heterogeneous environments.

  *) 32 and 64 bit, little-endian architectures can communicate
freely, assuming, of course, that integers do not overflow on 32bit.
This is tested at runtime to prevent hard to debug errors.

  *) 32bit big-endian machines can communicate with each other freely.
 64 bit big-endians can communicate with each other freely, too, but
not necessarily with 32bit big-endians: values of type int64, etc.,
may not necessarily be communicated correctly.  Note, too, that you
should exclusively use the automatically generated converters on
big-endian machines.  The hand-written (slower) ones for the basic
types are intended mostly for testing purposes only, and will not work
when mixed with a different endianness (they assume little-endians).

Thus, it certainly makes sense to package the binary protocol for
big-endians, too, as long as people are informed of what works.  Since
almost nobody uses big-endian machines, most users won't care.  But
I'd surely be happy to see a patch to fully support all
architectures...

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Thanks for the updates! I can compile now, but still get the following
errors in top-level:
# #use "topfind";;
- : unit = ()
... some output ...

# #require "core";;
... several libraries loaded ...
/Users/ashish/godi/lib/ocaml/site-lib/core/core.cma: loaded
Cannot load required shared library dllcore_stubs.
Reason: /Users/ashish/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so:
Symbol not found: _uerror
  Referenced from:
/Users/ashish/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so
  Expected in: flat namespace
.
#

The above is on Mac OS X. I get a slightly different error on a Linux
machine:
Cannot load required shared library dllcore_stubs.
Reason: /home1/aa544/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so:
/home1/aa544/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so: undefined
symbol: setsockopt_int.



On Sun, May 4, 2008 at 10:22 PM, Markus Mottl <markus.mottl@gmail.com>
wrote:

> 2008/5/3 Ashish Agarwal <agarwal1975@gmail.com>:
> > When doing "make", I get the error below. I think I have sexplib
> installed
> > correctly (through godi), and have tried on both Mac OS and Linux.
> Anyone
> > else get this error?
>
> Just update the sexplib package in Godi.  The new version should work fine
> now.
>
> Regards,
> Markus
>
> --
> Markus Mottl http://www.ocaml.info markus.mottl@gmail.com
>

[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

Re: [Caml-list] Core has landed

[Markus Mottl]

On Tue, May 6, 2008 at 4:38 PM, Ashish Agarwal <agarwal1975@gmail.com> wrote:
> Thanks for the updates! I can compile now, but still get the following
> errors in top-level:
[snip]
> The above is on Mac OS X. I get a slightly different error on a Linux
> machine:
> Cannot load required shared library dllcore_stubs.
> Reason: /home1/aa544/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so:
> /home1/aa544/godi/lib/ocaml/site-lib/stublibs/dllcore_stubs.so: undefined
> symbol: setsockopt_int.

We have already encountered this problem, too.  This should be fixed
in the next release.  In the meanwhile you may want to execute
"#thread" before "requiring" the core library.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Erick Tryzelaar]

On Fri, May 2, 2008 at 2:39 PM, Yaron Minsky <yminsky@janestcapital.com> wrote:
> We are proud to announce the first public release of core, Jane
>  Street's own alternative to OCaml's standard library.  We use this
>  library as the base for our own development, and we hope people on the
>  outside will find some use for it as well.

This looks really interesting! Do you plan on putting this on a public
repository, such as git, or do you plan on just doing version drops?
It would make it a little easier for our project to track if we could
pull it directly into our repository.

Do you also plan on accepting patches for new functionality?

And finally, think you could put ocamldoc documentation online?

-e

Re: [Caml-list] Core has landed

[Yaron Minsky]

On Tue, 2008-05-06 at 22:10 -0700, Erick Tryzelaar wrote:
> On Fri, May 2, 2008 at 2:39 PM, Yaron Minsky <yminsky@janestcapital.com> wrote:
> > We are proud to announce the first public release of core, Jane
> >  Street's own alternative to OCaml's standard library.  We use this
> >  library as the base for our own development, and we hope people on the
> >  outside will find some use for it as well.
> 
> This looks really interesting! Do you plan on putting this on a public
> repository, such as git, or do you plan on just doing version drops?
> It would make it a little easier for our project to track if we could
> pull it directly into our repository.

For the moment, just version drops.

> Do you also plan on accepting patches for new functionality?

Yes.  (depending on the functionality provided and the quality of
implementation of course.  And I don't think we'll accept any patches
without corresponding unit tests.)

> And finally, think you could put ocamldoc documentation online?

We haven't gotten it to build yet, sadly.  There are some issues with
preprocessing and with packed libraries that we haven't gotten around to
fixing.  Once we (or anyone else) fixes it, we'll post the docs.

> -e
> 
-- 
Yaron Minsky <yminsky@janestcapital.com>

Re: [Caml-list] Core has landed

[Stefano Zacchiroli]

On Tue, May 06, 2008 at 05:02:05PM -0400, Markus Mottl wrote:
> We have already encountered this problem, too.  This should be fixed
> in the next release.  In the meanwhile you may want to execute
> "#thread" before "requiring" the core library.

I encountered a problem which is probably already solved for the next
release then: doing #require "core" in the top-level tries to access the
(thread-specific) Condition module, that would fail unless "#thread" has
been enabled in the top-level.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Core has landed

[Richard Jones]

On Sat, May 03, 2008 at 06:56:44AM -0400, Yaron Minsky wrote:
> On Sat, May 3, 2008 at 4:19 AM, Richard Jones <rich@annexia.org> wrote:
> > It would be nice to have bin-prot working on all architectures ...
> We're happy to receive bug reports....

My mail's been all over the place recently so sorry for the late reply ...

... but yes we found a build problem on PPC & PPC64.  It looks very
simple to fix actually, but if you did want to take a look then here's
the build log:

http://koji.fedoraproject.org/koji/getfile?taskID=605617&name=build.log (ppc)
http://koji.fedoraproject.org/koji/getfile?taskID=605621&name=build.log (ppc64)
(from: http://koji.fedoraproject.org/koji/taskinfo?taskID=605616)

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Stefano Zacchiroli]

On Thu, May 15, 2008 at 01:27:19PM +0100, Richard Jones wrote:
> ... but yes we found a build problem on PPC & PPC64.  It looks very
> simple to fix actually, but if you did want to take a look then here's
> the build log:
> 
> http://koji.fedoraproject.org/koji/getfile?taskID=605617&name=build.log (ppc)

Curiously enough this didn't show up in Debian. Is it related to gcc in
use? Debian buildd is here:

  http://buildd.debian.org/fetch.cgi?pkg=bin-prot;ver=1.0.5-4;arch=powerpc;stamp=1210555306

and was using gcc 4.3.0.

Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Core has landed

[Markus Mottl]

On Thu, May 15, 2008 at 8:40 AM, Stefano Zacchiroli <zack@upsilon.cc> wrote:
> On Thu, May 15, 2008 at 01:27:19PM +0100, Richard Jones wrote:
>  > ... but yes we found a build problem on PPC & PPC64.  It looks very
>  > simple to fix actually, but if you did want to take a look then here's
>  > the build log:
>  >
>  > http://koji.fedoraproject.org/koji/getfile?taskID=605617&name=build.log (ppc)
>
>  Curiously enough this didn't show up in Debian. Is it related to gcc in
>  use? Debian buildd is here:
>
>   http://buildd.debian.org/fetch.cgi?pkg=bin-prot;ver=1.0.5-4;arch=powerpc;stamp=1210555306
>
>  and was using gcc 4.3.0.

It seems that this is related to the PPC architecture defaulting to
unsigned as opposed to signed chars.  I don't know what the best fix
for this is.  Maybe gcc 4.3.0 changes this default behavior.  I guess
that using the compilation flag "-no-unsigned-char" might fix this in
the general case.  Could anybody with access to a PPC platform please
verify, whether this flag solved this compilation problem for the
binary protocol library?  Thanks!

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Markus Mottl]

On Thu, May 15, 2008 at 11:59 AM, Markus Mottl <markus.mottl@gmail.com> wrote:
>  It seems that this is related to the PPC architecture defaulting to
>  unsigned as opposed to signed chars.  I don't know what the best fix
>  for this is.  Maybe gcc 4.3.0 changes this default behavior.  I guess
>  that using the compilation flag "-no-unsigned-char" might fix this in
>  the general case.  Could anybody with access to a PPC platform please
>  verify, whether this flag solved this compilation problem for the
>  binary protocol library?  Thanks!

Oops, typo.  The compilation flag in question is called "-fno-unsigned-char".

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] Core has landed

[Richard Jones]

On Thu, May 15, 2008 at 12:03:43PM -0400, Markus Mottl wrote:
> On Thu, May 15, 2008 at 11:59 AM, Markus Mottl <markus.mottl@gmail.com> wrote:
> >  It seems that this is related to the PPC architecture defaulting to
> >  unsigned as opposed to signed chars.  I don't know what the best fix
> >  for this is.  Maybe gcc 4.3.0 changes this default behavior.  I guess
> >  that using the compilation flag "-no-unsigned-char" might fix this in
> >  the general case.  Could anybody with access to a PPC platform please
> >  verify, whether this flag solved this compilation problem for the
> >  binary protocol library?  Thanks!
> 
> Oops, typo.  The compilation flag in question is called "-fno-unsigned-char".

Yes, unfortunately our dev ppc64 machine has been broken for a little
while, hence why I didn't fix this!  I'll try a build on the main
build servers tomorrow & see if this flag fixes it.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Richard Jones]

On Thu, May 15, 2008 at 08:06:08PM +0100, Richard Jones wrote:
> On Thu, May 15, 2008 at 12:03:43PM -0400, Markus Mottl wrote:
> > Oops, typo.  The compilation flag in question is called "-fno-unsigned-char".
> 
> Yes, unfortunately our dev ppc64 machine has been broken for a little
> while, hence why I didn't fix this!  I'll try a build on the main
> build servers tomorrow & see if this flag fixes it.

OK, it compiles with this flag but the unit tests fail.  For example
on ppc:

http://koji.fedoraproject.org/koji/getfile?taskID=612162&name=build.log

I still have no ppc/ppc64 machine to test on unfortunately.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Core has landed

[Markus Mottl]

On Fri, May 16, 2008 at 6:16 AM, Richard Jones <rich@annexia.org> wrote:
>  OK, it compiles with this flag but the unit tests fail.  For example
>  on ppc:
>
>  http://koji.fedoraproject.org/koji/getfile?taskID=612162&name=build.log

Oh, that's clear, of course: the PPC architecture is big endian...

The unit tests may fail, but it should still be possible to use the
binary protocol on this platform (and then only there!) if you stick
to using converters derived by the macro preprocessor.

If anybody feels like implementing the feature that big endian
architectures write out data in little endian format (only the C-stubs
need to be adapted), then I'll gladly add that patch...

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com