From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id EC3DF7FD92 for ; Sat, 11 Jun 2016 10:53:44 +0200 (CEST) IronPort-PHdr: 9a23:Y/AydBKChJAD8GtG2NmcpTZWNBhigK39O0sv0rFitYgVK/vxwZ3uMQTl6Ol3ixeRBMOAu6MC1Lud6/6ocFdDyKjCmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TWM5DIfUi/yKRBybrysXNWC3oLohqvqpdX6WEZhunmUWftKNhK4rAHc5IE9oLBJDeIP8CbPuWZCYO9MxGlldhq5lhf44dqsrtY4q3wD89pozcNLUL37cqIkVvQYSW1+ayFmrPHs4FOMdheG4HoHVmwI2jNSHwXA7xD8Ed255iLgqud42yicFcLzRLEwHz+l6vE4chLwjDY7MGtt8mbRosJ+h6Vduw+luxV5yJ+Sa4aQYqlQZKTYKJk4QmwJfs9LTStZSMvobYwJJ+QPNuFetJT5u1IIpgr4Dg6pUrC8ggRUj2P7iPVpm98qFhvLiUl5R4oD Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=jon@ffconsultancy.com; spf=None smtp.mailfrom=jon@ffconsultancy.com; spf=None smtp.helo=postmaster@avasout03.plus.net Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jon@ffconsultancy.com) identity=pra; client-ip=84.93.230.244; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jon@ffconsultancy.com"; x-sender="jon@ffconsultancy.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jon@ffconsultancy.com) identity=mailfrom; client-ip=84.93.230.244; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jon@ffconsultancy.com"; x-sender="jon@ffconsultancy.com"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@avasout03.plus.net) identity=helo; client-ip=84.93.230.244; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jon@ffconsultancy.com"; x-sender="postmaster@avasout03.plus.net"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0DcAgC20FtXfvTmXVRdgnCBJH2qCWOENo15FwEMhXMCgSc8EAEBAQEBAQEBEQEBCQ0JCSEvgjCCGgEBAQQIAhkFBUELDAEDAgkRBAEBDhMHAwICGQgbCQEDAQUIAgQTCwUCCYd7AxsKrRCGJIY3AwqDfgEBAQEBAQEDAQEBAQEBAQEBAQEciXGBA4JDgWc6EoITOBOCRwWGRQyRXDQBgVqEKYYqhDGMaYgBh20PJoImHIFMbQGKBwEBAQ X-IPAS-Result: A0DcAgC20FtXfvTmXVRdgnCBJH2qCWOENo15FwEMhXMCgSc8EAEBAQEBAQEBEQEBCQ0JCSEvgjCCGgEBAQQIAhkFBUELDAEDAgkRBAEBDhMHAwICGQgbCQEDAQUIAgQTCwUCCYd7AxsKrRCGJIY3AwqDfgEBAQEBAQEDAQEBAQEBAQEBAQEciXGBA4JDgWc6EoITOBOCRwWGRQyRXDQBgVqEKYYqhDGMaYgBh20PJoImHIFMbQGKBwEBAQ X-IronPort-AV: E=Sophos;i="5.26,455,1459807200"; d="scan'208,217";a="222021528" Received: from avasout03.plus.net ([84.93.230.244]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-SHA; 11 Jun 2016 10:53:43 +0200 Received: from XPS ([80.189.16.107]) by avasout03 with smtp id 5Ltf1t0042Jc0sY01Ltgrx; Sat, 11 Jun 2016 09:53:42 +0100 X-CM-Score: 0.00 X-CNFS-Analysis: v=2.1 cv=KLCndwxo c=1 sm=1 tr=0 a=IT7DEMVM+saEYhBIg8ZHbA==:117 a=IT7DEMVM+saEYhBIg8ZHbA==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=DAwyPP_o2Byb1YXLmDAA:9 a=pGLkceISAAAA:8 a=r2vSxAw-AAAA:8 a=AiHQdgjdAAAA:8 a=p0WdMEafAAAA:8 a=YG86fJyiAAAA:8 a=vlzoIh7UAAAA:8 a=NEAV23lmAAAA:8 a=ZOzjf2MOAAAA:8 a=CjxXgO3LAAAA:8 a=RxU19Olzw2t91CwLxtkA:9 a=D1NLBPH8IgknicmQ:21 a=5xe7Lvu9k2QAUy7N:21 a=QEXdDO2ut3YA:10 a=5kWise4xawAA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=N7xMmoTWSnIUNmbZ:21 a=sSAghyFkKY6eHH0k:21 a=ZHRnd-lkB491vG9X:21 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=6kGIvZw6iX1k4Y-7sg4_:22 a=X9RvXkcXfCZWqhcwqo90:22 a=M-fgipkQCoj_B38W52kE:22 a=KNX8SZJeBz7yKKbDRToh:22 a=2rTcZijYBW9OfK4jDYag:22 a=iH0rUA9h8FOUXzGSNSYd:22 a=Bn2pgwyD2vrAyMmN8A2t:22 a=1Mhi-5-LkjG4w5oc0yAU:22 a=_E8DXSXWC0jInl61zbkG:22 a=BKKCjISod1eDJeS0ORpz:22 a=zjWhRoSqWz9hl55Hdlzg:22 X-AUTH: jdh302@:2500 Reply-To: From: "Jon Harrop" To: "'Stanislav Artemkin'" Cc: "'caml users'" References: <00c801d1c357$95598ca0$c00ca5e0$@ffconsultancy.com> In-Reply-To: Date: Sat, 11 Jun 2016 09:53:40 +0100 Organization: Flying Frog Consultancy Ltd. Message-ID: <00db01d1c3be$c140ff80$43c2fe80$@ffconsultancy.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_00DC_01D1C3C7.2306EE20" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQISLKA/Zy9ItN2+dVFL0ec6gqjD+gEQ754JAmnAEuyfR0tqkA== Content-Language: en-gb Subject: RE: [Caml-list] Measuring GC latencies for OCaml program This is a multipart message in MIME format. ------=_NextPart_000_00DC_01D1C3C7.2306EE20 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable =C3=98 It seems it was completely broken C++ solution =20 For this particular problem (which is essentially building a stock exchange= ) C++ is only ok for the initial core. As soon as you start adding features= that interact with each other a decent solution becomes intractable and yo= u end up with 40+ developers writing millions of lines of code that deep co= py data structures to avoid memory management problems, share heavily conte= nded locks, uses thousands of OS threads (and you have to start tweaking th= e default stack size to fit them all into RAM) and so on. Everybody loses t= rack of the big picture and after 15 years of this you=E2=80=99ve got an un= maintainable code base with runaway costs. =20 =C3=98 Do I correctly understand that OCaml is suitable for latencies ~10m= s and worse? =20 I=E2=80=99m seeing 95% of messages handles in under 13 microseconds. =20 =C3=98 Also, there is still an issue with multithreading. Did you use any = existing solution? =20 My OCaml solution is single threaded. Latency is great but throughput could= be a lot better. In particular, serialization and deserialization to each = client is embarrassingly parallel but done in series by OCaml. =20 Cheers, Jon. =20 From: Stanislav Artemkin [mailto:artemkin@gmail.com]=20 Sent: 10 June 2016 22:34 To: jon@ffconsultancy.com Cc: Gabriel Scherer; caml users; Damien Doligez Subject: Re: [Caml-list] Measuring GC latencies for OCaml program =20 Very interesting! It seems it was completely broken C++ solution. Wish I co= uld use OCaml for current project, but we still have to use C++ to get micr= osecond latencies. =20 Do I correctly understand that OCaml is suitable for latencies ~10ms and wo= rse? =20 Also, there is still an issue with multithreading. Did you use any existing= solution? =20 Thank you =20 On Sat, Jun 11, 2016 at 12:35 AM, Jon Harrop wrote: Very interesting, thank you! We just implemented a substantial client and server system for the finance = sector with the "low" latency server written in OCaml. I have done this bef= ore in other languages and seen it done in many more languages. The OCaml i= s by far the consistently-fastest solution I have ever seen. Orders of magn= itude faster than the last C++ solution I saw. In particular, compared to J= ava and .NET where we see substantial latencies from the GC at around 100ms= , with OCaml there is no visible peak at high latency due to the GC at all.= And this project was implemented to a very short deadline with no time for= optimisation at all. On a related note, we used Jane St.'s Core and Async libraries as well as C= ohttp and found them all to be *phenomenally* efficient and robust. In case anyone is interested, the only pain point I had was the development= environment. I actually prototyped all my hard code in simplified F# in Vi= sual Studio on Windows and then ported to OCaml. Emacs and Merlin crash and= hang a lot for me: maybe 50 times per day. Hence my other post. :-) In terms of the language, OCaml was very well suited to this task. Lots of = purely functional data structures forming in-memory databases that can be q= ueried in different ways and have many different versions of them stored in= different places at different times. Perhaps the main language feature I m= issed from F# was (surprisingly!) reflection. My F# client code uses reflec= tion to serialize and deserialize messages. With no reflection I couldn't d= o that in OCaml so I used reflection in F# to autogenerate the OCaml code. Cheers, Jon. -----Original Message----- From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On Beh= alf Of Gabriel Scherer Sent: 30 May 2016 20:48 To: caml users Cc: Damien Doligez Subject: [Caml-list] Measuring GC latencies for OCaml program Dear caml-list, You may be interested in the following blog post, in which I give instructi= ons to measure the worst-case latencies incurred by the GC: Measuring GC latencies in Haskell, OCaml, Racket http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-= ocaml-racket/ In summary, the commands to measure latencies look something like: % build the program with the instrumented runtime ocamlbuild -tag "runtime_variant(i)" myprog.native % run with instrumentation enabled OCAML_INSTR_FILE=3D"ocaml.log" ./main.native % visualize results from the raw log $(OCAML_SOURCES)/tools/ocaml-instr-graph ocaml.log $(OCAML_SOURCES)/tools/ocaml-instr-report ocaml.log While the OCaml GC has had a good incremental mode with low latencies for m= ost workloads for a long time, the ability to instrument it to actually mea= sure latencies is still in its infancy: it is a side-result of Damien Dolig= ez's stint at Jane Street last year, and 4.03.0 is the first release in which this work is available. A practical consequence of this youth is that the "user experience" of actu= ally performing these measurements is currently very bad. The GC measuremen= ts are activated in an instrumented runtime variant (OCaml supports having = several variants of the runtime available, and deciding which one to use fo= r a specific program at link-time), which is the right design choice, but r= ight now this variant is not built by default by the compiler distribution = -- building it is a configure-time option disabled by default. This means t= hat, as a user interested in doing the measurements, you have to compile an= alternative OCaml compiler. Furthermore, processing the raw instrumented log requires tool that are als= o in their infancy, and are currently included in the compiler distribution= sources but not installed -- so you have to have a source checkout availab= le to use them. In contrast, GHC's instrumentation is enabled by just passi= ng the option "+RTS -s" to the Haskell program of interest; this is superbl= y convenient and something we should aim at. I discussed with Damien whether we should enable building the instrumented = runtime by default (for example pass the --with-instrumented-runtime option= to the opam switches people are using, and encourage distributions to use = it in their packages as well). Of course there is a cost/benefit trade-off:= currently virtually nobody is using this instrumentation, but enabling it = by default would increase the compilation time of the compiler distribution= for everyone. (On my machine it only adds 5 seconds to total build time.) I personally think that we should aim for a rock-solid experience for profi= ling and instrumenting OCaml program enabled by default=C2=B9. It is worth = making it slightly longer for anyone to install the compiler if we can make= it vastly easier to measure GC pauses in our program when the need arises = (even if it's not very often). But that is a discussion that should be had = before making any choice. Regarding the log analysis tools, before finding about Damien's included-bu= t-not installed tools (a shell and an awk script, in the finest Unix tradit= ion) I built a quick&dirty OCaml script to do some measurements, which can = be found in the benchmark repository below. It would not be much more work = to grow this in a reusable library to extract the current log format into a= structured data structure -- the format is undocumented but the provided s= cripts in tools/ have enough information to infer the structure. Such a scr= ipt/library would, of course, remain tightly coupled to the OCaml version, = but I think it could be useful to have it packaged for users to play with. https://gitlab.com/gasche/gc-latency-experiment/blob/master/parse_ocaml_l= og.ml =C2=B9: We cannot expect users to effectively write performant code if they= don't have the tool support for it. The fact that lazyness in Haskell make= s it harder for users to reason about efficiency or memory usage has made t= he avaibility of excellent performance tooling *necessary*, where it is mer= ely nice-to-have in OCaml. Rather ironically, Haskell tooling is now much b= etter than OCaml's in this area, to the point that it can be easier to writ= e efficient code in Haskell. Three side-notes on profiling tools: 1. `perf record --call-graph=3Ddwarf` works fine for ocamlopt binaries (no need for a frame-pointers switch), and this is documented: https://ocaml.org/learn/tutorials/performance_and_profiling.html#Usingp= erfonLinux 2. Thomas Leonard has done excellent work on domain-specific profiling tools for Mirage, and this is the kind of tool support that I think should be available to anyone out of the box. http://roscidus.com/blog/blog/2014/08/15/optimising-the-unikernel/ http://roscidus.com/blog/blog/2014/10/27/visualising-an-asynchronous-m= onad/ 3. There is currently more debate than anyone could wish for around a pull request of Mark Shinwell for runtime support for dynamic call graph construction and its use for memory profiling. https://github.com/ocaml/ocaml/pull/585 4. Providing a good user experience for performance or space profiling is a fundamentally harder problem than for GC pauses. It may require specially-compiled versions of the libraries used by your program, and thus a general policy/agreement across the ecosystem. Swapping a different runtime at link-time is very easy. -- Caml-list mailing list. Subscription management and archives: https://sympa.inria.fr/sympa/arc/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs=3D -- Caml-list mailing list. Subscription management and archives: https://sympa.inria.fr/sympa/arc/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs =20 ------=_NextPart_000_00DC_01D1C3C7.2306EE20 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

=C3=98  = It seems it= was completely broken C++ solution

 

For this particular problem (which is essentially building a stock exc= hange) C++ is only ok for the initial core. As soon as you start adding fea= tures that interact with each other a decent solution becomes intractable a= nd you end up with 40+ developers writing millions of lines of code that de= ep copy data structures to avoid memory management problems, share heavily = contended locks, uses thousands of OS threads (and you have to start tweaki= ng the default stack size to fit them all into RAM) and so on. Everybody lo= ses track of the big picture and after 15 years of this you=E2=80=99ve got = an unmaintainable code base with runaway costs.

 

=C3=98  Do I correctl= y understand that OCaml is suitable for latencies ~10ms and worse?

 

I=E2=80=99m seeing 95% of messages handles in under 1= 3 microseconds.

 = ;

=C3=98  Also, there is still an issue with multithrea= ding. Did you use any existing solution?

 

My O= Caml solution is single threaded. Latency is great but throughput could be = a lot better. In particular, serialization and deserialization to each clie= nt is embarrassingly parallel but done in series by OCaml.

 

Cheers,

Jon.=

 <= /p>

From: Stanislav Artemkin [m= ailto:artemkin@gmail.com]
Sent: 10 June 2016 22:34
To:= jon@ffconsultancy.com
Cc: Gabriel Scherer; caml users; Damien Do= ligez
Subject: Re: [Caml-list] Measuring GC latencies for OCaml p= rogram

 

=

Very interesting! It = seems it was completely broken C++ solution. Wish I could use OCaml for cur= rent project, but we still have to use C++ to get microsecond latencies.

 

Do I correctly understand that OCaml is suitable for = latencies ~10ms and worse?

 

Also, there is stil= l an issue with multithreading. Did you use any existing solution?

 

Thank you

 

On Sat, Jun = 11, 2016 at 12:35 AM, Jon Harrop <jon@ffconsultancy.com> wrote:


Very interesting, thank you!

We just implement= ed a substantial client and server system for the finance sector with the &= quot;low" latency server written in OCaml. I have done this before in = other languages and seen it done in many more languages. The OCaml is by fa= r the consistently-fastest solution I have ever seen. Orders of magnitude f= aster than the last C++ solution I saw. In particular, compared to Java and= .NET where we see substantial latencies from the GC at around 100ms, with = OCaml there is no visible peak at high latency due to the GC at all. And th= is project was implemented to a very short deadline with no time for optimi= sation at all.

On a related note, we used Jane St.'s Core and Async = libraries as well as Cohttp and found them all to be *phenomenally* efficie= nt and robust.

In case anyone is interested, the only pain point I h= ad was the development environment. I actually prototyped all my hard code = in simplified F# in Visual Studio on Windows and then ported to OCaml. Emac= s and Merlin crash and hang a lot for me: maybe 50 times per day. Hence my = other post. :-)

In terms of the language, OCaml was very well suited= to this task. Lots of purely functional data structures forming in-memory = databases that can be queried in different ways and have many different ver= sions of them stored in different places at different times. Perhaps the ma= in language feature I missed from F# was (surprisingly!) reflection. My F# = client code uses reflection to serialize and deserialize messages. With no = reflection I couldn't do that in OCaml so I used reflection in F# to autoge= nerate the OCaml code.

Cheers,
Jon.


-----Original Message-----
From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On B= ehalf Of Gabriel Scherer
Sent: 30 May 2016 20:48
To: caml users
Cc= : Damien Doligez
Subject: [Caml-list] Measuring GC latencies for OCaml p= rogram

Dear caml-list,

You may be interested in the following= blog post, in which I give instructions to measure the worst-case latencie= s incurred by the GC:

  Measuring GC latencies in Haskell, OCam= l, Racket
  http://prl.cc= s.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/

In summary, the commands to measure latencies look something like= :

    % build the program with the instrumented runtime    ocamlbuild -tag "runtime_variant(i)" myprog.native=

    % run with instrumentation enabled
    O= CAML_INSTR_FILE=3D"ocaml.log" ./main.native

    = % visualize results from the raw log
    $(OCAML_SOURCES)/tool= s/ocaml-instr-graph ocaml.log
    $(OCAML_SOURCES)/tools/ocaml= -instr-report ocaml.log

While the OCaml GC has had a good incrementa= l mode with low latencies for most workloads for a long time, the ability t= o instrument it to actually measure latencies is still in its infancy: it i= s a side-result of Damien Doligez's stint at Jane Street last year, and
= 4.03.0 is the first release in which this work is available.

A pract= ical consequence of this youth is that the "user experience" of a= ctually performing these measurements is currently very bad. The GC measure= ments are activated in an instrumented runtime variant (OCaml supports havi= ng several variants of the runtime available, and deciding which one to use= for a specific program at link-time), which is the right design choice, bu= t right now this variant is not built by default by the compiler distributi= on -- building it is a configure-time option disabled by default. This mean= s that, as a user interested in doing the measurements, you have to compile= an alternative OCaml compiler.
Furthermore, processing the raw instrume= nted log requires tool that are also in their infancy, and are currently in= cluded in the compiler distribution sources but not installed -- so you hav= e to have a source checkout available to use them. In contrast, GHC's instr= umentation is enabled by just passing the option "+RTS -s" to the= Haskell program of interest; this is superbly convenient and something we = should aim at.

I discussed with Damien whether we should enable buil= ding the instrumented runtime by default (for example pass the --with-instr= umented-runtime option to the opam switches people are using, and encourage= distributions to use it in their packages as well). Of course there is a c= ost/benefit trade-off: currently virtually nobody is using this instrumenta= tion, but enabling it by default would increase the compilation time of the= compiler distribution for everyone. (On my machine it only adds 5 seconds = to total build time.)

I personally think that we should aim for a ro= ck-solid experience for profiling and instrumenting OCaml program enabled b= y default=C2=B9. It is worth making it slightly longer for anyone to instal= l the compiler if we can make it vastly easier to measure GC pauses in our = program when the need arises (even if it's not very often). But that is a d= iscussion that should be had before making any choice.

Regarding the= log analysis tools, before finding about Damien's included-but-not install= ed tools (a shell and an awk script, in the finest Unix tradition) I built = a quick&dirty OCaml script to do some measurements, which can be found = in the benchmark repository below. It would not be much more work to grow t= his in a reusable library to extract the current log format into a structur= ed data structure -- the format is undocumented but the provided scripts in= tools/ have enough information to infer the structure. Such a script/libra= ry would, of course, remain tightly coupled to the OCaml version, but I thi= nk it could be useful to have it packaged for users to play with.

&n= bsp; https://gitlab.com/gasche/gc-latency= -experiment/blob/master/parse_ocaml_log.ml

=C2=B9: We cannot exp= ect users to effectively write performant code if they don't have the tool = support for it. The fact that lazyness in Haskell makes it harder for users= to reason about efficiency or memory usage has made the avaibility of exce= llent performance tooling *necessary*, where it is merely nice-to-have in O= Caml. Rather ironically, Haskell tooling is now much better than OCaml's in= this area, to the point that it can be easier to write efficient code in H= askell.

Three side-notes on profiling tools:

1. `perf record = --call-graph=3Ddwarf` works fine for ocamlopt binaries
  (no need f= or a frame-pointers switch), and this is documented:
    https://ocaml.org/learn/tutorials/performan= ce_and_profiling.html#UsingperfonLinux

2. Thomas Leonard has don= e excellent work on domain-specific profiling
   tools for Mir= age, and this is the kind of tool support that I think
   shou= ld be available to anyone out of the box.
     http://roscidus.com/blog/blog/2014/08/15/optimising-the-unikern= el/
     http://roscid= us.com/blog/blog/2014/10/27/visualising-an-asynchronous-monad/

3= . There is currently more debate than anyone could wish for around
 = ;  a pull request of Mark Shinwell for runtime support for dynamic cal= l
   graph construction and its use for memory profiling.
&= nbsp;    https://github.com/ocaml/ocaml/pull/585

4. Providi= ng a good user experience for performance or space profiling
  &nbs= p;is a fundamentally harder problem than for GC pauses. It may
  &n= bsp;require specially-compiled versions of the libraries used by your
&n= bsp;  program, and thus a general policy/agreement across the
 = ;  ecosystem. Swapping a different runtime at link-time is very easy.<= br>
--
Caml-list mailing list.  Subscription management and arch= ives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: <= a href=3D"http://groups.yahoo.com/group/ocaml_beginners" target=3D"_blank">= http://groups.yahoo.com/group/ocaml_beginners

Bug reports: http://caml.inria.fr/bin/caml-bugs=3D<= /o:p>



--
Caml-list mailing lis= t.  Subscription management and archives:
https://sympa.inria.fr/symp= a/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocam= l_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

=

 

= ------=_NextPart_000_00DC_01D1C3C7.2306EE20--