[ANN] coThreads 0.10

[ANN] coThreads 0.10

[Zheng Li]

Hi,

I'd like to announce the first public release of coThreads, a 
concurrent programming library for OCaml. 

It originated as STMlib [1], and was accepted as a Google Summer 
of Code 2007 project [2] mentored by Yoriyuki Yamagata from the 
Free Software Initiative of Japan [3].   The official website of 
coThreads is http://cothreads.sourceforge.net where you can find 
the source code and some preliminary documents. It's a wiki-based 
website, so feel free to register and contribute.    A byproduct 
of the project, the Vprint [4] module, was already announced 
independently a few weeks ago on this list.   Enjoy.   [1] 
http://www.pps.jussieu.fr/~li/software/index.html#stmlib [2] 
http://code.google.com/soc/2007/ [3] http://fsij.org [4] 
http://www.pps.jussieu.fr/~li/software/index.html#vprint   From 
the README: 
============================================================================== 
= Description =  coThreads is a concurrent programming library for 
OCaml. It enhances the Threads library of the standard OCaml 
distribution in two dimensions:  * coThreads implements the same 
API of the standard Threads library on different execution engines 
(process, netwoker(todo)), so that a single copy of source code 
can be compiled and deployed to different environments without 
modification  * coThreads is also a super set of the standard 
Threads library, with extra components (STM etc.), functions 
(spawn etc.) and features (object-level compatibility etc.)   = 
Features =  The design of coThreads brings several advantages:  == 
Powerfulness ==  * The process engine can give you real speedup on 
multi-core and multi-processor machines, the networker engine 
(todo) will give you both speedup and scalability.   * Combining 
the original components from the Threads library and the newly 
added ones, coThreads is a full-fledged toolbox covering two main 
concurrent programming paradigms, namely shared-memory and message 
passing, with both high-level and low-level constructors.   * All 
constructors (e.g. thread, mutex, condition, event, channel, tvar, 
stm etc.) are first class values that can be communicated and 
shared between independent threads.   == Compatibility ==  * Full 
compatibility with the original Threads library (systhreads and 
vmthreads), so that you can now deploy your legacy code to new 
environment for free — without modifying source code, without 
learning something new.   * Both source-level and object-level 
compatibility among different execution engines, so that you can 
keep a single copy of source code as well as a single copy of 
objects files.   == Convenience ==  * The full library is 
implemented in user-space, no modification to the OCaml compiler 
and runtime, so that you can use it as a set of plain modules.   * 
Switching engines is as easy as changing the include paths of 
compilation, and it's quite easy to automate the building process 
over a set of engines (e.g. with lines of pattern rules in your 
Makefile)   -- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

Really sorry for the format in the previous posts. Something must be wrong with
my Emacs setting. Anyway, here is the reformatted one, hope it works.

I'd like to announce the first public release of coThreads, a concurrent
programming library for OCaml.

It originated as STMlib [1], and was accepted as a Google Summer of Code 2007
project [2] mentored by Yoriyuki Yamagata from the Free Software Initiative of
Japan [3].

The official website of coThreads is http://cothreads.sourceforge.net where you
can find the source code and some preliminary documents. It's a wiki-based
website, so feel free to register and contribute.

A byproduct of the project, the Vprint [4] module, was already announced
independently a few weeks ago on this list.   

Enjoy.   

[1] http://www.pps.jussieu.fr/~li/software/index.html#stmlib
[2] http://code.google.com/soc/2007/ 
[3] http://fsij.org 
[4] http://www.pps.jussieu.fr/~li/software/index.html#vprint


>From the README:
==============================================================================

= Description =  

coThreads is a concurrent programming library for OCaml. It enhances the
Threads library of the standard OCaml distribution in two dimensions:

* coThreads implements the same API of the standard Threads library on
  different execution engines (process, netwoker(todo)), so that a single copy
  of source code can be compiled and deployed to different environments without
  modification
* coThreads is also a super set of the standard Threads library, with extra
  components (STM etc.), functions (spawn etc.) and features (object-level
  compatibility etc.)

= Features =

The design of coThreads brings several advantages: 

== Powerfulness ==  

* The process engine can give you real speedup on multi-core and
  multi-processor machines, the networker engine (todo) will give you both
  speedup and scalability. 
* Combining the original components from the Threads library and the newly
  added ones, coThreads is a full-fledged toolbox covering two main concurrent
  programming paradigms, namely shared-memory and message passing, with both
  high-level and low-level constructors. 
* All constructors (e.g. thread, mutex, condition, event, channel, tvar, stm
  etc.) are first class values that can be communicated and shared between
  independent threads.

== Compatibility ==  

* Full compatibility with the original Threads library (systhreads and
  vmthreads), so that you can now deploy your legacy code to new environment
  for free — without modifying source code, without learning something new.
* Both source-level and object-level compatibility among different execution
  engines, so that you can keep a single copy of source code as well as a
  single copy of objects files.

== Convenience ==  

* The full library is implemented in user-space, no modification to the OCaml
compiler and runtime, so that you can use it as a set of plain modules.
* Switching engines is as easy as changing the include paths of compilation,
and it's quite easy to automate the building process over a set of engines
(e.g. with lines of pattern rules in your Makefile)

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:

> * The process engine can give you real speedup on multi-core and
>   multi-processor machines, the networker engine (todo) will give you both
>   speedup and scalability. 

I'm curious how this can be possible**.. is this only with the message
passing model? 

** Since Ocaml can't multi-process and neither the compiler
nor library are modified ..

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [ANN] coThreads 0.10

[Zheng Li]

skaller <skaller@users.sourceforge.net> writes:
> On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:
>
>> * The process engine can give you real speedup on multi-core and
>>   multi-processor machines, the networker engine (todo) will give you both
>>   speedup and scalability. 
> I'm curious how this can be possible**.. is this only with the message
> passing model? 
> ** Since Ocaml can't multi-process and neither the compiler
> nor library are modified ..

Well, in order to have semantic consistency among different engines without
modifying standard OCaml itself, shard-memory style concurrency should only be
achieved through the STM module. It's documented in the pitfall pages: 

http://cothreads.sourceforge.net/doc/pitfall

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Mon, 2007-09-17 at 19:51 +0200, Zheng Li wrote:
> skaller <skaller@users.sourceforge.net> writes:
> > On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:
> >
> >> * The process engine can give you real speedup on multi-core and
> >>   multi-processor machines, the networker engine (todo) will give you both
> >>   speedup and scalability. 
> > I'm curious how this can be possible**.. is this only with the message
> > passing model? 
> > ** Since Ocaml can't multi-process and neither the compiler
> > nor library are modified ..
> 
> Well, in order to have semantic consistency among different engines without
> modifying standard OCaml itself, shard-memory style concurrency should only be
> achieved through the STM module. It's documented in the pitfall pages: 
> 
> http://cothreads.sourceforge.net/doc/pitfall

Yes but what I'm getting at is the claim of multi-processing,
which Ocaml 3.10 at least cannot do. If you run Ocaml on a multi-core
machine, even if you have 8 cores and 8 threads, one on each core,
only one will ever run at once, so there is no performance gain.
In fact, it will be slower than a single core.

If you are *enforcing* message passing and using separate processes
instead of threads, then 8 processes will run in parallel on 8 cores,
and you'll get roughly 8 times speedup.

So I'm not asking about how to ensure the code is consistent
over the various models, but rather how you can get ANY genuine
concurrency WITHOUT using message passing and processes
(in which case the networking model should be easy to implement)

Whether you explicitly send messages or use transactional memory
or whatever to wrap the message passing isn't the question:
I can see how that can work. However note, the message passing
has to be used for immutable values too, if the threads are
represented by processes in separate address spaces.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi, 

skaller <skaller@users.sourceforge.net> writes:
> Yes but what I'm getting at is the claim of multi-processing,
> which Ocaml 3.10 at least cannot do. If you run Ocaml on a multi-core
> machine, even if you have 8 cores and 8 threads, one on each core,
> only one will ever run at once, so there is no performance gain.
> In fact, it will be slower than a single core.
>
> If you are *enforcing* message passing and using separate processes
> instead of threads, then 8 processes will run in parallel on 8 cores,
> and you'll get roughly 8 times speedup.
Yes, you are right. The process engine is implemented with process. That's how
it can speedup and that's why it's called ``process engine''.  

> So I'm not asking about how to ensure the code is consistent
> over the various models, but rather how you can get ANY genuine
> concurrency WITHOUT using message passing and processes
> (in which case the networking model should be easy to implement)
I'm sorry if the document misleadingly made you think it get speedup without
using process and message passing. Threads and related tools is just the
abstract interface on which all engines agree, they implement it with various
facilities from VM threads, system threads to  process or independent programs.

> Whether you explicitly send messages or use transactional memory
> or whatever to wrap the message passing isn't the question:
> I can see how that can work. However note, the message passing
> has to be used for immutable values too, if the threads are
> represented by processes in separate address spaces.
For immutable value, it's not different from the traditional threads, at least
from the library users' point of view: if a variable is already inside the
scope of a threads (i.e. it's global and created before launching the thread),
there is no necessity to message passing it; if not, you should anyway do it
explicitly even with the traditional threads.

Regards
-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Tue, 2007-09-18 at 00:37 +0200, Zheng Li wrote:

> For immutable value, it's not different from the traditional threads, at least
> from the library users' point of view: if a variable is already inside the
> scope of a threads (i.e. it's global and created before launching the thread),
> there is no necessity to message passing it; if not, you should anyway do it
> explicitly even with the traditional threads.

Yes, however such a value can't be shared by parallel control paths
(in ocaml) because the collector won't allow it. Of course
you can *copy* it eg with fork() and run separate collectors.
Then the functional value is the same, but the object identity
(machine address) may differ.

BTW: an interesting application would be to adapt GLR parser
to actually use multi-core CPU to speed up parsing by spawning
alternate parses as parallel control streams.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [ANN] coThreads 0.10

[Zheng Li]

skaller <skaller@users.sourceforge.net> writes:
> On Tue, 2007-09-18 at 00:37 +0200, Zheng Li wrote:
>> For immutable value, it's not different from the traditional threads, at least
>> from the library users' point of view: if a variable is already inside the
>> scope of a threads (i.e. it's global and created before launching the thread),
>> there is no necessity to message passing it; if not, you should anyway do it
>> explicitly even with the traditional threads.
>
> Yes, however such a value can't be shared by parallel control paths
> (in ocaml) because the collector won't allow it. Of course
> you can *copy* it eg with fork() and run separate collectors.
Given that the copy-on-write strategy is widely adopted in the *nix family, I
hope it's not a big problem.
> Then the functional value is the same, but the object identity
> (machine address) may differ.
Yes. For immutable value, physical equivalence is not preserved across
*multiple* rounds of message passing. This is another slight semantic
difference I should document on the pitfall page. As mutable value, if shared
through STM, their inter-references are actually preserved.

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

Zheng Li wrote:

> The official website of coThreads is http://cothreads.sourceforge.net where you

Is anybody else having difficulty downloading this? I have tried 
a large number of different sourceforge mirrors and none of them
seem to have this file whic means I re-directed back and forth
across the net til one of these things redirects me to this:

    http://.dl.sourceforge.net/sourceforge/cothreads/cothreads_0.10.tar.bz2

which isn't even a legal host name.

I've also tried wget with no success.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I want to make sure (a user) can't get through ... an online
experience without hitting a Microsoft ad."
 - Microsoft CEO, Steve Ballmer on the Microsoft search engine.

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

Erik de Castro Lopo wrote:

> Is anybody else having difficulty downloading this? I have tried 

This works:

    svn co https://cothreads.svn.sourceforge.net/svnroot/cothreads/trunk cothreads

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I run Linux on pretty much everything except the microwave and
washing machine. Those are tempting targets but would probably
make Telsa extremely cross." -- Alan Cox

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Tue, 2007-09-18 at 10:53 +1000, Erik de Castro Lopo wrote:
> Zheng Li wrote:
> 
> > The official website of coThreads is http://cothreads.sourceforge.net where you
> 
> Is anybody else having difficulty downloading this?

Confirmed .. but then we live next door :)

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

Zheng Li wrote:

> I'd like to announce the first public release of coThreads, a concurrent
> programming library for OCaml.

What wasn't mentioned in this announcement was the existance of a
mailing list. I posted to the mailing list:

    http://sourceforge.net/mailarchive/forum.php?forum_name=cothreads-discuss

but I suspect that I may be the only person on the list :-).

Cheers,
Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
Microsoft VISTA : Virus Infection Spyware Trojans and Adware!

Can coThreads be used for message passing architecture?

[Jan Kybic]

>> I'd like to announce the first public release of coThreads, a concurrent
>> programming library for OCaml.

I have had only a quick look at the web page but I did not understand
whether and how can coThreads be used on a cluster of independent
computers (Beowulf-style) that are only connected by network (no
shared memory) for which I normally use MPI (message passing). 
Is coThreads usable in such a scenario? What is the backend - does it
run on the top of PVM, MPI, or similar?

Thanks for clarification.

Jan

-- 
-------------------------------------------------------------------------
Jan Kybic <kybic@fel.cvut.cz>                       tel. +420 2 2435 5721
http://cmp.felk.cvut.cz/~kybic                      ICQ 200569450

Re: [Caml-list] Can coThreads be used for message passing architecture?

[Erik de Castro Lopo]

Jan Kybic wrote:

> I have had only a quick look at the web page but I did not understand
> whether and how can coThreads be used on a cluster of independent
> computers (Beowulf-style) that are only connected by network (no
> shared memory) for which I normally use MPI (message passing). 

>From the TODO file, it seems that the networker engine is not yet 
complete which means the answer to your question is no (at least
for now).

I can however report that on a single machine with multiple cores,
message passing between CoThreads using the Event module is an absolute
pleasure to use.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I have found that Big Design Up Front environments attract people who
want to talk about software development, while iterative environments
attract people who want to develop software. Is it any surpise that in
companies where BDUF is predominant, nobody wants to stoop to coding?
Everyone wants to be an 'architect' or a 'Business Analyst' or a
'Product Manager'." -- Reg Braithwaite

Re: Can coThreads be used for message passing architecture?

[Zheng Li]

Hi,

Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> Jan Kybic wrote:
>> I have had only a quick look at the web page but I did not understand
>> whether and how can coThreads be used on a cluster of independent
>> computers (Beowulf-style) that are only connected by network (no
>> shared memory) for which I normally use MPI (message passing). 
>
>>From the TODO file, it seems that the networker engine is not yet 
> complete which means the answer to your question is no (at least
> for now).
Yes, he is right. What you're thinking about is the networker engine. It is not
available for now but planned for the future. You'll still be able to have
shared memory through STM, though the communication cost will be larger. 

The implementation is no more difficult than the process engine, on the
contrary it can reuse most of the techniques developed with process
engine. But I'm kind of busy in this period, and won't be able to start the
implementation right now.

--
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Can coThreads be used for message passing architecture?

[Jon Harrop]

On Wednesday 19 September 2007 12:13:46 Erik de Castro Lopo wrote:
> I can however report that on a single machine with multiple cores,
> message passing between CoThreads using the Event module is an absolute
> pleasure to use.

Carry on... :-)

Any examples?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] Can coThreads be used for message passing architecture?

[Erik de Castro Lopo]

Jon Harrop wrote:

> Carry on... :-)
> 
> Any examples?

All my examples so far are trivial, like the following which creates a 
Cothread and then polls for an event sent by the child.

-------------------------------------------------------------------
(*
**	Needs Cothreads installed :
**		http://cothreads.sourceforge.net/
**
**	Run with :
**		ocaml -I +process unix.cma cothreads.cma poll_event.ml
**
**	Native compile with :
**		ocamlopt -I +process unix.cmxa cothreads.cmxa poll_event.ml -o poll_event
*)

let child_thread chan =
	(* Print thread id (actually process id) and flush stdout. *)
	Printf.printf "child_thread : %d\n%!" (Cothread.id (Cothread.self ())) ;
	(* Hang about for a bit. *)
	Cothread.delay 2.5 ;
	(* Send an event. *)
	Event.sync (Event.send chan "Event from child.") ;
	(* Hang about a bit more. *)
	Cothread.delay 1.5 ;
	Cothread.exit ()

let () =
	(* Create an event channel for sending stuff between main and child. *)
	let chan = Event.new_channel () in
	(* Create the child thread and pass it the channel. *)
	let t1 = Cothread.create child_thread chan in
	(* Set the exit condition to false. *)
	let fini = ref false in
	while not !fini do
		(* Now we can poll on the incoming event. *)
		match (Event.poll (Event.receive chan)) with
		|	None -> Printf.printf "Nothing.\n%!" ; Cothread.delay 0.3
		|	Some s -> fini := true ; print_endline s
		done ;
	(* Wait for child thread to exit. *)
	Printf.printf "Waiting for child thread.\n%!" ;
	Cothread.join t1 ;
	print_endline "Done."

-------------------------------------------------------------------

I intend to blog about this over the weekend. I'll post a URL when
its done.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I believe C++ instills fear in programmers, fear that the
interaction of some details causes unpredictable results. Its
unmanageable complexity has spawned more fear-preventing tools
than any other language, but the solution _should_ have been
to create and use a language that does not overload the
whole goddamn human brain with irrelevant details."
-- Erik Naggum

Re: Can coThreads be used for message passing architecture?

[Zheng Li]

Hi,

Nice try!

Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> All my examples so far are trivial, like the following which creates a 
> Cothread and then polls for an event sent by the child.
>
> -------------------------------------------------------------------
> (*
> **	Needs Cothreads installed :
> **		http://cothreads.sourceforge.net/
> **
> **	Run with :
> **		ocaml -I +process unix.cma cothreads.cma poll_event.ml
> **
> **	Native compile with :
> **		ocamlopt -I +process unix.cmxa cothreads.cmxa poll_event.ml -o poll_event
> *)
One comment: you don't have to hard-wired "Cothread.xxx" into source, since all
the functions you're using are compatible between Thread and Cothread modules,
and moreover, coThreads does provide a compatible version of threads.cm(x)a and
Thread module being able to work with process.

So by "open Cothread" or "open Thread" or "module Thread=Cothread", you'll be
given more flexibility to change easily back and forth between coThreads and
standard Threads (image sometime later you need to deploy the same code to a
standard-OCaml-only environment ...)

> let child_thread chan =
> 	(* Print thread id (actually process id) and flush stdout. *)
> 	Printf.printf "child_thread : %d\n%!" (Cothread.id (Cothread.self ())) ;
> 	(* Hang about for a bit. *)
> 	Cothread.delay 2.5 ;
> 	(* Send an event. *)
> 	Event.sync (Event.send chan "Event from child.") ;
> 	(* Hang about a bit more. *)
> 	Cothread.delay 1.5 ;
> 	Cothread.exit ()
>
> let () =
> 	(* Create an event channel for sending stuff between main and child. *)
> 	let chan = Event.new_channel () in
> 	(* Create the child thread and pass it the channel. *)
> 	let t1 = Cothread.create child_thread chan in
> 	(* Set the exit condition to false. *)
> 	let fini = ref false in
> 	while not !fini do
> 		(* Now we can poll on the incoming event. *)
> 		match (Event.poll (Event.receive chan)) with
> 		|	None -> Printf.printf "Nothing.\n%!" ; Cothread.delay 0.3
> 		|	Some s -> fini := true ; print_endline s
> 		done ;
> 	(* Wait for child thread to exit. *)
> 	Printf.printf "Waiting for child thread.\n%!" ;
> 	Cothread.join t1 ;
> 	print_endline "Done."
>
> -------------------------------------------------------------------
>
> I intend to blog about this over the weekend. I'll post a URL when
> its done.

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: Can coThreads be used for message passing architecture?

[Zheng Li]

Hi Jon,

Jon Harrop <jon@ffconsultancy.com> writes:
> On Wednesday 19 September 2007 12:13:46 Erik de Castro Lopo wrote:
>> I can however report that on a single machine with multiple cores,
>> message passing between CoThreads using the Event module is an absolute
>> pleasure to use.
> Carry on... :-)
> Any examples?

There is a replanted version of your ray tracer in the /example directory:
ray.ml is the module containing common computation functions, ray_col.ml and
ray_nocol.ml are two different parallel engines. There's nothing new in this
example, it's traditional Thread and Event, but running on process.

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Vu Ngoc San]

Hi

this looks like something I'd be very happy to use. Have you tried 
compiling lablgtk2 with it ?

(so that I could get rid of the following:)
[
The files /usr/lib/ocaml/3.10.0/process/thread.cmi
and /usr/lib/ocaml/3.10.0/lablgtk2/gtkThread.cmi
make inconsistent assumptions over interface Thread
]

San

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

Vu Ngoc San <san.vu-ngoc@ujf-grenoble.fr> writes:
> this looks like something I'd be very happy to use. Have you tried compiling
> lablgtk2 with it ?
>
> (so that I could get rid of the following:)
> [
> The files /usr/lib/ocaml/3.10.0/process/thread.cmi
> and /usr/lib/ocaml/3.10.0/lablgtk2/gtkThread.cmi
> make inconsistent assumptions over interface Thread
> ]

I'm not sure what you're trying to do, but the reason of error seems obvious:
lablgtk2 seems to have been compiled with the Thread module of system threads
or VM thread (both come with OCaml's standard distribution), while you're
linking them with other modules compiled against the Thread module of process
engine.

The Thread module interfaces from the three engines differ with each
other! Note that this is not the problem from coThreads, it's a problem of
standard Threads library:

 - The Thread interfaces of system threads and VM threads coming with the
   standard distribution *already* differ with each other, so that the objects
   compiled with system Thread won't be able to mix with objects compiled with
   VM thread, vice versa. (The problem is the same as in your case)

 - We have no way to unify the three of them without modifying the files from
   original distribution, which is exactly what we manage to avoid.

On the other hand, the Cothread module (a super set of the Thread interface) is
provided to achieve object-level compatibility, so that the objects compiled
with different engines can mix. All other modules from Threads and coThreads,
e.g. Mutex, Event, don't have this problem.

You'd like to read the compatibility page for more explanation:
http://cothreads.sourceforge.net/doc/compatibility

Regards
-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Wed, 2007-09-19 at 22:10 +0200, Zheng Li wrote:

> The Thread module interfaces from the three engines differ with each
> other! Note that this is not the problem from coThreads, it's a problem of
> standard Threads library:

Why don't you just use a different module name?


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

skaller wrote:

> Why don't you just use a different module name?

I agree. Although it is nice the have the Cothreads API be so similar
to the native Threads module, I can't possibly imagine ever wanting
to go from Cothreads back to Threads.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"That being done, all you have to do next is call free() slightly
less often than malloc(). You may want to examine the Solaris
system libraries for a particularly ambitious implementation of
this technique."
-- Eric O'Dell (comp.lang.dylan)

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Thu, 2007-09-20 at 14:29 +1000, Erik de Castro Lopo wrote:
> skaller wrote:
> 
> > Why don't you just use a different module name?
> 
> I agree. Although it is nice the have the Cothreads API be so similar
> to the native Threads module, I can't possibly imagine ever wanting
> to go from Cothreads back to Threads.

Undesirable, but there is an issue which might lead to that,
namely the licence.

CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
whereas my project, for example, is FFAU/BSD, and only tolerates
non FFAU/BSD licences in Ocaml standard distro (not 3PLs).

GPL isn't so bad for executables, but it isn't very good for
libraries.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

skaller wrote:

> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,

I posted a request to the CoThreads mailing list for a licence change
to LGPL with linking exception.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
Linux : Think of it as 'free' as in 'free speech' not 'free beer'.

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> skaller wrote:
>> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
> I posted a request to the CoThreads mailing list for a licence change
> to LGPL with linking exception.

Yes. I have no problem to change the licence to another one if desired by the
community. I can change it since next release. Is LGPL+linking exception the
right one?

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Thu, 2007-09-20 at 10:37 +0200, Zheng Li wrote:
> Hi,
> 
> Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> > skaller wrote:
> >> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
> > I posted a request to the CoThreads mailing list for a licence change
> > to LGPL with linking exception.
> 
> Yes. I have no problem to change the licence to another one if desired by the
> community. I can change it since next release. Is LGPL+linking exception the
> right one?

It depends how restrictive you *intend* to be. BSD, MIT, Boost, 
Creative Commons, etc are for genuinely free software. 
Free as in FFAU: Free for Any Use. Do what you like with it!

GPL is if you want to prevent anyone using it who hides any 
source code (most commercial use). 

LGPL+linking X is when you  don't mind if people hide 
their own source, as long as they  don't hide yours.

My opinion is: for a MAJOR public executable like gcc,
GPL is OK. For major libraries like glibc, LGPL (with
linking permitted) would be ok.

For code which has hardly any developers or users, if you
wish to encourage the widest possible use, the most 
permissive licence (eg BSD) is probably the best:
why exclude potential users who may help with bug reports
or patches? IMHO *especially* commercial users who put funds
into it and make a commitment, are even more likely to help
with maintenance and development.

Unless or until you expect to actually make money out of the
code itself, restrictive licences are counter-productive ..
in my opinion of course.

EG: GHC Haskell is currently trying to dump GMP because
of the licence. GMP is LGPL, GHC is BSD: dependence on GMP
is therefore a serious barrier to people using Haskell in
industry.. not what the authors want. (Plain LGPL doesn't
have a static linking exception as Ocaml's libraries do).

Another *famous* example is 'readline()', the code that
does line editing for bash. Why is Ocaml's top level
so crappy, with no editing? Because of readline()'s GPL
licence it can't be used, because GPL is a virus, it
propagates to anything it touches.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Matthew Hannigan]

On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
> Another *famous* example is 'readline()', the code that
> does line editing for bash. Why is Ocaml's top level
> so crappy, with no editing? Because of readline()'s GPL

Surely that's not a good reason anymore; there are non-GPL
workalikes to readline().

> licence it can't be used, because GPL is a virus, it
> propagates to anything it touches.

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Thu, 2007-09-20 at 20:44 +1000, Matthew Hannigan wrote:
> On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
> > Another *famous* example is 'readline()', the code that
> > does line editing for bash. Why is Ocaml's top level
> > so crappy, with no editing? Because of readline()'s GPL
> 
> Surely that's not a good reason anymore; there are non-GPL
> workalikes to readline().

Well one might ask Inria why they don't use it?

[BTW: Python does, yet some maintains FFAU licence]

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Christophe Raffalli]

[-- Attachment #1.1: Type: text/plain, Size: 1213 bytes --]

skaller a écrit :
> On Thu, 2007-09-20 at 20:44 +1000, Matthew Hannigan wrote:
>   
>> On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
>>     
>>> Another *famous* example is 'readline()', the code that
>>> does line editing for bash. Why is Ocaml's top level
>>> so crappy, with no editing? Because of readline()'s GPL
>>>       
>> Surely that's not a good reason anymore; there are non-GPL
>> workalikes to readline().
>>     
>
> Well one might ask Inria why they don't use it?
>
> [BTW: Python does, yet some maintains FFAU licence]
>
>   
There is even one (http://pauillac.inria.fr/~ddr/ledit)  written in
OCaml ... and easy to add to any OCaml
program ...

-- 
Christophe Raffalli
Universite de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tel: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution 
can check this signature. The public key is
stored on www.keyserver.net
---------------------------------------------


[-- Attachment #1.2: Christophe.Raffalli.vcf --]
[-- Type: text/x-vcard, Size: 310 bytes --]

begin:vcard
fn:Christophe Raffalli
n:Raffalli;Christophe
org:LAMA (UMR 5127)
email;internet:christophe.raffalli@univ-savoie.fr
title;quoted-printable:Ma=C3=AEtre de conf=C3=A9rences
tel;work:+33 4 79 75 81 03
note:http://www.lama.univ-savoie.fr/~raffalli
x-mozilla-html:TRUE
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Thu, 2007-09-20 at 17:07 +0200, Christophe Raffalli wrote:
> skaller a écrit :
>      
> >> Surely that's not a good reason anymore; there are non-GPL
> >> workalikes to readline().

> There is even one (http://pauillac.inria.fr/~ddr/ledit)  written in
> OCaml ... and easy to add to any OCaml
> program ...

I'm aware of ledit, but it is hardly a readline workalike.
It provides very minimal editing capability -- better than
nothing certainly!

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Florian Weimer]

>> Surely that's not a good reason anymore; there are non-GPL
>> workalikes to readline().
>
> Well one might ask Inria why they don't use it?

If you don't need autocompletion, you could use rlwrap anyway.

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Vincent Aravantinos]

Le 20 sept. 07 à 18:26, Florian Weimer a écrit :

>>> Surely that's not a good reason anymore; there are non-GPL
>>> workalikes to readline().
>>
>> Well one might ask Inria why they don't use it?
>
> If you don't need autocompletion, you could use rlwrap anyway.

rlwrap has basic completion: ie. for already-seen words, and you can  
provide it with a file with keywords.
(much) better than nothing :)

--
Vincent Aravantinos
PhD Student - LIG - CAPP Team

Re: readline license (was: [ANN] coThreads 0.10)

[Xavier Leroy]

>>>Another *famous* example is 'readline()', the code that
>>>does line editing for bash. Why is Ocaml's top level
>>>so crappy, with no editing? Because of readline()'s GPL
>>
>>Surely that's not a good reason anymore; there are non-GPL
>>workalikes to readline().
>
>
> Well one might ask Inria why they don't use it?

(Just an aside: please say "the Caml development team" rather than
"Inria", these are two different things.)

So, yes, readline being GPL is a serious obstacle to using it within
the OCaml toplevel.  More details in this old thread:

http://groups.google.com/group/fa.caml/browse_thread/thread/e7266ab6e6df91e7/e5ce04a21c80cac0

This said, rlwrap works pretty well and I don't see many advantages to
linking directly with readline.

- Xavier Leroy

Re: [Caml-list] Re: readline license (was: [ANN] coThreads 0.10)

[Matthew William Cox]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Fri, Sep 21, 2007 at 06:33:34PM +0200, Xavier Leroy wrote:
> This said, rlwrap works pretty well and I don't see many advantages to
> linking directly with readline.

First off, agree with you that rlwrap works well. That being said,
having an rl-alike linked with Ocaml's toplevel would add some
capabilities:
	* tab completion of identifiers currently in the environment
	* ability to reject malformed/non-parsing lines from the history

They're not killer features by any stretch, but they are a nice touch.

Cheers,
Matthew Cox

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [Caml-list] Re: readline license (was: [ANN] coThreads 0.10)

[skaller]

On Fri, 2007-09-21 at 18:33 +0200, Xavier Leroy wrote:

> >>Surely that's not a good reason anymore; there are non-GPL
> >>workalikes to readline().
> >
> >
> > Well one might ask Inria why they don't use it?
> 
> (Just an aside: please say "the Caml development team" rather than
> "Inria", these are two different things.)

Sorry! Hmm .. is there a shorter form you'd find appropriate?

> So, yes, readline being GPL is a serious obstacle to using it within
> the OCaml toplevel.  More details in this old thread:
> 
> http://groups.google.com/group/fa.caml/browse_thread/thread/e7266ab6e6df91e7/e5ce04a21c80cac0
> 
> This said, rlwrap works pretty well and I don't see many advantages to
> linking directly with readline.

I had no idea this program existed. I tried it an am impressed.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: readline license

[Markus E L]

skaller wrote:

> On Fri, 2007-09-21 at 18:33 +0200, Xavier Leroy wrote:
>
>> >>Surely that's not a good reason anymore; there are non-GPL
>> >>workalikes to readline().
>> >
>> >
>> > Well one might ask Inria why they don't use it?
>> 
>> (Just an aside: please say "the Caml development team" rather than
>> "Inria", these are two different things.)

How about Caml breeders or CDT :-))). Yes, I know, I'm ridiculous.

Regards -- Markus

Re: [Caml-list] Re: readline license

[Daniel Bünzli]

Le 21 sept. 07 à 23:51, Markus E L a écrit :

> How about Caml breeders

I guess not, it used to make Xavier feel like a nuclear reactor [1].

Daniel

[1] http://caml.inria.fr/pub/ml-archives/caml-list/2005/01/ 
bc2a91d581783fcd0c0f28d6c6116229.fr.html

Re: [Caml-list] Re: readline license (was: [ANN] coThreads 0.10)

[Matthew Hannigan]

On Sat, Sep 22, 2007 at 04:05:50AM +1000, skaller wrote:
> 
> I had no idea this program [rlwrap] existed. I tried it an am impressed.
> 

rlwrap and ledit are mentioned here:
    http://www.cocan.org/tips_for_using_the_ocaml_toplevel

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Florian Weimer]

> It depends how restrictive you *intend* to be. BSD, MIT, Boost, 
> Creative Commons, etc are for genuinely free software. 
> Free as in FFAU: Free for Any Use. Do what you like with it!

The majority of Creative Commons license is definitely not "FFAU".

> EG: GHC Haskell is currently trying to dump GMP because
> of the licence. GMP is LGPL, GHC is BSD: dependence on GMP
> is therefore a serious barrier to people using Haskell in
> industry.. not what the authors want. (Plain LGPL doesn't
> have a static linking exception as Ocaml's libraries do).

It does, and it's quite straightforward to fulfill.  I don't think
this is a significant issue, and the "get rid of GMP" project doesn't
appear to be terribly successful.

Re: [Caml-list] Re: [ANN] coThreads 0.10

[skaller]

On Thu, 2007-09-20 at 13:39 +0200, Florian Weimer wrote:

> (Plain LGPL doesn't
> > have a static linking exception as Ocaml's libraries do).
> 
> It does, and it's quite straightforward to fulfill. 

A derivative work has conditions imposed on it which are contrary 
to normal (closed source/proprietary) commercial practice.

As I read the licence, the provider must make it possible for the
user to link an alternate or modified library. Static linked executables
provided by commercial vendors very rarely provide any way to do that.

Whether or not the conditions are straightforward to fulfil
depends on the producers goals. If you were making an embedded
application it would almost certainly not be, and it would
be a legal impediment. Do you know any vendors of console
games, for example, that tell you how to modify a part of the
game? The software build processes used are almost certainly
trade secrets.

Even in my own FFAU product, whilst the whole of the sources
is available for inspection and modification, I certainly
don't document all of it -- if you want to modify a library
you'll have to go read all the source and find out yourself.

It isn't clear that satisfies the "make it possible for the
client to replace the library" requirement.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Ken Rose]

skaller wrote:

> Whether or not the conditions are straightforward to fulfil
> depends on the producers goals. If you were making an embedded
> application it would almost certainly not be, and it would
> be a legal impediment. Do you know any vendors of console
> games, for example, that tell you how to modify a part of the
> game? The software build processes used are almost certainly
> trade secrets.
> 
> Even in my own FFAU product, whilst the whole of the sources
> is available for inspection and modification, I certainly
> don't document all of it -- if you want to modify a library
> you'll have to go read all the source and find out yourself.
> 
> It isn't clear that satisfies the "make it possible for the
> client to replace the library" requirement.


There was a discussion ten years ago or so, perhaps on the cross-gcc
list, that Stallman joined.  He said something to the effect that if you
provided object code in a linkable format (even partially linked) so
that a user could link in the library in question, you'd satisfied the
license.  You didn't have to provide instructions, guidance, or help.
If the binary then had to be embedded in a masked ROM in the middle of
your giant ASIC, that was the hacker's problem, not yours.

 - ken

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> skaller wrote:
>> Why don't you just use a different module name?
> I agree. Although it is nice the have the Cothreads API be so similar
> to the native Threads module, I can't possibly imagine ever wanting
> to go from Cothreads back to Threads.

It's easy to neglect that coThreads actually comes with a compatible
threads.cm(x)a for the process engine. So if you're working with legacy code
but wish to try the process engine, you can still using Thread module and
threads library, and don't have to change anything except the include path. 

On the other hand, if you're using shared-memory concurrency in your legacy
code, you'll have to port it with STM --- the only compatible way to have
shared-memory across different engines.

Regards
-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

First note that the naming convention follows the standard threads library in
OCaml: Threads and coThreads (both with 's') refer to the library (*.cm(x)a);
Thread and Cothread (neither with 's') are modules (*.cm[ox]) inside.

skaller <skaller@users.sourceforge.net> writes:
>> The Thread module interfaces from the three engines differ with each
>> other! Note that this is not the problem from coThreads, it's a problem of
>> standard Threads library:
>
> Why don't you just use a different module name?
The different module name *is* Cothread, a compatible super set of Thread
module. Use this module instead if you want to have object-level
compatibility. In short, all the three engines have the two libraries with
isomorphic structures (quite simple) :

<code>
  threads.cm(x)a = threads.cm[ox], mutex.cm[ox], condition.cm[ox], event.cm[ox]
cothreads.cm(x)a = threads.cm[ox], mutex.cm[ox], condition.cm[ox], event.cm[ox], cothread.cm[ox], stm.cm[ox]
</code>

See, we also provide the compatible "threads.cm(x)a" for the process
engine. Think about the following scenario:

 - If you're working on legacy code, you don't care STM and don't care
   object-level compatibility, the only thing you're interested is to running
   your code with process to speedup, then the only thing to change in your
   Makefile is the include path e.g. "-I +threads" -> "-I +process", you can
   still using Thread module and "threads.cma", they are present in process
   engine.

 - You have several projects, some of them using traditional Threads, some
   using coThreads, you don't want to bother to remember that. So simply
   changing all linking library from threads.cma to cothreads.cma is fine,
   because coThreads library contains every modules in Threads library.

 - You have some legacy code written in standard Threads, and you'd like to
   have it run with processes. Though you don't care about the newly introduced
   modules of coThreads, you do want to have a single copy of object files for
   each engines. In such case, you need to add one line to any source code
   which makes use of the Thread module:

   module Thread = Cothread

   because Thread module (the only module in Threads and coThreads) don't have
   object-level compatibility, and change any occurrence of “threads.cm(x)a” to
   “cothreads.cm(x)a” in your Makefile. 

For more explanation, see http://cothreads.sourceforge.net/doc/compatibility

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [Caml-list] [ANN] coThreads 0.10

[Erik de Castro Lopo]

Zheng Li wrote:

> 
> Hi,
> 
> I'd like to announce the first public release of coThreads, a 
> concurrent programming library for OCaml.

I've compiled this with Ocaml 3.09.1 on x86 Linux and x86-64 Linux.

Looking at the example program ray_nocol.ml, it works correctly
on x86 Linux but fails on x86-64 [0]:

    # > ./ray_nocol.proc.opt 9 1024 1 ray_nocol.proc.opt.pgm
    Fatal error: exception Unix.Unix_error(12, "lseek", "")

I've also tried the non-opt version and the nath version. All fail
in the same way.

The errno value of 12 means "Out of memory". I did a little debugging
but couldn't find anything obvious.

Erik

[0] This is the standard 3.09.1 compiler shipped with Ubuntu Feisty.
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
The Earth is around 70% water. Fish rule the seas.
Humans are over 90% water. It's only a matter of time.

Re: [Caml-list] [ANN] coThreads 0.10

[skaller]

On Tue, 2007-09-18 at 12:10 +1000, Erik de Castro Lopo wrote:
> Zheng Li wrote:
> 
> > 
> > Hi,
> > 
> > I'd like to announce the first public release of coThreads, a 
> > concurrent programming library for OCaml.
> 
> I've compiled this with Ocaml 3.09.1 on x86 Linux and x86-64 Linux.
> 
> Looking at the example program ray_nocol.ml, it works correctly
> on x86 Linux but fails on x86-64 [0]:
> 
>     # > ./ray_nocol.proc.opt 9 1024 1 ray_nocol.proc.opt.pgm
>     Fatal error: exception Unix.Unix_error(12, "lseek", "")
> 
> I've also tried the non-opt version and the nath version. All fail
> in the same way.
> 
> The errno value of 12 means "Out of memory". I did a little debugging
> but couldn't find anything obvious.

But it isn't an errno value that is reported, it is the index
of the variant that is 12, and that is "Invalid Argument".

I think I found it:

let fresh_number =
  let usable_size = Sys.word_size -2 in
  let bits_of_id = 16 in (* Should be sufficient in most OS *)
  let bits_of_num = usable_size - bits_of_id in
  let counter = ref 0 in
  fun () ->
    let self_id = id (self ()) in
    let id_part = bit_chop_to_n bits_of_id self_id in
    let num_part = 
      counter := bit_chop_to_n bits_of_num (!counter + 1);
      !counter in
    (id_part lsl bits_of_num) + num_part

produces:

1695112678495748100

which seems a little big for lseek on a zero size file .. 

My man page says:

       lseek - reposition read/write file offset

RETURN VALUE
       Upon successful completion, lseek() returns the resulting offset
location  as
       measured  in  bytes  from  the  beginning of the file.
Otherwise, a value of
       (off_t)-1 is returned and errno is set to indicate the error.

It is not clear that a seek to an invalid position in the file
is going to succeed. It's also not clear to me that the seek
argument isn't 32 bits (depends on complex ugly GNU macro
hackery what type off_t is .. my Caml got built with
the LARGE_FILE macro thing so it should be 64 bits).




-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] [ANN] coThreads 0.10

[Erik de Castro Lopo]

skaller wrote:

> But it isn't an errno value that is reported, it is the index
> of the variant that is 12, and that is "Invalid Argument".

Ok.

> It is not clear that a seek to an invalid position in the file
> is going to succeed. It's also not clear to me that the seek
> argument isn't 32 bits (depends on complex ugly GNU macro
> hackery what type off_t is .. my Caml got built with
> the LARGE_FILE macro thing so it should be 64 bits).

Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see 
this problem on x86-64 and Sys.word_size is 64.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I run Linux on pretty much everything except the microwave and
washing machine. Those are tempting targets but would probably
make Telsa extremely cross." -- Alan Cox

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,


Erik de Castro Lopo <mle+ocaml@mega-nerd.com> writes:
> skaller wrote:
>
>> But it isn't an errno value that is reported, it is the index
>> of the variant that is 12, and that is "Invalid Argument".
>
> Ok.
>
>> It is not clear that a seek to an invalid position in the file
>> is going to succeed. It's also not clear to me that the seek
>> argument isn't 32 bits (depends on complex ugly GNU macro
>> hackery what type off_t is .. my Caml got built with
>> the LARGE_FILE macro thing so it should be 64 bits).
>
> Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see 
> this problem on x86-64 and Sys.word_size is 64.

Thank you for the bug report. 

Unfortunately I don't have a x64 machine to test, and know quite little about
64bit machine (that's why I made the mistake). 

But the bug reason seems obvious: on a x64 machine, one have to use
Unix.LargeFile.lseek to seek the address in the extra space. 

On the other hand, I think 2^14 locks (the x86 case) are fairly enough in
most situation. So we don't have to bother with Int64 and LargeFile. Could you
(or anyone else with a x64 machine) help to test the simple solution: change
"Sys.word_size - 2"  in the definition of "fresh_number" to 
"(min Sys.word_size 32) - 2" as follows

let fresh_number =
  let usable_size = (min Sys.word_size 32) -2 in
  let bits_of_id = 16 in (* Should be sufficient in most OS *)
  ........

and report whether it works? Thanks

-- 
Zheng Li
http://www.pps.jussieu.fr/~li

Re: [ANN] coThreads 0.10

[Zheng Li]

Hi,

Thanks skaller for testing the fix on his machine. Here is a few updates:

 - I chose the simple fix: just restrict the upper bound of fresh_number
   based on the word_size of 32bit machine. It's only used to generate fresh
   number, name and offset, so I suspect the range is fairly enough. On the
   other hand, it won't be difficult to make use int64 and LargeFile for 64bit
   machine if necessary, but I'd better to get some x64 machine around for
   testing first.

 - A README for /example is added (also attached to the end of this post) 

 - Some minor updates to a few examples

 - I have no idea what's wrong with the sourceforge file download system. It
   did work when I uploaded the file a few days ago. Anyway, I re-uploaded the
   package (with the above modifications), hope it works.

Best regards.

-- 
Zheng Li
http://www.pps.jussieu.fr/~li


README for examples
===================

== BUILD ==

Just ''make all'' after you've got coThreads installed on your system.
Type ''make clean'' to remove all imtermediate and final building results.

== LIST ==

* coth (use: Thread (or Cothread), Mutex)
  Simple test of mutex. A set of threads tries to grab a single mutex and
  release it for random times 

* evt (use: Thread (or Cothread), Event)
  Simple test of event. Most examples are directly from the OCaml OReilly
  book. The execution won't exist, this is intentional.

* lock (use: Thread (or Cothread), Mutex)
  Simple test of mutex. A set of threads try to grab two mutex. Each thread
  first must grab the first mutex before the second mutex, then release the
  second mutex and the first one.

* mcast (use: Thread (or Cothread), Stm)
  STM example from [1], contributed by Yoriyuki Yamagata

* merge (use: Thread (or Cothread), Stm)
  STM example from [1], contributed by Yoriyuki Yamagata

* mvar (use: Thread (or Cothread), Stm)
  STM example from [1], contributed by Yoriyuki Yamagata

* phil (use: Thread (or Cothread), Stm)
  Classical philosophers dinning problem written in STM. Launch it with 
  [./phil n], where n is the number of philosophers and chopsticks.

* ray_col, ray_nocol (use: Cothread, Event)
  Replanted versions of Jon Harrop's ray tracer [3]. ray.ml is the module
  containing common computation functions, ray_xxx.ml are parallel engines.
  In ray_nocol.ml, the workers don't send the results back to master, instead
  they write them directly to the output file; in ray_col.ml, the workers send
  results back to the master, and the master write them to the output file. 
  Launch it with [./ray_xxx level size degree outputfile], where [level] and
  [size] are about the quality of output image, and [degree] is the parallel
  degree which should equal or greater than the cores or cpus of your machine
  if you'd like to get the most speedup. Or you may just lanch it with
  [./ray_xxx] which takes the default setting [./ray_xxx 9 512 2 ray_xxx.pgm]

* santa (use: Thread (or Cothread), Stm)
  The Santa Clause problem documented in [2]. The haskell version is attached
  as comment at the end of the file.

* sing (use: Thread (or Cothread), Stm)
  Simple test of Stm. Two threads constantly update a single tvar.

* test (use: Thread (or Cothread), Stm)
  Simple test of Stm to calculate the sum of [0..n-1] with n threads. The i_th
  thread is responsible for adding i to the sum. Its action is not allowed to
  take place until the current sum exceed sum (i/10).

* The Makefile itself is an example. It shows that how you can build your
  applications against a set of execution engines with just a few lines of
  pattern rules.

[1] http://research.microsoft.com/users/simonpj/papers/stm/index.htm#composble
[2] https://research.microsoft.com/users/simonpj/papers/stm/index.htm#beautiful
[3] http://www.ffconsultancy.com/languages/ray_tracer/index.html


Zheng Li <li@pps.jussieu.fr> writes:
> Thank you for the bug report. 
>
> Unfortunately I don't have a x64 machine to test, and know quite little about
> 64bit machine (that's why I made the mistake). 
>
> But the bug reason seems obvious: on a x64 machine, one have to use
> Unix.LargeFile.lseek to seek the address in the extra space. 
>
> On the other hand, I think 2^14 locks (the x86 case) are fairly enough in
> most situation. So we don't have to bother with Int64 and LargeFile. Could you
> (or anyone else with a x64 machine) help to test the simple solution: change
> "Sys.word_size - 2"  in the definition of "fresh_number" to 
> "(min Sys.word_size 32) - 2" as follows
>
> let fresh_number =
>   let usable_size = (min Sys.word_size 32) -2 in
>   let bits_of_id = 16 in (* Should be sufficient in most OS *)
>   ........
>
> and report whether it works? Thanks

Re: [Caml-list] Re: [ANN] coThreads 0.10

[Erik de Castro Lopo]

Zheng Li wrote:

> Thanks skaller for testing the fix on his machine.

Fix confirmed for me as well. Thanks.

>  - A README for /example is added (also attached to the end of this post) 

Ahh, good, I was going to ask for that :-).

Cheers,
Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"Every time microshaft's stock price drops again, I rejoice. I
want to see that bunch of criminals brought to their knees.
Preferably at the chopping block."
-- rixt in http://linuxtoday.com/stories/20659_flat.html

Re: [Caml-list] [ANN] coThreads 0.10

[skaller]

On Tue, 2007-09-18 at 16:23 +1000, Erik de Castro Lopo wrote:
> skaller wrote:

> > It is not clear that a seek to an invalid position in the file
> > is going to succeed. It's also not clear to me that the seek
> > argument isn't 32 bits (depends on complex ugly GNU macro
> > hackery what type off_t is .. my Caml got built with
> > the LARGE_FILE macro thing so it should be 64 bits).
> 
> Shouldn't off_t always be 64 bits on a 64 bit CPU? 

I don't know. I traced through the code and did some debugging,
and the problem is that system wide mutex is emulated by using
locking on an lseek on a (deleted) lock file. The seek address
is a magic number which is rather large, and this appears to
cause the EINVAL errno on the lseek call.

let lock_fd = 
  print_endline "CREATING MUTEX";
  let lock_name = fresh_name "_mutex" in
  remove_exists lock_name; 
  let fd = openfile lock_name [O_WRONLY; O_CREAT] file_perm in
  remove_exists lock_name;
  fd

type t = int (* The offset *)

let create = fresh_number

let rec lock lk = 
  print_endline ("LOCK " ^ string_of_int lk);
  if lk <> lseek lock_fd lk SEEK_SET then assert false;
  lockf lock_fd F_LOCK 1

So basically a mutex is modelled by an offset,
which is created by fresh_number routine, and using
Unix discretionary locks with lockf function, which
is based on the current position in the file.

Well, I don't see how the lseek can fail unless there
is a filesize constraint somewhere detecting that

1695112678495748100

is an invalid offset for lseek. 

A possible patch is to take this in process.coordinator

(* fresh_number fresh_name ensure that there won't exist number/name
   confliction between running processes.
*)
let fresh_number =
  let usable_size = Sys.word_size -2 in
  let bits_of_id = 16 in (* Should be sufficient in most OS *)
  let bits_of_num = usable_size - bits_of_id in
  let counter = ref 0 in
  fun () ->
    let self_id = id (self ()) in
    let id_part = bit_chop_to_n bits_of_id self_id in
    let num_part = 
      counter := bit_chop_to_n bits_of_num (!counter + 1);
      !counter in
    (id_part lsl bits_of_num) + num_part


and use the line 2:

  let usable_size = 30 in

instead (though I didn't try it, some other places may need
to replace Sys.word_size the same way for it to work
properly).

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] [ANN] coThreads 0.10

[Markus E L]

Erik de Castro Lopo wrote:

> skaller wrote:
>
>> But it isn't an errno value that is reported, it is the index
>> of the variant that is 12, and that is "Invalid Argument".
>
> Ok.
>
>> It is not clear that a seek to an invalid position in the file
>> is going to succeed. It's also not clear to me that the seek
>> argument isn't 32 bits (depends on complex ugly GNU macro
>> hackery what type off_t is .. my Caml got built with
>> the LARGE_FILE macro thing so it should be 64 bits).
>
> Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see 
> this problem on x86-64 and Sys.word_size is 64.

This might be, or might be not. "The standard has nothing to say about
this, but usually it's at least long int whatever that is at the
corresponing platform. In contrast, I find in unix.ml

   external lseek : file_descr -> int -> seek_command -> int = "unix_lseek"

but also in otherlibs/unix/lseek.c

  ret = lseek(Int_val(fd), Long_val(ofs),
                       seek_command_table[Int_val(cmd)]);

(see the Long_Val).

Am I only imagining potential problems with variable width here?

Regards -- Markus