From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by yquem.inria.fr (Postfix) with ESMTP id 58746BBAF for ; Fri, 12 Mar 2010 12:56:02 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AoQBAJ+6mUvVhjEXkGdsb2JhbACbAAEBAQEJCQwHEwMfuj2EewQ X-IronPort-AV: E=Sophos;i="4.49,625,1262559600"; d="scan'208";a="46407998" Received: from ihsmtp01voda.lis.interhost.com (HELO ihsmtp01cons.lis.interhost.com) ([213.134.49.23]) by mail3-smtp-sop.national.inria.fr with ESMTP; 12 Mar 2010 12:56:01 +0100 Received: from [192.168.1.64] ([95.136.106.136]) by ihsmtp01cons.lis.interhost.com with Microsoft SMTPSVC(6.0.3790.3959); Fri, 12 Mar 2010 11:56:00 +0000 Message-ID: <4B9A2BCB.3040607@inescporto.pt> Date: Fri, 12 Mar 2010 11:55:55 +0000 From: Hugo Ferreira User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: caml-list@yquem.inria.fr Subject: Shared memory parallel application: kernel threads Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 12 Mar 2010 11:56:00.0715 (UTC) FILETIME=[FBED61B0:01CAC1DA] X-Spam: no; 0.00; parallelism:01 inherently:01 parallelism:01 read-only:01 ocaml:01 ocaml:01 ocamlnet:01 val:01 recursive:01 libref:01 threadunix:01 queued:98 threads:01 threads:01 unix:01 Hello, I need to implement (meta) heuristic algorithms that uses parallelism in order to (attempt to) solve a (hard) machine learning problem that is inherently exponential. The aim is to take maximum advantage of the multi-core processors I have access to. To that effect I have revisited many of the lively discussions in threads related to concurrency, parallelism and shared memory in this mailing list. I however still have many doubts, some of which are very basic. My initial objective is to make a very simple tests that launches k jobs. Each of these jobs must access a common data set that is read-only. Each of the k threads in turn generates its own data. The data generated by the k jobs are then placed in a queue for further processing. The process continues by launching (or reusing) k/2 jobs. Each job consumes two elements from the queue that where previously generated (the common data set must still be available). The process repeats itself until k=1. Note that the queued data is not small nor can I determine a fixed maximum size for it. I have opted to use "kernel-level threads" that allow use of the (multi-core) processors but still allow "easy" access to shared memory". I have done a cursory look at: - Ocaml.Threads - Ocaml.Unix (LinuxThreads) - coThreads - Ocamlnet2/3 (netshm, netcamlbox) (An eThreads library exists in the forge but I did not examine this) My first concern is to take advantage of the multi-cores so: 1. The thread library is not the answer Chapter 24 - "The threads library is implemented by time-sharing on a single processor. It will not take advantage of multi-processor machines." [1] 2. LinuxThreads seems to be what I need "The main strength of this approach is that it can take full advantage of multiprocessors." [2] Issue 1 In the manual [3] I see only references to function for the creation and use of processes. I see no calls that allow me to simply generate and assign a function (job) to a thread (such as val create : ('a -> 'b) -> 'a -> t in the Thread module). The unix library where LinuxThreads is now integrated shows the same API. Am I missing something or is their no way to launch "threaded functions" from the Unix module? Naturally I assume that threads and processes are not the same thing. Issue 2 If I cannot launch kernel-threads to allow for easy memory sharing, what other options do I have besides netshm? The data I must share is defined by a recursive variant and is not simple numerical data. I would appreciate any comments. TIA, Hugo F. [1] http://caml.inria.fr/pub/docs/manual-ocaml/manual038.html [2] http://pauillac.inria.fr/~xleroy/linuxthreads/ [3] http://caml.inria.fr/pub/docs/manual-ocaml/libref/ThreadUnix.html [4] http://caml.inria.fr/pub/docs/manual-ocaml/manual035.html