Re: [Caml-list] Smart ways to implement worker threads

[Goswin von Brederlow <goswin-v-b@web.de>]

Romain Beauxis <toots@rastageeks.org> writes:

> Le jeudi 15 juillet 2010 12:46:53, Goswin von Brederlow a écrit :
>> I don't see where that helps at all. I don't want to offload the IO into
>> threads and schedule them and Duppy seems to only handle IO tasks.
>
> I don't understand what you mean by IO tasks. Tasks in duppy are scheduled 
> according to some events which, since it is select-based, are either an event 
> on a socket or a timeout.

But I don't have a socket. The main thread runs and at some point it has
a Buffer.t that it needs to have checksummed. So that leaves timeout.

> Once scheduled, the action that the task does is anything you programmed. Once 
> finished, the tasks can return an array of new tasks which are then put in the 
> queue.
>
> In your case, you probably only need the timeout event, which would mean that 
> as soon as you have a new tasks to perform, you submit it to the scheduler 
> with timeout 0 and it will be processed by one of the threads as soon as 
> possible..

So the code would be something like this?

let with_checksum buf sum _ =
  reply_request ();
  []

let do_checksum buf _ =
  let sum = ...
  in
    { priority = 0; events = [`Delay 0.]; handler = with_checksum buf sum; }

let main_task events =
  let rec loop tasks = function
      [] -> tasks
    | (`Read fd)::events ->
         let buf = parse_request fd in
         let task1 = { priority = 1; events = [`Delay 0.]; handler = do_checksum buf; } in
         let task2 = { priority = 0; events = [`Read fd]; handler = main_task; }
         in
           loop (task1::task2::tasks) events
  in
    loop events

let main () =
  let scheduler = Duppy.create ()
  in
    for i = 1 to num_cores do
      Thread.create (Duppy.queue ~priorities=(fun x -> x = 1) scheduler "worker") ();
    done;
    Duppy.Task.add scheduler
     { priority = 0; events = [`Read server_socket]; handler = main_task; }
    Duppy.queue ~priorities=(fun x -> x = 0) scheduler "main") ();


The main task will only process priority 0 events and bounce between
main_task and with_checksum while the worker threads process priority 1
events and do_checksum.

Correct?

>> Except if I pick Solution 1 and then it still doesn't help anything
>> since I can already run select in every thread. The IO should not be
>> scheduled by priorities and isn't the bottleneck anyway. Seems this
>> would just add overhead.
>
> The idea of duppy was to have only one select running among the multiple queue 
> threads.
>
>
> Romain

MfG
        Goswin