From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by sympa.inria.fr (Postfix) with ESMTPS id 8BD0D7F1C3 for ; Wed, 28 Nov 2012 02:43:39 +0100 (CET) Received-SPF: None (mail4-smtp-sop.national.inria.fr: no sender authenticity information available from domain of mlin@mlin.net) identity=pra; client-ip=209.85.215.54; receiver=mail4-smtp-sop.national.inria.fr; envelope-from="mlin@mlin.net"; x-sender="mlin@mlin.net"; x-conformance=sidf_compatible Received-SPF: None (mail4-smtp-sop.national.inria.fr: no sender authenticity information available from domain of mlin@mlin.net) identity=mailfrom; client-ip=209.85.215.54; receiver=mail4-smtp-sop.national.inria.fr; envelope-from="mlin@mlin.net"; x-sender="mlin@mlin.net"; x-conformance=sidf_compatible Received-SPF: None (mail4-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-la0-f54.google.com) identity=helo; client-ip=209.85.215.54; receiver=mail4-smtp-sop.national.inria.fr; envelope-from="mlin@mlin.net"; x-sender="postmaster@mail-la0-f54.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjoCAHhrtVDRVdc2jWdsb2JhbABFh0QjrzCJAwgWDgEBAQEJCQsJEgYjgh4BAQQBeQULCwsDOCISAQUBHAYTCAGHfgYMoHuOI5AijDqEQQOIXo0jgRyNSBYpglCBYoF/ X-IronPort-AV: E=Sophos;i="4.84,174,1355094000"; d="scan'208";a="164023411" Received: from mail-la0-f54.google.com ([209.85.215.54]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 28 Nov 2012 02:43:38 +0100 Received: by mail-la0-f54.google.com with SMTP id j13so17139073lah.27 for ; Tue, 27 Nov 2012 17:43:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:x-gm-message-state; bh=7O/xV5JOUz++GQjlt/SruLiNkq/wZuappfyR7ldw7lQ=; b=fb+YSRaVjKLKQ6GNMtwn7p1pnhUdO2xKd8SHxtikZWljmOxuPF5rbdDJlRRE9I6PAX e8IBS4A6bUIXk6PQnqtHKj8otJFs/B7SPwHLMDnIrYhBRuJWBPZLr9+8S4tj7Ud2pEIM FXvACQfZPbCYqyswPqhaZq9Awydqa1FQYbT4bKbGI856j9aCV1dtN7Zxfulkr3UVtUsi HRf9QyvCQBL4h+0hWIyKooOKwXJIIQEucWtgML17szXSJCkHyB9x1oKXqXC54OEfHLmC 7I2Hx3mgf1Y6L7z4EV/1fEr2NBM3hqM5QYFBYUYBYdv81w8jKb7qqIucMQlPo0ECHtoS j+gg== Received: by 10.152.111.68 with SMTP id ig4mr16655118lab.50.1354067017398; Tue, 27 Nov 2012 17:43:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.154.165 with HTTP; Tue, 27 Nov 2012 17:43:17 -0800 (PST) X-Originating-IP: [76.191.184.242] In-Reply-To: <1354021383.3169.2@samsung> References: <1354021383.3169.2@samsung> From: Mike Lin Date: Tue, 27 Nov 2012 17:43:17 -0800 Message-ID: To: Gerd Stolpmann Cc: ocamlnet-devel , "caml-list@inria.fr" Content-Type: multipart/alternative; boundary=f46d04083b5192f9ac04cf844b98 X-Gm-Message-State: ALoCoQkTThM/ZHER444AoJVlxqkANjA2cnxoTRP/hfNAIwgvHx6mCC+wjkErI5o8tmCpZL4SSn5R X-Validation-by: mlin@mlin.net Subject: [Caml-list] Re: Multithreaded https requests in ocamlnet netclient --f46d04083b5192f9ac04cf844b98 Content-Type: text/plain; charset=ISO-8859-1 Thanks, Gerd! FWIW I could not reproduce the crash by using ocaml-ssl's blocking operations directly. https://gist.github.com/4152047#file_ssl_threads.ml This works fine- so, perhaps something nasty arises from using nonblocking I/O on ssl sockets from multiple threads. I'm sure if there is any other critical difference with how netclient/equeue-ssl and my example use ocaml-ssl. I also don't have time to pursue this much further, so I will try to put all of my http operations on one thread as your example suggests. Best, Mike On Tue, Nov 27, 2012 at 5:03 AM, Gerd Stolpmann wrote: > Hi Mike, > > I can confirm the problem, but I'm not able to track it down. > > First of all, I'm quite sure that you cannot share SSL contexts between > threads. But even if I fix this, I get crashes - usually early during > SSL_connect. > > I'm wondering whether ocaml-ssl initializes openssl correctly for > multi-threading. In my version of openssl the prototype is > > int CRYPTO_THREADID_set_callback(**void (*threadid_func)(CRYPTO_**THREADID > *)); > > but in ocaml-ssl the CRYPTO_THREADID pointer is ignored (it assumes void > here). So, maybe this is the reason. I currently don't have time to follow > this idea. > > If everything fails, there is a way to use Netclient in threaded > applications so that only one thread is doing the networking stuff, see > this example: > > https://godirepo.camlcity.org/**svn/lib-ocamlnet2/trunk/code/** > examples/netclient/simple/**http_mt.ml > > As Netclient is event-driven, the requests are processed in parallel (with > one exception, though: DNS lookups are synchronous, but this can be worked > around, as the DNS lookup function is settable in Netclient). > > Gerd > > Am 27.11.2012 03:54:16 schrieb(en) Mike Lin: > > Hello Gerd, all, >> >> I'm trying to find the right incantations to make parallel https requests >> from multiple threads using Http_client. (I know the library is capable of >> multiple requests from one thread -- I have other strong reasons to use >> multithreading.) >> >> https://gist.github.com/**4152047 >> >> This program generally segfaults for me at some random point during the 10 >> requests. Compiled as in the comment in line 2 (ubuntu quantal, >> libocamlnet-ocaml-dev 3.5.1, ocaml 3.12.1). >> >> Permutations attempted: >> >> 1) Change https to http in google URL -- works as expected >> 2) Serialize requests by uncommenting line 35 -- works as expected >> 3) Move invocations of Ssl.init, Ssl.create_context, >> HTTPS.https_transport_channel_**type in and out of fresh_pipeline (which >> runs >> in the worker threads) -- no help >> >> Any suggestions? Thanks! >> >> Mike Lin >> >> > > > -- > ------------------------------**------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > Creator of GODI and camlcity.org. > Contact details: http://www.camlcity.org/**contact.html > Company homepage: http://www.gerd-stolpmann.de > ------------------------------**------------------------------ --f46d04083b5192f9ac04cf844b98 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks, Gerd!

FWIW I could not reproduce the = crash by using ocaml-ssl's blocking operations directly.
=

This works fine- so, perhaps something nasty aris= es from using nonblocking I/O on ssl sockets from multiple threads. I'm= sure if there is any other critical difference with how netclient/equeue-s= sl and my example use ocaml-ssl.

I also don't have time to pursue this much further,= so I will try to put all of my http operations on one thread as your examp= le suggests.

Best,
Mike


On Tue, Nov 27, 2012 at 5:03 AM, Gerd St= olpmann <info@gerd-stolpmann.de> wrote:
Hi Mike,

I can confirm the problem, but I'm not able to track it down.

First of all, I'm quite sure that you cannot share SSL contexts between= threads. But even if I fix this, I get crashes - usually early during SSL_= connect.

I'm wondering whether ocaml-ssl initializes openssl correctly for multi= -threading. In my version of openssl the prototype is

int CRYPTO_THREADID_set_callback(void (*threadid_func)(CRYPTO_THREADID *));

but in ocaml-ssl the CRYPTO_THREADID pointer is ignored (it assumes void he= re). So, maybe this is the reason. I currently don't have time to follo= w this idea.

If everything fails, there is a way to use Netclient in threaded applicatio= ns so that only one thread is doing the networking stuff, see this example:=

https://godirepo.camlcit= y.org/svn/lib-ocamlnet2/trunk/code/examples/netclient/simple/= http_mt.ml

As Netclient is event-driven, the requests are processed in parallel (with = one exception, though: DNS lookups are synchronous, but this can be worked = around, as the DNS lookup function is settable in Netclient).

Gerd

Am 27.11.2012 03:54:16 schrieb(en) Mike Lin:

Hello Gerd, all,

I'm trying to find the right incantations to make parallel https reques= ts
from multiple threads using Http_client. (I know the library is capable of<= br> multiple requests from one thread -- I have other strong reasons to use
multithreading.)

https://gist.= github.com/4152047

This program generally segfaults for me at some random point during the 10<= br> requests. =A0Compiled as in the comment in line 2 (ubuntu quantal,
libocamlnet-ocaml-dev 3.5.1, ocaml 3.12.1).

Permutations attempted:

1) Change https to http in google URL -- works as expected
2) Serialize requests by uncommenting line 35 -- works as expected
3) Move invocations of Ssl.init, Ssl.create_context,
HTTPS.https_transport_channel_type in and out of fresh_pipeline (whi= ch runs
in the worker threads) -- no help

Any suggestions? Thanks!

Mike Lin




--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany =A0 =A0gerd@gerd-stolpmann.de
Creator of GODI and camlc= ity.org.
Contact details: =A0 =A0 =A0 =A0http://www.camlcity.org/contact.html
Company homepage: =A0 =A0 =A0 http://www.gerd-stolpmann.de
------------------------------------------------------------<= /span>

--f46d04083b5192f9ac04cf844b98--