From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id E2C7A7F6CC for ; Thu, 5 Feb 2015 00:51:26 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.126.187; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.126.187; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.kundenserver.de) identity=helo; client-ip=212.227.126.187; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@mout.kundenserver.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0ANAQA8sNJUm7t+49RagmR0WQSydI8/CoVxAoEiQwEBAQEBEQEBAQEBBgsLCRQuhAwBAQEDAVUZCwULBQYYDSFFBAENBhMJCAGIBwMJCQMBCNB0A4VgAQEBAQEFAQEBAQEBHIoOhRIlMweCLUyBMAWESwp4iUU9ARaCfIZwNoRvJIYJhEKBOIQRbgGBASSBHAEBAQ X-IPAS-Result: A0ANAQA8sNJUm7t+49RagmR0WQSydI8/CoVxAoEiQwEBAQEBEQEBAQEBBgsLCRQuhAwBAQEDAVUZCwULBQYYDSFFBAENBhMJCAGIBwMJCQMBCNB0A4VgAQEBAQEFAQEBAQEBHIoOhRIlMweCLUyBMAWESwp4iUU9ARaCfIZwNoRvJIYJhEKBOIQRbgGBASSBHAEBAQ X-IronPort-AV: E=Sophos;i="5.09,521,1418079600"; d="scan'208";a="120310862" Received: from mout.kundenserver.de ([212.227.126.187]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 05 Feb 2015 00:51:26 +0100 Received: from office1.lan.sumadev.de ([88.69.151.168]) by mrelayeu.kundenserver.de (mreue003) with ESMTPSA (Nemesis) id 0M2puc-1XSqaZ11qo-00sfhh; Thu, 05 Feb 2015 00:51:25 +0100 Received: from gps.dynxs.de (localhost [IPv6:::1]) by office1.lan.sumadev.de (Postfix) with ESMTP id 98180DC05D; Thu, 5 Feb 2015 00:51:24 +0100 (CET) Received: from 204.14.239.105 (SquirrelMail authenticated user gerd) by gps.dynxs.de with HTTP; Thu, 5 Feb 2015 00:51:24 +0100 Message-ID: <6395dd48f0abe859ff44f5095631d32c.squirrel@gps.dynxs.de> In-Reply-To: <20150204164702.GA14942@birba.invalid> References: <20150202103256.GA30053@birba.invalid> <20150204164702.GA14942@birba.invalid> Date: Thu, 5 Feb 2015 00:51:24 +0100 From: "Gerd Stolpmann" To: "Enrico Tassi" Cc: caml-list@inria.fr User-Agent: SquirrelMail/1.4.23 [SVN] MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 X-Priority: 3 (Normal) Importance: Normal Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K0:6jaOShwcMZ7nTU8hAEwt9aWPMuMKaESkLEVeBXIBSZd+Aezf4BY HFwTAHyoTSR6mK+izjbS2akW/bXESVe26VDFtPrI0WKBTLdSGie8AX5H6Lgxq6Pha8b/Lm5 DFdFc4QtCH+W8RmExKMiZu7vB/KBwxr/KP/BkfQT8KJL1Hc628megfL59x7Th/v/J61tIRL kXOAbXU2ukqpWr531KlCg== X-UI-Out-Filterresults: notjunk:1; Subject: Re: [Caml-list] unmarshaling large data from string on 32 bits What about this: you change the protocol so that there is a single character, say an 'X', before any marshalled value. The 'X' is something you can use for non-blocking reads. So (if ch is the input channel): Unix.set_nonblock (Unix.descr_of_in_channel ch); let x =3D input_char ch in (* or Sys_blocked_io *) assert(x =3D 'X'); Unix.clear_nonblock (Unix.descr_of_in_channel ch); let v =3D Marshal.from_channel ch This will also work when there are several messages in the input buffer, as input_char then simply succeeds. If you get a Sys_blocked_io, you can even revert to using select() because you know that the buffer is empty then. Gerd > On Mon, Feb 02, 2015 at 01:00:53PM +0100, Gabriel Scherer wrote: >> If you don't mind going through a temporary file, >> Marshal.{to,from}_channel should work fine. > > Thanks for the suggestion, if Windows is as smart as Linux than a > tmpfile should work fine. If not, well, better than nothing. > >> You should consider opening a problem report to OCaml upstream ( >> http://caml.inria.fr/mantis/ ) explaining the use-case and asking for >> a large-string-safe API (eg. taking and returning lists of strings). > > The chain of workarounds that leads here is long an ugly :-/ > > 1. I have a problems with threads on Windows and (rarely) on Linux. > The model is simple, Coq sits between 1 user interface and many > (usually only 1) worker process. Coq's main thread talks to the > UI via a socket and does blocking calls; worker manager > threads (1 per worker) do the same with their respective workers. > At some point all threads are blocked reading. Then > a worker process writes data but no thread is woken up. > On Linux I need at least 2 worker manager threads to see the problem, > on Windows 1 is enough. All that using the channels API and Marshal. > > OK, I say, let's go back to the old good Unix.select to read only when > some > data is there. > > 2. The Unix module lets you get the fd number associated to the channel > and you can use Unix.select with it. And you can still use the > channels > API to Marshal.from_channel. Looks good but I still a problem. I have > LARGE and small messages. The small ones fit, largely, in the > channels buffer. Result: you have 2 "values" in the buffers of the > OS. Select tells you that you can read. You Marshal.from_channel. > Both values are moved in the channel buffer, but clearly > "input_value" reads only the first one. You select again, but this > time the OS buffers are empty. So you wait until next message > arrives to discover the one forgotten in the channel buffer. > > I can't bet all my money on the correctness of this diagnoses, > but that seemed the cause at the time. Artificially inflating > messages was working, but this is not what you want. There is no > API, at least in 3.12, to peek a channel and see if there is > data (and if so, don't call select). I tried with non blocking > channels, but I could not succeed using input_value there (I don't > recall if input_value is always blocking or something else went > wrong). > > OK, I say, let's not use the channels and do old good Unix select and > read. Unfortunately the size of buffers, strings, is limited and the > LARGE messages I have do not fit. > > So yes, Marshal.from_string_list would be an option here. > > I still have around a simple example that locks up on Windows, > I'll open a bug for that. > > Best, > -- > Enrico Tassi > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------