From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id B024F7EEF6 for ; Wed, 17 Jun 2015 10:09:37 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of dhouse@janestreet.com) identity=pra; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="dhouse@janestreet.com"; x-sender="dhouse@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of dhouse@janestreet.com designates 38.105.200.112 as permitted sender) identity=mailfrom; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="dhouse@janestreet.com"; x-sender="dhouse@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="dhouse@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0B7AQC5KoFVnHDIaSZbgkWBfgaDGMJaAoFFB0wBAQEBAQESAQEBAQEGFglPhCIBAQEDARIRBBkBATcBBAsJAgsNIAEJAgIhARIBBQEcBhMIGod4AwoIA5kKkGs+MYpPcIRkAQWMEQ2FQQEBAQEBBQEBAQEBAQEBFAYKizqCTYFuSweCaIFDk22JZoFhgTSHE4hDgzuCERIjgRWCLh+BU26BAySBIQEBAQ X-IPAS-Result: A0B7AQC5KoFVnHDIaSZbgkWBfgaDGMJaAoFFB0wBAQEBAQESAQEBAQEGFglPhCIBAQEDARIRBBkBATcBBAsJAgsNIAEJAgIhARIBBQEcBhMIGod4AwoIA5kKkGs+MYpPcIRkAQWMEQ2FQQEBAQEBBQEBAQEBAQEBFAYKizqCTYFuSweCaIFDk22JZoFhgTSHE4hDgzuCERIjgRWCLh+BU26BAySBIQEBAQ X-IronPort-AV: E=Sophos;i="5.13,631,1427752800"; d="scan'208";a="165749614" Received: from mxout1.mail.janestreet.com ([38.105.200.112]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-SHA; 17 Jun 2015 10:09:36 +0200 Received: from tot-qpr-mailcore2.delacy.com ([172.27.56.106] helo=tot-qpr-mailcore2) by mxout1.mail.janestreet.com with smtp (Exim 4.82) (envelope-from ) id 1Z58Pf-0008CF-0d for caml-list@inria.fr; Wed, 17 Jun 2015 04:09:35 -0400 X-JS-Flow: external Received: by tot-qpr-mailcore2 with JS-mailcore (0.1) (envelope-from ) id BVgSs_-AAACkb-AK; 2015-06-17 04:09:35.006113-04:00 Received: from mail-qk0-f174.google.com ([209.85.220.174]) by mxgoog1.mail.janestreet.com with esmtps (UNKNOWN:AES128-GCM-SHA256:128) (Exim 4.72) (envelope-from ) id 1Z58Pe-0000oy-TF for caml-list@inria.fr; Wed, 17 Jun 2015 04:09:34 -0400 Received: by qkbp125 with SMTP id p125so17996091qkb.2 for ; Wed, 17 Jun 2015 01:09:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bcC4navgHGgzRssVpG93ilDFt00bwibskS1Mw0+APDY=; b=XEcyS5slqqYmjHuqe/tElWOmnAJMYcDPYI8wbyE78N8+eRKFBXUjVyr+KRh1XXbAop HLHgOGjizCjKQAIQvxoj8E0fQlteUFL1m0DMD8pJf5XbBa2gW5ZyzBOK0a635yG5VMYj sWrb9QPuvH6oEYxMtSNCmtKs8KCiCpA0Bpqv0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=bcC4navgHGgzRssVpG93ilDFt00bwibskS1Mw0+APDY=; b=IDBy0xS6ik6BUUNdEudkd1zqy5IM7P1u4ZMBExGQhFYgcMFZXh61TvXYG2k1sfpmow MpmgjlPi6FpssJclBAU0JyU7fDP7+OMcV0JeNUfVcLeiNY5peE1KBzg5MbXyDpfF8InI bJBTCmHkXjpl7KQTdMYB8c1KXu+vb9tz0AjjAn6i8IB7pOPFN3SF3etaNd7gGRPYqiKG hktMWZ9aH2gvZ4LTZChbHRcNDyPL0APYVeBbtug70IOqDqHsnO+vdCw54nVHIx5pM0j1 KWQGuQ4Jp1fRu+4JgHbvUAWh1x37mkZ4CsQyqNPmCfAhppKJ+upUSjlevBHj4LbCUe3+ ivcg== X-Gm-Message-State: ALoCoQlkEd/e3eoGCHLXTsWAElUx8MeGwKJvR69HBU5cQ5bzCV7f35plwx5Z7wslg1n3YTLVTE1DSLRkLfH+lfKWwAoSm+8w26O93N9crIlIzPGLm1N57W7YvOn6S8nexj74Zui1ZPhQ X-Received: by 10.55.42.1 with SMTP id q1mr10380719qkh.3.1434528574619; Wed, 17 Jun 2015 01:09:34 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.55.42.1 with SMTP id q1mr10380702qkh.3.1434528574506; Wed, 17 Jun 2015 01:09:34 -0700 (PDT) Received: by 10.140.107.102 with HTTP; Wed, 17 Jun 2015 01:09:34 -0700 (PDT) In-Reply-To: References: Date: Wed, 17 Jun 2015 09:09:34 +0100 Message-ID: From:David House To:Kenneth Adam Miller Cc:caml users Content-Type: multipart/alternative; boundary=001a11473db01aa2c30518b2368a X-JS-Processed-by: mailcore Subject: Re: [Caml-list] Async Server not executing --001a11473db01aa2c30518b2368a Content-Type: text/plain; charset=UTF-8 Right, length-prefixing is another common trick. In fact, this is exactly what Writer.send does. Your code at the moment is a little weird because it uses Reader.recv, whose documentation says "[recv t] returns a string that was written with Writer.send", but you're not using Writer.send. I think if you use those pairs of things then you get the semantics you want. More generally, async has good support for reading chunks of data, which might contain partial messages, and then putting those chunks together to form full messages. See Unpack_sequence in async_extra, which works with Unpack_buffer in core_kernel. On 16 June 2015 at 18:50, Kenneth Adam Miller wrote: > Ok, I have one last question - I've gotten my unit tests to succeed > properly, but I'm concerned. > > At one point in the Pipe documentation, it talked about how writes a recvs > would guarantee progress by returning early if they had something at all. I > just need some kind of guarantee of the semantics of a send and recv, so > that if I send a string of length 4, I don't receive multiple fragments. > > On Tue, Jun 16, 2015 at 11:48 AM, Kenneth Adam Miller < > kennethadammiller@gmail.com> wrote: > >> Ah. Well I think I can incorporate from that what I can, but my unit >> tests need to be reflective of the use case I have. I'm sending protobuf >> encoded messages between two processes on one machine-whatever the size of >> the message, that's the size that should be received on the other end (I've >> read about the returning partial bytes, I can't decode a part of a proto >> message, it has to be the right size). >> >> I can't use write_line, I'll have to find a way to delimit the messages >> based on size. I think I'll just prepend every message with the size that >> it should expect, and then read a integer off the stream and then that many >> bytes. >> >> On Tue, Jun 16, 2015 at 11:41 AM, David House >> wrote: >> >>> Ah, now I read your code in more detail I think I see why. >>> >>> Reader.contents on the server side will block until eof. The client side >>> sends some stuff, but does not close the writer, so the server never sees >>> eof. (Recall that sockets are not like files: it's possible to read all of >>> the available data right now, but not reach eof.) >>> >>> Network protocols normally have some explicit "this is the end of one >>> message" marker, like a newline or something similar. Then the server just >>> reads chunks until it sees that marker, at which point it can put the >>> message together and to something with it. >>> >>> For example, you could use Writer.write_line on the client side and >>> Reader.read_line on the server side. >>> >>> On 16 June 2015 at 16:36, Kenneth Adam Miller < >>> kennethadammiller@gmail.com> wrote: >>> >>>> So, now I can get server received if I add that into the callback, but >>>> at "writing shutdown to server" I don't see response received or even >>>> something for Eof. >>>> >>>> On Tue, Jun 16, 2015 at 11:09 AM, David House >>>> wrote: >>>> >>>>> The first thing to try is to make sure that everything is getting >>>>> flushed. For temporary debugging messages I strongly recommend just using >>>>> [Core.Std.eprintf "\n%!"]. >>>>> >>>>> On 16 June 2015 at 16:03, Kenneth Adam Miller < >>>>> kennethadammiller@gmail.com> wrote: >>>>> >>>>>> I'm having trouble with OCaml Async. I wrote a small server with it, >>>>>> and right now I'm trying to unit test that server. Here's my code for the >>>>>> server: >>>>>> >>>>>> >>>>>> let _main ()= >>>>>> print_endline "Server running"; >>>>>> let handler = print_endline in >>>>>> let socket = Tcp.on_port 5554 in >>>>>> let server = Tcp.Server.create socket (fun addr r w -> >>>>>> (Reader.contents r) >>| handler; (Writer.write w "got it")) in >>>>>> server >>>>>> >>>>>> >>>>>> >>>>>> In my unit test code I have: >>>>>> >>>>>> let test_shutdown test_ctxt = Thread_safe.block_on_async_exn (fun () >>>>>> -> ( >>>>>> print_endline "test_shutdown"; >>>>>> let server = Server._main () in >>>>>> server >>= fun server -> >>>>>> let where = Tcp.to_host_and_port "127.0.0.1" 5554 in >>>>>> Tcp.connect where >>= fun s -> >>>>>> let socket, r, w = s in >>>>>> ignore (Writer.write w "kill"); >>>>>> ignore (Writer.flushed w); >>>>>> (Reader.recv r >>> function >>>>>> | `Ok result -> print_endline ("writing shutdown to >>>>>> server" ^ result) >>>>>> | `Eof -> ()); >>>>>> return () >>>>>> )); () >>>>>> >>>>>> >>>>>> >>>>>> I see test_shutdown and Server running, but not sign of "writing >>>>>> shutdown to server" or even "got it"; why isn't my server or even any of >>>>>> the connection executing? >>>>>> >>>>> >>>>> >>>> >>> >> > --001a11473db01aa2c30518b2368a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Right, length-prefixing is another common trick.

<= /div>
In fact, this is exactly what Writer.send does. Your code at the = moment is a little weird because it uses Reader.recv, whose documentation s= ays "[recv t] returns a string that was written with Writer.send"= , but you're not using Writer.send. I think if you use those pairs of t= hings then you get the semantics you want.

More ge= nerally, async has good support for reading chunks of data, which might con= tain partial messages, and then putting those chunks together to form full = messages. See Unpack_sequence in async_extra, which works with Unpack_buffe= r in core_kernel.

On 16 June 2015 at 18:50, Kenneth Adam Miller <kenne= thadammiller@gmail.com> wrote:
Ok, I have one last question - I've gotten my unit= tests to succeed properly, but I'm concerned.

At on= e point in the Pipe documentation, it talked about how writes a recvs would= guarantee progress by returning early if they had something at all. I just= need some kind of guarantee of the semantics of a send and recv, so that i= f I send a string of length 4, I don't receive multiple fragments.
<= br>
On Tue, Jun 16, 2015 at 11:48 AM, Kenneth Ada= m Miller <kennethadammiller@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Ah. Well I think I can incor= porate from that what I can, but my unit tests need to be reflective of the= use case I have. I'm sending protobuf encoded messages between two pro= cesses on one machine-whatever the size of the message, that's the size= that should be received on the other end (I've read about the returnin= g partial bytes, I can't decode a part of a proto message, it has to be= the right size).=C2=A0

I can't use write_line, I= 9;ll have to find a way to delimit the messages based on size. I think I= 9;ll just prepend every message with the size that it should expect, and th= en read a integer off the stream and then that many bytes.
=

On Tue, Jun = 16, 2015 at 11:41 AM, David House <dhouse@janestreet.com> wrote:
Ah, now I read = your code in more detail I think I see why.

Reader.conte= nts on the server side will block until eof. The client side sends some stu= ff, but does not close the writer, so the server never sees eof. (Recall th= at sockets are not like files: it's possible to read all of the availab= le data right now, but not reach eof.)

Network pro= tocols normally have some explicit "this is the end of one message&quo= t; marker, like a newline or something similar. Then the server just reads = chunks until it sees that marker, at which point it can put the message tog= ether and to something with it.

For example, you c= ould use Writer.write_line on the client side and Reader.read_line on the s= erver side.

On 16 June 2015 at 16:36, Kenneth Adam Miller <= kennethadammiller@gmail.com> wrote:
So, now I can get server received if I add that i= nto the callback, but at "writing shutdown to server" I don't= see response received or even something for Eof.

On Tue, Jun 16, 2015 at 11:= 09 AM, David House <dhouse@janestreet.com> wrote:
The first thing to try is to m= ake sure that everything is getting flushed. For temporary debugging messag= es I strongly recommend just using [Core.Std.eprintf "<message>\= n%!"].

On 16 June 2015 at 16:03, Kenneth Adam Miller &= lt;kenneth= adammiller@gmail.com> wrote:
I'm having trouble with OCaml Async. I wrote a small serv= er with it, and right now I'm trying to unit test that server. Here'= ;s my code for the server:


<= /div>
let _main=C2=A0()=3D
=C2=A0 print_endline=C2=A0"= Server running";
= =C2=A0 let handler=C2=A0=3Din=3D=C2=A0Tcp.on_port=C2=A05554=C2=A0in
=C2= =A0 let server=C2=A0=3D= =C2=A0Tcp.Server.cr= eate socket=C2=A0(fun add= r r w=C2=A0->
=C2= =A0 =C2=A0 =C2=A0=C2=A0(Reader.contents r)=C2=A0>>|=C2=A0handler;=C2=A0(Writer.write w= =C2=A0"got it"))=C2=A0in
=C2=A0 server



In my unit test code I have:

let test_shutdown test_ctxt=C2=A0=3D=C2=A0Thr= ead_safe.block_on_async_e= xn=C2=A0(fun=C2=A0= ()=C2=A0->=C2=A0(
=C2=A0 =C2=A0 =C2=A0 print_endline=C2=A0"test_shutdown";
=C2=A0 =C2=A0 =C2=A0 let server=C2=A0=3D=C2=A0Server._main=C2=A0()=C2=A0in
=C2=A0 =C2=A0 =C2=A0 server=C2=A0= >>=3D=C2=A0fun server=C2=A0->
=C2=A0 =C2=A0 =C2=A0 let=C2=A0where=C2=A0=3D=C2=A0Tcp.to_host_and_port=C2=A0"127.0.0.1"=C2=A05554=C2=A0in
=C2=A0 =C2=A0 =C2=A0=C2=A0Tcp.connect=C2= =A0where=C2=A0>>=3D=C2=A0fun s=C2=A0->=C2=A0
=C2=A0 =C2=A0 =C2=A0= =C2=A0 let socket,=C2=A0= r,=C2=A0w=C2=A0=3D=C2=A0s=C2=A0in
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ignore=C2= =A0(Writer.write w=C2=A0"kill");
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 ignore=C2=A0(= Writer.);
=C2=A0 = =C2=A0 =C2=A0 =C2=A0=C2=A0(Reader.recv r=C2=A0= >>>=C2=A0= function
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0=C2=A0|=C2= =A0`Ok result -> =C2=A0print_endline ("wri= ting shutdown to server" ^ result)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 | `Eof=C2=A0->=C2=A0());
=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0= return=C2=A0()
=C2=A0 =C2=A0=C2=A0));=C2=A0()



<= /div>
I see test_shutdow= n and Server running, but not sign of "writing shutdown to server"= ; or even "got it"; why isn't my server or even any of the co= nnection executing?






--001a11473db01aa2c30518b2368a--