* [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
@ 2017-07-05 22:55 Shayne Fletcher
2017-07-06 5:23 ` Nicolás Ojeda Bär
0 siblings, 1 reply; 5+ messages in thread
From: Shayne Fletcher @ 2017-07-05 22:55 UTC (permalink / raw)
To: caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 2257 bytes --]
Can anyone help me out with this?
In short, I'm doing a HTTP POST with a handler along the lines of:
```
fun ((resp : Cohttp_async.Response.t)
, (body : Cohttp_async.Body.t)) : unit Or_error.t ->
let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
let _ =
Async.Pipe.iter
r
~continue_on_error:true
~f:(fun s -> Async.return (printf "%s" s)) in
(Ok () : unit Or_error.t)
```
Each time I invoke the program I get, more or less output (very
occasionally none) but never all of it. The response header says it's
"fixed 2700816" and indeed, if I replace the code above with
```
fun ((resp : Cohttp_async.Response.t)
, (body : Cohttp_async.Body.t)) : unit Or_error.t ->
let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
let n : int Deferred.t =
Async.Pipe.fold
r
~init:0
~f:(fun acc s ->
printf "acc : %d\n" acc; Async.return (acc + String.length s)
) in
let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in
(Ok () : unit Or_error.t)
```
then (for example), on the first run I might see
```
acc : 0
acc : 1118
acc : 7503
acc : 8780
```
and then on the second run I might see
```
acc : 0
acc : 1118
acc : 4949
```
but never have I seen "Chars read : 2700816".
I've tried a bunch of different things:
- `Pipe.read_all`:
```
let _ =
Async.Pipe.read_all r >>|
fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in
...
```
No output.
- `Pipe.drain_and_count`:
```
let _ =
Async.Pipe.drain_and_count r >>|
fun n -> printf "Count %d\n" n in
...
```
No output.
- `Pipe.to_list`:
```
let _ =
Async.Pipe.to_list r >>|
fun l -> printf "%s" (String.concat ~sep:"" l) in
...
```
No output.
Also tried the obvious:
- `Cohttp_async.Body.to_string`:
```
let s = Cohttp_async.Body.to_string body in
let _ = s >>| fun s -> printf "%s" s in
...
```
No output.
I'm new to this and obviously missing something fundamental :) Is there a
kind soul out there who can give me a hint on how to proceed please?
Thanks!
--
Shayne Fletcher
[-- Attachment #2: Type: text/html, Size: 9210 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body 2017-07-05 22:55 [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body Shayne Fletcher @ 2017-07-06 5:23 ` Nicolás Ojeda Bär 2017-07-06 5:37 ` Yaron Minsky 0 siblings, 1 reply; 5+ messages in thread From: Nicolás Ojeda Bär @ 2017-07-06 5:23 UTC (permalink / raw) To: Shayne Fletcher; +Cc: caml-list@inria.fr users [-- Attachment #1: Type: text/plain, Size: 2670 bytes --] Hi Shayne, I am not very familiar with Async, but did you remember to run the scheduler (Scheduler.go () or similar) ? Cheers, Nicolas On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher < shayne.fletcher.50@gmail.com> wrote: > Can anyone help me out with this? > > In short, I'm doing a HTTP POST with a handler along the lines of: > ``` > fun ((resp : Cohttp_async.Response.t) > , (body : Cohttp_async.Body.t)) : unit Or_error.t -> > let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in > let _ = > Async.Pipe.iter > r > ~continue_on_error:true > ~f:(fun s -> Async.return (printf "%s" s)) in > (Ok () : unit Or_error.t) > > ``` > > Each time I invoke the program I get, more or less output (very > occasionally none) but never all of it. The response header says it's > "fixed 2700816" and indeed, if I replace the code above with > ``` > fun ((resp : Cohttp_async.Response.t) > , (body : Cohttp_async.Body.t)) : unit Or_error.t -> > let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in > let n : int Deferred.t = > Async.Pipe.fold > r > ~init:0 > ~f:(fun acc s -> > printf "acc : %d\n" acc; Async.return (acc + String.length s) > ) in > let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in > (Ok () : unit Or_error.t) > > ``` > then (for example), on the first run I might see > ``` > acc : 0 > acc : 1118 > acc : 7503 > acc : 8780 > > ``` > > and then on the second run I might see > ``` > acc : 0 > acc : 1118 > acc : 4949 > > ``` > but never have I seen "Chars read : 2700816". > > I've tried a bunch of different things: > > - `Pipe.read_all`: > ``` > let _ = > Async.Pipe.read_all r >>| > fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in > ... > > ``` > No output. > > - `Pipe.drain_and_count`: > ``` > let _ = > Async.Pipe.drain_and_count r >>| > fun n -> printf "Count %d\n" n in > ... > > ``` > No output. > > - `Pipe.to_list`: > ``` > let _ = > Async.Pipe.to_list r >>| > fun l -> printf "%s" (String.concat ~sep:"" l) in > ... > > ``` > No output. > > Also tried the obvious: > > - `Cohttp_async.Body.to_string`: > ``` > let s = Cohttp_async.Body.to_string body in > let _ = s >>| fun s -> printf "%s" s in > ... > > ``` > No output. > > I'm new to this and obviously missing something fundamental :) Is there a > kind soul out there who can give me a hint on how to proceed please? > > Thanks! > > -- > Shayne Fletcher > [-- Attachment #2: Type: text/html, Size: 9948 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body 2017-07-06 5:23 ` Nicolás Ojeda Bär @ 2017-07-06 5:37 ` Yaron Minsky 2017-07-06 6:51 ` Shayne Fletcher 2017-07-06 17:20 ` Shayne Fletcher 0 siblings, 2 replies; 5+ messages in thread From: Yaron Minsky @ 2017-07-06 5:37 UTC (permalink / raw) To: Nicolás Ojeda Bär; +Cc: Shayne Fletcher, caml-list@inria.fr users [-- Attachment #1: Type: text/plain, Size: 3288 bytes --] The fact that you're ignoring the result of Pipe.iter is concerning. That function should return a deferred that becomes determined once the iteration is complete. It's hard to say without having a bit more of the program, but I suspect the bug is related to that. y On Thu, Jul 6, 2017 at 8:23 AM, Nicolás Ojeda Bär < nicolas.ojeda.bar@lexifi.com> wrote: > Hi Shayne, > > I am not very familiar with Async, but did you remember to run the > scheduler (Scheduler.go () or similar) ? > > Cheers, > Nicolas > > > On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher < > shayne.fletcher.50@gmail.com> wrote: > >> Can anyone help me out with this? >> >> In short, I'm doing a HTTP POST with a handler along the lines of: >> ``` >> fun ((resp : Cohttp_async.Response.t) >> , (body : Cohttp_async.Body.t)) : unit Or_error.t -> >> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body >> in >> let _ = >> Async.Pipe.iter >> r >> ~continue_on_error:true >> ~f:(fun s -> Async.return (printf "%s" s)) in >> (Ok () : unit Or_error.t) >> >> ``` >> >> Each time I invoke the program I get, more or less output (very >> occasionally none) but never all of it. The response header says it's >> "fixed 2700816" and indeed, if I replace the code above with >> ``` >> fun ((resp : Cohttp_async.Response.t) >> , (body : Cohttp_async.Body.t)) : unit Or_error.t -> >> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in >> let n : int Deferred.t = >> Async.Pipe.fold >> r >> ~init:0 >> ~f:(fun acc s -> >> printf "acc : %d\n" acc; Async.return (acc + String.length s) >> ) in >> let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in >> (Ok () : unit Or_error.t) >> >> ``` >> then (for example), on the first run I might see >> ``` >> acc : 0 >> acc : 1118 >> acc : 7503 >> acc : 8780 >> >> ``` >> >> and then on the second run I might see >> ``` >> acc : 0 >> acc : 1118 >> acc : 4949 >> >> ``` >> but never have I seen "Chars read : 2700816". >> >> I've tried a bunch of different things: >> >> - `Pipe.read_all`: >> ``` >> let _ = >> Async.Pipe.read_all r >>| >> fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in >> ... >> >> ``` >> No output. >> >> - `Pipe.drain_and_count`: >> ``` >> let _ = >> Async.Pipe.drain_and_count r >>| >> fun n -> printf "Count %d\n" n in >> ... >> >> ``` >> No output. >> >> - `Pipe.to_list`: >> ``` >> let _ = >> Async.Pipe.to_list r >>| >> fun l -> printf "%s" (String.concat ~sep:"" l) in >> ... >> >> ``` >> No output. >> >> Also tried the obvious: >> >> - `Cohttp_async.Body.to_string`: >> ``` >> let s = Cohttp_async.Body.to_string body in >> let _ = s >>| fun s -> printf "%s" s in >> ... >> >> ``` >> No output. >> >> I'm new to this and obviously missing something fundamental :) Is there a >> kind soul out there who can give me a hint on how to proceed please? >> >> Thanks! >> >> -- >> Shayne Fletcher >> > > [-- Attachment #2: Type: text/html, Size: 10757 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body 2017-07-06 5:37 ` Yaron Minsky @ 2017-07-06 6:51 ` Shayne Fletcher 2017-07-06 17:20 ` Shayne Fletcher 1 sibling, 0 replies; 5+ messages in thread From: Shayne Fletcher @ 2017-07-06 6:51 UTC (permalink / raw) To: Yaron Minsky; +Cc: Nicolás Ojeda Bär, caml-list@inria.fr users [-- Attachment #1: Type: text/plain, Size: 4303 bytes --] On Thu, Jul 6, 2017 at 1:37 AM, Yaron Minsky <yminsky@janestreet.com> wrote: > The fact that you're ignoring the result of Pipe.iter is concerning. That > function should return a deferred that becomes determined once the > iteration is complete. It's hard to say without having a bit more of the > program, but I suspect the bug is related to that. > > y > > Ah! Thank-you Yaron. Is something close to this schematic looking more plausible? let run : Command.t = let cmd_proc : (unit -> unit Deferred.Or_error.t) Param.t = let open Command.Let_syntax in [%map_open let uri = ... and ... in fun () -> Async_client.post ... uri >>| fun (_, body) -> let r = Cohttp_async.Body.to_pipe body in Async.Pipe.iter r ~continue_on_error:true ~f:(fun s -> Async.return (printf "%s" s)) >>| fun () -> Ok () ] in Command.async_or_error' ~summary:"..." cmd_proc let main : Command.t = Command.group ~summary:"..." [ "run", run ] let () = Command.run ~v main On Thu, Jul 6, 2017 at 8:23 AM, Nicolás Ojeda Bär < > nicolas.ojeda.bar@lexifi.com> wrote: > >> Hi Shayne, >> >> I am not very familiar with Async, but did you remember to run the >> scheduler (Scheduler.go () or similar) ? >> >> Cheers, >> Nicolas >> >> >> On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher < >> shayne.fletcher.50@gmail.com> wrote: >> >>> Can anyone help me out with this? >>> >>> In short, I'm doing a HTTP POST with a handler along the lines of: >>> ``` >>> fun ((resp : Cohttp_async.Response.t) >>> , (body : Cohttp_async.Body.t)) : unit Or_error.t -> >>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body >>> in >>> let _ = >>> Async.Pipe.iter >>> r >>> ~continue_on_error:true >>> ~f:(fun s -> Async.return (printf "%s" s)) in >>> (Ok () : unit Or_error.t) >>> >>> ``` >>> >>> Each time I invoke the program I get, more or less output (very >>> occasionally none) but never all of it. The response header says it's >>> "fixed 2700816" and indeed, if I replace the code above with >>> ``` >>> fun ((resp : Cohttp_async.Response.t) >>> , (body : Cohttp_async.Body.t)) : unit Or_error.t -> >>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in >>> let n : int Deferred.t = >>> Async.Pipe.fold >>> r >>> ~init:0 >>> ~f:(fun acc s -> >>> printf "acc : %d\n" acc; Async.return (acc + String.length s) >>> ) in >>> let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in >>> (Ok () : unit Or_error.t) >>> >>> ``` >>> then (for example), on the first run I might see >>> ``` >>> acc : 0 >>> acc : 1118 >>> acc : 7503 >>> acc : 8780 >>> >>> ``` >>> >>> and then on the second run I might see >>> ``` >>> acc : 0 >>> acc : 1118 >>> acc : 4949 >>> >>> ``` >>> but never have I seen "Chars read : 2700816". >>> >>> I've tried a bunch of different things: >>> >>> - `Pipe.read_all`: >>> ``` >>> let _ = >>> Async.Pipe.read_all r >>| >>> fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in >>> ... >>> >>> ``` >>> No output. >>> >>> - `Pipe.drain_and_count`: >>> ``` >>> let _ = >>> Async.Pipe.drain_and_count r >>| >>> fun n -> printf "Count %d\n" n in >>> ... >>> >>> ``` >>> No output. >>> >>> - `Pipe.to_list`: >>> ``` >>> let _ = >>> Async.Pipe.to_list r >>| >>> fun l -> printf "%s" (String.concat ~sep:"" l) in >>> ... >>> >>> ``` >>> No output. >>> >>> Also tried the obvious: >>> >>> - `Cohttp_async.Body.to_string`: >>> ``` >>> let s = Cohttp_async.Body.to_string body in >>> let _ = s >>| fun s -> printf "%s" s in >>> ... >>> >>> ``` >>> No output. >>> >>> I'm new to this and obviously missing something fundamental :) Is there >>> a kind soul out there who can give me a hint on how to proceed please? >>> >>> Thanks! >>> >>> -- >>> Shayne Fletcher >>> >> >> > -- Shayne Fletcher [-- Attachment #2: Type: text/html, Size: 13019 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body 2017-07-06 5:37 ` Yaron Minsky 2017-07-06 6:51 ` Shayne Fletcher @ 2017-07-06 17:20 ` Shayne Fletcher 1 sibling, 0 replies; 5+ messages in thread From: Shayne Fletcher @ 2017-07-06 17:20 UTC (permalink / raw) To: Yaron Minsky; +Cc: Nicolás Ojeda Bär, caml-list@inria.fr users [-- Attachment #1: Type: text/plain, Size: 752 bytes --] On Thu, Jul 6, 2017 at 1:37 AM, Yaron Minsky <yminsky@janestreet.com> wrote: > The fact that you're ignoring the result of Pipe.iter is concerning. That > function should return a deferred that becomes determined once the > iteration is complete. It's hard to say without having a bit more of the > program, but I suspect the bug is related to that. > The following fragment seems to do the trick. ``` fun () : unit Deferred.Or_error.t -> Async_client.post ... uri >>= fun (_, body) -> Async.Pipe.iter ~continue_on_error:true ~f:(fun s -> Deferred.return (printf "%s" s)) (Cohttp_async.Body.to_pipe body) >>| fun () -> Ok () ``` Thanks again. -- Shayne Fletcher [-- Attachment #2: Type: text/html, Size: 1761 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-07-06 17:21 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-07-05 22:55 [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body Shayne Fletcher 2017-07-06 5:23 ` Nicolás Ojeda Bär 2017-07-06 5:37 ` Yaron Minsky 2017-07-06 6:51 ` Shayne Fletcher 2017-07-06 17:20 ` Shayne Fletcher
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).