* [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
@ 2017-07-05 22:55 Shayne Fletcher
2017-07-06 5:23 ` Nicolás Ojeda Bär
0 siblings, 1 reply; 5+ messages in thread
From: Shayne Fletcher @ 2017-07-05 22:55 UTC (permalink / raw)
To: caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 2257 bytes --]
Can anyone help me out with this?
In short, I'm doing a HTTP POST with a handler along the lines of:
```
fun ((resp : Cohttp_async.Response.t)
, (body : Cohttp_async.Body.t)) : unit Or_error.t ->
let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
let _ =
Async.Pipe.iter
r
~continue_on_error:true
~f:(fun s -> Async.return (printf "%s" s)) in
(Ok () : unit Or_error.t)
```
Each time I invoke the program I get, more or less output (very
occasionally none) but never all of it. The response header says it's
"fixed 2700816" and indeed, if I replace the code above with
```
fun ((resp : Cohttp_async.Response.t)
, (body : Cohttp_async.Body.t)) : unit Or_error.t ->
let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
let n : int Deferred.t =
Async.Pipe.fold
r
~init:0
~f:(fun acc s ->
printf "acc : %d\n" acc; Async.return (acc + String.length s)
) in
let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in
(Ok () : unit Or_error.t)
```
then (for example), on the first run I might see
```
acc : 0
acc : 1118
acc : 7503
acc : 8780
```
and then on the second run I might see
```
acc : 0
acc : 1118
acc : 4949
```
but never have I seen "Chars read : 2700816".
I've tried a bunch of different things:
- `Pipe.read_all`:
```
let _ =
Async.Pipe.read_all r >>|
fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in
...
```
No output.
- `Pipe.drain_and_count`:
```
let _ =
Async.Pipe.drain_and_count r >>|
fun n -> printf "Count %d\n" n in
...
```
No output.
- `Pipe.to_list`:
```
let _ =
Async.Pipe.to_list r >>|
fun l -> printf "%s" (String.concat ~sep:"" l) in
...
```
No output.
Also tried the obvious:
- `Cohttp_async.Body.to_string`:
```
let s = Cohttp_async.Body.to_string body in
let _ = s >>| fun s -> printf "%s" s in
...
```
No output.
I'm new to this and obviously missing something fundamental :) Is there a
kind soul out there who can give me a hint on how to proceed please?
Thanks!
--
Shayne Fletcher
[-- Attachment #2: Type: text/html, Size: 9210 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
2017-07-05 22:55 [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body Shayne Fletcher
@ 2017-07-06 5:23 ` Nicolás Ojeda Bär
2017-07-06 5:37 ` Yaron Minsky
0 siblings, 1 reply; 5+ messages in thread
From: Nicolás Ojeda Bär @ 2017-07-06 5:23 UTC (permalink / raw)
To: Shayne Fletcher; +Cc: caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]
Hi Shayne,
I am not very familiar with Async, but did you remember to run the
scheduler (Scheduler.go () or similar) ?
Cheers,
Nicolas
On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher <
shayne.fletcher.50@gmail.com> wrote:
> Can anyone help me out with this?
>
> In short, I'm doing a HTTP POST with a handler along the lines of:
> ```
> fun ((resp : Cohttp_async.Response.t)
> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
> let _ =
> Async.Pipe.iter
> r
> ~continue_on_error:true
> ~f:(fun s -> Async.return (printf "%s" s)) in
> (Ok () : unit Or_error.t)
>
> ```
>
> Each time I invoke the program I get, more or less output (very
> occasionally none) but never all of it. The response header says it's
> "fixed 2700816" and indeed, if I replace the code above with
> ```
> fun ((resp : Cohttp_async.Response.t)
> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
> let n : int Deferred.t =
> Async.Pipe.fold
> r
> ~init:0
> ~f:(fun acc s ->
> printf "acc : %d\n" acc; Async.return (acc + String.length s)
> ) in
> let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in
> (Ok () : unit Or_error.t)
>
> ```
> then (for example), on the first run I might see
> ```
> acc : 0
> acc : 1118
> acc : 7503
> acc : 8780
>
> ```
>
> and then on the second run I might see
> ```
> acc : 0
> acc : 1118
> acc : 4949
>
> ```
> but never have I seen "Chars read : 2700816".
>
> I've tried a bunch of different things:
>
> - `Pipe.read_all`:
> ```
> let _ =
> Async.Pipe.read_all r >>|
> fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in
> ...
>
> ```
> No output.
>
> - `Pipe.drain_and_count`:
> ```
> let _ =
> Async.Pipe.drain_and_count r >>|
> fun n -> printf "Count %d\n" n in
> ...
>
> ```
> No output.
>
> - `Pipe.to_list`:
> ```
> let _ =
> Async.Pipe.to_list r >>|
> fun l -> printf "%s" (String.concat ~sep:"" l) in
> ...
>
> ```
> No output.
>
> Also tried the obvious:
>
> - `Cohttp_async.Body.to_string`:
> ```
> let s = Cohttp_async.Body.to_string body in
> let _ = s >>| fun s -> printf "%s" s in
> ...
>
> ```
> No output.
>
> I'm new to this and obviously missing something fundamental :) Is there a
> kind soul out there who can give me a hint on how to proceed please?
>
> Thanks!
>
> --
> Shayne Fletcher
>
[-- Attachment #2: Type: text/html, Size: 9948 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
2017-07-06 5:23 ` Nicolás Ojeda Bär
@ 2017-07-06 5:37 ` Yaron Minsky
2017-07-06 6:51 ` Shayne Fletcher
2017-07-06 17:20 ` Shayne Fletcher
0 siblings, 2 replies; 5+ messages in thread
From: Yaron Minsky @ 2017-07-06 5:37 UTC (permalink / raw)
To: Nicolás Ojeda Bär; +Cc: Shayne Fletcher, caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 3288 bytes --]
The fact that you're ignoring the result of Pipe.iter is concerning. That
function should return a deferred that becomes determined once the
iteration is complete. It's hard to say without having a bit more of the
program, but I suspect the bug is related to that.
y
On Thu, Jul 6, 2017 at 8:23 AM, Nicolás Ojeda Bär <
nicolas.ojeda.bar@lexifi.com> wrote:
> Hi Shayne,
>
> I am not very familiar with Async, but did you remember to run the
> scheduler (Scheduler.go () or similar) ?
>
> Cheers,
> Nicolas
>
>
> On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher <
> shayne.fletcher.50@gmail.com> wrote:
>
>> Can anyone help me out with this?
>>
>> In short, I'm doing a HTTP POST with a handler along the lines of:
>> ```
>> fun ((resp : Cohttp_async.Response.t)
>> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body
>> in
>> let _ =
>> Async.Pipe.iter
>> r
>> ~continue_on_error:true
>> ~f:(fun s -> Async.return (printf "%s" s)) in
>> (Ok () : unit Or_error.t)
>>
>> ```
>>
>> Each time I invoke the program I get, more or less output (very
>> occasionally none) but never all of it. The response header says it's
>> "fixed 2700816" and indeed, if I replace the code above with
>> ```
>> fun ((resp : Cohttp_async.Response.t)
>> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
>> let n : int Deferred.t =
>> Async.Pipe.fold
>> r
>> ~init:0
>> ~f:(fun acc s ->
>> printf "acc : %d\n" acc; Async.return (acc + String.length s)
>> ) in
>> let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in
>> (Ok () : unit Or_error.t)
>>
>> ```
>> then (for example), on the first run I might see
>> ```
>> acc : 0
>> acc : 1118
>> acc : 7503
>> acc : 8780
>>
>> ```
>>
>> and then on the second run I might see
>> ```
>> acc : 0
>> acc : 1118
>> acc : 4949
>>
>> ```
>> but never have I seen "Chars read : 2700816".
>>
>> I've tried a bunch of different things:
>>
>> - `Pipe.read_all`:
>> ```
>> let _ =
>> Async.Pipe.read_all r >>|
>> fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in
>> ...
>>
>> ```
>> No output.
>>
>> - `Pipe.drain_and_count`:
>> ```
>> let _ =
>> Async.Pipe.drain_and_count r >>|
>> fun n -> printf "Count %d\n" n in
>> ...
>>
>> ```
>> No output.
>>
>> - `Pipe.to_list`:
>> ```
>> let _ =
>> Async.Pipe.to_list r >>|
>> fun l -> printf "%s" (String.concat ~sep:"" l) in
>> ...
>>
>> ```
>> No output.
>>
>> Also tried the obvious:
>>
>> - `Cohttp_async.Body.to_string`:
>> ```
>> let s = Cohttp_async.Body.to_string body in
>> let _ = s >>| fun s -> printf "%s" s in
>> ...
>>
>> ```
>> No output.
>>
>> I'm new to this and obviously missing something fundamental :) Is there a
>> kind soul out there who can give me a hint on how to proceed please?
>>
>> Thanks!
>>
>> --
>> Shayne Fletcher
>>
>
>
[-- Attachment #2: Type: text/html, Size: 10757 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
2017-07-06 5:37 ` Yaron Minsky
@ 2017-07-06 6:51 ` Shayne Fletcher
2017-07-06 17:20 ` Shayne Fletcher
1 sibling, 0 replies; 5+ messages in thread
From: Shayne Fletcher @ 2017-07-06 6:51 UTC (permalink / raw)
To: Yaron Minsky; +Cc: Nicolás Ojeda Bär, caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 4303 bytes --]
On Thu, Jul 6, 2017 at 1:37 AM, Yaron Minsky <yminsky@janestreet.com> wrote:
> The fact that you're ignoring the result of Pipe.iter is concerning. That
> function should return a deferred that becomes determined once the
> iteration is complete. It's hard to say without having a bit more of the
> program, but I suspect the bug is related to that.
>
> y
>
>
Ah! Thank-you Yaron. Is something close to this schematic looking more
plausible?
let run : Command.t =
let cmd_proc : (unit -> unit Deferred.Or_error.t) Param.t =
let open Command.Let_syntax in
[%map_open
let uri = ...
and ... in
fun () ->
Async_client.post ... uri >>|
fun (_, body) ->
let r = Cohttp_async.Body.to_pipe body in
Async.Pipe.iter
r
~continue_on_error:true
~f:(fun s -> Async.return (printf "%s" s)) >>|
fun () -> Ok ()
] in
Command.async_or_error' ~summary:"..." cmd_proc
let main : Command.t =
Command.group ~summary:"..."
[
"run", run
]
let () = Command.run ~v main
On Thu, Jul 6, 2017 at 8:23 AM, Nicolás Ojeda Bär <
> nicolas.ojeda.bar@lexifi.com> wrote:
>
>> Hi Shayne,
>>
>> I am not very familiar with Async, but did you remember to run the
>> scheduler (Scheduler.go () or similar) ?
>>
>> Cheers,
>> Nicolas
>>
>>
>> On Thu, Jul 6, 2017 at 12:55 AM, Shayne Fletcher <
>> shayne.fletcher.50@gmail.com> wrote:
>>
>>> Can anyone help me out with this?
>>>
>>> In short, I'm doing a HTTP POST with a handler along the lines of:
>>> ```
>>> fun ((resp : Cohttp_async.Response.t)
>>> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
>>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body
>>> in
>>> let _ =
>>> Async.Pipe.iter
>>> r
>>> ~continue_on_error:true
>>> ~f:(fun s -> Async.return (printf "%s" s)) in
>>> (Ok () : unit Or_error.t)
>>>
>>> ```
>>>
>>> Each time I invoke the program I get, more or less output (very
>>> occasionally none) but never all of it. The response header says it's
>>> "fixed 2700816" and indeed, if I replace the code above with
>>> ```
>>> fun ((resp : Cohttp_async.Response.t)
>>> , (body : Cohttp_async.Body.t)) : unit Or_error.t ->
>>> let r : string Async.Pipe.Reader.t = Cohttp_async.Body.to_pipe body in
>>> let n : int Deferred.t =
>>> Async.Pipe.fold
>>> r
>>> ~init:0
>>> ~f:(fun acc s ->
>>> printf "acc : %d\n" acc; Async.return (acc + String.length s)
>>> ) in
>>> let _ : unit Deferred.t = n >>| fun n -> printf "Chars read : %d" n in
>>> (Ok () : unit Or_error.t)
>>>
>>> ```
>>> then (for example), on the first run I might see
>>> ```
>>> acc : 0
>>> acc : 1118
>>> acc : 7503
>>> acc : 8780
>>>
>>> ```
>>>
>>> and then on the second run I might see
>>> ```
>>> acc : 0
>>> acc : 1118
>>> acc : 4949
>>>
>>> ```
>>> but never have I seen "Chars read : 2700816".
>>>
>>> I've tried a bunch of different things:
>>>
>>> - `Pipe.read_all`:
>>> ```
>>> let _ =
>>> Async.Pipe.read_all r >>|
>>> fun q -> Queue.iter q ~f:(fun s -> printf "%s" s) in
>>> ...
>>>
>>> ```
>>> No output.
>>>
>>> - `Pipe.drain_and_count`:
>>> ```
>>> let _ =
>>> Async.Pipe.drain_and_count r >>|
>>> fun n -> printf "Count %d\n" n in
>>> ...
>>>
>>> ```
>>> No output.
>>>
>>> - `Pipe.to_list`:
>>> ```
>>> let _ =
>>> Async.Pipe.to_list r >>|
>>> fun l -> printf "%s" (String.concat ~sep:"" l) in
>>> ...
>>>
>>> ```
>>> No output.
>>>
>>> Also tried the obvious:
>>>
>>> - `Cohttp_async.Body.to_string`:
>>> ```
>>> let s = Cohttp_async.Body.to_string body in
>>> let _ = s >>| fun s -> printf "%s" s in
>>> ...
>>>
>>> ```
>>> No output.
>>>
>>> I'm new to this and obviously missing something fundamental :) Is there
>>> a kind soul out there who can give me a hint on how to proceed please?
>>>
>>> Thanks!
>>>
>>> --
>>> Shayne Fletcher
>>>
>>
>>
>
--
Shayne Fletcher
[-- Attachment #2: Type: text/html, Size: 13019 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body
2017-07-06 5:37 ` Yaron Minsky
2017-07-06 6:51 ` Shayne Fletcher
@ 2017-07-06 17:20 ` Shayne Fletcher
1 sibling, 0 replies; 5+ messages in thread
From: Shayne Fletcher @ 2017-07-06 17:20 UTC (permalink / raw)
To: Yaron Minsky; +Cc: Nicolás Ojeda Bär, caml-list@inria.fr users
[-- Attachment #1: Type: text/plain, Size: 752 bytes --]
On Thu, Jul 6, 2017 at 1:37 AM, Yaron Minsky <yminsky@janestreet.com> wrote:
> The fact that you're ignoring the result of Pipe.iter is concerning. That
> function should return a deferred that becomes determined once the
> iteration is complete. It's hard to say without having a bit more of the
> program, but I suspect the bug is related to that.
>
The following fragment seems to do the trick.
```
fun () : unit Deferred.Or_error.t ->
Async_client.post ... uri >>= fun (_, body) ->
Async.Pipe.iter
~continue_on_error:true
~f:(fun s -> Deferred.return (printf "%s" s))
(Cohttp_async.Body.to_pipe body) >>| fun () ->
Ok ()
```
Thanks again.
--
Shayne Fletcher
[-- Attachment #2: Type: text/html, Size: 1761 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-07-06 17:21 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-05 22:55 [Caml-list] [Async / Cohttp_async] Problem getting all of the data in a response body Shayne Fletcher
2017-07-06 5:23 ` Nicolás Ojeda Bär
2017-07-06 5:37 ` Yaron Minsky
2017-07-06 6:51 ` Shayne Fletcher
2017-07-06 17:20 ` Shayne Fletcher
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).