Netclient

Netclient

[Jean-David]

Hello there,
I was wondering about the best way to create a stream
from a remote file for which I have the URL

open Http_client
open Stream

let body = Stream.of_string Convenience.http_get
"http://dfl.sjsu.edu";;

Is the content of body downloaded completely before
being streamed(which defeats the purpose of streams)
or not? and if not how can I get the content of an URL
gradually (without getting the whole body into
memory)?

Thank you,
JD

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Caml-list] Netclient

[Gerd Stolpmann]

Am Montag, den 13.03.2006, 00:03 -0800 schrieb Jean-David:
> Hello there,
> I was wondering about the best way to create a stream
> from a remote file for which I have the URL
> 
> open Http_client
> open Stream
> 
> let body = Stream.of_string Convenience.http_get
> "http://dfl.sjsu.edu";;
> 
> Is the content of body downloaded completely before
> being streamed(which defeats the purpose of streams)
> or not? and if not how can I get the content of an URL
> gradually (without getting the whole body into
> memory)?

Yes, Convenience.http_get always downloads the whole body into a string.

For a stream-like download, you must use the pipeline interface. For
example, to store the body into a file do:

let call = new get "http://dfl.sjsu.edu" in
call # set_response_body_storage (`File (fun () -> "filename"));
let pipeline = new pipeline in
pipeline # set_proxy_from_environment();
pipeline # run()

Now check call#status whether the HTTP request was successfully
responded.

If you want to postprocess the body while it is downloaded, things will
get more complicated. In particular, it is possible to store the body in
an object that fulfils the Netmime.mime_body class type. Netclient
invokes the open_value_wr method and outputs into this netchannel. Using
this mechanism to define a callback-style interface (i.e. a function is
called whenever a chunk of data arrives) can be implemented in a
straight-forward manner.

If you really need a "string Stream.t": This is possible because
Netclient is programmed in an event-driven way, but is very difficult,
because you must arrange a so-called control inversion. This is
absolutely non-trivial but implementable in less than 100 lines of code.

Hope this helps,

Gerd

> 
> Thank you,
> JD
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Netclient

[Gerd Stolpmann]

Am Montag, den 13.03.2006, 13:38 +0100 schrieb Gerd Stolpmann:

Sorry, one statement is missing here:

> let call = new get "http://dfl.sjsu.edu" in
> call # set_response_body_storage (`File (fun () -> "filename"));
> let pipeline = new pipeline in
> pipeline # set_proxy_from_environment();

pipeline # add call;

> pipeline # run()

Gerd 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Netclient

[Jean-David]

Thank you for making things clear!
I will pipeline the URL to a hard disk file for now
even though it would be nicer to have a direct Stream
but 100 lines !? I thought we were talking dozens of
lines... hehe Thank you for the expert's insight
JD

--- Gerd Stolpmann <info@gerd-stolpmann.de> wrote:

> Am Montag, den 13.03.2006, 00:03 -0800 schrieb
> Jean-David:
> > Hello there,
> > I was wondering about the best way to create a
> stream
> > from a remote file for which I have the URL
> > 
> > open Http_client
> > open Stream
> > 
> > let body = Stream.of_string Convenience.http_get
> > "http://dfl.sjsu.edu";;
> > 
> > Is the content of body downloaded completely
> before
> > being streamed(which defeats the purpose of
> streams)
> > or not? and if not how can I get the content of an
> URL
> > gradually (without getting the whole body into
> > memory)?
> 
> Yes, Convenience.http_get always downloads the whole
> body into a string.
> 
> For a stream-like download, you must use the
> pipeline interface. For
> example, to store the body into a file do:
> 
> let call = new get "http://dfl.sjsu.edu" in
> call # set_response_body_storage (`File (fun () ->
> "filename"));
> let pipeline = new pipeline in
> pipeline # set_proxy_from_environment();
> pipeline # run()
> 
> Now check call#status whether the HTTP request was
> successfully
> responded.
> 
> If you want to postprocess the body while it is
> downloaded, things will
> get more complicated. In particular, it is possible
> to store the body in
> an object that fulfils the Netmime.mime_body class
> type. Netclient
> invokes the open_value_wr method and outputs into
> this netchannel. Using
> this mechanism to define a callback-style interface
> (i.e. a function is
> called whenever a chunk of data arrives) can be
> implemented in a
> straight-forward manner.
> 
> If you really need a "string Stream.t": This is
> possible because
> Netclient is programmed in an event-driven way, but
> is very difficult,
> because you must arrange a so-called control
> inversion. This is
> absolutely non-trivial but implementable in less
> than 100 lines of code.
> 
> Hope this helps,
> 
> Gerd
> 
> > 
> > Thank you,
> > JD
> > 
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> > http://mail.yahoo.com 
> > 
> > _______________________________________________
> > Caml-list mailing list. Subscription management:
> >
>
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> > Archives: http://caml.inria.fr
> > Beginner's list:
> http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> > 
> -- 
>
------------------------------------------------------------
> Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt *
> Germany 
> gerd@gerd-stolpmann.de         
> http://www.gerd-stolpmann.de
> Phone: +49-6151-153855                  Fax:
> +49-6151-997714
>
------------------------------------------------------------
> 
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com