[Caml-list] https... which lib?

[Caml-list] https... which lib?

[oliver]

Hi,

AFAIK I can't use ocamlnet for https-files?!

Which lib do I need for that?

(Or does the new ocamlnet provide this functionality?)


Ciao,
   Oliver


-- 
  Dec. 25th:
    http://stallman.org/grav-mass.html

Re: [Caml-list] https... which lib?

[Török Edwin]

On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> Hi,
> 
> AFAIK I can't use ocamlnet for https-files?!
> 
> Which lib do I need for that?

I used libcurl-ocaml for https recently.

> 
> (Or does the new ocamlnet provide this functionality?)



Best regards,
--Edwin

Re: [Caml-list] https... which lib?

[oliver]

On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> > Hi,
> > 
> > AFAIK I can't use ocamlnet for https-files?!
> > 
> > Which lib do I need for that?
> 
> I used libcurl-ocaml for https recently.
[...]

OK, thanks.
I think I should try it.

Ciao,
   Oliver

Re: [Caml-list] https... which lib?

[oliver]

On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> > Hi,
> > 
> > AFAIK I can't use ocamlnet for https-files?!
> > 
> > Which lib do I need for that?
> 
> I used libcurl-ocaml for https recently.
[...]

http://curl.haxx.se/libcurl/ocaml/

Has a simple easy-to-start example...
... and https works out of the box.


So, thanks again for pointing me to that lib. :)


Ciao,
   Oliver

Re: [Caml-list] https... which lib?

[oliver]

On Sat, Dec 25, 2010 at 07:30:21PM +0100, oliver@first.in-berlin.de wrote:
> On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
> > On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> > > Hi,
> > > 
> > > AFAIK I can't use ocamlnet for https-files?!
> > > 
> > > Which lib do I need for that?
> > 
> > I used libcurl-ocaml for https recently.
> [...]
> 
> http://curl.haxx.se/libcurl/ocaml/
> 
> Has a simple easy-to-start example...
> ... and https works out of the box.
[...]

But documentation lacks....

The functions seem not to have the same names as the C-counterparts...

What function do I need for not getting the webpage printed to stdout,
but into a string?


Ciao,
   Oliver

Re: [Caml-list] https... which lib?

[Török Edwin]

On 2010-12-25 22:21, oliver@first.in-berlin.de wrote:
> On Sat, Dec 25, 2010 at 07:30:21PM +0100, oliver@first.in-berlin.de wrote:
>> On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
>>> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
>>>> Hi,
>>>>
>>>> AFAIK I can't use ocamlnet for https-files?!
>>>>
>>>> Which lib do I need for that?
>>>
>>> I used libcurl-ocaml for https recently.
>> [...]
>>
>> http://curl.haxx.se/libcurl/ocaml/
>>
>> Has a simple easy-to-start example...
>> ... and https works out of the box.
> [...]
> 
> But documentation lacks....
> 
> The functions seem not to have the same names as the C-counterparts...
> 
> What function do I need for not getting the webpage printed to stdout,
> but into a string?

Here's a snippet from my code:

let new_curl_connection () =
    let buffer = Buffer.create 4096
    and connection = new Curl.handle in
    let write_function str =
        Buffer.add_string buffer str;
        String.length str;
    in
    connection#set_writefunction write_function;
    connection#set_useragent Config.user_agent;
    connection#set_timeout Config.timeout;
    connection#set_followlocation true;
    connection#set_connecttimeout Config.connect_timeout;
(*  connection#set_verbose true;*)
    (connection, buffer);;

let _ =
	let conn, buffer = new_curl_connection() in
	conn#set_url url;
	conn#perform;
	let http_code = conn#get_responsecode in
	if http_code < 400 then
		let result = Buffer.contents buffer;
....

Also since you're using HTTPS you might want this:
            sslconn#set_sslverifypeer true;
            (* verify that certificate is valid for target host *)
            sslconn#set_sslverifyhost Curl.SSLVERIFYHOST_HOSTNAME;
            (* only accept our servers, optional *)
            sslconn#set_cainfo "ourserver_ca.pem";

Re: [Caml-list] https... which lib?

[oliver]

On Sat, Dec 25, 2010 at 10:26:08PM +0200, Török Edwin wrote:
> On 2010-12-25 22:21, oliver@first.in-berlin.de wrote:
> > On Sat, Dec 25, 2010 at 07:30:21PM +0100, oliver@first.in-berlin.de wrote:
> >> On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
> >>> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> >>>> Hi,
> >>>>
> >>>> AFAIK I can't use ocamlnet for https-files?!
> >>>>
> >>>> Which lib do I need for that?
> >>>
> >>> I used libcurl-ocaml for https recently.
> >> [...]
> >>
> >> http://curl.haxx.se/libcurl/ocaml/
> >>
> >> Has a simple easy-to-start example...
> >> ... and https works out of the box.
> > [...]
> > 
> > But documentation lacks....
> > 
> > The functions seem not to have the same names as the C-counterparts...
> > 
> > What function do I need for not getting the webpage printed to stdout,
> > but into a string?
> 
> Here's a snippet from my code:
> 
> let new_curl_connection () =
>     let buffer = Buffer.create 4096
>     and connection = new Curl.handle in
>     let write_function str =
>         Buffer.add_string buffer str;
>         String.length str;
[...]

OK, that's nice.


Thanks a lot again. :)

Ciao,
   Oliver

Re: [Caml-list] https... which lib?

[oliver]

Hi,

some more questions...

On Sat, Dec 25, 2010 at 10:26:08PM +0200, Török Edwin wrote:
> On 2010-12-25 22:21, oliver@first.in-berlin.de wrote:
> > On Sat, Dec 25, 2010 at 07:30:21PM +0100, oliver@first.in-berlin.de wrote:
> >> On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
> >>> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
> >>>> Hi,
> >>>>
> >>>> AFAIK I can't use ocamlnet for https-files?!
> >>>>
> >>>> Which lib do I need for that?
> >>>
> >>> I used libcurl-ocaml for https recently.
> >> [...]
> >>
> >> http://curl.haxx.se/libcurl/ocaml/
> >>
> >> Has a simple easy-to-start example...
> >> ... and https works out of the box.
> > [...]
> > 
> > But documentation lacks....
> > 
> > The functions seem not to have the same names as the C-counterparts...
> > 
> > What function do I need for not getting the webpage printed to stdout,
> > but into a string?
> 
[...]


>     connection#set_timeout Config.timeout;
>     connection#set_followlocation true;
>     connection#set_connecttimeout Config.connect_timeout;
[...]


The   method  set_timeout           and
the   method  set_connecttimeout,
what are they for and which value do they use?

I would assume to set one timeout, not two.

Is it in milli-seconds?


btw: I didn't found the docs for the OCaml-bindings of above mentioned methods.

Also in the curl-lib-binding it's mentioned that only the "easy-interface" is bound,
but the OCaml-bindings offer much more functions than those some of the easy-interface
(http://curl.haxx.se/libcurl/c/libcurl-easy.html => 26 functions, the mli contains a lot more) .
So, somehow at some place the docs seem to be outdated (?).
If you have a link to a more up-to-date documentatiopn, if this exists,
could you sent me a link to that?!

Ciao,
   Oliver

P.S.: Does the   method  set_followlocation
      set, if that curllib-connection is follwoing a redirect?

Re: [Caml-list] https... which lib?

[Török Edwin]

On 2010-12-26 01:10, oliver@first.in-berlin.de wrote:
> Hi,
> 
> some more questions...
> 
> On Sat, Dec 25, 2010 at 10:26:08PM +0200, Török Edwin wrote:
>> On 2010-12-25 22:21, oliver@first.in-berlin.de wrote:
>>> On Sat, Dec 25, 2010 at 07:30:21PM +0100, oliver@first.in-berlin.de wrote:
>>>> On Sat, Dec 25, 2010 at 10:07:16AM +0200, Török Edwin wrote:
>>>>> On 2010-12-25 02:44, oliver@first.in-berlin.de wrote:
>>>>>> Hi,
>>>>>>
>>>>>> AFAIK I can't use ocamlnet for https-files?!
>>>>>>
>>>>>> Which lib do I need for that?
>>>>>
>>>>> I used libcurl-ocaml for https recently.
>>>> [...]
>>>>
>>>> http://curl.haxx.se/libcurl/ocaml/
>>>>
>>>> Has a simple easy-to-start example...
>>>> ... and https works out of the box.
>>> [...]
>>>
>>> But documentation lacks....
>>>
>>> The functions seem not to have the same names as the C-counterparts...
>>>
>>> What function do I need for not getting the webpage printed to stdout,
>>> but into a string?
>>
> [...]
> 
> 
>>     connection#set_timeout Config.timeout;
>>     connection#set_followlocation true;
>>     connection#set_connecttimeout Config.connect_timeout;
> [...]
> 
> 
> The   method  set_timeout           and

See CURLOPT_TIMEOUT, I think it is timeout for entire transfer,
including DNS lookup.

> the   method  set_connecttimeout,

CURLOPT_CONNECTTIMEOUT, timeout for connect().

> what are they for and which value do they use?
> 
> I would assume to set one timeout, not two.
> 
> Is it in milli-seconds?

No, it is in seconds. See CURLOPT_TIMEOUT, if you want miliseconds use
CURLOPT_TIMEOUT_MS.

BTW I remember some curl versions having trouble with multiple threads
and timeouts, not sure if that is still the case but it is something to
keep in mind if your OCaml app is multithreaded.

> 
> 
> btw: I didn't found the docs for the OCaml-bindings of above mentioned methods.
> 
> Also in the curl-lib-binding it's mentioned that only the "easy-interface" is bound,
> but the OCaml-bindings offer much more functions than those some of the easy-interface
> (http://curl.haxx.se/libcurl/c/libcurl-easy.html => 26 functions, the mli contains a lot more) .
> So, somehow at some place the docs seem to be outdated (?).
> If you have a link to a more up-to-date documentatiopn, if this exists,
> could you sent me a link to that?!

All I have are the ocamldoc-generated ones, which don't say what each
method does, but it is not hard to guess.
For example: 'val set_timeout : t -> int -> unit' is just a convenience
function for 'setopt h CURLOPT_TIMEOUT(..)', that is curl_easy_setopt(..
CURLOPT_TIMEOUT ...). So read the docs for CURLOPT_TIMEOUT.

Similarly each function also has an OO equivalent.

> 
> Ciao,
>    Oliver
> 
> P.S.: Does the   method  set_followlocation
>       set, if that curllib-connection is follwoing a redirect?

Yes, it doesn't do that by default.

Best regards,
--Edwin

Re: [Caml-list] https... which lib?

[oliver]

On Sat, Dec 25, 2010 at 10:26:08PM +0200, Török Edwin wrote:
[...]
> Also since you're using HTTPS you might want this:
>             sslconn#set_sslverifypeer true;
>             (* verify that certificate is valid for target host *)
>             sslconn#set_sslverifyhost Curl.SSLVERIFYHOST_HOSTNAME;
>             (* only accept our servers, optional *)
>             sslconn#set_cainfo "ourserver_ca.pem";

OK.... can I also disable all these checks?
I got a
  Fatal error: exception Curl.CurlException(60, 60, "CURLE_SSL_CACERT")
on a http-page, and the tool I'm working on is simple downloading stuff,
and it doesn't matter if the docs are coming from the 55.th planet of the solar system or
from somewhere else... ;)
As far as I remember, my ocamlnet stuff could just download this stuff...

So, somehow just being able to read and download would be fine.

Btw, when I use the "simple" example for the webpage that fails,
I get an error, which tells me, that I'm not allowed to access that page,
but wget and Browsers display me the page without problems.

This is strange... it looks like I have to identify?!
This problem only occurs with curllib (with the code I sused from
your example, as welklk as the simple.ml from the curl-ocaml-binding-page).


Also a problem I have: the curl-lib seems to retrieve the data
in background as default, without me setting options for background working.
Can I disable this feature? I need to know whe the stuff access is ready.

Even my normal http-requests now fail, and in this respect it's a step backward
from ocamlnet (which also is better documented)...

A retry-option also would be fine...


Ciao,
   Oliver

Re: [Caml-list] https... which lib?

[oliver]

On Sun, Dec 26, 2010 at 12:38:08AM +0100, oliver@first.in-berlin.de wrote:
> On Sat, Dec 25, 2010 at 10:26:08PM +0200, Török Edwin wrote:
> [...]
> > Also since you're using HTTPS you might want this:
> >             sslconn#set_sslverifypeer true;
> >             (* verify that certificate is valid for target host *)
> >             sslconn#set_sslverifyhost Curl.SSLVERIFYHOST_HOSTNAME;
> >             (* only accept our servers, optional *)
> >             sslconn#set_cainfo "ourserver_ca.pem";
> 
> OK.... can I also disable all these checks?
[...]

Solution:

   sslconn#set_sslverifypeer false;


This makes me happy again. :)


Regarding the background-working, which I mourned on: this was nonsense.
I just had not enabled my printings which show me the current action ;)

So.... as a compensation for flooding this list (and thank for your help)
I may publish the tool, when it's ready... 


Ciao,
   Oliver

P.S.: ...now I should go sleeping, not again being up until 9 am ... ;)