[Caml-list] Ocaml-Weblib?

[Caml-list] Ocaml-Weblib?

[Oliver Bandel]

Hello,

is there a Library for parsing/analysing Webpages?

I want to write a tool, that helps me searching
on websites, and I don't want to rewrite such a library from scratch.

Is there something for Ocaml, similar (or even better :))
like Web-/CGI-Libs for Perl?

TIA,
   Oliver


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.08.21 20:49 schrieb(en) Oliver Bandel:
> Hello,
> 
> is there a Library for parsing/analysing Webpages?
> 
> I want to write a tool, that helps me searching
> on websites, and I don't want to rewrite such a library from scratch.
> 
> Is there something for Ocaml, similar (or even better :))
> like Web-/CGI-Libs for Perl?

You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
It includes an HTML parser, and CGI support. For HTTP operations, netclient
http://www.ocaml-programming.de/programming/netclient.html may be helpful.

Gerd
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

Hello Gerd,


On Thu, 22 Aug 2002, Gerd Stolpmann wrote:

> 
> Am 2002.08.21 20:49 schrieb(en) Oliver Bandel:
> > Hello,
> > 
> > is there a Library for parsing/analysing Webpages?
> > 
> > I want to write a tool, that helps me searching
> > on websites, and I don't want to rewrite such a library from scratch.
> > 
> > Is there something for Ocaml, similar (or even better :))
> > like Web-/CGI-Libs for Perl?
> 
> You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> It includes an HTML parser, and CGI support. For HTTP operations, netclient
> http://www.ocaml-programming.de/programming/netclient.html may be helpful.

Well I think it is designed to be used under bigger
programming environments (gtk is necessary).

Isn't there a lib, which does not require such GUI-stuff like
gtk?
Isn't there a pure lib for handling the networkstuff?

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.08.24 01:38 schrieb(en) Oliver Bandel:
> Hello Gerd,
> 
> 
> On Thu, 22 Aug 2002, Gerd Stolpmann wrote:
> 
> > 
> > Am 2002.08.21 20:49 schrieb(en) Oliver Bandel:
> > > Hello,
> > > 
> > > is there a Library for parsing/analysing Webpages?
> > > 
> > > I want to write a tool, that helps me searching
> > > on websites, and I don't want to rewrite such a library from scratch.
> > > 
> > > Is there something for Ocaml, similar (or even better :))
> > > like Web-/CGI-Libs for Perl?
> > 
> > You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> > It includes an HTML parser, and CGI support. For HTTP operations, netclient
> > http://www.ocaml-programming.de/programming/netclient.html may be helpful.
> 
> Well I think it is designed to be used under bigger
> programming environments (gtk is necessary).
> 
> Isn't there a lib, which does not require such GUI-stuff like
> gtk?
> Isn't there a pure lib for handling the networkstuff?

This is wrong. Neither ocamlnet nor netclient require gtk. (I really wonder
where you have found this; there is no published piece of code from me that
uses gtk.)

Of course, these libraries have requirements, mostly because of coding 
economy. I try to avoid to program functions twice, or to distribute the
same function twice under different names. (Nevertheless this happens, e.g.
there are parsers for mail (RFC822) messages in both ocamlnet and netclient.)
You are right that "bigger programming environments" profit more from this
approach, because code is shared. It makes it also simpler for me, the author,
because I need to maintain smaller amounts of code.

In particular, ocamlnet needs PCRE, and netclient needs (besides ocamlnet) 
equeue, and optionally xstr (for multi-threaded apps). A quite short 
dependency chain, IMHO. It might be possible that netclient and equeue are 
integrated into ocamlnet in the future, but a lot of consolidation is needed
first.

Gerd
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

Hi,

On Sat, 24 Aug 2002, Gerd Stolpmann wrote:

> 
> Am 2002.08.24 01:38 schrieb(en) Oliver Bandel:
> > Hello Gerd,
> > 
> > 
> > On Thu, 22 Aug 2002, Gerd Stolpmann wrote:
> > 
> > > 
> > > Am 2002.08.21 20:49 schrieb(en) Oliver Bandel:
> > > > Hello,
> > > > 
> > > > is there a Library for parsing/analysing Webpages?
> > > > 
> > > > I want to write a tool, that helps me searching
> > > > on websites, and I don't want to rewrite such a library from scratch.
> > > > 
> > > > Is there something for Ocaml, similar (or even better :))
> > > > like Web-/CGI-Libs for Perl?
> > > 
> > > You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> > > It includes an HTML parser, and CGI support. For HTTP operations, netclient
> > > http://www.ocaml-programming.de/programming/netclient.html may be helpful.
> > 
> > Well I think it is designed to be used under bigger
> > programming environments (gtk is necessary).
> > 
> > Isn't there a lib, which does not require such GUI-stuff like
> > gtk?
> > Isn't there a pure lib for handling the networkstuff?
> 
> This is wrong. Neither ocamlnet nor netclient require gtk. (I really wonder
> where you have found this; there is no published piece of code from me that
> uses gtk.)
[...]
> In particular, ocamlnet needs PCRE, and netclient needs (besides ocamlnet) 
> equeue, and optionally xstr (for multi-threaded apps). A quite short 
> dependency chain, IMHO. It might be possible that netclient and equeue are 
> integrated into ocamlnet in the future, but a lot of consolidation is needed
> first.

Ok, I have looked into a lib called netpath or so.
This was before I you send your mail; and I have
thought netpath was netclient.

I will look forward to use your libraries then.

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Sat, 24 Aug 2002, Gerd Stolpmann wrote:
[...]
> In particular, ocamlnet needs PCRE,

Hmhhh

configure says, that findlib is not installed (this is correct)
and that pcre is not installed (this is wrong).

My pcre.cmi is in directory /usr/local/lib/ocaml/contrib/pcre/
but it seems that it is not found.

Did I anything wrong during installation of pcre?
Is this an uncommon directory for pcre?
Or does the ocamlnet's configure look not there?

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Alain Frisch]

On Sun, 25 Aug 2002, Oliver Bandel wrote:

> configure says, that findlib is not installed (this is correct)
> and that pcre is not installed (this is wrong).
>
> My pcre.cmi is in directory /usr/local/lib/ocaml/contrib/pcre/
> but it seems that it is not found.

You have to install pcre with findlib, as stated in the src/INSTALL file
from the ocamlnet package ...

-- Alain

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Sun, 25 Aug 2002, Alain Frisch wrote:

> On Sun, 25 Aug 2002, Oliver Bandel wrote:
> 
> > configure says, that findlib is not installed (this is correct)
> > and that pcre is not installed (this is wrong).
> >
> > My pcre.cmi is in directory /usr/local/lib/ocaml/contrib/pcre/
> > but it seems that it is not found.
> 
> You have to install pcre with findlib, as stated in the src/INSTALL file
> from the ocamlnet package ...

Well I once have installed pcre as stand-alone.
So, I may now have to install it a second time?
(Or add some links?)

Confused,
      Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Stefano Zacchiroli]

On Sun, Aug 25, 2002 at 11:33:52PM +0200, Oliver Bandel wrote:
> Well I once have installed pcre as stand-alone.
> So, I may now have to install it a second time?
> (Or add some links?)

Ocamlnet "configure" script can use findlib or do a 'manual' search.

Seems to me that you doesn't have findlib, so configure script will look
for "pcre.cmi" in the standard ocaml directory (obtained with "ocamlc
-where") and fail because your pcre.cmi isn't there.

You can solve the configure problem adding your dir to the "incdirs"
variable in the configure script, but probably you will need to add some
more switches to reach the end of the compilation.

Obviously installing findlib seems to be easier :-)
Consider also that you can install findlib and not reinstall pcre, just
add your "/usr/local/lib/ocaml/contrib" to the findlib search path.

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Mon, 26 Aug 2002, Stefano Zacchiroli wrote:

> On Sun, Aug 25, 2002 at 11:33:52PM +0200, Oliver Bandel wrote:
> > Well I once have installed pcre as stand-alone.
> > So, I may now have to install it a second time?
> > (Or add some links?)
> 

Hmhhhh.

Ok, I now had installed the findlib.
And then the ocamlnet-configure-script detected both:
findlib and pcre.

But when typing "make" the result was, that pcre was'nt found.

What is the best installing directory for stuff like pcre,
which normally will be searched by default?

When directly using pcre from within Ocaml-Programs, I had
the ssame problem, that the pcre-stuff was not in the
default-search-path of the compilers.

Are there installation-driectories, which can be recommended
for such packagaes?

I reinstall pcre then (nice, that it has a "make uninstall")
in the better directory.

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Stefano Zacchiroli]

On Fri, Aug 30, 2002 at 06:16:02PM +0200, Oliver Bandel wrote:
> But when typing "make" the result was, that pcre was'nt found.

Uhm ... reporting the error message can help here, but anyway ...

The problem is that probably you installed pcre and thus the META file
that ocamlfind is looking for in a directory unknown to ocamlfind.
You can fix the problem changing the "path" setting in
/etc/ocamlfind.conf, just add to hit the directory _above_ the directory
where you installed pcre. (e.g. you have installed pcre in
/usr/local/lib/ocaml/pcre, then add /usr/local/lib/ocaml to the path
value in ocamlfind.conf; this value is a ":" separated list of dirs).

> What is the best installing directory for stuff like pcre,
> which normally will be searched by default?
> 
> When directly using pcre from within Ocaml-Programs, I had
> the ssame problem, that the pcre-stuff was not in the
> default-search-path of the compilers.

Usually is better to install them in a subdir of the ocaml library
directory (which you can get typing "ocamlc -where"). Don't know which
is your setup but on debian this directory is /usr/lib/ocaml and by
default from the ocaml sources I guess it can be /usr/local/lib/ocaml,
but I'm not sure.

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

Hello,

On Fri, 30 Aug 2002, Oliver Bandel wrote:

> On Mon, 26 Aug 2002, Stefano Zacchiroli wrote:
> 
> > On Sun, Aug 25, 2002 at 11:33:52PM +0200, Oliver Bandel wrote:
> > > Well I once have installed pcre as stand-alone.
> > > So, I may now have to install it a second time?
> > > (Or add some links?)
> > 
> 
> Hmhhhh.
> 
> Ok, I now had installed the findlib.
> And then the ocamlnet-configure-script detected both:
> findlib and pcre.
> 
> But when typing "make" the result was, that pcre was'nt found.


I fixed the problem.

I have now found the META-file in the pcre-src-directory,
added a pcre-directory beyond /usr/local/lib/ocaml/site-lib/
and copied the META-file into this pcre-directory.

Now findlib (ocamlfind) found the pcre and the ocamlnet-lib
could be compiled.

Next question: Is there any more detailed documentation somewhere?
Or do I have to browse the *.ml/*.mli-files?



Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Stefano Zacchiroli]

On Sun, Sep 01, 2002 at 06:45:22PM +0200, Oliver Bandel wrote:
> Next question: Is there any more detailed documentation somewhere?
> Or do I have to browse the *.ml/*.mli-files?

Documentation of what? PCRE or findlib?

Anyway, PCRE have a good .mli which is commented in ocamldoc style so
you can generate a good documentation using ocamldoc on it. The debian
package of pcre ships this documentation already build so you can unpack
the debian package and copy the /usr/share/doc/libpcre-ocaml-dev/html/
from it.

Regarding findlib it ships a good manual which is contained in the
original tarball IIRC.

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

Hello,

On Mon, 2 Sep 2002, Stefano Zacchiroli wrote:

> On Sun, Sep 01, 2002 at 06:45:22PM +0200, Oliver Bandel wrote:
> > Next question: Is there any more detailed documentation somewhere?
> > Or do I have to browse the *.ml/*.mli-files?
> 
> Documentation of what? PCRE or findlib?
> 
> Anyway, PCRE have a good .mli which is commented in ocamldoc style so
> you can generate a good documentation using ocamldoc on it. The debian
> package of pcre ships this documentation already build so you can unpack
> the debian package and copy the /usr/share/doc/libpcre-ocaml-dev/html/
> from it.
> 
> Regarding findlib it ships a good manual which is contained in the
> original tarball IIRC.

Yes, findlib has a quickstart and a doc. It seems to be good,
but I haven't read it (I may do it later, if I provide own modules).
What I meant was the ocamlnet-lib; I browsed a littltbid through
the *.mli-files. Hmhh, well there are comments. I hope they are
suficcient for me to understand usage. Or if there is a tutorial,
I would read it.

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Stefano Zacchiroli]

On Mon, Sep 02, 2002 at 12:47:00PM +0200, Oliver Bandel wrote:
> What I meant was the ocamlnet-lib; I browsed a littltbid through

Ocamlnet haven't yet a good documentation because, I suppose, it is
still a work in progress.
Anyway Gerd's sources and expecially .mli are often well commented and
should also be possible IIRC to build some kind of documentation from it
using an house made tool, anyway ask him for more information.

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.09.02 12:47 schrieb(en) Oliver Bandel:
> Hello,
> 
> On Mon, 2 Sep 2002, Stefano Zacchiroli wrote:
> 
> > On Sun, Sep 01, 2002 at 06:45:22PM +0200, Oliver Bandel wrote:
> > > Next question: Is there any more detailed documentation somewhere?
> > > Or do I have to browse the *.ml/*.mli-files?
> > 
> > Documentation of what? PCRE or findlib?
> > 
> > Anyway, PCRE have a good .mli which is commented in ocamldoc style so
> > you can generate a good documentation using ocamldoc on it. The debian
> > package of pcre ships this documentation already build so you can unpack
> > the debian package and copy the /usr/share/doc/libpcre-ocaml-dev/html/
> > from it.
> > 
> > Regarding findlib it ships a good manual which is contained in the
> > original tarball IIRC.
> 
> Yes, findlib has a quickstart and a doc. It seems to be good,
> but I haven't read it (I may do it later, if I provide own modules).
> What I meant was the ocamlnet-lib; I browsed a littltbid through
> the *.mli-files. Hmhh, well there are comments. I hope they are
> suficcient for me to understand usage. Or if there is a tutorial,
> I would read it.

No, I have not yet found the time to do it. There are some aspects
that cannot be explained in mli files well enough, and a tutorial
would be a great thing. E.g. why ocamlnet has an object-oriented
layer on top of channels (netchannels). You find them everywhere,
but no introduction.

There is an "examples" directory containing some very simple, and
some advanced examples, especially for CGI programming.

> Ciao,
>    Oliver
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> 
> 
> 

------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Tue, 3 Sep 2002, Gerd Stolpmann wrote:

[...]
> No, I have not yet found the time to do it. There are some aspects
> that cannot be explained in mli files well enough, and a tutorial
> would be a great thing. E.g. why ocamlnet has an object-oriented
> layer on top of channels (netchannels). You find them everywhere,
> but no introduction.
> 
> There is an "examples" directory containing some very simple, and
> some advanced examples, especially for CGI programming.

Yes, I have seen it.
The simple CGI-example is very nice. :)
Looks like easy programming. :)
The advanced examples are a littlebid too large for
Ocaml-and-ocamlnet-beginners. ;-)

I first asked, because I want to write two different
tools:

first one: a wget-like tool, which can parse the html-pages
(if possible, this f..... javascript-stuff too) and
can download not only recursively, but can also
select the pages for download e.g. by pattern-matching
on href-tags (url or text of the link) or by selection
of filesizes or so.

And the second tool I wanted to write was a similar tool
for nntp-protocol: Download by attributes (size, date,
MsgID, Subject, author, thread-length, ...).
(I once wrote such stuff (not completed) in Perl
 and after the program grew more and more, it
 becomes more and more a mess...).


So I need access to sockets, some low-level stuff
(Unix.read) and such, or a good library, which helps
here. I need a library, which can parse me the
html-pages and maybe nntp-headers, and I want only
to implement the logic of the tool, and let the
network stuff programming be the work, that the
lib can do.

And I hope the ocamlnet/netstring can help here.
But if it will be more effort to understand the library
than writing the networking-code by myself, then
I will write the sockets-stuff by myself.

That's all about it.

Ciao,
   Oliver

P.S.: Don't forget: I'm not Ocaml-user since years...
      ...since some months - and ONLY IF I HAD SOME
      TIME - could do some Ocaml-programming. So a
      documentation/tutorial would help a lot.

P.P.S.: IMHO OCaml is a very complex language.


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.09.03 01:24 schrieb(en) Oliver Bandel:
> On Tue, 3 Sep 2002, Gerd Stolpmann wrote:
> 
> [...]
> > No, I have not yet found the time to do it. There are some aspects
> > that cannot be explained in mli files well enough, and a tutorial
> > would be a great thing. E.g. why ocamlnet has an object-oriented
> > layer on top of channels (netchannels). You find them everywhere,
> > but no introduction.
> > 
> > There is an "examples" directory containing some very simple, and
> > some advanced examples, especially for CGI programming.
> 
> Yes, I have seen it.
> The simple CGI-example is very nice. :)
> Looks like easy programming. :)
> The advanced examples are a littlebid too large for
> Ocaml-and-ocamlnet-beginners. ;-)
> 
> I first asked, because I want to write two different
> tools:
> 
> first one: a wget-like tool, which can parse the html-pages
> (if possible, this f..... javascript-stuff too) and
> can download not only recursively, but can also
> select the pages for download e.g. by pattern-matching
> on href-tags (url or text of the link) or by selection
> of filesizes or so.

HTML parsing can be done with Nethtml. Simple example:

Nethtml.parse 
  (new Netchannels.input_string "<HTML><HEAD>...</HEAD><BODY>...</BODY></HTML>")

Returns something like

[ Element("html",[], [ Element("head",[], [ ... ]);
                       Element("body",[], [ ... ]) ) ]

just try it in the toploop.

If there are attributes (e.g. <BODY BGCOLOR="#EEFF54">) you get them
instead of [], e.g. Element("body",["bgcolor", "#EEFF54"], [ ... ]).
Note that all names are returned in lower-case.

If you want to read from a file instead of a string, just use

  new Netchannels.input_channel ch 

instead of input_string (where ch is an open in_channel).

There is no parser for javascript.

To download the HTML pages you can use netclient (distributed
separately). Simple example:

Http_client.Convenience.http_get "http://caml.inria.fr"

returns the contents of this location. If you need the HTTP headers
(sometimes announcing file sizes), you can use

let m = Http_client.Convenience.http_get_message "http://caml.inria.fr" in
let contents = m # get_resp_body () in
let size = m # assoc_resp_header "content-length" in ... (* may raise Not_found *)

If you use http_head_message, only the headers are requested from the server,
so you have the chance to determine the file size before downloading. In my
own experiments I found out that there are HTTP servers that handle HEAD like
GET causing protocol errors. So be prepared that you can get a strange exception
when you call http_head_message.
> 
> And the second tool I wanted to write was a similar tool
> for nntp-protocol: Download by attributes (size, date,
> MsgID, Subject, author, thread-length, ...).
> (I once wrote such stuff (not completed) in Perl
>  and after the program grew more and more, it
>  becomes more and more a mess...).

As far as I know there is no ready-to-use NNTP client. There are important
components for an NNTP client, though. For example, there are parsers for
messages in email format, and there is the working implementation for the
POP protocol that has some similarities.

To parse an email message, you can call Netmime.read_mime_message, e.g.

Netmime.read_mime_message
  (new Netchannels.input_string "subject: xxx\nsize: 50\n...\n\nbody")

This returns a structure like

  (header, `Body(body))

where "header" and "body" are objects:

  header # field "subject"    returns "xxx"
  header # field "size"       returns "50"
  body # value                returns "body"

Note that Netmime.read_mime_message decodes multipart messages by default,
and you can also get something like

  (header, `Parts [ (part1_header, `Body part1_body); ... (partN_header, `Body partN_body)]

or even deeper nested structures. You can control this by passing the argument
multipart_style.

There is also the function Netmime.read_mime_header returning only the
header, but it is a bit more complicated to use. To parse a string:

let header = Netmime.read_mime_header 
               (new Netstream.input_stream
                  (new Netchannels.input_string "..."))

There is another object involved (input_stream) that has no effect if you
read only from a string, but that allows you to read the header from 
non-seekable files (e.g. pipelines or sockets). But this is definitely
a feature for experts.

> 
> So I need access to sockets, some low-level stuff
> (Unix.read) and such, or a good library, which helps
> here. 

See the sources in netpop.ml for an example how to write a "telnet-style"
client. Note that netpop.ml does not use sockets, it expects that the
user of this module passes channels that are already connected sockets.
See the example mbox_list.ml for the socket stuff (very simple).

>I need a library, which can parse me the
> html-pages and maybe nntp-headers, and I want only
> to implement the logic of the tool, and let the
> network stuff programming be the work, that the
> lib can do.
> 
> And I hope the ocamlnet/netstring can help here.
> But if it will be more effort to understand the library
> than writing the networking-code by myself, then
> I will write the sockets-stuff by myself.

I hope this short introduction gives you the right impression of
the library.

Gerd
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Tue, 3 Sep 2002, Gerd Stolpmann wrote:

> 
> Am 2002.09.03 01:24 schrieb(en) Oliver Bandel:
> > On Tue, 3 Sep 2002, Gerd Stolpmann wrote:
[...]

> HTML parsing can be done with Nethtml. Simple example:
> 
> Nethtml.parse 
>   (new Netchannels.input_string "<HTML><HEAD>...</HEAD><BODY>...</BODY></HTML>")
> 
> Returns something like
> 
> [ Element("html",[], [ Element("head",[], [ ... ]);
>                        Element("body",[], [ ... ]) ) ]
> 
> just try it in the toploop.
[...]

OK, done. Is good... very good.
Yields expected results similar to the Perl-Libs for
such work; but it's soooo much better here, because it's
cleaner ( typesafety and clearer structure than the Perl-stuff :) ).

So, this is, what I have looked for. :)

Are there "opposite" tools, to out back the datastructure,
which will get here from html-parsing back into a html-file?
I think about reading a html-file, throwing out unnecessary
contents, adding interesting stuff, re-layouting the page
and create new pages, better for viewing as output.

(Often webpages are messy (called "good designed";-)) and
 I only need two or three parts of it. Creating a pagelaout,
 which matches my needs is intended here.)

Can this be done by the cgi-module?
Or is it part of the nethtml-lib?


[...]
> As far as I know there is no ready-to-use NNTP client. There are important
> components for an NNTP client, though. For example, there are parsers for
> messages in email format, and there is the working implementation for the
> POP protocol that has some similarities.

OK, if I can rely on the ocamlnet's sub-libraries, then I may write
such a client based on it.

[...]
> There is another object involved (input_stream) that has no effect if you
> read only from a string, but that allows you to read the header from 
> non-seekable files (e.g. pipelines or sockets). But this is definitely
> a feature for experts.

Well, I have done socket-programming .... in C. :)


[...]
> See the sources in netpop.ml for an example how to write a "telnet-style"
> client. Note that netpop.ml does not use sockets, it expects that the
> user of this module passes channels that are already connected sockets.

Hmhhh... how can be sockets read like channels?
Sockets must be handled unbuffered... Unix.read and such
functions... hmhhh is this done by Stream-module or similar
things? (Or could it be done in this way?)


[...]
> I hope this short introduction gives you the right impression of
> the library.

Yes. It's powerful (and wonderful:)).

With each more of impression of Ocaml's features (e.g.
module system) and such libraries, I see, that my interest
in perl and other languages fades away....

Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.08.25 23:06 schrieb(en) Alain Frisch:
> On Sun, 25 Aug 2002, Oliver Bandel wrote:
> 
> > configure says, that findlib is not installed (this is correct)
> > and that pcre is not installed (this is wrong).
> >
> > My pcre.cmi is in directory /usr/local/lib/ocaml/contrib/pcre/
> > but it seems that it is not found.
> 
> You have to install pcre with findlib, as stated in the src/INSTALL file
> from the ocamlnet package ...

This is the recommended way. Alternatively, ocamlnet can be installed without
findlib (configure -disable-findlib -I /where/to/find/pcre) but you will
certainly run into problems when you try to install packages depending on
ocamlnet.

Gerd

------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Gerd Stolpmann]

Am 2002.08.25 21:55 schrieb(en) Oliver Bandel:
> 
> On Sat, 24 Aug 2002, Gerd Stolpmann wrote:
> [...]
> > In particular, ocamlnet needs PCRE,
> 
> Hmhhh
> 
> configure says, that findlib is not installed (this is correct)
> and that pcre is not installed (this is wrong).
> 
> My pcre.cmi is in directory /usr/local/lib/ocaml/contrib/pcre/
> but it seems that it is not found.
> 
> Did I anything wrong during installation of pcre?
> Is this an uncommon directory for pcre?
> Or does the ocamlnet's configure look not there?

It looks only at places where it has been told to do so: 
configure -I <directory>

There is no automatic search.

Gerd

------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Dmitry Bely]

Gerd Stolpmann <info@gerd-stolpmann.de> writes:

> You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> It includes an HTML parser, and CGI support. For HTTP operations, netclient
> http://www.ocaml-programming.de/programming/netclient.html may be
> helpful.

BTW, is there any FTP client, written in Ocaml?

- Dmitry Bely


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Alan Schmitt]

* Dmitry Bely (dbely@mail.ru) wrote:
> Gerd Stolpmann <info@gerd-stolpmann.de> writes:
> 
> > You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> > It includes an HTML parser, and CGI support. For HTTP operations, netclient
> > http://www.ocaml-programming.de/programming/netclient.html may be
> > helpful.
> 
> BTW, is there any FTP client, written in Ocaml?

There is a simple ftp client library in cdk, that I'm thinking of
pulling out and make independent (I wrote this library initially for
camlgrenouille, then integrated it with the cdk ... as camlgrenouille
might have a new release soon, and the cdk does not seem to be going
anywhere, that's why I'll des-integrate it ;-)

So I guess you could write a simple text interface on top of this
library (that let you do active and passive put and get, cd, delete, and
size).

Alan

-- 
The hacker: someone who figured things out and made something cool happen.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Oliver Bandel]

On Fri, 30 Aug 2002, Alan Schmitt wrote:

> * Dmitry Bely (dbely@mail.ru) wrote:
> > Gerd Stolpmann <info@gerd-stolpmann.de> writes:
> > 
> > > You may have a look at ocamlnet, http://sourceforge.net/projects/ocamlnet.
> > > It includes an HTML parser, and CGI support. For HTTP operations, netclient
> > > http://www.ocaml-programming.de/programming/netclient.html may be
> > > helpful.
> > 
> > BTW, is there any FTP client, written in Ocaml?
> 
> There is a simple ftp client library in cdk, that I'm thinking of


cdk?

Is this the *big* library, what I found a lot of months ago
and didn't find again since then?


Ciao,
   Oliver

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Re: Ocaml-Weblib?

[Michaël Grünewald]

Oliver Bandel <oliver@first.in-berlin.de> writes:


[...]

> cdk?
> 
> Is this the *big* library, what I found a lot of months ago
> and didn't find again since then?

You got it back:
http://pauillac.inria.fr/cdk/

What a lucky day!
-- 
Michaël Grünewald <michael-grunewald@wanadoo.fr>  - RSA PGP Key ID: 0x20D90C12
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Alan Schmitt]

* Oliver Bandel (oliver@first.in-berlin.de) wrote:
> cdk?
> 
> Is this the *big* library, what I found a lot of months ago
> and didn't find again since then?

Well, it's still at http://pauillac.inria.fr/cdk/

Alan

-- 
The hacker: someone who figured things out and made something cool happen.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Dmitry Bely]

Alan Schmitt <alan.schmitt@inria.fr> writes:

>> BTW, is there any FTP client, written in Ocaml?
>
> There is a simple ftp client library in cdk, that I'm thinking of
> pulling out and make independent (I wrote this library initially for
> camlgrenouille, then integrated it with the cdk ... as camlgrenouille
> might have a new release soon, and the cdk does not seem to be going
> anywhere, that's why I'll des-integrate it ;-)

Hmm, I cannot find it in CDK CVS. Could you tell me the exact place where
it is supposed to be present?

> So I guess you could write a simple text interface on top of this
> library (that let you do active and passive put and get, cd, delete, and
> size).

That's enough. The library (not stand-alone client) is what I need.

- Dmitry Bely


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Alan Schmitt]

* Dmitry Bely (dbely@mail.ru) wrote:
> Alan Schmitt <alan.schmitt@inria.fr> writes:
> 
> >> BTW, is there any FTP client, written in Ocaml?
> >
> > There is a simple ftp client library in cdk, that I'm thinking of
> > pulling out and make independent (I wrote this library initially for
> > camlgrenouille, then integrated it with the cdk ... as camlgrenouille
> > might have a new release soon, and the cdk does not seem to be going
> > anywhere, that's why I'll des-integrate it ;-)
> 
> Hmm, I cannot find it in CDK CVS. Could you tell me the exact place where
> it is supposed to be present?
> 

http://camlcvs.inria.fr/cgi-bin/cvsweb.cgi/cdk/
It is between "cash" and "geneweb" on the public caml cvs repository.

Alan Schmitt


-- 
The hacker: someone who figured things out and made something cool happen.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Dmitry Bely]

Alan Schmitt <alan.schmitt@inria.fr> writes:

>> >> BTW, is there any FTP client, written in Ocaml?
>> >
>> > There is a simple ftp client library in cdk, that I'm thinking of
>> > pulling out and make independent (I wrote this library initially for
>> > camlgrenouille, then integrated it with the cdk ... as camlgrenouille
>> > might have a new release soon, and the cdk does not seem to be going
>> > anywhere, that's why I'll des-integrate it ;-)
>> 
>> Hmm, I cannot find it in CDK CVS. Could you tell me the exact place where
>> it is supposed to be present?
>> 
> http://camlcvs.inria.fr/cgi-bin/cvsweb.cgi/cdk/
> It is between "cash" and "geneweb" on the public caml cvs repository.

I realize that :-) Where is you library _inside_ cdk and how it is called?
I don't like to download all cdk for a couple of files...

- Dmitry Bely


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml-Weblib?

[Alan Schmitt]

* Dmitry Bely (dbely@mail.ru) wrote:
> Alan Schmitt <alan.schmitt@inria.fr> writes:
> 
> >> >> BTW, is there any FTP client, written in Ocaml?
> >> >
> >> > There is a simple ftp client library in cdk, that I'm thinking of
> >> > pulling out and make independent (I wrote this library initially for
> >> > camlgrenouille, then integrated it with the cdk ... as camlgrenouille
> >> > might have a new release soon, and the cdk does not seem to be going
> >> > anywhere, that's why I'll des-integrate it ;-)
> >> 
> >> Hmm, I cannot find it in CDK CVS. Could you tell me the exact place where
> >> it is supposed to be present?
> >> 
> > http://camlcvs.inria.fr/cgi-bin/cvsweb.cgi/cdk/
> > It is between "cash" and "geneweb" on the public caml cvs repository.
> 
> I realize that :-) Where is you library _inside_ cdk and how it is called?
> I don't like to download all cdk for a couple of files...

Oops, sorry. It's in network, the module is called Ftp. It depends on
Abort, Debut, Timeout (all in the same directory) and unfortunately also
on Unix2 (for the function getdomainname). I just had a quick look, and
that should be it.

Best,

Alan

-- 
The hacker: someone who figured things out and made something cool happen.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners