[Caml-list] Web Development with OCaml

[Caml-list] Web Development with OCaml

[Jimmie Houchin]

Hello,

I am new to OCaml and Functional programming in general. I am building a
website and would like to know if OCaml is being used for web development.

OCaml has some very intriguing benefits and I am in the process of learning.

I am also considering Erlang for web development.
With Erlang it is a pretty clear path for web development.

I haven't read into that part of the Erlang documentation but has anyone used
OCaml to write any extensions for Erlang?

It would seem to be an interesting combination. Maybe it's just me. :)
Instead of dropping into C drop into OCaml. :)

Any help or pointers greatly appreciated.

Jimmie Houchin
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jean-Christophe Filliatre]

Jimmie Houchin writes:
 > 
 > I am new to OCaml and Functional programming in general. I am building a
 > website and would like to know if OCaml is being used for web development.

The Caml Hump is already pointing at several web tools in/for ocaml:

    http://caml.inria.fr/hump.html#web

-- 
Jean-Christophe Filliatre
  mailto:Jean-Christophe.Filliatre@lri.fr
  http://www.lri.fr/~filliatr
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

[Caml-list] Web Development with OCaml

[Tom Hirschowitz]

Hi. You may have a look at the "making of" Alain Frisch's page.

http://www.eleves.ens.fr:8080/home/frisch/realis.html
 
Jimmie Houchin writes:
 > Hello,
 > 
 > I am new to OCaml and Functional programming in general. I am building a
 > website and would like to know if OCaml is being used for web development.
 > 
 > OCaml has some very intriguing benefits and I am in the process of learning.
 > 
 > I am also considering Erlang for web development.
 > With Erlang it is a pretty clear path for web development.
 > 
 > I haven't read into that part of the Erlang documentation but has anyone used
 > OCaml to write any extensions for Erlang?
 > 
 > It would seem to be an interesting combination. Maybe it's just me. :)
 > Instead of dropping into C drop into OCaml. :)
 > 
 > Any help or pointers greatly appreciated.
 > 
 > Jimmie Houchin
 > -------------------
 > Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
 > To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr
 > 
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[David Mentre]

Tom Hirschowitz <Tom.Hirschowitz@inria.fr> writes:

> http://www.eleves.ens.fr:8080/home/frisch/realis.html

Unfortunately, this page is an french (ok, not a problem for me ;) and
the caml source is not available.

d.
-- 
 David.Mentre@inria.fr
 Opinions expressed here are only mine.
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Alain Frisch]

On Tue, 10 Jul 2001, Tom Hirschowitz wrote:

> 
> Hi. You may have a look at the "making of" Alain Frisch's page.
> 
> http://www.eleves.ens.fr:8080/home/frisch/realis.html

Well, this is largely out-dated, and is only about producing static
html pages. The method also works for dynamic content, but the issues
are quite different (Jimmie asked anout "web development", which involves
issues as data persistency, query to databases, application structure
to handle connections; html layout of produced pages is not a central
problem).

I've used several different system to build my homepage. The idea was
always to factor out from the documents what can be factorized (design of
the site, automatic generation of indexes).  Now I use my HereDoc
preprocessor (http://www.eleves.ens.fr:8080/home/frisch/soft), which has
system of templates (documents with holes that can be filled with the
result of Caml expressions); HereDoc has nothing specific to html. The
source for my homepage can be found there:  
http://www.eleves.ens.fr:8080/home/frisch/source


As I said above, web development is more than producing html pages.
OCaml has already most of the library needed:
- to process CGI arguments: Netstring's Cgi module does the job
- to handle connection to databases, there are several bindings
  to access PostgreSQL, MySQL, Oracle,.. databases
- XML parser: Pxp
- string manipulation and regular expressions à la Perl: Str, Pcre
- this may be useful too: http://www.eleves.ens.fr:8080/home/mine/ocamlhtml
- the OCaml Link Database is a good example of web application written
  is OCaml

In addition to my static homepage, I use OCaml for almost all the dynamic
content (and automatic generation of indexes) for the web server I
administrate (www.eleves.ens.fr:8080). I also used OCaml to prototype
quickly a semi-industrial web site which featured, among other things,
many different CGI forms, cookies, connections, upload and validation of
XML document, dynamic html generation from templates. It was a real
pleasure to use OCaml, and made it possible to develop all these things in
a few hours (the main, actually the only, problem was the lack of OCaml
developpers to maintain the code).

So yes, I believe that OCaml is a great tool for web development.
It would be great to have some integration between Cgi,
regexp manipulation, document generation, and a general application
framework dealing with persistency and database connections.
With Camlp4, it should be possible to define a comfortable 
semi-specialized syntax. Does anyone have the experience to design
such a framework, and want to launch a project ?


> Jimmie Houchin writes:
>  > Hello,
>  > 
>  > I am new to OCaml and Functional programming in general. I am building a
>  > website and would like to know if OCaml is being used for web development.



Alain Frisch

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jimmie Houchin]

First, I would like to thank all who replied.

Your replies encouraged me to start printing the documentation and start
learning OCaml. :)
I am also working my way thru the archives of the mailing list I've
downloaded.
Thanks for making them available. That is a great resource.

I have been keeping an eye on OCaml because of messages in
comp.lang.functional, the ICFP (?) competition results and then more recently
the Bagley Shootout results. 

The speed and memory results in the shootout are amazing. But what compounds
it is the LOC result. To me that indicates that a language does not have to be
verbose to be fast. 


My thoughts on web development (currently) :) and what brought me to OCaml.

I've been working with Python/Zope for web development. But I've been giving a
lot of thought to the architecture I want for my website.

I, like many entrepreneurs, have great hopes for my website. Because of this I
want my website to be fast, user-friendly and scalable.

Currently fast web servers like Tux 10-15,000+ rps (requests per second),
Apache 2-4,000 rps aren't dynamic (at those speeds). Dynamic website tools
like Zope, etc are either like 20-80 rps or some I've seen up to 100-200 rps.
This is a great disparity.

I want a dynamic website which gets as close to static speeds as possible.

Several ways to approach.

1. Tux. Use Tux as the front end webserver.
       Allow Tux to do as much serving as possible.
       Images, static content, etc.

2. Apache. Put Apache behind Tux.
       Use Apache, FastCGI, mod_ocaml(?) or some such
       for generating dynamic content.

3. Tux module. Create an OCaml user space module for Tux.
       This would be interesting and would probably be very fast.
       This is also beyond my skill.

4. OCaml CGI. Use traditional CGI with OCaml behind Tux.
       If OCaml's cgi spawns very fast or I guess is 
       preforked with a pool of processes. This could be pretty fast.

5. An OCaml Web Server. I don't know what's available here yet.
       I haven't had the time to research yet.
       This could be a very interesting option, especially if it's fast.
       This could easily be an excellent option to put behind Tux.
       Tux serves number 1. above and passes all dynamic services to this
server.

6. OCaml web app server.
       If all data was in a database for persistence and most pages were
templates,
       then many of the pages could be dynamically generated static pages.
       If most of the page is static then what options are available to allow
       the fastest server serve the static data and either the server Tux(?)
or Apache 
       request the dynamic data via SSI or the client request it via
Javascript.
       I do generally prefer the server-side option. I generally don't like
requiring 
       anything client-side, but if necessary or advantageous will take
advantage 
       of those who allow/permit it.

I believe there is an oportunity here to bring speed to dynamic websites with
OCaml.

Most people don't want to program websites in C. 
Perl, Python, et al don't compare speedwise.
Performance does matter.
Scalability does matter.
OCaml seems like a great tool to build a fast, developer friendly web app
server(?)/toolkit.

Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
I haven't a clue on how to develop either but could possibly learn. :)
After learning OCaml (at least some) first.

Those who are using OCaml for web development. 
Are you doing traditional CGI behind Apache?

I have just read a babelfish version of Alain's page. Not perfect but
understandable.
We seem to have some common concepts/beliefs.

I hope some of what I wrote above is understandable, hopefully logical.
Maybe I'm just crazy. :)
Hopefully as I come up to speed with OCaml and what's available, I'll be more
coherent and rational in my questions or statements.

Any comments, pointers, wisdom is greatly appreciated.

Thanks again.

Jimmie Houchin
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Alexander V. Voinov]

Hi

Jimmie Houchin wrote:
> The speed and memory results in the shootout are amazing. But what compounds
> it is the LOC result. To me that indicates that a language does not have to be
> verbose to be fast.
...
> Currently fast web servers like Tux 10-15,000+ rps (requests per second),
> Apache 2-4,000 rps aren't dynamic (at those speeds). Dynamic website tools
> like Zope, etc are either like 20-80 rps or some I've seen up to 100-200 rps.
> This is a great disparity.
> 
> I want a dynamic website which gets as close to static speeds as possible.

> 3. Tux module. Create an OCaml user space module for Tux.
>       This would be interesting and would probably be very fast.
>       This is also beyond my skill.

I'd like to seriously wonder about the impact of garbage collection in
such a persistent module, which has to reclaim stuff time to time. And
given that there will be many linked (recursive) data structures, the
process of reclaiming would take time. Unless it may be run as a
separate thread on a separate CPU, and the process of getting space for
new stuff doesn't heavily depend on the collection of the old one (say,
when it's not a problem to malloc one more gig). My suspicion is that
the currently widely discussed shootout doesn't catch the effect of GC.

With all this I mean that you are to maintain a kind of _conceptual_
cache in your dynamic server, like filesystem cache serves for static
pages you mentioned, otherwise it would be little point to apply the
strength of FP. Say, if you store your stuff just in a database, and use
some WebDB, or so, you do not have to bother about FP. (But it all is
complete IMHO)

Alexander
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Alain Frisch]

On Wed, 11 Jul 2001, Jimmie Houchin wrote:

> Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
> I haven't a clue on how to develop either but could possibly learn. :)
> After learning OCaml (at least some) first.

If I understand correctly what you mean by mod_ocaml (packaging
OCaml runtime environment in a Apache module, in order to avoid to
load it for every request), Samuel Heriard (in Cc)
started such a project; I don't know how the progress status.

The GC should'nt be a big problem: between two requests, the runtime
environment can completely flush the heap (excepted for the
connection/persistency manager). But I may miss some important issues.


-- 
  Alain Frisch

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Samuel Heriard Dubreuil]

> > Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
> > I haven't a clue on how to develop either but could possibly learn. :)
> > After learning OCaml (at least some) first.
> 
> If I understand correctly what you mean by mod_ocaml (packaging
> OCaml runtime environment in a Apache module, in order to avoid to
> load it for every request), Samuel Heriard (in Cc)
> started such a project; I don't know how the progress status.

That's right, I started to write a mod_caml (a la mod_perl). I've not been
working on it since six months, but it does work under apache 1.3 (at
least "Hello world !").
I didn't worry too much about performance issues because I wrote it in
one afternoon mainly to learn the apache api. But I'd like to restart working
on it.
The principle was to associate the caml handler to .cmo files, dynamically
load  $DocumentRoot/bar.cmo on a request to http://foo.com/bar.cmo, let the cmo
do the job, and then unload it. The goal was to create something like jsp.
I know it's pretty inefficient, but with a cache system, you would not
have to load/unload a .cmo for each request.


> The GC should'nt be a big problem: between two requests, the runtime
> environment can completely flush the heap (excepted for the
> connection/persistency manager). But I may miss some important issues.

The connection manager is an orthogonal problem. A connection may use
several instances of the runtime (several apache process) so I think it
has nothing to deal with the garbage collector (one can use files, shared
memory or db to store the connection informations).

So if you're interested in working an a mod_ocaml, let me know, I'll send
you the code (actually not more than a hundred lines of caml/C if I
remember).

--
Samuel
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jimmie Houchin]

I would love to see your mod_ocaml code.
I don't know how much I will be able contribute as I am learning.
However it would be a good learning opportunity for both OCaml and the
Apache API.

A quality mod_ocaml would potentially help OCaml gain mindshare in the
web development arena. Since OCaml is a good general language, that use
could spill over.

A good architecture first, then performance.
It would be nice to have either in this module or loadable from this
module a module for a pool of database handles. That way we could have
persistent connections to the database.

Just some thoughts.

Thanks for you input (both of you).
I look forward to seeing your code.
Email it to me or send me a link.

Thanks.

Jimmie Houchin

Samuel Heriard Dubreuil wrote:
> 
> > > Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
> > > I haven't a clue on how to develop either but could possibly learn. :)
> > > After learning OCaml (at least some) first.
> >
> > If I understand correctly what you mean by mod_ocaml (packaging
> > OCaml runtime environment in a Apache module, in order to avoid to
> > load it for every request), Samuel Heriard (in Cc)
> > started such a project; I don't know how the progress status.
> 
> That's right, I started to write a mod_caml (a la mod_perl). I've not been
> working on it since six months, but it does work under apache 1.3 (at
> least "Hello world !").
> I didn't worry too much about performance issues because I wrote it in
> one afternoon mainly to learn the apache api. But I'd like to restart working
> on it.
> The principle was to associate the caml handler to .cmo files, dynamically
> load  $DocumentRoot/bar.cmo on a request to http://foo.com/bar.cmo, let the cmo
> do the job, and then unload it. The goal was to create something like jsp.
> I know it's pretty inefficient, but with a cache system, you would not
> have to load/unload a .cmo for each request.
> 
> > The GC should'nt be a big problem: between two requests, the runtime
> > environment can completely flush the heap (excepted for the
> > connection/persistency manager). But I may miss some important issues.
> 
> The connection manager is an orthogonal problem. A connection may use
> several instances of the runtime (several apache process) so I think it
> has nothing to deal with the garbage collector (one can use files, shared
> memory or db to store the connection informations).
> 
> So if you're interested in working an a mod_ocaml, let me know, I'll send
> you the code (actually not more than a hundred lines of caml/C if I
> remember).
> 
> --
> Samuel
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jimmie Houchin]

I don't know if what I have in my mind does justice to functional
programming or not as I do not fully have a grasp as to what that is. My
approach is somewhat pragmatic.

I would like to code in a language that is both fast to run and fast to
code in. I want a language with automatic memory management. I am not a
C/C++ programmer. I don't particularly want to program in Java. Nor do I
find Java too advantageous. Now that I've made everyone ill. :) ...

OCaml seems to fit the bill reasonably. The largest challenge to me with
OCaml is it has a foreign syntax (to me) and it is functional
programming. So I need to learn a new syntax and program
paradigm/philosophy. But the benefits for doing so seem to be there.

The strength of FP in my web application, from my understanding, will be
a simpler design. Most web app servers and such are OO and can be quite
complex.

In a simple direct comparison to Perl or Python it would seem OCaml
would perform better.

>From a pragmatic view if simple Python pages (non web app server,
mod_python, etc.) were served at the rate of 100-200 rps and OCaml could
better that by a reasonable margin, then it would be a big win. As tools
for OCaml web development mature and grow the win could become even
larger. Just thoughts, hypothesis, and theories. No concrete facts as of
yet. :)

To me if a Functional Language such as OCaml became more widely used for
practical, pragmatic reasons. It would still be a win for Functional
Programming and OCaml.

So yes, I could probably be using other tools but if OCaml out performs
them, then it is a reasonable pragmatic choice. And if I can end up
being a better programmer using a better tool, well... :)

I am currently exploring and evaluating my options. It just seemed to me
from surface appearances that OCaml could be an excellent, potentially
superior choice.

Hopefully in the near future I'll be able to put together some test
pages which I can use Apache/Zope, Apache/Webware, Apache/OCaml,
Erlang/INETS and see what comes out.

Just my thoughts and opinions. 
Thanks for your input.

Jimmie Houchin


"Alexander V. Voinov" wrote:
> 
> Hi
> 
> Jimmie Houchin wrote:
> > The speed and memory results in the shootout are amazing. But what compounds
> > it is the LOC result. To me that indicates that a language does not have to be
> > verbose to be fast.
> ...
> > Currently fast web servers like Tux 10-15,000+ rps (requests per second),
> > Apache 2-4,000 rps aren't dynamic (at those speeds). Dynamic website tools
> > like Zope, etc are either like 20-80 rps or some I've seen up to 100-200 rps.
> > This is a great disparity.
> >
> > I want a dynamic website which gets as close to static speeds as possible.
> 
> > 3. Tux module. Create an OCaml user space module for Tux.
> >       This would be interesting and would probably be very fast.
> >       This is also beyond my skill.
> 
> I'd like to seriously wonder about the impact of garbage collection in
> such a persistent module, which has to reclaim stuff time to time. And
> given that there will be many linked (recursive) data structures, the
> process of reclaiming would take time. Unless it may be run as a
> separate thread on a separate CPU, and the process of getting space for
> new stuff doesn't heavily depend on the collection of the old one (say,
> when it's not a problem to malloc one more gig). My suspicion is that
> the currently widely discussed shootout doesn't catch the effect of GC.
> 
> With all this I mean that you are to maintain a kind of _conceptual_
> cache in your dynamic server, like filesystem cache serves for static
> pages you mentioned, otherwise it would be little point to apply the
> strength of FP. Say, if you store your stuff just in a database, and use
> some WebDB, or so, you do not have to bother about FP. (But it all is
> complete IMHO)
> 
> Alexander
> -------------------
> Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
> To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

RE: [Caml-list] Web Development with OCaml

[Francois Rouaix]

To add a few cents to the discussion

> 5. An OCaml Web Server. I don't know what's available here yet.
>        I haven't had the time to research yet.
>        This could be a very interesting option, especially if it's fast.
>        This could easily be an excellent option to put behind Tux.
>        Tux serves number 1. above and passes all dynamic services to this
> server.

Back in 1996, I had something called V6 running in OCaml; it was an HTTP
proxy with
a bunch of interesting features, and could serve as a Web server. However,
this was
a long time ago, and iI implemented only HTTP 1.0, not 1.1. Also, we didn't
have threads in
native mode in those days. The speed was still reasonable. I remember that I
could
easily use 60% of  our 10 Mb/s network (network still being used by other
stuff in
the lab).

> Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
> I haven't a clue on how to develop either but could possibly learn. :)
> After learning OCaml (at least some) first.

Actually, I did start an mod_ocaml project a couple of years ago. I had the
core
working, meaning that I produce a very simple page with the module.
However, I stopped before moving forward because of the intricacy of
correctly
designing the whole thing (configuration, separation of name spaces between
user-modules, compilation and loading on demand, etc...).
Also, I had doubts about the usefulness of a mod_ocaml. In practice (meaning
in the real .com world, where I was working at the time), one uses multiple
tier architectures. And you want something in the frontend that is simple
enough
that non-real programmers can still tweak; something that only does layout,
and
no logic.

If someone is interested, I might be able to retrieve the source. There is
a good OReilly book on Apache modules (although Apache 2.0 may make
this obsolete altogether).

--f
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

RE: [Caml-list] Web Development with OCaml

[Gerd Stolpmann]

On Wed, 11 Jul 2001, Francois Rouaix wrote:
>Also, I had doubts about the usefulness of a mod_ocaml. In practice (meaning
>in the real .com world, where I was working at the time), one uses multiple
>tier architectures. And you want something in the frontend that is simple
>enough
>that non-real programmers can still tweak; something that only does layout,
>and
>no logic.

A good comment. I am currently doing web development, and from my experience the
separation of layout and logic is very useful for real-world projects. Not only
because non-programmers can contribute, but also because of general
maintainability (customers often want only layout changes).

200 requests per second for a web application? Well, I must admit that my web
apps are much slower, maybe you can get 5 rps on a single CPU. But my
applications are very complex, and a clean design was more important than
speed. So I think a high-speed web platform isn't what I need.

My apps are always mixtures of languages and principles. Most of them use Perl
scripts to describe the front-logic, O'Caml as powerful background engine, and
XML to describe the layout. Many applications are simply CGIs. If speed did not
suffice, I switched to a simple application server architecture: A CGI program
is the connection between the web server and the application server, and the
latter simply forks for every new request. This works fine, is rock-stable, and
is fast enough (~0.5 seconds request/response time).

I already mentioned earlier this year on this list that I have written an
XML-based template system that makes it easy to separate layout and logic.
Sorry, I have not found the time to release it. I hope I can do it in the next
weeks (my boss is in holidays).

I have further plans with PXP, my XML parser. The idea is to have a camlp4
syntax extension for embedded XML. You can simply write something like

let table_row (a,b) =
  <:xml< <tr><td>This is $a</td><td>And this is $b</td></tr> >>

let table rows =
  <:xml< <table><?list rows?></table> >>

let my_tab =
  table (List.map table_row ["the first row","another cell"; 
                             "the second row", "nonsense"])

(I am not sure about the syntax.)

A converter from XML to HTML is very simple to write, such that a web app could
dynamically create the XML tree in memory, optionally validate it, and write it
as HTML code. I think that would already simplify web development a lot.

For a complex web application, one needs more library modules. For example a
module that manages the "micro-state" of the application (i.e. the state that
is not stored by the database but simply passed through all requests).
Furthermore modules that help to create and manage HTML widgets.

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[John Max Skaller]

Jimmie Houchin wrote:

> I would like to code in a language that is both fast to run and fast to
> code in. I want a language with automatic memory management. I am not a
> C/C++ programmer. I don't particularly want to program in Java. Nor do I
> find Java too advantageous. Now that I've made everyone ill. :) ...
 
> OCaml seems to fit the bill reasonably. The largest challenge to me with
> OCaml is it has a foreign syntax (to me) and it is functional
> programming. So I need to learn a new syntax and program
> paradigm/philosophy. But the benefits for doing so seem to be there.

	Ocaml supports ordinary procedural and object oriented programming.
It is a traditional language in this sense: all languages provide
expressions, which is all that functional programming is.
The main differences is emphasis, and the fact that in Ocaml you 
have first class functions. Uses for this will come naturally
from your application. Soon, you will not be able to go
back. Be warned!!

	For me, the main obstacle to learning was, and still is,
the syntax. It takes time to mentally 'pattern match' against
a new syntax.

	Take the time. It's worth it :-)

> In a simple direct comparison to Perl or Python it would seem OCaml
> would perform better.

	I think this is less relevant than the tradeoff between
rapid prototyping of small applications in Python/Perl, compared
to much better assurances of correctness for larger projects
in Ocaml (due to a combination of strict static typing and
extremely compact code which is easier to analyse).
 
-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au 
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
New generation programming language Felix  http://felix.sourceforge.net
Literate Programming tool Interscript     
http://Interscript.sourceforge.net
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jimmie Houchin]

I would be interested in seeing any code you have available for release.
Any code is at least usable for learning.

A multi-tiered architecture is advocated by some but not all.
Philip Greenspun wrote a book called 
Philip and Alex's Guide to Web Publishing
http://www.amazon.com/exec/obidos/ASIN/1558605347/
view it online at:
http://www.arsdigita.com/books/panda/

He advocates using a two or one tier system. In the book he talks about
AolServer with it's built-in Tcl interpreter and his ACS, Arsdigita Community
Service web app.
http://www.arsdigita.com/products/
http://www.arsdigita.com/

He argues against the three-tiered model. Quite effectively I might add. You
might not agree with everything he says, but he does make a provocative
argument. In his book he talks bad about Java. He is actually a Lisp (Scheme)
proponent. Unfortunately his company took some VC money and now sings the Java
song. Blech. I don't think he liked it tho'.

If there were a sufficient group of people wanting to work towards such, his
ACS provides an excellent model from which to base ideas.

I think that
A web server written in OCaml which is extensible by OCaml modules,
With a built-in templating system based on HereDoc, Pxp, or Camlp4 and
A built-in persistent database connection pool
Could be the basis of an excellent two-tiered system.

>From this you could build a web app such as ACS.

There are several good models from which to get ideas which are implemented in
reasonably friendly languages. There are many templating tools from which to
borrow ideas. The nice thing is to find the tool we like and port it or
rewrite it for and in OCaml.

Am I wrong? Zope, Python can provide 20-80 rps.
Python is not known for it's speed.
Couldn't an implementation in OCaml perform better without sacrificing much in
a user friendly deployment language?

Erlang, an interpreted functional language, has it's own web server and
capacity to extend in Erlang.
It also has an excellent distributed database.
It performs equal to or better than Python's setup. (From my understanding)
Could not OCaml do equal or better?

Can't OCaml do better than the Java ...? :)


I would think OCaml could do well better than all of the above.
Am I wrong?
Am I wrong that there would be benefit to such?

This could be done well with an OCaml web server or with a mod_ocaml.

Behind either you develop the web app server.

Advantages mod_ocaml.
  Let the Apache group do what they do best.
  Ride their coattails and use their tools.
  Use OCaml for building the web app and site.
  Increased mind share for OCaml.

Advantages OCaml web server.
  One language, one tool.
  OCaml and go...

They are not mutually exclusive. :)

If the benefits I see are real and exist. 
Then the main issue is creating the community which develops these tools.
I believe this can come from those in this community who have done parts of
the tools.
I also believe that this is a project which when successfully organized and
active can attract other developers.

Well enough for now. I need to get back to the docs and see if I can wrap my
brain around OCaml. :)
If I can and choose to use OCaml for my web site I'll be happy to contribute
any non-site specific code to the project.

Jimmie Houchin



Francois Rouaix wrote:
> 
> To add a few cents to the discussion
> 
> > 5. An OCaml Web Server. I don't know what's available here yet.
> >        I haven't had the time to research yet.
> >        This could be a very interesting option, especially if it's fast.
> >        This could easily be an excellent option to put behind Tux.
> >        Tux serves number 1. above and passes all dynamic services to this
> > server.
> 
> Back in 1996, I had something called V6 running in OCaml; it was an HTTP
> proxy with
> a bunch of interesting features, and could serve as a Web server. However,
> this was
> a long time ago, and iI implemented only HTTP 1.0, not 1.1. Also, we didn't
> have threads in
> native mode in those days. The speed was still reasonable. I remember that I
> could
> easily use 60% of  our 10 Mb/s network (network still being used by other
> stuff in
> the lab).
> 
> > Is there any interest in a mod_ocaml or a fast-cgi module for OCaml?
> > I haven't a clue on how to develop either but could possibly learn. :)
> > After learning OCaml (at least some) first.
> 
> Actually, I did start an mod_ocaml project a couple of years ago. I had the
> core
> working, meaning that I produce a very simple page with the module.
> However, I stopped before moving forward because of the intricacy of
> correctly
> designing the whole thing (configuration, separation of name spaces between
> user-modules, compilation and loading on demand, etc...).
> Also, I had doubts about the usefulness of a mod_ocaml. In practice (meaning
> in the real .com world, where I was working at the time), one uses multiple
> tier architectures. And you want something in the frontend that is simple
> enough
> that non-real programmers can still tweak; something that only does layout,
> and
> no logic.
>
> If someone is interested, I might be able to retrieve the source. There is
> a good OReilly book on Apache modules (although Apache 2.0 may make
> this obsolete altogether).
> 
> --f
> -------------------
> Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
> To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[William Chesters]

   I once had ideas for an ocaml-based webapp framework, which ended
up as Melati http://melati.org --- in Java (servlets), for the usual
reasons :(.

   Melati is not quite as tidy or as fast as it would have been in
ocaml, but we think it has some good ideas, and it has been used for
enough commercial projects to prove them sound and practical.  All of
them would work very nicely in ocaml, because that's what they were
really inspired by!

   Incidentally I can't see any reason why one shouldn't connect an
ocaml servlet runner to Apache/Tomcat: the protocol is well defined
and not Java-specific.

   The key points are: object database with access protection,
type-sensitive templating system, and generic admin system.

   Apologies that this is rather off topic for caml-list --- maybe
we should take the web development discussion to sourceforge!


   -- An object-oriented database on top of JDBC (SQL).  You specify
a "table hierarchy" using a Java-like schema file, which maps onto a
machine-generated Java class hierarchy plus a set of SQL tables and
indexes to serve as backing store.

   Then accessing your data is as easy as product.getDescription(),
product.setDescription("blah"), or book.getAuthor().getName() with
inter-object references being resolved automatically.
What goes on behind your back is:

      o  Objects---rows, as far as SQL is concerned---are stored on
         the Java side in an LRU cache.  (For some applications,
         this cacheing is enough of a speed win to offset all the
         overhead associated with the OODB and then some.)

      o  You can add your own Java methods to objects, alongside
         the automatically generated accessors.

      o  Each thread is associated with its own SQL transaction.
         Consistency is ensured by object- and where necessary
         table-level locking.

      o  At the end of a transaction, changes you have made to
         objects via set-methods are either written down to SQL
         and committed, or rolled back if an exception was thrown.

      o  Reads and writes of object properties are access-checked.
         The db administrator can protect tables via a user/group/
         capability system, and the programmer can implement
         programmatic checks where necessary.

      o  Each thread is considered to be running on behalf of a
         user (or more generally has an "access token" granting
         certain capabilities).  The standard Melati servlet
         base class sets up the user from a cookie or from
         HTTP basic authentication; the login process is
         triggered transparently to the programmer by access
         exceptions which arise when data is not accessible to
         the "guest" user.

The object database provides the ideal platform on which to rest your
application logic and dynamic content generation: you have a much less
verbose, less fussy, more abstract, more programmable, and safer
interface to your data.

   You do not have to worry about SQL connections, result sets,
transactions, access checking, login, or _anything_.  It is all
completely transparent, so the data really just look like standard
Java objects to the application logic and content generator.  Indeed
the SQL backend could be replaced entirely by something else, except
that we provide access to literal SQL queries (returning object
streams or just fields) for those times when they are very handy.

   camlp4 is perfect for compiling the schema, ocaml objects are
the perfect model for such an OODB, and the details of things like
building up and verifying the SQL database at startup time would be
much easier to handle in ocaml than Java.

   Furthermore, such a thing might be nice to have for all kinds of
non-web ocaml applications.


   -- A "templating engine" for rendering content pages.  We almost
always use webmacro or velocity with Melati.  All that is required is
to map constructs like

     <P>From: $message.author.name</P>

onto

     output_string o "<P>From: ";
     output_string o message#author#name;
     output_string o "</P>"

plus conditional and iteration constructs.  This is laughably trivial
using camlp4 and ocaml classes: it would be easy to achieve much
better efficiency and type-safety than webmacro, without sacrificing
convenience.

   One thing we have found extremely useful is the facility to say
in a template

     $html.input($message.SubjectField)

and have that expand to an appropriate HTML input box, or dropdown, or
whatever as appropriate, depending on the type of the field.
We use the object database's type system, which is an enriched version
of SQL's type system, to name sub-templates (or "templets"), such
as this one for integer values:

    ## File org.melati.poem.IntegerPoemType.wm : the templet used
    ## for rendering fields of Integer type.

    <INPUT NAME="field_$ml.rendered($field.Name)" SIZE=$field.Width
           VALUE="$ml.Attr.rendered($field.RawString)">

    ## Add hooks to validate values entered in the field using
    ## our client-side Javascript library

    <SCRIPT LANGUAGE=JavaScript1.2>
      add_integer("field_$ml.escaped($field.Name)",
                  "$ml.escaped($field.DisplayName)",
                  !$field.Type.Nullable)
    </SCRIPT>

Administrators can tweak the way individual fields are rendered by
setting parameters such as field width through the admin interface,
and if necessary can override the type-default templet for a field
with something completely different.


   -- A web-based data browsing, data entry and admin system.  As soon
as you start up your Melati application, you can call up forms for
searching, data entry etc.  This is also where the admin can do user
management and access control.  See for instance

      http://www.melati.org/melati/org.melati.admin.Admin/melatitest/Main


   Anyway, there it is.  As you can probably tell, I would love
to discuss our experiences with Melati, and more details of scheme we
have worked out, if it helps something similar come to life in my
favourite language!  There is much more to say and our code is
probably readable enough to serve as a source of inspiration.
Unfortunately I don't have a lot of time to do actual coding.

   Note by the way that Melati is structured as a set of auxiliary
libraries, not an overarching "environment" which smothers you and
constrains your freedom.  Your web app is structured as a standard
Java servlet, i.e. a fairly normal program, or it could even be CGI or
anything else.  But it still gets to leverage Melati's productivity- and
quality-enhancing facilities.

   The sources are at http://melati.org/cgi-bin/cvsweb.cgi/org/melati .
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Alain Frisch]

On Fri, 13 Jul 2001, William Chesters wrote:

>    Apologies that this is rather off topic for caml-list --- maybe
> we should take the web development discussion to sourceforge!

Yes. Recent discussions show that many ocaml'ers would be interested
in contributing to a web application architecture for OCaml.

Before creating the project, we need:
- to find a name
- to define the general shape of the architecture, the motivation
  and the aim of the project, and its organisation

I propose to continue discussions on a specific mailing-list:
web-caml@quatramaran.ens.fr that I just created.
To subscribe, just send a mail to majordomo@quatramaran.ens.fr
with body "subscribe web-caml".

If we create the project on Sourceforge, the list will move.
The first technical discussions will be about the architecture;
people interested in such discussions are welcome even if they
don't want to code a single line.

Hope to see you there !


Alain Frisch

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Vitaly Lugovsky]

On Fri, 13 Jul 2001, Alain Frisch wrote:

> On Fri, 13 Jul 2001, William Chesters wrote:

> Before creating the project, we need:
> - to find a name
> - to define the general shape of the architecture, the motivation
>   and the aim of the project, and its organisation

 As I already mentioned here, we need a possibility to load a modules in
runtime. Bytecode is fast, but not enough for a killer web architecture.
So, we need a JIT, or, which will be much better, an object format
including intermediate lambda-trees to compile it and link at runtime.
Even a PIC code for native will not be enough, 'cause of a great overhead.


-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Xavier Leroy]

>  As I already mentioned here, we need a possibility to load a modules in
> runtime. Bytecode is fast, but not enough for a killer web architecture.
> So, we need a JIT, or, which will be much better, an object format
> including intermediate lambda-trees to compile it and link at runtime.
> Even a PIC code for native will not be enough, 'cause of a great overhead.

I fail to see the logic in this argument.

First, position-independent native code is indeed a bit slower than
statically-linked code on some platforms, but surely JIT-produced code
is much worse.  If you don't believe me, compare performance between
gcc -fpic -O and any Java implementation :-)

In other terms: those who can, compile ahead of time; those who can't,
compile just in time.  A prime example of those who can't is Java:
they screwed the definition of their language so well that it makes
traditional compilation impossible; hence their propaganda about JITs.
We (Caml) can...

Second, you (and others) seem to take for granted that a "servlet
container" must load servlets dynamically into its own memory image.
That's the Java way, but again that's not the only way.  Using
separate processes for the HTTP server/servlet container and for the
servlets (but not starting a new servlet process on each request like
CGI does) might very well work better: I'll bet the performance hit
would be insignificant, and you get a clean, well-specified
communication protocol between server and servlets.

That's just one idea, but my general point is that the Java (or .NET)
approach is not necessarily the right approach.

- Xavier Leroy
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Lyn A Headley]

    Alain> I propose to continue discussions on a specific
    Alain> mailing-list: web-caml@quatramaran.ens.fr that I just
    Alain> created.  To subscribe, just send a mail to
    Alain> majordomo@quatramaran.ens.fr with body "subscribe
    Alain> web-caml".

cool.  I won't be participating in the design, but I would appreciate
it if you all would inform us over here on the main list of major
developments from time to time.

I'd also like to make a point in passing.  Dynamic web development
performance has less to do with the implementation language of the web
application than it does with the database backend and associated
indexes, and with your caching and concurrency strategy.  I believe,
if performance is a main concern, that you will have to commit deep
thought to those issues especially.

-Lyn
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Miles Egan]

On Fri, Jul 13, 2001 at 12:29:26PM +0200, Alain Frisch wrote:
> Before creating the project, we need:
> - to find a name

I've always thought "bedouin" would be a cool name for an ocaml web server.

> I propose to continue discussions on a specific mailing-list:
> web-caml@quatramaran.ens.fr that I just created.
> To subscribe, just send a mail to majordomo@quatramaran.ens.fr
> with body "subscribe web-caml".

A mailing list is a good idea; I agree.  
 
> If we create the project on Sourceforge, the list will move.
> The first technical discussions will be about the architecture;
> people interested in such discussions are welcome even if they
> don't want to code a single line.

Why not just start out with Sourceforge?

-- 
miles
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [web-caml] Re: [Caml-list] Web Development with OCaml

[Vitaly Lugovsky]

On Fri, 13 Jul 2001, Xavier Leroy wrote:

> >  As I already mentioned here, we need a possibility to load a modules in
> > runtime. Bytecode is fast, but not enough for a killer web architecture.
> > So, we need a JIT, or, which will be much better, an object format
> > including intermediate lambda-trees to compile it and link at runtime.
> > Even a PIC code for native will not be enough, 'cause of a great overhead.
>
> I fail to see the logic in this argument.
>
> First, position-independent native code is indeed a bit slower than
> statically-linked code on some platforms,

 Is there any benchmarks for Caml? I tryed to modify an i386 compiler
backend to produce a PIC, but I failed. For Alpha we don't need a PIC,
so it will work as fast as statically linked native (but without
inlining...)

> but surely JIT-produced code
> is much worse.  If you don't believe me, compare performance between
> gcc -fpic -O and any Java implementation :-)

 JVM is not the best architecture for a JIT compilation. When I'm talking
about JIT for Caml, I really mean a late compilation and linking, using
the same intermediate formats as in a current native compiler. You only
have to serialize it into object file, and include some parts of compiler
into runtime. I understood, that OCaml bytecode is not well suited for
JVM-like JIT. Even much more worse then JVM itself.

> In other terms: those who can, compile ahead of time; those who can't,
> compile just in time.  A prime example of those who can't is Java:
> they screwed the definition of their language so well that it makes
> traditional compilation impossible; hence their propaganda about JITs.
> We (Caml) can...

 Btw., Java permits a traditional compilation: take a look at GCJ, it
produces a very nice code, except the stupid GC...

> Second, you (and others) seem to take for granted that a "servlet
> container" must load servlets dynamically into its own memory image.

 No. I only don't want to recompile the whole server when I add some new
functions. I don't even think about dynamic RELOADING of objects. It's a
big problem even for a Java (just restarted again this stupid Tomcat...).

> That's the Java way, but again that's not the only way.  Using
> separate processes for the HTTP server/servlet container and for the
> servlets (but not starting a new servlet process on each request like
> CGI does) might very well work better: I'll bet the performance hit
> would be insignificant, and you get a clean, well-specified
> communication protocol between server and servlets.

 But what if we have only one big servlet, which operates with a lot of
different templates/"... server pages", and using a lot of external
libraries?

> That's just one idea, but my general point is that the Java (or .NET)
> approach is not necessarily the right approach.

 Sure, there could be some other ways. But this one seems to me as the
most easy and historically proven. If we want to beat the Java, we have to
show a very close, but better alternative to those java programmers who
was bored with all that disgusting JSPs, Servlets, Oracle Business Objects
and so on. I'm the one of them - I'm using Java, I hate Java, but I can't
prove to my collegues that even Kawa-based solutions will be much
better...


-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[William Chesters]

Lyn A Headley writes:
 > I'd also like to make a point in passing.  Dynamic web development
 > performance has less to do with the implementation language of the web
 > application than it does with the database backend and associated
 > indexes, and with your caching and concurrency strategy.

That has been our experience with Melati; one can easily end up
database-bound---or more likely waiting on the staggeringly slow
JDBC driver (try postgresql's some time ...).

Melati's cacheing can be very helpful.  But of course concurrency
tends to work against cacheing ...

 > I believe, if performance is a main concern, that you will have to
 > commit deep thought to those issues especially.

Depends on the kind of market you are thinking of.  The performance we
get out of Melati+postgres on a cheap PC is adequate for the great
majority of all known websites, without really trying.  If you put a
good ocaml-based equivalent on a 4-way SMP box talking to a database
on another box, you would be comfortably into the hundreds of hits per
second I reckon.
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Jimmie Houchin]

Thanks Alain,

This is great. I had thought about suggesting the creation of a project
and so much as intimated towards such in one of my messages, but thought
it would be presumptuous of me to do so, as I am still only working
through the docs. :)

I also think it was a good idea to delay posting of messages 'til
Monday, giving all time to subscribe.
By the way, what time do the doors open on Monday? :)

Upon visiting your website here are a couple of other architectures you
might add for visiting.

ACS, Arsdigita Community System
High performance reasonably complete web toolkit. Modules for many
things.
Originally written in Tcl and SQL for Oracle and AolServer.
The Tcl version is probably somewhat FPish in that it was written by
Scheme advocates from MIT.
The new Java version is probably going OO.
It is GPL'ed.
http://www.arsdigita.com/products
A port to PostgreSQL is OpenACS is at:
http://www.openacs.org

Another servlet oriented webkit is Webware for Python.
http://webware.sourceforge.net

Personally I'm for simple elegant solutions.
As simple as possible anyway.
Like the XP (eXtreme Programming) people like to say: Do the simplest
thing that works.

Woohoo!!!

Jimmie Houchin

Alain Frisch wrote:
> 
> On Fri, 13 Jul 2001, William Chesters wrote:
> 
> >    Apologies that this is rather off topic for caml-list --- maybe
> > we should take the web development discussion to sourceforge!
> 
> Yes. Recent discussions show that many ocaml'ers would be interested
> in contributing to a web application architecture for OCaml.
> 
> Before creating the project, we need:
> - to find a name
> - to define the general shape of the architecture, the motivation
>   and the aim of the project, and its organisation
> 
> I propose to continue discussions on a specific mailing-list:
> web-caml@quatramaran.ens.fr that I just created.
> To subscribe, just send a mail to majordomo@quatramaran.ens.fr
> with body "subscribe web-caml".
> 
> If we create the project on Sourceforge, the list will move.
> The first technical discussions will be about the architecture;
> people interested in such discussions are welcome even if they
> don't want to code a single line.
> 
> Hope to see you there !
> 
> Alain Frisch
> 
> -------------------
> Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
> To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Ari Heitner]

[apologies to all, i should let sleepng dogs lie. but hey, i go to cmu,
where John Harper gives 15-312 exam questions about java braindamage]

On Fri, Jul 13, 2001 at 04:04:28PM +0200, Xavier Leroy wrote:
> >  As I already mentioned here, we need a possibility to load a modules in
> > runtime. Bytecode is fast, but not enough for a killer web architecture.
> > So, we need a JIT, or, which will be much better, an object format
> > including intermediate lambda-trees to compile it and link at runtime.
> > Even a PIC code for native will not be enough, 'cause of a great overhead.
> 
> I fail to see the logic in this argument.
> 
> First, position-independent native code is indeed a bit slower than
> statically-linked code on some platforms, but surely JIT-produced code
> is much worse.  If you don't believe me, compare performance between
> gcc -fpic -O and any Java implementation :-)
> 
> In other terms: those who can, compile ahead of time; those who can't,
> compile just in time.  A prime example of those who can't is Java:
> they screwed the definition of their language so well that it makes
> traditional compilation impossible; hence their propaganda about JITs.
> We (Caml) can...

I fail to follow this. Isn't Java really no different than C++ except
 - Method calls are dynamic (my tests show this is maybe a 10%
   slowdown. question to java gurus: does "final" make the call static?)
 - Can't use primitive types in containers (big hit)
 - No type parameterization, so containers throw away all type info (yuck!).
   So you have to keep doing expensive rtti (array typing is also broken, so
   you have to typecheck all array access at runtime)
 - of course we all know the java gc's suck

I've definitely seen Java behave slowly. But today I did a dead-simple test
(designed to avoid all the Java weaknesses): a recursive fib, and got
interesting results:
 - Java was a touch faster than gcc (~10% penalty for not making the call 
   "static")
 - gcc (2.95.4) was ~20% slower for pic. I seem to recall gcc 3.0 being
   maybe 10% faster, tho it may be quite a bit faster on less moronic code
   (i've heard up to 40%). but this is a moronic measure that only leans
   on function-call cost -- in real code it should be much less noticeable.
   no one seems to complain about either using shared libraries, or (to 
   be like the java servlet arch below) explicitly loading code via dlopen.
 - ocaml is about 40% faster than java/gcc
 
...

As was mentioned elsewhere, you *can* compile java to native.

And of course I've seen more signficant java stuff just crawl. But my
(decidedly unscientific) guess would be the real penalty is in data
management -- a combination of the weight of the class primitives
(capital-I-Integer and friends) and all the bloody expense of of runtime
typechecking everything (might as well use Python).

It certainly doesn't seem to me that except for braindamaged oo decisions,
the basic language is any harder to optimize than C or C++.

> 
> Second, you (and others) seem to take for granted that a "servlet
> container" must load servlets dynamically into its own memory image.
> That's the Java way, but again that's not the only way.  Using
> separate processes for the HTTP server/servlet container and for the
> servlets (but not starting a new servlet process on each request like
> CGI does) might very well work better: I'll bet the performance hit
> would be insignificant, and you get a clean, well-specified
> communication protocol between server and servlets.

Aha! the real point.

CGIs roll over when people keep hitting the system, and apache (not fast in
the first place) croaks as it spawns a new process for every request -- the
system rapidly has too many processes, and swaps itself to death.

The sane thing to do seems to be to have only a few more processes than
CPUs, and drive those processes has hard as possible with async io. There's
no strong reason to make either your httpd or your database process even
multithreaded...

that aside, using java just to get around switching to a new version of the
servlet without downing the server is even worse -- just bloody use
dlopen...



cheers,

ari

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Gerd Stolpmann]

On Sun, 15 Jul 2001, Ari Heitner wrote:
>[apologies to all, i should let sleepng dogs lie. but hey, i go to cmu,
>where John Harper gives 15-312 exam questions about java braindamage]
>
>> 
>> Second, you (and others) seem to take for granted that a "servlet
>> container" must load servlets dynamically into its own memory image.
>> That's the Java way, but again that's not the only way.  Using
>> separate processes for the HTTP server/servlet container and for the
>> servlets (but not starting a new servlet process on each request like
>> CGI does) might very well work better: I'll bet the performance hit
>> would be insignificant, and you get a clean, well-specified
>> communication protocol between server and servlets.
>
>Aha! the real point.
>
>CGIs roll over when people keep hitting the system, and apache (not fast in
>the first place) croaks as it spawns a new process for every request -- the
>system rapidly has too many processes, and swaps itself to death.

Don't be so unfair. Native-code CGIs are relatively fast. Unix has been
optimized for starting new processes very quickly. Of course, this is only true
if most of the process image is read-only, so the image can be shared by
several processes (my argument doesn't apply to all interpreter languages
which load the code into read-write sections). I have seen expensive SMP
systems that most of their time compile Perl programs. I have also done some
experiments with native-code O'Caml programs called as CGIs, and this is _much_
faster. (I've just done some simple benchmarks: a 500k executable that queries
the state of a daemon and prints it as HTML page, it runs with 40 requests per
second, and the CPU is still 50% idle (I think it's slowed down by the query,
and my benchmark is too stupid). Not bad for a simple technique like CGI.)

Okay, if there are too many processes, the hardware will become too ineffective
(too many cache misses). But this is also true for threads.

>The sane thing to do seems to be to have only a few more processes than
>CPUs, and drive those processes has hard as possible with async io. There's
>no strong reason to make either your httpd or your database process even
>multithreaded...

Most multithreading programs aren't so stupid that they create a new thread for
every request. There is usually a pool of waiting threads that are
activated when new requests arrive. If there are more requests than threads,
the requests must wait.

Because of this, I see no fundamental difference between multi-process and
multi-threaded programs. You can implement the "workers" that wait for requests
both as threads or as processes. However, the communication between threads
is simpler than between processes. (On the other hand, it is possible to
restart hanging processes which is not possible with threads.)

>that aside, using java just to get around switching to a new version of the
>servlet without downing the server is even worse -- just bloody use
>dlopen...

But dlopen is not available for caml (it is not possible to create PIC). I
would suggest to implement the pool of workers as pool of processes (to get
most out of SMP boxes) that can be dynamically changed without breaking the
service. There could be one "master process" knowing which workers are
available and for which tasks they are programmed. The master process assigns
the requests to the workers. (The master process should be as lightweight as
possible, perhaps not even a process.)

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[wakita]

i'm sorry that my comment is off the topic of web development...

A few months ago, I ran the Fibonacci test for gcc 2.95 and ocamlopt
on the Intel platform and saw the similar result; I was not interested
in PIC at the moment.  First, I thought the difference was due to
inlining done by the ocamlopt but it was not the case for Fibonacci
function.  Comparison of the assembly code produced by the both
compiler suggests that the difference is due primariry to the fact
that gcc does not optimize enough argument passing to functions.

Ken Wakita
Tokyo Institute of Technology

In message (<20010715140351.B20044@andrew.cmu.edu>)
from Ari Heitner <aheitner@andrew.cmu.edu>,
talking about "Re: [Caml-list] Web Development with OCaml",
on Sun, 15 Jul 2001 14:03:51 -0400

> I've definitely seen Java behave slowly. But today I did a dead-simple test
> (designed to avoid all the Java weaknesses): a recursive fib, and got
> interesting results:
>  - Java was a touch faster than gcc (~10% penalty for not making the call 
>    "static")
>  - gcc (2.95.4) was ~20% slower for pic. I seem to recall gcc 3.0 being
>    maybe 10% faster, tho it may be quite a bit faster on less moronic code
>    (i've heard up to 40%). but this is a moronic measure that only leans
>    on function-call cost -- in real code it should be much less noticeable.
>    no one seems to complain about either using shared libraries, or (to 
>    be like the java servlet arch below) explicitly loading code via dlopen.
>  - ocaml is about 40% faster than java/gcc
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[John Max Skaller]

Ari Heitner wrote:
> > Second, you (and others) seem to take for granted that a "servlet
> > container" must load servlets dynamically into its own memory image.
> > That's the Java way, but again that's not the only way.  Using
> > separate processes for the HTTP server/servlet container and for the
> > servlets (but not starting a new servlet process on each request like
> > CGI does) might very well work better: I'll bet the performance hit
> > would be insignificant, and you get a clean, well-specified
> > communication protocol between server and servlets.
> 
> Aha! the real point.
> 
> CGIs roll over when people keep hitting the system, and apache (not fast in
> the first place) croaks as it spawns a new process for every request -- the
> system rapidly has too many processes, and swaps itself to death.
> 
> The sane thing to do seems to be to have only a few more processes than
> CPUs, and drive those processes has hard as possible with async io. There's
> no strong reason to make either your httpd or your database process even
> multithreaded...

	Poor OS design is the problem. The correct way to service
HTTP requests is to have a queue of pairs:

	(user-id, request)

and an object for each user. One then checks the user-id of the 
oldest message in the queue, and calls the resume method 
of the object for that user, passing the request.

The key design issue here is how to map user ids onto objects.
By considering how many connectsion you expect, and using
a hash table, you can get near constant time context switching
and support millions of connections, even on a small box.
Distributing the connection service between CPUs is trivial.

There are two big problems here. The first is that writing
event driven code is very hard. Felix solves that problem.
One writes a 'thread' which reads messages, and the compiler
translates the code into event driven form. 
Because Felix generates C++ which is compiled to a shared library, 
the resulting 'service logic' executes very quickly, and the
available 'servlets' can be dynamically extended 
(and even upgraded) while the service process is running.

The second problem is the design of operating
system schedulers. It is necessary to do TCP/IP
on a single channel. Most OS's cannot do this.
They insist on spawning a separate socket for each connection.
Then there is no fast way to find which socket is ready
with a complete message. This is because the scheduling
algorithms are general purpose.

So, if there is anyone that knows enough TCP/IP to provide
the required logic from a RAW (connectionless) socket,
then there is a solution. I believe FreeBSD provides
such sockets. The TCP layer must still be implemented.
There may be a way to hack Linux to intercept the 
construction of messages per socket (and steal the 
message away, and put it into the message queue).

It is possible that Felix can be modified/enhanced to
generate Ocaml instead of C++: the key requirement
is for classes with a resume method. [The biggest
technical problem is that efficient implementation
of control structures requires a goto which Caml
doesn't have, but there is a slightly less efficient
workaround already available]

The main problem with an Ocaml solution is that it doesn't 
support dynamic loading. It may be possible to circumvent
that issue by either using a process per program
(just ONE) or by using the bytecode interpreter.

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au 
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
New generation programming language Felix  http://felix.sourceforge.net
Literate Programming tool Interscript     
http://Interscript.sourceforge.net
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[Ari Heitner]

On Wed, Jul 18, 2001 at 02:18:58AM +1000, John Max Skaller wrote:
> Ari Heitner wrote:
> > The sane thing to do seems to be to have only a few more processes than
> > CPUs, and drive those processes has hard as possible with async io. There's
> > no strong reason to make either your httpd or your database process even
> > multithreaded...
> 
> 	Poor OS design is the problem. The correct way to service
> HTTP requests is to have a queue of pairs:
> 	(user-id, request)
> and an object for each user. One then checks the user-id of the 
> oldest message in the queue, and calls the resume method 
> of the object for that user, passing the request.
> 
> There are two big problems here. The first is that writing
> event driven code is very hard. Felix solves that problem.
> [...]
> The second problem is the design of operating
> system schedulers. It is necessary to do TCP/IP
> on a single channel. Most OS's cannot do this.
> 
> So, if there is anyone that knows enough TCP/IP to provide
> the required logic from a RAW (connectionless) socket,
> then there is a solution. I believe FreeBSD provides
> such sockets. The TCP layer must still be implemented.
> There may be a way to hack Linux to intercept the 
> construction of messages per socket (and steal the 
> message away, and put it into the message queue).

Wow. You're a mind reader :)

This is how I've been wanting to do communication for some time.

My background for this is actually videogames -- something I started doing
in high school in a DOS extender.

Similar situation -- you've got a render pipeline (initially a hand-coded
software render, later a hardware card), representing a serial resource. And
you've got a bunch of concrete conceptual models to render, each of which
can be broken into little parts.

What you don't have is overhead ;) since we wrote everything ourselves. Our
homebrew OS had non-premptive multiprocessing via an event-driven model --
a straightforward minimalist RTOS. 120 times a second you process necessary
physics. The rest of your spare time, you activate render events, which give
every game object an equal shot at rendering a pass (think "sending out a
packet").

And of course it worked fine on top of Windows or Linux or whatever -- since
the render pipeline is handled at a very low level, traditionally (if it was
like TCP/IP, it would be like having GL replaced by a scenegraph system).

> 
> It is possible that Felix can be modified/enhanced to
> generate Ocaml instead of C++: the key requirement
> is for classes with a resume method. [The biggest
> technical problem is that efficient implementation
> of control structures requires a goto which Caml
> doesn't have, but there is a slightly less efficient
> workaround already available]

I've looked at Felix a tiny bit, but not enough to know how it interprets
event-driven stuff...I never felt uncomfortable programming event-driven
stuff in C++, but then, I literally grew up on it :) you just have to keep a
good tag on how how long any even takes :)

...

I never seriously looked at implementing any of this -- it's way off topic
for my normal stuff. But surely Linux or the BSDs have some kind of raw
ethernet interface? I mean, eth0 is a character device, right? Except of
course for the weird magical distinction between network devices, which
exist in their own magical TCP/IP-land-namespace, and all other devices
(serial ports, usb ports, parallel ports, hard drives, floppy drives, smoke
signals, hamsters with laser pointers) all have /dev/ entries and can
presumably be accessed in raw mode (oh wait, "smoke signals" are a
networking device). 

I suspect half the problem is the evils of TCP/IP ("a hoagie with many many
layers"). But then I never took 15-411 Networks, I took 15-412 OS. So aside
from the fact I know every packet has a lot of bytes worth of headers around
the payload, I have no real clue how hard it would be, or what libs exist,
for assembling packets on a raw interface.

...

Our scheme in our video game engine was simple: all objects can register for
ticks (120 times a second) and renders (whenever there's free resources).
Everyone plays nice. Easy enough to do in o'caml, given the right libs.

...

But doesn't this sound a lot like what's happening in kernel space anyhow? I
mean, you've got a queue of packets to send (skb_buf's, in Linux, iirc).
Those include the actual data you've produced, which can even be
scatter-gathered to avoid copying data -- it's just a collection of pages to
send directly to the card. And the kernel has what we descreibed above --
when the card generates a RTS interrupt, the driver pulls more packets off
the queue.

So what's the hangup? I'm no longer sure :) Is it just the expense of having
zillions of threads/processes managing all these connections, and context
switching expensively to get tiny bits of work done? You can't easily get
away w/o at least a process or thread for your DB work, right, because the
database needs fore each query are complicated and it would be *really* hard
to manage that in an event-oriented way. So even if your webserver was a
single thread, or even entirely in kernel space, you spend a lot on managing
the assebly of data from your DB -- which has to be done (at least) a thread
per conceptual request?

Or does Felix do that nicely too? Or did I totally miss the point?

[or am i just rambling?]

ari
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Web Development with OCaml

[John Max Skaller]

Ari Heitner wrote:

> This is how I've been wanting to do communication for some time.
> 
> My background for this is actually videogames -- something I started doing
> in high school in a DOS extender.
>
	I did that too, but commercially. Didn't pan out.
But I got to use the best C++ compiler around (Metaware: had nested
functions with closures, and iterators modelled as continuations!)

> I've looked at Felix a tiny bit, but not enough to know how it interprets
> event-driven stuff.

	The current, deficient, model is easy to explain.
You write:

	// blah blah sequential code
	read x;
	// blah blah using x
	read x;

and when you hit 'read x', you lose control.

The dispatch loop then grabs the next
message from the input queue, and writes it into the object 
that should get it, and then 'resumes' the code just after
the 'read' it did. Its done by:

	switch(pc) {
	case 1: ...
	case 2:
		// read x
		data_required_in = &x;
		pc = 3;
		return this;
	case 3:
		...

This is exactly what the OS does when you say 'fread()' in C.

The difference is that your Felix code yields to YOUR driver
program, not the OS. You have to write that driver yourself.

>..I never felt uncomfortable programming event-driven
> stuff in C++, 

	Try something difficult. Like writing a compiler
whose parser is _called_ with each line of the source,
instead of reading it. :-)

> but then, I literally grew up on it :) you just have to keep a
> good tag on how how long any even takes :)

	That's not the problem: the difficulty is that you
are back to programming in a flat data space without
a stack. That is, you're back to computer science as it was
pre- structured programming. You're writing COBOL or worse.

	Now look at the immense power of functional programming
languages and imagine what would happen if you didn't have a stack.
It's all gone. No recursion, for example! So you have to emulate
a stack. You have to write your own homebrew interpreter/OS.

> I never seriously looked at implementing any of this -- it's way off topic
> for my normal stuff. But surely Linux or the BSDs have some kind of raw
> ethernet interface? 

	Sure, but thats WAY too low level for me!

> But doesn't this sound a lot like what's happening in kernel space anyhow? 

	Absolutely. The problem, as I said, is that the OS provides a rich
set of facilities for scheduling. As a result, context switching
is extremely slow. For many applications, such as GUI and telephony
and web servers, you just don't need that rich set of tools for the
bulk of the work (you DO need a few processes/threads, but NOT one
per connection!)

> So what's the hangup? I'm no longer sure :) 

	A Web server must play the TCP/IP game because that's
what clients are playing. That means, the server must obey the
TCP/IP protocol. 

> Is it just the expense of having
> zillions of threads/processes managing all these connections, and context
> switching expensively to get tiny bits of work done? 

	Yes. The problem is deciding which thread to run next,
out of the 100,000 you have waiting on the 100,000 sockets you have open
for each of the 100,000 users buying your merchandise by e-commerce.
[Imagine Amazon, or an online newspaper, or a bank .. you could easily
have that
many users connected]

You can't easily get
> away w/o at least a process or thread for your DB work, 

	Of course. You need some threads/processes. But not one per connection:
the connections are all typically idle most of the time, and the 
response will be maximised by serialising the requests (into one stream
per CPU).

> database needs fore each query are complicated and it would be *really* hard
> to manage that in an event-oriented way. 

	Event driven programming of anything non-trivial is almost impossible
by hand, because you have to maintain ALL the state yourself ..
including
the current locus of control (program counter) .. because the stack
has to be empty at each blocking point.

> So even if your webserver was a
> single thread, or even entirely in kernel space, you spend a lot on managing
> the assebly of data from your DB -- which has to be done (at least) a thread
> per conceptual request?

	No. Access to the DB is done by passing messages to the DB server.
Typically, thats a small collection of processes. Your request just
joins the queue. That is, it must be done asynchronously. You don't
get an answer back immediately: you have to 'read' the answer
(that is, block until the DB gets around to responding to your request).

> Or does Felix do that nicely too? Or did I totally miss the point?
> 
> [or am i just rambling?]

	Felix doesn't do ANY of the threading work. It doesn't
even do any scheduling. You have to write all that in C/C++ (or Ocaml,
or whatever). What Felix does is builds objects with resume methods
that you can call from your driver, and it builds them from ML like
code in which you 'read' data, but you actually get called _with_ 
the data.

	BTW: none of this is mentioned in the tutorial yet.
One reason is that the design is deficient: there's only ONE
input queue, because you can only say 'read x'. You can't
read _from_ something. Which means you can't write a Felix dispatcher
in Felix, which isn't acceptable. At present, the continuations
are transparent: they need to be brought into the type system
(and I don't know how to do that) Its possible that reading
_from_ a channel isn't the right idea (JoCaml doesn't do that,
for example).

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au 
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
New generation programming language Felix  http://felix.sourceforge.net
Literate Programming tool Interscript     
http://Interscript.sourceforge.net
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr