Re: [Caml-list] Web Development with OCaml

[John Max Skaller <skaller@maxtal.com.au>]

Ari Heitner wrote:

> This is how I've been wanting to do communication for some time.
> 
> My background for this is actually videogames -- something I started doing
> in high school in a DOS extender.
>
	I did that too, but commercially. Didn't pan out.
But I got to use the best C++ compiler around (Metaware: had nested
functions with closures, and iterators modelled as continuations!)

> I've looked at Felix a tiny bit, but not enough to know how it interprets
> event-driven stuff.

	The current, deficient, model is easy to explain.
You write:

	// blah blah sequential code
	read x;
	// blah blah using x
	read x;

and when you hit 'read x', you lose control.

The dispatch loop then grabs the next
message from the input queue, and writes it into the object 
that should get it, and then 'resumes' the code just after
the 'read' it did. Its done by:

	switch(pc) {
	case 1: ...
	case 2:
		// read x
		data_required_in = &x;
		pc = 3;
		return this;
	case 3:
		...

This is exactly what the OS does when you say 'fread()' in C.

The difference is that your Felix code yields to YOUR driver
program, not the OS. You have to write that driver yourself.

>..I never felt uncomfortable programming event-driven
> stuff in C++, 

	Try something difficult. Like writing a compiler
whose parser is _called_ with each line of the source,
instead of reading it. :-)

> but then, I literally grew up on it :) you just have to keep a
> good tag on how how long any even takes :)

	That's not the problem: the difficulty is that you
are back to programming in a flat data space without
a stack. That is, you're back to computer science as it was
pre- structured programming. You're writing COBOL or worse.

	Now look at the immense power of functional programming
languages and imagine what would happen if you didn't have a stack.
It's all gone. No recursion, for example! So you have to emulate
a stack. You have to write your own homebrew interpreter/OS.

> I never seriously looked at implementing any of this -- it's way off topic
> for my normal stuff. But surely Linux or the BSDs have some kind of raw
> ethernet interface? 

	Sure, but thats WAY too low level for me!

> But doesn't this sound a lot like what's happening in kernel space anyhow? 

	Absolutely. The problem, as I said, is that the OS provides a rich
set of facilities for scheduling. As a result, context switching
is extremely slow. For many applications, such as GUI and telephony
and web servers, you just don't need that rich set of tools for the
bulk of the work (you DO need a few processes/threads, but NOT one
per connection!)

> So what's the hangup? I'm no longer sure :) 

	A Web server must play the TCP/IP game because that's
what clients are playing. That means, the server must obey the
TCP/IP protocol. 

> Is it just the expense of having
> zillions of threads/processes managing all these connections, and context
> switching expensively to get tiny bits of work done? 

	Yes. The problem is deciding which thread to run next,
out of the 100,000 you have waiting on the 100,000 sockets you have open
for each of the 100,000 users buying your merchandise by e-commerce.
[Imagine Amazon, or an online newspaper, or a bank .. you could easily
have that
many users connected]

You can't easily get
> away w/o at least a process or thread for your DB work, 

	Of course. You need some threads/processes. But not one per connection:
the connections are all typically idle most of the time, and the 
response will be maximised by serialising the requests (into one stream
per CPU).

> database needs fore each query are complicated and it would be *really* hard
> to manage that in an event-oriented way. 

	Event driven programming of anything non-trivial is almost impossible
by hand, because you have to maintain ALL the state yourself ..
including
the current locus of control (program counter) .. because the stack
has to be empty at each blocking point.

> So even if your webserver was a
> single thread, or even entirely in kernel space, you spend a lot on managing
> the assebly of data from your DB -- which has to be done (at least) a thread
> per conceptual request?

	No. Access to the DB is done by passing messages to the DB server.
Typically, thats a small collection of processes. Your request just
joins the queue. That is, it must be done asynchronously. You don't
get an answer back immediately: you have to 'read' the answer
(that is, block until the DB gets around to responding to your request).

> Or does Felix do that nicely too? Or did I totally miss the point?
> 
> [or am i just rambling?]

	Felix doesn't do ANY of the threading work. It doesn't
even do any scheduling. You have to write all that in C/C++ (or Ocaml,
or whatever). What Felix does is builds objects with resume methods
that you can call from your driver, and it builds them from ML like
code in which you 'read' data, but you actually get called _with_ 
the data.

	BTW: none of this is mentioned in the tutorial yet.
One reason is that the design is deficient: there's only ONE
input queue, because you can only say 'read x'. You can't
read _from_ something. Which means you can't write a Felix dispatcher
in Felix, which isn't acceptable. At present, the continuations
are transparent: they need to be brought into the type system
(and I don't know how to do that) Its possible that reading
_from_ a channel isn't the right idea (JoCaml doesn't do that,
for example).

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au 
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
New generation programming language Felix  http://felix.sourceforge.net
Literate Programming tool Interscript     
http://Interscript.sourceforge.net
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr