async networking

async networking

[Rick Richardson]

I found a post on the list from about 4 years ago where a gentleman
expressed interest in developing an async net library in/for ocaml
modelled after medusa. 

I have the exact same aspirations, actually, I'd rather just *use* a
quality async library than build one. Can anyone suggest such a library?

Thanks,
Rick Richardson

Re: [Caml-list] async networking

[Gerd Stolpmann]

Maybe you interested in my equeue library for multiplexing I/O. There is
now a lot of code using equeue (HTTP client, FTP client, RPC
implementation), so I think it is very stable.

http://www.ocaml-programming.de/programming/equeue.html

Gerd

On So, 2006-02-05 at 00:52 -0800, Rick Richardson wrote:
> I found a post on the list from about 4 years ago where a gentleman
> expressed interest in developing an async net library in/for ocaml
> modelled after medusa. 
> 
> I have the exact same aspirations, actually, I'd rather just *use* a
> quality async library than build one. Can anyone suggest such a library?
> 
> Thanks,
> Rick Richardson
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] async networking

[Pierre Etchemaïté]

Le Sun, 05 Feb 2006 00:52:10 -0800, Rick Richardson <rick@eltopia.com> a écrit :


> I have the exact same aspirations, actually, I'd rather just *use* a
> quality async library than build one. Can anyone suggest such a library?

I came across a wrapper for the libevent library...
(No personnal experience with it, sorry)

	http://www.xs4all.nl/~mmzeeman/ocaml/

Re: async networking

[Bardur Arantsson]

Gerd Stolpmann wrote:
> Maybe you interested in my equeue library for multiplexing I/O. There is
> now a lot of code using equeue (HTTP client, FTP client, RPC
> implementation), so I think it is very stable.
> 
> http://www.ocaml-programming.de/programming/equeue.html
> 

If you want ease of use and all-round stability I can also recommend
Gerd's Equeue. However, if you want very high-performance networking
you'd be better off with something closer to the metal, i.e. something
like a libevent wrapper combined with a memory-mapped ring buffer to
avoid unnecessary copying and minimize the costs associated with string
allocation and GC. I have developed a highly efficient ring buffer 
library (ocaml-rbuffer) which can be downloaded at

    http://www.imada.sdu.dk/~bardur/personal/45-programs/index.html

Cheers,

-- 
Bardur Arantsson
<bardurREMOVE@THISimada.sdu.dk>
<bardurREMOVE@THISscientician.net>

- Sometimes I think Trent just needs a cup of hot chocolate and a
blankie.
                           Tori Amos commenting on Nine Inch Nails

Re: [Caml-list] Re: async networking

[skaller]

On Mon, 2006-02-06 at 19:34 +0100, Bardur Arantsson wrote:

> However, if you want very high-performance networking
> you'd be better off with something closer to the metal, i.e. something
> like a libevent wrapper 

Argg no. Libevent isn't a library, it doesn't control invert.
It is a monolithic framework. Therefore it is not very useful because
your code will no longer be composable. In particular,
there is no way to compose two such frameworks, for example
you cannot use it with an event driven GUI framework.

To quote the manpage:

"In order to process events, an application needs to call
event_dispatch(). This function only returns on error, and should
replace the event core of the application program."

Just try telling a GUI that *also* demands you use ITS 
event loop that!

The only proper way to use libevent would be to spawn the
dispatch loop inside a pthread, and use the hook functions
to communicate with your program via channels. It isn't
immediately clear this would work (libevent's single page
of documentation doesn't explain if it works when spawned
inside a pthread).

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: async networking

[Bardur Arantsson]

skaller wrote:
> On Mon, 2006-02-06 at 19:34 +0100, Bardur Arantsson wrote:
> 
>> However, if you want very high-performance networking
>> you'd be better off with something closer to the metal, i.e. something
>> like a libevent wrapper 
> 
> Argg no. Libevent isn't a library, it doesn't control invert.
> It is a monolithic framework. Therefore it is not very useful because
> your code will no longer be composable. In particular,
> there is no way to compose two such frameworks, for example
> you cannot use it with an event driven GUI framework.
> 

Note that I said 'high-performance'.

Point #1: select() and anything based on it (I believe Equeue still is 
though I haven't looked at it for quite a while) is woefully inadequate 
for high performance I/O except in very specific circumstances.

Point #2: It is not customary for UI applications to require 
particularly high-performance I/O, thus rendering the non-composability 
issue moot.

I'm _not_ recommending libevent for general use, just if you want high 
performance with an easily switchable backend implementation.

Cheers,

-- 
Bardur Arantsson
<bardurREMOVE@THISimada.sdu.dk>
<bardurREMOVE@THISscientician.net>

- Your Honor, let the record reflect that the Defense just grazed
my Johnson.
                                               John, 'Ally McBeal'

Re: [Caml-list] Re: async networking

[Rick Richardson]

> Note that I said 'high-performance'.
> 
> Point #1: select() and anything based on it (I believe Equeue still is 
> though I haven't looked at it for quite a while) is woefully inadequate 
> for high performance I/O except in very specific circumstances.
> 
I was planning on a select() based system. Which architecture would you
recommend? I like the idea of the ring buffer. Especially for switch
type applications which are going in one port and out the other. For a
standard system that could involve 2-3 different copies when 1 would be
adequate. 

> Point #2: It is not customary for UI applications to require  
> particularly high-performance I/O, thus rendering the non-composability 
> issue moot.
> 
> I'm _not_ recommending libevent for general use, just if you want high 
> performance with an easily switchable backend implementation.

I haven't fully looked at the libevent system, but any such framework
would seem like overkill to me, when really I just need a lightweight
"deferred" class which maintains a simple callback list. In fact.. I
could leave that out of this library entirely as it may be out of the
scope of my simple project. 

The medusa/asyncore project is 847 lines of c++ code including a test
case and demo. I was hoping to stay around that number. 

Thanks, everyone for your input so far, I am rather new to ocaml and all
the resources that have been pointed out to me have been enlightening on
more subjects than this.  

Re: [Caml-list] Re: async networking

[Gerd Stolpmann]

Am Dienstag, den 07.02.2006, 18:44 +0100 schrieb Bardur Arantsson:
> skaller wrote:
> > On Mon, 2006-02-06 at 19:34 +0100, Bardur Arantsson wrote:
> > 
> >> However, if you want very high-performance networking
> >> you'd be better off with something closer to the metal, i.e. something
> >> like a libevent wrapper 
> > 
> > Argg no. Libevent isn't a library, it doesn't control invert.
> > It is a monolithic framework. Therefore it is not very useful because
> > your code will no longer be composable. In particular,
> > there is no way to compose two such frameworks, for example
> > you cannot use it with an event driven GUI framework.
> > 
> 
> Note that I said 'high-performance'.
> 
> Point #1: select() and anything based on it (I believe Equeue still is 
> though I haven't looked at it for quite a while) is woefully inadequate 
> for high performance I/O except in very specific circumstances.

Yes, the default Equeue implementation bases simply on select(). It is,
however, possible to develop alternate implementations. Currently, there
are three of them which integrate into the event loops of labltk,
lablgtk1 and lablgtk2. One could, for example, easily add an
implementation for advanced kernel interfaces like epoll. Or one that
sits upon libevent. There is, however, the basic design decision that
all events pass the same queue. This mainly has a certain scheduling
effect (application-driven scheduling), and increases latency if the
queue gets too long, but shows good behaviour under high load.

select() is, as far as I know, only bad if the file descriptors are
linked with many different processes, because all that processes must be
waked up in order to check the descriptors (even if no I/O can happen).
But if you only have Internet sockets, I expect that select() performs
well.

> Point #2: It is not customary for UI applications to require 
> particularly high-performance I/O, thus rendering the non-composability 
> issue moot.

There are many aspects of high performance, for example throughput,
latency, and whether a low or high number of descriptors are watched. UI
applications are often interested in low latency for a moderate number
of descriptors.

I think there is another point why it is a bad idea to distinguish
between, say UI and server applications. Network components should be
shareable between all types of applications. For example, an HTTP
component is useful for both, so why should we develop an extra one for
specific needs of high performance?

Gerd

> I'm _not_ recommending libevent for general use, just if you want high 
> performance with an easily switchable backend implementation.
> 
> Cheers,
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Telefon: 06151/153855                  Telefax: 06151/997714
------------------------------------------------------------

Re: async networking

[Bardur Arantsson]

Gerd Stolpmann wrote:
> Am Dienstag, den 07.02.2006, 18:44 +0100 schrieb Bardur Arantsson:
>> skaller wrote:
>>> On Mon, 2006-02-06 at 19:34 +0100, Bardur Arantsson wrote:
>>>
>>>> However, if you want very high-performance networking
>>>> you'd be better off with something closer to the metal, i.e. something
>>>> like a libevent wrapper 
>>> Argg no. Libevent isn't a library, it doesn't control invert.
>>> It is a monolithic framework. Therefore it is not very useful because
>>> your code will no longer be composable. In particular,
>>> there is no way to compose two such frameworks, for example
>>> you cannot use it with an event driven GUI framework.
>>>
>> Note that I said 'high-performance'.
>>
>> Point #1: select() and anything based on it (I believe Equeue still is 
>> though I haven't looked at it for quite a while) is woefully inadequate 
>> for high performance I/O except in very specific circumstances.
> 
> Yes, the default Equeue implementation bases simply on select(). It is,
> however, possible to develop alternate implementations. Currently, there
> are three of them which integrate into the event loops of labltk,
> lablgtk1 and lablgtk2. One could, for example, easily add an
> implementation for advanced kernel interfaces like epoll.

Actually, it might be quite interesting to see one based on 
poll()/epoll(), but I suspect it wouldn't matter much compared to all 
the other stuff that's going in the Equeue code. This is just a very 
vague hunch, though.

[--snip--]
> 
> select() is, as far as I know, only bad if the file descriptors are
> linked with many different processes, because all that processes must be
> waked up in order to check the descriptors (even if no I/O can happen).
> But if you only have Internet sockets, I expect that select() performs
> well.

There are lots of problems with select(). For example, it generally 
requires copying the file descriptor sets from user-space to kernel 
space for every select() invocation, the scanning of active file 
descriptors afterward can be inefficient(*), from OCaml it also causes 
more pressure on the GC through list allocations for result sets, etc. etc.

(*) If your file descriptor set has large "holes" -- which can develop 
if clients have very variable connection lifetimes -- you'll end up 
scanning lots of memory you don't really need to scan. Algorithmically 
it's all still O(n), but the constants can be _very_ different.

> 
>> Point #2: It is not customary for UI applications to require 
>> particularly high-performance I/O, thus rendering the non-composability 
>> issue moot.
> 
> There are many aspects of high performance, for example throughput,
> latency, and whether a low or high number of descriptors are watched.

Sure. It's just that select() sucks performance-wise for most purposes. 
The alternatives like the OpenBSD kqueue and poll() are usually much better.

> UI applications are often interested in low latency for a moderate number
> of descriptors.

Sure, but not at the level I'm talking. I'm talking about saturating 
high-bandwidth links, ultra-low latency -- though usually not at the 
same time, obviously ;). You know, the kind of stuff you might even 
consider using C/C++ (shudder) for just to avoid GC.

> 
> I think there is another point why it is a bad idea to distinguish
> between, say UI and server applications. Network components should be
> shareable between all types of applications.

Yup. Though all non-blocking I/O has the basic problem of "one event 
loop to rule them all" unless you go with several threads, each running 
their own event loop... At which point you might as well have each of 
those components just using regular blocking I/O and using threads as 
appropriate. Note, I'm not talking about performance characteristics 
here, just design. Also, exception handling and such can be handled much 
more nicely with a threaded design since they don't "invert" the control 
logic the same way non-blocking I/O tends to do.

> For example, an HTTP component is useful for both, so why should we develop an extra one for
> specific needs of high performance?
> 

Sometimes you just need that last bit of performance where a "nice" 
implementation just won't do.

Cheers,

-- 
Bardur Arantsson
<bardurREMOVE@THISimada.sdu.dk>
<bardurREMOVE@THISscientician.net>

[Postmodernism] is more like a three year old throwing his blocks
around the room because he got frustrated with his failed
attempts to stack them higher.
                                Black Parrot @ http://slashdot.org

Re: [Caml-list] Re: async networking

[Gerd Stolpmann]

Am Dienstag, den 07.02.2006, 20:43 +0100 schrieb Bardur Arantsson:
> > Yes, the default Equeue implementation bases simply on select(). It is,
> > however, possible to develop alternate implementations. Currently, there
> > are three of them which integrate into the event loops of labltk,
> > lablgtk1 and lablgtk2. One could, for example, easily add an
> > implementation for advanced kernel interfaces like epoll.
> 
> Actually, it might be quite interesting to see one based on 
> poll()/epoll(), but I suspect it wouldn't matter much compared to all 
> the other stuff that's going in the Equeue code. This is just a very 
> vague hunch, though.

The Equeue core is quite light-weight, there are just some modules on
top of it that make life easier at the user's option. So it would
matter.

> > select() is, as far as I know, only bad if the file descriptors are
> > linked with many different processes, because all that processes must be
> > waked up in order to check the descriptors (even if no I/O can happen).
> > But if you only have Internet sockets, I expect that select() performs
> > well.
> 
> There are lots of problems with select(). For example, it generally 
> requires copying the file descriptor sets from user-space to kernel 
> space for every select() invocation, the scanning of active file 
> descriptors afterward can be inefficient(*), from OCaml it also causes 
> more pressure on the GC through list allocations for result sets, etc. etc.

That depends all very much on the average data case. E.g. if all the
descriptors generate events at the same time, there is nothing bad with
select(). If the typical case is that only very few descriptors "fire"
it has very much overhead.

The poll() interface fixes a number of these issues. Anyway, in practice
it is not much better than select() just because it is also based on
passive iteration. One needs an actively notifying interface to improve
that.

> Sure, but not at the level I'm talking. I'm talking about saturating 
> high-bandwidth links, ultra-low latency -- though usually not at the 
> same time, obviously ;). You know, the kind of stuff you might even 
> consider using C/C++ (shudder) for just to avoid GC.

Given that data also needs to be processed by the application, it would
be quite interesting whether the choice of the language makes a
difference here. I would guess the time for GC can be kept low in
relation to the costs for context switches, even for the
high-performance case.

And in O'Caml one can prefer to program in a style that reduces the load
on the GC. A much nicer alternative than considering C/C++ too early.

> > I think there is another point why it is a bad idea to distinguish
> > between, say UI and server applications. Network components should be
> > shareable between all types of applications.
> 
> Yup. Though all non-blocking I/O has the basic problem of "one event 
> loop to rule them all" unless you go with several threads, each running 
> their own event loop... At which point you might as well have each of 
> those components just using regular blocking I/O and using threads as 
> appropriate. Note, I'm not talking about performance characteristics 
> here, just design. Also, exception handling and such can be handled much 
> more nicely with a threaded design since they don't "invert" the control 
> logic the same way non-blocking I/O tends to do.

Really? From my experience I would say that exception handling is much
easier in the non-blocking case. Exceptions may raise complicated
synchronisation issues in the threaded design.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Telefon: 06151/153855                  Telefax: 06151/997714
------------------------------------------------------------

Re: [Caml-list] Re: async networking

[Remi Vanicat]

2006/2/7, Bardur Arantsson <spam@scientician.net>:
> Gerd Stolpmann wrote:
> > Yes, the default Equeue implementation bases simply on select(). It is,
> > however, possible to develop alternate implementations. Currently, there
> > are three of them which integrate into the event loops of labltk,
> > lablgtk1 and lablgtk2. One could, for example, easily add an
> > implementation for advanced kernel interfaces like epoll.
>
> Actually, it might be quite interesting to see one based on
> poll()/epoll(), but I suspect it wouldn't matter much compared to all
> the other stuff that's going in the Equeue code. This is just a very
> vague hunch, though.

I believe that labgtk (and so equeue over lablgtk) use poll. Note that
you are adding another layer : glib, so you might not have gain a lot.

Re: [Caml-list] Re: async networking

[Rick Richardson]

On Tue, 2006-02-07 at 18:44 +0100, Bardur Arantsson wrote: 
> skaller wrote:
> > On Mon, 2006-02-06 at 19:34 +0100, Bardur Arantsson wrote:
> > 
> >> However, if you want very high-performance networking
> >> you'd be better off with something closer to the metal, i.e. something
> >> like a libevent wrapper 
> > 
> > Argg no. Libevent isn't a library, it doesn't control invert.
> > It is a monolithic framework. Therefore it is not very useful because
> > your code will no longer be composable. In particular,
> > there is no way to compose two such frameworks, for example
> > you cannot use it with an event driven GUI framework.
> > 
> 
> Note that I said 'high-performance'.
> 
> Point #1: select() and anything based on it (I believe Equeue still is 
> though I haven't looked at it for quite a while) is woefully inadequate 
> for high performance I/O except in very specific circumstances.

My only interest lies in high performance networking, specifically in a
high connection / low data volume scenario (at this juncture at least).
The optimal solution is some form of iocp with multiple threads (1 per
open socket initiated at startup). I think at some level of high
performance networking the whole file analogy goes out the window.

The multiple serving threads could actually make for an simple api,
actually. A simple function to add a receive callback for a port would
be all you'd need. You could even pass in a buffer to that callback that
a person could respond directly to for socket send, since the same
thread is handling the send requests as well. 

> 
> Point #2: It is not customary for UI applications to require 
> particularly high-performance I/O, thus rendering the non-composability 
> issue moot.
> 
> I'm _not_ recommending libevent for general use, just if you want high 
> performance with an easily switchable backend implementation.
> 
> Cheers,
> 

Re: [Caml-list] Re: async networking

[Jonathan Roewen]

> The multiple serving threads could actually make for an simple api,
> actually. A simple function to add a receive callback for a port would
> be all you'd need. You could even pass in a buffer to that callback that
> a person could respond directly to for socket send, since the same
> thread is handling the send requests as well.

I haven't checked the systhreads implementation (as I don't use it),
but if it's anything similar to the vmthreads implementation, it is
not very high performance.

For example: to reschedule a thread, vmthreads a) uses select (not
sure if this is a problem), and b) does a linear search for a runnable
thread (I can't remember if it stops early, I'd have to have another
check).

The point is, in this case, you'd need to rewrite it to use thread
queues for the various states to get something closer to an O(1)
scheduler.

Jonathan

Re: [Caml-list] Re: async networking

[Rick Richardson]

On Wed, 2006-02-08 at 11:03 +1300, Jonathan Roewen wrote:
> > The multiple serving threads could actually make for an simple api,
> > actually. A simple function to add a receive callback for a port would
> > be all you'd need. You could even pass in a buffer to that callback that
> > a person could respond directly to for socket send, since the same
> > thread is handling the send requests as well.
> 
> I haven't checked the systhreads implementation (as I don't use it),
> but if it's anything similar to the vmthreads implementation, it is
> not very high performance.
> 
> For example: to reschedule a thread, vmthreads a) uses select (not
> sure if this is a problem), and b) does a linear search for a runnable
> thread (I can't remember if it stops early, I'd have to have another
> check).
> 
> The point is, in this case, you'd need to rewrite it to use thread
> queues for the various states to get something closer to an O(1)
> scheduler.
I technically don't need to officially reschedule.. since they will all
be doing the same thing, I'll sleep the thread, then when data becomes
available pass it the new fd and wake it up. A simple queue would be
fine for that. 

> 
> Jonathan

Re: [Caml-list] Re: async networking

[skaller]

On Tue, 2006-02-07 at 13:29 -0800, Rick Richardson wrote:

> My only interest lies in high performance networking, specifically in a
> high connection / low data volume scenario (at this juncture at least).
> The optimal solution is some form of iocp with multiple threads (1 per
> open socket initiated at startup). 

The optimal solution is to use MLton, Haskell, or Felix,
which all give you a threading API without needing pthreads.

Ocaml bytecode with vmthreads does too, but I think it needs to be 
fixed to schedule using event notifications instead of select.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Re: async networking

[Markus Mottl]

On 2/7/06, Rick Richardson <rick@eltopia.com> wrote:
> I technically don't need to officially reschedule.. since they will all
> be doing the same thing, I'll sleep the thread, then when data becomes
> available pass it the new fd and wake it up. A simple queue would be
> fine for that.

You had mentioned that you mostly care about performance in an
environment with low data volume and a high number of transactions
(connections).  Passing off data between threads requires a
context-switch, which is going to kill your performance if you have to
do that very often (as is the case here).

It would be better to attempt "straight-through processing" (STP): let
the thread that initially received the data execute the job, and only
if this would take too long (e.g. because it might block) should you
consider passing off the work to a helper thread.  In that case it
would be advisable to use two queues if you want to reduce the number
of locks required to access new jobs.  By using STP you can quite
likely improve performance by an order of magnitude for your kind of
problem.

We have implemented a library for STP and are probably going to
release it to the general public within the next couple of weeks
anyway, but the hints above might still be useful to you if you want
to implement a solution yourself (good exercise! ;).

Regards,
Markus

--
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com