Re: [Caml-list] Re: thousands of CPU cores

[Kuba Ober <ober.14@osu.edu>]

On Tuesday 15 July 2008, you wrote:
> Am Dienstag, den 15.07.2008, 11:57 -0400 schrieb Kuba Ober:
> > On Thursday 10 July 2008, Gerd Stolpmann wrote:
> > > Am Donnerstag, den 10.07.2008, 21:02 +0000 schrieb Sylvain Le Gall:
> > > > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > > > Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> > > > >> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > > >> > In Ocaml you can exploit multi-core currently only by using
> > > > >> > multi-processing parallel programs that communicate over message
> > > > >> > passing (and only on Unix). Actually, it's an excellent language
> > > > >> > for this style.
> > > > >>
> > > > >> Why only on Unix ?
> > > > >
> > > > > No fork() on Windows. And emulating its effects is hard.
> > > >
> > > > open_process + stdin/stdout should do the trick... at least i think
> > > > so.
> > >
> > > After having ported godi to mingw I am not sure whether this works at
> > > all. The point is that you usually want to inherit OS resources to the
> > > child process (e.g. sockets). The CreateProcess Win32 call
> > > (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> > > that you can inherit handles, but I would be careful with the
> > > information given in MSDN. Often it works only as far as the presented
> > > examples. Windows isn't written for multi-processing, and its syscalls
> > > aren't as orthogonal as in Unix-type systems.
> >
> > Windows syscalls are quite reasonable IMHO, if a tad undocumented.
> > ReactOS folk have done a great job of reimplementing most of them, and
> > there isn't anything mucho broken about those. In fact, I'd posit that
> > Windows native syscalls expose some functionality that's traditionally
> > unavailable on unices and requires hacks to achieve (usually via
> > executable code injection). Just look at what Wine folks had to do in
> > order to emulate some win32 (not even native!) API on Linux: a lot of
> > hard work for what amounts to a single API call. This of course works
> > both ways, and on Windows, while a fork() implementation is simple, it
> > AFAIK requires a custom loader or some other ingenuity to work.
>
> Sure, both systems follow different philosophies.
>
> > > Furthermore, it looks like a pain in the ass - often you want to run
> > > some initialization code, and without fork() you have to run it as
> > > often as you start processes.
> >
> > On Windows, there's the native API, which is then used by the win32
> > subsystem and posix subsystems to do the job. Native API allows fork()
> > implementations mostly on par with what you get on Unices. MS has a posix
> > subsystem on which fork() performs in the same ballpark as fork() on
> > linux, and make Cygwin's fork() look bad like it deserves. About the only
> > good thing about Cygwin's fork() is that it works on win9x.
>
> Well, there's now SFU for Windows (but only for XP Professional and
> Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
> when you want to run Win32 and POSIX programs on the same system, and
> maybe an alternative to using virtualization. But it is nothing for
> developing consumer programs on Windows.
>
> Btw, has something tried to compile O'Caml on SFU? It's a 230M free
> download. There seems to be gcc and lots of GNU stuff, too (yes, it's
> from MS...).
>
> > > Also, Windows is just a bad platform for event-based programs, and you
> > > want to do it to some extent (e.g. for watching all your child
> > > processes). Only for socket handles there is a select() call. For all
> > > other types of handles you cannot find out in advance whether the
> > > operation would block or not.
> >
> > This is misinformation at best, FUD at worst. I'm no Microsoft fanboy,
> > but the reality is quite opposite to what you claim. Windows has quite
> > robust asynchronous I/O support.
>
> No, this is not misinformation, this is the result of digging deeply
> into the Win32 API for an attempt to port Ocamlnet to Win32 (which will
> finally happen to some degree).

If you limit yourself to Win32 API, you're right that you won't get it
to work the way you want to. As soon as you start digging into the DDK,
there are ways to do it.

> There's overlapped I/O, but the 
> difficulty is that you have to start an operation before you can watch
> asynchronously for its completion. There is no way to check in advance
> for that (and that was my claim). Also, there is a quite small limit for
> the number of resources you can watch at the same time (I think 32 or
> 64).
>
> Look at what Cygwin has done. Basically, they start helper threads for
> emulating select(). For some cases, there is no real select() support,
> e.g. the output side of pipes is always considered as writable. Only the
> input side can be watched.

A lot of that is done since they limit themselves to the API, and old
API at that. SFU achieves it without much in the way of limits by using
admittedly undocumented, albeit relatively understandable native APIs,
and by gaining more direct access to the network stack. There's no reason
nowadays why Cygwin folks couldn't do the same.

Cheers, Kuba