[Caml-list] Re: thousands of CPU cores

[Caml-list] Re: thousands of CPU cores

[Damien Guichard]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]


> And I am not sure that Ocaml has been ported to non-unix free 
> software... Does Ocaml runs on Hurd, or Syllable, or EROS?

OCaml 3.10.2 binary package for SyllableOS: http://web.syllable.org/pages/resources.html 

Regards.
-- damien

[-- Attachment #2: Type: text/html, Size: 761 bytes --]

Re: [Caml-list] Re: thousands of CPU cores

[Kuba Ober]

On Tuesday 15 July 2008, you wrote:
> Am Dienstag, den 15.07.2008, 11:57 -0400 schrieb Kuba Ober:
> > On Thursday 10 July 2008, Gerd Stolpmann wrote:
> > > Am Donnerstag, den 10.07.2008, 21:02 +0000 schrieb Sylvain Le Gall:
> > > > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > > > Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> > > > >> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > > >> > In Ocaml you can exploit multi-core currently only by using
> > > > >> > multi-processing parallel programs that communicate over message
> > > > >> > passing (and only on Unix). Actually, it's an excellent language
> > > > >> > for this style.
> > > > >>
> > > > >> Why only on Unix ?
> > > > >
> > > > > No fork() on Windows. And emulating its effects is hard.
> > > >
> > > > open_process + stdin/stdout should do the trick... at least i think
> > > > so.
> > >
> > > After having ported godi to mingw I am not sure whether this works at
> > > all. The point is that you usually want to inherit OS resources to the
> > > child process (e.g. sockets). The CreateProcess Win32 call
> > > (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> > > that you can inherit handles, but I would be careful with the
> > > information given in MSDN. Often it works only as far as the presented
> > > examples. Windows isn't written for multi-processing, and its syscalls
> > > aren't as orthogonal as in Unix-type systems.
> >
> > Windows syscalls are quite reasonable IMHO, if a tad undocumented.
> > ReactOS folk have done a great job of reimplementing most of them, and
> > there isn't anything mucho broken about those. In fact, I'd posit that
> > Windows native syscalls expose some functionality that's traditionally
> > unavailable on unices and requires hacks to achieve (usually via
> > executable code injection). Just look at what Wine folks had to do in
> > order to emulate some win32 (not even native!) API on Linux: a lot of
> > hard work for what amounts to a single API call. This of course works
> > both ways, and on Windows, while a fork() implementation is simple, it
> > AFAIK requires a custom loader or some other ingenuity to work.
>
> Sure, both systems follow different philosophies.
>
> > > Furthermore, it looks like a pain in the ass - often you want to run
> > > some initialization code, and without fork() you have to run it as
> > > often as you start processes.
> >
> > On Windows, there's the native API, which is then used by the win32
> > subsystem and posix subsystems to do the job. Native API allows fork()
> > implementations mostly on par with what you get on Unices. MS has a posix
> > subsystem on which fork() performs in the same ballpark as fork() on
> > linux, and make Cygwin's fork() look bad like it deserves. About the only
> > good thing about Cygwin's fork() is that it works on win9x.
>
> Well, there's now SFU for Windows (but only for XP Professional and
> Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
> when you want to run Win32 and POSIX programs on the same system, and
> maybe an alternative to using virtualization. But it is nothing for
> developing consumer programs on Windows.
>
> Btw, has something tried to compile O'Caml on SFU? It's a 230M free
> download. There seems to be gcc and lots of GNU stuff, too (yes, it's
> from MS...).
>
> > > Also, Windows is just a bad platform for event-based programs, and you
> > > want to do it to some extent (e.g. for watching all your child
> > > processes). Only for socket handles there is a select() call. For all
> > > other types of handles you cannot find out in advance whether the
> > > operation would block or not.
> >
> > This is misinformation at best, FUD at worst. I'm no Microsoft fanboy,
> > but the reality is quite opposite to what you claim. Windows has quite
> > robust asynchronous I/O support.
>
> No, this is not misinformation, this is the result of digging deeply
> into the Win32 API for an attempt to port Ocamlnet to Win32 (which will
> finally happen to some degree).

If you limit yourself to Win32 API, you're right that you won't get it
to work the way you want to. As soon as you start digging into the DDK,
there are ways to do it.

> There's overlapped I/O, but the 
> difficulty is that you have to start an operation before you can watch
> asynchronously for its completion. There is no way to check in advance
> for that (and that was my claim). Also, there is a quite small limit for
> the number of resources you can watch at the same time (I think 32 or
> 64).
>
> Look at what Cygwin has done. Basically, they start helper threads for
> emulating select(). For some cases, there is no real select() support,
> e.g. the output side of pipes is always considered as writable. Only the
> input side can be watched.

A lot of that is done since they limit themselves to the API, and old
API at that. SFU achieves it without much in the way of limits by using
admittedly undocumented, albeit relatively understandable native APIs,
and by gaining more direct access to the network stack. There's no reason
nowadays why Cygwin folks couldn't do the same.

Cheers, Kuba

Re: [Caml-list] Re: thousands of CPU cores

[Erik de Castro Lopo]

Richard Jones wrote:

> On a different, but not unrelated topic, Debian have a cross-compiler
> (based on MinGW) so you don't need to leave the safety & comfort of
> Linux in order to build Windows DLLs and binaries.
> 
>   http://packages.debian.org/search?keywords=mingw32

I am the main author of a libsndfile (a library for reading/writing
audio files like WAV, AIFF etc) written in C and widely used across
all the major platforms.

I have recently switched to doing all my windows builds for libsndfile
on a Debian/Ubuntu box, cross-compiling using these MinGW tools and
running the test suite under Wine (the windows emulator).

For me, this is about 100 times easier than dealing with the pain
that is windows.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"Arguing that Java is better than C++ is like arguing that
grasshoppers taste better than tree bark." -- Thant Tessman

Re: [Caml-list] Re: thousands of CPU cores

[Gerd Stolpmann]

Am Mittwoch, den 16.07.2008, 10:59 +0200 schrieb Michaël Grünewald:
> Gerd Stolpmann wrote:
> 
> > Well, there's now SFU for Windows (but only for XP Professional and
> > Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
> > when you want to run Win32 and POSIX programs on the same system, and
> > maybe an alternative to using virtualization. But it is nothing for
> > developing consumer programs on Windows.
> > 
> > Btw, has something tried to compile O'Caml on SFU? It's a 230M free
> > download. There seems to be gcc and lots of GNU stuff, too (yes, it's
> > from MS...).
> 
> I did this a few monthes ago, I followed the NetBSD way, since SFU is 
> supported by NetBSD's `pkgsrc'. This was really *easy*, thanks to the 
> efforts of the `pkgsrc' maintainers. However, I did not play that much 
> with the system, my point was to test SFU by running very Unix-oriented 
> and complex proecdures in it.
> 
> See http://www.netbsd.org/docs/software/packages.html
> for general information about NetBSD's pkgsrc; Microsoft SFU is refered 
> to as Interix here, e.g. in the ``Supported Platforms'' section.

Good to know.

> The `pkgsrc' software is a port infrastructure similar to what is found 
> on *BSD and MacPorts, if you have used one of them, you certainly will 
> feel comfortable with `pkgsrc'. Documentation for `pkgsrc' is available 
> at http://www.netbsd.org/docs/pkgsrc/, besides the introduction, see 
> especially sections 3.2 (Bootstrapping) and 4.2 (Installing ports), it 
> shall be enough to get started!

Yes, I know pkgsrc very well. I used it years ago to build software on
Solaris. Later I took it as starting point for GODI.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: thousands of CPU cores

[Richard Jones]

On a different, but not unrelated topic, Debian have a cross-compiler
(based on MinGW) so you don't need to leave the safety & comfort of
Linux in order to build Windows DLLs and binaries.

  http://packages.debian.org/search?keywords=mingw32

Fedora are going to offer a MinGW cross-compiler and libraries too,
shortly:

  https://fedoraproject.org/wiki/SIGs/MinGW

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Re: thousands of CPU cores

[Michaël Grünewald]

Gerd Stolpmann wrote:

> Well, there's now SFU for Windows (but only for XP Professional and
> Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
> when you want to run Win32 and POSIX programs on the same system, and
> maybe an alternative to using virtualization. But it is nothing for
> developing consumer programs on Windows.
> 
> Btw, has something tried to compile O'Caml on SFU? It's a 230M free
> download. There seems to be gcc and lots of GNU stuff, too (yes, it's
> from MS...).

I did this a few monthes ago, I followed the NetBSD way, since SFU is 
supported by NetBSD's `pkgsrc'. This was really *easy*, thanks to the 
efforts of the `pkgsrc' maintainers. However, I did not play that much 
with the system, my point was to test SFU by running very Unix-oriented 
and complex proecdures in it.

See http://www.netbsd.org/docs/software/packages.html
for general information about NetBSD's pkgsrc; Microsoft SFU is refered 
to as Interix here, e.g. in the ``Supported Platforms'' section.

The `pkgsrc' software is a port infrastructure similar to what is found 
on *BSD and MacPorts, if you have used one of them, you certainly will 
feel comfortable with `pkgsrc'. Documentation for `pkgsrc' is available 
at http://www.netbsd.org/docs/pkgsrc/, besides the introduction, see 
especially sections 3.2 (Bootstrapping) and 4.2 (Installing ports), it 
shall be enough to get started!
-- 
Cheers,
Michaël

Re: [Caml-list] Re: thousands of CPU cores

[Adrien]

2008/7/15 Adrien <camaradetux@gmail.com>:
> 2008/7/15 Gerd Stolpmann <info@gerd-stolpmann.de>:
>>
>> Well, there's now SFU for Windows (but only for XP Professional and
>> Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
>> when you want to run Win32 and POSIX programs on the same system, and
>> maybe an alternative to using virtualization. But it is nothing for
>> developing consumer programs on Windows.
>>
>> Btw, has something tried to compile O'Caml on SFU? It's a 230M free
>> download. There seems to be gcc and lots of GNU stuff, too (yes, it's
>> from MS...).
>
> Well, I did a few months ago. For those who don't know, SFU (Services
> For Unix) provides approximately the same features as cygwin.
> SFU works really well and is professional : you use the installer and
> you're done. It's usually easier to get an SFU build than a mingw or a
> cygwin one.
>
> There's a catch however : it works too well !
> It's certainly surprising but SFU gives you an Unix-like environment
> and you'll certainly try to interact with windows and that's where the
> problem lies. For example, paths will clash iirc.
> However it is worth being investigated. These are s-nt.h :
>
> First, the official one for windows :
>
> #define OCAML_OS_TYPE "Win32"
>
> #undef BSD_SIGNALS
> #define HAS_STRERROR
> #define HAS_SOCKETS
> #define HAS_GETCWD
> #define HAS_UTIME
> #define HAS_DUP2
> #define HAS_GETHOSTNAME
> #define HAS_MKTIME
> #define HAS_PUTENV
> #define HAS_LOCALE
>
>
> Then the one you can obtain when configure'ing under SFU :
>
> #define OCAML_OS_TYPE "Unix"
> #define OCAML_STDLIB_DIR "/usr/local/lib/ocaml"
> #define POSIX_SIGNALS
> #define HAS_GETRUSAGE
> #define HAS_TIMES
> #define HAS_TERMCAP
> #define HAS_SOCKETS
> #define HAS_INET_ATON
> #define HAS_UNISTD
> #define HAS_OFF_T
> #define HAS_DIRENT
> #define HAS_REWINDDIR
> #define HAS_LOCKF
> #define HAS_MKFIFO
> #define HAS_GETCWD
> #define HAS_GETWD
> #define HAS_GETPRIORITY
> #define HAS_UTIME
> #define HAS_DUP2
> #define HAS_FCHMOD
> #define HAS_TRUNCATE
> #define HAS_SYS_SELECT_H
> #define HAS_SELECT
> #define HAS_SYMLINK
> #define HAS_WAITPID
> #define HAS_GETGROUPS
> #define HAS_TERMIOS
> #define HAS_SETITIMER
> #define HAS_GETHOSTNAME
> #define HAS_UNAME
> #define HAS_GETTIMEOFDAY
> #define HAS_MKTIME
> #define HAS_SETSID
> #define HAS_PUTENV
> #define HAS_LOCALE
> #define HAS_MMAP
> #define HAS_SIGWAIT
>
>
> It's a bit longer !
>
> I have since uninstalled SFU but could reinstall it, virtualBox will
> be handy (except that compiling software inside a virtual machine is
> everything but funny).
> By the way, SFU is available for Vista and should work under XP Home
> (Home is like Pro with a few things removed). Microsoft announced
> there would be no other version of SFU [1] but SFU is not their
> product, it is Interix's and I think Interix will continue to ship new
> versions though these probably won't be free.
>
> Also, the utilities are not necessarily Gnu, they provide BSD ones. ;)
> But, but, ... I just saw debian-interix which claims "since the last
> buildd run there are now over 1000 packages available for
> interix-i386", and even better, the project is still active, the
> latest is dated from "2008-06-30". I definitely have to install SFU
> again ! (I'll just need a windows partition bigger than 4GB...).
>
> [1] : http://www.microsoft-watch.com/content/operating_systems/its_the_end_of_the_line_for_microsofts_services_for_unix_product.html
>
>
> The not-so-funny part is that applications compiled under SFU need SFU
> installed to run so this limits portability but ocaml-compiled
> programs may not have this problem. The only way to know more about
> this is certainly to experiment. Also, I don't know how free is SFU's
> license.
>
>
>  ---
>
> Adrien Nader
>


It may not run on XP Home contrary to what I stated. Debian-interix
says the following in its INSTALL file :
  * On Windows XP make sure, you
     DON'T "Use simple file sharing"
XP Home can only "use simple file sharing" so sfu may not work on XP
Home (though you can get the required functionnality).


 ---

Adrien Nader

Re: [Caml-list] Re: thousands of CPU cores

[Adrien]

2008/7/15 Gerd Stolpmann <info@gerd-stolpmann.de>:
>
> Well, there's now SFU for Windows (but only for XP Professional and
> Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
> when you want to run Win32 and POSIX programs on the same system, and
> maybe an alternative to using virtualization. But it is nothing for
> developing consumer programs on Windows.
>
> Btw, has something tried to compile O'Caml on SFU? It's a 230M free
> download. There seems to be gcc and lots of GNU stuff, too (yes, it's
> from MS...).

Well, I did a few months ago. For those who don't know, SFU (Services
For Unix) provides approximately the same features as cygwin.
SFU works really well and is professional : you use the installer and
you're done. It's usually easier to get an SFU build than a mingw or a
cygwin one.

There's a catch however : it works too well !
It's certainly surprising but SFU gives you an Unix-like environment
and you'll certainly try to interact with windows and that's where the
problem lies. For example, paths will clash iirc.
However it is worth being investigated. These are s-nt.h :

First, the official one for windows :

#define OCAML_OS_TYPE "Win32"

#undef BSD_SIGNALS
#define HAS_STRERROR
#define HAS_SOCKETS
#define HAS_GETCWD
#define HAS_UTIME
#define HAS_DUP2
#define HAS_GETHOSTNAME
#define HAS_MKTIME
#define HAS_PUTENV
#define HAS_LOCALE


Then the one you can obtain when configure'ing under SFU :

#define OCAML_OS_TYPE "Unix"
#define OCAML_STDLIB_DIR "/usr/local/lib/ocaml"
#define POSIX_SIGNALS
#define HAS_GETRUSAGE
#define HAS_TIMES
#define HAS_TERMCAP
#define HAS_SOCKETS
#define HAS_INET_ATON
#define HAS_UNISTD
#define HAS_OFF_T
#define HAS_DIRENT
#define HAS_REWINDDIR
#define HAS_LOCKF
#define HAS_MKFIFO
#define HAS_GETCWD
#define HAS_GETWD
#define HAS_GETPRIORITY
#define HAS_UTIME
#define HAS_DUP2
#define HAS_FCHMOD
#define HAS_TRUNCATE
#define HAS_SYS_SELECT_H
#define HAS_SELECT
#define HAS_SYMLINK
#define HAS_WAITPID
#define HAS_GETGROUPS
#define HAS_TERMIOS
#define HAS_SETITIMER
#define HAS_GETHOSTNAME
#define HAS_UNAME
#define HAS_GETTIMEOFDAY
#define HAS_MKTIME
#define HAS_SETSID
#define HAS_PUTENV
#define HAS_LOCALE
#define HAS_MMAP
#define HAS_SIGWAIT


It's a bit longer !

I have since uninstalled SFU but could reinstall it, virtualBox will
be handy (except that compiling software inside a virtual machine is
everything but funny).
By the way, SFU is available for Vista and should work under XP Home
(Home is like Pro with a few things removed). Microsoft announced
there would be no other version of SFU [1] but SFU is not their
product, it is Interix's and I think Interix will continue to ship new
versions though these probably won't be free.

Also, the utilities are not necessarily Gnu, they provide BSD ones. ;)
But, but, ... I just saw debian-interix which claims "since the last
buildd run there are now over 1000 packages available for
interix-i386", and even better, the project is still active, the
latest is dated from "2008-06-30". I definitely have to install SFU
again ! (I'll just need a windows partition bigger than 4GB...).

[1] : http://www.microsoft-watch.com/content/operating_systems/its_the_end_of_the_line_for_microsofts_services_for_unix_product.html


The not-so-funny part is that applications compiled under SFU need SFU
installed to run so this limits portability but ocaml-compiled
programs may not have this problem. The only way to know more about
this is certainly to experiment. Also, I don't know how free is SFU's
license.


 ---

Adrien Nader

Re: [Caml-list] Re: thousands of CPU cores

[Gerd Stolpmann]

Am Dienstag, den 15.07.2008, 11:57 -0400 schrieb Kuba Ober:
> On Thursday 10 July 2008, Gerd Stolpmann wrote:
> > Am Donnerstag, den 10.07.2008, 21:02 +0000 schrieb Sylvain Le Gall:
> > > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > > Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> > > >> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > >> > In Ocaml you can exploit multi-core currently only by using
> > > >> > multi-processing parallel programs that communicate over message
> > > >> > passing (and only on Unix). Actually, it's an excellent language for
> > > >> > this style.
> > > >>
> > > >> Why only on Unix ?
> > > >
> > > > No fork() on Windows. And emulating its effects is hard.
> > >
> > > open_process + stdin/stdout should do the trick... at least i think so.
> >
> > After having ported godi to mingw I am not sure whether this works at
> > all. The point is that you usually want to inherit OS resources to the
> > child process (e.g. sockets). The CreateProcess Win32 call
> > (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> > that you can inherit handles, but I would be careful with the
> > information given in MSDN. Often it works only as far as the presented
> > examples. Windows isn't written for multi-processing, and its syscalls
> > aren't as orthogonal as in Unix-type systems.
> 
> Windows syscalls are quite reasonable IMHO, if a tad undocumented. ReactOS
> folk have done a great job of reimplementing most of them, and there isn't
> anything mucho broken about those. In fact, I'd posit that Windows native
> syscalls expose some functionality that's traditionally unavailable on unices
> and requires hacks to achieve (usually via executable code injection). Just
> look at what Wine folks had to do in order to emulate some win32 (not even
> native!) API on Linux: a lot of hard work for what amounts to a single API
> call. This of course works both ways, and on Windows, while a fork()
> implementation is simple, it AFAIK requires a custom loader or some other
> ingenuity to work.

Sure, both systems follow different philosophies.

> > Furthermore, it looks like a pain in the ass - often you want to run
> > some initialization code, and without fork() you have to run it as often
> > as you start processes.
> 
> On Windows, there's the native API, which is then used by the win32 subsystem
> and posix subsystems to do the job. Native API allows fork() implementations
> mostly on par with what you get on Unices. MS has a posix subsystem on which
> fork() performs in the same ballpark as fork() on linux, and make Cygwin's
> fork() look bad like it deserves. About the only good thing about Cygwin's 
> fork() is that it works on win9x.

Well, there's now SFU for Windows (but only for XP Professional and
Windows 2003, not for XP Home and Vista, AFAIK). That's a cool solution
when you want to run Win32 and POSIX programs on the same system, and
maybe an alternative to using virtualization. But it is nothing for
developing consumer programs on Windows.

Btw, has something tried to compile O'Caml on SFU? It's a 230M free
download. There seems to be gcc and lots of GNU stuff, too (yes, it's
from MS...).

> > Also, Windows is just a bad platform for event-based programs, and you
> > want to do it to some extent (e.g. for watching all your child
> > processes). Only for socket handles there is a select() call. For all
> > other types of handles you cannot find out in advance whether the
> > operation would block or not.
> 
> This is misinformation at best, FUD at worst. I'm no Microsoft fanboy,
> but the reality is quite opposite to what you claim. Windows has quite
> robust asynchronous I/O support.

No, this is not misinformation, this is the result of digging deeply
into the Win32 API for an attempt to port Ocamlnet to Win32 (which will
finally happen to some degree). There's overlapped I/O, but the
difficulty is that you have to start an operation before you can watch
asynchronously for its completion. There is no way to check in advance
for that (and that was my claim). Also, there is a quite small limit for
the number of resources you can watch at the same time (I think 32 or
64).

Look at what Cygwin has done. Basically, they start helper threads for
emulating select(). For some cases, there is no real select() support,
e.g. the output side of pipes is always considered as writable. Only the
input side can be watched.

Of course, these difficulties result from porting Unix libraries to
Windows. You can say with some right: If you want to program in the
event-based way on Windows, you must do it the Windows style. Sure, we
are running into the same problems as with fork() - different
philosophies make it problematic to write portable programs. What's
quite interesting is that the Win32 APIs are less powerful in these
areas (process creation, watching events) than the Unix counterpart.
That's my whole claim. If I had to develop programs only for Windows,
I'd do it multi-threaded because Win32 is much better there.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: thousands of CPU cores

[Kuba Ober]

On Friday 11 July 2008, Sylvain Le Gall wrote:
> On 11-07-2008, Jon Harrop <jon@ffconsultancy.com> wrote:
> > On Friday 11 July 2008 07:26:44 Sylvain Le Gall wrote:
> >> On 10-07-2008, Oliver Bandel <oliver@first.in-berlin.de> wrote:
> >
> > However, any serious power users will already be on 64-bit where these
> > limits have been relegated to quaint stories your grandpa will tell you.
>
> As you cannot ignore people running on Windows, you cannot ignore people
> running on older hardware.
>
> If you plan to program a big DB that will use more than 3GB on 32 bits
> hardware, you should take care of this memory limit and consider
> splitting your application into several process...

Re-mapping stuff into and out of virtual memory space is relatively trivial
and doesn't require multiple processes. Multiple processes only give you
the advantage of having all this memory mapped at once, split across
processes though. So it may or may not be an actual improvement, depending
on the application...

Cheers, Kuba

Re: [Caml-list] Re: thousands of CPU cores

[Kuba Ober]

On Thursday 10 July 2008, Gerd Stolpmann wrote:
> Am Donnerstag, den 10.07.2008, 21:02 +0000 schrieb Sylvain Le Gall:
> > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> > >> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > >> > In Ocaml you can exploit multi-core currently only by using
> > >> > multi-processing parallel programs that communicate over message
> > >> > passing (and only on Unix). Actually, it's an excellent language for
> > >> > this style.
> > >>
> > >> Why only on Unix ?
> > >
> > > No fork() on Windows. And emulating its effects is hard.
> >
> > open_process + stdin/stdout should do the trick... at least i think so.
>
> After having ported godi to mingw I am not sure whether this works at
> all. The point is that you usually want to inherit OS resources to the
> child process (e.g. sockets). The CreateProcess Win32 call
> (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> that you can inherit handles, but I would be careful with the
> information given in MSDN. Often it works only as far as the presented
> examples. Windows isn't written for multi-processing, and its syscalls
> aren't as orthogonal as in Unix-type systems.

Windows syscalls are quite reasonable IMHO, if a tad undocumented. ReactOS
folk have done a great job of reimplementing most of them, and there isn't
anything mucho broken about those. In fact, I'd posit that Windows native
syscalls expose some functionality that's traditionally unavailable on unices
and requires hacks to achieve (usually via executable code injection). Just
look at what Wine folks had to do in order to emulate some win32 (not even
native!) API on Linux: a lot of hard work for what amounts to a single API
call. This of course works both ways, and on Windows, while a fork()
implementation is simple, it AFAIK requires a custom loader or some other
ingenuity to work.

> Furthermore, it looks like a pain in the ass - often you want to run
> some initialization code, and without fork() you have to run it as often
> as you start processes.

On Windows, there's the native API, which is then used by the win32 subsystem
and posix subsystems to do the job. Native API allows fork() implementations
mostly on par with what you get on Unices. MS has a posix subsystem on which
fork() performs in the same ballpark as fork() on linux, and make Cygwin's
fork() look bad like it deserves. About the only good thing about Cygwin's 
fork() is that it works on win9x.

> Also, Windows is just a bad platform for event-based programs, and you
> want to do it to some extent (e.g. for watching all your child
> processes). Only for socket handles there is a select() call. For all
> other types of handles you cannot find out in advance whether the
> operation would block or not.

This is misinformation at best, FUD at worst. I'm no Microsoft fanboy,
but the reality is quite opposite to what you claim. Windows has quite
robust asynchronous I/O support.

Cheers, Kuba

Re: [Caml-list] Re: thousands of CPU cores

[Kuba Ober]

On Thursday 10 July 2008, Gerd Stolpmann wrote:
> Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > > In Ocaml you can exploit multi-core currently only by using
> > > multi-processing parallel programs that communicate over message
> > > passing (and only on Unix). Actually, it's an excellent language for
> > > this style.
> >
> > Why only on Unix ?
>
> No fork() on Windows. And emulating its effects is hard.
>
> I would subsume Cygwin under "pseudo-Unix", and its fork emulation is so
> slow that it would be a problem for speedy programs.

AFAIK, Cygwin's fork() emulation is quite limited since Cygwin didn't go the 
way of doing a custom process loader. I.e. instead of having a *tiny* 
statically linked loader executable which then actually brings in code and
data pages into the process space (and would allow sharing them copy-on-write
with other processes), they just use Windows for that, and that's why they
have to emulate (and make up) their process IDs. Doing an exec() when you have
your own loader is trivial: just tell the loader to deallocate all of
the process's virtual memory (save for that of the loader's), and load 
something else instead.

I have fiddled some time ago with a very basic custom loader which uses native
API and it worked OK for what it did. Of course Cygwin has to work under 
win 95, so it has no access to native API there, but on anything modern it
could easily have speeds comparable to native. I mean, Microsoft has
implemented the Posix subsystem using same native API and its fork() works
just fine (read: way better than Cygwin's).

I have last delved into all this ~5+ years ago, so things may have changed
in the meantime...

Cheers, Kuba

Re: [Caml-list] Re: thousands of CPU cores

[Jon Harrop]

On Friday 11 July 2008 07:26:44 Sylvain Le Gall wrote:
> On 10-07-2008, Oliver Bandel <oliver@first.in-berlin.de> wrote:
> > Using multi-processes instead of multi-threads is the
> > usual way on Unix, and it has a lot of advantages.
> > Threads-apologetes often say, threads are the ultimative
> > technology... but processes have the advantage of encapsulation
> > of the wohole environment of the program.
>
> There is also the fact that using multi process allow you to go further
> than the memory limit per process...

Yes.

> (3GB for Linux/

Is that for 32-bit Linux?

> 1GB for Windows)...  

32-bit Windows XP has a 2Gb default process memory limit:

  http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
  http://msdn.microsoft.com/en-us/library/aa366778.aspx

32-bit Windows Server can be increased to 3Gb.

However, any serious power users will already be on 64-bit where these limits 
have been relegated to quaint stories your grandpa will tell you.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Re: thousands of CPU cores

[Gerd Stolpmann]

Am Donnerstag, den 10.07.2008, 22:35 +0100 schrieb Jon Harrop:
> On Thursday 10 July 2008 22:19:05 Gerd Stolpmann wrote:
> > After having ported godi to mingw I am not sure whether this works at
> > all. The point is that you usually want to inherit OS resources to the
> > child process (e.g. sockets). The CreateProcess Win32 call
> > (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> > that you can inherit handles, but I would be careful with the
> > information given in MSDN. Often it works only as far as the presented
> > examples. Windows isn't written for multi-processing, and its syscalls
> > aren't as orthogonal as in Unix-type systems.
> >
> > Furthermore, it looks like a pain in the ass - often you want to run
> > some initialization code, and without fork() you have to run it as often
> > as you start processes.
> >
> > Also, Windows is just a bad platform for event-based programs, and you
> > want to do it to some extent (e.g. for watching all your child
> > processes). Only for socket handles there is a select() call. For all
> > other types of handles you cannot find out in advance whether the
> > operation would block or not.
> >
> > So... if there is any chance you can select the OS, keep away from
> > Windows for this type of program.
> 
> I think your conclusion needs qualification.
> 
> You are trying to shoehorn an existing process-based Linux solution onto 
> Windows and discovering that it does not work well. However, Windows already 
> provides different ways of achieving the same thing, e.g. F# with its 
> first-class events and built-in asynchronous programming syntax using 
> the .NET thread pool and mailboxes for message passing.
> 
> Moreover, the idiomatic solution on Windows is almost certainly faster than 
> anything you can reasonably write under Linux in the absence of a concurrent 
> GC (i.e. off the JVM).

You are right, this is pretty much incomparable. MS has its own way of
doing things, making it hard to write portable code. I mean other OS
vendors managed to jump on the POSIX train.

We should stop here, it's getting off-topic.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: thousands of CPU cores

[Jon Harrop]

On Thursday 10 July 2008 22:19:05 Gerd Stolpmann wrote:
> After having ported godi to mingw I am not sure whether this works at
> all. The point is that you usually want to inherit OS resources to the
> child process (e.g. sockets). The CreateProcess Win32 call
> (http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
> that you can inherit handles, but I would be careful with the
> information given in MSDN. Often it works only as far as the presented
> examples. Windows isn't written for multi-processing, and its syscalls
> aren't as orthogonal as in Unix-type systems.
>
> Furthermore, it looks like a pain in the ass - often you want to run
> some initialization code, and without fork() you have to run it as often
> as you start processes.
>
> Also, Windows is just a bad platform for event-based programs, and you
> want to do it to some extent (e.g. for watching all your child
> processes). Only for socket handles there is a select() call. For all
> other types of handles you cannot find out in advance whether the
> operation would block or not.
>
> So... if there is any chance you can select the OS, keep away from
> Windows for this type of program.

I think your conclusion needs qualification.

You are trying to shoehorn an existing process-based Linux solution onto 
Windows and discovering that it does not work well. However, Windows already 
provides different ways of achieving the same thing, e.g. F# with its 
first-class events and built-in asynchronous programming syntax using 
the .NET thread pool and mailboxes for message passing.

Moreover, the idiomatic solution on Windows is almost certainly faster than 
anything you can reasonably write under Linux in the absence of a concurrent 
GC (i.e. off the JVM).

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Re: thousands of CPU cores

[Gerd Stolpmann]

Am Donnerstag, den 10.07.2008, 21:02 +0000 schrieb Sylvain Le Gall:
> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> >
> > Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> >> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> >> > In Ocaml you can exploit multi-core currently only by using
> >> > multi-processing parallel programs that communicate over message passing
> >> > (and only on Unix). Actually, it's an excellent language for this style.
> >> 
> >> Why only on Unix ?
> >
> > No fork() on Windows. And emulating its effects is hard.
> >
> 
> open_process + stdin/stdout should do the trick... at least i think so.

After having ported godi to mingw I am not sure whether this works at
all. The point is that you usually want to inherit OS resources to the
child process (e.g. sockets). The CreateProcess Win32 call
(http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx) mentions
that you can inherit handles, but I would be careful with the
information given in MSDN. Often it works only as far as the presented
examples. Windows isn't written for multi-processing, and its syscalls
aren't as orthogonal as in Unix-type systems.

Furthermore, it looks like a pain in the ass - often you want to run
some initialization code, and without fork() you have to run it as often
as you start processes.

Also, Windows is just a bad platform for event-based programs, and you
want to do it to some extent (e.g. for watching all your child
processes). Only for socket handles there is a select() call. For all
other types of handles you cannot find out in advance whether the
operation would block or not.

So... if there is any chance you can select the OS, keep away from
Windows for this type of program.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: thousands of CPU cores

[Jon Harrop]

On Thursday 10 July 2008 21:48:54 Basile STARYNKEVITCH wrote:
> Sylvain Le Gall wrote:
> > On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> >> In Ocaml you can exploit multi-core currently only by using
> >> multi-processing parallel programs that communicate over message passing
> >> (and only on Unix). Actually, it's an excellent language for this style.
> >
> > Why only on Unix ?
>
> Rumors (and my remembering of old benchmarks) say than inter-process
> communication -using pipes, fifos, or unix sockets- on recent Unix
> systems (in particular Linux) are significantly faster than the
> equivalent Windows counterpart.

While that may be true, no Windows developer in their right mind would use 
such Linux technologies when Windows is pioneering vastly simpler and faster 
alternatives: .NET and the TPL.

> (I am still not sure that the current implementation of Ocaml would
> nicely run on a thousand cores machine; and I am not sure that such a
> machine would run the current Linux)

Linux is run on today's thousand-core shared-memory supercomputers.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Re: thousands of CPU cores

[Basile STARYNKEVITCH]

Sylvain Le Gall wrote:
> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
>> In Ocaml you can exploit multi-core currently only by using
>> multi-processing parallel programs that communicate over message passing
>> (and only on Unix). Actually, it's an excellent language for this style.
> 
> Why only on Unix ?


Rumors (and my remembering of old benchmarks) say than inter-process 
communication -using pipes, fifos, or unix sockets- on recent Unix 
systems (in particular Linux) are significantly faster than the 
equivalent Windows counterpart.

Also, I don't know Windows but it could happen that some system calls in 
Windows useful for communication might be missing in Ocaml (but I really 
don't know; I never coded on Windows).

Besides, who want to code parallel program with Ocaml or Jocaml on non 
Unix plateforms? :-) :-)

And I am not sure that Ocaml has been ported to non-unix free 
software... Does Ocaml runs on Hurd, or Syllable, or EROS?

But don't trust me on all that. I am only familiar with Unix. Ask some 
more knowledgable guy!

(I am still not sure that the current implementation of Ocaml would 
nicely run on a thousand cores machine; and I am not sure that such a 
machine would run the current Linux)


Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [Caml-list] Re: thousands of CPU cores

[Gerd Stolpmann]

Am Donnerstag, den 10.07.2008, 20:07 +0000 schrieb Sylvain Le Gall:
> On 10-07-2008, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > In Ocaml you can exploit multi-core currently only by using
> > multi-processing parallel programs that communicate over message passing
> > (and only on Unix). Actually, it's an excellent language for this style.
> 
> Why only on Unix ?

No fork() on Windows. And emulating its effects is hard.

I would subsume Cygwin under "pseudo-Unix", and its fork emulation is so
slow that it would be a problem for speedy programs.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------