GODI News: RocketBoost Beta

[Gerd Stolpmann <info@gerd-stolpmann.de>]

Hi,

In the past weeks I've worked hard to finish the next GODI release,
focusing on portability. A beta version of the new bootstrap and
godi_console, called RocketBoost, is now available, and it would be
great if it were tested at large.

There are not many new features in this release, so there is no reason
to switch to it if you already have GODI.

The big news is that GODI now supports the MinGW port of OCaml for
Windows (besides the Cygwin port). This means that it is now possible to
create native Windows applications with GODI. Note, however, that the
porting effort is still in its beginning - GODI itself works, but most
packages aren't ported yet, and won't work.

The MinGW support has become possible because large parts of the GODI
core have been rewritten. In particular, the C programs accompanying
godi_console are now coded in O'Caml (godi_make, godi_add, etc.), so
porting was possible by enhancing a few O'Caml libraries. This
refactoring of GODI also increases the portability in the Unix world -
effectively it should now run on all platforms where OCaml runs, where
gcc is available, and where POSIX-compliant shell utilities are
available.

If you would like to test it, follow these instructions. For all ports,
you need the bootstrap tarball from:

http://download.camlcity.org/download/godi-rocketboost-20080615.tar.gz



--- Installation for Unix ---

For Unix platforms (including MacOSX), just download
godi-rocketboost-20080615.tar.gz, unpack it, run ./bootstrap, and follow
the instructions.



--- Installation for Windows ---

For Windows you always need Cygwin, even for the MinGW port (it is
needed as scripting environment, please don't question that). Get it
from cygwin.com. Install everything that is needed (binutils, bzip2,
diffutils, file, findutils, gawk, gcc-core, gcc-mingw-core, grep, groff,
gzip, m4, make, man, mingw-runtime, patch, rxvt, sed, tar, w32api, wget
- hope this list is complete).

IMPORTANT: When you install software you need for GODI, choose paths
that do not contain space characters. This is incompatible with the
shell scripts. So don't install into "My Files".

Download godi-rocketboost-20080615.tar.gz, and unpack it:

$ tar xzf godi-rocketboost-20080615.tar.gz
$ cd godi-rocketboost-20080615

Now invoke bootstrap. You probably want to set the path where it is
going to be installed with -prefix. Furthermore, you can now select
whether you want the Cygwin or the MinGW port. For the latter, pass
"-w32port mingw" as extra argument.

IMPORTANT: Remember that GODI itself relies on Cygwin scripting. If you
pass paths to GODI (including godi_console, godi_add, etc.) these are
Cygwin paths, even if you build the MinGW port.

$ ./bootstrap -prefix /home/gerd/godi -w32port mingw

This will build boot_console. Then you need an Internet connection, and
do:

$ export PATH=/home/gerd/godi/bin:/home/gerd/godi/sbin:$PATH
$ ./bootstrap_stage2

This will install the minimal GODI core.

IMPORTANT: godi_console isn't ported to the Windows console window, and
for now only supports terminal windows that can deal with ANSI control
codes. Use Cygwin's rxvt as terminal (or any other capable terminal like
xterm). Furthermore, there is an issue with the MinGW port setting some
terminal features. As workaround, you have to set the TTY environment
variable to the Cygwin tty device, i.e. TTY=`tty`.

Note that the MinGW port is only available for ocaml-3.10. Passing
"-section 3.09" to bootstrap won't work.


--- List of problems ---

The final part of this message lists some problems that popped up during
the MinGW porting effort. I have often found workarounds, but maybe the
reader knows better solutions.

* Cygwin interoperability

A lot of the porting effort was about Cygwin interoperability.
godi_console (which is a native Win32 binary) can translate Cygwin paths
to native Windows paths (by reading the Cygwin mount table in the
registry). However, some problems turned out to be hard:

Starting a Cygwin binary with CreateProcess from godi_console turned out
to be unreliable when godi_console was itself called from a Cygwin
program. So the calling chain Cygwin_pgm -> Win32_pgm -> Cygwin_pgm
did not always work (symptom: the called program immediately exits). A
workaround was to use cmd.exe as intermediate instance:
Cygwin_pgm -> Win32_pgm -> cmd.exe -> Cygwin_pgm

Maybe related to that, it wasn't possible to run stty without explicit
device argument. (godi_console needs to call it to talk to the Cygwin
tty driver.) stty (and tty, too) always complained that stdin wasn't a
tty. The workaround is to ask the user to set the environment variable
TTY to the tty device.

* Cygwin vs. Windows file names

The approach is that GODI itself is seen as a Cygwin program - although
godi_console is a native program. As already mentioned, godi_console is
capable of translating Cygwin paths without using Cygwin.

At some time, it is required to translate paths to the Windows style,
because programs like ocaml, ocamlfind, etc. are Windows executables
without Cygwin path translation capability. In GODI Makefiles, there is
now always the variable $(LOCALBASE_NATIVE), set to the Windows path
(using forward slashes). The porting effort was effectively quite low -
in a few places I had to use ${LOCALBASE_NATIVE} instead of
${LOCALBASE}.

* CR/LF

In godi_console, it was required to look carefully over the code, and to
switch between ASCII and binary channel modes where needed. As native
Windows executable, godi_console uses CR/LF as EOL characters.

It turned out that Cygwin's bash has sometimes problems with CR/LF.
Especially, something like

stdlib=`ocamlc -where`

does not work - the CR character remains as part of the result value.
The workaround was to use Cygwin's alternate shell, ash, in these cases,
which handles that better.

* cygpcre DLL, and DLL lookup by PATH

Cygwin includes a PCRE DLL. Of course, this DLL cannot be used in
non-Cygwin programs, so GODI always installs its own PCRE DLL.

It turns out that the MinGW install of PCRE produces a DLL with the same
name as the Cygwin version (cygpcre-0.dll, as far as I remember). As
DLL's are looked up by PATH, and we _need_ the Cygwin directories in
PATH, the problem arises how to direct MinGW executables to use the
right DLL.

For now the workaround is that the MinGW version has a different version
number (cygpcre-7.dll). It would be nice, however, to get rid of the
"cyg" prefix for a clearer separation of DLL spaces.

* Ocaml does not install ocamldep.opt.exe

in the MinGW port. Maybe an oversight?

* No support for ocamlmklib

A lot of work has been put into working around the missing ocamlmklib.
The difficulty is that stub libraries need to be compiled with different
flags when a DLL is to be produced, i.e. the C compiler is invoked
differently in this case. This is incompatible with many build Makefiles
that can only produce one version of .o files from a .c file. Because of
this difficulty, ocamlmklib isn't available.

For ocamlnet, I worked around by using two scripts, stubcc, and
mkstublib, available from

https://godirepo.camlcity.org/svn/godi-build/trunk/godi/godi-ocamlnet/files/

Effectively, stubcc calls the C compiler twice, and produces a .d.o
intended for DLL inclusion, and a normal .o for the static library. I'm
wondering whether such a tool could be included in the ocaml
distribution, or whether ocamlc could have a special -for-stublib option
so that it invokes gcc twice for .c files.

* Windows "execv" is broken with respect to "wait"

When a windows process calls execv to invoke another binary, it is
signaled to the parent process that the program is finished. (The
correct POSIX behavior is to wait until the second binary is finished.)
In many places, it was easy to work around the problem by using the
spawn calls. At one place, this is not possible, and godi_console has to
use "execv" to avoid deadlocks. As consequence, it may happen that the
caller of godi_console thinks that the program is done although it is
still running.

No workaround yet.

* No real argv support in Windows

There are no clear rules how to quote arguments in Windows (cmd.exe).
(Actually, there is no list of arguments (argv) in Windows, but only a
command line string.) In GODI, it sometimes happens that quoted
arguments need to be quoted (and more complicated), and this did not
work in Windows. The workaround is to write complex commands to
temporary files, and to run these by /bin/sh.

* No signals in Windows

There is no clear way how to signal a process to terminate itself. As a
consequence, aborting builds is broken in godi_console.

* argv.(0) is broken

I don't know who is to blame for this, either Windows, or Cygwin. At
least it wasn't possible to pass a name as argv.(0) that differed from
the executable name.

The workaround was to use wrapper programs.



Anyway, I hope you like the new port. And remember: be patient. Windows
is very slow when used for shell-style scripting.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------