question: hard-coded file descriptors in stdin/stdout/stderr

question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

Greetings,

I just noticed that the file descriptors in stdin.c, stdout.c, and 
stderr.c do not use the #defines from <unistd.h> (namely STDIN_FILENO, 
STDOUT_FILENO, and STDERR_FILENO), but are rather hard-coded (as 0, 1, 
and 2 respectively).  I was therefore wondering whether there was a 
special reason for that?  With POSIX systems this would normally not be 
an issue, however there are still some systems out there with standard 
file descriptor numbers which are different...

On that same note: wouldn't it make sense to slightly modify unistd.h so 
that it first checks whether STDIN_FILENO, etc. have already been 
defined?  That would allow a system with different standard file 
descriptor numbers to define them in one of the /arch headers, yet 
enable the default for systems that use the "normal" numbers.  The 
relevant section would then read:


#ifndef STDIN_FILENO
   #define STDIN_FILENO  0
#endif

#ifndef STDOUT_FILENO
   #define STDOUT_FILENO 1
#endif

#ifndef STDERR_FILENO
   #define STDERR_FILENO 2
#endif


Sincerley,
Zvi Gilboa

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Szabolcs Nagy]

* Zvi Gilboa <zg7s@eservices.virginia.edu> [2013-03-14 12:18:53 -0400]:
> I just noticed that the file descriptors in stdin.c, stdout.c, and
> stderr.c do not use the #defines from <unistd.h> (namely
> STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO), but are rather
> hard-coded (as 0, 1, and 2 respectively).  I was therefore wondering

these numbers are fixed on posix (and linux and all historical unices)

the macro definitions are there for conformance reasons

it's like using 0 instead of NULL or -1U instead of UINT_MAX


> this would normally not be an issue, however there are still some
> systems out there with standard file descriptor numbers which are
> different...

which ones?

> On that same note: wouldn't it make sense to slightly modify
> unistd.h so that it first checks whether STDIN_FILENO, etc. have

i dont think it makes sense for a linux libc
to have these numbers configurable

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> which ones?

... since you are asking...  inspired by musl-libc, I am currently 
writing a win32/win64 open-source library that implements/provides POSIX 
system calls (see note below).  I believe that having a powerful libc 
with an MIT license available on Windows would actually be of great 
value to the open source community for all possible reasons, but that is 
of course irrelevant to my question:)

The main issue here is that the standard file descriptors on Windows are 
-10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12 
(STD_ERROR_HANDLE).  I could of course compensate for that in my code 
and "translate" the POSIX special file descriptor numbers to the Windows 
ones, however it would be more elegant if musl-libc used the descriptors 
defined in <unistd.h>  instead of hard-coded numbers.

* as for psxcalls: this is a C library, that exclusively uses the Native 
API.  It attaches to ntdll.dll during run-time, and can thus be compiled 
as a "native" static library with no external dependencies.  While it is 
currently in its very initial stage (and not yet online), the major 
"obstacle" features -- including fork() -- have already been 
implemented.  To remove all doubts, I am aware of Cygwin's existence, 
yet am looking for a high-performance solution that would be both 
"clean" (psxcalls-->libc-->user-library-or-application), and flexibly 
licensed.

Best regards,
Zvi




On 03/14/2013 01:17 PM, Szabolcs Nagy wrote:
> * Zvi Gilboa <zg7s@eservices.virginia.edu> [2013-03-14 12:18:53 -0400]:
>> I just noticed that the file descriptors in stdin.c, stdout.c, and
>> stderr.c do not use the #defines from <unistd.h> (namely
>> STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO), but are rather
>> hard-coded (as 0, 1, and 2 respectively).  I was therefore wondering
> these numbers are fixed on posix (and linux and all historical unices)
>
> the macro definitions are there for conformance reasons
>
> it's like using 0 instead of NULL or -1U instead of UINT_MAX
>
>
>> this would normally not be an issue, however there are still some
>> systems out there with standard file descriptor numbers which are
>> different...
> which ones?
>
>> On that same note: wouldn't it make sense to slightly modify
>> unistd.h so that it first checks whether STDIN_FILENO, etc. have
> i dont think it makes sense for a linux libc
> to have these numbers configurable

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Szabolcs Nagy]

* Zvi Gilboa <zg7s@eservices.virginia.edu> [2013-03-14 13:51:19 -0400]:
> ... since you are asking...  inspired by musl-libc, I am currently
> writing a win32/win64 open-source library that implements/provides
> POSIX system calls (see note below).  I believe that having a
> powerful libc with an MIT license available on Windows would
> actually be of great value to the open source community for all

ok

> The main issue here is that the standard file descriptors on Windows
> are -10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12

ouch

i think windows file handles have very
different semantics than posix fds
(dup2, fork,..)

so you need to do hacks if you want posix
on windows

> * as for psxcalls: this is a C library, that exclusively uses the
> Native API.  It attaches to ntdll.dll during run-time, and can thus
> be compiled as a "native" static library with no external
> dependencies.  While it is currently in its very initial stage (and
> not yet online), the major "obstacle" features -- including fork()
> -- have already been implemented.  To remove all doubts, I am aware
> of Cygwin's existence, yet am looking for a high-performance
> solution that would be both "clean"
> (psxcalls-->libc-->user-library-or-application), and flexibly
> licensed.

how do you implement fork?

(windows version of some tools actually allow redirecting
io of child processes i wonder how your fork would interact
with that, eg gawk can read/write to "/dev/fd/n" so what
would happen if you fork such a child, can you communicate
with it with pipes?)

(at some point there was discussion about porting musl to
various systems including windows and the conclusion was
that it should be done on the syscall layer and fork would
be ugly and slow no matter what you do)

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> how do you implement fork?

My main objective was to be efficient and avoid "ugliness," that is, 
eliminate all improbabilities on the parent's side (and obviously also 
the child's), and accordingly ensure that the state following each step 
in the implementation is exactly what I expect it to be.  If you are 
familiar with the Cygwin implementation, then this must ring a bell with 
respect to LoadLibrary (kernel32.dll) and the fact that does not allow 
you to specify the library's base address.

It is important to note that one main problem with the Cygwin 
implementation (which I believe has been acknowledged by their 
developers as well), is that it does not "disable" the loading of 
hard-linked (compile-time linked) libraries, and thus has no control 
over the bases addresses of these modules.  This forces the parent to 
check "after the fact" whether the base addresses of modules in the 
parent and child agree, and accordingly "die" if they don't... with 
respect to dynamically-linked libraries, the Cygwin fork() tries very 
hard to "help" Windows choose the right base-address, yet with no 
guarantee as for the outcome of that effort (so more improbability and 
after-the-fact checking).  For these and several other reasons, I have 
taken a rather different approach.

Following that very long introduction, here is an outline of what I 
do... quite a few steps, yet each of them is extremely fast and fully 
determinable (note that I'm only using the Native API, so "create 
process" refers to NtCreateProcess in ntdll.dll, not to the various APIs 
in kernel32.dll)


0) normalize the heap and perform some basic sanity checks

1) "backup" the parent's import table, mark all relevant handles as 
inheritable

2) nullify the parent's import table, then obtain handle to its 
executable section (ZwOpenSection)

3) create process with NO console attached --> using the above section, 
then adjust necessary settings (process group [aka "job"], etc.)

4) clone process sections (enumerate the sections, map them to the 
child, efficient crc32 checks to verify synchronization allow us to take 
advantage of the speed of mapping, and accordingly use 
ZwWriteProcessMemory only where necessary)

--> the main advantage of using ZwOpenSection and ZwMapViewOfSection 
(which lies at the heart of the whole thing) is that it allows us to 
specify the section's (aka module's, dll's) base address.

5) clone process thread with new EIP (&child)
--> this clones the current thread's stack and sets the EIP in the child 
to &child

6) resume thread:
--> the child is in child(), waiting for an event to be signaled (to be 
sure, at hand is an NT signal)
--> since child has resumed, we now have access to all features of 
PEB_LDR_DATA
--> nonetheless, the child is waiting for us, and could wait forever -- 
until we either signal the event, or terminate it
--> which guarantees accurate knowledge of the parent about the state of 
the child process

7) update child's PEB_LDR_DATA, and gracefully "restore" its import table
--> so that GetModuleHandle returns a HANDLE
--> so that resources can be retrieved
--> and so that DllMain gets called whenever a new thread is being created

* the mapped sections (modules) are already there, so there is no need 
to call LdrLoadLibrary :)
* as long as the parent and child have the same loaded modules, no true 
knowledge of what LdrLoadLibrary does is necessary.  This is true since 
the entire PEB_LDR_DATA resides in the process's address space.
* in other words: we do not implement LdrLoadLibrary, but rather ensure 
that in the end of the day, there would be no difference between what we 
did, and what LdrLoadLibrary would have done...

8) clone process heap (not for the faint of heart)

9) re-attach to console (if needed)

10) signal event so that the child process continues to execute

11) child: set EAX to 0, restore esp, ret --> so that fork() returns zero

12) parent: restore the import table (this is not needed in and of 
itself, but better safe than sorry)

13) parent: return child_pid as required by fork()


AND THAT'S IT!:)  As said above, many steps, yet all are extremely fast 
and fully determinable.


 >> so you need to do hacks if you want posix
on windows


YES, indeed.  Although I'd rather call them "translations":)  But again, 
it would be nice if I didn't have to worry about about hard-coded 
numbers.  Correct use of the pipes is possible as well.  It is actually 
quite easy to duplicate handles on Windows, as well as connect multiple 
processes to the same console and/or console handles.  One simply needs 
to experiment with the low-level functions (for instance 
WriteConsoleInput, which in fact has some basic documentation on MSDN).


So, all in all taken, does that mean my suggestion has been accepted?








On 03/14/2013 02:17 PM, Szabolcs Nagy wrote:
> * Zvi Gilboa <zg7s@eservices.virginia.edu> [2013-03-14 13:51:19 -0400]:
>> ... since you are asking...  inspired by musl-libc, I am currently
>> writing a win32/win64 open-source library that implements/provides
>> POSIX system calls (see note below).  I believe that having a
>> powerful libc with an MIT license available on Windows would
>> actually be of great value to the open source community for all
> ok
>
>> The main issue here is that the standard file descriptors on Windows
>> are -10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12
> ouch
>
> i think windows file handles have very
> different semantics than posix fds
> (dup2, fork,..)
>
> so you need to do hacks if you want posix
> on windows
>
>> * as for psxcalls: this is a C library, that exclusively uses the
>> Native API.  It attaches to ntdll.dll during run-time, and can thus
>> be compiled as a "native" static library with no external
>> dependencies.  While it is currently in its very initial stage (and
>> not yet online), the major "obstacle" features -- including fork()
>> -- have already been implemented.  To remove all doubts, I am aware
>> of Cygwin's existence, yet am looking for a high-performance
>> solution that would be both "clean"
>> (psxcalls-->libc-->user-library-or-application), and flexibly
>> licensed.
> how do you implement fork?
>
> (windows version of some tools actually allow redirecting
> io of child processes i wonder how your fork would interact
> with that, eg gawk can read/write to "/dev/fd/n" so what
> would happen if you fork such a child, can you communicate
> with it with pipes?)
>
> (at some point there was discussion about porting musl to
> various systems including windows and the conclusion was
> that it should be done on the syscall layer and fork would
> be ugly and slow no matter what you do)

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

[-- Attachment #1: Type: text/plain, Size: 2894 bytes --]

More information on the project will become available soon, along with 
two portable (compiler-dependent), header-only libraries that are almost 
complete, and which provide information about the host system (library 
names are NTAPI, which covers Native API types, structs, and function 
typedefs, and PESPECS, which covers the specifications and format (types 
and structs) of the PE format).

The idea to make NTAPI and PESPECS available as separate header-only 
libraries is to relieve developers who write applications targeting the 
Native API and/or the PE format from the current pain of 
header-mishmash.  If nobody objects, I'll announce them on the musl list 
as well.

As for the alternatives: what makes musl the most attractive (in my 
opinion, of course), is not only the high quality of its code and 
performance, but also its *readability*.  I believe that the experience 
of reading the source of a C runtime library should resemble that of 
reading a well-written, inspiring textbook, and musl-libc is the only 
libc project I found that provided me with that kind of reading experience.

[Note, however, that provided an appropriate musl-gcc script, MinGW 
could still serve as the host build environment, so fortunately no need 
to create anything from scratch on that front...]

By all means(!!), it would be great to obtain help with the 
implementation of the various Posix SysCalls/APIs, so the project's 
web-page is actually going to include a "sign-up" sheet:)

Till soon,
Zvi






On 03/14/2013 02:40 PM, LM wrote:
> On Thu, Mar 14, 2013 at 1:51 PM, Zvi Gilboa <zg7s@eservices.virginia.edu> wrote:
>> ... since you are asking...  inspired by musl-libc, I am currently writing a
>> win32/win64 open-source library that implements/provides POSIX system calls
>> (see note below).  I believe that having a powerful libc with an MIT license
>> available on Windows would actually be of great value to the open source
>> community for all possible reasons, but that is of course irrelevant to my
>> question:)
> That is a great idea.  Would very much like to hear how this
> progresses.  Do you have any more information on the project
> available?
>
> Don't know if it helps, but Watcom has an Open Source C library.
> Don't remember the exact license.  Also, I've seen a couple of
> projects on Sourceforge that tried to create an alternate C library
> for MinGW with more POSIX support.  Again, not sure of the licenses.
> You might be able to get some coding ideas from some of these projects
> if the licenses are suitable.  An MIT licensed option on Windows would
> be very nice.
>
> I have code for some POSIX functions like fnmatch that I've been using
> for various Open Source programs that need it in order to port to
> Windows.  I do try to find MIT, BSD or other less stringent licenses
> than GPL/LGPL when I can.  If that would be helpful, let me know.


[-- Attachment #2: Type: text/html, Size: 3726 bytes --]

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rich Felker]

On Thu, Mar 14, 2013 at 01:51:19PM -0400, Zvi Gilboa wrote:
> >> which ones?
> 
> .... since you are asking...  inspired by musl-libc, I am currently
> writing a win32/win64 open-source library that implements/provides
> POSIX system calls (see note below).  I believe that having a
> powerful libc with an MIT license available on Windows would
> actually be of great value to the open source community for all
> possible reasons, but that is of course irrelevant to my question:)

This is actually something I've wanted to see done for a long time, so
I wish you the greatest success on your project. Despite my general
aversion to using the STDIN_FILENO etc. macros, if your project takes
off and this issue is a major obstacle to pulling up-to-date code from
musl, I would probably consider just using them. But...

> The main issue here is that the standard file descriptors on Windows
> are -10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12
> (STD_ERROR_HANDLE).  I could of course compensate for that in my
> code and "translate" the POSIX special file descriptor numbers to
> the Windows ones, however it would be more elegant if musl-libc used
> the descriptors defined in <unistd.h>  instead of hard-coded
> numbers.

When I originally proposed an idea similar to what you're doing, one
of the open-ended questions was "how much" of POSIX such a libc (for
Windows) should aim for. Since we weren't actually doing the project,
a lot of questions were left unanswered, but I think this sort of file
descriptor remapping is something you really should do. Not only are
the numbers 0/1/2 specified and widely hard-coded in applications; the
Windows "file handle" values for stdin/out/err are not usable as file
descriptors because they are negative. They definitely aren't
compatible with select(), and probably have a lot of other issues
where they clash with the interpretation of a negative return value as
an error (e.g. dup2(fd,-10) would return -10, which applications would
interpret as an error). If you want to make sockets and regular file
file descriptors a common space (which is needed for close() to work,
as well as select()/poll()) I think that also requires a remapping of
windows file/socket handles to POSIX-style file descriptor numbers.

> * as for psxcalls: this is a C library, that exclusively uses the
> Native API.  It attaches to ntdll.dll during run-time, and can thus
> be compiled as a "native" static library with no external
> dependencies.  While it is currently in its very initial stage (and
> not yet online), the major "obstacle" features -- including fork()
> -- have already been implemented.  To remove all doubts, I am aware
> of Cygwin's existence, yet am looking for a high-performance
> solution that would be both "clean"
> (psxcalls-->libc-->user-library-or-application), and flexibly
> licensed.

Nice to hear. Incidentally, fork() was one of the interfaces I thought
we could just sacrifice in doing such a libc (posix_spawn or the
similar Windows function can replace most correct uses of fork()), so
it's nice to hear you've solved the problem.

Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[LM]

On Fri, Mar 15, 2013 at 4:33 AM, Rich Felker <dalias@aerifal.cx> wrote:
> Not only are
> the numbers 0/1/2 specified and widely hard-coded in applications; the
> Windows "file handle" values for stdin/out/err are not usable as file
> descriptors because they are negative.

Was just thinking about that this morning.  I've been trying to port
msh to Windows and it uses hard-coded 0, 1 and 2 with pipes and
expects them to mean stdin, stdout and stderr.  I'm guessing MinGW
does something in its code to deal with the situation, because it
works with these hard-coded numbers when I build msh.  Would be nice
if it worked the same way with a replacement library such as musl if
used on Windows.

> Nice to hear. Incidentally, fork() was one of the interfaces I thought
> we could just sacrifice in doing such a libc (posix_spawn or the
> similar Windows function can replace most correct uses of fork()), so
> it's nice to hear you've solved the problem.

That's pretty much what I do when I port applications to Windows; I
substitute spawn.  Seems easier to replace vfork than fork that way
though.

One other issue I hit when porting to Windows is the lack of certain
signal handling capabilities.  For instance, when attempting to port
flrec, in place of  kill(pid_sox,SIGSTOP);, I'm trying to use
NtSuspendProcess and in place of  SIGCONT, I'm trying to use
NtResumeProcess.  Some of the other signal information used in the
program, I'm not even sure if I can duplicate on Windows yet.

Would be nice if musl does get ported to Windows.  It could do a lot
towards simplifying porting of other applications.

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> [Rich wrote:] Not only are the numbers 0/1/2 specified and widely 
hard-coded in applications;

 >> [LM wrote:] I've been trying to port msh to Windows and it uses 
hard-coded 0, 1 and 2 with pipes and expects them to mean stdin, stdout 
and stderr.

Thank you for this valuable input!  Clearly, remapping 0/1/2 from within 
psxcalls would be the only responsible approach:)


 >> [LM wrote, regarding the absence of fork() on Windows:] That's 
pretty much what I do when I port applications to Windows; I substitute 
spawn.  Seems easier to replace vfork than fork that way though.

This seems to be the common in-lack-of-a-better-solution approach... but 
I'm sure you'd agree it is both inaccurate and inconvenient, not to 
mention all those file handles and .data and .bss variables that have to 
be manually synchronized...


 >> [LM wrote:] One other issue I hit when porting to Windows is the 
lack of certain signal handling capabilities.  For instance, when 
attempting to port flrec, in place of kill(pid_sox,SIGSTOP);, I'm trying 
to use NtSuspendProcess and in place of  SIGCONT, I'm trying to use 
NtResumeProcess.

That kind of "translation" is an essential part of the psxcalls 
project.  Surprisingly (or not), the vast majority of differences 
between the Native API and POSIX are about syntax, not functionality.  
In other words, there exist only very few POSIX features that cannot be 
implemented using the Native API in a straight-forward way.  There are 
for sure some cases that require programming acrobatics (fork() and 
exec()) being the most obvious examples), but that's the exception, not 
the rule.  Whenever I implement one of the functions, I try to phrase 
the task in the form of a question, for instance: "using the Native API, 
how would you map a memory page to the address space of a user 
applications?"  That kind of approach to writing the library surely 
entails a lot of work, but is also very exciting -- like solving a 
puzzle every single day...









On 03/15/2013 04:33 AM, Rich Felker wrote:
> On Thu, Mar 14, 2013 at 01:51:19PM -0400, Zvi Gilboa wrote:
>>>> which ones?
>> .... since you are asking...  inspired by musl-libc, I am currently
>> writing a win32/win64 open-source library that implements/provides
>> POSIX system calls (see note below).  I believe that having a
>> powerful libc with an MIT license available on Windows would
>> actually be of great value to the open source community for all
>> possible reasons, but that is of course irrelevant to my question:)
> This is actually something I've wanted to see done for a long time, so
> I wish you the greatest success on your project. Despite my general
> aversion to using the STDIN_FILENO etc. macros, if your project takes
> off and this issue is a major obstacle to pulling up-to-date code from
> musl, I would probably consider just using them. But...
>
>> The main issue here is that the standard file descriptors on Windows
>> are -10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12
>> (STD_ERROR_HANDLE).  I could of course compensate for that in my
>> code and "translate" the POSIX special file descriptor numbers to
>> the Windows ones, however it would be more elegant if musl-libc used
>> the descriptors defined in <unistd.h>  instead of hard-coded
>> numbers.
> When I originally proposed an idea similar to what you're doing, one
> of the open-ended questions was "how much" of POSIX such a libc (for
> Windows) should aim for. Since we weren't actually doing the project,
> a lot of questions were left unanswered, but I think this sort of file
> descriptor remapping is something you really should do. Not only are
> the numbers 0/1/2 specified and widely hard-coded in applications; the
> Windows "file handle" values for stdin/out/err are not usable as file
> descriptors because they are negative. They definitely aren't
> compatible with select(), and probably have a lot of other issues
> where they clash with the interpretation of a negative return value as
> an error (e.g. dup2(fd,-10) would return -10, which applications would
> interpret as an error). If you want to make sockets and regular file
> file descriptors a common space (which is needed for close() to work,
> as well as select()/poll()) I think that also requires a remapping of
> windows file/socket handles to POSIX-style file descriptor numbers.
>
>> * as for psxcalls: this is a C library, that exclusively uses the
>> Native API.  It attaches to ntdll.dll during run-time, and can thus
>> be compiled as a "native" static library with no external
>> dependencies.  While it is currently in its very initial stage (and
>> not yet online), the major "obstacle" features -- including fork()
>> -- have already been implemented.  To remove all doubts, I am aware
>> of Cygwin's existence, yet am looking for a high-performance
>> solution that would be both "clean"
>> (psxcalls-->libc-->user-library-or-application), and flexibly
>> licensed.
> Nice to hear. Incidentally, fork() was one of the interfaces I thought
> we could just sacrifice in doing such a libc (posix_spawn or the
> similar Windows function can replace most correct uses of fork()), so
> it's nice to hear you've solved the problem.
>
> Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rich Felker]

On Fri, Mar 15, 2013 at 10:46:54AM -0400, Zvi Gilboa wrote:
> >> [Rich wrote:] Not only are the numbers 0/1/2 specified and widely
> hard-coded in applications;
> 
> >> [LM wrote:] I've been trying to port msh to Windows and it uses
> hard-coded 0, 1 and 2 with pipes and expects them to mean stdin,
> stdout and stderr.
> 
> Thank you for this valuable input!  Clearly, remapping 0/1/2 from
> within psxcalls would be the only responsible approach:)

Note that even if the shell doesn't hard-code them, every single
non-trivial shell script ever written hard-codes them. I think this
was the motivation for POSIX requiring that they have the specific
values 0/1/2.

> That kind of "translation" is an essential part of the psxcalls
> project.  Surprisingly (or not), the vast majority of differences
> between the Native API and POSIX are about syntax, not
> functionality.  In other words, there exist only very few POSIX
> features that cannot be implemented using the Native API in a
> straight-forward way.  There are for sure some cases that require

That's nice to hear. I suspect there are a lot more subtleties like
different error conditions and corner cases, which probably need
consideration on a case-by-case basis whether it's important to match
the POSIX behavior (or whether it's even possible to match it
exactly). Still, I think most such problems are things that can be
solved incrementally, and shouldn't stall the project.

> programming acrobatics (fork() and exec()) being the most obvious
> examples), but that's the exception, not the rule.  Whenever I
> implement one of the functions, I try to phrase the task in the form
> of a question, for instance: "using the Native API, how would you
> map a memory page to the address space of a user applications?"
> That kind of approach to writing the library surely entails a lot of
> work, but is also very exciting -- like solving a puzzle every
> single day...

Keep up the good spirits -- I'm glad to see this finally happening. My
hope for the past 7+ years has been that we could get to a point where
software (or at least FOSS) could be written with a portable core
targetting POSIX rather than having windows-specific hacks all over
the core to deal with the badness of msvcrt, or bloated "portable
runtime" libraries that layer abstractions on top of the standardized
abstractions.

Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

Thank you for the encouragement, Rich! It is great to know that the 
project will be warmly welcome by the musl community.  As for the 
incremental "case-by-case basis" translation of error codes, as well as 
handling of corner cases: my method thus far has been to create raw text 
files with the basic mapping information, and then generate the header 
and source files using command-line tools such as awk, gawk, sed, etc 
(which forces me to actually understand what I need to achieve...)  This 
means that covering more cases in the future by me or other developers 
should be a rather self-contained task, requiring only minimal knowledge 
about the library as a whole.

 >> "hacks all over the core to deal with the badness of msvcrt"

No way I could express this better:)

Zvi



On 03/15/2013 02:43 PM, Rich Felker wrote:
> On Fri, Mar 15, 2013 at 10:46:54AM -0400, Zvi Gilboa wrote:
>>>> [Rich wrote:] Not only are the numbers 0/1/2 specified and widely
>> hard-coded in applications;
>>
>>>> [LM wrote:] I've been trying to port msh to Windows and it uses
>> hard-coded 0, 1 and 2 with pipes and expects them to mean stdin,
>> stdout and stderr.
>>
>> Thank you for this valuable input!  Clearly, remapping 0/1/2 from
>> within psxcalls would be the only responsible approach:)
> Note that even if the shell doesn't hard-code them, every single
> non-trivial shell script ever written hard-codes them. I think this
> was the motivation for POSIX requiring that they have the specific
> values 0/1/2.
>
>> That kind of "translation" is an essential part of the psxcalls
>> project.  Surprisingly (or not), the vast majority of differences
>> between the Native API and POSIX are about syntax, not
>> functionality.  In other words, there exist only very few POSIX
>> features that cannot be implemented using the Native API in a
>> straight-forward way.  There are for sure some cases that require
> That's nice to hear. I suspect there are a lot more subtleties like
> different error conditions and corner cases, which probably need
> consideration on a case-by-case basis whether it's important to match
> the POSIX behavior (or whether it's even possible to match it
> exactly). Still, I think most such problems are things that can be
> solved incrementally, and shouldn't stall the project.
>
>> programming acrobatics (fork() and exec()) being the most obvious
>> examples), but that's the exception, not the rule.  Whenever I
>> implement one of the functions, I try to phrase the task in the form
>> of a question, for instance: "using the Native API, how would you
>> map a memory page to the address space of a user applications?"
>> That kind of approach to writing the library surely entails a lot of
>> work, but is also very exciting -- like solving a puzzle every
>> single day...
> Keep up the good spirits -- I'm glad to see this finally happening. My
> hope for the past 7+ years has been that we could get to a point where
> software (or at least FOSS) could be written with a portable core
> targetting POSIX rather than having windows-specific hacks all over
> the core to deal with the badness of msvcrt, or bloated "portable
> runtime" libraries that layer abstractions on top of the standardized
> abstractions.
>
> Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rich Felker]

On Fri, Mar 15, 2013 at 02:55:44PM -0400, Zvi Gilboa wrote:
> Thank you for the encouragement, Rich! It is great to know that the
> project will be warmly welcome by the musl community.  As for the

No problem. To clarify a bit, musl itself will most likely remain
Linux-only (or rather Linux-syscall-API-only) in the main repository,
because I don't like the complexity cost (which you can see in its
extreme form in glibc :) of abstracting for that kind of underlying
system diversity. The approach I would recommend to you if you want to
use musl for this is to fork files which need to be seriously
different on Windows, and keep a separate list of files that can be
synchonized automatically or with minimal manual intervention.

If there end up being things that are gratuitously difficult to reuse
in their current form, that could be changed in musl without making it
more complex or having other undesirable side effects, I think we
could probably make changes that make your work easier.

If on the other hand your work provides a pseudo-syscall-interface
musl could ride on top of, it might be possible to use most of musl
almost-unmodified on top of windows.

Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> If on the other hand your work provides a pseudo-syscall-interface 
musl could ride on top of, it might be possible to use most of musl 
almost-unmodified on top of windows.


That's perfectly understood!  I'm indeed writing psxcalls with musl in 
mind, with the goal that the library would provide musl with all it 
needs to treat it just like any other linux system. This means that the 
only "specialties" would exist where they are present for other 
platforms as well, namely as sub-folders under /src/thread or 
/src/internal.  As for /src/math, I have already created some basic sed 
scripts that convert the Linux .S files to MinGW ones.

There are probably some fine details of which I am not yet aware, 
however I'm overall optimistic about musl's eventual ability to 
integrate win32 and win64 transparently, just like it would with any 
other Linux platform.

Zvi





On 03/15/2013 03:03 PM, Rich Felker wrote:
> On Fri, Mar 15, 2013 at 02:55:44PM -0400, Zvi Gilboa wrote:
>> Thank you for the encouragement, Rich! It is great to know that the
>> project will be warmly welcome by the musl community.  As for the
> No problem. To clarify a bit, musl itself will most likely remain
> Linux-only (or rather Linux-syscall-API-only) in the main repository,
> because I don't like the complexity cost (which you can see in its
> extreme form in glibc :) of abstracting for that kind of underlying
> system diversity. The approach I would recommend to you if you want to
> use musl for this is to fork files which need to be seriously
> different on Windows, and keep a separate list of files that can be
> synchonized automatically or with minimal manual intervention.
>
> If there end up being things that are gratuitously difficult to reuse
> in their current form, that could be changed in musl without making it
> more complex or having other undesirable side effects, I think we
> could probably make changes that make your work easier.
>
> If on the other hand your work provides a pseudo-syscall-interface
> musl could ride on top of, it might be possible to use most of musl
> almost-unmodified on top of windows.
>
> Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rob Landley]

On 03/14/2013 12:51:19 PM, Zvi Gilboa wrote:
> >> which ones?
> 
> ... since you are asking...  inspired by musl-libc, I am currently  
> writing a win32
> /win64 open-source library that implements/provides POSIX system  
> calls (see note
> below).  I believe that having a powerful libc with an MIT license  
> available on
> Windows would actually be of great value to the open source community  
> for all
> possible reasons, but that is of course irrelevant to my question:)

All possible reasons? I can't think of one.

I have actually thought about this sort of thing. For example, I did a  
talk at the recent emebedded Linux conference explaining _why_ toybox  
and musl are important to Android:

http://www.youtube.com/watch?v=SGmtP5Lg_t0

> The main issue here is that the standard file descriptors on Windows  
> are -10
> (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12  
> (STD_ERROR_HANDLE).

In DOS, stdin, stdout, and stderr were 0, 1, and 2. What you just  
listed is an explicit violation of posix.

>  I could of course compensate for that in my code and "translate" the  
> POSIX special
> file descriptor numbers to the Windows ones, however it would be more  
> elegant if musl
> -libc used the descriptors defined in <unistd.h>  instead of  
> hard-coded numbers.

You want to introduce non-posix assumptions into musl to make windows  
code more elegant.

Really?

> 
> * as for psxcalls: this is a C library, that exclusively uses the  
> Native API.  It
> attaches to ntdll.dll during run-time, and can thus be compiled as a  
> "native" static
> library with no external dependencies.

Doesn't mingw already exist?

>  While it is currently in its very initial
> stage (and not yet online), the major "obstacle" features --  
> including fork()
> -- have already been implemented.  To remove all doubts,

Remove all doubts? Really? All of them?

> I am aware of Cygwin's existence,

But apparently not mingw.

> yet am looking for a high-performance solution that would be both  
> "clean" (psxcalls-->libc-->user-library-or-application), and flexibly  
> licensed.

How on earth does licensing on WINDOWS matter, since the base OS is  
proprietary?

So this is explicitly "provide free stuff to make paying money to  
Microsoft more appealing"?

Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rich Felker]

On Sun, Mar 17, 2013 at 10:14:00PM -0500, Rob Landley wrote:
> On 03/14/2013 12:51:19 PM, Zvi Gilboa wrote:
> >>> which ones?
> >
> >... since you are asking...  inspired by musl-libc, I am currently
> >writing a win32
> >/win64 open-source library that implements/provides POSIX system
> >calls (see note
> >below).  I believe that having a powerful libc with an MIT license
> >available on
> >Windows would actually be of great value to the open source
> >community for all
> >possible reasons, but that is of course irrelevant to my question:)
> 
> All possible reasons? I can't think of one.
> 
> I have actually thought about this sort of thing. For example, I did
> a talk at the recent emebedded Linux conference explaining _why_
> toybox and musl are important to Android:
> 
> http://www.youtube.com/watch?v=SGmtP5Lg_t0
> 
> >The main issue here is that the standard file descriptors on
> >Windows are -10
> >(STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12
> >(STD_ERROR_HANDLE).
> 
> In DOS, stdin, stdout, and stderr were 0, 1, and 2. What you just
> listed is an explicit violation of posix.

This issue was already resolved. There's no need to flame...

> > I could of course compensate for that in my code and "translate"
> >the POSIX special
> >file descriptor numbers to the Windows ones, however it would be
> >more elegant if musl
> >-libc used the descriptors defined in <unistd.h>  instead of
> >hard-coded numbers.
> 
> You want to introduce non-posix assumptions into musl to make
> windows code more elegant.
> 
> Really?

No, he just wanted to eliminate some assumptions (guaranteed by POSIX)
and use macros instead of hard-coded magic numbers. But this turned
out not to be necessary or useful anyway, for reasons connected to the
ones you cited. We arrived at this conclusion with no need for
flaming. :-)

> >* as for psxcalls: this is a C library, that exclusively uses the
> >Native API.  It
> >attaches to ntdll.dll during run-time, and can thus be compiled as
> >a "native" static
> >library with no external dependencies.
> 
> Doesn't mingw already exist?

mingw is not a libc. It uses msvcrt, which is horrible. The reason for
getting a real proper libc on windows is not to make windows' users
lives easier, or at least that's not my reason for wanting it. Rather,
it makes OUR lives easier, because FOSS projects can just target POSIX
and keep their cores simple, rather than being full of #ifdef hell or
"portable runtime" hell.

Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Strake]

On 17/03/2013, Rich Felker <dalias@aerifal.cx> wrote:
> Rather,
> it makes OUR lives easier, because FOSS projects can just target POSIX
> and keep their cores simple

Actually, we can do that anyhow.

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rich Felker]

On Sun, Mar 17, 2013 at 10:50:01PM -0500, Strake wrote:
> On 17/03/2013, Rich Felker <dalias@aerifal.cx> wrote:
> > Rather,
> > it makes OUR lives easier, because FOSS projects can just target POSIX
> > and keep their cores simple
> 
> Actually, we can do that anyhow.

Well of course those of us who don't care about Windows support can do
that. The problem is that many people do care about Windows support,
and thus we're stuck with lots of mess. I'd really like to see a next
generation of applications that aren't full of hacks for
"portability".

Rich

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rob Landley]

On 03/17/2013 11:08:24 PM, Rich Felker wrote:
> On Sun, Mar 17, 2013 at 10:50:01PM -0500, Strake wrote:
> > On 17/03/2013, Rich Felker <dalias@aerifal.cx> wrote:
> > > Rather,
> > > it makes OUR lives easier, because FOSS projects can just target  
> POSIX
> > > and keep their cores simple
> >
> > Actually, we can do that anyhow.
> 
> Well of course those of us who don't care about Windows support can do
> that. The problem is that many people do care about Windows support,
> and thus we're stuck with lots of mess. I'd really like to see a next
> generation of applications that aren't full of hacks for
> "portability".

If Windows gets left behind on the PC the way Dec's Unicos got left  
behind on the minicomputer, and in the new world of smartphones nobody  
does Windows, problem solved. (And speaking of portability hacks, 64  
bit Windows is LLP64, not LP64. On 64 bit windows, "long" is 32 bits.)

I understand what Cygwin tried to do, and why it's a mess. (Timesys  
supported its own fork of that when I worked there.) I understand what  
mingw tried to do, and why it's a different mess. (I used that to test  
tinycc's windows mode under wine.) I don't understand why this new  
approach thinks it won't encounter the problems of either previous  
project. I'd wait to see code, except I haven't got a windows test  
environment and don't want one.

Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rob Landley]

On 03/17/2013 10:26:31 PM, Rich Felker wrote:
> > You want to introduce non-posix assumptions into musl to make
> > windows code more elegant.
> >
> > Really?
> 
> No, he just wanted to eliminate some assumptions (guaranteed by POSIX)
> and use macros instead of hard-coded magic numbers. But this turned
> out not to be necessary or useful anyway, for reasons connected to the
> ones you cited. We arrived at this conclusion with no need for
> flaming. :-)

Sorry, I used to work on OS/2. My response to Microsoft is the same as  
Vogon Grandmothers and the Ravenous Bugbladder Beast of Traal. "In  
brief: avoid." (Pure reflex at this point.)

> > >* as for psxcalls: this is a C library, that exclusively uses the
> > >Native API.  It
> > >attaches to ntdll.dll during run-time, and can thus be compiled as
> > >a "native" static
> > >library with no external dependencies.
> >
> > Doesn't mingw already exist?
> 
> mingw is not a libc. It uses msvcrt, which is horrible.

Oh agreed. I set up a mingw environment to run under wine to test my  
old tinycc fork with at one point. It's just _less_ horrible than  
cygwin (which I've also had to use).

> The reason for
> getting a real proper libc on windows is not to make windows' users
> lives easier, or at least that's not my reason for wanting it. Rather,
> it makes OUR lives easier, because FOSS projects can just target POSIX
> and keep their cores simple, rather than being full of #ifdef hell or
> "portable runtime" hell.

That was the part that was confusing me, POSIX _does_ say that 0 is  
stdin, 1 is stdout, and 2 is stderr. "Targeting POSIX" would mean  
write(2, "error message"); is allowed.

Also kinda hard to do shell scripting without that definition:

   $ cd ~/linux/scripts
   $ grep '2>&1' * | wc -l
   32

Windows _isn't_ a unix. I don't see how you're going to fix the things  
cygwin did without either reinventing their huge glue layer, and the  
obvious alternative is mingw's not dealing with any of it...

*shrug* As long as I can continue not to care, it isn't really my  
problem. But this is a can of worms.

Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> Doesn't mingw already exist?

Of course it does, but it does not allow one to compile unmodified posix 
code.


 >> How on earth does licensing on WINDOWS matter, since the base OS is 
proprietary? So this is explicitly "provide free stuff to make paying 
money to Microsoft more appealing"?

My approach on that issue is apparently rather different.  But as often 
happens, the greatest resistance comes not from those who oppose one's 
goal, but rather from those who share the same goal, yet differ in their 
vision as to how it should be reached.  My hope is that as my project 
evolves, you, too, will become one of its supports.

Best regards,
Zvi






On 03/17/2013 11:14 PM, Rob Landley wrote:
> On 03/14/2013 12:51:19 PM, Zvi Gilboa wrote:
>> >> which ones?
>>
>> ... since you are asking...  inspired by musl-libc, I am currently 
>> writing a win32
>> /win64 open-source library that implements/provides POSIX system 
>> calls (see note
>> below).  I believe that having a powerful libc with an MIT license 
>> available on
>> Windows would actually be of great value to the open source community 
>> for all
>> possible reasons, but that is of course irrelevant to my question:)
>
> All possible reasons? I can't think of one.
>
> I have actually thought about this sort of thing. For example, I did a 
> talk at the recent emebedded Linux conference explaining _why_ toybox 
> and musl are important to Android:
>
> http://www.youtube.com/watch?v=SGmtP5Lg_t0
>
>> The main issue here is that the standard file descriptors on Windows 
>> are -10
>> (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12 (STD_ERROR_HANDLE).
>
> In DOS, stdin, stdout, and stderr were 0, 1, and 2. What you just 
> listed is an explicit violation of posix.
>
>>  I could of course compensate for that in my code and "translate" the 
>> POSIX special
>> file descriptor numbers to the Windows ones, however it would be more 
>> elegant if musl
>> -libc used the descriptors defined in <unistd.h>  instead of 
>> hard-coded numbers.
>
> You want to introduce non-posix assumptions into musl to make windows 
> code more elegant.
>
> Really?
>
>>
>> * as for psxcalls: this is a C library, that exclusively uses the 
>> Native API.  It
>> attaches to ntdll.dll during run-time, and can thus be compiled as a 
>> "native" static
>> library with no external dependencies.
>
> Doesn't mingw already exist?
>
>>  While it is currently in its very initial
>> stage (and not yet online), the major "obstacle" features -- 
>> including fork()
>> -- have already been implemented.  To remove all doubts,
>
> Remove all doubts? Really? All of them?
>
>> I am aware of Cygwin's existence,
>
> But apparently not mingw.
>
>> yet am looking for a high-performance solution that would be both 
>> "clean" (psxcalls-->libc-->user-library-or-application), and flexibly 
>> licensed.
>
> How on earth does licensing on WINDOWS matter, since the base OS is 
> proprietary?
>
> So this is explicitly "provide free stuff to make paying money to 
> Microsoft more appealing"?
>
> Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rob Landley]

On 03/17/2013 10:28:51 PM, Zvi Gilboa wrote:
> >> Doesn't mingw already exist?
> 
> Of course it does, but it does not allow one to compile unmodified  
> posix code.
> 
> 
> >> How on earth does licensing on WINDOWS matter, since the base OS  
> is proprietary? So this is explicitly "provide free stuff to make  
> paying money to Microsoft more appealing"?
> 
> My approach on that issue is apparently rather different.  But as  
> often
> happens, the greatest resistance comes not from those who oppose  
> one's goal,
> but rather from those who share the same goal, yet differ in their  
> vision as
> to how it should be reached.  My hope is that as my project evolves,  
> you,
> too, will become one of its supports.

Oh no, I oppose your goal entirely. I think that a posix libc  
attempting to support windows is a bad idea.

My concern isn't that you'll succeed or fail either way, it's that in  
trying to do it you'll mess up the nice clean Linux C library Rich has  
made by forcing a bunch of non-posix assumptions on it. (I.E. fork it  
all you like, that's merely useless, but pushing this stuff upstream  
makes no sense to me and seems actively harmful.)

This assumption that started this thread is a perfect example. The  
posix-2008 stdin definition under system interfaces in the function  
list explicitly says that stdin is 0, stdout is 1, and stderr is 2. So  
musl relying on what posix said _is_ what you were objecting to.  
Meanwhile Rich is saying that letting windows programs rely on posix is  
the advantage of your approach. This seems like a direct conflict to me.

It is of course Rich's call not mine. But I'm not following the logic  
at all.

Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Zvi Gilboa]

 >> My concern isn't that you'll succeed or fail either way, it's that 
in trying to do it you'll mess up the nice clean Linux C library Rich 
has made by forcing a bunch of non-posix assumptions on it.

No need to worry, I shall never do something like that.  As mentioned a 
couple of days ago, my library is written so that all additions to musl 
will take place in the form of architecture-specific sub-folders, as is 
the case today with mips, x86_64, etc.


 >> This assumption that started this thread is a perfect example... so 
musl relying on what posix said _is_ what you were objecting to.

Not really an assumption, just an attempt to better understand things.  
The answers I received were enlightening and very useful, and thus saved 
me a lot of time.  I also do not recall trying to object to the current 
musl implementation, hence the use of question marks and the subjunctive 
mood in my original post:)


 >> I don't understand why this new approach thinks it won't encounter 
the problems of either previous project. I'd wait to see code...

You know, sometimes dreams come true.  Maybe mine will be one of them.

ZG






On 03/18/2013 12:22 AM, Rob Landley wrote:
> On 03/17/2013 10:28:51 PM, Zvi Gilboa wrote:
>> >> Doesn't mingw already exist?
>>
>> Of course it does, but it does not allow one to compile unmodified 
>> posix code.
>>
>>
>> >> How on earth does licensing on WINDOWS matter, since the base OS 
>> is proprietary? So this is explicitly "provide free stuff to make 
>> paying money to Microsoft more appealing"?
>>
>> My approach on that issue is apparently rather different.  But as often
>> happens, the greatest resistance comes not from those who oppose 
>> one's goal,
>> but rather from those who share the same goal, yet differ in their 
>> vision as
>> to how it should be reached.  My hope is that as my project evolves, 
>> you,
>> too, will become one of its supports.
>
> Oh no, I oppose your goal entirely. I think that a posix libc 
> attempting to support windows is a bad idea.
>
> My concern isn't that you'll succeed or fail either way, it's that in 
> trying to do it you'll mess up the nice clean Linux C library Rich has 
> made by forcing a bunch of non-posix assumptions on it. (I.E. fork it 
> all you like, that's merely useless, but pushing this stuff upstream 
> makes no sense to me and seems actively harmful.)
>
> This assumption that started this thread is a perfect example. The 
> posix-2008 stdin definition under system interfaces in the function 
> list explicitly says that stdin is 0, stdout is 1, and stderr is 2. So 
> musl relying on what posix said _is_ what you were objecting to. 
> Meanwhile Rich is saying that letting windows programs rely on posix 
> is the advantage of your approach. This seems like a direct conflict 
> to me.
>
> It is of course Rich's call not mine. But I'm not following the logic 
> at all.
>
> Rob

Re: question: hard-coded file descriptors in stdin/stdout/stderr

[Rob Landley]

On 03/14/2013 11:18:53 AM, Zvi Gilboa wrote:
> Greetings,
> 
> I just noticed that the file descriptors in stdin.c, stdout.c, and  
> stderr.c do not use the #defines from <unistd.h> (namely  
> STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO), but are rather  
> hard-coded (as 0, 1, and 2 respectively).

Because these have been constant values since 1969, were adopted by DOS  
in 1983, and the numerical values are SPECIFIED BY POSIX:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html

   STDIN_FILENO - Standard input value, stdin. Its value is 0.
   STDOUT_FILENO - Standard output value, stdout. Its value is 1.
   STDERR_FILENO - Standard error value, stderr. Its value is 2.

> I was therefore wondering whether there was a special reason for that?

Because MUSL is simple and using a macro to specify a constant that's  
standardized by posix, has been stable for over 40 years, and is  
regularly used on the command line, is pointless?

> With POSIX systems this would normally not be an issue, however there  
> are still some
> systems out there with standard file descriptor numbers which are  
> different...

Posix specifies the value of the constant. Supporting systems that  
violate posix was not, last I checked, part of musl's goals?

> On that same note: wouldn't it make sense to slightly modify unistd.h  
> so that it
> first checks whether STDIN_FILENO, etc. have already been defined?   
> That would allow
> a system with different standard file descriptor numbers to define  
> them in one of
> the /arch headers, yet enable the default for systems that use the  
> "normal" numbers.
> The relevant section would then read:

What system are you referring to? "There are systems" means what,  
exactly>

Not Linux, not Mac, not Android, not iOS... what are you referring to?

Rob