From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/2915 Path: news.gmane.org!not-for-mail From: Zvi Gilboa Newsgroups: gmane.linux.lib.musl.general Subject: Re: question: hard-coded file descriptors in stdin/stdout/stderr Date: Thu, 14 Mar 2013 15:34:02 -0400 Message-ID: <5142262A.4060606@eservices.virginia.edu> References: <5141F86D.8010000@eservices.virginia.edu> <20130314171752.GB19010@port70.net> <51420E17.9030305@eservices.virginia.edu> <20130314181759.GC19010@port70.net> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1363289654 3002 80.91.229.3 (14 Mar 2013 19:34:14 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 14 Mar 2013 19:34:14 +0000 (UTC) To: Original-X-From: musl-return-2916-gllmg-musl=m.gmane.org@lists.openwall.com Thu Mar 14 20:34:39 2013 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1UGDvC-0007SR-8S for gllmg-musl@plane.gmane.org; Thu, 14 Mar 2013 20:34:38 +0100 Original-Received: (qmail 12010 invoked by uid 550); 14 Mar 2013 19:34:15 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 12002 invoked from network); 14 Mar 2013 19:34:14 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 In-Reply-To: <20130314181759.GC19010@port70.net> X-Originating-IP: [68.229.98.213] Xref: news.gmane.org gmane.linux.lib.musl.general:2915 Archived-At: >> how do you implement fork? My main objective was to be efficient and avoid "ugliness," that is, eliminate all improbabilities on the parent's side (and obviously also the child's), and accordingly ensure that the state following each step in the implementation is exactly what I expect it to be. If you are familiar with the Cygwin implementation, then this must ring a bell with respect to LoadLibrary (kernel32.dll) and the fact that does not allow you to specify the library's base address. It is important to note that one main problem with the Cygwin implementation (which I believe has been acknowledged by their developers as well), is that it does not "disable" the loading of hard-linked (compile-time linked) libraries, and thus has no control over the bases addresses of these modules. This forces the parent to check "after the fact" whether the base addresses of modules in the parent and child agree, and accordingly "die" if they don't... with respect to dynamically-linked libraries, the Cygwin fork() tries very hard to "help" Windows choose the right base-address, yet with no guarantee as for the outcome of that effort (so more improbability and after-the-fact checking). For these and several other reasons, I have taken a rather different approach. Following that very long introduction, here is an outline of what I do... quite a few steps, yet each of them is extremely fast and fully determinable (note that I'm only using the Native API, so "create process" refers to NtCreateProcess in ntdll.dll, not to the various APIs in kernel32.dll) 0) normalize the heap and perform some basic sanity checks 1) "backup" the parent's import table, mark all relevant handles as inheritable 2) nullify the parent's import table, then obtain handle to its executable section (ZwOpenSection) 3) create process with NO console attached --> using the above section, then adjust necessary settings (process group [aka "job"], etc.) 4) clone process sections (enumerate the sections, map them to the child, efficient crc32 checks to verify synchronization allow us to take advantage of the speed of mapping, and accordingly use ZwWriteProcessMemory only where necessary) --> the main advantage of using ZwOpenSection and ZwMapViewOfSection (which lies at the heart of the whole thing) is that it allows us to specify the section's (aka module's, dll's) base address. 5) clone process thread with new EIP (&child) --> this clones the current thread's stack and sets the EIP in the child to &child 6) resume thread: --> the child is in child(), waiting for an event to be signaled (to be sure, at hand is an NT signal) --> since child has resumed, we now have access to all features of PEB_LDR_DATA --> nonetheless, the child is waiting for us, and could wait forever -- until we either signal the event, or terminate it --> which guarantees accurate knowledge of the parent about the state of the child process 7) update child's PEB_LDR_DATA, and gracefully "restore" its import table --> so that GetModuleHandle returns a HANDLE --> so that resources can be retrieved --> and so that DllMain gets called whenever a new thread is being created * the mapped sections (modules) are already there, so there is no need to call LdrLoadLibrary :) * as long as the parent and child have the same loaded modules, no true knowledge of what LdrLoadLibrary does is necessary. This is true since the entire PEB_LDR_DATA resides in the process's address space. * in other words: we do not implement LdrLoadLibrary, but rather ensure that in the end of the day, there would be no difference between what we did, and what LdrLoadLibrary would have done... 8) clone process heap (not for the faint of heart) 9) re-attach to console (if needed) 10) signal event so that the child process continues to execute 11) child: set EAX to 0, restore esp, ret --> so that fork() returns zero 12) parent: restore the import table (this is not needed in and of itself, but better safe than sorry) 13) parent: return child_pid as required by fork() AND THAT'S IT!:) As said above, many steps, yet all are extremely fast and fully determinable. >> so you need to do hacks if you want posix on windows YES, indeed. Although I'd rather call them "translations":) But again, it would be nice if I didn't have to worry about about hard-coded numbers. Correct use of the pipes is possible as well. It is actually quite easy to duplicate handles on Windows, as well as connect multiple processes to the same console and/or console handles. One simply needs to experiment with the low-level functions (for instance WriteConsoleInput, which in fact has some basic documentation on MSDN). So, all in all taken, does that mean my suggestion has been accepted? On 03/14/2013 02:17 PM, Szabolcs Nagy wrote: > * Zvi Gilboa [2013-03-14 13:51:19 -0400]: >> ... since you are asking... inspired by musl-libc, I am currently >> writing a win32/win64 open-source library that implements/provides >> POSIX system calls (see note below). I believe that having a >> powerful libc with an MIT license available on Windows would >> actually be of great value to the open source community for all > ok > >> The main issue here is that the standard file descriptors on Windows >> are -10 (STD_INPUT_HANDLE), -11 (STD_OUTPUT_HANDLE), and -12 > ouch > > i think windows file handles have very > different semantics than posix fds > (dup2, fork,..) > > so you need to do hacks if you want posix > on windows > >> * as for psxcalls: this is a C library, that exclusively uses the >> Native API. It attaches to ntdll.dll during run-time, and can thus >> be compiled as a "native" static library with no external >> dependencies. While it is currently in its very initial stage (and >> not yet online), the major "obstacle" features -- including fork() >> -- have already been implemented. To remove all doubts, I am aware >> of Cygwin's existence, yet am looking for a high-performance >> solution that would be both "clean" >> (psxcalls-->libc-->user-library-or-application), and flexibly >> licensed. > how do you implement fork? > > (windows version of some tools actually allow redirecting > io of child processes i wonder how your fork would interact > with that, eg gawk can read/write to "/dev/fd/n" so what > would happen if you fork such a child, can you communicate > with it with pipes?) > > (at some point there was discussion about porting musl to > various systems including windows and the conclusion was > that it should be done on the syscall layer and fork would > be ugly and slow no matter what you do)