Pipes and Standard Output / Input on Windows

Pipes and Standard Output / Input on Windows

[John Whitington]

Hullo.

I'm compiling OCaml command-line software with the MSVC toolchain on  
Windows. Users have noticed two problems:

(a) When output (such as the --help) is done on Windows, the help  
doesn't all show (I fixed this by flushing stdout manually - but  
shouldn't stdout be flushed when the process exits anyway?)

(b) Chaining invocations of the tool together using pipes on Windows  
often fails. The second process in the chain gets an End_Of_File after  
only a few hundred bytes of data. I've confirmed the data is all being  
output by the first process, and the first process is exiting cleanly.  
All the open_in and open_out calls are using the _bin variant. The  
data being sent down the pipe is a PDF file (which contains binary  
sections).

Neither of these problems occur on Linux / Mac builds - is there  
something about windows pipes I should know?

-- 
John Whitington
Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: Pipes and Standard Output / Input on Windows

[Sylvain Le Gall]

On 29-05-2008, John Whitington <john@coherentgraphics.co.uk> wrote:
> Hullo.
>
> I'm compiling OCaml command-line software with the MSVC toolchain on  
> Windows. Users have noticed two problems:
>
> (b) Chaining invocations of the tool together using pipes on Windows  
> often fails. The second process in the chain gets an End_Of_File after  
> only a few hundred bytes of data. I've confirmed the data is all being  
> output by the first process, and the first process is exiting cleanly.  
> All the open_in and open_out calls are using the _bin variant. The  
> data being sent down the pipe is a PDF file (which contains binary  
> sections).
>
> Neither of these problems occur on Linux / Mac builds - is there  
> something about windows pipes I should know?
>

Yep, on windows GetStdHandle doesn't always return the same kind of
handle depending on the context:
- if the output/input is console, it is console handle (i.e. need to be
  manipulated through console function)
- if the output/input is a pipe, it is a pipe handle (i.e. need to be
  manipulated through pipe function)

And since we are in the good old world of windows, the return type is an
HANDLE whatever it is ;-)

I think your issue is related to this difference. It can be fixed, but
you need a little bit of knowledge about windows pipe/standard handle.

Regards,
Sylvain Le Gall

RE: [Caml-list] Re: Pipes and Standard Output / Input on Windows

[David Allsopp]

> Yep, on windows GetStdHandle doesn't always return the same kind of
> handle depending on the context:
> - if the output/input is console, it is console handle (i.e. need to be
>   manipulated through console function)
> - if the output/input is a pipe, it is a pipe handle (i.e. need to be
>   manipulated through pipe function)

Both these kinds of HANDLE work transparently with ReadFile/WriteFile API
functions so should be fine with the underlying OCaml calls (OCaml's file
I/O is based on the C I/O routines which won't be using
ReadConsole/WriteConsole) - what scenario are you referring to where the
difference between a console handle and a pipe would be a problem?


David

Re: Pipes and Standard Output / Input on Windows

[Sylvain Le Gall]

On 29-05-2008, David Allsopp <dra-news@metastack.com> wrote:
>> Yep, on windows GetStdHandle doesn't always return the same kind of
>> handle depending on the context:
>> - if the output/input is console, it is console handle (i.e. need to be
>>   manipulated through console function)
>> - if the output/input is a pipe, it is a pipe handle (i.e. need to be
>>   manipulated through pipe function)
>
> Both these kinds of HANDLE work transparently with ReadFile/WriteFile API
> functions so should be fine with the underlying OCaml calls (OCaml's file
> I/O is based on the C I/O routines which won't be using
> ReadConsole/WriteConsole) - what scenario are you referring to where the
> difference between a console handle and a pipe would be a problem?
>

"Unix.select"?

Regards,
Sylvain Le Gall

RE: [Caml-list] Pipes and Standard Output / Input on Windows

[David Allsopp]

> All the open_in and open_out calls are using the _bin variant. The  
> data being sent down the pipe is a PDF file (which contains binary  
> sections).

Why is open_in_bin needed to access stdin and stdout? I've personally had no
problems chaining processes in OCaml under Windows (and the particular
chaining is the control program for a Win32 Service so it's a long-lived
process chain) though I did need to use set_binary_mode_in and
set_binary_mode_out on stdin, stdout and stderr to disable OCaml's \r\n
masking but I think that was for an application-specific reason rather than
because something didn't work with the pipes.


David

Re: Pipes and Standard Output / Input on Windows

[Sylvain Le Gall]

On 29-05-2008, John Whitington <john@coherentgraphics.co.uk> wrote:
> Hullo.
>
> I'm compiling OCaml command-line software with the MSVC toolchain on  
> Windows. Users have noticed two problems:
>
> (a) When output (such as the --help) is done on Windows, the help  
> doesn't all show (I fixed this by flushing stdout manually - but  
> shouldn't stdout be flushed when the process exits anyway?)
>
> (b) Chaining invocations of the tool together using pipes on Windows  
> often fails. The second process in the chain gets an End_Of_File after  
> only a few hundred bytes of data. I've confirmed the data is all being  
> output by the first process, and the first process is exiting cleanly.  
> All the open_in and open_out calls are using the _bin variant. The  
> data being sent down the pipe is a PDF file (which contains binary  
> sections).
>
> Neither of these problems occur on Linux / Mac builds - is there  
> something about windows pipes I should know?
>

I am not sure if it is related, but i have a problem with "|" command
line pipe. It seems that this way channel are not binary one... If you
happen to transmit binary data through it, you could have problem...

I detect it, by trying "output_value stdin x".

Do you think your problem is related?

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: Pipes and Standard Output / Input on Windows

[John Whitington]

On 1 Jun 2008, at 22:06, Sylvain Le Gall wrote:

> On 29-05-2008, John Whitington <john@coherentgraphics.co.uk> wrote:
>>
>> I'm compiling OCaml command-line software with the MSVC toolchain on
>> Windows. Users have noticed two problems:
>>
>> (a) When output (such as the --help) is done on Windows, the help
>> doesn't all show (I fixed this by flushing stdout manually - but
>> shouldn't stdout be flushed when the process exits anyway?)

Still no clue on this one, I'm afraid.

>> (b) Chaining invocations of the tool together using pipes on Windows
>> often fails. The second process in the chain gets an End_Of_File  
>> after
>> only a few hundred bytes of data. I've confirmed the data is all  
>> being
>> output by the first process, and the first process is exiting  
>> cleanly.
>> All the open_in and open_out calls are using the _bin variant. The
>> data being sent down the pipe is a PDF file (which contains binary
>> sections).
>>
>> Neither of these problems occur on Linux / Mac builds - is there
>> something about windows pipes I should know?
>>
>
> I am not sure if it is related, but i have a problem with "|" command
> line pipe. It seems that this way channel are not binary one... If you
> happen to transmit binary data through it, you could have problem...
>
> I detect it, by trying "output_value stdin x".
>
> Do you think your problem is related?

It seems this is correct. See the section "Problems with Pipes" here:

http://dev.mysql.com/doc/refman/5.0/en/windows-vs-unix.html

So I'll need to encode the data somehow, or introduce my own chaining
separator on the command line.

-- 
John Whitington
Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] Re: Pipes and Standard Output / Input on Windows

['Sylvain Le Gall']

On Mon, Jun 02, 2008 at 11:32:18AM +0100, David Allsopp wrote:
> > It seems this is correct. See the section "Problems with Pipes" here:
> >
> > http://dev.mysql.com/doc/refman/5.0/en/windows-vs-unix.html
> 
> See http://archives.postgresql.org/pgsql-hackers-win32/2005-01/msg00227.php
> for an equivalent Postgres post but with a solution :o)
> 
> As I mentioned in my previous post, you need to use set_binary_mode_in and
> set_binary_mode_out on both stdin (to prevent ^Z ending the file) and stdout
> (to prevent \n being translated to \r\n). Having done this, I get no
> problems with ^Z characters in a test piping script. Does it fix your
> problem?

Sorry, i must have read too quickly.

It solve my problem indeed.

Regards
Sylvain Le Gall

RE: [Caml-list] Re: Pipes and Standard Output / Input on Windows

[David Allsopp]

> It seems this is correct. See the section "Problems with Pipes" here:
>
> http://dev.mysql.com/doc/refman/5.0/en/windows-vs-unix.html

See http://archives.postgresql.org/pgsql-hackers-win32/2005-01/msg00227.php
for an equivalent Postgres post but with a solution :o)

As I mentioned in my previous post, you need to use set_binary_mode_in and
set_binary_mode_out on both stdin (to prevent ^Z ending the file) and stdout
(to prevent \n being translated to \r\n). Having done this, I get no
problems with ^Z characters in a test piping script. Does it fix your
problem too?


David

Re: [Caml-list] Re: Pipes and Standard Output / Input on Windows

[John Whitington]

On 2 Jun 2008, at 13:46, David Allsopp wrote:

>> It seems this is correct. See the section "Problems with Pipes" here:
>>
>> http://dev.mysql.com/doc/refman/5.0/en/windows-vs-unix.html
>
> See http://archives.postgresql.org/pgsql-hackers-win32/2005-01/msg00227.php
> for an equivalent Postgres post but with a solution :o)
>
> As I mentioned in my previous post, you need to use  
> set_binary_mode_in and
> set_binary_mode_out on both stdin (to prevent ^Z ending the file)  
> and stdout
> (to prevent \n being translated to \r\n). Having done this, I get no
> problems with ^Z characters in a test piping script. Does it fix your
> problem too?


This does indeed fix the problem here. Thanks!

-- 
John Whitington
Coherent Graphics Ltd
http://www.coherentpdf.com/