[9fans] APE select() and awkward Python subprocess PIPEfitting

[9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

I've been tracking down little errors in the APE select() function that cropped up when trying to use Python's subprocess module.  After a few too many hours of investigation, I've come to the conclusion that the code that causes error is intentional code to handle a specific case for X (I'll assume X11 until corrected):

; diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
292a293
> printf("no buffered %d\n", i);
294a296
> /*
298c300
< 				errno = EBADF;		/* how X tells a client is gone */
---
> 				errno = EBADF;		// how X tells a client is gone
300a303
> */



By removing the above, Python code that uses subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE) will now work correctly, without having to resort to os.popen3(cmd) attempts to get around the select.error: (4, 'Bad file number') that would crop up without the change.

My question is: does anyone still use the X11 code based on APE?  Is this section safe to remove in sources?  Or do you have additional recommendations to work around the select() error?

If not, I'll prep a patch.  Thanks.

-jas

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

Actually, that printf was for my debugging:

diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
294a295
> /*
298c299
< 				errno = EBADF;		/* how X tells a client is gone */
---
> 				errno = EBADF;		// how X tells a client is gone
300a302
> */


On Feb 25, 2013, at 8:29 PM, Jeff Sickel <jas@corpus-callosum.com> wrote:

> I've been tracking down little errors in the APE select() function that cropped up when trying to use Python's subprocess module.  After a few too many hours of investigation, I've come to the conclusion that the code that causes error is intentional code to handle a specific case for X (I'll assume X11 until corrected):
> 
> ; diff /sys/src/ape/lib/ap/plan9/_buf.c _buf.c
> 292a293
>> printf("no buffered %d\n", i);
> 294a296
>> /*
> 298c300
> < 				errno = EBADF;		/* how X tells a client is gone */
> ---
>> 				errno = EBADF;		// how X tells a client is gone
> 300a303
>> */
> 
> 
> 
> By removing the above, Python code that uses subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE) will now work correctly, without having to resort to os.popen3(cmd) attempts to get around the select.error: (4, 'Bad file number') that would crop up without the change.
> 
> My question is: does anyone still use the X11 code based on APE?  Is this section safe to remove in sources?  Or do you have additional recommendations to work around the select() error?
> 
> If not, I'll prep a patch.  Thanks.
> 
> -jas
> 
> 

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Yaroslav]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

2013/2/26 Jeff Sickel <jas@corpus-callosum.com>

> 've come to the conclusion that the code that causes error is intentional


We're facing problems in ceratin python apps too[1].
We used to use X11/equis mainly for rdesktop until a native port[2] became
available.

Does you change break X11/equis if relinked?

- Yaroslav

_____
[1] http://9fans.net/archive/2013/01/95
[2] http://9fans.net/archive/2013/02/58

[-- Attachment #2: Type: text/html, Size: 1235 bytes --]

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

On 26 February 2013 11:25, Yaroslav <yarikos@gmail.com> wrote:

> 've come to the conclusion that the code that causes error is intentional


I don't think that code is right. It's correct to return with a bit set in
rfds for a file descriptor that's ready with only an end of file (zero
read).

EBADF should only be given for a file descriptor number that's invalid.

If X11 is reacting to EBADF, I'd expect that would only be because it knows
that it will close a file descriptor when a client vanishes (say), but in
that case,
it's emulated file descriptor in APE should also be closed, and it's the
subsequent attempt to use it (in select or elsewhere) that should produce
EBADF.

In other words, I'd fix select, and then change whatever else needs to be
changed (if anything does) if X11 then stops working.

[-- Attachment #2: Type: text/html, Size: 1403 bytes --]

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

On Feb 26, 2013, at 5:41 AM, Charles Forsyth <charles.forsyth@gmail.com> wrote:

> In other words, I'd fix select, and then change whatever else needs to be changed (if anything does) if X11 then stops working.


If I'm interpreting this correctly, as well as the select() code,
then by removing that block from the "ensure buffered" loop the
slot selection loop should work as intended and return if any
error or read fds are ready.

It is possible that _buf.c:327 may be incorrect for actually
handling an error.  This requires some additional setup and
testing before being able to be proven.  I'm also looking for
additional APE examples that actually use select() to see if
there are any other changes required.  So far I've only found
Python and X11.

If someone has X11 and wants to test this change w/o completely
replacing /$objtype/lib/ape/libap.a, drop the edited _buf.c file
into your X11 source and add _buf.$O to your mkfile w/

pcc -c -I/sys/src/ape/lib/ap/plan9 -D_POSIX_SOURCE -D_PLAN9_SOURCE -D_BSD_EXTENSION _buf.c


-jas

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

On Feb 26, 2013, at 5:25 AM, Yaroslav <yarikos@gmail.com> wrote:

> Does you change break X11/equis if relinked?

I'm not quite sure as I don't use X11 on Plan 9 for these reasons:

1) "You really don’t want to use this. It’s old and slow and only
    works well on 8-bit displays."

2) When look at my available hardware and hosted OSs I realize
   even on my FreeBSD machines I don't have X11 installed. So,
   there's no good way for me to test.

But if the X11 port does expect EBADF on the EOF, then I'd expect
it to break.

-jas

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

On 26 February 2013 15:29, Jeff Sickel <jas@corpus-callosum.com> wrote:

> But if the X11 port does expect EBADF on the EOF, then I'd expect
> it to break.
>

It breaks anyway:
http://lists.opensuse.org/opensuse-bugs/2012-08/msg02390.html

Note, however, that the EBADF in that example is on the output side (and
outputs are always
marked as available in APE's select).

More important, let's look at the Linux source code, since that will do as
an example: http://lxr.linux.no/linux+v2.6.39/fs/select.c#L595
(mind yer lunch!).

It generates EBADF only when a file descriptor in the read or write sets
isn't open.
If APE's select does the same, it will be fine, without breaking Python's
pipes.

In that loop i from 0 to OPEN_MAX-1, on read and write sides, it should
check each f in the read and write sets for  (f->flags&FD_ISOPEN) != 0, or
set EBADF and return -1.

[-- Attachment #2: Type: text/html, Size: 1841 bytes --]

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

On 26 February 2013 16:20, Charles Forsyth <charles.forsyth@gmail.com>wrote:

> If APE's select does the same, it will be fine, without breaking Python's
> pipes.


"it will be fine" -- I meant X11 will be fine, since it's relying on
closing a file descriptor in
one part of the program to cause EBADF in the select in another part, which
will cause it
to check connections. But, but, why doesn't it just ... ?  Never mind, I
don't think we need to work that out,
just behave in the same way as Linux select.

[-- Attachment #2: Type: text/html, Size: 949 bytes --]

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

On Feb 26, 2013, at 10:26 AM, Charles Forsyth <charles.forsyth@gmail.com> wrote:

> 
> On 26 February 2013 16:20, Charles Forsyth <charles.forsyth@gmail.com> wrote:
> If APE's select does the same, it will be fine, without breaking Python's pipes.
> 
> "it will be fine" -- I meant X11 will be fine, since it's relying on closing a file descriptor in
> one part of the program to cause EBADF in the select in another part, which will cause it
> to check connections. But, but, why doesn't it just ... ?  Never mind, I don't think we need to work that out,
> just behave in the same way as Linux select.

Hopefully with enough time to recover for another meal.

Of course, if we're going the Linux route, we'll need to up FD_SETSIZE to at
least 1024, we're currently at 96.  Seems that 1024 has become the default
these days.

http://www.freebsd.org/cgi/man.cgi?query=select&sektion=2&manpath=FreeBSD+9.1-RELEASE
http://svnweb.freebsd.org/base/stable/9/sys/sys/select.h?view=markup
http://fxr.watson.org/fxr/source/kern/sys_generic.c?v=FREEBSD91#L893

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On 26 February 2013 17:01, Jeff Sickel <jas@corpus-callosum.com> wrote:

> Of course, if we're going the Linux route, we'll need to up FD_SETSIZE to
> at
> least 1024, we're currently at 96.  Seems that 1024 has become the default
> these days.
>

I meant only for trying to work out what seems to satisfy X11 as regards
error returns,
and even X11 is unlikely to need 1024 connections.

[-- Attachment #2: Type: text/html, Size: 762 bytes --]

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Anthony Sorace]

On Feb 26, 2013, at 10:29, Jeff Sickel <jas@corpus-callosum.com> wrote:

> 1) "You really don’t want to use this. It’s old and slow and only
>    works well on 8-bit displays."

I believe that's about the old X11 port. fgb's equis is much newer and nicer (aside from, y'know, X11). That's the one forest against (for anyone who's going to try).

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

On Feb 26, 2013, at 10:48 AM, Anthony Sorace <a@9srv.net> wrote:

> I believe that's about the old X11 port. fgb's equis is much newer and nicer (aside from, y'know, X11). That's the one forest against (for anyone who's going to try).


While we're at it, does anyone have an answer to why we have both

	/sys/include/ape/select.h
	/sys/include/ape/sys/select.h

From what I can tell it was a result of the cut being copied
to the new release:

; ls -l select.h sys/select.h
--rw-rw-r-- M 3198 glenda sys 779 Dec 11  1999 select.h
--rw-rw-r-- M 3198 glenda sys 779 Feb 28  2002 sys/select.h


I motion we rm /sys/include/ape/select.h and keep the <sys/select.h>
per most other posix-like systems.

-jas

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Federico G. Benavento]

you know that there's that I ported X11 years ago which does the 32bit if your
displays supports it, how else did people manage to run opera, firefox and others
on linuxemu?

On Feb 26, 2013, at 12:29 PM, Jeff Sickel <jas@corpus-callosum.com> wrote:

> 1) "You really don’t want to use this. It’s old and slow and only
>    works well on 8-bit displays."

---
Federico G. Benavento
benavento@gmail.com

Re: [9fans] APE select() and awkward Python subprocess PIPEfitting

[Jeff Sickel]

Do you know how your port of X11 handles the select call?  It looks like the
old code I've commented out was there to support he old X11 port.

Guess I'm also an unusual 9fan, not running X11 or Linux as a comparison.

-jas

On Feb 26, 2013, at 8:03 PM, Federico G. Benavento <benavento@gmail.com> wrote:

> you know that there's that I ported X11 years ago which does the 32bit if your
> displays supports it, how else did people manage to run opera, firefox and others
> on linuxemu?
> 
> On Feb 26, 2013, at 12:29 PM, Jeff Sickel <jas@corpus-callosum.com> wrote:
> 
>> 1) "You really don’t want to use this. It’s old and slow and only
>>   works well on 8-bit displays."