Re: [9fans] Mac p9p snarf buffer

Re: [9fans] Mac p9p snarf buffer

[Russ Cox]

I made some X fixes that solve at least half the problem Rob reported.

	cd $PLAN9/src/cmd/devdraw; mk install; mk clean

on the machines where the apps are running (which might not
be the Mac if you are using ssh -X to run remote X apps).

I have no good answer for Skip's problem -- X11 shouldn't even
be involved.  The rest of this mail is an explanation of the 
various problems.

There are many different moving parts here, and each configuration can
produce a different failure (or success).


+ X11 selections v.  Pasteboard

X11 doesn't have a snarf buffer.  Instead it has an idea of which
window currently "owns" the snarf (X11 would say `selection'), and
when you want to find the snarf contents you go ask the current owner.
There is no central buffer like on Plan 9's /dev/snarf or the Windows
clipboard.  (In addition to making things a lot more complicated, this
means that snarf contents do not persist once their owner exits.  But
that particular problem isn't relevant here.)

OS X does have a snarf buffer, called the Pasteboard.  There are
programs pbcopy and pbpaste that come with OS X that write to and read
from the Pasteboard.

The OS X X11 server has the unenviable task of making the X11 madness
interoperate with native OS X. As far as the snarf buffer is
concerned, it's job should be to synchronize in both directions:

  * Every time an OS X app writes to the pasteboard, X11 should claim
  to own the snarf, and when apps request the snarf from it, it should
  serve the request using the pasteboard contents.

  * Every time an X11 app claims the snarf, X11 should fetch the snarf
  from it and write to the pasteboard.

Bug #1: some versions of the OS X X11 server require you to select
"Copy" from the X11 Edit menu before fetching the snarf and writing it
to the pasteboard.

To work around bug #1, p9p apps on OS X write directly to the
pasteboard themselves rather than depend on the X11 server.  nm
$PLAN9/bin/devdraw | grep Pasteboard should confirm this.

Unfortunately, this workaround does not work when running an app on a
remote machine via ssh -X.  Then the only connection to the local
pasteboard is via X11 so apps are at its mercy.


+ X11 selection updates

As best I can tell, if you own the snarf then another app can send you
an XSelectionRequest request, which turns into an
XSelectionRequestEvent event.  Then you copy the snarf into a public
attribute on the X11 server using XChangeProperty and use XSendEvent
to send an XSelectionEvent to the requestor (the id is in the initial
request) to tell it that the snarf is ready.

Bug #2.  On OS X using ssh -X forwarding, the X11 server does not let
the final XSendEvent run.  Instead it returns with an error and the
app often gives up.  I was trying this with "X11 1.1.3 - XFree86
4.4.0", ssh -X to Linux, and then running xterm or gtkedit or leafpad,
and they all crashed after the server rejected one message or another.
So remote apps seem not to be very reliable in some versions of X11 on
OS X.


+ X11 selection formats

X11 allows (nay, encourages) selections to be arbitrary blocks of
data.  When you fetch the snarf from an owner, the first thing you do
is ask for a list of possible formats.  Then you look in the format
list for one that sounds good to you.  The most common is "STRING".
Other apps are free to make up names for other strings.  Most seem to
have standardized on "UTF8_STRING" for a UTF-8 string, though some do
things like "text/plain;charset=UTF-8".

Bug #3 (fixed).  It had been working okay to respond to all these
various kinds of strings with UTF-8 data and to ask for plain "STRING"
and hope it would be UTF-8.  Apparently on OS X that is not okay
anymore -- when you ask for "STRING" from the X11 server it replaces
non-ASCII characters with ?.

I changed p9p to try "UTF8_STRING" before trying "STRING", and now
pasting UTF-8 into p9p apps on OS X works for me.

I tried to reproduce the "garbage out when copying UTF-8 from p9p to
OS X" but it works fine for me locally.  Because of Bug #2 I can't
tell whether copying UTF-8 from a remote X11 window to a local OS X
app produces garbage or not.  I changed the returned format list to
mention "UTF8_STRING" before "STRING", so that apps that request the
format list (not all do) and look for the first one they like will
prefer UTF-8.


+ Avoiding X11

Paul Lalonde suggested using native OS X bindings instead of relying
on X11.  This is a good idea and one that I hope will happen soon,
but it won't entirely solve the problem.  People do run remote
X apps via ssh -X, and most of the problems (like Bug #3) would
have eventually cropped up on the other X-based systems if left
unfixed.  So while I encourage having a native OS X devdraw,
we still need to figure out how to play nice with the OS X X11.

Russ

Re: [9fans] Mac p9p snarf buffer

[Kris Maglione]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Thu, May 03, 2007 at 05:20:08PM -0400, Russ Cox wrote:
>+ X11 selections v.  Pasteboard
>
>X11 doesn't have a snarf buffer.  Instead it has an idea of which
>window currently "owns" the snarf (X11 would say `selection'), and
>when you want to find the snarf contents you go ask the current owner.
>There is no central buffer like on Plan 9's /dev/snarf or the Windows
>clipboard.  (In addition to making things a lot more complicated, this
>means that snarf contents do not persist once their owner exits.  But
>that particular problem isn't relevant here.)

As if that weren't bad enough, X11 *also* has central buffers, "cut 
buffers", which are deprecated, but still used by some apps. And there 
are 3 selections, PRIMARY, SECONDARY, and CLIPBOARD. Some apps, 
expecially browsers and Gnome/KDEish apps, tend to use the PRIMARY and 
CLIPBOARD selections, depending on what you're doing (middle click 
pastes the PRIMARY, ^V pastes the CLIPBOARD). It's a terrific mess.

But let's not sully this list with X11 insanity any more.

-- 
Kris Maglione

If you lived here you'd be home now.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [9fans] Mac p9p snarf buffer

[Scott Schwartz]

| X11 doesn't have a snarf buffer.  Instead it has an idea of which
| window currently "owns" the snarf (X11 would say `selection'), and
| when you want to find the snarf contents you go ask the current owner.
| There is no central buffer like on Plan 9's /dev/snarf or the Windows
| clipboard.  (In addition to making things a lot more complicated, this
| means that snarf contents do not persist once their owner exits.  But
| that particular problem isn't relevant here.)
 
No, it really does have a CLIPBOARD property, separate from selections
like PRIMARY.  A program like xclipboard or kde's klipper or whatever
the gnome thing is, maintains this.

Re: [9fans] Mac p9p snarf buffer

[Derek Fawcus]

On Thu, May 03, 2007 at 08:00:50PM -0400, Kris Maglione wrote:
> 
> As if that weren't bad enough, X11 *also* has central buffers, "cut 
> buffers", which are deprecated, but still used by some apps. And there 
> are 3 selections, PRIMARY, SECONDARY, and CLIPBOARD. Some apps, 
> expecially browsers and Gnome/KDEish apps, tend to use the PRIMARY and 
> CLIPBOARD selections, depending on what you're doing (middle click 
> pastes the PRIMARY, ^V pastes the CLIPBOARD). It's a terrific mess.

There is an attempt underway to get some sanity in use...

http://standards.freedesktop.org/clipboards-spec/clipboards-0.1.txt

As to cut buffers,  there is no agreed data type in them,  so one
cannot know if they are ascii,  latin1,  utf-8,  or something else.

DF

Re: [9fans] Mac p9p snarf buffer

[Derek Fawcus]

On Thu, May 03, 2007 at 05:20:08PM -0400, Russ Cox wrote:
> + X11 selection formats
> 
> X11 allows (nay, encourages) selections to be arbitrary blocks of
> data.  When you fetch the snarf from an owner, the first thing you do
> is ask for a list of possible formats.  Then you look in the format
> list for one that sounds good to you.  The most common is "STRING".
> Other apps are free to make up names for other strings.  Most seem to
> have standardized on "UTF8_STRING" for a UTF-8 string, though some do
> things like "text/plain;charset=UTF-8".

( For anyone interested,  there are gory details at
  http://www.pps.jussieu.fr/~jch/software/UTF8_STRING/UTF8_STRING.text )

> Bug #3 (fixed).  It had been working okay to respond to all these
> various kinds of strings with UTF-8 data and to ask for plain "STRING"
> and hope it would be UTF-8.  Apparently on OS X that is not okay
> anymore -- when you ask for "STRING" from the X11 server it replaces
> non-ASCII characters with ?.

That sounds like a bug in the apple stuff.  AFAICR "STRING" encoding
means 8 bit latin1 (8859-1) characters.  So unless the apple code is
complaining about bytes for which there is no 8859-1 character,  I'm
not sure what's happening.

DF

Re: [9fans] Mac p9p snarf buffer

[Kris Maglione]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

On Fri, May 04, 2007 at 01:29:03PM +0100, Derek Fawcus wrote:
>That sounds like a bug in the apple stuff.  AFAICR "STRING" encoding
>means 8 bit latin1 (8859-1) characters.  So unless the apple code is
>complaining about bytes for which there is no 8859-1 character,  I'm
>not sure what's happening.

The behavior is undefined if a STRING is not in the 'Host Portable 
Character Set', but it's not the X server which causes the problem, it's 
the owner of the selection which converts it to a STRING from UTF-8. I'd 
guess that most apps use iconv (ick) for the conversion. Some apps give 
you a '?', some give \xXXXX, \uXXXX, or the like.

Incidentally, snarfer is still broken in this manner.

-- 
Kris Maglione

If at first you don't succeed, try something else.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [9fans] Mac p9p snarf buffer

[Russ Cox]

> Yes, this bothered the hell out of me, too. I fixed my copy of
> devdraw(1) to ask for a UTF8_STRING before a STRING, as suggested by the
> comment, which seems to imply that the extra polling is not worth the
> effort:

This is now fixed in the latest CVS,
as is the other direction (a completely
different fix, of course).

Russ

Re: [9fans] Mac p9p snarf buffer

[Kris Maglione]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

On Thu, May 03, 2007 at 10:52:26AM -0700, Rob Pike wrote:
> The snarf issue that bothers me most these days is that the
> plan9port programs, at least on the mac, don't handle UTF-8
> in the snarf buffer correctly.  UTF-8 going out turns into junk;
> going in it turns into question marks.

Yes, this bothered the hell out of me, too. I fixed my copy of 
devdraw(1) to ask for a UTF8_STRING before a STRING, as suggested by the 
comment, which seems to imply that the extra polling is not worth the 
effort:

	 * We should try to go for _x.utf8string instead of XA_STRING,
	 * but that would add to the polling.

I hacked this together in about a minute, so it's not beautiful, but it 
works.

$PLAN9/src/cmd/devdraw/x11-itrans.c:439:
	prop = 1;
	data = nil;
	XChangeProperty(_x.display, _x.drawable, prop, _x.utf8string, 8, PropModeReplace, (uchar*)"", 0);
	XConvertSelection(_x.display, clipboard, _x.utf8string, prop, _x.drawable, CurrentTime);
	XFlush(_x.display);
	lastlen = 0;
	for(i=0; i<10 || (lastlen!=0 && i<30); i++){
		usleep(100*1000);
		XGetWindowProperty(_x.display, _x.drawable, prop, 0, 0, 0, AnyPropertyType,
			&type, &fmt, &dummy, &len, &data);
		if(lastlen == len && len > 0)
			break;
		lastlen = len;
	}
	if(lastlen == 0) {
		XChangeProperty(_x.display, _x.drawable, prop, XA_STRING, 8, PropModeReplace, (uchar*)"", 0);
		XConvertSelection(_x.display, clipboard, XA_STRING, prop, _x.drawable, CurrentTime);
		XFlush(_x.display);
		lastlen = 0;
		for(i=0; i<10 || (lastlen!=0 && i<30); i++){
			usleep(100*1000);
			XGetWindowProperty(_x.display, _x.drawable, prop, 0, 0, 0, AnyPropertyType,
				&type, &fmt, &dummy, &len, &data);
			if(lastlen == len && len > 0)
				break;
			lastlen = len;
		}
	}
	if(lastlen == 0)
		goto out;

-- 
Kris Maglione


[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [9fans] Mac p9p snarf buffer

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The right answer is for someone motivated (it won't be me until late  
summer, I'm afraid) to take the OS X native drawterm back end and  
drop it into p9p.  And maybe fix the couple of bugs left in there  
(changing screen resolution confuses it because the window resize  
routine doesn't block to the plan 9 desktop size, and the wnidow-on- 
top code is borked, making drawterm sit on top of the spotlight  
pulldown, for example).

Paul

On May 3, 2007, at 10:52 AM, Rob Pike wrote:

> The snarf issue that bothers me most these days is that the
> plan9port programs, at least on the mac, don't handle UTF-8
> in the snarf buffer correctly.  UTF-8 going out turns into junk;
> going in it turns into question marks.
>
> -rob

- -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFGOiVEpJeHo/Fbu1wRAvSHAJ0XuMNcxYwW0JQn25UevrZr84gE2wCglTms
qTDsRzjsxgmgHMLBAHhhV0w=
=P3Eo
- -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFGOiVNpJeHo/Fbu1wRAuUaAJ4rzuS5eoo4WvJWf9tESn33plSUMACgjxyV
SJ7Xdt56CDFJxRyCngvLdss=
=9npk
-----END PGP SIGNATURE-----

Re: [9fans] Mac p9p snarf buffer

[Skip Tavakkolian]

> if you have a program called $PLAN9/bin/devdraw then you
> should not be having this problem.

i do, and i am, pradoxically ☺
i tried snarfer -v, same result. i'm running xdarwin 1.4a1 (xfree86 4.5.0)

Re: [9fans] Mac p9p snarf buffer

[Rob Pike]

The snarf issue that bothers me most these days is that the
plan9port programs, at least on the mac, don't handle UTF-8
in the snarf buffer correctly.  UTF-8 going out turns into junk;
going in it turns into question marks.

-rob

Re: [9fans] Mac p9p snarf buffer

[David Leimbach]

On 5/3/07, Rob Pike <robpike@gmail.com> wrote:
> On 5/3/07, Skip Tavakkolian <9nut@9netics.com> wrote:
> > snarf'ed content is not passed between mac apps and p9p apps. is it a xdarwin
> > problem?
> >
> >
>
> It is if you ask it to be.  Use the copy command in X to move the data out.
> Or you can do it automatically; rsc has a program somewhere that runs as
> a daemon doing the copy for you.
>
> -rob
>

When I was using acme all the time, so I'd just select the text I need
and then Edit it out to "pbcopy".

Then it's in the clipboard for whatever else should need it.

Re: [9fans] Mac p9p snarf buffer

[Russ Cox]

> snarf'ed content is not passed between mac apps and p9p apps. is it a xdarwin
> problem?

yes, but i fixed it (by writing directly to the apple clipboard) last year.
if you have a program called $PLAN9/bin/devdraw then you
should not be having this problem.

russ

Re: [9fans] Mac p9p snarf buffer

[Anthony Sorace]

are you running a current p9p? it didn't used to be, but has been for
quite some time. there's a program called 'snarfer' that used to be
needed, but it's not running on my system.

are you using a window manager other than quartz-wm? if so, you'll
need to run: 'quartz-wm --only-proxy &' before starting your wm to
bridge the x11/quartz snarf buffers.
anthony

Re: [9fans] Mac p9p snarf buffer

[Rob Pike]

On 5/3/07, Skip Tavakkolian <9nut@9netics.com> wrote:
> snarf'ed content is not passed between mac apps and p9p apps. is it a xdarwin
> problem?
>
>

It is if you ask it to be.  Use the copy command in X to move the data out.
Or you can do it automatically; rsc has a program somewhere that runs as
a daemon doing the copy for you.

-rob

[9fans] Mac p9p snarf buffer

[Skip Tavakkolian]

snarf'ed content is not passed between mac apps and p9p apps. is it a xdarwin
problem?