Re: [9fans] QTCTL?

[erik quanstrom <quanstro@quanstro.net>]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

i haven't thought this through, but perhaps this would
be an easier problem if we didn't change 9p, but we changed
the model of the caching server.

the current proposals assume that the caching servers don't
know that other caches are out there.  alternatively,
the caching servers participate in a coherency protocol with
9p clients none-the-wiser.

- erik

[-- Attachment #2: Type: message/rfc822, Size: 6133 bytes --]

From: "Russ Cox" <rsc@swtch.com>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] QTCTL?
Date: Thu, 1 Nov 2007 10:28:48 -0400
Message-ID: <20071101142849.BA1A61E8C4C@holo.morphisms.net>

> Do you recall what the issues were?

The main proposal I remember was to give out read and write
tokens and then revoke them as appropriate.  The two big problems
with this were that it meant having spontaneous server->client->server
message exchange instead of the usual client->server->client
(a very large semantic break with existing 9P) and that once you
had granted a read token to a client the client could simply stop
responding (acknowledging the revocation) and slow down the
rest of the system.

I think that an ideal caching solution would have the following
properties.

1. For 9P servers that don't care to let their files be cached,
the effort in doing so should be minimal.  For example, perhaps
servers would simply respond to a Tcache message with Rerror,
like non-authenticating servers respond to Tauth with Rerror.
Anything more is not going to get implemented.

2. All 9P messages would still be client->server->client.
This fits with #1, but also excludes solutions that introduce
new server->client->server messages after a successful Tcache.

3. If a client that is caching some data stops responding, the rest
of the system can continue to function without it: slow clients
don't slow the entire system.

4. Except for timing, the cached behavior is identical to what 
you'd see in an uncached setting, not some relaxed semantics.
For example, suppose you adopted a model where each server
response could have some cache invalidations piggybacked on it.
This would provide a weak but precise consistency model in that
any cached behaviors observed interacting with that server
would be the same as shifting uncached behavior back in time a bit.
It could be made to appear that the machine was just a few seconds
behind the server, but otherwise fully consistent.  The problem with
this is when multiple machines are involved, and since Plan 9 is
a networked system, this happens.  For example, a common setup
is for one machine to spool mail into /mail/queue and then run
rx to another machine to kick the queue processor (the mail sender).
If the other machine is behaving like it's 5 seconds behind, then it
won't see the mail that just got spooled, making the rx kick worthless.

5. It is easy to get right.

#1 is trivial.  #2 and #3 are difficult and point to some kind of 
lease-based solution instead of tokens.  

#4 keeps us honest: weakened consistency like in my example
or in cfs(4) or in recover(4) might occasionally be useful, but it
will break important and subtle real-world cases and make the
system a lot more fragile.  If you pile up enough things that only
work 99% of the time, you very quickly end up with a crappy system.
(If that's what you want, might I suggest Linux?)

#5 is probably wishful thinking on my part.

> Wouldn't something like load-linked/store-conditional suffice
> if the common case is a single writer?  When the client does
> a "read-linked" call, the server sends an ID along with the
> data. The client can then do a "write-conditional" by passing
> the original ID and new data.  If the ID is not valid anymore
> (if someone else wrote in the meantime) the write fails.  The
> server doesn't have to keep any client state or inform anyone
> about an invalid cache.  Of course, if any client fails to
> follow this protocol things fall apart but at least well
> behaved clients can get coherency. And this would work for
> cases such as making changes on a disconnected laptop and
> resyncing to the main server on the next connect.  You
> wouldn't use this for synthetic files.  This ID can be as
> simple as a file "generation" number incremented on each
> write or crypto strong checksum.

This doesn't solve the problem of one client caching the file
contents and another writing to the file; how does the first
find out that the file has changed before it uses the cached 
contents again?

Russ