From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <914d0489b3dee458eceaf94593dbc71d@quanstro.net> To: 9fans@cse.psu.edu Subject: Re: [9fans] QTCTL? From: erik quanstrom Date: Thu, 1 Nov 2007 10:38:22 -0400 In-Reply-To: <20071101142849.BA1A61E8C4C@holo.morphisms.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="upas-opejjktcgijpbsdacyxpzmpugq" Topicbox-Message-UUID: e2ccec24-ead2-11e9-9d60-3106f5b1d025 This is a multi-part message in MIME format. --upas-opejjktcgijpbsdacyxpzmpugq Content-Disposition: ainline Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit i haven't thought this through, but perhaps this would be an easier problem if we didn't change 9p, but we changed the model of the caching server. the current proposals assume that the caching servers don't know that other caches are out there. alternatively, the caching servers participate in a coherency protocol with 9p clients none-the-wiser. - erik --upas-opejjktcgijpbsdacyxpzmpugq Content-Type: message/rfc822 Content-Disposition: ainline Received: from mail.cse.psu.edu ([130.203.4.6]) by ladd; Thu Nov 1 10:29:48 EDT 2007 Received: from psuvax1.cse.psu.edu (localhost [127.0.0.1]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id 31FADFAB5 for ; Thu, 1 Nov 2007 10:29:47 -0400 (EDT) X-Original-To: 9fans@cse.psu.edu Delivered-To: 9fans@cse.psu.edu Received: from localhost (localhost [127.0.0.1]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id 306E7FA6F for <9fans@cse.psu.edu>; Thu, 1 Nov 2007 10:28:57 -0400 (EDT) Received: from mail.cse.psu.edu ([127.0.0.1]) by localhost (psuvax1 [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 18943-01-36 for <9fans@cse.psu.edu>; Thu, 1 Nov 2007 10:28:55 -0400 (EDT) Received: from holo.morphisms.net (holo.morphisms.net [216.254.78.15]) by mail.cse.psu.edu (CSE Mail Server) with ESMTP id 6575DF9DA for <9fans@cse.psu.edu>; Thu, 1 Nov 2007 10:28:50 -0400 (EDT) Received: by holo.morphisms.net (Postfix, from userid 1002) id BA1A61E8C4C; Thu, 1 Nov 2007 10:28:49 -0400 (EDT) To: 9fans@cse.psu.edu Subject: Re: [9fans] QTCTL? From: "Russ Cox" Date: Thu, 1 Nov 2007 10:28:48 -0400 In-Reply-To: <20071101062133.49DF25B30@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Message-Id: <20071101142849.BA1A61E8C4C@holo.morphisms.net> X-Virus-Scanned: by amavisd-new at cse.psu.edu X-BeenThere: 9fans@cse.psu.edu X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu> List-Id: Fans of the OS Plan 9 from Bell Labs <9fans.cse.psu.edu> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: 9fans-bounces+quanstro=quanstro.net@cse.psu.edu Errors-To: 9fans-bounces+quanstro=quanstro.net@cse.psu.edu > Do you recall what the issues were? The main proposal I remember was to give out read and write tokens and then revoke them as appropriate. The two big problems with this were that it meant having spontaneous server->client->server message exchange instead of the usual client->server->client (a very large semantic break with existing 9P) and that once you had granted a read token to a client the client could simply stop responding (acknowledging the revocation) and slow down the rest of the system. I think that an ideal caching solution would have the following properties. 1. For 9P servers that don't care to let their files be cached, the effort in doing so should be minimal. For example, perhaps servers would simply respond to a Tcache message with Rerror, like non-authenticating servers respond to Tauth with Rerror. Anything more is not going to get implemented. 2. All 9P messages would still be client->server->client. This fits with #1, but also excludes solutions that introduce new server->client->server messages after a successful Tcache. 3. If a client that is caching some data stops responding, the rest of the system can continue to function without it: slow clients don't slow the entire system. 4. Except for timing, the cached behavior is identical to what you'd see in an uncached setting, not some relaxed semantics. For example, suppose you adopted a model where each server response could have some cache invalidations piggybacked on it. This would provide a weak but precise consistency model in that any cached behaviors observed interacting with that server would be the same as shifting uncached behavior back in time a bit. It could be made to appear that the machine was just a few seconds behind the server, but otherwise fully consistent. The problem with this is when multiple machines are involved, and since Plan 9 is a networked system, this happens. For example, a common setup is for one machine to spool mail into /mail/queue and then run rx to another machine to kick the queue processor (the mail sender). If the other machine is behaving like it's 5 seconds behind, then it won't see the mail that just got spooled, making the rx kick worthless. 5. It is easy to get right. #1 is trivial. #2 and #3 are difficult and point to some kind of lease-based solution instead of tokens. #4 keeps us honest: weakened consistency like in my example or in cfs(4) or in recover(4) might occasionally be useful, but it will break important and subtle real-world cases and make the system a lot more fragile. If you pile up enough things that only work 99% of the time, you very quickly end up with a crappy system. (If that's what you want, might I suggest Linux?) #5 is probably wishful thinking on my part. > Wouldn't something like load-linked/store-conditional suffice > if the common case is a single writer? When the client does > a "read-linked" call, the server sends an ID along with the > data. The client can then do a "write-conditional" by passing > the original ID and new data. If the ID is not valid anymore > (if someone else wrote in the meantime) the write fails. The > server doesn't have to keep any client state or inform anyone > about an invalid cache. Of course, if any client fails to > follow this protocol things fall apart but at least well > behaved clients can get coherency. And this would work for > cases such as making changes on a disconnected laptop and > resyncing to the main server on the next connect. You > wouldn't use this for synthetic files. This ID can be as > simple as a file "generation" number incremented on each > write or crypto strong checksum. This doesn't solve the problem of one client caching the file contents and another writing to the file; how does the first find out that the file has changed before it uses the cached contents again? Russ --upas-opejjktcgijpbsdacyxpzmpugq--