Re: [9fans] QTCTL?

[Latchesar Ionkov <lionkov@lanl.gov>]

The problem is that the clients with higher latencies badly need to  
be able to cache. And the ones with better latencies can afford not  
caching :)

On Nov 1, 2007, at 12:03 PM, Francisco J Ballesteros wrote:

> But 5 seconds would be enough to be convinced that a client is not  
> properly
> responding to invalidation requests, and cease all its leases. Why  
> make other
> clients wait for more? I mean, assuming a central FS and clients
> connected on star
> to it.
>
> On 11/1/07, Latchesar Ionkov <lionkov@lanl.gov> wrote:
>> The 5 seconds lease might work in the local network case, but not
>> caching at all is going to work out pretty well too. What if you want
>> to cache over Internet and you round-trip is 3-4 seconds :)
>>
>> On Nov 1, 2007, at 11:03 AM, Russ Cox wrote:
>>
>>>> The fact is we have loose consistency now, we just don't call it
>>>> that.
>>>> Anytime you are running a file from a server, you have loose
>>>> consistency. It works ok in most cases.
>>>
>>> Because all reads and writes go through to the
>>> server, all file system operations on a particular
>>> server are globally ordered, *and* that global ordering
>>> matches the actual sequence of events in the
>>> physical world (because clients wait for R-messages).
>>> That's a pretty strong consistency statement!
>>>
>>> Any revocation-based system has to have the server
>>> wait for an acknowledgement from the client.
>>> If there is no wait, then between the time that the server
>>> sends the "oops, stop caching this" and the client
>>> processes it, the client might incorrectly use the
>>> now-invalid data.  That's why a change file doesn't
>>> provide the same consistency guarantees as pushing
>>> all reads/writes to the file server.  To get those, revocations
>>> fundamentally must wait for the client.
>>>
>>> It's also why this doesn't work:
>>>
>>>> Tcache       asks whether the server is prepared to cache
>>>> Rcache       makes lease available with parameters, Rerror says no.
>>>>
>>>> Tlease       says, ok start my lease now (almost immediately  
>>>> follows Rache)
>>>> Rlease       lease expired or lease needs to be given back early
>>>>
>>>> Tcache       done with old lease (may immediately ask for a new  
>>>> lease)
>>>> etc.
>>>
>>> because the Rlease/Tcache sequence is a s->c->s
>>> message.  If a client doesn't respond with the Tcache
>>> to formally give up the lease, the server has no choice
>>> but to wait.
>>>
>>> If you are willing to assume that each machine has
>>> a real-time clock that runs approximately at the
>>> same rate (so that different machines agree on
>>> what 5 seconds means, but not necessarily what
>>> time it is right now), then you can fix the above messages
>>> by saying that the client lease is only good for a fixed
>>> time period (say 5 seconds) from the time that the
>>> client sent the Tlease.  Then the server can overestimate
>>> the lease length as 5 seconds from when it sent the
>>> Rlease, and everything is safe.  And if the server
>>> sends a Rlease and the client doesn't respond with
>>> a Tcache to officially renounce the lease, the server
>>> can just wait until Tlease + 5 seconds and go on.
>>> But that means the client has to be renewing the
>>> lease every 5 seconds (more frequently, actually).
>>>
>>> Also, in the case where the lease is just expiring
>>> but not being revoked, then you have to have some
>>> mechanism for establishing the new lease before
>>> the old one runs out.  If there is a time between
>>> two leases when you don't hold any leases, then
>>> all your cached data becomes invalid.
>>>
>>> The following works:
>>>
>>> Tnewlease     asks for a new lease
>>> Rnewlease     grants the lease, for n seconds starting at time of
>>> Tnewlease
>>>
>>> Trenewlease   asks to renew the lease
>>> Rrenewlease   grants the renewal for n seconds starting at time of
>>> Trenewlease
>>>
>>> Now if the server needs to revoke the lease, it just
>>> refuses to renew and waits until the current lease expires.
>>>
>>> You can add a pseudo-callback to speed up revocation
>>> with a cooperative client:
>>>
>>> Tneeditback   offers to give lease back to server early
>>> Rneeditback   says I accept your offer, please do
>>>
>>> Tdroplease    gives lease back
>>> Rdroplease    says okay I got it (not really necessary)
>>>
>>> The lease ends when the client sends Tdroplease,
>>> *not* when the server sends Rneeditback.  It can't end
>>> at Rneeditback for the same reason change files don't work.
>>> And this can *only* be an optimization, because it
>>> depends on the client sending Tdroplease.  To get
>>> something that works in the presence of misbehaved
>>> clients you have to be able to fall back on the
>>> "wait it out" strategy.
>>>
>>> One could, of course, use a different protocol with
>>> a 9P front end.  That's okay for clients, but you'd still
>>> have to teach the back-end server (i.e. fossil) to speak
>>> the other protocol directly in order to get any guarantees.
>>> (If 9P doesn't cut it then anything that's just in front of
>>> (not in place of) a 9P server can't solve the problem.)
>>>
>>> Russ
>>>
>>
>>