Re: [9fans] replica (was: ipv6)

Re: [9fans] replica (was: ipv6)

[Russ Cox]

> Your message arrived here while I had replica/pull running.
> I don't know whether this already came up in the replica
> discussion so far... (or can be R in TFM):
> if you push while I'm pulling, do I still get
> a 'consistent' set of files?

Yes, unless you get extraordinarily unlucky.

Pushing consists of two operations: copying
the file to sources and updating
/n/sources/plan9/dist/replica/plan9.log
to reflect the change.  Replica/pull consults
the log, sees that a new file is available,
and copies the new file.

First, the log:

The log is an append-only file that you copy
to your local machine via tail.  If more data arrives
while you're copying, you'll just get the new
data.  No harm done.  Updatedb uses buffered I/O
to write the log, so it is possible that on a really
big update (more than 8kb worth of log), you'd
get a log ending in a half-entry.  Replica/applylog
(the guts of replica/pull) would then complain,
and if you ran replica/pull again, the rest of the
log would come down and you'd be fine.

When the log gets far too large, I compress it,
removing the old one and writing an equivalent
smaller one.  If this happens at exactly the right
moment, pull (a lowly shell script) could get
confused, trying to grab the tail of a file that
has changed underfoot.  The change always makes
the log smaller, though, so if you've updated
at all since installing, tail and thus pull will
fail.  Running replica/pull again will then work.

It is possible (though extraordinarily unlikely)
that if you install Plan 9 and then do an update
just as I replace the log with a compressed version,
tail will succeed but leave you with your original
log and then some arbitrary suffix of the new log.
At this point, the log is probably corrupt,
so replica/pull will complain.  If the log isn't
corrupt but you downloaded less than a kilobyte
for the suffix, then replica/pull will download
the whole log afresh the next time.
If, by chance, you get a well-formed log AND the
new suffix you downloaded is more than a kilobyte,
then you'll never notice the screwup.  This is
exceedingly unlikely even if I were compressing
the log often.  I intend to compress the log
only when it grows to about 1.6MB, twice its normal
size.  If we sustain our current rate of change,
I'll have to compress the log every two years.
Oh, and you had to install when the log was at its
minimal size.  Installing with a recent packaging
would have gotten you a bigger log and the tail
would have failed.  I'm just not worried.

Second, copying:

If a file changes before being mentioned in the
log (but is also mentioned earlier in the log)
you might copy the new file expecting to get an older
version of it.  That's okay, replica/pull will notice
the unexpectedly new time stamp and record that it
has the new file.  When (later) it sees the
entry in the log for the new version, it won't
bother copying the file a second time.

When replica/pull downloads a file, it spools
it to /tmp and then once it has the whole file
copies it from /tmp to its destination.  If the
copy into /tmp fails (say, because of a network
hangup), then an error is printed and the /tmp
copy is tossed.  Pull stats the file before and
after the copy into /tmp.  It repeats the copy
until the qid stops changing.

In summary:

Downloading the new log while it changes won't
leave you inconsistent unless you install Plan 9
right after I compress the log (an annual or biannual
event), and then wait to run replica/pull until the
exact instant I'm about to compress the log again.

Downloading files while they change won't leave
you in an inconsistent state.  Ever.

Russ

Re: [9fans] replica (was: ipv6)

[Micah Stetson]

> > How do I know if I have to schedule a new pull because I got a
> > 'partial' update?
>
> You don't, really.  That would require predicting the future.
> At the time you disconnected, there was no way to know
> that, say, foo.c was going to get changed two minutes later.

What if replica/pull complained about unexpected time
stamps?  That would at least tell the user that something
might be awry.

Micah

Re: [9fans] replica (was: ipv6)

[rsc]

> So I can get incompatible files in 'partial' updates:
> - current log tells me I need foo.h and foo.c
> - pull gets foo.h
> - someone updates foo.h and foo.c (but wont enter log until next
> cron run)
> - pull then gets (newer) foo.c

Damn!  I failed to confuse you.  ☺

Yes, you're absolutely right.

> I've now got a wrong foo.h/foo.c pair and they wont compile.
>
> My foo.h is correct as per the log info
> Pull can spot that foo.c is more up to date but without
> dependency info cannot determine that it needs to re-fetch foo.h

Yes.

> This is resolved by re-running pull after the next log update.
> This raises two questions:
> How do I know when the log is up to date (mk scan has run)?

You have to trust us to run mk scan after we make changes.
If, after pulling, you run

	date `{tail -1 /dist/replica/client/plan9.log | awk '{print $1}'}

and that was more than half an hour ago, then you
have everything we've put out on sources.  We use a
script that pushes the changes over en masse, so we're
talking maybe a second between updates, not large
amounts of time.

> How do I know if I have to schedule a new pull because I got a
> 'partial' update?

You don't, really.  That would require predicting the future.
At the time you disconnected, there was no way to know
that, say, foo.c was going to get changed two minutes later.

Still, if you just did a pull and didn't get any changes less
than half an hour old (run the date command above for
the time of the last change), you've probably got a decent set,
even assuming we forgot to run mk scan manually.

We usually mention the big change sets on 9fans;
that's a reasonable way to know when there are big
things waiting to be picked up.

If you find something that you think is wrong,
you can always ask on 9fans.  Sometimes we just
miss files, though we've got tools that try to keep
that from happening.  Let us know.

Russ

Re: [9fans] replica (was: ipv6)

[Chris Hollis-Locke]

So I can get incompatible files in 'partial' updates:
- current log tells me I need foo.h and foo.c
- pull gets foo.h
- someone updates foo.h and foo.c (but wont enter log until next
cron run)
- pull then gets (newer) foo.c

I've now got a wrong foo.h/foo.c pair and they wont compile.

My foo.h is correct as per the log info
Pull can spot that foo.c is more up to date but without
dependency info cannot determine that it needs to re-fetch foo.h

This is resolved by re-running pull after the next log update.
This raises two questions:
How do I know when the log is up to date (mk scan has run)?
How do I know if I have to schedule a new pull because I got a
'partial' update?

Re: [9fans] replica (was: ipv6)

[rsc]

> The file is consistent but is the directory? I.e. if I was copying over a whole bunch of files,
> could he not have gotten in when I was half way through?

Pull doesn't walk the entire file system looking
for changes.  Instead, it relies on changes being
listed in a change log /n/sources/dist/replica/plan9.log.

If I run a pull (so I'm up-to-date) and then you
change files on sources, future pulls won't notice
any of them until they're added to the change log.

The change log gets updated by running mk scan
in /sys/lib/dist.  This happens half-hourly via cron.
If the cron job happened while the copy was in
progress, then yes, until the next scan you'd only
get half the new files.  If the cron job didn't exist,
so that mk scan couldn't happen while you were
halfway through updating things, then there'd be
no way for me to get just some of the changes if I
were up-to-date except for those changes.

I guess my point in all this was that you cannot
get into a state where the client machine has the
wrong idea of what it has and doesn't have.

For example, a priori you might worry that if I copy
down a file as it is changing, I'll get a weird merge
of the two files that will persist until the file changes
again.  Or I'd get the old file but think I had the new
one, so that I wouldn't download the new one the
next time around.  None of those things can happen.

Russ

Re: [9fans] replica (was: ipv6)

[presotto]

[-- Attachment #1: Type: text/plain, Size: 153 bytes --]

The file is consistent but is the directory? I.e. if I was copying over a whole bunch of files,
could he not have gotten in when I was half way through?

[-- Attachment #2: Type: message/rfc822, Size: 1967 bytes --]

From: rsc@plan9.bell-labs.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] replica (was: ipv6)
Date: Tue, 28 May 2002 12:55:16 -0400
Message-ID: <38e0f7c1cf60157c1e74879ac4909e6c@plan9.bell-labs.com>

> Could there be a problem if pull has already opened the file
> and is copying it at the time an update is written to it.
> pull would have got the metadata of the file pre update.
> I'm assuming your local updates are faster than remote pulls
> so local update writes can 'overtake' the reads of a remote pull.
> Unlikely I know, but possible?

Nope.  Pull stats the file before and after copying.
If a local write happened in the middle, the qid
returned by the second stat will be different, so
pull will copy it again.  This repeats until the qid
stops changing.

Russ

Re: [9fans] replica (was: ipv6)

[rsc]

> Could there be a problem if pull has already opened the file
> and is copying it at the time an update is written to it.
> pull would have got the metadata of the file pre update.
> I'm assuming your local updates are faster than remote pulls
> so local update writes can 'overtake' the reads of a remote pull.
> Unlikely I know, but possible?

Nope.  Pull stats the file before and after copying.
If a local write happened in the middle, the qid
returned by the second stat will be different, so
pull will copy it again.  This repeats until the qid
stops changing.

Russ

Re: [9fans] replica (was: ipv6)

[forsyth]

[-- Attachment #1: Type: text/plain, Size: 145 bytes --]

inconsistencies arise at a different level too:
i noticed that the lookupcmd code appeared throughout ip(v4)
only to vanish again just now!


[-- Attachment #2: Type: message/rfc822, Size: 4537 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 26 bytes --]

I'm pretty sure you don't.

[-- Attachment #2.1.2: Type: message/rfc822, Size: 2898 bytes --]

From: Axel Belinfante <Axel.Belinfante@cs.utwente.nl>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] replica (was: ipv6)
Date: Tue, 28 May 2002 15:26:22 +0200
Message-ID: <200205281326.g4SDQMx27511@zamenhof.cs.utwente.nl>

> I moved our ipv4+6 stack onto sources.cs.bell-labs.com.

Your message arrived here while I had replica/pull running.
I don't know whether this already came up in the replica
discussion so far... (or can be R in TFM):
if you push while I'm pulling, do I still get
a 'consistent' set of files?

Axel.

Re: [9fans] replica (was: ipv6)

[postmaster]

Could there be a problem if pull has already opened the file
and is copying it at the time an update is written to it.
pull would have got the metadata of the file pre update.
I'm assuming your local updates are faster than remote pulls
so local update writes can 'overtake' the reads of a remote pull.

Unlikely I know, but possible?

Chris.

Re: [9fans] replica (was: ipv6)

[Russ Cox]

That whole message was about whether the replica
metadata and the file system remain consistent.

If you haven't updated for a long time and update
after new files have been written but before the log
is updated, you might get a new file that requires
other new or missing files to build correctly.
In that sense, you could end up with an inconsistent
set of files.  But pulling again once the log has
been updated will fix that.

Russ

Re: [9fans] replica (was: ipv6)

[presotto]

[-- Attachment #1: Type: text/plain, Size: 26 bytes --]

I'm pretty sure you don't.

[-- Attachment #2: Type: message/rfc822, Size: 2898 bytes --]

From: Axel Belinfante <Axel.Belinfante@cs.utwente.nl>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] replica (was: ipv6)
Date: Tue, 28 May 2002 15:26:22 +0200
Message-ID: <200205281326.g4SDQMx27511@zamenhof.cs.utwente.nl>

> I moved our ipv4+6 stack onto sources.cs.bell-labs.com.

Your message arrived here while I had replica/pull running.
I don't know whether this already came up in the replica
discussion so far... (or can be R in TFM):
if you push while I'm pulling, do I still get
a 'consistent' set of files?

Axel.

Re: [9fans] replica (was: ipv6)

[Axel Belinfante]

> I moved our ipv4+6 stack onto sources.cs.bell-labs.com.

Your message arrived here while I had replica/pull running.
I don't know whether this already came up in the replica
discussion so far... (or can be R in TFM):
if you push while I'm pulling, do I still get
a 'consistent' set of files?

Axel.