[9fans] Replica - just a thought.

[9fans] Replica - just a thought.

[Lucio De Re]

This is probably boring _and_ ignorant, but maybe just once I've
hit on a good idea.  You be the judge.

I've been downloading a fresh Plan 9 image weekly, religiously
adding it to a 9660 "dump" image that has now grown to a little
less than 500 megabytes.  When I get an opportunity, I burn the
dump image onto a rewritable CD and so I have a little bit of
development history that may come in handy one day.

In the meantime, I've been wondering how hard it would be to unwind
the CD dump for various uses, but I always refrained from inspecting
the actual dump9660 sources as probably much too complex for my
limited understanding.

The recent oddities with replica, whatever their nature, as well
as a personal need for replication that does not seem to coincide
exactly with replica's capabilities has made me think about a
convergence of replica and dump9660 that might just be preferable
to the present situation.

As I see it, replica effectively records externally what dump9660
records implicitly.  Again, without having studied either of these
I may be barking up the wrong tree, but those in the know may be
able to correct me.  If we imagine dump9660 as two parts communicating
via some channel (in a figurative sense), the first comparing the
fresh filesystem against an archive copy and the second actually
recording the differences by allocating new disk space to new copies
of the affected files, do we not come pretty close to replica's
operation at least in principle?

The next step would then be to define the protocol between these
two portions of dump9660 formally and arrange for it to be exchanged
between replica masters and slaves.  It may need enhancing to cater
for changes to the metadata (and then it may not), but largely
there would no longer be two or more replication mechanisms, one
would be sufficient and would be merely a formalisation of a feature
(the dump) that Plan 9 has had for many years.  Whether we can have
the granularity of the FS dump, I'm not sure, but file level
granularity may even be preferable.

Am I missing something important and therefore spouting nonsense,
or have I struck on an idea that hasn't yet been obvious to anyone
else?  Please let me know.  I don't mind trying to implement this
myself (and more USB devices and SCSI LUs and one or two new SCSI
controller drivers and S/MIME and, and, and - life's too short by
far) if it isn't flawed in some fatal way.

++L

PS: To put it another way, from the CD as described above, I can
update my system by selecting the image I want, the most recent
being the norm.  But it uses the replica database information to
perform the update and I believe that, possibly with some adjustments
to the format of the metadata, the information required is already
in the structure of the dump and ought to be used instead.  The
reason I can't do that is that there is no formal specification
that would then perhaps be more complete than is the case presently.
Being able to derive it instead of recording it separately should
provide a degree of greater reliability.  And one could then make
an external copy at any stage as well.

Re: [9fans] Replica - just a thought.

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

perhaps I didn't understand you, but if you dump
a replica client, as I think you did, then you'd need
to sync to the original sources to detect conflicts.

AFAIK, the replica scheme works as long as each
client gets in sync with the same master.

And anyway, if a corrupt entry arises, you'd get it
anyway (be it dump or replica), so there's no protection
but to be able to recover if something goes wrong.

But maybe I'm missing your point.

[-- Attachment #2: Type: message/rfc822, Size: 5276 bytes --]

From: Lucio De Re <lucio@proxima.alt.za>
To: 9fans mailing list <9fans@cse.psu.edu>
Subject: [9fans] Replica - just a thought.
Date: Tue, 10 Feb 2004 12:47:25 +0200
Message-ID: <20040210124725.K17981@cackle.proxima.alt.za>

This is probably boring _and_ ignorant, but maybe just once I've
hit on a good idea.  You be the judge.

I've been downloading a fresh Plan 9 image weekly, religiously
adding it to a 9660 "dump" image that has now grown to a little
less than 500 megabytes.  When I get an opportunity, I burn the
dump image onto a rewritable CD and so I have a little bit of
development history that may come in handy one day.

In the meantime, I've been wondering how hard it would be to unwind
the CD dump for various uses, but I always refrained from inspecting
the actual dump9660 sources as probably much too complex for my
limited understanding.

The recent oddities with replica, whatever their nature, as well
as a personal need for replication that does not seem to coincide
exactly with replica's capabilities has made me think about a
convergence of replica and dump9660 that might just be preferable
to the present situation.

As I see it, replica effectively records externally what dump9660
records implicitly.  Again, without having studied either of these
I may be barking up the wrong tree, but those in the know may be
able to correct me.  If we imagine dump9660 as two parts communicating
via some channel (in a figurative sense), the first comparing the
fresh filesystem against an archive copy and the second actually
recording the differences by allocating new disk space to new copies
of the affected files, do we not come pretty close to replica's
operation at least in principle?

The next step would then be to define the protocol between these
two portions of dump9660 formally and arrange for it to be exchanged
between replica masters and slaves.  It may need enhancing to cater
for changes to the metadata (and then it may not), but largely
there would no longer be two or more replication mechanisms, one
would be sufficient and would be merely a formalisation of a feature
(the dump) that Plan 9 has had for many years.  Whether we can have
the granularity of the FS dump, I'm not sure, but file level
granularity may even be preferable.

Am I missing something important and therefore spouting nonsense,
or have I struck on an idea that hasn't yet been obvious to anyone
else?  Please let me know.  I don't mind trying to implement this
myself (and more USB devices and SCSI LUs and one or two new SCSI
controller drivers and S/MIME and, and, and - life's too short by
far) if it isn't flawed in some fatal way.

++L

PS: To put it another way, from the CD as described above, I can
update my system by selecting the image I want, the most recent
being the norm.  But it uses the replica database information to
perform the update and I believe that, possibly with some adjustments
to the format of the metadata, the information required is already
in the structure of the dump and ought to be used instead.  The
reason I can't do that is that there is no formal specification
that would then perhaps be more complete than is the case presently.
Being able to derive it instead of recording it separately should
provide a degree of greater reliability.  And one could then make
an external copy at any stage as well.

Re: [9fans] Replica - just a thought.

[Lucio De Re]

On Tue, Feb 10, 2004 at 12:14:31PM +0100, Fco.J.Ballesteros wrote:
>
> perhaps I didn't understand you, but if you dump
> a replica client, as I think you did, then you'd need
> to sync to the original sources to detect conflicts.
>
Well, no, although what you say is right.  (a) I actually mounted one
ISO image and merged it into the dump, struck me as obvious; (b) It is
hard to keep two sites in sync without an arbitration, and in my case
I want to keep a pristine copy of the Plan 9 distribution handy and so
does Bell Labs.

> AFAIK, the replica scheme works as long as each
> client gets in sync with the same master.
>
Yes, that's the arbitration point.  It eliminates the need to define
masters and slaves, too.

> And anyway, if a corrupt entry arises, you'd get it
> anyway (be it dump or replica), so there's no protection
> but to be able to recover if something goes wrong.
>
The recent spate of problems seem to me to be finger trouble, like
compiling kernels on the distribution server.  Replica lacks
checksumming and that is a serious flaw, but it's not a problem my
approach addresses.  To me the benefit lies in the ditraibution
format, file-level dump instructions, if such things can be
represented.

> But maybe I'm missing your point.

No, I'm the one with the half-baked idea.  But I think that
discussion will lead to a ripe fruit :-)

++L

Re: [9fans] Replica - just a thought.

[Fco.J.Ballesteros]

> To me the benefit lies in the ditraibution
> format, file-level dump instructions, if such things can be
> represented.

But I love to "9fs sources" and then "diff" or
whatever wrt the exact thing I'm pulling from.

Although agree that for some with connection problems
or w/o a Plan 9 already working, it's handly to have a CD
at hand. Wait!, I think even in the last case I'd prefer to
boot as a term from a kind soul's machine :-)

Re: [9fans] Replica - just a thought.

[Lucio De Re]

On Tue, Feb 10, 2004 at 01:00:32PM +0100, Fco.J.Ballesteros wrote:
>
> > To me the benefit lies in the ditraibution
> > format, file-level dump instructions, if such things can be
> > represented.
>
> But I love to "9fs sources" and then "diff" or
> whatever wrt the exact thing I'm pulling from.
>
My problem, which my solution may not address at all, is that I
don't have enough faith in the replica process.  When it reports
"locally modified" during a pull to a pristine previous distribution
I feel that much of its purpose is being defeated.

I also greatly appreciated sources and specially sourcesdump.  What
I'm suggesting is that we replace replica with the equivalent of
a remote dump.  As you point out, there is a need for a "release"
base, the clever approach is to be able to identify the base point
instead of dictating it, assuming such a thing is feasible.  It
is, by default, in the "dump".

> Although agree that for some with connection problems
> or w/o a Plan 9 already working, it's handly to have a CD
> at hand. Wait!, I think even in the last case I'd prefer to
> boot as a term from a kind soul's machine :-)

That's Plan 9 at its most powerful and is quite irresistible.  But
I believe Plan 9 can deliver reliable replication without imitating
Microsoft and in my opinion that fits well with the dump/Venti
model for extending distributed processing even beyond the boundaries
of decent bandwidth, like here in Africa.

To put all my card on the table, I'm looking to replicate my mailbox
because my network is at the far end of a low-reliability link and
when it fails there is no one to restart it, yet that's where my
Fossil/Venti server resides.  Replication is preferable to mapping
the two endpoints to the same, frequently inaccessible file service.

++L

Re: [9fans] Replica - just a thought.

[Prem Mallappa]

Is there any way that i can get this CD ( download????)

plz point to me
thx 
regards
prem

On Tue, 10 Feb 2004 10:48:32 +0000, Lucio De Re wrote:

> This is probably boring _and_ ignorant, but maybe just once I've
> hit on a good idea.  You be the judge.
> 
> I've been downloading a fresh Plan 9 image weekly, religiously
> adding it to a 9660 "dump" image that has now grown to a little
> less than 500 megabytes.  When I get an opportunity, I burn the
> dump image onto a rewritable CD and so I have a little bit of
> development history that may come in handy one day.

Re: [9fans] Replica - just a thought.

[andrey mirtchovski]

> Is there any way that i can get this CD ( download????)
>
> plz point to me
> thx
> regards
> prem
>

google for "Plan 9 from Bell Labs"?  failing that you may want to go to
http://plan9.bell-labs.com/plan9dist

Re: [9fans] Replica - just a thought.

[Lucio De Re]

On Thu, Feb 19, 2004 at 10:27:52AM +0000, Prem Mallappa wrote:
>
> Is there any way that i can get this CD ( download????)
>
If you're after the "dump9660" image, no, I don't think it's even
remotely practical to serve such a large image as web-retrievable.  I
think it shrinks down to well over 100 Meg with bzip2, but even that
is too large to distribute.

Sorry about that.

++L
>
> On Tue, 10 Feb 2004 10:48:32 +0000, Lucio De Re wrote:
>
> > This is probably boring _and_ ignorant, but maybe just once I've
> > hit on a good idea.  You be the judge.
> >
> > I've been downloading a fresh Plan 9 image weekly, religiously
> > adding it to a 9660 "dump" image that has now grown to a little
> > less than 500 megabytes.  When I get an opportunity, I burn the
> > dump image onto a rewritable CD and so I have a little bit of
> > development history that may come in handy one day.