[9fans] silly replica question (repeated m msgs won't go away)

[9fans] silly replica question (repeated m msgs won't go away)

[Axel Belinfante]

Today I updated my fake-worm fs from sources (first time since summer).

every time I replica/pull, I get lots of msgs like:

m sys/lib/antiword/8859-1.txt 644 sys sys 1069793886
m sys/lib/antiword/8859-10.txt 644 sys sys 1069793886
m sys/lib/antiword/8859-13.txt 644 sys sys 1069793886
m sys/lib/antiword/8859-14.txt 644 sys sys 1069793886
m sys/lib/antiword/8859-15.txt 644 sys sys 1069793886

and whenever I now run replica/pull again, I get the same...

adding -s flag to replica/pull did not help
(fair enough, it serves a different purpose)

What am I doing wrong, or overlooking?
I _think_ have the fake-worm fs permissions open enough, or???

Thanks,
Axel.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[mirtchov]

> Today I updated my fake-worm fs from sources (first time since summer).
>
> every time I replica/pull, I get lots of msgs like:
>
> m sys/lib/antiword/8859-1.txt 644 sys sys 1069793886
> m sys/lib/antiword/8859-10.txt 644 sys sys 1069793886
> m sys/lib/antiword/8859-13.txt 644 sys sys 1069793886
> m sys/lib/antiword/8859-14.txt 644 sys sys 1069793886
> m sys/lib/antiword/8859-15.txt 644 sys sys 1069793886
>
> and whenever I now run replica/pull again, I get the same...
>
> adding -s flag to replica/pull did not help
> (fair enough, it serves a different purpose)
>
> What am I doing wrong, or overlooking?
> I _think_ have the fake-worm fs permissions open enough, or???
>
> Thanks,
> Axel.

there were broken /sys/lib/antiword and /sys/src/cmd/aux/antiword with
bad permissions on sources.  looks like you're stuck with a bad copy.
try chmod-ing your own version to something world-readable.

or deleting them altogether and trying the -s flag to pull..

andrey

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Adrian Tritschler]

mirtchov@cpsc.ucalgary.ca wrote (sometime last December):

>>Today I updated my fake-worm fs from sources (first time since summer).
>>
>>every time I replica/pull, I get lots of msgs like:
>>
>>m sys/lib/antiword/8859-1.txt 644 sys sys 1069793886
>>m sys/lib/antiword/8859-10.txt 644 sys sys 1069793886
>>m sys/lib/antiword/8859-13.txt 644 sys sys 1069793886
>>m sys/lib/antiword/8859-14.txt 644 sys sys 1069793886
>>m sys/lib/antiword/8859-15.txt 644 sys sys 1069793886
>>
>>and whenever I now run replica/pull again, I get the same...
>>
>>adding -s flag to replica/pull did not help
>>(fair enough, it serves a different purpose)
>>
>>What am I doing wrong, or overlooking?
>>I _think_ have the fake-worm fs permissions open enough, or???
>>
>>Thanks,
>>Axel.

> there were broken /sys/lib/antiword and /sys/src/cmd/aux/antiword with
> bad permissions on sources.  looks like you're stuck with a bad copy.
> try chmod-ing your own version to something world-readable.
> 
> or deleting them altogether and trying the -s flag to pull..
> 
> andrey

I'm suffering the same problem, every pull that I do from sources is 
producing two lines for each file in sys/lib/antiword and 
sys/src/cmd/aux/antiword.  I've tried removing all the files and redoing 
the pull as "pull -s sys/src/cmd/aux", but that then failed because 
/n/kfs/sys/src/cmd/aux/antiword didn't exist.  I created the directory, 
then the next time I did a pull I got
  :
a sys/src/cmd/aux/antiword/xml.c 644 sys sys 1069795877
  :
m sys/src/cmd/aux/antiword/xml.c 664 sys sys 1069795877

Then on every subsequent pull I get the two:
  :
m sys/src/cmd/aux/antiword/xml.c 644 sys sys 1069795877
  :
m sys/src/cmd/aux/antiword/xml.c 644 sys sys 1069795877

Is there any way to convince replica that I really do have all the files 
on my machine, and for it to stop flipping the permissions back and 
forth between 644 and 664?

RTFM accepted, but can some kind sole point me to the right part of the FM?

	Adrian

---------------------------------------------------------------
Adrian Tritschler                          mailto:ajft@ajft.org
Latitude 38°S, Longitude 145°E, Altitude 50m,      Shoe size 44
---------------------------------------------------------------

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Geoff Collyer]

> RTFM accepted, but can some kind sole point me to the right part of the FM?

I believe that would be the BUGS section.  I've been suffering with
this too and have resorted to grepping out the antiword messages.
replica can flip its bits to its heart's content.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Lucio De Re]

On Mon, Feb 09, 2004 at 09:07:38PM -0800, Geoff Collyer wrote:
>
> > RTFM accepted, but can some kind sole point me to the right part of the FM?
>
> I believe that would be the BUGS section.  I've been suffering with
> this too and have resorted to grepping out the antiword messages.
> replica can flip its bits to its heart's content.

I've edited various instances of the .log and .db files (master or
slave) and I have no idea which one did the trick, but the messages
have ceased.

++L

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Charles Forsyth]

i think the replica scheme is actually quite pretty,
but there's a flaw that happens to afflict 'm' particularly.

the easiest way to fix this without code change is to use
-c and -s as desired (carefully!)to resolve every other inconsistency,
so that plan9.time will finally be updated, and it won't process the 'm' entries again.
better still, your subsequent updates will be quicker too because it won't replay so much
of the log!  perhaps just do one replica/pull -v ... to get a list of differences to check later,
and fetch anything new, then do replica/pull -vc ... to retain your current client state.
a subsequent replica/pull -v ... shouldn't show the dreaded 'm' entries.

it might be helpful to look at one case to see what went wrong.

the file on sources is
--rw-rw-r-- M 119 rsc sys 10583 Nov 25 20:58 /n/sources/plan9/sys/lib/antiword/8859-15.txt

the m messages are repeated because the log contains several batches of records like the following:

1069794016 8 a sys/lib/antiword/8859-15.txt - 640 sys sys 1069793886 10583
1069799416 4 m sys/lib/antiword/8859-15.txt - 644 sys sys 1069793886 10583
1069801217 4 m sys/lib/antiword/8859-15.txt - 664 sys sys 1069793886 10583

(with other records interspersed).  note that the mtime is the same in each record
(correctly, as it happens).

the 'a' record has the test:
			if(dbd.mtime >= rd.mtime)	/* already created this file; ignore */
				break;
which prevents it from going any further.

the 'm' record has the test:
			if(!(dbd.mode&DMDIR) && dbd.mtime > rd.mtime)		/* have newer version; ignore */
				break;
			if((dbd.mode&DMDIR) && dbd.mtime > now)
				break;
where `now' is the date of the log entry (its first field), not the current time.
the test for `newer version' is > not >= so each 'm' is considered further.

one of the tests is whether the local file mode matches the remote mode,
where the `remote mode' is the mode in the log entry not the state of the current remote file.
they don't match, so the local file's mode is changed by the first 'm', then changed again
by the second 'm', and since it has done something you get a comment each time,
and an update in the database.

the file's mtime is the same in each log entry, because chmod correctly affects the parent directory,
not the file itself.  the -c or -s options have no effect because although they correctly update
the metadata (.db) to set the appropriate state, all the 'm' entries have the same
mtime and thus the newer-version test will continue to be false and they'll
rerun each time.  the trouble is that the .db has got only the mtime to resolve
`newer action', not the log sequence number for that action.

is there a simple change to the code? it can't simply set the mtime in .db
to the log entry time because it needs the correct last-modified time
for the file contents to be in .db to test later 'a'/'c'/'d' requests.  it isn't correct
to use >= in the mtime comparison because that would prevent 'm' from having much effect generally.
it would help to know which log entry it has seen.
there might be another way short of storing the log sequence in .db,
but i couldn't see it in the time i was prepared to spend on it.
i could see several partial/incorrect changes (eg, stat the actual remote file).

i looked at replica's approach for possible use in Inferno, because it
was compact and seemed simpler (in code size) compared to some
alternatives.  as it happens, the replica problem has its subtle
points (even for a problem simpler than tra's).  i quickly resorted to
a case analysis on my own, and wrote Limbo code based on that.
i noticed at least one other difference in
my analysis and applylog's; i must check my notes when i remember
where i filed them.  i ended up deciding to process the log files differently,
more precisely as with a (class of) log-structured file system (i was
doing a lot with them at the time).  there is a particular invariant
for the contents of the client's .log.  i could possibly have used a
modified .db structure (including log sequence numbers), but in my
scheme that's just an optimisation (and there was another way to do that).
of course, something else might be wrong ...  we'll see.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Lucio De Re]

On Tue, Feb 10, 2004 at 12:22:28PM +0000, Charles Forsyth wrote:
>
> i think the replica scheme is actually quite pretty,
> but there's a flaw that happens to afflict 'm' particularly.
>
Replica is extremely useful, specially in the role it serves to
most of this community.  Its minor flaws are sufficiently trivial
to be ignored and workarounds applied as the occasion demands.

But replication deserves being done "right" rather than heuristically
and Plan 9 seems to have all the ingredients to provide the final
word in replication.  Further, as I mentioned in another thread,
replication enhances the "distributability" and Plan 9 can only
benefit from such an ugly word.

If I'm right that the dump (in its ISO-9660 guise) and replica can
converge and provide the ideal tool, then I'm willing to make myself
unpopular and challenge the present replica wisdom.  With all due
respect, of course, to those who first trod this path.

If I'm being stupid, then I won't mind acknowledging it once my nose
has been rubbed in it.

++L

Re: [9fans] silly replica question (repeated m msgs won't go away)

[David Presotto]

On Tue Feb 10 07:39:37 EST 2004, lucio@proxima.alt.za wrote:
> But replication deserves being done "right" rather than heuristically

Describe right a little more.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Fco.J.Ballesteros]

>> But replication deserves being done "right" rather than heuristically
>
> Describe right a little more.

tra


PS: Couldn't resist :-)

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Lucio De Re]

On Tue, Feb 10, 2004 at 10:00:21AM -0500, David Presotto wrote:
>
> On Tue Feb 10 07:39:37 EST 2004, lucio@proxima.alt.za wrote:
> > But replication deserves being done "right" rather than heuristically
>
> Describe right a little more.

I do really mean that in the grey area between exact and heuristically
adequate, I'd like to be closer to exact.  My opinion is that
replica has weaknesses that diminish the "faith" factor whereas
"dump" certainly doesn't or doesn't seem to.

What might be a different approach would be to provide a measurement
of the uncertainty in replica, but somehow I can't help believing
that there is a better paradigm implicit in dump9660 that can be
exploited instead.

Does the above help?  "Right" as in as accurate as produced by the
dump9660 procedure just ahead of generating the new release.  For
all I know this is already the case and it's the baseline that
triggers replica errors, but that needs proof.

Sorry if I seem to be pontificating, as I mentioned it's a half
formed idea and I'm looking to give it a little more solidity.  If
it's preferable to continue this in private mail, please don't
hesitate to point it out.

++L

Re: [9fans] silly replica question (repeated m msgs won't go away)

[boyd, rounin]

> Describe right a little more.

wobber/prusker siphon.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Charles Forsyth]

i don't think i'd describe what replica attempts as `heuristic'.
there's a little graph of possibilities and it either chooses the safe
path or requests guidance.

an SHA1 hash would be helpful; by using mtime+length it can moan about
insignificant changes but then again, with some things it's possible
the mtime matters, and perhaps it should moan.

having mentioned SHA1: there might be an interesting variant that uses
venti scores.

Re: [9fans] silly replica question (repeated m msgs won't go away)

[Lucio De Re]

On Tue, Feb 10, 2004 at 04:10:20PM +0000, Charles Forsyth wrote:
>
> i don't think i'd describe what replica attempts as `heuristic'.
> there's a little graph of possibilities and it either chooses the safe
> path or requests guidance.
>
I really hope I don't sound confrontational, there's much I don't
know and can only guess at, but there's definitely more than a
trivial amount of slipperiness about replica that makes it unsuitable
for the type of replication that Microsoft used (or used to use)
to propagate changes to the users database between the PDC and BDCs
(if you don't know the acronym, you're missing nothing) or what
Lotus Notes uses (this is NetNews in disguise or is it on steroids?)
or, for that matter, the OpenLDAP replicator.

> an SHA1 hash would be helpful; by using mtime+length it can moan about
> insignificant changes but then again, with some things it's possible
> the mtime matters, and perhaps it should moan.
>
I think there's a philosophy behind replica that suggests humans
are involved therefore there's scope for some fuzzy logic.  I'm
not even arguing the merits thereof, it's a perfectly good approach.
What I prefer, however, is the solidity of the dump and a tool to
extract the updates list in a format that makes it possible to
revert them and/or apply them elsewhere.

I'd better be careful, however, I'm beginning to recognise signs
that my attraction for CVS is coming through.

> having mentioned SHA1: there might be an interesting variant that uses
> venti scores.

I won't try to follow your thinking there, it's way beyond my
processing powers :-)

++L