[9fans] some thoughts on replica and irreproducible results

[9fans] some thoughts on replica and irreproducible results

[Mike Haertel]

So, I just ran replica again to get this morning's batch of changes.
Here's what I saw:

term% replica/pull -v /dist/replica/network
c 386/bin/dd
c sys/src/cmd/dd.c
c sys/src/cmd/acme/fsys.c
c sys/src/cmd/upas/fs/fs.c
c sys/src/9/pc/sd53c8xx.c
c sys/src/9/pc/sd53c8xx.n

The issue: some of the maintainers are updating the corresponding
binaries to updated sources, while others are not.

Unfortunately, if I go and update these binaries myself, and then
later someone else updates them on sources, replica assumes they are
locally modified and stops tracking the version on sources.  I can
override this with replica/pull -s, but it's not safe to be doing that
by default and it's potentially error prone to go and do it manually
if I have genuine local changes that I've since forgotten about.  (Ok,
admittedly the #1 solution to the latter is to keep all my genuine
local changes in a nonstandard location...)

Now, for an experiment and then a suggestion:

term% mkdir /tmp/cmd
term% bind -bc /tmp/cmd /sys/src/cmd
term% cd /sys/src/cmd
term% mk 8.dd
8^c -FVw dd.c
8^l  -o 8.dd dd.8
term% cmp 8.dd /386/bin/dd
term%

Cool, no differences!  So nobody is doing anything dumb like putting
gratuitous timestamps in binaries.  If I build the same binary twice
I get the same bytes.  This is a feature.

Therefore one simple improvement to replica might be: if a file
appears to be locally modified, but is actually byte-for-byte
identical to the corresponding file on sources, then replica should
"update" the file after all (including changing the mtime to match the
file on sources).

However the problem is wider than it first appears.  In the course of
investigating this problem I also discovered (picking "ls" at random):

term% mk 8.ls
8^c -FVw ls.c
8^l  -o 8.ls ls.8
term% cmp 8.ls /386/bin/ls
8.ls /386/bin/ls differ: char 20

This happened on the very first command other than "dd" that I tried
to build, and it suggests that quite a few binaries may be out of date
with the sources.

So perhaps it might also be worthwhile to have a cron job on sources
that would periodically rebuild all the binaries and compare with
the installed versions.  However, some obnoxious programs like "ar"
keep timestamps in binaries that are gratuitiously updated even when
the object file contents themselves haven't changed.  Furthermore
The lack of shared libraries would also cause a lot of binary turnover
anytime somebody updated a popular routine in the C library.  Designing
an automatic build that wouldn't produce a huge volume of updates
would be hard.

Re: [9fans] some thoughts on replica and irreproducible results

[rsc]

> term% replica/pull -v /dist/replica/network
> c 386/bin/dd
> c sys/src/cmd/dd.c
> c sys/src/cmd/acme/fsys.c
> c sys/src/cmd/upas/fs/fs.c
> c sys/src/9/pc/sd53c8xx.c
> c sys/src/9/pc/sd53c8xx.n
>
> The issue: some of the maintainers are updating the corresponding
> binaries to updated sources, while others are not.

We're still working out the kinks in the update system.

The only real bug here is the lack of 386/bin/upas/fs,
which you'll find is now there.  Acme didn't really change:
the file just got rewritten.  As for sd53c8xx, it hardly seemed
worth making people download new kernels to get a fix to
a problem that no one will run into before the next time
kernels really have to be updated.

> Unfortunately, if I go and update these binaries myself, and then
> later someone else updates them on sources, replica assumes they are
> locally modified and stops tracking the version on sources.  I can
> override this with replica/pull -s, but it's not safe to be doing that
> by default and it's potentially error prone to go and do it manually
> if I have genuine local changes that I've since forgotten about.  (Ok,
> admittedly the #1 solution to the latter is to keep all my genuine
> local changes in a nonstandard location...)

Exactly.  It seems safe to me to use

	replica/pull -sv /dist/replica/network 386/bin 386/lib
	replica/pull -v /dist/replica/network

If you smash a locally changed binary, you can always rebuild it.

> Now, for an experiment and then a suggestion:
>
> term% mkdir /tmp/cmd
> term% bind -bc /tmp/cmd /sys/src/cmd
> term% cd /sys/src/cmd
> term% mk 8.dd
> 8^c -FVw dd.c
> 8^l  -o 8.dd dd.8
> term% cmp 8.dd /386/bin/dd
> term%
>
> Cool, no differences!  So nobody is doing anything dumb like putting
> gratuitous timestamps in binaries.  If I build the same binary twice
> I get the same bytes.  This is a feature.

You got lucky.  There are various reasons why this
won't be true in general.  See below.

> Therefore one simple improvement to replica might be: if a file
> appears to be locally modified, but is actually byte-for-byte
> identical to the corresponding file on sources, then replica should
> "update" the file after all (including changing the mtime to match the
> file on sources).

Then you have to spend time downloading the
remote file when most of the time you're going to throw it away
because the local file _has_ been locally modified.
I could have included sha1 hashes in the replica(8) metadata.
Oh well.  I expected this not to happen very often,
and so far have not been proved wrong.

> However the problem is wider than it first appears.  In the course of
> investigating this problem I also discovered (picking "ls" at random):
>
> term% mk 8.ls
> 8^c -FVw ls.c
> 8^l  -o 8.ls ls.8
> term% cmp 8.ls /386/bin/ls
> 8.ls /386/bin/ls differ: char 20
>
> This happened on the very first command other than "dd" that I tried
> to build, and it suggests that quite a few binaries may be out of date
> with the sources.

They're not out of date, just different.  In this case, there was
a one-line change to the fmt routines in libc that makes
it compile cleanly on non-Plan 9 systems, and we removed
an old prototype from libc.h.  The first caused all the binaries
to change with new code.  The second caused the symbol tables
in all the binaries to change.  Neither is worth making everyone
download all new binaries for.  We can work around the second
change, but not the first.

> So perhaps it might also be worthwhile to have a cron job on sources
> that would periodically rebuild all the binaries and compare with
> the installed versions.  However, some obnoxious programs like "ar"
> keep timestamps in binaries that are gratuitiously updated even when
> the object file contents themselves haven't changed.  Furthermore
> The lack of shared libraries would also cause a lot of binary turnover
> anytime somebody updated a popular routine in the C library.  Designing
> an automatic build that wouldn't produce a huge volume of updates
> would be hard.

Exactly.  That's why we haven't done that.

As you concluded, this is a difficult problem.
It's also not particularly well understood.  We've
got a reasonable solution in place, one that we
expect to evolve as we learn more about the
problem being solved.

Russ