[9fans] booting problems with 3rd ed.

[9fans] booting problems with 3rd ed.

[James A. Robinson]

This weekend I tried booting up the 3rd edition on two systems that
used to run the 2nd edition.  I'm unable to get either one up to the
point where it tries to start rio. One of them, the auth server, simply
pauses after device probe.  The other, my former file server, panics.
Both systems currently have OpenBSD installed (using all disk space). And
I've executed a bunch of bytebench and lmbench runs to stress test the CPU,
memory, file, and net i/o.  So I think the hardware is still solid. Of
course, the problem could be Plan 9 being much more sensitive to something
in the mainboards or CPUs. =(

The two systems, as I stated above, ran 2ed Plan 9 without any trouble.
Only two things have changed with the systems since then: 1) both systems
now have Intel PRO/100+ ethernet cards (the chips and OpenBSD indicate
they are i82557 cards) instead of the old 3com 3c509 combo cards. 2) both
systems are using different video cards. For the auth server, I swapped
out the 512k no-name card for an ATI Mach64 GU card. For the file server,
I stuck in the S3 Trio/64 that I used to have in my 2ed Plan 9 terminal.


My former Plan 9 auth/cpu server doesn't get past the device probe. The
details are in

  http://highwire.stanford.edu/~jimr/plan9/pensito.9disk.9fd

This machine has an ASUS P2B BX single-processor motherboard with a
400MHz PII.  It has 256 megs of ECC ram. The disks are a 2 gig IDE and
an ATAPI CD-ROM.  Detailed information from the OpenBSD dmesg is at the
end of the file.


My former Plan 9 file server gets further, but then panics. The full
details are in the page

  http://highwire.stanford.edu/~jimr/plan9/spensa.9disk.9fd

The hex numbers printed out at the panic seem to be different each time.

This one is using a Tyan Tiger LX multi-processor motherboard with
one 233MHz PII. It has 128 megs of non-ECC ram, an NCR53c875 "Diamond
Fireport" scsi card, and a 9 gig scsi. In addition, it has a 2gig IDE
drive.  Detailed information from the OpenBSD dmesg is at the end of
the file.

I was hoping maybe some of the good folks here could suggest possible
reasons for such odd behaviour.  I've scanned through the last 187
messages from Jim and Russ, but nothing clicked as possibly being the
same problem. =(


Jim

Re: [9fans] booting problems with 3rd ed.

[James A. Robinson]

Many thanks to Jim, Russ, and Geoff for responding. The problem with
spensa has been solved. Jim noted the problem was that spensa was mapping
the vga framebuffer in a place which clashed with Plan 9's kernel virtual
mapping.  He said to remove the 'linear=1' for my card entry in vgadb.
This worked, rio came up, and everything was happy. =)


> My former Plan 9 file server gets further, but then panics. The full
> details are:

> dosserv: serving #s/dos
> panic: mmukmap2: pa 80000000 entry 63
> 
> ktrace /kernel/path 8010649e 80406eac
> 80406e4c=801063a3 80406e5c=80170289 80406e78=8016581e 80406e80=8010ee12
> 80406e9c=8010649e 80406e9c=8010ef86 80406ea8=8010649e 80406eb0=8013a086
> 80406eb4=8013a47b 80406f00=80170289 80406f0c=8015d45c 80406f20=8015d45c
> 80406f28=80182c62 80406f30=80182c62 80406f34=801281aa 80406f40=80172089
> 80406f44=8018117b 80406f58=801705e2 80406f5c=801281aa 80406f68=80170289
> 80406f6c=8018117b 80406f84=80170289 80406fa4=8015d45c 80406fb0=8016fdfd
> 80406fb8=8016fdfd 80406fc4=80170289 80406fc8=80103daa 80406fd0=80105182 
> 80406fdc=80105318 8040701c=80104873 80407038=801604d6 80407040=80118b81
> 8040705c=80160927 804070a0=8014398a 804070c0=80150d46 804070d8=80183f44 
> 804070f4=801842f6 80407100=80184383 8040711c=801701d1 80407124=8015d633
> 80407140=80150fff 80407160=8016d3e4 804071a0=8016d3e4 804071cc=80100962
> 80407208=801004ad
> cpu0: exiting

Re: [9fans] rev control

[presotto]

Russ has a point.  A project with a small group, 10 or less, is
more easily managed by really communicating than by using a
program that lets you avoid communicating.  However, to do
it, you need a group dynamic that works.  There were usually
5 of us running around in the same code on Plan 9.  Our
programming styles eventually synthesized into something that
we could all tolerate.  We learned when and who to warn when we
are about to do something, and who to ask when we get in
trouble.  We got used to fixing each other's mistakes.  It
got to the point that I'ld make some changes, compile, find
a few syntax errors, and hear Rob or Phil yell from across
the room that they were fixing my syntax errors so they could
finish their own compile.  A text editor that tells you when
you're about to write back a file that was changed while you
were editing is a HUGE help.  And of course, that doesn't work
unless you're also living in a common file system.  It also
helps that the people are also relatively close by, reachable
instantly by phone, email, or a loud yell.

This dynamic takes time to create. I hadn't really
realized how well we worked together till Sape Mullender
joined the group.  It's taken months before Sape and
I can work in the same source without constantly stepping
on each other's toes even though we're using the same tools
that worked so well with the rest of us.

The dump file system was invaluable.  However, its utility
was not in synchronizing our actions.  It was
1) a crutch that gave us a certain fearlessness when attacking
  code.  We always knew that there was an indelible version
  from the night before.
2) a history of changes that can easily be perused to correlate
  a code change to a new behavior in the system.

Source control systems like CVS, SCCS, etc all try to provide
these two things, i.e., a history of changes and some
synchronization.  I've worked as a developer on 5 OS's now.
The worst/biggest was the #5ESS project with millions of lines of
code and thousands of  programmers.  The project was subdivided
to compartmentalize the work.  However, it was still not uncommon
for people that didn't even know of each other's existance to
be working on the same piece of code.  There wasn't a file system
big enough to handle everyone so it was necessary to check things
in and out.  It was also impossible to have an intelligent
conversation between 2 people wanting to make contradictory
changes because they were probably each trying to satisfy
requirements from different parts of the company and each was
in some way justified.  We needed to branch the code in
infitesimal ways to satisfy all the intended customers.  NO was
never an answer since the requirements were usually the result of
some law passed in the community being delivered to.  No 2 #5ESS's
have ever been delivered running the same code.  Never the less, we
have to keep track of all versions of the code that have ever been
shipped, be able to debug them, and apply the fixes to all versions
that they may apply to.  This is far beyond the ability of mortals
not to mention tools.  We had almost as many people working
on the control tools as the code itself.  The were as many/more people
adjudicating changes and doing ode reviews as there were people writing
code.  It was not unusual to take a week to write 25 lines of code
because of all the controls, testing, checking in and out, and
code reviews.

We recently tried a much smaller project, PathStar, and ran into the same
problem.  Every customer wanted/needed a different set of changes
to the code.  To control all the branching, we once more had
to result to a code management system.  The system wasn't really
needed to synchronize multiple people changing the code for a
common cause.  There were 100+ programmers and they wrote their
code on the same Plan 9 file server.  They all pretty much knew
who the others were that were working on the same code and they
could use all the mechanisms that we could.  However very few
of them managed to fall into the same work mode that we did.
Because of what they were used to they tended to copy the whole
source tree once a week, work on that, and then check the changes
back in.

I'm not sure what the moral is here.  I've always believed it was
make the project small and get a dozen or less smart peolple to work
together on it.  However, some problems don't cut up nicely into
small pieces and it's hard to get a dozen smart people into one
place so you use CVS or Sablime and deal with the heartache.

Re: [9fans] rev control

[James A. Robinson]

> Russ has a point.  A project with a small group, 10 or less, is
> more easily managed by really communicating than by using a
> program that lets you avoid communicating.  However, to do
> it, you need a group dynamic that works.  There were usually
> 5 of us running around in the same code on Plan 9.  Our
> programming styles eventually synthesized into something that
> we could all tolerate.  We learned when and who to warn when we
> are about to do something, and who to ask when we get in
> trouble.  We got used to fixing each other's mistakes.  It
> got to the point that I'ld make some changes, compile, find
> a few syntax errors, and hear Rob or Phil yell from across
> the room that they were fixing my syntax errors so they could
> finish their own compile.  A text editor that tells you when
> you're about to write back a file that was changed while you
> were editing is a HUGE help.  And of course, that doesn't work
> unless you're also living in a common file system.  It also
> helps that the people are also relatively close by, reachable
> instantly by phone, email, or a loud yell.
> 
> This dynamic takes time to create. I hadn't really
> realized how well we worked together till Sape Mullender
> joined the group.  It's taken months before Sape and
> I can work in the same source without constantly stepping
> on each other's toes even though we're using the same tools
> that worked so well with the rest of us.
> 
> The dump file system was invaluable.  However, its utility
> was not in synchronizing our actions.  It was
> 1) a crutch that gave us a certain fearlessness when attacking
>   code.  We always knew that there was an indelible version
>   from the night before.
> 2) a history of changes that can easily be perused to correlate
>   a code change to a new behavior in the system.
> 
> Source control systems like CVS, SCCS, etc all try to provide
> these two things, i.e., a history of changes and some
> synchronization.  I've worked as a developer on 5 OS's now.
> The worst/biggest was the #5ESS project with millions of lines of
> code and thousands of  programmers.  The project was subdivided
> to compartmentalize the work.  However, it was still not uncommon
> for people that didn't even know of each other's existance to
> be working on the same piece of code.  There wasn't a file system
> big enough to handle everyone so it was necessary to check things
> in and out.  It was also impossible to have an intelligent
> conversation between 2 people wanting to make contradictory
> changes because they were probably each trying to satisfy
> requirements from different parts of the company and each was
> in some way justified.  We needed to branch the code in
> infitesimal ways to satisfy all the intended customers.  NO was
> never an answer since the requirements were usually the result of
> some law passed in the community being delivered to.  No 2 #5ESS's
> have ever been delivered running the same code.  Never the less, we
> have to keep track of all versions of the code that have ever been
> shipped, be able to debug them, and apply the fixes to all versions
> that they may apply to.  This is far beyond the ability of mortals
> not to mention tools.  We had almost as many people working
> on the control tools as the code itself.  The were as many/more people
> adjudicating changes and doing ode reviews as there were people writing
> code.  It was not unusual to take a week to write 25 lines of code
> because of all the controls, testing, checking in and out, and
> code reviews.
> 
> We recently tried a much smaller project, PathStar, and ran into the same
> problem.  Every customer wanted/needed a different set of changes
> to the code.  To control all the branching, we once more had
> to result to a code management system.  The system wasn't really
> needed to synchronize multiple people changing the code for a
> common cause.  There were 100+ programmers and they wrote their
> code on the same Plan 9 file server.  They all pretty much knew
> who the others were that were working on the same code and they
> could use all the mechanisms that we could.  However very few
> of them managed to fall into the same work mode that we did.
> Because of what they were used to they tended to copy the whole
> source tree once a week, work on that, and then check the changes
> back in.
> 
> I'm not sure what the moral is here.  I've always believed it was
> make the project small and get a dozen or less smart peolple to work
> together on it.  However, some problems don't cut up nicely into
> small pieces and it's hard to get a dozen smart people into one
> place so you use CVS or Sablime and deal with the heartache.
> 

Re: [9fans] rev control

[James A. Robinson]

With a fairly small group of 15 where I work, handling around 200K lines
of code, I find the following trend with projects in CVS:

	people do make sure things compile, but don't necessarily make
	sure things work

	CVS diff is helpful, but it's still a pain in the ass to deal with merges

	at least it's more helpful than straight RCS since we each have our
	own play pen.  We only have to deal with merges when we've got a
	new feature or bug fix ready to commit back to production

	comments logged for commited code changes are useless one-liners like
	"changed stuff in function x" or "added feature y"

	differences in coding style makes sharing a tree a pain in the ass
	when it comes to doing the aforementioned diff.

Essentially, I don't see a whole lot of use in the system other than syncing
the project back into a main source tree. And that could be done with a few
shell scripts if all the other things like yesterday(1) and history(1) were
available to use on Unix.

My views would probably change with huge projects, but for medium size ones
I would prefer the plan 9 dump approach.


Jim

Re: [9fans] rev control

[James A. Robinson]

Damn it. Sorry about the quoted posted.

Re: [9fans] rev control

[Steve Kilbane]

I was thinking about the CVS issue earlier today, and I couldn't see
any problems normally "solved" by CVS and its ilk that couldn't be
addressed by Plan 9's inherent facilities, and a little scripting
(compared to a lot of scripting for a UNIX system without a version
control system).

Obviously, yesterday is the heart of things. Equally important is
the namespace; multiple views through a global set of files is
the norm, not a special feature.

Tagging could be as simple as doing a find of a tree, and saving
the result in a file named after the tag.

The biggest gap appears to be a change log for commentary. Even
there, Plan 9's append-only file permissions reduce the need for
something complex down to "cat".

I'm not saying that it all becomes trivial; any large and/or complex
project tends to bite you in ways you didn't expect. I do think that
Plan 9 is a better place to start from, though.

Having said all that, I wonder whether there isn't a system in place
already because of the way Plan 9 itself has been developed. presotto
mentions a large project where no two versions ran the same code.
In constrast, almost all of Plan 9 is single-source. I suspect this
has been highly significant in the level of perceived need for a
management system.

steve

Re: [9fans] rev control

[Steve Kilbane]

jim:
> 	people do make sure things compile, but don't necessarily make
> 	sure things work
> 
> 	CVS diff is helpful, but it's still a pain in the ass to deal with merges
> 
> 	comments logged for commited code changes are useless one-liners like
> 	"changed stuff in function x" or "added feature y"
> 
> 	differences in coding style makes sharing a tree a pain in the ass
> 	when it comes to doing the aforementioned diff.

Well, sure, but this is just people being people (i.e. lazy). CVS is
a tool; it doesn't stop you screwing things up if you want. By and large,
you've also mentioned things that Plan 9 won't help with:

- people don't make sure things work. Well...

- merging sources is a pain; Plan 9 doesn't help with this, if you happen to
  be maintaining and merging multiple versions (e.g. development, release and
  historical versions of a product).

- the need for change logs are a function of the size of a project and the
  awareness of each programmer wrt the whole project. If you need 'em, you
  need 'em. Plan 9, of itself, doesn't help.

- if everyone's playing with their own coding style, then yes, it's going to
  be a pain merging. As long as you *are* merging, that is.

steve

Re: [9fans] rev control

[Boyd Roberts]

at digital, SRC and PRL has a 'package' mechanism.  you
could get a copy of the code, fix the bugs and then 'ship'
it and it would be installed at SRC [PA] and PRL [paris].
it was ted wobber's work.  it was kinda neat.  a bit
gluggy, but it showed the power of distributed bug
fixing.  of course, it required discipline because you
could break a lotta stuff or add in a lotta crap.
it'd never work with the linux [ab]users.

Re: [9fans] rev control

[Boyd Roberts]

From: <presotto@plan9.bell-labs.com>
> 
> I'm not sure what the moral is here.  I've always believed it was
> make the project small and get a dozen or less smart peolple to work
> together on it.  However, some problems don't cut up nicely into
> small pieces and it's hard to get a dozen smart people into one
> place so you use CVS or Sablime and deal with the heartache.
>

you got it.