[9fans] booting problems with 3rd ed.

[9fans] booting problems with 3rd ed.

[James A. Robinson]

This weekend I tried booting up the 3rd edition on two systems that
used to run the 2nd edition.  I'm unable to get either one up to the
point where it tries to start rio. One of them, the auth server, simply
pauses after device probe.  The other, my former file server, panics.
Both systems currently have OpenBSD installed (using all disk space). And
I've executed a bunch of bytebench and lmbench runs to stress test the CPU,
memory, file, and net i/o.  So I think the hardware is still solid. Of
course, the problem could be Plan 9 being much more sensitive to something
in the mainboards or CPUs. =(

The two systems, as I stated above, ran 2ed Plan 9 without any trouble.
Only two things have changed with the systems since then: 1) both systems
now have Intel PRO/100+ ethernet cards (the chips and OpenBSD indicate
they are i82557 cards) instead of the old 3com 3c509 combo cards. 2) both
systems are using different video cards. For the auth server, I swapped
out the 512k no-name card for an ATI Mach64 GU card. For the file server,
I stuck in the S3 Trio/64 that I used to have in my 2ed Plan 9 terminal.


My former Plan 9 auth/cpu server doesn't get past the device probe. The
details are in

  http://highwire.stanford.edu/~jimr/plan9/pensito.9disk.9fd

This machine has an ASUS P2B BX single-processor motherboard with a
400MHz PII.  It has 256 megs of ECC ram. The disks are a 2 gig IDE and
an ATAPI CD-ROM.  Detailed information from the OpenBSD dmesg is at the
end of the file.


My former Plan 9 file server gets further, but then panics. The full
details are in the page

  http://highwire.stanford.edu/~jimr/plan9/spensa.9disk.9fd

The hex numbers printed out at the panic seem to be different each time.

This one is using a Tyan Tiger LX multi-processor motherboard with
one 233MHz PII. It has 128 megs of non-ECC ram, an NCR53c875 "Diamond
Fireport" scsi card, and a 9 gig scsi. In addition, it has a 2gig IDE
drive.  Detailed information from the OpenBSD dmesg is at the end of
the file.

I was hoping maybe some of the good folks here could suggest possible
reasons for such odd behaviour.  I've scanned through the last 187
messages from Jim and Russ, but nothing clicked as possibly being the
same problem. =(


Jim

Re: [9fans] booting problems with 3rd ed.

[James A. Robinson]

Many thanks to Jim, Russ, and Geoff for responding. The problem with
spensa has been solved. Jim noted the problem was that spensa was mapping
the vga framebuffer in a place which clashed with Plan 9's kernel virtual
mapping.  He said to remove the 'linear=1' for my card entry in vgadb.
This worked, rio came up, and everything was happy. =)


> My former Plan 9 file server gets further, but then panics. The full
> details are:

> dosserv: serving #s/dos
> panic: mmukmap2: pa 80000000 entry 63
> 
> ktrace /kernel/path 8010649e 80406eac
> 80406e4c=801063a3 80406e5c=80170289 80406e78=8016581e 80406e80=8010ee12
> 80406e9c=8010649e 80406e9c=8010ef86 80406ea8=8010649e 80406eb0=8013a086
> 80406eb4=8013a47b 80406f00=80170289 80406f0c=8015d45c 80406f20=8015d45c
> 80406f28=80182c62 80406f30=80182c62 80406f34=801281aa 80406f40=80172089
> 80406f44=8018117b 80406f58=801705e2 80406f5c=801281aa 80406f68=80170289
> 80406f6c=8018117b 80406f84=80170289 80406fa4=8015d45c 80406fb0=8016fdfd
> 80406fb8=8016fdfd 80406fc4=80170289 80406fc8=80103daa 80406fd0=80105182 
> 80406fdc=80105318 8040701c=80104873 80407038=801604d6 80407040=80118b81
> 8040705c=80160927 804070a0=8014398a 804070c0=80150d46 804070d8=80183f44 
> 804070f4=801842f6 80407100=80184383 8040711c=801701d1 80407124=8015d633
> 80407140=80150fff 80407160=8016d3e4 804071a0=8016d3e4 804071cc=80100962
> 80407208=801004ad
> cpu0: exiting

Re: [9fans] rev control

[presotto]

Russ has a point.  A project with a small group, 10 or less, is
more easily managed by really communicating than by using a
program that lets you avoid communicating.  However, to do
it, you need a group dynamic that works.  There were usually
5 of us running around in the same code on Plan 9.  Our
programming styles eventually synthesized into something that
we could all tolerate.  We learned when and who to warn when we
are about to do something, and who to ask when we get in
trouble.  We got used to fixing each other's mistakes.  It
got to the point that I'ld make some changes, compile, find
a few syntax errors, and hear Rob or Phil yell from across
the room that they were fixing my syntax errors so they could
finish their own compile.  A text editor that tells you when
you're about to write back a file that was changed while you
were editing is a HUGE help.  And of course, that doesn't work
unless you're also living in a common file system.  It also
helps that the people are also relatively close by, reachable
instantly by phone, email, or a loud yell.

This dynamic takes time to create. I hadn't really
realized how well we worked together till Sape Mullender
joined the group.  It's taken months before Sape and
I can work in the same source without constantly stepping
on each other's toes even though we're using the same tools
that worked so well with the rest of us.

The dump file system was invaluable.  However, its utility
was not in synchronizing our actions.  It was
1) a crutch that gave us a certain fearlessness when attacking
  code.  We always knew that there was an indelible version
  from the night before.
2) a history of changes that can easily be perused to correlate
  a code change to a new behavior in the system.

Source control systems like CVS, SCCS, etc all try to provide
these two things, i.e., a history of changes and some
synchronization.  I've worked as a developer on 5 OS's now.
The worst/biggest was the #5ESS project with millions of lines of
code and thousands of  programmers.  The project was subdivided
to compartmentalize the work.  However, it was still not uncommon
for people that didn't even know of each other's existance to
be working on the same piece of code.  There wasn't a file system
big enough to handle everyone so it was necessary to check things
in and out.  It was also impossible to have an intelligent
conversation between 2 people wanting to make contradictory
changes because they were probably each trying to satisfy
requirements from different parts of the company and each was
in some way justified.  We needed to branch the code in
infitesimal ways to satisfy all the intended customers.  NO was
never an answer since the requirements were usually the result of
some law passed in the community being delivered to.  No 2 #5ESS's
have ever been delivered running the same code.  Never the less, we
have to keep track of all versions of the code that have ever been
shipped, be able to debug them, and apply the fixes to all versions
that they may apply to.  This is far beyond the ability of mortals
not to mention tools.  We had almost as many people working
on the control tools as the code itself.  The were as many/more people
adjudicating changes and doing ode reviews as there were people writing
code.  It was not unusual to take a week to write 25 lines of code
because of all the controls, testing, checking in and out, and
code reviews.

We recently tried a much smaller project, PathStar, and ran into the same
problem.  Every customer wanted/needed a different set of changes
to the code.  To control all the branching, we once more had
to result to a code management system.  The system wasn't really
needed to synchronize multiple people changing the code for a
common cause.  There were 100+ programmers and they wrote their
code on the same Plan 9 file server.  They all pretty much knew
who the others were that were working on the same code and they
could use all the mechanisms that we could.  However very few
of them managed to fall into the same work mode that we did.
Because of what they were used to they tended to copy the whole
source tree once a week, work on that, and then check the changes
back in.

I'm not sure what the moral is here.  I've always believed it was
make the project small and get a dozen or less smart peolple to work
together on it.  However, some problems don't cut up nicely into
small pieces and it's hard to get a dozen smart people into one
place so you use CVS or Sablime and deal with the heartache.

Re: [9fans] rev control

[James A. Robinson]

> Russ has a point.  A project with a small group, 10 or less, is
> more easily managed by really communicating than by using a
> program that lets you avoid communicating.  However, to do
> it, you need a group dynamic that works.  There were usually
> 5 of us running around in the same code on Plan 9.  Our
> programming styles eventually synthesized into something that
> we could all tolerate.  We learned when and who to warn when we
> are about to do something, and who to ask when we get in
> trouble.  We got used to fixing each other's mistakes.  It
> got to the point that I'ld make some changes, compile, find
> a few syntax errors, and hear Rob or Phil yell from across
> the room that they were fixing my syntax errors so they could
> finish their own compile.  A text editor that tells you when
> you're about to write back a file that was changed while you
> were editing is a HUGE help.  And of course, that doesn't work
> unless you're also living in a common file system.  It also
> helps that the people are also relatively close by, reachable
> instantly by phone, email, or a loud yell.
> 
> This dynamic takes time to create. I hadn't really
> realized how well we worked together till Sape Mullender
> joined the group.  It's taken months before Sape and
> I can work in the same source without constantly stepping
> on each other's toes even though we're using the same tools
> that worked so well with the rest of us.
> 
> The dump file system was invaluable.  However, its utility
> was not in synchronizing our actions.  It was
> 1) a crutch that gave us a certain fearlessness when attacking
>   code.  We always knew that there was an indelible version
>   from the night before.
> 2) a history of changes that can easily be perused to correlate
>   a code change to a new behavior in the system.
> 
> Source control systems like CVS, SCCS, etc all try to provide
> these two things, i.e., a history of changes and some
> synchronization.  I've worked as a developer on 5 OS's now.
> The worst/biggest was the #5ESS project with millions of lines of
> code and thousands of  programmers.  The project was subdivided
> to compartmentalize the work.  However, it was still not uncommon
> for people that didn't even know of each other's existance to
> be working on the same piece of code.  There wasn't a file system
> big enough to handle everyone so it was necessary to check things
> in and out.  It was also impossible to have an intelligent
> conversation between 2 people wanting to make contradictory
> changes because they were probably each trying to satisfy
> requirements from different parts of the company and each was
> in some way justified.  We needed to branch the code in
> infitesimal ways to satisfy all the intended customers.  NO was
> never an answer since the requirements were usually the result of
> some law passed in the community being delivered to.  No 2 #5ESS's
> have ever been delivered running the same code.  Never the less, we
> have to keep track of all versions of the code that have ever been
> shipped, be able to debug them, and apply the fixes to all versions
> that they may apply to.  This is far beyond the ability of mortals
> not to mention tools.  We had almost as many people working
> on the control tools as the code itself.  The were as many/more people
> adjudicating changes and doing ode reviews as there were people writing
> code.  It was not unusual to take a week to write 25 lines of code
> because of all the controls, testing, checking in and out, and
> code reviews.
> 
> We recently tried a much smaller project, PathStar, and ran into the same
> problem.  Every customer wanted/needed a different set of changes
> to the code.  To control all the branching, we once more had
> to result to a code management system.  The system wasn't really
> needed to synchronize multiple people changing the code for a
> common cause.  There were 100+ programmers and they wrote their
> code on the same Plan 9 file server.  They all pretty much knew
> who the others were that were working on the same code and they
> could use all the mechanisms that we could.  However very few
> of them managed to fall into the same work mode that we did.
> Because of what they were used to they tended to copy the whole
> source tree once a week, work on that, and then check the changes
> back in.
> 
> I'm not sure what the moral is here.  I've always believed it was
> make the project small and get a dozen or less smart peolple to work
> together on it.  However, some problems don't cut up nicely into
> small pieces and it's hard to get a dozen smart people into one
> place so you use CVS or Sablime and deal with the heartache.
> 

Re: [9fans] rev control

[James A. Robinson]

With a fairly small group of 15 where I work, handling around 200K lines
of code, I find the following trend with projects in CVS:

	people do make sure things compile, but don't necessarily make
	sure things work

	CVS diff is helpful, but it's still a pain in the ass to deal with merges

	at least it's more helpful than straight RCS since we each have our
	own play pen.  We only have to deal with merges when we've got a
	new feature or bug fix ready to commit back to production

	comments logged for commited code changes are useless one-liners like
	"changed stuff in function x" or "added feature y"

	differences in coding style makes sharing a tree a pain in the ass
	when it comes to doing the aforementioned diff.

Essentially, I don't see a whole lot of use in the system other than syncing
the project back into a main source tree. And that could be done with a few
shell scripts if all the other things like yesterday(1) and history(1) were
available to use on Unix.

My views would probably change with huge projects, but for medium size ones
I would prefer the plan 9 dump approach.


Jim

Re: [9fans] rev control

[James A. Robinson]

Damn it. Sorry about the quoted posted.

Re: [9fans] rev control

[Steve Kilbane]

I was thinking about the CVS issue earlier today, and I couldn't see
any problems normally "solved" by CVS and its ilk that couldn't be
addressed by Plan 9's inherent facilities, and a little scripting
(compared to a lot of scripting for a UNIX system without a version
control system).

Obviously, yesterday is the heart of things. Equally important is
the namespace; multiple views through a global set of files is
the norm, not a special feature.

Tagging could be as simple as doing a find of a tree, and saving
the result in a file named after the tag.

The biggest gap appears to be a change log for commentary. Even
there, Plan 9's append-only file permissions reduce the need for
something complex down to "cat".

I'm not saying that it all becomes trivial; any large and/or complex
project tends to bite you in ways you didn't expect. I do think that
Plan 9 is a better place to start from, though.

Having said all that, I wonder whether there isn't a system in place
already because of the way Plan 9 itself has been developed. presotto
mentions a large project where no two versions ran the same code.
In constrast, almost all of Plan 9 is single-source. I suspect this
has been highly significant in the level of perceived need for a
management system.

steve

Re: [9fans] rev control

[Steve Kilbane]

jim:
> 	people do make sure things compile, but don't necessarily make
> 	sure things work
> 
> 	CVS diff is helpful, but it's still a pain in the ass to deal with merges
> 
> 	comments logged for commited code changes are useless one-liners like
> 	"changed stuff in function x" or "added feature y"
> 
> 	differences in coding style makes sharing a tree a pain in the ass
> 	when it comes to doing the aforementioned diff.

Well, sure, but this is just people being people (i.e. lazy). CVS is
a tool; it doesn't stop you screwing things up if you want. By and large,
you've also mentioned things that Plan 9 won't help with:

- people don't make sure things work. Well...

- merging sources is a pain; Plan 9 doesn't help with this, if you happen to
  be maintaining and merging multiple versions (e.g. development, release and
  historical versions of a product).

- the need for change logs are a function of the size of a project and the
  awareness of each programmer wrt the whole project. If you need 'em, you
  need 'em. Plan 9, of itself, doesn't help.

- if everyone's playing with their own coding style, then yes, it's going to
  be a pain merging. As long as you *are* merging, that is.

steve

Re: [9fans] rev control

[Boyd Roberts]

at digital, SRC and PRL has a 'package' mechanism.  you
could get a copy of the code, fix the bugs and then 'ship'
it and it would be installed at SRC [PA] and PRL [paris].
it was ted wobber's work.  it was kinda neat.  a bit
gluggy, but it showed the power of distributed bug
fixing.  of course, it required discipline because you
could break a lotta stuff or add in a lotta crap.
it'd never work with the linux [ab]users.

Re: [9fans] rev control

[Boyd Roberts]

From: <presotto@plan9.bell-labs.com>
> 
> I'm not sure what the moral is here.  I've always believed it was
> make the project small and get a dozen or less smart peolple to work
> together on it.  However, some problems don't cut up nicely into
> small pieces and it's hard to get a dozen smart people into one
> place so you use CVS or Sablime and deal with the heartache.
>

you got it.
 

Re: [9fans] rev control

[Boyd Roberts]

/n/dump

Re: [9fans] rev control

[greg andruk]

Thanks to all who replied.  Especially helpful was the description of
how Bell Labs put the tools to use -- oh, if only all projects could
be allowed by their respective manglement entities to run that way!

Just a couple notes, since I gather that more people end up using
source control by mandate rather than choice...

There really are reasons for wanting commit logs outside blame
assignment.  For example, I tend to (ab)use RCS as a journal of sorts,
recording random ideas like folllow-ons I might want to consider on
some change I just made, or even something unrelated I might want to
think about putting in later.  If I don't write these things down
right away, I tend to forget them.  Maybe some of this could into the
source files, but sources get seriously ugly that way.

Tags have bailed me out on a few occasions when files were moved to
other places and filesystem timestamps weren't preserved.  A long
history's a really nice thing to have, but it can be too much of a
good thing if you're not quite sure where to start looking.

Anyway... I can get all of what I really want with a little wrapper
script around the editor, and perhaps a mkfile rule here and there for
packaging stuff that escapes outside.  There are only two of us
working on this stuff here, so elaborate lockout schemes would be
overkill.

I did look at porting RCS, but on realizing it wants a fancier flavor
of diff than what Plan 9 provides I made it go away.  Having things
break from guessing incorectly which of the SysV, BSD and GNU versions
of every freaking Unix futility is needed, is a situation I really
don't care to help replicate.

The CVS client port, OTOH, will be a big help in slurping some sources
from places I can't easily get them in another way (Sourceforge etc.),
thanks for that, Russ.

Re: [9fans] rev control

[Russ Cox]

  What is the Plan 9 answer to stuff like SCCS and RCS?  The yesterday
  facility takes care of the "how do I back out of this mess?" problem,
  but I'm not sure what would be analogous to tags and commit logs.  Are
  there other tools I've overlooked?

As Scott said, yesterday is it.  For the most part,
changes happen slowly enough that the granularity of a day
is just fine.  history -D prints a diff log, which is
not quite a commit log but just as useful.  If you really
want to know, the file server keeps track of who last
touched each file, and this information is in the dump
too, so it's not hard to assign blame for old changes.

I've worked on projects that required CVS, and I've
worked on Plan 9.  Not surprisingly, I like the Plan 9 
way better.  From a usability standpoint, I like
being able to run any command I want on the dump
file system.  There's nothing to learn except the
typing of "9fs dump".  Contrast this with CVS, where
you have to learn the CVS lexicon, which is just close
enough to familiar tools to be incredibly frustrating,
in order to do anything meaningful.

The more compelling issue, though, is that many people
treat CVS as a license to commit changes: if the changes break
something, someone else can easily back them out.  
But the time spent figuring out that it's not your fault
that the program is crashing, and then finding the correct
change to back out, is lost.  The Plan 9 way requires
more communication than the CVS way, which means most
people have a better sense of what's going on and
how the changes will affect them.

I spent the summer working on a programming languages
book: the text, the associated code, and the build
scripts were all under CVS control.  Oh, and I was
using Plan 9 (hence the CVS port) but the professor
with whom I was working was using Linux.  We fell into
the CVS trap -- instead of talking through changes
before they happened, many of the discussions happened
afterward, when one of us couldn't build the world after
the other's changes.  We got a lot done, but the lack
of extensive communication about who currently owned what
meant we stepped on each other's feet more often 
than we would have liked.

I think a single common source with communication between
the (human) writers is still the best way to go.  

Russ

Re: [9fans] rev control

[Scott Schwartz]

> What is the Plan 9 answer to stuff like SCCS and RCS? 

If you run the real file server, cd `{yesterday}.

Re: [9fans] rev control

[William K. Josephson]

On Tue, Jan 02, 2001 at 05:47:58PM +0000, greg andruk wrote:

> What is the Plan 9 answer to stuff like SCCS and RCS?  The yesterday
> facility takes care of the "how do I back out of this mess?" problem,
> but I'm not sure what would be analogous to tags and commit logs.  Are
> there other tools I've overlooked?

There is a port of CVS (and so RCS) somewhere.  Russ?

 -WJ

[9fans] rev control

[greg andruk]

I have a nagging feeling that I'm missing something painfully obvious
here, but I give up on trying to figure this one out myself.

What is the Plan 9 answer to stuff like SCCS and RCS?  The yesterday
facility takes care of the "how do I back out of this mess?" problem,
but I'm not sure what would be analogous to tags and commit logs.  Are
there other tools I've overlooked?

Thanks.