[9fans] Installing the updates

[9fans] Installing the updates

[Doug Henderson]

In one of my porting project, <stdio.h> is included twice. This fails,
because, for instance, the way getchar is declared, first as an external
function, then as a macro. As a result, I added the usual protection code
(#ifndef <tag>; #define <tag>; <original body of header>; #endif) to
/sys/include/stdio.h, and others. I have also added some defines to allow
<event.h> and <mouse.h> to both be included without error/warning messages.
And I messed around with the ethernet driver code for my card (RealTeck
8029AS) trying to find out why it didn't work. Etc, etc.

I understand that wrap/inst will not overwrite my modified files, unless I
force it to do so. But I want to get a list of them, stash the source
changes to safety,
and overwrite all files with those from the release. Then, if still
necessary, I
will reapply the parts that actually fix a problem, instead of help diagnose
one. I want to have a clean updated system, with well known modifications.

Is there a wrap/inst expert out there that can provide some quick help.

How do I list the contents of a wrap archive?

How do I list the conflicts between a wrap archive and the current versions
(without actually installing anything)?

Thanks,
Doug
--  --
Doug Henderson, Calgary, Alberta, Canada

Re: [9fans] Installing the updates

[Scott Schwartz]

One thing you can do, at the cost of some preparatory work, is to
create a tree of source directories in your home directory, and then
bind -ac each one in front of the corresponding system source directory.
Then your changes stay private, and you don't have to run as adm or use
kfscmd allow.

(With some more preparation you could write a fileserver that just
interposed itself at the root of a subtree, and let the originals
show through.)

Speaking of inst, one probablu ought to unpack with -r /n/kfs, so that
bindings in one's profile don't confuse it.

An example of such confusion was that my bin/386/ip was was bound ahead
of /bin/ip.  /lib/namespace binds /386/bin to /bin, instead of unioning
/386/bin onto /bin, so later binds leak names into /386.  I think it
ought to make the union right at the outset, instead.

Re: [9fans] Installing the updates

[Steve Simon]

Scott Schwartz wrote:
[snip]
> 
> (With some more preparation you could write a fileserver that just
> interposed itself at the root of a subtree, and let the originals
> show through.)
> 
[snip]

I always liked the idea of a small cache file system like that used on
The fs.

A user could have the distribution source on a CDROM and view it via a
cache file
server. This would store only the files that have changed - using copy
on write.
I suspose this could be extended to storing only the blocks that have
changed, or
even file diff(1) output, or wrap(8) file even...

I'am not sure if this is interesting or hedious :-)

-Steve

Re: [9fans] Installing the updates

[rob pike]

> In one of my porting project, <stdio.h> is included twice.

Pardon me for doing a boyd here, but you only need to include
it once - you get everything defined the first time.

Seriously, we deliberately refused to buy into that multiple
inclusion dance, which is a hideous non-solution for the problem
of undefined dependency order on #includes.  Why not use the
occasion to clean up the code so you only include <stdio.h> once?

I'll let Russ answer your wrap questions; he's the author.

-rob

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

rob pike wrote:
> Seriously, we deliberately refused to buy into that multiple
> inclusion dance, which is a hideous non-solution for the problem
> of undefined dependency order on #includes.  Why not use the
> occasion to clean up the code so you only include <stdio.h> once?

One doesn't always have that option; for example, a header file
might be part of somebody else's project on a machine far, far away.
/* imported.h */
#include <stdio.h> /* need FILE decl */
extern imp_read( FILE*, char *whatever );

Many developers take the approach that each interface header
should be self-contained, so that the user of the header doesn't
need to know anything about the details of the implementation of
that header.  Information hiding, you know.

Since the C standard definitely requires idempotency for the
standard headers (except <assert.h>), at least the APE headers
ought to conform, to ease the burden of importing non-native code.

Re: [9fans] Installing the updates

[Howard Trickey]

> From: "Douglas A. Gwyn" <gwyn@arl.army.mil>
> Since the C standard definitely requires idempotency for the
> standard headers (except <assert.h>), at least the APE headers
> ought to conform, to ease the burden of importing non-native code.

The APE headers do indeed conform.
There are two stdio.h's; the one in /sys/include is for the "plan 9 way"
of programming (no include protection), and the one in /sys/include/ape
is for importing non-native code.

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

> The APE headers do indeed conform.

Okay, problem solved.  The Plan9 native headers can
certainly do whatever Rob and crew think is best.
We already know that it is non-trivial to port much
non-Plan9 code to Plan9-native and vice versa.

Re: [9fans] Installing the updates

[Ralph Corderoy]

Hi,

> rob pike wrote:
> > Seriously, we deliberately refused to buy into that multiple
> > inclusion dance, which is a hideous non-solution for the problem of
> > undefined dependency order on #includes.  Why not use the occasion
> > to clean up the code so you only include <stdio.h> once?
> 
> ...
> 
> Many developers take the approach that each interface header should
> be self-contained, so that the user of the header doesn't need to
> know anything about the details of the implementation of that header.
> Information hiding, you know.

I think he's heard the argument but doesn't agree.

    "Simple rule: include files should never include include files."
        -- http://www.lysator.liu.se/c/pikestyle.html

It's well worth a read if you haven't already.

(The document is a sane view in a profession contorted by ridculous
company coding standards that forbid short variable names and insist on
`add 1 to the object's reference counter' comments.  Normally also the
place where code reviews are a farce where everyone tries to detect a
couple of spelling errors, even the same spelling errors as everyone
else, so they can show they studied the code.)


Ralph.

Re: [9fans] Installing the updates

[Greg Hudson]

> I think he's heard the argument but doesn't agree.

It would be nice if Pike could present a compelling argument.  His web
page states only:

	Multiple inclusions are a bane of systems programming.  It's
	not rare to have files included five or more times to compile
	a single C source file.  The Unix /usr/include/sys stuff is
	terrible this way.

Where's the horror here?  Computers are fast.  Pushing extra work on
programmers and creating an unnecessary portability issue is a high
cost.  Reading a header file five or more times during compilation is
a low cost (and one which can be optimized away for ifdef-protected
headers; I'm told gcc does so).

Re: [9fans] Installing the updates

[James A. Robinson]

> It would be nice if Pike could present a compelling argument. 
> [...]
> Where's the horror here?  Computers are fast.  Pushing extra work on
> programmers and creating an unnecessary portability issue is a high
> cost.  Reading a header file five or more times during compilation is
> a low cost (and one which can be optimized away for ifdef-protected
> headers; I'm told gcc does so).

The horror is that many header files don't get 'the dance' right. Even OS
header files are screwed up sometimes. Before I read Notes on Programming
in C I had put all #include calls into my program's local header file
because that was how I had always seen it done.  When I started writing
libraries, I found all kinds of problems with the compiler complaining
about multiple definitions (both my own and the system headers).

I think it even makes good sense from an interface perspective. The header
file shows the interface for program file. The program should hide all
the implementation details, which means you shouldn't be able to tell
which system calls the program makes to get the job done. Realizing that
you keep including the same files may also force an eye toward keeping
solid boundries across the different files (one does x, and only x. The
next only handles y).  Of course, if the file hides a bad job (say,
calls to strtok(2)) then you're screwed anyway.

Why do people object to this rule? It's not hard to follow, right?
It's really not that much extra work, in my opinion. Of course there
are a few that always get included, stdlib for example, but I've found
only a few overlaps in a library I've just written. It's spread over
9 files, and you can see that there isn't *that* much overlap:

; grep '#include' *.c|cut -d: -f2|sort|uniq -c|sort -nr
      8	#include <stdlib.h>
      5	#include <string.h>
      5	#include <limits.h>
      5	#include <errno.h>
      4	#include <unistd.h>
      4	#include <sys/types.h>
      4	#include <sys/stat.h>
      4	#include <stdio.h>
      4	#include <fcntl.h>
      2	#include <time.h>
      2	#include <stdarg.h>
      2	#include <pthread.h>
      1	#include <sys/poll.h>
      1	#include <sys/file.h>
      1	#include <strings.h>
      1	#include <ctype.h>
      1	#include <assert.h>

Jim

Re: [9fans] Installing the updates

[James A. Robinson]

Here's an even better example. Rob's sam distribution:
note how the system headers are only called once.

; grep '#include' *.c|cut -d: -f2|sort|uniq -c|sort -nr
     19	#include "sam.h"
      4	#include "parse.h"
      2	#include "config.h"
      1	#include <string.h>
      1	#include <stdlib.h>
      1	#include <varargs.h>
      1 #include <sys/wait.h>
      1 #include <sys/stat.h>
      1 #include <stdio.h>
      1 #include <stdarg.h>
      1 #include <signal.h>
      1 #include "sam.h"

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

Greg Hudson wrote:
> a low cost (and one which can be optimized away for ifdef-protected
> headers; I'm told gcc does so).

The general trick is for the preprocessor to maintain
a table of (lock_symbol,header_name) entries, and when
a new header file is read, to make a note whether it
consists of <white>#ifndef symbol...#endif<white>
(after comment processing).  If so, the symbol and
file name go into the table.  #include processing
always checks the table and if there is an entry for
the file, if the associated lock_symbol is defined,
the file is not even opened.  This trick does no harm.

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

Ralph Corderoy wrote:
> > Many developers take the approach that each interface header should
> > be self-contained, so that the user of the header doesn't need to
> > know anything about the details of the implementation of that header.
> > Information hiding, you know.
> I think he's heard the argument but doesn't agree.
>         -- http://www.lysator.liu.se/c/pikestyle.html
> (The document is a sane view in a profession contorted by ridculous
> company coding standards that forbid short variable names and insist on
> `add 1 to the object's reference counter' comments.  Normally also the
> place where code reviews are a farce where everyone tries to detect a
> couple of spelling errors, even the same spelling errors as everyone
> else, so they can show they studied the code.)

I wasn't saying that *either* approach is always best,
just that there could be solid reasons why the standard
headers need to be idempotent as specified.

Making interface headers self-contained is hardly in
the same category as the "coding standards" you cite.

Re: [9fans] Installing the updates

[ozan s. yigit]

ghudson@mit.edu (Greg Hudson) writes:

> Where's the horror here?  Computers are fast.  Pushing extra work on
> programmers and creating an unnecessary portability issue is a high
> cost.

alas these costs are not calculated on ones armchair easily. i would say
that in my industry experience, we pay high porting and maintenance cost
for code that does not pay enough attention to its includes or screws
around with many nested ifdefs, especially for products on multiple
architectures. [ymmv, especially if you have all the time
in the world.]

oz
---
freedom has a mental cost. -- peter roosen-runge

Re: [9fans] Installing the updates

[jmk]

Doug Henderson <djhender@telusplanet.net>:
	And I messed around with the ethernet driver code for my card (RealTeck
	8029AS) trying to find out why it didn't work. Etc, etc.

I have an ether8029.c, it's really just a front end to the 8390 code
that autoconfigures. Yours if you want it.

--jim

Re: [9fans] Installing the updates

[Russ Cox]

	How do I list the contents of a wrap archive?

gunzip < arch.9gz | disk/mkext -h

	How do I list the conflicts between a wrap archive and the current versions
	(without actually installing anything)?

wrap/inst -x arch.9gz
will print what it would do, without actually doing anything.

wrap/wdiff -l arch.9gz
may be worth trying as well.

Russ

Re: [9fans] Installing the updates

[Russ Cox]

	A user could have the distribution source on a CDROM and view it via a
	cache file
	server. This would store only the files that have changed - using copy
	on write.
	I suspose this could be extended to storing only the blocks that have
	changed, or
	even file diff(1) output, or wrap(8) file even...
	
	I'am not sure if this is interesting or hedious :-)

I think it's interesting, at least until you
start talking about saving individual blocks
or diff output, at which point it gets complicated
enough to perhaps qualify as hideous.
Especially given that binaries don't change much.

Re: [9fans] Installing the updates

[Russ Cox]

[didn't mean to send that first one -- misclicked]

	A user could have the distribution source on a CDROM and view it via a
	cache file
	server. This would store only the files that have changed - using copy
	on write.
	I suspose this could be extended to storing only the blocks that have
	changed, or
	even file diff(1) output, or wrap(8) file even...
	
	I'am not sure if this is interesting or hedious :-)

I think it's interesting, at least until you
start talking about saving individual blocks
or diff output, at which point it gets complicated
enough to perhaps qualify as hideous.
Especially since with the exception of log files
and the fortune database, files tend to be rewritten
completely when they change.

I've thought about writing such a thing a number
of times, but been scared away both by not having
a name for it and by the complexity of the state
machine you'd need to use only a constant number of
processes.

I've since come up with a name -- stitch -- and
I think that you can use libthread to great advantage
to manage the complexity, but I haven't tried it.

If you had such a thing, you could try out
new wrap updates by doing

	gunzip < /tmp/new.9gz >/tmp/new.9
	archfs /tmp/new.9
	stitch -b /tmp/new.9 /

which would be neat.

Russ

Re: [9fans] Installing the updates

[rob pike]

> Since the C standard definitely requires idempotency for the
> standard headers (except <assert.h>), at least the APE headers
> ought to conform, to ease the burden of importing non-native code.

Yes, and our APE headers do conform.  But ANSI C made a blunder
on this one, I think. They should at least have deprecated the practice.

But for the native environment, we corrected the error and did it right.

Plan 9 is non-standard and proud of it.

-rob

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

rob pike wrote:
> Plan 9 is non-standard and proud of it.

That's fine; research involves trying out different ideas.

If and when it makes importing code a big enough problem,
then it could be changed (without adversely impacting
on existing code).

Re: [9fans] Installing the updates

[rob pike]

	A user could have the distribution source on a CDROM and view it via a
	cache file
	server. This would store only the files that have changed - using copy
	on write.
	I suspose this could be extended to storing only the blocks that have
	changed, or
	even file diff(1) output, or wrap(8) file even...
	
	I'am not sure if this is interesting or hedious :-)

This is the essence of what BSD calls union directories.

-rob

Re: [9fans] Installing the updates

[Lucio De Re]

On Tue, Aug 01, 2000 at 09:06:08AM -0400, rob pike wrote:
> 	
> 	I'am not sure if this is interesting or hedious :-)
> 
> This is the essence of what BSD calls union directories.
> 
In which case, it is hideous: there are bugs lurking in union
directories and filesystems (I also can't quite tell them apart, let's
call them "mounts" for convenience) that _nobody_ wants to tackle.
Mostly related to locking, if I'm not mistaken.

++L

Re: [9fans] Installing the updates

[rob pike]

Perhaps I should explain better.

The only tenable argument in favor of allowing #include files to
include #include files is that a lot of software does that, so
permitting it makes software easy to port.  That is, I'm sure, why the
ANSI C committee accepted it: it's standard practice.

Nonetheless, it's a cop-out.  It's a terrible practice that leads to
n-squared inclusion of files, painful overwork by the C preprocessor,
and a bad name for the idea of #include files in the first place.

When designing the Plan 9 environment, we started with ANSI C for the
language.  Other than a couple of extensions to the language proper,
we took ANSI C at face value.  The library was also accepted pretty
much as is, with a few minor differences.  However, the C preprocessor
was unacceptable to us because a) it was very complex; b) it had lots
of new stuff designed by committee, a recipe for disaster; and c) most
of the new stuff encouraged conditional compilation and other horrors
we were trying to get away from.

Our idea was to build a portable system, one whose software could be
compiled without change - and that means without conditional
compilation, since you always have to change the #ifdefs when you
port; that is the very definition of config - as we changed
architectures.  Just architectures, not operating systems: we did not
expect Plan 9 programs to be compilable on Unix systems, but we did
expect a Plan 9 program that runs on a 386 to work on a SPARC, say,
without change, just by compiling it.

To meet that goal, we had to break with the ANSI include rules.

To import external code, we (well, Howard) built a fairly strict ANSI
C/POSIX environment that follows all the rules, including getting the
libraries and C preprocessors to have standard behavior.  If you want
to import code, that's the minimum effort path.

But if you want to build Plan 9 code, part of the environment is that
#include files don't #include each other.  That is just the way it
works, like not running X or using mk instead of make or setting
objtype=mips to build for the Mips.

In short, the argument that existing software #includes like that so
Plan 9 should is bogus, *because Plan 9 software does not*.

Plan 9 is a different environment, but it's different for a reason.
In the case of #include files, it's to make software easier to port
and simpler to maintain.  Within its own, admittedly smaller, world,
it succeeds.  Show me another system that builds all its software on
multiple architectures without #ifdefs or config.  Show me another
system where I can compile on a SPARC, link on a 386, and run on a
Mips.  Show me another system where I can update an application for
every architecture by typing the equivalent of
	mk installall
I'd love to hear of it!  But in this #ifdef, config-riddled world I
don't think it exists.

-rob

Re: [9fans] Installing the updates

[Greg Hudson]

> but we did expect a Plan 9 program that runs on a 386 to work on a
> SPARC, say, without change, just by compiling it.
[...]
> Show me another system that builds all its software on multiple
> architectures without #ifdefs or config.

I don't see what the big deal here is.  Processor architecture has
never been a big source of portability problems for Unix software.
Software has to actually go and do something weird to uncover a
portability problem relating to the hardware platform.  It's true,
some software does weird stuff (playing fast and loose with
representations of integers, or having native assembly code to speed
up certain operations), but it's not the majority of software and such
software would presumably have the same problems doing those things on
different architectures all running Plan 9.

Portability problems arise mainly from operating systems, not from
processors.

I also don't see what this has to do with multiple include protection
and headers including other headers.  With regard to that issue, I'm
fine with the Plan 9 designers making the choice they made, but it
puzzles me when I see statements like this one:

	It's a terrible practice that leads to n-squared inclusion of
	files, painful overwork by the C preprocessor, and a bad name
	for the idea of #include files in the first place.

which dramatically overstate the (largely nonexistent) problems
resulting from the alternative.

Re: [9fans] Installing the updates

[Douglas A. Gwyn]

rob pike wrote:
> To meet that goal, we had to break with the ANSI include rules.

I don't see why.  The C standard does *not* say that
you *must* include any header multiple times; you
can enforce Plan9 programming discipline regardless
of whether headers have internal idempotency locks.
The presence or absence of such locks has no effect
on Plan9-style applications that #include every header
needed, once apiece in whatever the proper order is.

Indeed, the Plan9 header discipline imposes an
interdepency ordering upon the system headers,
such that e.g. <libsec.h> and <libmp.h> cannot
both define something that the other header needs,
which might otherwise be a natural thing to do.

It is nice that Plan9 programs don't individually
use #ifdefs and config, but really the use of
centrally-defined types and macros in a common
environment avoids such stuff.  In effect, the
differences are factored out to a global level.

The reason for remaining #ifdefs in many actual
applications is that they have to work in many
environments that differ in much more radical ways
than any two instances of Plan9.  There are ways
of factoring out the differences, and I generally
recommend them in the design phase, but most
applications are created by people who don't know
(or care) about all the ways other environments
differ from the one where they do their development,
and some other poor sap later on is stuck with
porting their code.  Repackaging the whole thing
might not be feasible, so patching places where
problems crop up using #ifdefs is expedient.
(More work for the *next* poor sap.)  It would be
nice if everybody had the experience and foresight
to design his code properly in the first place,
with all system dependencies isolated modularly,
but in the real world that seldom seems to happen.

Re: [9fans] Installing the updates

[rob pike]

> It would be nice if Pike could present a compelling argument.  His web
> page states only:

[The `web page' is just a very old class handout that propagated
far beyond its intended audience.]

My long note earlier today was supposed to explain better.

Here's a short version:
	Avoid complexity.
Flat structures are simpler than nested ones: simpler to write
and, more important, simpler to maintain.

If that doesn't compel you, I have nothing more to say.

As I said, in our world, our rules work well.  We cope with
outside software by following outside rules.  I do not expect
the world to switch to our way of handling #includes; I merely
ask that people writing code for Plan 9 try the Plan 9 way.

-rob

Re: [9fans] Installing the updates

[Steve Simon]

rob pike wrote:
> 
> > It would be nice if Pike could present a compelling argument.  His web
> > page states only:
> 
> [The `web page' is just a very old class handout that propagated
> far beyond its intended audience.]
> 
[snip]

Hi,

Please don't belittle your handout.

I found it a revelation, I honestly think it is the best document
on programming in C I have ever seen - the Principals of Programming
Practice says other things and explains in more depth but your handout
is
still the "essence".

-Steve

Re: [9fans] Installing the updates

[rob pike]

> I think he's heard the argument but doesn't agree.
> 
>     "Simple rule: include files should never include include files."
>         -- http://www.lysator.liu.se/c/pikestyle.html
> 

Yup.  That little note, written for a CS class in 1987, has been
expanded into a book, published last year, called The Practice
of Programming.

-rob

Re: [9fans] Installing the updates

[Ralph Corderoy]

Hi,

> Yup.  That little note, written for a CS class in 1987, has been
> expanded into a book, published last year, called The Practice of
> Programming.

For those that haven't already bought and read it see

    http://cm.bell-labs.com/cm/cs/tpop/index.html

and then go and buy and read it.


Ralph.

Re: [9fans] Installing the updates

[Russ Cox]

	Where's the horror here?  Computers are fast.  Pushing extra work on
	programmers and creating an unnecessary portability issue is a high
	cost.  Reading a header file five or more times during compilation is
	a low cost (and one which can be optimized away for ifdef-protected
	headers; I'm told gcc does so).

While it may be simpler when it works, when
it fails it does so in mysterious ways.  I don't know 
how many times I've tried to figure out why 
some header file I wanted wasn't getting included,
only to find that it had _already_ been included,
by someone else, with different things #defined,
so the definition or prototype_I_ wanted wasn't
there.

I'd much rather have the compiler barf on a 
#define or something like that, than hop over
the whole file as though it weren't there.

Russ

Re: [9fans] Installing the updates

[Bruce G. Stewart]

Russ Cox wrote:
> 
>         Where's the horror here?  Computers are fast.  Pushing extra work on
>         programmers and creating an unnecessary portability issue is a high
>         cost.  Reading a header file five or more times during compilation is
>         a low cost (and one which can be optimized away for ifdef-protected
>         headers; I'm told gcc does so).
> 
> While it may be simpler when it works, when
> it fails it does so in mysterious ways.  I don't know
> how many times I've tried to figure out why
> some header file I wanted wasn't getting included,
> only to find that it had _already_ been included,
> by someone else, with different things #defined,
> so the definition or prototype_I_ wanted wasn't
> there.
> 
> I'd much rather have the compiler barf on a
> #define or something like that, than hop over
> the whole file as though it weren't there.
> 
> Russ

I think nested includes also lead to excessive factoring - there is a
folklore that says small include files are better because fewer things
need to be recompiled when a header is changed.

More includes makes for more interdependencies, so one resorts to
nesting to reduce the amount of typing. Then one needs a tool to find
all the dependencies to create the mkfile. One wants to run it whenever
a header changes in case some new nested dependency has been introduced.
Perhaps a mkfile to mk your mkfile is the answer.. and on it goes.

I also find the one library / one h principal attractive. I'd rather
have the compiler wasting its time scanning one behemoth text than
scanning a lot of little files looking for #endif.

Re: [9fans] Installing the updates

[forsyth]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

>>We already know that it is non-trivial to port much
>>non-Plan9 code to Plan9-native and vice versa.

i have found porting well written Unix
code to native plan 9 (libc.h, 8[cl] etc)
to be straightforward.  the main exceptions
are socket calls but since those involve deleting
lots of ugly lines of code, it's satisfying, and since
they mainly involve call setup with the I/O the
same read/write after that, it's not too hard.
the compilers sometimes pick up
old problems that were lurking though.


[-- Attachment #2: Type: message/rfc822, Size: 1891 bytes --]

To: cse.psu.edu!9fans
Subject: Re: [9fans] Installing the updates
Date: Wed, 2 Aug 2000 09:11:37 GMT
Message-ID: <3987E345.4B8F3643@home.com>

> The APE headers do indeed conform.

Okay, problem solved.  The Plan9 native headers can
certainly do whatever Rob and crew think is best.
We already know that it is non-trivial to port much
non-Plan9 code to Plan9-native and vice versa.

Re: [9fans] Installing the updates

[Boyd Roberts]

From: <forsyth@vitanuova.com>
> 
> i have found porting well written Unix
> code to native plan 9 (libc.h, 8[cl] etc)

so have i.  in two days with sam i very roughly ported
my MH like user agent.  of, course i cheated with the
odd hack define.

the worst part was getting the mailbox locking right
and those damn morF's!