Re: [9fans] Install from CD fails

Re: [9fans] Install from CD fails

[erik quanstrom]

On Wed Apr 19 20:05:43 CDT 2006, rog@vitanuova.com wrote:
> i had es in mind when i wrote the inferno shell (vita nuova's, not the original tiny shell).

the inferno shell is quite nice.  good job.

> i thought then that some of the es concepts were nice, but the whole complete functional language
> thing was overkill and didn't get one very far (why make it possible to redefine
> core shell concepts such as the pipe? - it just confuses everyone if you make use of this.)

for starters, it makes a nice paper.  ;-)   also it was used for the es debugger.  i never found it confusing
(due to lexical bindings) but never made use of it, either.

> for instance:
> > 	; if {condition1} {body1} {condition2} {body2}
>
> is not implemented by the core inferno shell, but by an externally loaded
> module (std).
>

pretty nice!

>
> the main problem is that a shell of this style
> has no concept of storage of any item but a string (ok, a list of strings), so it is not
> possible to manipulate other data items directly. [...]
>
> i don't think it's possible to address these problems without
> breaking the simplicity of the whole thing; the moment you
> introduce lexical binding, differently typed variables, etc,
> a whole raft of other issues starts to drift into view.
> another kind of language might begin to help, but that's another
> story.

i think you're right.  i wrote a shell in college that was similar to rc, but
lists were allowed to contain lists.  even this proved to be unworkable, as
"for(i in *.c *.h) echo $i" would print
	a.c b.c
	x.h y.h z.h
and not what was expected.

>
> just having shell blocks as values is a big win, in my book.
> i don't think it would be hard to do this in rc.

i don't think it would be hard, but it would break td's basic design of compiling
everything at input time.  one way to thinking of es' rewriting is that it is
like rc's bytecode exposed.  i don't think it really hangs together though.  is this
	a | b
or this
	%pipe {a} 0 1 {b}
es code?

also, what does a traditional rc function do with a shell block?

>
> > a trivial grep module could make
> >	; grep {^fn} *.[ch]
> > acceptable syntax.
>
> i'm sorry, i don't see why this is preferable to the original.
> it has the same number of characters.

it doesn't!  i didn't intend to imply that it did.  it was a trivial example.

- erik

Re: [9fans] Install from CD fails

[Charles Forsyth]

> also, what does a traditional rc function do with a shell block?

invoke it.  i did something in a variant of 7th edition shell that was roughly similar.
i remember!  i converted {block} as arguments into shell functions exported to the environment
(having added 8th edition style shell functions and exported shell functions to 7th edition shell).
the shell replaced the {} block by the function name before calling the command.
thus allowing
	find usual-find-syntax -exec {mv $file /n/distant/$file}
where the modified find put the current name in the environment $file, removing the need for {} and the \;
find saw ... -exec my-exported-fn-name, and execvp would invoke /bin/sh -c my-exported-fn-name.
	time {a | pipe | line}; nohup {another; sequence; of; commands}
this relied on the environment not being shared, of course
and having the source to 7th edition.

[9fans] [OT] URI

[Charles Forsyth]

it's a pity there isn't something similar to the loader modification
recently discussed, but for eliminating the chaff from RFCs
and other standards.  for instance, RFC3986 (URI generic syntax)
defines the syntax of `uniform resource identifiers' and they
are so `uniform' that it takes a mere 56 pages excluding index to do it.
and that's just the framework!  perhaps! well, you've got to laugh;
especially when you start reading:

	For most systems, an unreserved character appearing within a URI
	component is interpreted as representing the data octet corresponding
	to that character's encoding in US-ASCII.  Consumers of URIs assume
	that the letter "X" corresponds to the octet "01011000", and even
	when that assumption is incorrect, there is no harm in making it.   [p. 15]

Re: [9fans] [OT] URI

[geoff]

An interesting idea, especially if you could tell it what to cull as
absolutely uninteresting, e.g., no OSI, no ATM, no SONET, no MPLS, no
SNMP, no RSVP, no OSPF, no XML, no UTF-7.  You'd need to have some way
to evaluate questions like `is the anal-retention and micromanagement
in SMTP (821/2821) irrelevant and thus removable?'.  I suppose any
algorithm that could cope with the Book of Armaments should be good
enough.

Re: [9fans] [OT] URI

[Brantley Coile]

There should be a service that sells cliff notes for standards.  They read
the standard and write a new one that states what we need to know to
write a program using the standard in much fewer lines.  I'd pay for such
a service.

> An interesting idea, especially if you could tell it what to cull as
> absolutely uninteresting, e.g., no OSI, no ATM, no SONET, no MPLS, no
> SNMP, no RSVP, no OSPF, no XML, no UTF-7.  You'd need to have some way
> to evaluate questions like `is the anal-retention and micromanagement
> in SMTP (821/2821) irrelevant and thus removable?'.  I suppose any
> algorithm that could cope with the Book of Armaments should be good
> enough.

Re: [9fans] [OT] URI

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

O'Reilly.

And the quality varies just as much as Cliff's notes.

Paul

On 20-Apr-06, at 8:01 AM, Brantley Coile wrote:

> There should be a service that sells cliff notes for standards.
> They read
> the standard and write a new one that states what we need to know to
> write a program using the standard in much fewer lines.  I'd pay
> for such
> a service.
>
>> An interesting idea, especially if you could tell it what to cull as
>> absolutely uninteresting, e.g., no OSI, no ATM, no SONET, no MPLS, no
>> SNMP, no RSVP, no OSPF, no XML, no UTF-7.  You'd need to have some
>> way
>> to evaluate questions like `is the anal-retention and micromanagement
>> in SMTP (821/2821) irrelevant and thus removable?'.  I suppose any
>> algorithm that could cope with the Book of Armaments should be good
>> enough.
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFER6l7pJeHo/Fbu1wRAhdyAKDhJY/TMnsKnHO+lnD+vCVLX98eTACdFXs9
aVJvHzMpODLMoEt5mDV42P0=
=Klk7
-----END PGP SIGNATURE-----

Re: [9fans] [OT] URI

[Brantley Coile]

But O'Reilly books are bigger than the rfc's.  I want less to read.

As an aside, have you noticed that Prentice Hall is printing
K&R on thicker paper to make the book look larger?  Seems too
many people don't judge a book by it's cover, but by the pound.

> O'Reilly.
>
> And the quality varies just as much as Cliff's notes.
>
> Paul

Re: [9fans] [OT] URI

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Many of them are bigger.  Some of them are actually smaller.  In a
prior life I had to care about .NET, and the O'Reilly books were the
only way to navigate the runtime environment.  And their VI book is
still the best...

It's sad how few computer (as opposed to computing science) books are
happy addressing an expert audience; writing anything that assumes
programming skills and intelligence would probably reduce the
audience to much less than a printing :-(

Paul

On 20-Apr-06, at 8:49 AM, Brantley Coile wrote:

> But O'Reilly books are bigger than the rfc's.  I want less to read.
>
> As an aside, have you noticed that Prentice Hall is printing
> K&R on thicker paper to make the book look larger?  Seems too
> many people don't judge a book by it's cover, but by the pound.
>
>> O'Reilly.
>>
>> And the quality varies just as much as Cliff's notes.
>>
>> Paul
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFER7EipJeHo/Fbu1wRAiTkAJwJwDsuvLJyjDluD3Kyl01oHVB3tQCgun8L
pKG8enKrsqfB42XYIEGrAxA=
=rr3/
-----END PGP SIGNATURE-----

Re: [9fans] [OT] URI

[Jack Johnson]

On 4/20/06, Paul Lalonde <plalonde@telus.net> wrote:
> It's sad how few computer (as opposed to computing science) books are
> happy addressing an expert audience; writing anything that assumes
> programming skills and intelligence would probably reduce the
> audience to much less than a printing :-(

I would disagree with that assessment.

As neither an expert nor a novice, I find that most texts are at the
extremes.  It's either iPod Syncing for Dummies or Implementing
IKE-DNS Interfaces using Visual Studio, with a dearth of titles that
assume some middling knowledge by the reader.  It's either 1200 pages
of hand-holding or some tome written for some poor schmuck who has to
read it to keep his job tending the same Oracle DB day after day.

Rarely do you see From Smalltalk to Ruby, Plain Old C for the C++
Brainwashed, Your First Device Driver, or Understanding the
Model-Viewer-Controller Model.  Try finding an introduction to
assembly that assumes you haven't already done it before.  It's easy
to find architecture-specific references, but the general theory stuff
is just gone.  Either you lived through it, you went to school for it,
or you're on your own.  I'd like to see guides for the not-yet-lost.

The Practice of Programming is probably the best recent book for the
neither-drooler-nor-brainiac crowd.  I find myself picking up stuff
that's out of print, just for the perspective: Software Tools,
Starting FORTH, Project Oberon, etc.  I've heard good things about the
How to Think Like a Computer Scientist variations, but haven't read
them.

-Jack

Re: [9fans] [OT] URI

[Moritz Kiese]

On Thu, 20 Apr 2006, Jack Johnson wrote:

> On 4/20/06, Paul Lalonde <plalonde@telus.net> wrote:

[snip: NEWS: good books are rare]

> Rarely do you see From Smalltalk to Ruby, Plain Old C for the C++
> Brainwashed, Your First Device Driver, or Understanding the
> Model-Viewer-Controller Model.  Try finding an introduction to
> assembly that assumes you haven't already done it before.  It's easy
> to find architecture-specific references, but the general theory stuff
> is just gone.  Either you lived through it, you went to school for it,
> or you're on your own.  I'd like to see guides for the not-yet-lost.

I found
Computer Organization and Design. The Hardware/Software Interface. by
David A. Patterson, John L. Hennessy
combined with a (M)MIX intro by some German guys (don't have it on my
office shelf) a viable approach for the assembly part. I agree on the rest
though.

++mbk

Re: [9fans] [OT] URI

[Brantley Coile]

> I found
> Computer Organization and Design. The Hardware/Software Interface. by
> David A. Patterson, John L. Hennessy

The two books by Patterson and Hennessy are just a couple of exceptions
to the `large id bad' generalizations, not that you mention them.  I even
find one or the other of them in my local Borders book store.

Re: [9fans] [OT] URI

[quanstro]

this isn't unique to computer science.

the content of calculus textbooks is inversely proportional to their
length.

- erik

On Thu Apr 20 10:53:08 CDT 2006, brantley@coraid.com wrote:
> But O'Reilly books are bigger than the rfc's.  I want less to read.
>
> As an aside, have you noticed that Prentice Hall is printing
> K&R on thicker paper to make the book look larger?  Seems too
> many people don't judge a book by it's cover, but by the pound.
>
> > O'Reilly.
> >
> > And the quality varies just as much as Cliff's notes.
> >
> > Paul
>

1

Re: [9fans] Install from CD fails

[erik quanstrom]

the fact that the unix environment has gotten out of control had
made things that should be simple (like man), quite difficult.

unfortunately, that's a different problem.  my main issue with
tcl is this:

	; man -k tcl|wc -l
	7191

that's just a wee bit out of control, no?

- erik

On Mon Apr 24 21:03:56 CDT 2006, rvs@sun.com wrote:
> > i'm probablly on my own in this, but tcl is just strange, and its
> > linux implementation never ceases to annoy, and it's way too big.
> > since when is \t anything other than whitespace?  (python, unfortunately, did
> > the same thing).  when i try to "man -k" on a system with tcl,
> > i get a hundred of useless tcl/tk commands.
>
>   Yeap. In fact, I still don't understand why $ man foo/bar doesn't do
>   what I perceive to be the right thing...
>
> Thanks,
> Roman.

Re: [9fans] Install from CD fails

[erik quanstrom]

that mkfile is way too big.  how are we going to fit it on these new 750GB drives
(http://hardware.slashdot.org/hardware/06/04/20/2358213.shtml)?

- erik

On Fri Apr 21 10:48:02 CDT 2006, 9nut@9netics.com wrote:
> > skip had a pre-release a year ago.
>
> cpu% ls -l /usr/brucee/inferno/appl
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/cmd
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/dispatch
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/funnel
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/journal
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/legacy
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/lib
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/mash
> --rw-rw-r-- M 30904 brucee brucee 106 May 14  2005 /usr/brucee/inferno/appl/mkfile
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/stuff
> d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/xml
> cpu%
>

Re: [9fans] Install from CD fails

[erik quanstrom]

perhaps you did things more elegantly than i.  when i wrote my shell back when,
getting this kind of stuff to work took quite a bit of effort

	fn f{
		if($1)
			$2
	}

	f {~ 1 1} {echo fu}

- erik

On Thu Apr 20 01:18:42 CDT 2006, forsyth@terzarima.net wrote:
> > also, what does a traditional rc function do with a shell block?
>
> invoke it.  i did something in a variant of 7th edition shell that was roughly similar.
> i remember!  i converted {block} as arguments into shell functions exported to the environment
> (having added 8th edition style shell functions and exported shell functions to 7th edition shell).
> the shell replaced the {} block by the function name before calling the command.
> thus allowing
> 	find usual-find-syntax -exec {mv $file /n/distant/$file}
> where the modified find put the current name in the environment $file, removing the need for {} and the \;
> find saw ... -exec my-exported-fn-name, and execvp would invoke /bin/sh -c my-exported-fn-name.
> 	time {a | pipe | line}; nohup {another; sequence; of; commands}
> this relied on the environment not being shared, of course
> and having the source to 7th edition.
>

Re: [9fans] Install from CD fails

[erik quanstrom]

On Wed Apr 19 23:03:19 CDT 2006, rvs@sun.com wrote:
>   I think Tcl has struck a nice balance here -- everything is a command
>   taking a number of arguments. Or are you talking about a different
>   sort of interface ?

i mean the dynamic module interface.

>
> > 2. what hooks are provided by the shell.  the "es" shell provided a hook
> > for darn near every language construct there was.  for example, it was possible
> > to redefine globbing, piping, the if statement, etc.
>
>   That is a bit too much, in my oppinion. I think what the 'core' shell
>   is supposed to do is provide a nice glue for the rest of the dynamic
>   functionality. Sort of like file system is a universal glue for every
>   other application running on the system. I would say that this is the
>   most challenging aspect of designing my dream 'shell' -- the glue is
>   supposed to be easy enough for me to understand, yet powerful enough
>   to express simple things in simple terms. Tcl comes pretty close
>   to being that glue.

give it a try:
	ftp://ftp.sys.toronto.edu/pub/es/es-0.9-beta1.tar.gz
it is very interesting, if not completely successful.

i'm probablly on my own in this, but tcl is just strange, and its
linux implementation never ceases to annoy, and it's way too big.
since when is \t anything other than whitespace?  (python, unfortunately, did
the same thing).  when i try to "man -k" on a system with tcl,
i get a hundred of useless tcl/tk commands.

- erik

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/20/06, erik quanstrom <quanstro@quanstro.net> wrote:
> On Wed Apr 19 23:03:19 CDT 2006, rvs@sun.com wrote:
> >   I think Tcl has struck a nice balance here -- everything is a command
> >   taking a number of arguments. Or are you talking about a different
> >   sort of interface ?
>
> i mean the dynamic module interface.
>
> >
> > > 2. what hooks are provided by the shell.  the "es" shell provided a hook
> > > for darn near every language construct there was.  for example, it was possible
> > > to redefine globbing, piping, the if statement, etc.
> >
> >   That is a bit too much, in my oppinion. I think what the 'core' shell
> >   is supposed to do is provide a nice glue for the rest of the dynamic
> >   functionality. Sort of like file system is a universal glue for every
> >   other application running on the system. I would say that this is the
> >   most challenging aspect of designing my dream 'shell' -- the glue is
> >   supposed to be easy enough for me to understand, yet powerful enough
> >   to express simple things in simple terms. Tcl comes pretty close
> >   to being that glue.
>
> give it a try:
>         ftp://ftp.sys.toronto.edu/pub/es/es-0.9-beta1.tar.gz
> it is very interesting, if not completely successful.
>
> i'm probablly on my own in this, but tcl is just strange, and its
> linux implementation never ceases to annoy, and it's way too big.
> since when is \t anything other than whitespace?  (python, unfortunately, did
> the same thing).  when i try to "man -k" on a system with tcl,
> i get a hundred of useless tcl/tk commands.
>

Nah, I'm not a fan of Tcl either really...

Re: [9fans] Install from CD fails

[Skip Tavakkolian]

> i'm probablly on my own in this, but tcl is just strange

you're not alone.

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Thu, Apr 20, 2006 at 05:30:12AM -0500, erik quanstrom wrote:
> On Wed Apr 19 23:03:19 CDT 2006, rvs@sun.com wrote:
> >   I think Tcl has struck a nice balance here -- everything is a command
> >   taking a number of arguments. Or are you talking about a different
> >   sort of interface ?
>
> i mean the dynamic module interface.
>
> >
> > > 2. what hooks are provided by the shell.  the "es" shell provided a hook
> > > for darn near every language construct there was.  for example, it was possible
> > > to redefine globbing, piping, the if statement, etc.
> >
> >   That is a bit too much, in my oppinion. I think what the 'core' shell
> >   is supposed to do is provide a nice glue for the rest of the dynamic
> >   functionality. Sort of like file system is a universal glue for every
> >   other application running on the system. I would say that this is the
> >   most challenging aspect of designing my dream 'shell' -- the glue is
> >   supposed to be easy enough for me to understand, yet powerful enough
> >   to express simple things in simple terms. Tcl comes pretty close
> >   to being that glue.
>
> give it a try:
> 	ftp://ftp.sys.toronto.edu/pub/es/es-0.9-beta1.tar.gz
> it is very interesting, if not completely successful.

 I most certainly will. Thanks a lot.

> i'm probablly on my own in this, but tcl is just strange, and its
> linux implementation never ceases to annoy, and it's way too big.
> since when is \t anything other than whitespace?  (python, unfortunately, did
> the same thing).  when i try to "man -k" on a system with tcl,
> i get a hundred of useless tcl/tk commands.

  Yeap. In fact, I still don't understand why $ man foo/bar doesn't do
  what I perceive to be the right thing...

Thanks,
Roman.

Re: [9fans] Install from CD fails

[erik quanstrom]

es scans the bit inside the {} as a string and does not parse it until it is used.

- erik

On Thu Apr 20 01:09:02 CDT 2006, forsyth@terzarima.net wrote:
> >> es did this: anything passed to a function inside curly braces was passed verbatim.
> >
> > that was the main bit of es that i stole for the inferno shell. it works well,
> > and combined with dynamically loaded shell modules, makes many nice things quite
> > easy.
>
> i thought that in the inferno sh the bit inside the {} was parsed, and later converted
> back to a string if passed to a command (uniform treatment of {} as in `{}, and you
> discover your syntax errors in good time).  i don't remember what es did.
>

Re: [9fans] Install from CD fails

[R]

On 4/20/06, erik quanstrom <quanstro@quanstro.net> wrote:
> es scans the bit inside the {} as a string and does not parse it until it is used.

no.

body    : cmd                   { $$ = $1; }
        | cmdsan body           { $$ = mkseq("%seq", $1, $2); }
...
comword : param                         { $$ = $1; }
        | '(' nlwords ')'               { $$ = $2; }
        | '{' body '}'                  { $$ = thunkify($2); }

Re: [9fans] Install from CD fails

[erik quanstrom]

i believe it's inferno's answer to mk.

http://www.vitanuova.com/inferno/man/1/mash.html

On Wed Apr 19 20:08:23 CDT 2006, rvs@sun.com wrote:
> On Thu, Apr 20, 2006 at 05:42:15AM +1000, Bruce Ellis wrote:
> > that is how mash works ... you augment it's command set by loading
> > modules (for example "make").
>
>   what's mash ?
>
> Thanks,
> Roman.
>
> P.S. And yes, I've tried google ;-)

Re: [9fans] Install from CD fails

[Michael Baldwin]

> [re: mash] i believe it's inferno's answer to mk.

I believe that now Inferno's answer to mk is mk (in Limbo).  Mash
never quite made it as a build tool in Genuine Inferno, from what I
could tell.  Of course, in the mythical Oz species, mash probably
does everything, and more.  I'm curious: what are the actual build
tools used today at Vita Nuova, in the Heart of Inferno: Inferno mk?
Plan 9 mk? mash? nmake?

Re: [9fans] Install from CD fails

[Charles Forsyth]

>I'm curious: what are the actual build
>tools used today at Vita Nuova, in the Heart of Inferno: Inferno mk?
>Plan 9 mk? mash? nmake?

mk.  probably the main problem with mash was that for various
reasons we lost contact with its source from the start (nothing to do
with either us or brucee, or oz, or with mash for that matter).
nmake?  ha ha, ha ha ha ha. it's the way you tell them!

i don't think any of them is quite right, though, but mk
survives and i've not really fleshed out anything better.
possibly thinking about dependencies, variants, etc etc
might help do that, but more likely it will just make me
feel a little bored.

Re: [9fans] Install from CD fails

[Bruce Ellis]

i take offense at the term mythical.  skip had a pre-release a year ago.
there is a more up to date version on boyd's laptop, whoever has it.
i gave a demo to ems in the last month.

put your $ down ... bet me.

brucee

On 4/20/06, Michael Baldwin <michael@cibernet.com> wrote:
> > [re: mash] i believe it's inferno's answer to mk.
>
> I believe that now Inferno's answer to mk is mk (in Limbo).  Mash
> never quite made it as a build tool in Genuine Inferno, from what I
> could tell.  Of course, in the mythical Oz species, mash probably
> does everything, and more.  I'm curious: what are the actual build
> tools used today at Vita Nuova, in the Heart of Inferno: Inferno mk?
> Plan 9 mk? mash? nmake?

Re: [9fans] Install from CD fails

[Skip Tavakkolian]

> skip had a pre-release a year ago.

cpu% ls -l /usr/brucee/inferno/appl
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/cmd
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/dispatch
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/funnel
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/journal
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/legacy
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/lib
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/mash
--rw-rw-r-- M 30904 brucee brucee 106 May 14  2005 /usr/brucee/inferno/appl/mkfile
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/stuff
d-rwxrwxr-x M 30904 brucee brucee   0 May 14  2005 /usr/brucee/inferno/appl/xml
cpu%

Re: [9fans] Install from CD fails

[Jack Johnson]

On 4/21/06, Bruce Ellis <bruce.ellis@gmail.com> wrote:
> i take offense at the term mythical.

I understand your offense, but look at it this way: only a handful of
people can claim to have seen architeuthis, but we tend to believe
that architeuthis exists.  The rest of us we have to rely on anecdote.
That reliance on anecdote makes it mythical, but we still believe it
exists.

Kind of like Natalie Portman's bedroom....

I heard Teller (of Penn & Teller) on the radio a couple of months
back, but I've never actually seen him speak.  I still wonder if it's
just a brilliantly subduded joke:

http://www.npr.org/templates/story/story.php?storyId=4750603

Maybe Rob has seen his lips move.

-Jack

Re: [9fans] Install from CD fails

[YAMANASHI Takeshi]

> the boundary is a bit blurred on inferno,
> because the explicitly module loading there
> is most commonly used to load what on other
> systems would be libraries.

I also have an feeling that the boundary between
library call and exec'ing is blurred on inferno.
--

Re: [9fans] Install from CD fails

[Mike Haertel]

>I once knew
>someone who thought that a context switch was less expensive using
>shared libraries than without.

That could actually be true in some situations, for the following reason:
if program A and B have small unshared amounts of memory, but are sharing
many megabytes of libraries, potentially there are far fewer cache misses
after a context switches than there would be in the case that A and B
were each statically linked with their own copies of the many megabytes
of libraries.

However, some of this benefit is eaten by TLB misses: dynamically
linked programs tend to have much more total active virtual address
space than the corresponding statically linked programs that link only
the code they need.  And TLBs usually get flushed by context switches.
So some of the savings gained by not thrashing things in and out of the
cache is lost to thrashing things in and out of the TLBs.  TLB misses
are usually more expensive than cache misses, since a single TLB miss
can spawn many cache misses.

Re: [9fans] Install from CD fails

[erik quanstrom]

i think this is a case of robust agreement;
russ is just much better explaining himself than i.

- erik

On Tue Apr 18 21:53:00 CDT 2006, rsc@swtch.com wrote:
> > dynamic linking allows one to extend a program without inventing a metalanguage.
> > i believe there is a paper on how inferno's shell uses this to nice effect.
>
> shared libraries != dynamic loading of modules.
> both require a dynamic linking implementation
> but they are not at all the same.
>
> shared libraries are just a bad replacement
> for static libraries.  they're used implicitly without
> a program having to ask for anything, and there
> is never an appropriate situation in which
> to use them.
>
> dynamic loading of modules can be a very
> powerful method of extension.  i have been
> meaning for a long time to convert snoopy to
> make the protocol parsers dynamically loaded
> instead of having one huge binary.  the inferno
> shell is another good example.
>
> the boundary is a bit blurred on inferno,
> because the explicitly module loading there
> is most commonly used to load what on other
> systems would be libraries.  but the result,
> at least as implemented, has a very different
> feel from the shared library hell on unix
> and windows.
>
> russ
>

Re: [9fans] Install from CD fails

[erik quanstrom]

i think the right solution to this problem is to run a password server at
a known "address" that is independent of namespace.  only the c library needs
to know about this. (there are other unix traditions than can be kept in a world
were normal users can mount and bind this way.)


- erik

On Tue Apr 18 14:53:50 CDT 2006, leimy2k@gmail.com wrote:
> On 4/18/06, Russ Cox <rsc@swtch.com> wrote:
> > > A masking bind over /etc/passwd could be disasterous
> > > on Unix and I don't think anyone has really solved this problem yet
> >
> > this is trivial to solve.  setuid binaries should run
> > in the default system name space instead of inheriting
> > the one in use where they are started.
> >
> > russ
>
> Hmmm, anyone spoken to Al Viro about this?

Re: [9fans] Install from CD fails

[erik quanstrom]

i think there is a #3 here.  extension.

dynamic linking allows one to extend a program without inventing a metalanguage.
i believe there is a paper on how inferno's shell uses this to nice effect.

- erik

On Tue Apr 18 16:56:40 CDT 2006, leimy2k@gmail.com wrote:
> >   And you would have to go through all of the aforementioned troubles
> >   to achieve exactly what ? What is it, that shared libraries are good
> >   at ?
>
> #1 Maintenance - If you have 50 programs that depend on one library
> and you have a fix for the library how many things do you want to
> "remember to build"?  (though people are throwing up straw man
> arguments for this too.  I suspect the worst case scenario is not
> always the common case though.)
>
> #2 Supposed physical memory savings - libSystem on Mac OS X only
> exists in memory 1 time for all the programs that use it... sorta.
> Read Only pages are shared, writable pages are COW and yes, this adds
> a good deal of complexity to the VM of the OS to have this.
>

Re: [9fans] Install from CD fails

[Russ Cox]

> dynamic linking allows one to extend a program without inventing a metalanguage.
> i believe there is a paper on how inferno's shell uses this to nice effect.

shared libraries != dynamic loading of modules.
both require a dynamic linking implementation
but they are not at all the same.

shared libraries are just a bad replacement
for static libraries.  they're used implicitly without
a program having to ask for anything, and there
is never an appropriate situation in which
to use them.

dynamic loading of modules can be a very
powerful method of extension.  i have been
meaning for a long time to convert snoopy to
make the protocol parsers dynamically loaded
instead of having one huge binary.  the inferno
shell is another good example.

the boundary is a bit blurred on inferno,
because the explicitly module loading there
is most commonly used to load what on other
systems would be libraries.  but the result,
at least as implemented, has a very different
feel from the shared library hell on unix
and windows.

russ

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 09:35:27PM -0500, erik quanstrom wrote:
> i think there is a #3 here.  extension.
>
> dynamic linking allows one to extend a program without inventing a metalanguage.
> i believe there is a paper on how inferno's shell uses this to nice effect.

  I think you're confusing two notions here. What you're talking about
  sounds more like dynamic loading, not dynamic linking. And with dynamic
  loading the control is *explicitly* at client's possession. You're
  supposed to know what to dlopen, what to look for inside, etc. Such
  a controlled environment lets you be much more precise and avoid
  many of the shortcomings of the true "dynamic linking".

  Now, it would be interesting to know what others think about the need
  for dynamic loading in Plan9.

  Personally, my dream has always been to make all of the applications
  which rely heavily on console input-output to be dynamically loaded
  on top of each other, so that when I do something like:

    $ bash
    <long session with bash, with lots of useful stuff in history and such>
    $ gdb

  I don't leave shell (and lose all of the context) but I rather have
  my shell augmented with gdb commands. Sort of like Tcl works with
  external modules.

  Have anybody thought about anything likes this ?

Thanks,
Roman.

Re: [9fans] Install from CD fails

[Bruce Ellis]

that is how mash works ... you augment it's command set by loading
modules (for example "make").

brucee

On 4/20/06, Roman Shaposhnick <rvs@sun.com> wrote:
> On Tue, Apr 18, 2006 at 09:35:27PM -0500, erik quanstrom wrote:
> > i think there is a #3 here.  extension.
> >
> > dynamic linking allows one to extend a program without inventing a metalanguage.
> > i believe there is a paper on how inferno's shell uses this to nice effect.
>
>  I think you're confusing two notions here. What you're talking about
>  sounds more like dynamic loading, not dynamic linking. And with dynamic
>  loading the control is *explicitly* at client's possession. You're
>  supposed to know what to dlopen, what to look for inside, etc. Such
>  a controlled environment lets you be much more precise and avoid
>  many of the shortcomings of the true "dynamic linking".
>
>  Now, it would be interesting to know what others think about the need
>  for dynamic loading in Plan9.
>
>  Personally, my dream has always been to make all of the applications
>  which rely heavily on console input-output to be dynamically loaded
>  on top of each other, so that when I do something like:
>
>    $ bash
>    <long session with bash, with lots of useful stuff in history and such>
>    $ gdb
>
>  I don't leave shell (and lose all of the context) but I rather have
>  my shell augmented with gdb commands. Sort of like Tcl works with
>  external modules.
>
>  Have anybody thought about anything likes this ?
>
> Thanks,
> Roman.
>

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Thu, Apr 20, 2006 at 05:42:15AM +1000, Bruce Ellis wrote:
> that is how mash works ... you augment it's command set by loading
> modules (for example "make").

  what's mash ?

Thanks,
Roman.

P.S. And yes, I've tried google ;-)

Re: [9fans] Install from CD fails

[Jack Johnson]

On 4/19/06, Roman Shaposhnick <rvs@sun.com> wrote:
>   what's mash ?

http://www.vitanuova.com/inferno/man/1/mash.html

-Jack

Re: [9fans] Install from CD fails

[Charles Forsyth]

>   Now, it would be interesting to know what others think about the need
>   for dynamic loading in Plan9.

the mechanisms are probably not much used yet outside Plan 9's emu for Inferno
but [58q]l support dynamic loading in a way that unusually provides type checking,
and structured in a similar way to Inferno's Dis modules.
?l can produce loadable modules that contain two sets of type-tagged symbols,
one set to be imported from the surrounding environment and the other to be
exported to it.  there is a small supporting library.
economically, it uses the same type checking mechanism that is used statically by ?l.
or to put it another way, the mechanism was designed to support both.

Re: [9fans] Install from CD fails

[Brantley Coile]

>   Now, it would be interesting to know what others think about the need
>   for dynamic loading in Plan9.

If you want Oberon you know where to get it?
	:)

(I like Oberon--the system and the language.)

Re: [9fans] Install from CD fails

[quanstro]

yes.  there are a couple (okay, three) problems to address here.  byron and paul haahr
addressed some of the problems with a shell called "es" years ago, even though they
didn't have dynamic modules.

1.  what interface do external modules get to see?  perhaps masquerading as
a shell function would be good enough.

2. what hooks are provided by the shell.  the "es" shell provided a hook
for darn near every language construct there was.  for example, it was possible
to redefine globbing, piping, the if statement, etc.

i thought the result was effective, but somewhat like pounding a nail with
a jackhammer. (traditional shell syntax was first converted from
	a | b
to es's perferred syntax
	%pipe {a} 0 1 {b}
then bound as
	$&pipe {a} {b},
assuming one hadn't redefined %pipe.)

3. shell syntax vs. command syntax.  the shell's syntax often gets in the way
of command syntax.  mike hartel wrote up some stuff about this years ago.
basically, it's a pain that the shell and, say, grep, want to use the same characters.
i can't just type
	; grep ^fn *.[ch]
i must type
	; grep '^fn' *.[ch]
instead.  this would get very difficult if one added awk-like functions to the shell.
es did this: anything passed to a function inside curly braces was passed verbatim.
thus "if" in es looks like a normal es function call:
	; if {condition1} {body1} {condition2} {body2}
a trivial grep module could make
	; grep {^fn} *.[ch]
acceptable syntax.

- erik


On Wed Apr 19 14:35:12 CDT 2006, rvs@sun.com wrote:
>
>   Personally, my dream has always been to make all of the applications
>   which rely heavily on console input-output to be dynamically loaded
>   on top of each other, so that when I do something like:
>
>     $ bash
>     <long session with bash, with lots of useful stuff in history and such>
>     $ gdb
>
>   I don't leave shell (and lose all of the context) but I rather have
>   my shell augmented with gdb commands. Sort of like Tcl works with
>   external modules.
>
>   Have anybody thought about anything likes this ?
>
> Thanks,
> Roman.

Re: [9fans] Install from CD fails

[rog]

> yes.  there are a couple (okay, three) problems to address here.  byron and paul haahr
> addressed some of the problems with a shell called "es" years ago, even though they
> didn't have dynamic modules.

i had es in mind when i wrote the inferno shell (vita nuova's, not the original tiny shell).
i thought then that some of the es concepts were nice, but the whole complete functional language
thing was overkill and didn't get one very far (why make it possible to redefine
core shell concepts such as the pipe? - it just confuses everyone if you make use of this.)

inferno made it very easy to provide "internal" modules (which i called "builtins";
i never really thought of a satisfactory name) - the interface they get to
see is a restricted version of the interface seen by the shell itself internally.
i tried to keep this as clean as possible. (by contrast, mash, which did a similar
thing, had quite a complex interface at this level).

> es did this: anything passed to a function inside curly braces was passed verbatim.

that was the main bit of es that i stole for the inferno shell. it works well,
and combined with dynamically loaded shell modules, makes many nice things quite
easy.

for instance:
> 	; if {condition1} {body1} {condition2} {body2}

is not implemented by the core inferno shell, but by an externally loaded
module (std).

since then, shell modules have been implemented for all kinds of things (e.g. graphics,
s-expression parsing, mash-like make).
externally implemented programs are still preferable (smaller interface,
the shell doesn't break if the program does), for interfaces which
need to maintain state across calls or wish to call back into the shell,
a shell module can work well. it also makes it easy (and efficient)
for a function in such a module to return a list of values (the fundamental
type).

the main problem is that a shell of this style
has no concept of storage of any item but a string (ok, a list of strings), so it is not
possible to manipulate other data items directly. also, the interface makes
it quite natural to pass code fragments to external programs to execute,
e.g.
	listen 'tcp!*!6666' {export /&}
but the external programs don't have access to the original shell's
environment (e.g. loaded shell modules).

i don't think it's possible to address these problems without
breaking the simplicity of the whole thing; the moment you
introduce lexical binding, differently typed variables, etc,
a whole raft of other issues starts to drift into view.
another kind of language might begin to help, but that's another
story.

just having shell blocks as values is a big win, in my book.
i don't think it would be hard to do this in rc.

> a trivial grep module could make
>	; grep {^fn} *.[ch]
> acceptable syntax.

i'm sorry, i don't see why this is preferable to the original.
it has the same number of characters.

Re: [9fans] Install from CD fails

[Charles Forsyth]

>> es did this: anything passed to a function inside curly braces was passed verbatim.
>
> that was the main bit of es that i stole for the inferno shell. it works well,
> and combined with dynamically loaded shell modules, makes many nice things quite
> easy.

i thought that in the inferno sh the bit inside the {} was parsed, and later converted
back to a string if passed to a command (uniform treatment of {} as in `{}, and you
discover your syntax errors in good time).  i don't remember what es did.

Re: [9fans] Install from CD fails

[rog]

> i thought that in the inferno sh the bit inside the {} was parsed, and later converted
> back to a string if passed to a command (uniform treatment of {} as in `{}, and you
> discover your syntax errors in good time).  i don't remember what es did.

that's true, but es did exactly the same thing. the internal representation is the
shell's syntax tree (which means that evaluation of sub-blocks is reasonably efficient,
unlike tcl, which has to rescan the string each time through), but the string representation
should generate exactly the same tree when parsed, so for most practical purposes,
it can be treated as if it *was* verbatim (excess white space is removed, as are comments).

i could have used a similar approach to your v7 "export a function containing the code", but
it's nice to be able to pass code fragments around to unrelated processes, for instance
to remotely execute some code on a file server.

the other thing the inferno shell allows, which i thought quite neat at the time,
is that a running program can load itself as a module into an instance of the shell,
and define shell "builtins" that call back into the program, thus giving scriptability
at little cost. for instance, the window manager does this for its initialisation script:
the "menu" primitive adds a new item to the window manager menu (its argument is,
of course, a shell command to run when the item is invoked).

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Wed, Apr 19, 2006 at 04:46:47PM -0500, quanstro@quanstro.net wrote:
> yes.  there are a couple (okay, three) problems to address here.  byron and paul haahr
> addressed some of the problems with a shell called "es" years ago, even though they
> didn't have dynamic modules.
>
> 1.  what interface do external modules get to see?  perhaps masquerading as
> a shell function would be good enough.

  I think Tcl has struck a nice balance here -- everything is a command
  taking a number of arguments. Or are you talking about a different
  sort of interface ?

> 2. what hooks are provided by the shell.  the "es" shell provided a hook
> for darn near every language construct there was.  for example, it was possible
> to redefine globbing, piping, the if statement, etc.

  That is a bit too much, in my oppinion. I think what the 'core' shell
  is supposed to do is provide a nice glue for the rest of the dynamic
  functionality. Sort of like file system is a universal glue for every
  other application running on the system. I would say that this is the
  most challenging aspect of designing my dream 'shell' -- the glue is
  supposed to be easy enough for me to understand, yet powerful enough
  to express simple things in simple terms. Tcl comes pretty close
  to being that glue.

  Any other suggestions ?

> 3. shell syntax vs. command syntax.  the shell's syntax often gets in the way
> of command syntax.

  I think this is unavoidable. But somehow, I don't really perceive it to
  be that big of a deal, anyway. Quoting is quite useful and not really
  painful. At least as long as you understand the rules involved. That's
  why I like Tcl so much -- even though you have to use extra quote or eval
  from time to time, you know exactly how everything gets parsed because
  there's only 11 rules to describe the entire language machinery.

> mike hartel wrote up some stuff about this years ago.
> basically, it's a pain that the shell and, say, grep, want to use the same characters.
> i can't just type
> 	; grep ^fn *.[ch]
> i must type
> 	; grep '^fn' *.[ch]
> instead.  this would get very difficult if one added awk-like functions to the shell.
> es did this: anything passed to a function inside curly braces was passed verbatim.
> thus "if" in es looks like a normal es function call:
> 	; if {condition1} {body1} {condition2} {body2}

   That sounds exactly like Tcl.

Thanks,
Roman.

[9fans] Install from CD fails

[Bakul Shah]

Installing from the latest CD (fetched today from tip9ug.jp)
fails in the copydist step with the following message

--------------------------------------
usage: replica/pull [-nv] [-c name] [-s name] replica-name [paths]

--------------------------------------

I took a quick look at /sys/lib/dist/pc/inst/copydist and
think the culprit is this line:

	 replica/pull -c /rc/bin/inst/replcfg

Doesn't it need an extra argument now?  I doubt it matters
but this was an install in a qemu VM.  Thanks!

Re: [9fans] Install from CD fails

[Russ Cox]

> Installing from the latest CD (fetched today from tip9ug.jp)
> fails in the copydist step with the following message

I think I fixed this.  The fix is s!-c!-c /! in the command.
There's a new CD on the Bell Labs site.

In other news, the floppy distribution doesn't fit on
a floppy anymore.

Russ

Re: [9fans] Install from CD fails

[Gorka guardiola]

I am curious about this. What has grown so much, the number of
drivers?.

On 4/18/06, Russ Cox <rsc@swtch.com> wrote:
> > Installing from the latest CD (fetched today from tip9ug.jp)
> > fails in the copydist step with the following message
>
> I think I fixed this.  The fix is s!-c!-c /! in the command.
> There's a new CD on the Bell Labs site.
>
> In other news, the floppy distribution doesn't fit on
> a floppy anymore.
>
> Russ
>
>


--
- curiosity sKilled the cat

Re: [9fans] Install from CD fails

[Russ Cox]

> I am curious about this. What has grown so much, the number of
> drivers?.

The binaries keep getting bigger.
Fossil is much bigger than kfs ever was.
It's just little by little.  I don't know for
sure but I doubt the kernel is the actual
problem.

The current floppy contains over 4MB of
root file system (mostly binaries),
compressed very carefully down to
around 700kB.  If I sat down and tweaked
the compression some more I'm sure I
could get the current binaries onto the disk,
but I don't care enough.  Maybe later.

Much of the bloat is from library routines
that don't actually get called in practice
but end up linked into every binary anyway
(like the formatters for %e, %f, and %g).

Shared libraries would help a lot here.
Maybe that should be a summer of code
project, eh, Uriel?

Russ

Re: [9fans] Install from CD fails

[Ronald G Minnich]

Russ Cox wrote:

> Shared libraries would help a lot here.

That's one option.

There were linkers written, long ago, that would extract only the code
from a library that was used by a program, and the binary got pretty
compact. You got almost as good an effect as if you had compiled the
program with the shared library source. Wonder how much this would help?

People got pretty smart back when 256KB was a lot of memory.

Shared libraries are going to need mmap() or a thing like it. Is the
seg() stuff enough like mmap()? I've never tried it out. Is it worth it?
  How messy will things get once that goes in?

ron

Re: [9fans] Install from CD fails

[Charles Forsyth]

> There were linkers written, long ago, that would extract only the code
> from a library that was used by a program, and the binary got pretty

we do that in inferno's 5l.  richard miller implemented it originally in 5l
but the changes vanished with the business unit and we redid it years later,
and perhaps not in quite the same way (i don't know).

Re: [9fans] Install from CD fails

[andrey mirtchovski]

> we do that in inferno's 5l.  richard miller implemented it originally in 5l
> but the changes vanished with the business unit and we redid it years later,
> and perhaps not in quite the same way (i don't know).

did it work out? did you see a good reduction in size?

Re: [9fans] Install from CD fails

[Bruce Ellis]

that's a great addition to the loader, and relatively easy to implement.
grafting shared libraries into plan9 will make presto go into a rage.
i think it is important to have an install floppy.  or throw out machines
without CD drives.

brucee

On 4/19/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > There were linkers written, long ago, that would extract only the code
> > from a library that was used by a program, and the binary got pretty
>
> we do that in inferno's 5l.  richard miller implemented it originally in 5l
> but the changes vanished with the business unit and we redid it years later,
> and perhaps not in quite the same way (i don't know).

Re: [9fans] Install from CD fails

[Charles Forsyth]

> grafting shared libraries into plan9 will make presto go into a rage.

he's not the only one...

Re: [9fans] Install from CD fails

[Brantley Coile]

>> grafting shared libraries into plan9 will make presto go into a rage.
>
> he's not the only one...

Just want to second that one.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > grafting shared libraries into plan9 will make presto go into a rage.
>
> he's not the only one...
>
>

Don't shared libraries also typically provide memory savings?  One
version of your c library "resident" for all VM spaces to map?

Re: [9fans] Install from CD fails

[Bruce Ellis]

i'd rather buy more memory than suffer from a shared library problem.
what version do you need? do you get wafers with it?

brucee

On 4/19/06, David Leimbach <leimy2k@gmail.com> wrote:
> On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > > grafting shared libraries into plan9 will make presto go into a rage.
> >
> > he's not the only one...
> >
> >
>
> Don't shared libraries also typically provide memory savings?  One
> version of your c library "resident" for all VM spaces to map?
>

Re: [9fans] Install from CD fails

[Francisco J Ballesteros]

And another question: will you break all your binaries just by
replacing a broken
version of the lib. with a fixed one?

Also, isn´t the kernel enough as our shared library? :-)

On 4/18/06, Bruce Ellis <bruce.ellis@gmail.com> wrote:
> i'd rather buy more memory than suffer from a shared library problem.
> what version do you need? do you get wafers with it?
>
> brucee
>
> On 4/19/06, David Leimbach <leimy2k@gmail.com> wrote:
> > On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > > > grafting shared libraries into plan9 will make presto go into a rage.
> > >
> > > he's not the only one...
> > >
> > >
> >
> > Don't shared libraries also typically provide memory savings?  One
> > version of your c library "resident" for all VM spaces to map?
> >
>
>

Re: [9fans] Install from CD fails

[Skip Tavakkolian]

> Also, isn´t the kernel enough as our shared library? :-)

that's my view too, including sharing via file servers. 

Re: [9fans] Install from CD fails

[Charles Forsyth]

>>
>> Don't shared libraries also typically provide memory savings?  One
>> version of your c library "resident" for all VM spaces to map?
>>

that's often quoted as a consequence, but in practice,
not that i've seen in ...  what is it now?  ...  at least six or
seven different systems.  i think the trouble is that to get savings
that make the pain worthwhile you still need various forms of
discipline, but with shared libraries, people are even less concerned.
and RSS continues up.

another is bug fixing at a stroke, but it also allows
bug and trapdoor introduction at a stroke.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> >>
> >> Don't shared libraries also typically provide memory savings?  One
> >> version of your c library "resident" for all VM spaces to map?
> >>
>
> that's often quoted as a consequence, but in practice,
> not that i've seen in ...  what is it now?  ...  at least six or
> seven different systems.  i think the trouble is that to get savings
> that make the pain worthwhile you still need various forms of
> discipline, but with shared libraries, people are even less concerned.
> and RSS continues up.

That's interesting to know.  I've heard a lot of the theory but have
experienced far too little of the reality of these systems (at least
in ways in which it seems to affect me).

So the reality of actually seen improvements from shared libs is really.

a) Not having to rebuild everything that's dynamically linked.
b) space savings of binary files.

b - is not such a big deal.  Hard disks are huge (I remember my 60MB
Hard disk on my old 286... I was soo impressed, it seemed like
infinity, I was 10 years old.)

That plus it sounds like in cases where it really matters, we have
good alternatives to deal with this.

a - this becomes more of a time saver and supposedly simplifies
knowing what things need to be rebuilt.  Vulnerabilities in a shared
lib are "automatically" fixed.

However, for the same reasons a is good, it's also bad.  Lots of
binaries might look for libqt.so and expect qt-1.x.  Your friendly
sysadmin may have installed libqt.so but it symlinks to libqt-4.x.

Apple tried to solve this by introducing "two-level" namespaces.  Not
only is the library linked but the path to it.  Again, the theory
sounds good but it's not convincing in practice.  The BSDs went with
variant symlinks (FreeBSD and Dragonfly will or do both have these)

The interesting thing is that Plan 9's great namespace manipulation
functionality + the fact that each process can have a private
namespace means that Plan 9 probably has the best shot at dealing with
"DLL-hell", like when 10 programs need 10 different versions of the
same shared library to run respectively.  A simple script wrapped
around the loading of a program can set up a namespace such that
ambiguities don't exist.

This is why I think "cheap" and correct private namespaces are
interesting to Unix actually.  The problem is too much is stored in
the filesystem and accessible by users who really don't need to see
certain files.  A masking bind over /etc/passwd could be disasterous
on Unix and I don't think anyone has really solved this problem yet on
Linux where it seems the only way to make private namespaces "safe" is
to allow only the root user to create a new namespace and binding
mounts.

On the surface it appears easier for Plan 9 to adopt shared libs than
it would be for Unix to do namespaces properly :).  That's really the
only reason I'm interested in shared libs really. (especially since
the claim of memory savings appears to be false).

Dave

>
> another is bug fixing at a stroke, but it also allows
> bug and trapdoor introduction at a stroke.
>
>

Re: [9fans] Install from CD fails

[lucio]

> The interesting thing is that Plan 9's great namespace manipulation
> functionality + the fact that each process can have a private
> namespace means that Plan 9 probably has the best shot at dealing with
> "DLL-hell", like when 10 programs need 10 different versions of the
> same shared library to run respectively.  A simple script wrapped
> around the loading of a program can set up a namespace such that
> ambiguities don't exist.

Not quite.  It compounds DLL hell, instead.  Consider the flaming
hoops NetBSD needs to go through to retain backwards compatibility of
all symbols that have existed since DLLs were introduced.  To the
extent that "libc" cannot have its major version number bumped.  Which
means that there is a single "libc" DLL with everything since the ark
in it.

Now add the ability for the _user_ to construct the namespace.  No,
thank you, I think there must be less painful ways to die.

What Russ may have overlooked, for the particular problem involved
here (too large for a floppy) is stripping the binaries.  Or he may
already have thought of it.  I am surprised, however, that ?l binds
the full library rather than only the essential components.  Even
early MSDOS compiler/linker combinations knew how to do this.  But I
may have misunderstood.

++L

Re: [9fans] Install from CD fails

[Russ Cox]

> A masking bind over /etc/passwd could be disasterous
> on Unix and I don't think anyone has really solved this problem yet

this is trivial to solve.  setuid binaries should run
in the default system name space instead of inheriting
the one in use where they are started.

russ

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Russ Cox <rsc@swtch.com> wrote:
> > A masking bind over /etc/passwd could be disasterous
> > on Unix and I don't think anyone has really solved this problem yet
>
> this is trivial to solve.  setuid binaries should run
> in the default system name space instead of inheriting
> the one in use where they are started.
>
> russ

Hmmm, anyone spoken to Al Viro about this?

Re: [9fans] Install from CD fails

[Latchesar Ionkov]

Another solution would be to refuse to honor setuid if the
namespace allows a user to mount arbitrary file systems.

	Lucho

On Tue, Apr 18, 2006 at 03:34:53PM -0500, Russ Cox said:
> > A masking bind over /etc/passwd could be disasterous
> > on Unix and I don't think anyone has really solved this problem yet
>
> this is trivial to solve.  setuid binaries should run
> in the default system name space instead of inheriting
> the one in use where they are started.
>
> russ

Re: [9fans] Install from CD fails

[Ronald G Minnich]

David Leimbach wrote:

> a) Not having to rebuild everything that's dynamically linked.

until gnu introduced symbol versions. binaries are now locked to a .so
build.

oh well.

ron

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Ronald G Minnich <rminnich@lanl.gov> wrote:
> David Leimbach wrote:
>
> > a) Not having to rebuild everything that's dynamically linked.
>
> until gnu introduced symbol versions. binaries are now locked to a .so
> build.
>
> oh well.

Actually symbol versioning is meant to guarantee binary compatibility
by letting you advertise that you've broken the binary interface. 
This doesn't necessarily mean that you have to break the interface..
it just means you have to do more to support it.

But I can see what you mean.  If opening the dynamic library box to
solve the 2 problems I've noted opens up 90 more cans of worms, I'm
going fishing?

Dave

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 12:22:30PM -0700, David Leimbach wrote:
> On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> The interesting thing is that Plan 9's great namespace manipulation
> functionality + the fact that each process can have a private
> namespace means that Plan 9 probably has the best shot at dealing with
> "DLL-hell", like when 10 programs need 10 different versions of the
> same shared library to run respectively.  A simple script wrapped
> around the loading of a program can set up a namespace such that
> ambiguities don't exist.

  And you would have to go through all of the aforementioned troubles
  to achieve exactly what ? What is it, that shared libraries are good
  at ?

Thanks,
Roman.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Roman Shaposhnick <rvs@sun.com> wrote:
> On Tue, Apr 18, 2006 at 12:22:30PM -0700, David Leimbach wrote:
> > On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > The interesting thing is that Plan 9's great namespace manipulation
> > functionality + the fact that each process can have a private
> > namespace means that Plan 9 probably has the best shot at dealing with
> > "DLL-hell", like when 10 programs need 10 different versions of the
> > same shared library to run respectively.  A simple script wrapped
> > around the loading of a program can set up a namespace such that
> > ambiguities don't exist.
>
>   And you would have to go through all of the aforementioned troubles
>   to achieve exactly what ? What is it, that shared libraries are good
>   at ?

#1 Maintenance - If you have 50 programs that depend on one library
and you have a fix for the library how many things do you want to
"remember to build"?  (though people are throwing up straw man
arguments for this too.  I suspect the worst case scenario is not
always the common case though.)

#2 Supposed physical memory savings - libSystem on Mac OS X only
exists in memory 1 time for all the programs that use it... sorta. 
Read Only pages are shared, writable pages are COW and yes, this adds
a good deal of complexity to the VM of the OS to have this.

We can stand up and push down arguments all day.  At the end it's up
to someone to decide if they're worth the trouble.  It seems that I'm
alone in my thinking that they might be of some use :-).

I'm actually more concerned with #1 overall.  If #2 is a real issue
then there may be something else wrong with the system, though I'd be
kind of interested to see what the real performance gains of shared
library systems are vs static library systems... I suspect this
measurement hasn't really been done adequately.  Then again, if it's
not a real problem, no sense trying to fix it either.

As for #1, I suppose that can be handled through a well constructed
build system also without making the kernel bend in knots to support
shared libs and mmap'ing and sharing pages or not.

I'm actually not really convinced myself that shared libs are good...
I prefer to remain somewhat agnostic on the topic.  I've not heard
particularly convincing arguments in favor of them and the best and
most reasonable argument not to have them is increased complexity.

Dave

Re: [9fans] Install from CD fails

[Brantley Coile]

System designers have a responsibility to help the rest of the
system's community by providing a useable system.  I'll make the
following observations regarding shared libraries as if you and I were
debating putting shared libraries into a system.  I have worked both
sides of this issue, so I have an experienced opinion, not just apriori
idea of good system hygiene.

First, let me deal with your two `advantages.'

> #1 Maintenance - If you have 50 programs that depend on one library
> and you have a fix for the library how many things do you want to
> "remember to build"?  (though people are throwing up straw man
> arguments for this too.  I suspect the worst case scenario is not
> always the common case though.)

This has an ugly other side.  If I have 50 programs that depend on one
library and fix a bug in the library, I now have 50 live programs in
production that haven't been tested.  I don't know if my change has
broken something else and won't know until I re-test all 50 programs.
I now have to do all that testing at once.  With static libraries I
have the luxury of going thru the 50 programs one at a time and
relinking them with the new library or reverting to an old library if
there's a problem.
>
> #2 Supposed physical memory savings - libSystem on Mac OS X only
> exists in memory 1 time for all the programs that use it... sorta.
> Read Only pages are shared, writable pages are COW and yes, this adds
> a good deal of complexity to the VM of the OS to have this.

Before X windows, 10% of programs were from the library.  With X
windows that number ballooned to 90%, so there is an apparent (see
below) space savings with X windows.  This is because when to touch
anything you get everything.  That, I would argue, is bad design.  (I
added shared library to a custom embedded Unix I did 15 years ago
because the application was also mis-designed.) The X library is
really a sub-system that should be somewhere other than the user's
program.


Also on the down side, shared libraries make it hard to distrubute
binaries without also sending out the library.  The binary, or even
the files in the same directory, are no longer all that is required to
run the program.  You also have to have the correct version of the
library.  Since you get different binaries that need different
versions of the library, you now wind up with three or six or more
slight variations of the same library, all get loaded into core.  Now,
where's my space saving?  Remove one and now you have a bunch of
programs that don't work, but you don't know it yet.

To load a binary that is linked to a library takes longer to load.


I have seen code sharing done right, such as Oberon, but every version
of shared libraries in Unix I've seen cost more than they were worth.
Including my own.

Hope the above arguments at least seem rational.  They are off the top
of my head.  I'm sure others on 9fans have other, better data points
that do the engineering calculation on the cost/benefits of shared
libraries.


 bc

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 07:19:08PM -0400, Brantley Coile wrote:
> > #2 Supposed physical memory savings - libSystem on Mac OS X only
> > exists in memory 1 time for all the programs that use it... sorta.
> > Read Only pages are shared, writable pages are COW and yes, this adds
> > a good deal of complexity to the VM of the OS to have this.
>
> Before X windows, 10% of programs were from the library.  With X
> windows that number ballooned to 90%, so there is an apparent (see
> below) space savings with X windows.  This is because when to touch
> anything you get everything.  That, I would argue, is bad design.  (I
> added shared library to a custom embedded Unix I did 15 years ago
> because the application was also mis-designed.) The X library is
> really a sub-system that should be somewhere other than the user's
> program.

  Speaking of which: since I'm a bit behind when it comes to the
  modern X, it was only a week ago that I discovered a global
  conspiracy called Font rendering via RENDER extension. It looks
  to me that the usual confusion about who's the server and who's
  the client in X-world has finally reached its peak. So much so, that
  a simple task of rendering a string of text with a particular
  font (which 5 years ago required a couple of X* calls) now
  requires each client to have a deep and profound knowledge of
  what hardware each Xserver is running on. And of course, requires
  an application as simple as Xterm being dynamically linked with
  5-6(!) additional libraries. Worse yet -- it requires *EVERY*
  application to be linked with these libraries since now its
  client's responsibility to render text:

     $ ldd /usr/X11/bin/xterm | wc -l
     20
     $ ldd /opt/MozillaFirefox/lib/firefox-bin  | wc -l
     52

   And did I mention the use of XML config files to manage all of the
   complexity the client has to face from now on ?

   Madness... :-(

> Also on the down side, shared libraries make it hard to distrubute
> binaries without also sending out the library.  The binary, or even
> the files in the same directory, are no longer all that is required to
> run the program.  You also have to have the correct version of the
> library.  Since you get different binaries that need different
> versions of the library, you now wind up with three or six or more
> slight variations of the same library, all get loaded into core.  Now,
> where's my space saving?  Remove one and now you have a bunch of
> programs that don't work, but you don't know it yet.

  It gets even more entertaining when it comes to doing forensics on
  'core' files generated by a binary with an ldd output longer than
  Bill Gates's tax return. :-(

Thanks,
Roman.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Brantley Coile <brantley@coraid.com> wrote:
> System designers have a responsibility to help the rest of the
> system's community by providing a useable system.  I'll make the
> following observations regarding shared libraries as if you and I were
> debating putting shared libraries into a system.  I have worked both
> sides of this issue, so I have an experienced opinion, not just apriori
> idea of good system hygiene.
>
> First, let me deal with your two `advantages.'
>
> > #1 Maintenance - If you have 50 programs that depend on one library
> > and you have a fix for the library how many things do you want to
> > "remember to build"?  (though people are throwing up straw man
> > arguments for this too.  I suspect the worst case scenario is not
> > always the common case though.)
>
> This has an ugly other side.  If I have 50 programs that depend on one
> library and fix a bug in the library, I now have 50 live programs in
> production that haven't been tested.  I don't know if my change has
> broken something else and won't know until I re-test all 50 programs.
> I now have to do all that testing at once.  With static libraries I
> have the luxury of going thru the 50 programs one at a time and
> relinking them with the new library or reverting to an old library if
> there's a problem.

This is definitely a point of view I've not heard before, regarding
suddenly changed but now untested binaries.  However let me say that I
have experienced this problem before.

Mac OS X Tiger betas shipped with new software that broke old software
that relied on glib, which relied on some old software behavior.  Lots
of stuff just stopped working as a result.

There goes my maintenance claim :)... and it's not just theory, it happend.

> >
> > #2 Supposed physical memory savings - libSystem on Mac OS X only
> > exists in memory 1 time for all the programs that use it... sorta.
> > Read Only pages are shared, writable pages are COW and yes, this adds
> > a good deal of complexity to the VM of the OS to have this.
>
> Before X windows, 10% of programs were from the library.  With X
> windows that number ballooned to 90%, so there is an apparent (see
> below) space savings with X windows.  This is because when to touch
> anything you get everything.  That, I would argue, is bad design.  (I
> added shared library to a custom embedded Unix I did 15 years ago
> because the application was also mis-designed.) The X library is
> really a sub-system that should be somewhere other than the user's
> program.
>
>
> Also on the down side, shared libraries make it hard to distrubute
> binaries without also sending out the library.  The binary, or even
> the files in the same directory, are no longer all that is required to
> run the program.  You also have to have the correct version of the
> library.  Since you get different binaries that need different
> versions of the library, you now wind up with three or six or more
> slight variations of the same library, all get loaded into core.  Now,
> where's my space saving?  Remove one and now you have a bunch of
> programs that don't work, but you don't know it yet.
>
> To load a binary that is linked to a library takes longer to load.
>
>
> I have seen code sharing done right, such as Oberon, but every version
> of shared libraries in Unix I've seen cost more than they were worth.
> Including my own.
>
> Hope the above arguments at least seem rational.  They are off the top
> of my head.  I'm sure others on 9fans have other, better data points
> that do the engineering calculation on the cost/benefits of shared
> libraries.
>
>

Sure these are good arguments.  For the record, I know people outside
the Plan 9 camp who stick to the "shared libraries are bad argument"
and I've always liked to think of myself as pretty open minded to
unpopular ideas.  I know from experience that popular doesn't mean
it's correct.

Dave

>  bc
>
>

Re: [9fans] Install from CD fails

[Charles Forsyth]

someone suggested culling the sysfatal calls from cat,
so aiming to please at least some people, i'll comply:

8L -KV cat.8
atexit unused
atexitdont unused
lock unused
canlock unused
unlock unused
atoi unused
sleep unused
_tas unused
term% size 8.out
1045t + 56d + 264b = 1365	8.out

because atexit isn't used, all the lock.c code, _tas and sleep vanish as well.

Re: [9fans] Install from CD fails

[Chad Dougherty]

David Leimbach wrote:
> This is definitely a point of view I've not heard before, regarding
> suddenly changed but now untested binaries.  However let me say that I
> have experienced this problem before.
>

yeah, same here.  i agree with Brantley's original point but one
counterpoint i can think of is that many (most?) of the bugs fixed in
shared libs fall into in the "just wrong, wrong, wrong and any
application that used this was just lucky to have worked in the first
place if it even worked at all" category.  breaking one program in order
to fix the other 49 that were just lucky to have worked in the first
place could still be considered a win.  however, i'm not advocating
shared libs in plan 9.  the issue is made pretty much moot by having a
sane and reliable build system.  build the new bins and move on with
your life...

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/19/06, Chad Dougherty <crd@andrew.cmu.edu> wrote:
> David Leimbach wrote:
> > This is definitely a point of view I've not heard before, regarding
> > suddenly changed but now untested binaries.  However let me say that I
> > have experienced this problem before.
> >
>
> yeah, same here.  i agree with Brantley's original point but one
> counterpoint i can think of is that many (most?) of the bugs fixed in
> shared libs fall into in the "just wrong, wrong, wrong and any
> application that used this was just lucky to have worked in the first
> place if it even worked at all" category.  breaking one program in order
> to fix the other 49 that were just lucky to have worked in the first
> place could still be considered a win.  however, i'm not advocating
> shared libs in plan 9.  the issue is made pretty much moot by having a
> sane and reliable build system.  build the new bins and move on with
> your life...
>

Is there a way to tell what version of a static library a potentially
old binary is linked against?  I don't like guessing games :)

Re: [9fans] Install from CD fails

[Federico G. Benavento]

> On 4/19/06, Chad Dougherty <crd@andrew.cmu.edu> wrote:

> Is there a way to tell what version of a static library a potentially
> old binary is linked against?  I don't like guessing games :)

ls -l, 9fs dump?

Federico G. Benavento

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/19/06, Federico G. Benavento <benavento@gmail.com> wrote:
> > On 4/19/06, Chad Dougherty <crd@andrew.cmu.edu> wrote:
>
> > Is there a way to tell what version of a static library a potentially
> > old binary is linked against?  I don't like guessing games :)
>
> ls -l, 9fs dump?
>
> Federico G. Benavento
>
>
I guess that lets me look at the library sizes to see if they've
changed then I can compare my binaries to see if they aren't the same.
 Still I don't think this ends up being a complete solution for
binaries that I'm not aware of the static library linkage.

Dave

Re: [9fans] Install from CD fails

[geoff]

If you're really worried about rebuilding all the right things, never
use `mk clean' and when you're ready to update the world,

	cd /sys/src; mk install	# or mk installall

I believe that only things that need to be rebuilt will be rebuilt and
installed.

Re: [9fans] Install from CD fails

[Russ Cox]

> Is there a way to tell what version of a static library a potentially
> old binary is linked against?  I don't like guessing games :)

look at the date on the file and then 9fs dump.

russ

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/19/06, Russ Cox <rsc@swtch.com> wrote:
> > Is there a way to tell what version of a static library a potentially
> > old binary is linked against?  I don't like guessing games :)
>
> look at the date on the file and then 9fs dump.
>
> russ
>
>
"touch" could be problematic here then I guess?

I guess also, if in doubt, just rebuild it, and if you want extra
assurance, md5 hash the files  that are built against the latest.

Dunno could get pretty crazy with this if I wanted to I suppose.

Re: [9fans] Install from CD fails

[Federico G. Benavento]

> "touch" could be problematic here then I guess?
>

lotte% history -D /386/bin/cat
Mar 30 15:06:05 ART 2006 /386/bin/cat 37482 [fgb]
Mar 30 15:06:05 ART 2006 /n/dump/2006/0419/386/bin/cat 37482 [fgb]
binary files /n/dump/2006/0419/386/bin/cat /n/dump/2006/0330/386/bin/cat differ
Oct 27 00:37:47 ART 2005 /n/dump/2006/0330/386/bin/cat 37465 [fgb]

Re: [9fans] Install from CD fails

[Charles Forsyth]

i regard the libraries themselves as irrelevant and focus on the things they contain.
those are the things that have different versions, and the version is determined by their signatures and
code (separately).   is there anything else that matters?

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/19/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> i regard the libraries themselves as irrelevant and focus on the things they contain.
> those are the things that have different versions, and the version is determined by their signatures and
> code (separately).   is there anything else that matters?
>
>
True, it's the final payload of bytes from building against static
libs that counts... so it seems checksums are good enough.

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 02:54:40PM -0700, David Leimbach wrote:
> On 4/18/06, Roman Shaposhnick <rvs@sun.com> wrote:
> > On Tue, Apr 18, 2006 at 12:22:30PM -0700, David Leimbach wrote:
> > > On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > > The interesting thing is that Plan 9's great namespace manipulation
> > > functionality + the fact that each process can have a private
> > > namespace means that Plan 9 probably has the best shot at dealing with
> > > "DLL-hell", like when 10 programs need 10 different versions of the
> > > same shared library to run respectively.  A simple script wrapped
> > > around the loading of a program can set up a namespace such that
> > > ambiguities don't exist.
> >
> >   And you would have to go through all of the aforementioned troubles
> >   to achieve exactly what ? What is it, that shared libraries are good
> >   at ?
>
> #1 Maintenance - If you have 50 programs that depend on one library
> and you have a fix for the library how many things do you want to
> "remember to build"?  (though people are throwing up straw man
> arguments for this too.  I suspect the worst case scenario is not
> always the common case though.)

  Perhaps. However, now that I've suffered my own share of .so madness by
  the virtue of working on Sun's C++ compiler, I really don't
  think that maintenance argument is a valid one. As always, it
  sounds good in theory, but as I've mentioned in my previous post
  an absolute lack of a decent management scheme kills it. The fix
  that benefits some application usually breaks others. What's even
  scarier is -- once broken they have to resort to LD_LIBRARY_PATH
  or some such to work around the issue. Not good.

  In a C++ case its even worse, because C++ runtime shared library is
  the worst of them all -- shared library other shared libraries
  might depend upon. It is really difficult to manage even the
  most trivial of cases: when a particular vendor distributes a
  C++ library as a shared object compiled with an older compiler
  and thus having a dependency on a previous generation of a C++
  runtime shared library and later on the customer uses the latest
  version of a compiler only to discover the joys of mixing two
  version of a C++ runtime in one application.

> #2 Supposed physical memory savings - libSystem on Mac OS X only
> exists in memory 1 time for all the programs that use it... sorta.
> Read Only pages are shared, writable pages are COW and yes, this adds
> a good deal of complexity to the VM of the OS to have this.

  This one I might even agree with. However, there are much better
  ways of saving memory: kernel + file servers clients interact with
  would be the one.

> I'm actually more concerned with #1 overall.

  And that's the problem. Once you start layering dependencies on
  top of each other -- you end up in a mess.

  I think the key issue here is that while it is safe to assume
  that you can control bits and pieces your application is built
  from during the build process itself, it is absolutely naive
  to assume that you have any guarantees of the same kind at a runtime
  if all that you give to a dynamic linker is just a name of an external
  entity (like a function call or a static variable). It sort of reminds
  me of the life in a flat file namespace, if you know what I mean.
  Painful and very difficult to manage right.

Thanks,
Roman.

Re: [9fans] Install from CD fails

[Federico G. Benavento]

> I prefer to remain somewhat agnostic on the topic.  I've not heard
> particularly convincing arguments in favor of them and the best and
> most reasonable argument not to have them is increased complexity.
>

what about what actually happens in practice?
10 programs needing 10 different versions of the same shared lib;
or when you want to install some binary app that needs libxxx.1.1.4b.so,
but you have libxx.1.3.4a.so.
Even when you try to keep things clean you end up in a mess.

btw, the fact that Plan 9 could deal better with the crap of shared libs
doesn't mean they are good at anything.

Federico G. Benavento

PS: ldd on your favorite app can be fun, if feel masochistic.

Re: [9fans] Install from CD fails

[Lyndon Nerenberg]

> what about what actually happens in practice?
> 10 programs needing 10 different versions of the same shared lib;
> or when you want to install some binary app that needs libxxx.1.1.4b.so,
> but you have libxx.1.3.4a.so.
> Even when you try to keep things clean you end up in a mess.

As a working example of this, At my day job I just spent the last while
configuring and building a statically linked apache+php binary for the
production web servers.  Why?  The combination of
expat+libxml+xmlrpc+libxslt versions required by php and apache combined
with the shared library dependency hell that is Linux resulted in a
dependency graph that *could* *not* be solved by any combination of
multiple versions of all these libraries without causing something to stop
functioning.

I fought the same battle six years ago while trying to get an IMAP server
running (Linux again).  Same solution then, too.

Shared libraries just aren't worth it.

--lyndon

Re: [9fans] Install from CD fails

[David Leimbach]

> Federico G. Benavento
>
> PS: ldd on your favorite app can be fun, if feel masochistic.
>
>

Yeah pick any GNOME application... or even KDE.  It can be rather
interesting :).

Re: [9fans] Install from CD fails

[Ronald G Minnich]

David Leimbach wrote:

> Yeah pick any GNOME application... or even KDE.  It can be rather
> interesting :).


you're getting way too fancy. /bin/date anyone?

sh-3.00$ export LD_KDEBUG=all
sh-3.00$ date
        515:
        515:	file=librt.so.1 [0];  needed by date [0]
        515:	find library=librt.so.1 [0]; searching
        515:	 search cache=/etc/ld.so.cache
        515:	  trying file=/lib/librt.so.1
        515:
        515:	file=librt.so.1 [0];  generating link map
        515:	  dynamic: 0x44d35ebc  base: 0x00000000   size: 0x00013ef8
        515:	    entry: 0x44d2f060  phdr: 0x44d2d034  phnum:          9
        515:
        515:
        515:	file=libc.so.6 [0];  needed by date [0]
        515:	find library=libc.so.6 [0]; searching
        515:	 search cache=/etc/ld.so.cache
        515:	  trying file=/lib/libc.so.6
        515:
        515:	file=libc.so.6 [0];  generating link map
        515:	  dynamic: 0x4423fd3c  base: 0x00000000   size: 0x00128bdc
        515:	    entry: 0x4412feaa  phdr: 0x4411b034  phnum:         10
        515:
        515:
        515:	file=libpthread.so.0 [0];  needed by /lib/librt.so.1 [0]
        515:	find library=libpthread.so.0 [0]; searching
        515:	 search cache=/etc/ld.so.cache
        515:	  trying file=/lib/libpthread.so.0
        515:
        515:	file=libpthread.so.0 [0];  generating link map
        515:	  dynamic: 0x4436ced0  base: 0x00000000   size: 0x000111c4
        515:	    entry: 0x44362804  phdr: 0x4435e034  phnum:          9
        515:
        515:	checking for version `GLIBC_2.2' in file /lib/librt.so.1
[0] required by file date [0]
        515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
[0] required by file date [0]
        515:	checking for version `GLIBC_2.3.4' in file /lib/libc.so.6
[0] required by file date [0]
        515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
required by file date [0]
        515:	checking for version `GLIBC_2.3' in file /lib/libc.so.6 [0]
required by file date [0]
        515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
required by file date [0]
        515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
required by file date [0]
        515:	checking for version `GLIBC_PRIVATE' in file
/lib/ld-linux.so.2 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.2' in file
/lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.3.3' in file
/lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.1' in file
/lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.0' in file
/lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_PRIVATE' in file
/lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
[0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.3.2' in file /lib/libc.so.6
[0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_PRIVATE' in file /lib/libc.so.6
[0] required by file /lib/librt.so.1 [0]
        515:	checking for version `GLIBC_2.1' in file /lib/ld-linux.so.2
[0] required by file /lib/libc.so.6 [0]
        515:	checking for version `GLIBC_2.3' in file /lib/ld-linux.so.2
[0] required by file /lib/libc.so.6 [0]
        515:	checking for version `GLIBC_PRIVATE' in file
/lib/ld-linux.so.2 [0] required by file /lib/libc.so.6 [0]
        515:	checking for version `GLIBC_2.1' in file /lib/ld-linux.so.2
[0] required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_PRIVATE' in file
/lib/ld-linux.so.2 [0] required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
[0] required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_2.3.2' in file /lib/libc.so.6
[0] required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
required by file /lib/libpthread.so.0 [0]
        515:	checking for version `GLIBC_PRIVATE' in file /lib/libc.so.6
[0] required by file /lib/libpthread.so.0 [0]
        515:
        515:	prelink checking: ok
        515:
        515:	conflict processing: date
        515:
        515:	calling init: /lib/libpthread.so.0
        515:
        515:
        515:	calling init: /lib/libc.so.6
        515:
        515:
        515:	calling init: /lib/librt.so.1
        515:
        515:
        515:	initialize program: date
        515:
        515:
        515:	transferring control: date
        515:
Wed Apr 19 08:29:23 MDT 2006
        515:
        515:	calling fini: date [0]
        515:
        515:
        515:	calling fini: /lib/librt.so.1 [0]
        515:
        515:
        515:	calling fini: /lib/libpthread.so.0 [0]
        515:
        515:
        515:	calling fini: /lib/libc.so.6 [0]
        515:
sh-3.00$ exit

oh yeah. /bin/date needs pthreads. oh yeah.

ron

Re: [9fans] Install from CD fails

[Tim Wiess]

wow... that's pretty insane.


> you're getting way too fancy. /bin/date anyone?
>
> sh-3.00$ export LD_KDEBUG=all
> sh-3.00$ date
>         515:
>         515:	file=librt.so.1 [0];  needed by date [0]
>         515:	find library=librt.so.1 [0]; searching
>         515:	 search cache=/etc/ld.so.cache
>         515:	  trying file=/lib/librt.so.1
>         515:
>         515:	file=librt.so.1 [0];  generating link map
>         515:	  dynamic: 0x44d35ebc  base: 0x00000000   size: 0x00013ef8
>         515:	    entry: 0x44d2f060  phdr: 0x44d2d034  phnum:          9
>         515:
>         515:
>         515:	file=libc.so.6 [0];  needed by date [0]
>         515:	find library=libc.so.6 [0]; searching
>         515:	 search cache=/etc/ld.so.cache
>         515:	  trying file=/lib/libc.so.6
>         515:
>         515:	file=libc.so.6 [0];  generating link map
>         515:	  dynamic: 0x4423fd3c  base: 0x00000000   size: 0x00128bdc
>         515:	    entry: 0x4412feaa  phdr: 0x4411b034  phnum:         10
>         515:
>         515:
>         515:	file=libpthread.so.0 [0];  needed by /lib/librt.so.1 [0]
>         515:	find library=libpthread.so.0 [0]; searching
>         515:	 search cache=/etc/ld.so.cache
>         515:	  trying file=/lib/libpthread.so.0
>         515:
>         515:	file=libpthread.so.0 [0];  generating link map
>         515:	  dynamic: 0x4436ced0  base: 0x00000000   size: 0x000111c4
>         515:	    entry: 0x44362804  phdr: 0x4435e034  phnum:          9
>         515:
>         515:	checking for version `GLIBC_2.2' in file /lib/librt.so.1
> [0] required by file date [0]
>         515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
> [0] required by file date [0]
>         515:	checking for version `GLIBC_2.3.4' in file /lib/libc.so.6
> [0] required by file date [0]
>         515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
> required by file date [0]
>         515:	checking for version `GLIBC_2.3' in file /lib/libc.so.6 [0]
> required by file date [0]
>         515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
> required by file date [0]
>         515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
> required by file date [0]
>         515:	checking for version `GLIBC_PRIVATE' in file
> /lib/ld-linux.so.2 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.2' in file
> /lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.3.3' in file
> /lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.1' in file
> /lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.0' in file
> /lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_PRIVATE' in file
> /lib/libpthread.so.0 [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
> [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
> required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
> required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
> required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.3.2' in file /lib/libc.so.6
> [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_PRIVATE' in file /lib/libc.so.6
> [0] required by file /lib/librt.so.1 [0]
>         515:	checking for version `GLIBC_2.1' in file /lib/ld-linux.so.2
> [0] required by file /lib/libc.so.6 [0]
>         515:	checking for version `GLIBC_2.3' in file /lib/ld-linux.so.2
> [0] required by file /lib/libc.so.6 [0]
>         515:	checking for version `GLIBC_PRIVATE' in file
> /lib/ld-linux.so.2 [0] required by file /lib/libc.so.6 [0]
>         515:	checking for version `GLIBC_2.1' in file /lib/ld-linux.so.2
> [0] required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_PRIVATE' in file
> /lib/ld-linux.so.2 [0] required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_2.1.3' in file /lib/libc.so.6
> [0] required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_2.3.2' in file /lib/libc.so.6
> [0] required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_2.0' in file /lib/libc.so.6 [0]
> required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_2.2' in file /lib/libc.so.6 [0]
> required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_2.1' in file /lib/libc.so.6 [0]
> required by file /lib/libpthread.so.0 [0]
>         515:	checking for version `GLIBC_PRIVATE' in file /lib/libc.so.6
> [0] required by file /lib/libpthread.so.0 [0]
>         515:
>         515:	prelink checking: ok
>         515:
>         515:	conflict processing: date
>         515:
>         515:	calling init: /lib/libpthread.so.0
>         515:
>         515:
>         515:	calling init: /lib/libc.so.6
>         515:
>         515:
>         515:	calling init: /lib/librt.so.1
>         515:
>         515:
>         515:	initialize program: date
>         515:
>         515:
>         515:	transferring control: date
>         515:
> Wed Apr 19 08:29:23 MDT 2006
>         515:
>         515:	calling fini: date [0]
>         515:
>         515:
>         515:	calling fini: /lib/librt.so.1 [0]
>         515:
>         515:
>         515:	calling fini: /lib/libpthread.so.0 [0]
>         515:
>         515:
>         515:	calling fini: /lib/libc.so.6 [0]
>         515:
> sh-3.00$ exit
>
> oh yeah. /bin/date needs pthreads. oh yeah.
>
> ron
>

Re: [9fans] Install from CD fails

[Wes Kussmaul]

Ronald G Minnich wrote:

> you're getting way too fancy. /bin/date anyone?

So let me ask a dumb question: why is it Linuxbios instead of 9bios?

Re: [9fans] Install from CD fails

[Ronald G Minnich]

Wes Kussmaul wrote:
> Ronald G Minnich wrote:
>
>> you're getting way too fancy. /bin/date anyone?
>
>
> So let me ask a dumb question: why is it Linuxbios instead of 9bios?
>
>
>

because the original intent was to use linux as the bios platform.

It's still not a bad idea, if you look at EFI you see a poorly-designed
OS masquerading as a BIOS. If you're going to use a poorly designed OS,
you might as well use one that has drivers and people already understand.

ron

Re: [9fans] Install from CD fails

[Jack Johnson]

On 4/20/06, Ronald G Minnich <rminnich@lanl.gov> wrote:
> It's still not a bad idea, if you look at EFI you see a poorly-designed
> OS masquerading as a BIOS. If you're going to use a poorly designed OS,
> you might as well use one that has drivers and people already understand.

Slightly interesting read; watch your sodium intake:

http://www.kerneltraffic.org/kernel-traffic/kt20030910_231.html#7

"The guys at Intel were having problems getting a traditional PC style
BIOS to run on the first Itaniums, realized they had a opportunity to
come up with a cleaner firmware interface and came up with EFI. Open
Firmware was considered but dropped because it was not compatible with
ACPI, and they did not want to dilute the momentum that had built up
for ACPI."

...which seems a shame (and may be apocryphal).  I never minded
OpenFirmware/OpenBoot, and until the EFI Macs shipped I still had my
hopes up.  Maybe they couldn't find enough Forth programmers.  :)

Linus' rant and the unofficial (but informed) Intel response that
followed are worth a read.

-Jack

Re: [9fans] Install from CD fails

[Ronald G Minnich]

EFI requires a FAT-12 formatted FLASH on your mainboard. Licensed IP
from M$. As is the PCOFF format the binaries use.

need I say more?

ron

Re: [9fans] Install from CD fails

[Dan Cross]

On Thu, Apr 20, 2006 at 03:20:56PM -0600, Ronald G Minnich wrote:
> EFI requires a FAT-12 formatted FLASH on your mainboard. Licensed IP
> from M$. As is the PCOFF format the binaries use.
>
> need I say more?

FAT-12?  Shesh.  You know, I wonder why a more reasonable filesystem format
can't be found for things like this.  I mean, it doesn't have to be fancy
(I've often thought that a simple extent-based filesystem with an extent-
based directory structure would be sufficient for, e.g., 9fat), but it doesn't
have to be FAT, either.

	- Dan C.

Re: [9fans] Install from CD fails

[Brantley Coile]

Gary Kindall had a great one.  http://www.dcast.vbox.co.uk/cpm.html.

> On Thu, Apr 20, 2006 at 03:20:56PM -0600, Ronald G Minnich wrote:
>> EFI requires a FAT-12 formatted FLASH on your mainboard. Licensed IP
>> from M$. As is the PCOFF format the binaries use.
>>
>> need I say more?
>
> FAT-12?  Shesh.  You know, I wonder why a more reasonable filesystem format
> can't be found for things like this.  I mean, it doesn't have to be fancy
> (I've often thought that a simple extent-based filesystem with an extent-
> based directory structure would be sufficient for, e.g., 9fat), but it doesn't
> have to be FAT, either.
>
> 	- Dan C.

Re: [9fans] Install from CD fails

[Ronald G Minnich]

Dan Cross wrote:

> FAT-12?  Shesh.  You know, I wonder why a more reasonable filesystem format
> can't be found for things like this.  I mean, it doesn't have to be fancy
> (I've often thought that a simple extent-based filesystem with an extent-
> based directory structure would be sufficient for, e.g., 9fat), but it doesn't
> have to be FAT, either.

it gets better. They also define a Flat File System (FFS) in the spec.
It has functions to look up and create files, including one function to
tell you if there are two files of the same name on the volume, and if
so, where the many files with the same name are. I kid you not. It's one
of the worst things I've ever seen.

ron

Re: [9fans] Install from CD fails

[Dan Cross]

On Thu, Apr 20, 2006 at 09:43:35PM -0600, Ronald G Minnich wrote:
> it gets better. They also define a Flat File System (FFS) in the spec.
> It has functions to look up and create files, including one function to
> tell you if there are two files of the same name on the volume, and if
> so, where the many files with the same name are. I kid you not. It's one
> of the worst things I've ever seen.

Sounds a lot like my final project for high school AP computer science
(back when it was in Pascal).

You know, the way the Intel engineer described EFI, some of it didn't
sound all together unreasonable.  I don't understand why they wouldn't
try and integrate ACPI support into Open Firmware and be done with it,
but hey, what do I know?  However, it seems like Dis would have been
a really interesting choice for the EFI bytecode engine.

	- Dan C.

Re: [9fans] Install from CD fails

[Ronald G Minnich]

Dan Cross wrote:

> You know, the way the Intel engineer described EFI, some of it didn't
> sound all together unreasonable.  I don't understand why they wouldn't
> try and integrate ACPI support into Open Firmware and be done with it,

They're very smart people at Intel. They're just not very good OS
designers. Few people are -- that's why we're on this list in the first
place ...

I've begun to realize in the last 6 years that every BIOS designer wants
to be an OS designer. I am working with a system now in which the BIOS
is actually a small message-passing kernel -- all running in 16-bit mode
in SM Interrupt space. Quite impressive ... and it all is running while
your real OS is running!

ron

Re: [9fans] Install from CD fails

[Wes Kussmaul]

Ronald G Minnich wrote:

> EFI requires a FAT-12 formatted FLASH on your mainboard. Licensed IP 
> from M$. 

Can you point us (me) to info about the licensed IP besides PE/COFF?

> need I say more?

Sure. Say something about getting LinuxBIOS, which AMD64 motherboards 
support it, availability on laptops, etc. The world wants to know about 
routes around proprietary walls.


-- 
Wes Kussmaul
CIO
The Village Group
738 Main Street
Waltham, MA 02451

781-647-7178


My uncle likes to say that the world’s biggest troubles started when the serpent said, “Try this fruit, and by the way if a bunch of people collectively calling themselves Arthur Andersen signs something it’s the same as if a person named Arthur Andersen signed it.” I don’t get the serpent and fruit part. Must be some Swiss mythology thing. He can be a bit obscure. 

                         P.K. Iggy
                         _How I Like Fixed The Internet_
                           (Tales from the Great Infodepression of 2009
                           and the prosperity that followed)

Re: [9fans] Install from CD fails

[Charles Forsyth]

> Can you point us (me) to info about the licensed IP besides PE/COFF?

microsoft stakes some IP claim to aspects of FAT.
i know ... i know.   http://www.terzarima.net/fatgen103.pdf
for FAT32.  as it happens, in the licence of that one, there are some
special royalty-free provisions for specific uses with EFI.

Re: [9fans] Install from CD fails

[Ronald G Minnich]

Wes Kussmaul wrote:

> Sure. Say something about getting LinuxBIOS, which AMD64 motherboards
> support it, availability on laptops, etc. The world wants to know about
> routes around proprietary walls.
>
>

we're working the laptop issue. It's been a bear. BUT: AMD has been
absolutely great to deal with. They just gave us a great deal of help
getting things going on the GX2 platform. They sent down a BIOS engineer
from colorado to work with us all week on the port. Just amazing.

I am hoping, in the next two weeks, to be booting Plan 9 out of the BIOS
FLASH part :-) (Which, BTW, will make Plan 9 my 9load replacement).

ron

Re: [9fans] Install from CD fails

[Dan Cross]

On Thu, Apr 20, 2006 at 09:45:41PM -0600, Ronald G Minnich wrote:
> I am hoping, in the next two weeks, to be booting Plan 9 out of the BIOS
> FLASH part :-) (Which, BTW, will make Plan 9 my 9load replacement).

Well then.  The IRC crowd had better get a move on!

	- Dan C.

Re: [9fans] Install from CD fails

[Christoph Lohmann]

Good day.

Am Fri, 21 Apr 2006 00:31:03 -0400 schrieb Dan Cross <cross@math.psu.edu>:

> Well then.  The IRC crowd had better get a move on!

Overhauling without catching up.

Sincerely,

Christoph

Re: [9fans] Install from CD fails

[lucio]

> (Which, BTW, will make Plan 9 my 9load replacement).

That will take a big excuse off Jim's list.  No more waiting for
Uriel :-)

++L

Re: [9fans] Install from CD fails

[jmk]

In the limiting case, where every binary needs a different version of
the shared library, you are back where we are today with Plan 9's statically
linked binaries, so why bother even thinking about it.

On Tue Apr 18 14:55:41 EDT 2006, forsyth@terzarima.net wrote:
> >>
> >> Don't shared libraries also typically provide memory savings?  One
> >> version of your c library "resident" for all VM spaces to map?
> >>
>
> that's often quoted as a consequence, but in practice,
> not that i've seen in ...  what is it now?  ...  at least six or
> seven different systems.  i think the trouble is that to get savings
> that make the pain worthwhile you still need various forms of
> discipline, but with shared libraries, people are even less concerned.
> and RSS continues up.
>
> another is bug fixing at a stroke, but it also allows
> bug and trapdoor introduction at a stroke.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, jmk@plan9.bell-labs.com <jmk@plan9.bell-labs.com> wrote:
> In the limiting case, where every binary needs a different version of
> the shared library, you are back where we are today with Plan 9's statically
> linked binaries, so why bother even thinking about it.

But by that same logic quicksort algorithms tend towards O(N^2) when
data is already sorted, so why use them ever either?  (though I wonder
how often quicksort is really rolled out in production... also... I
think I just defeated my own argument.)

Dave


>
> On Tue Apr 18 14:55:41 EDT 2006, forsyth@terzarima.net wrote:
> > >>
> > >> Don't shared libraries also typically provide memory savings?  One
> > >> version of your c library "resident" for all VM spaces to map?
> > >>
> >
> > that's often quoted as a consequence, but in practice,
> > not that i've seen in ...  what is it now?  ...  at least six or
> > seven different systems.  i think the trouble is that to get savings
> > that make the pain worthwhile you still need various forms of
> > discipline, but with shared libraries, people are even less concerned.
> > and RSS continues up.
> >
> > another is bug fixing at a stroke, but it also allows
> > bug and trapdoor introduction at a stroke.
>

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 07:54:16PM +0100, Charles Forsyth wrote:
> >>
> >> Don't shared libraries also typically provide memory savings?  One
> >> version of your c library "resident" for all VM spaces to map?
> >>
>
> that's often quoted as a consequence, but in practice,
> not that i've seen in ...  what is it now?  ...  at least six or
> seven different systems.  i think the trouble is that to get savings
> that make the pain worthwhile you still need various forms of
> discipline, but with shared libraries, people are even less concerned.
> and RSS continues up.
>
> another is bug fixing at a stroke, but it also allows
> bug and trapdoor introduction at a stroke.

  I seems that the majority of the problems associated with shared libraries
  are testaments to an absolute lack of any sensible management mechanism
  built into the system. LD_LIBRARY_PATH & co. look more like successful
  April Fools, than anything else.

Thanks,
Roman.

Re: [9fans] Install from CD fails

[Dan Cross]

On Tue, Apr 18, 2006 at 07:54:16PM +0100, Charles Forsyth wrote:
> that's often quoted as a consequence, but in practice,
> not that i've seen in ...  what is it now?  ...  at least six or
> seven different systems.  i think the trouble is that to get savings
> that make the pain worthwhile you still need various forms of
> discipline, but with shared libraries, people are even less concerned.
> and RSS continues up.

I think shared libraries are often times misunderstood.  I once knew
someone who thought that a context switch was less expensive using
shared libraries than without.

	- Dan C.

Re: [9fans] Install from CD fails

[Charles Forsyth]

> I think shared libraries are often times misunderstood.  I once knew
> someone who thought that a context switch was less expensive using
> shared libraries than without.

many, many people apparently that think that a context switch
must necessarily be much more expensive than a function call
and return (or two)

Re: [9fans] Install from CD fails

[Dan Cross]

On Tue, Apr 18, 2006 at 10:11:55PM +0100, Charles Forsyth wrote:
> > I think shared libraries are often times misunderstood.  I once knew
> > someone who thought that a context switch was less expensive using
> > shared libraries than without.
>
> many, many people apparently that think that a context switch
> must necessarily be much more expensive than a function call
> and return (or two)

Ooops, let me clarify.  The individual in question thought that a program
that was linked dynamically was less expensive to switch than one that was
linked statically.

	- Dan C.

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > I think shared libraries are often times misunderstood.  I once knew
> > someone who thought that a context switch was less expensive using
> > shared libraries than without.
>
> many, many people apparently that think that a context switch
> must necessarily be much more expensive than a function call
> and return (or two)
>
>

Doesn't a lot of that depend on how you define a context? :-)

Re: [9fans] Install from CD fails

[David Leimbach]

On 4/18/06, Dan Cross <cross@math.psu.edu> wrote:
> On Tue, Apr 18, 2006 at 07:54:16PM +0100, Charles Forsyth wrote:
> > that's often quoted as a consequence, but in practice,
> > not that i've seen in ...  what is it now?  ...  at least six or
> > seven different systems.  i think the trouble is that to get savings
> > that make the pain worthwhile you still need various forms of
> > discipline, but with shared libraries, people are even less concerned.
> > and RSS continues up.
>
> I think shared libraries are often times misunderstood.  I once knew
> someone who thought that a context switch was less expensive using
> shared libraries than without.
>
>         - Dan C.

It certainly wouldn't have any effect on the context switch itself
necessarily.  But it might have an effect on page faults for code
that's loaded dynamically and the swapping that would occur while
freeing up free memory to load multiple copies of the same code or
not.  (though it's been said over and over again that physical memory
savings of dynamic libraries rarely happens in practice.)

Re: [9fans] Install from CD fails

[geoff]

During the Christmas holidays in 1989, I ran a little benchmark
(`forktime') on various machines.  The program was roughly

	loop many times
		fork
		parent waits
		child exits

I ran it twice or more linked statically and twice or more linked
dynamically on each machine.  The machines were lightly loaded.  For
1000 iterations, user time is consistently about 100ms for
statically-linked forktime, but 4-11 times that when dynamically
linked (i.e., using shared libraries).

System times are more interesting.  SunOS 3.5 had no shared libraries,
so Sun seems to have sped up fork() by about 50% between SunOS 3.5 and
4.0 for statically-linked programs, but then lost all that performance
and more when using shared libraries on 4.0, which took almost 4 times
as much system time as statically-linked forktime on 4.0.

We scratched our heads about what in the world the kernel was doing
for 42ms per fork.  Even on a Sun 3/50, that's a lot of cycles.

This was a long time ago, but it's some actual measurements.

--- fork times ---
: utubrutus; time ./forktime 1000	# sun 3/50 sunos 3.5 diskless
       22.9 real         0.1 user        22.0 sys
       22.6 real         0.1 user        22.1 sys

: nil.ai; time 3s/forktime 1000		# sun 3/50 sunos 4.0 diskless -Bstatic
       12.0 real         0.1 user        11.7 sys
       11.9 real         0.1 user        11.7 sys
: nil.ai; time 3/forktime 1000		# sun 3/50 sunos 4.0 diskless dynamic
       46.8 real         0.8 user        41.7 sys
       46.9 real         1.1 user        42.4 sys
       45.9 real         0.9 user        42.2 sys

: neat.cs; time 4s/forktime 1000	# sun 4/280 sunos 4.0 -Bstatic
       12.7 real         0.1 user        11.8 sys
       12.0 real         0.1 user        11.2 sys
: neat.cs; time 4/forktime 1000		# sun 4/280 sunos 4.0 dynamic
       29.9 real         0.4 user        25.3 sys
       35.4 real         0.5 user        25.0 sys

---
/*
 * forktime - execute many forks for timing
 */

#define NULL 0

/* imports */
long atol();

main(argc, argv)
int argc;
char **argv;
{
	register long count = 1000;

	if (argc > 1)
		count = atol(argv[1]);
	while (count-- > 0) {
		register int pid = fork();

		if (pid < 0)
			perror(argv[0]);
		else if (pid == 0)	/* child */
			_exit(0);
		else {
			register int deadpid;

			while ((deadpid = wait((int *)NULL)) != pid && deadpid != -1)
				;
		}
	}
	_exit(0);
}

Re: [9fans] Install from CD fails

[Dan Cross]

On Tue, Apr 18, 2006 at 07:53:30PM -0700, geoff@collyer.net wrote:
> [...]
>
> We scratched our heads about what in the world the kernel was doing
> for 42ms per fork.  Even on a Sun 3/50, that's a lot of cycles.
>
> This was a long time ago, but it's some actual measurements.

Hmm.  Weird.  If it had been doing an exec, it would have been easy to
point the finger at the run time loader, but if all you did was fork?
My guess was reference counting pages in the shared libraries.  Did you
ever find out where the time went?

	- Dan C.

Re: [9fans] Install from CD fails

[geoff]

There's no exec; the actual program follows the timing data.

I don't believe that we ever found out where the time was going.

Re: [9fans] Install from CD fails

[Roman Shaposhnick]

On Tue, Apr 18, 2006 at 07:53:30PM -0700, geoff@collyer.net wrote:
> During the Christmas holidays in 1989, I ran a little benchmark
> (`forktime') on various machines.  The program was roughly
>
> 	loop many times
> 		fork
> 		parent waits
> 		child exits
>
> I ran it twice or more linked statically and twice or more linked
> dynamically on each machine.  The machines were lightly loaded.  For
> 1000 iterations, user time is consistently about 100ms for
> statically-linked forktime, but 4-11 times that when dynamically
> linked (i.e., using shared libraries).
>
> System times are more interesting.  SunOS 3.5 had no shared libraries,
> so Sun seems to have sped up fork() by about 50% between SunOS 3.5 and
> 4.0 for statically-linked programs, but then lost all that performance
> and more when using shared libraries on 4.0, which took almost 4 times
> as much system time as statically-linked forktime on 4.0.
>
> We scratched our heads about what in the world the kernel was doing
> for 42ms per fork.  Even on a Sun 3/50, that's a lot of cycles.
>
> This was a long time ago, but it's some actual measurements.

  It is still somewhat reproducible. We had a little chat about this
  issue with a colleague of mine, who happens to be an expert in
  performance tuning and the consensus seems to be that:

    1. First of all, you see dynamic linker resolving calls to _exit
    in every child process because of the default lazy binding policy.
    You can instruct it to do just the opposite by LD_BIND_NOW and
    cut the time in half.

    2. Even if you do #1 the dynamically linked binary is still inferior
    to the statically linked one because of the calls not being direct
    and going through the level of indirection.

Thanks,
Roman.

P.S. And yes, Solaris 10 banned static libraries :-(

Re: [9fans] Install from CD fails

[Dan Cross]

On Tue, Apr 18, 2006 at 02:24:49PM -0700, David Leimbach wrote:
> It certainly wouldn't have any effect on the context switch itself
> necessarily.  But it might have an effect on page faults for code
> that's loaded dynamically and the swapping that would occur while
> freeing up free memory to load multiple copies of the same code or
> not.  (though it's been said over and over again that physical memory
> savings of dynamic libraries rarely happens in practice.)

Well, this is Unix, so text segments are shared anyway.  The point is,
you're right: it wouldn't have an effect on the context switch itself.
The person in question was just plain wrong.  I tried to explain it to
him, and for a second he gave me that ``deer in the headlights'' blank
stare of non-comprehension, and then just went back to saying, ``write,
so it saves on the context switch....''  The guy was senior to me, so
we went with his idea, which introduced a new level of complexity that
was just ridiculous.

Of course, this was on a project written in C++ where each *class*
was in it's own directory.  That is, the headers were in a single
directory, and the ``.cpp'' (heaven forbide we should use .cc as the
extension for our C++ source files, since, you know, most of the guys
had a Windows background) were each in their own directory.  Make
was utterly defeated.

Ugh.

	- Dan C.

Re: [9fans] Install from CD fails

[Ronald G Minnich]

David Leimbach wrote:

> Don't shared libraries also typically provide memory savings?  One
> version of your c library "resident" for all VM spaces to map?



that's never been demonstrated in practice. The claim has been made, but
it's all notional and hand-waving.

ron

Re: [9fans] Install from CD fails

[LiteStar numnums]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

Didn't Burrough's MCP have this way back when? The public libraries even
functioned as IPC, although you could still link privately & get
normal static binaries...

On 4/18/06, Ronald G Minnich <rminnich@lanl.gov> wrote:
>
> David Leimbach wrote:
>
> > Don't shared libraries also typically provide memory savings?  One
> > version of your c library "resident" for all VM spaces to map?
>
>
>
> that's never been demonstrated in practice. The claim has been made, but
> it's all notional and hand-waving.
>
> ron
>



--
Nietzsche's first step is to accept what he knows. Atheism for him goes
without saying and is "contructive and
radical". Nietzsche's supreme vocation, so he says, is to provoke a kind of
crisis and a final decision about the
problem of atheism. The world continues on its course at random and there is
nothing final about it. Thus God
is useless, since He wants nothing in particular. If he wanted something --
and here we recognize the traditional
forumlation of the problem of evil -- He would have to assume responsiblity
for "a sum total of pain and inconsistency
which would debase the entire value of being born."
-- Albert Camus, L'Homme révolté

[-- Attachment #2: Type: text/html, Size: 1565 bytes --]

Re: [9fans] Install from CD fails

[Charles Forsyth]

> that's a great addition to the loader, and relatively easy to implement.

yes, i was going to try copying the changes into some-other-l tonight

Re: [9fans] Install from CD fails

[Bruce Ellis]

it would be great.  many programs have zero coverage of the
bits they can't get to.

brucee

On 4/19/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> > that's a great addition to the loader, and relatively easy to implement.
>
> yes, i was going to try copying the changes into some-other-l tonight

Re: [9fans] Install from CD fails

[uriel]

> grafting shared libraries into plan9 will make presto go into a rage.

I was going to say that the day Plan 9 gets shared libs I would shot
myself.  But given how many fans I have in 9fans that might have
motivated someone into actually implementing shard libs.

So maybe more persuasive is the fact that it would make boyd spin in his
grave until the Earth would end up torn to pieces.

> that's a great addition to the loader, and relatively easy to implement.

I like that idea too, simple and clean, I can't see any downsides.

uriel


P.S.: Unsurprisingly http://kencc.sf.net is nowhere to be seen in the SoC
site either... *sigh* hey, who knows, maybe in a couple of years more
we will have a website and a tarball for kencc!


> i think it is important to have an install floppy.  or throw out machines
> without CD drives.
>
> brucee
>
> On 4/19/06, Charles Forsyth <forsyth@terzarima.net> wrote:
>> > There were linkers written, long ago, that would extract only the code
>> > from a library that was used by a program, and the binary got pretty
>>
>> we do that in inferno's 5l.  richard miller implemented it originally in 5l
>> but the changes vanished with the business unit and we redid it years later,
>> and perhaps not in quite the same way (i don't know).

Re: [9fans] Install from CD fails

[Bruce Ellis]

boyd would take out the machine pistol and wipe out a few pigeons.
i know i did when they insisted on terror on my sleep.  shared
libraries are worse!

brucee

On 4/19/06, uriel@cat-v.org <uriel@cat-v.org> wrote:
> > grafting shared libraries into plan9 will make presto go into a rage.
>
> I was going to say that the day Plan 9 gets shared libs I would shot
> myself.  But given how many fans I have in 9fans that might have
> motivated someone into actually implementing shard libs.
>
> So maybe more persuasive is the fact that it would make boyd spin in his
> grave until the Earth would end up torn to pieces.
>
> > that's a great addition to the loader, and relatively easy to implement.
>
> I like that idea too, simple and clean, I can't see any downsides.
>
> uriel
>
>
> P.S.: Unsurprisingly http://kencc.sf.net is nowhere to be seen in the SoC
> site either... *sigh* hey, who knows, maybe in a couple of years more
> we will have a website and a tarball for kencc!
>
>
> > i think it is important to have an install floppy.  or throw out machines
> > without CD drives.
> >
> > brucee
> >
> > On 4/19/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> >> > There were linkers written, long ago, that would extract only the code
> >> > from a library that was used by a program, and the binary got pretty
> >>
> >> we do that in inferno's 5l.  richard miller implemented it originally in 5l
> >> but the changes vanished with the business unit and we redid it years later,
> >> and perhaps not in quite the same way (i don't know).
>
>

Re: [9fans] Install from CD fails

[Richard Miller]

>> There were linkers written, long ago, that would extract only the code
>> from a library that was used by a program, and the binary got pretty
>
> we do that in inferno's 5l.  richard miller implemented it originally in 5l

There may be a small misunderstanding here.

The change I made in 5l was to discard all unreachable code, whether
it came from a library object file or from the main program.  It was
mostly useful for tidying up programs which had been evolving for
a while, when the authors hadn't bothered to delete functions which
were no longer used.  But that wouldn't help with what Russ described:

> Much of the bloat is from library routines
> that don't actually get called in practice
> but end up linked into every binary anyway
> (like the formatters for %e, %f, and %g).

The floating point format routines are always reachable if you use any
functions of the print() family, because they're in the list of built-in
formats (/sys/src/libc/fmt/fmt.c:39,41).  There's no way to determine
statically at link time that they won't be called at runtime, because
format strings might be constructed dynamically or even read in.

There was a hack in some early Unixes to solve this particular
instance of the problem: the C compiler would generate an external
reference to pull in a "special" version of printf from the library --
the one with calls to floating-point conversion routines -- if any
floating-point instructions were generated during the compilation.  If
none of your program's object files contained this reference, then you
would get the integer-only printf.  This was in 1978 or so when memory
footprint really made a difference -- e.g. we ran a timesharing lab
with 7 terminals on an Interdata minicomputer with 192 kilobytes of
RAM and two 5-megabyte hard disks.

-- Richard

Re: [9fans] Install from CD fails

[Charles Forsyth]

> The change I made in 5l was to discard all unreachable code, whether
> it came from a library object file or from the main program.  It was
> mostly useful for tidying up programs which had been evolving for
> a while, when the authors hadn't bothered to delete functions which
> were no longer used.  But that wouldn't help with what Russ described:

it turned out to do more than that, for quite sensible programs.
it's now quite common to have object files in libraries that provide a set of
functions, only some of which are actually used by any particular program.
indeed, given a linker that works that way, it becomes more attractive
to move beyond `one function per file' and collect things in a more modular way.
you've mentioned one case that it wouldn't eliminate (functions referenced
through a static array).  another interesting case turned up in Tk and a few
other places where there were functions for use by debugging code
that wasn't compiled in, and so with the reachable-code change,
those functions were eliminated.

Re: [9fans] Install from CD fails

[geoff]

The hack Richard refers to was that, at least on the PDP-11, the C
compiler emitted a reference to a symbol called `_fltused' if and only
if a floating-point variable, function or constant was seen when
compiling a given source file (as I recall).  _fltused was defined
only in the file containing the %[efg] output conversions for printf.
The PDP-11 implementation relied on having a dummy version of the FP
routines later in the library (and upon the loader observing ordering
within a library) so that the generic printf could call the %[efg]
routines, but today we could probably do that with a function pointer
(nil if no %[efg] routines loaded, so don't call them).  It seemed to
me to be a simple and elegant hack that we could revive if the space
consumption really matters.

Re: [9fans] Install from CD fails

[Richard Miller]

> (nil if no %[efg] routines loaded, so don't call them).  It seemed to
> me to be a simple and elegant hack that we could revive if the space
> consumption really matters.

It's not clear that floating point print routines are worth
singling out any more.  Here's a list of all the library object
files with multiple copies occupying cumulatively more than
100000 bytes of text space in commands in /386/bin.  The three
columns of numbers are #copies, text size, and cumulative text size.

132   787  103884 /sys/src/libc/9sys/convM2D.c
 60  1738  104280 /sys/src/libc/9sys/qlock.c
 32  3302  105664 /sys/src/libc/9sys/fcallfmt.c
 13  9070  117910 /sys/src/lib9p/srv.c
209   600  125400 /sys/src/libc/port/frexp.c
  7 18078  126546 /sys/src/libsec/port/x509.c
 31  4830  149730 /sys/src/libdraw/init.c
  7 22109  154763 /sys/src/libsec/port/tlshand.c
  7 22626  158382 /sys/src/libmemdraw/draw.c
209   805  168245 /sys/src/libc/fmt/fmt.c
189  1598  302022 /sys/src/libc/port/malloc.c
209  2453  512677 /sys/src/libc/fmt/fltfmt.c
209  3006  628254 /sys/src/libc/port/strtod.c
209  3888  812592 /sys/src/libc/386/vlrt.c
209  5078 1061302 /sys/src/libc/fmt/dofmt.c
189  9957 1881873 /sys/src/libc/port/pool.c

If I was looking for ways to reduce space I would probably
start with vlrt.c and pool.c

-- Richard

Re: [9fans] Install from CD fails

[Charles Forsyth]

> If I was looking for ways to reduce space I would probably
> start with vlrt.c and pool.c

the quick linker changes i did this morning discard various
commonly unused functions in vlrt.c, and sometimes in pool.c.
it saves 12% on text and data for something simple like cat.c but as one might
expect it saves less and less as a percentage of
bigger things like plan 9 and inferno kernels (eg 1.4% or so).
i haven't tabulated savings overall.

a few represent obsolete things that could simply be removed,
but in quite a few cases (eg, atexit and canlock below) the relevant source
files have several functions (often to use internal functions and data marked static)
and in any given load some of them aren't used (eg, rc4skip and rc4back in rc4.c).
on the other hand, partly because of the simple-minded implementation,
it adds a bit to the linking time when used, notably on the kernels.

here's the list for cat.c
	atexit unused
	atexitdont unused
	canlock unused
	atoi unused
	_d2v unused
	_f2v unused
	_v2d unused
	_v2f unused
	vneg unused
	_divv unused
	_modv unused
	_rshav unused
	_rshlv unused
	_lshv unused
	_andv unused
	_orv unused
	_xorv unused
	_vpp unused
	_vmm unused
	_ppv unused
	_mmv unused
	_p2v unused
	_sl2v unused
	_ul2v unused
	_si2v unused
	_ui2v unused
	_sh2v unused
	_uh2v unused
	_sc2v unused
	_uc2v unused
	_v2sc unused
	_v2uc unused
	_v2sh unused
	_v2uh unused
	_v2sl unused
	_v2ul unused
	_v2si unused
	_v2ui unused
	_testv unused
	_eqv unused
	_nev unused
	_ltv unused
	_lev unused
	_gtv unused
	_gev unused
	_lov unused
	_lsv unused
	_hiv unused
	_hsv unused
	runenlen unused
	fmtfdflush unused
	fmtinstall unused
	_mulv unused
	_addv unused
	toupper unused
	modf unused
	muldiv unused
	$0.0 unused (data)
but that amounts to only 2844 bytes overall of the original 23110.
by the time you get to the kernels, you're talking real savings of, ooh, 30k
out of 1.7mbytes (text+data+bss).  it's slightly more helpful with inferno
than plan 9, because the kernel includes more library code, and sometimes that
includes functions that could be called to dump data structures when debugging
but aren't referenced normally.

Re: [9fans] Install from CD fails

[Dave Eckhardt]

> Shared libraries would help a lot here.

Available data seem to suggest it might be a pyrrhic victory.

There is a venerable hack which I think I saw in the BSD
world (but which is probably older than that) by which
N program's main()'s are renamed to be foo_main() and
everything is linked together into one huge executable
with a main() which dispatches based on argv[0].  It's
crufty but the win is you end up with 1 printf() instead
of N.

Dave Eckhardt

Re: [9fans] Install from CD fails

[Lyndon Nerenberg]

> There is a venerable hack which I think I saw in the BSD
> world

http://www.freebsd.org/cgi/man.cgi?query=crunchgen

Re: [9fans] Install from CD fails

[Bakul Shah]

> > Installing from the latest CD (fetched today from tip9ug.jp)
> > fails in the copydist step with the following message
>
> I think I fixed this.  The fix is s!-c!-c /! in the command.
> There's a new CD on the Bell Labs site.

Thanks for the quick fix.

> In other news, the floppy distribution doesn't fit on
> a floppy anymore.

Alan Shugart has a lot to answer for -- his circular magtape
is still wasting lots of time...

Does it make sense to move less frequently used bits to a 2nd
disk rather than take heroic measures to create a TARDIS[1]?

-- bakul

[1] TARDIS is a registered trademark of the BBC.

Re: [9fans] Install from CD fails

[Russ Cox]

> Does it make sense to move less frequently used bits to a 2nd
> disk rather than take heroic measures to create a TARDIS[1]?

I'm sure I can squeeze everything back onto a single disk
but I'm busy right now.  In a week or two I will.  It's bad enough
we have one disk.  I don't want two.

Russ