[TUHS] Unix stories

[TUHS] Unix stories

[Larry McVoy]

Inspired by:

> Stephen Bourne after some time wrote a cron job that checked whether an 
> update in a binary also resulted in an updated man page and otherwise 
> removed the binary. This is why these programs have man pages.

I want to tell a story about working at Sun.  I feel like I've sent this
but I can't find it in my outbox.  If it's a repeat chalk it up to old 
age.

I wanted to work there, they were the Bell Labs of the day, or as close
as you could get.

I got hired as a contractor through Lachman (anyone remember them?) to do
POSIX conformance in SunOS (the 4.x stuff, not that Solaris crap that I
hate).

As such, I was frequently the last guy to touch any file in the kernel,
my fingerprints were everywhere.  So when there was a panic, it was
frequently laid at my doorstep.

So here is how I got a pager and learned about source management.

Sun had two guys, who will remain nameless, but they were known as
"the SCSI twins".  These guys decided, based on feedback that "people
can interrupt sun install", to go into the SCSI tape driver and disable
SIGINT, in the driver.  The kernel model doesn't allow for drivers messing
with your signal mask so on exit, sometimes, we would get a "panic: psig".

Somehow, I sure was because of the POSIX stuff, I ended up debugging this
panic.  It had nothing to with me, I'm not a driver person (I've written
a few but I pretty much suck at them), but it landed in my lap.

Once I figured it out (which was not easy, you had to hit ^C to trigger it
so unless you did that, and who does that during an install) I tracked down
the code to SCSI twins.

No problem, everyone makes mistakes.  Oh, wait.  Over the next few months
I'm tracking down more problems, that were blamed on me since I'm all over
the kernel, but came from the twins.

Suns integration machines were argon, radon, and krypton.  I wrote
scripts, awk I think, that watched every update to the tree on all
of those machines and if anything came from the SCSI twins the script
paged me.

That way I could go build and test that kernel and get ahead of the bugs.
If I could fix up their bugs before the rest of the team saw it then I 
wouldn't get blamed for them.

I wish I could have figured out something like Steve did that would have
made them not screw up so much but this was the next best thing.  I actually
got bad reviews because of their crap.  My boss at the time, Eli Lamb, just
said "you are in kadb too much".  

--lm

[TUHS] Unix stories

[Nick Downing]

I can never emphasize enough how much damage it does to just get people in
on a contract and let them scribble all over the company's valuable IP.
Soon they are gone but the damage lives on making the real programmers'
lives very difficult. Luckily the thing that took the contract programmer
months to do can usually be redone by the real programmer in a day or two,
but if it's been released it takes careful planning and opportunism to get
all the breakage out of the system. This was my life when I worked on cash
registers since software was basically a pimple on the side of hardware, my
bosses didn't understand software (or worse, thought they understood it and
hence trivialized it) so this would happen a lot, the software was seen as
more of a vehicle to get the hardware where it needed to go, rather than an
end in itself. Working there and dealing with customer complaints and
overseeing the expansion of what was originally a few BASIC programs to
print a report of the day's transactions on the receipt printer... to a
thousands and thousands of lines of code so the cash registers could
participate in multiple networks, download software and price updates,
report on takings and performances of different categories, order stock,
track till balances etc etc etc... really taught me a MASSIVE respect for
code quality, I almost never meet anyone who cares as much about code
quality and careful analysis of the system's assumptions and invariants
(like the assumption about drivers modifying a process' s signal mask in
your example)... as I do.

I can remember a conversation I had with a new hire in my research group
later on (not cash registers this time)... this dude had a PhD and had
written a hugely successful open source package that is still standard
today for a lot of courses etc in our field... and was hired to rewrite
some similar stuff created by our research group that was a bit of a dogs
breakfast but nevertheless was in daily use and publicly disseminated. Well
I had hacked on this dude's code a lot and I hated it, way overcomplicated
and using a very awkward structure of millions of interdependent C++
templates and what-have-you. He showed me his progress after some months
and I showed him a competing implementation (very immature) that I had put
together in my summer holiday using Python. So I tried to sell him the idea
of doing it in Python and structuring it all for simplicity and
maintainability... he was not having it. I could see his code would rapidly
descend into a dogs breakfast as soon as it was used to solve any real
world problem, because he was repeating all the same mistakes as in his
open source package.

So fast forward 5 years or so and he has a useable system, it is in daily
use and is being publicly disseminated. It is not too bad, until one looks
under the hood. I used it as a module in one of my major research tools and
it is great that it's available, BUT, it falls over miserably when you
stray away from the normal standard use cases that his group have tested
and made to work by extensive layers of band-aid fixes, leaving the code in
an incomprehensible state. I would spend days debugging and then send him a
comprehensive report on my investigation including several proposals for
short and long term fixes, he was initially enthusiastic about this but
lately my reports get labelled "won't fix" with weak excuses about it being
outside the common use cases. "Can't fix" would be more accurate. In the
process of all this I looked at the changelogs for the releases. In the
past 3 years there were a couple of feature releases and about 30 bugfix
releases, each accompanied by a release note which just kind of casually
passes this off as no big deal and implies the code is approaching a
reliable state. Ha!!

By contrast in May/June this year I decided to enter my tool in a
competition run by my research group and open to outside entrants, I think
about 20 groups entered including 3 or 4 internal entries like mine. Well
my tool was far from perfect since I had embarked on a major rewrite of the
frontend some months earlier and it was hard to produce anything working at
all, let alone competition quality. Luckily I had help from the competition
organizer, since internal entries are not eligible for prizes he was happy
to alert me to any problems he found and let me submit fixes if it did not
mess up his schedule. Well he found quite a few issues and I fixed them and
ended up having the fastest and best tool in the competition even though it
was not eligible for prizes. But now to the point of the story: The
CHARACTER of the problems he found. So much do I care about code quality
that it turns out most of the problems amounted to basically an oversight,
a misplaced comma that was hard to see, a pointer violation that occurred
because a realloc had moved some data during the evaluation of an
expression, that sort of thing. The fix never required any significant
restructuring of the code, except in one case where it was basically caused
by my using that other broken software as a module and I had to work around
it. I am so happy that my basic asusumptions and algorithms turned out to
be robust, because this means that after some period of getting all the
typos and minor oversights out, I will have a tool that is close to perfect
despite its complexity and the things I still plan to refactor and rewrite.
The guys who do not understand code quality will never experience this.

cheers, Nick

On 01/01/2017 4:00 PM, "Larry McVoy" <lm at mcvoy.com> wrote:

> Inspired by:
>
> > Stephen Bourne after some time wrote a cron job that checked whether an
> > update in a binary also resulted in an updated man page and otherwise
> > removed the binary. This is why these programs have man pages.
>
> I want to tell a story about working at Sun.  I feel like I've sent this
> but I can't find it in my outbox.  If it's a repeat chalk it up to old
> age.
>
> I wanted to work there, they were the Bell Labs of the day, or as close
> as you could get.
>
> I got hired as a contractor through Lachman (anyone remember them?) to do
> POSIX conformance in SunOS (the 4.x stuff, not that Solaris crap that I
> hate).
>
> As such, I was frequently the last guy to touch any file in the kernel,
> my fingerprints were everywhere.  So when there was a panic, it was
> frequently laid at my doorstep.
>
> So here is how I got a pager and learned about source management.
>
> Sun had two guys, who will remain nameless, but they were known as
> "the SCSI twins".  These guys decided, based on feedback that "people
> can interrupt sun install", to go into the SCSI tape driver and disable
> SIGINT, in the driver.  The kernel model doesn't allow for drivers messing
> with your signal mask so on exit, sometimes, we would get a "panic: psig".
>
> Somehow, I sure was because of the POSIX stuff, I ended up debugging this
> panic.  It had nothing to with me, I'm not a driver person (I've written
> a few but I pretty much suck at them), but it landed in my lap.
>
> Once I figured it out (which was not easy, you had to hit ^C to trigger it
> so unless you did that, and who does that during an install) I tracked down
> the code to SCSI twins.
>
> No problem, everyone makes mistakes.  Oh, wait.  Over the next few months
> I'm tracking down more problems, that were blamed on me since I'm all over
> the kernel, but came from the twins.
>
> Suns integration machines were argon, radon, and krypton.  I wrote
> scripts, awk I think, that watched every update to the tree on all
> of those machines and if anything came from the SCSI twins the script
> paged me.
>
> That way I could go build and test that kernel and get ahead of the bugs.
> If I could fix up their bugs before the rest of the team saw it then I
> wouldn't get blamed for them.
>
> I wish I could have figured out something like Steve did that would have
> made them not screw up so much but this was the next best thing.  I
> actually
> got bad reviews because of their crap.  My boss at the time, Eli Lamb, just
> said "you are in kadb too much".
>
> --lm
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20170101/7d161d25/attachment.html>

[TUHS] Unix stories

[Ron Natalie]

> That way I could go build and test that kernel and get ahead of the bugs.
> If I could fix up their bugs before the rest of the team saw it, then I
wouldn't get blamed for them.

Ugh.   I had a programmer who was sent over from the other side of our
company to help out.   The guy was a complete dolt and broke more than he
fixed.    Mostly, he never really caught on to how dynamic memory worked in
C and was always trashing the malloc areas.  So I just backed out every
change he made (at least we had source code control going at that point, one
of the first things I insisted happen when I took over).     Finally, I was
on my way in to fire him when I found he had quit an hour previously (fine,
I love it when that works out).    Unfortunately, he checked in all his
"work in progress" which didn't even compile.   Backed all that out.

Several years later I get a hold of a tool called Purify which finds memory
leaks (among other things) In your code.    I find a piece of code written
by one our better programmers that predate the source code control system
that's leaking memory.   This can't be.    Look through the edit history and
there is one edit, my former programmer.   He's deleted the free() call in
the routine without explanation.   Obviously, he had corrupted the malloced
area one day, and it crashed in the subsequent free in this routine, so he
just deleted it.

Bad programmers can hurt you for a good long time after they leave.    The
only other guy I had to fire at least had done absolutely nothing over his
tenure, so we were unaffected in the long run.

[TUHS] Unix stories

[Steve Johnson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]


These stories certainly rang true to me.  I think it's interesting to
pose the question, to be a bit more contemporary, "What do we need to
do to make open source code higher quality?"  I think the original
arguments (that open source would be high quality because everybody
would read the code and fix the bugs) has a bit of validity.  But,
IMHO, it is swamped by the typical in-coherence of the software.

It seems to me to be glaringly obvious that if you add a single on/off
option to a program, and don't want the quality of the code to
decrease, you should _a_ _priori_ double the amount of testing you do
before it releasing it.  If you have a carefully designed program
with multiple phases with firewalls between them, you might be able to
get away with only 10 or 20% more testing.

So look at gcc with nearly 600 lines in the man page describing just
the _names_ of the options...   It seems obvious that the actual
amount of testing of these combinations is a set of measure 0 in the
space of all possible invocations.  And the observed fact is that if
you try to do something at all unusual, no matter how reasonable it
may seem, it is likely to fail.  Additionally, it is kind sad to see
the same functionality (e.g., increasing the default stack size) is
done so differently on different OS's.  Even within Linux, setting
the locale (at least at one time) was quite different from Linux to
Linux.

And I think you can argue that gcc is a success story...

But how much longer can this go on?  What can we do to fight the
exponential bloat of this and other open-souce projects.  All ideas
cheerfully entertained...

Steve


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20170101/c26dc7c5/attachment.html>

[TUHS] Unix stories

[Nick Downing]

This is something I think about a lot. And I can't really say how
adding an option to gcc (for example) should be tested. In fact I'm a
bit lazy when it comes to adding test cases even though I recognize
their validity. Programming by test case is a good idea, but one
problem is the setup cost, for a new project I tend to just write a
two-line Makefile and start coding, and I really want to keep doing
this. Once I start adding a testcase infrastructure then the Makefile
is a lot bigger (make tests and so forth) and relies on external tools
etc. Simplicity really can be better, I mean once you start using
tools like cmake/smake, ant, or (THE HORRORS) maven, it's a time
waste.

But what I really want to say here, is a lot of the bloat arises
because different ecosystems have different ways of doing things. For
instance the object model. Well C++ has an object model. And C has an
object model (based on void * pointers, lots of libraries export this
kind of object model including FILE * in stdio). And gnome has an
object model. And OpenOffice/StarOffice defines its own object model
which is similar to Microsoft's COM. And so on and so forth. So, much
as I like the democracy of an open ecosystem where everyone can define
a competing way of doing the same thing and de facto standards
develop... my own plan is a little bit different.

What I want to do is go back to a REALLY SIMPLE unbloated system,
which is why I am very interested in 2.11BSD (you probably saw my
earlier posts about the 2.11BSD system and potential port to Z180 and
so on). And then I want to define the ONE TRUE WAY(TM) of doing each
thing. But before I do this I want to go right back to basics and look
at the object model used in the operating system itself. For instance
the stuff like the oft (open file table), inode table, the filesystem
table (I mean Sun VFS which isn't in 2.11BSD AFAIK but eventually
should be), the device table and so on. And also the user-visible
objects like files, sockets etc which map to kernel objects.

So once I have sorted all this out and created an object model that
the kernel can use efficiently, with compiler support (like C++
without the bloat, like java with internal pointers and ability to get
at the bits and bytes of your objects, like C without all the void *
stuff and with automatic handling of stuff like vtables), and
converted the kernel and all drivers to use it, then I think I will
have an object model that is useful in practice. So my idea then is to
export it to userlevel, so that userland programs can be rewritten
into this new C-like object oriented language and calls like: count =
read(fd, buf, size) would change to: count = fd.read(buf, size) since
kernel objects are objects. There would also be a compatibility layer
that allows you to keep a table of integer fds and their kernel
objects in userspace for porting.

After this I would start to look at popular libraries like the C
library or the X-Window system, and convert them to use the new object
model, while also providing the compatibility layer as I did for the
kernel interface. Ultimately the result would be a bit like Java, but
using all the familiar Unix objects and familiar Unix calling
conventions (such as argument passing by reference or malloc/memcpy
stuff that Java can't do). Also without any header files or
boilerplate of any description, which is one of my pet peeves with
Java, C, C++ etc.

I really think that the solution to bloat is to go through and rewrite
everything to do things in a more standardized way with more reuse.
Also I think that the massive amount of bloat arises to some extent
because the environment lends itself to writing non maintainable code
(for example you have to write loads of boilerplate and synchronize
function definitions in various places, which discourages you from
changing a function definition even if that's the right thing to do in
a situation). So there's always the temptation to add another
compatibility layer rather than dealing with the bloat. Rewriting
things in a much more minimal and maintainable style is the answer.

Another reason for bloat is that authors have to support millions of
slightly different systems. My idea is to totally standardize it, like
POSIX but much more drastically so. Think about Java, it defines a
strict virtual machine so there's nothing to change when you port your
code to another platform. I haven't totally decided how to handle
word-size issues in this context, but I am sure there is a way.

Did i mention I also have plans for world domination? :)

cheers, Nick


On Mon, Jan 2, 2017 at 1:03 PM, Steve Johnson <scj at yaccman.com> wrote:
>
> These stories certainly rang true to me.  I think it's interesting to pose
> the question, to be a bit more contemporary, "What do we need to do to make
> open source code higher quality?"  I think the original arguments (that open
> source would be high quality because everybody would read the code and fix
> the bugs) has a bit of validity.  But, IMHO, it is swamped by the typical
> in-coherence of the software.
>
> It seems to me to be glaringly obvious that if you add a single on/off
> option to a program, and don't want the quality of the code to decrease, you
> should a priori double the amount of testing you do before it releasing it.
> If you have a carefully designed program with multiple phases with firewalls
> between them, you might be able to get away with only 10 or 20% more
> testing.
>
> So look at gcc with nearly 600 lines in the man page describing just the
> names of the options...   It seems obvious that the actual amount of testing
> of these combinations is a set of measure 0 in the space of all possible
> invocations.  And the observed fact is that if you try to do something at
> all unusual, no matter how reasonable it may seem, it is likely to fail.
> Additionally, it is kind sad to see the same functionality (e.g., increasing
> the default stack size) is done so differently on different OS's.  Even
> within Linux, setting the locale (at least at one time) was quite different
> from Linux to Linux.
>
> And I think you can argue that gcc is a success story...
>
> But how much longer can this go on?  What can we do to fight the exponential
> bloat of this and other open-souce projects.  All ideas cheerfully
> entertained...
>
> Steve
>

[TUHS] Unix stories

[Steve Nickolas]

On Mon, 2 Jan 2017, Nick Downing wrote:

> What I want to do is go back to a REALLY SIMPLE unbloated system,
> which is why I am very interested in 2.11BSD (you probably saw my
> earlier posts about the 2.11BSD system and potential port to Z180 and
> so on). And then I want to define the ONE TRUE WAY(TM) of doing each
> thing. But before I do this I want to go right back to basics and look
> at the object model used in the operating system itself. For instance
> the stuff like the oft (open file table), inode table, the filesystem
> table (I mean Sun VFS which isn't in 2.11BSD AFAIK but eventually
> should be), the device table and so on. And also the user-visible
> objects like files, sockets etc which map to kernel objects.
>
> So once I have sorted all this out and created an object model that
> the kernel can use efficiently, with compiler support (like C++
> without the bloat, like java with internal pointers and ability to get
> at the bits and bytes of your objects, like C without all the void *
> stuff and with automatic handling of stuff like vtables), and
> converted the kernel and all drivers to use it, then I think I will
> have an object model that is useful in practice. So my idea then is to
> export it to userlevel, so that userland programs can be rewritten
> into this new C-like object oriented language and calls like: count =
> read(fd, buf, size) would change to: count = fd.read(buf, size) since
> kernel objects are objects. There would also be a compatibility layer
> that allows you to keep a table of integer fds and their kernel
> objects in userspace for porting.
>
> After this I would start to look at popular libraries like the C
> library or the X-Window system, and convert them to use the new object
> model, while also providing the compatibility layer as I did for the
> kernel interface. Ultimately the result would be a bit like Java, but
> using all the familiar Unix objects and familiar Unix calling
> conventions (such as argument passing by reference or malloc/memcpy
> stuff that Java can't do). Also without any header files or
> boilerplate of any description, which is one of my pet peeves with
> Java, C, C++ etc.
>
> I really think that the solution to bloat is to go through and rewrite
> everything to do things in a more standardized way with more reuse.
> Also I think that the massive amount of bloat arises to some extent
> because the environment lends itself to writing non maintainable code
> (for example you have to write loads of boilerplate and synchronize
> function definitions in various places, which discourages you from
> changing a function definition even if that's the right thing to do in
> a situation). So there's always the temptation to add another
> compatibility layer rather than dealing with the bloat. Rewriting
> things in a much more minimal and maintainable style is the answer.
>
> Another reason for bloat is that authors have to support millions of
> slightly different systems. My idea is to totally standardize it, like
> POSIX but much more drastically so. Think about Java, it defines a
> strict virtual machine so there's nothing to change when you port your
> code to another platform. I haven't totally decided how to handle
> word-size issues in this context, but I am sure there is a way.

This vaguely reminds me of an idea I had a couple years ago, but never 
mentioned because I thought it might be perceived as daft.

I had the idea of writing a virtual machine specifically tailored to 
running C code, with a virtual operating system similar to Unix.  On the 
surface it would probably feel like running Unix under an emulator, but 
the design would probably be a bit less baroque, because everything would 
be designed to work together cleanly.

Like the mutant child of Unix and UCSD Pascal, I suppose.

-uso.

[TUHS] Unix stories

[Warren Toomey]

On Mon, Jan 02, 2017 at 01:01:32AM -0500, Steve Nickolas wrote:
> I had the idea of writing a virtual machine specifically tailored to running
> C code, with a virtual operating system similar to Unix.  On the surface it
> would probably feel like running Unix under an emulator, but the design
> would probably be a bit less baroque, because everything would be designed
> to work together cleanly.

I've been admiring Swieros: https://github.com/rswier/swieros
which sort of does this.

Cheers, Warren

[TUHS] Unix stories

[Nick Downing]

Yes, I agree, I want to do exactly that. And, I know my ideas are
probably also perceived as daft. But it's OK :)

Unfortunately a C interpreter does not really work in principle,
because of the fact you can jump into loops and other kinds of abuse.
So the correct execution model is a bytecode. This means the C
compiler is pretty much unchanged, so there are no real
simplifications there. Also, you can cast a pointer and access
individual bytes of a structure and that kind of abuse. So the data
model is pretty much unchanged from a real machine's too, and there
are no real simplifications there either. But in my opinion that's not
the end of the story.

What I would like to do is to implement pointer safety, I know this is
a pretty big ask and will break compatibility with a big number of C
applications out there. But it might be surprising how many actually
do use pointers in a safe way. So my idea was something like this:

Suppose there is "struct foo { char *p; int *q; long b; short a; };",
well the user wouldn't be allowed to access the individual bytes of
the pointers p and q. But they would be allowed to access the
individual bytes of a and b since this does not introduce anything
dangerous. Therefore the struct would be internally converted to
"struct  foo { char *p; int *q; char data[6] };" allowing 4 bytes for
the long and 2 for the short. Then, now suppose I call a function and
I pass it "&a". This would be internally converted to "data + 4". But
it would be passed as a triple (data, data + 4, data + 6) which gives
the bounds of the underlying array. Or suppose the virtual machine was
implemented in Java or Python or some such, it would be passed as a
pair (data, 4) where data is a 6-byte array and 4 is an integer index
into it.

I was then going to make the compiler recognize some idiomatic C
constructs like "struct foo *p = malloc(10 * sizeof(foo));" and
internally translate them to something like "struct foo *p = new
foo[10];". Hopefully there could eventually be discriminated unions so
that I could declare something like "union foo { char *p; int *q; long
b; short a; };" and this would be internally translated to "struct foo
{ int which; union { char *, int *, data[4] } };" and the field
"which" would be set to 0, 1 or 2 depending on whether a char *, an
int *, or plain old data had been stored there. That way, when you
access a given field of the union it can validate that the correct
thing was stored there earlier. Also, with a little bit of syntactic
sugar we could implement intptr_t as "union { void *p; int a; };" to
increase compatibility with C.

If we do it like this, then your P-system idea works quite well,
because the memory space doesn't exist anymore, it's an OO database.

cheers, Nick

On Mon, Jan 2, 2017 at 5:01 PM, Steve Nickolas <usotsuki at buric.co> wrote:
> On Mon, 2 Jan 2017, Nick Downing wrote:
>
>> What I want to do is go back to a REALLY SIMPLE unbloated system,
>> which is why I am very interested in 2.11BSD (you probably saw my
>> earlier posts about the 2.11BSD system and potential port to Z180 and
>> so on). And then I want to define the ONE TRUE WAY(TM) of doing each
>> thing. But before I do this I want to go right back to basics and look
>> at the object model used in the operating system itself. For instance
>> the stuff like the oft (open file table), inode table, the filesystem
>> table (I mean Sun VFS which isn't in 2.11BSD AFAIK but eventually
>> should be), the device table and so on. And also the user-visible
>> objects like files, sockets etc which map to kernel objects.
>>
>> So once I have sorted all this out and created an object model that
>> the kernel can use efficiently, with compiler support (like C++
>> without the bloat, like java with internal pointers and ability to get
>> at the bits and bytes of your objects, like C without all the void *
>> stuff and with automatic handling of stuff like vtables), and
>> converted the kernel and all drivers to use it, then I think I will
>> have an object model that is useful in practice. So my idea then is to
>> export it to userlevel, so that userland programs can be rewritten
>> into this new C-like object oriented language and calls like: count =
>> read(fd, buf, size) would change to: count = fd.read(buf, size) since
>> kernel objects are objects. There would also be a compatibility layer
>> that allows you to keep a table of integer fds and their kernel
>> objects in userspace for porting.
>>
>> After this I would start to look at popular libraries like the C
>> library or the X-Window system, and convert them to use the new object
>> model, while also providing the compatibility layer as I did for the
>> kernel interface. Ultimately the result would be a bit like Java, but
>> using all the familiar Unix objects and familiar Unix calling
>> conventions (such as argument passing by reference or malloc/memcpy
>> stuff that Java can't do). Also without any header files or
>> boilerplate of any description, which is one of my pet peeves with
>> Java, C, C++ etc.
>>
>> I really think that the solution to bloat is to go through and rewrite
>> everything to do things in a more standardized way with more reuse.
>> Also I think that the massive amount of bloat arises to some extent
>> because the environment lends itself to writing non maintainable code
>> (for example you have to write loads of boilerplate and synchronize
>> function definitions in various places, which discourages you from
>> changing a function definition even if that's the right thing to do in
>> a situation). So there's always the temptation to add another
>> compatibility layer rather than dealing with the bloat. Rewriting
>> things in a much more minimal and maintainable style is the answer.
>>
>> Another reason for bloat is that authors have to support millions of
>> slightly different systems. My idea is to totally standardize it, like
>> POSIX but much more drastically so. Think about Java, it defines a
>> strict virtual machine so there's nothing to change when you port your
>> code to another platform. I haven't totally decided how to handle
>> word-size issues in this context, but I am sure there is a way.
>
>
> This vaguely reminds me of an idea I had a couple years ago, but never
> mentioned because I thought it might be perceived as daft.
>
> I had the idea of writing a virtual machine specifically tailored to running
> C code, with a virtual operating system similar to Unix.  On the surface it
> would probably feel like running Unix under an emulator, but the design
> would probably be a bit less baroque, because everything would be designed
> to work together cleanly.
>
> Like the mutant child of Unix and UCSD Pascal, I suppose.
>
> -uso.

[TUHS] Unix stories

[arnold]

Nick Downing <downing.nick at gmail.com> wrote:

> What I want to do is go back to a REALLY SIMPLE unbloated system,

Ahem. This is not a new idea. Take a hard look at Plan 9 From Bell Labs,
where the original Unix guys decided that it was worth starting over.
They did A LOT of neat stuff.

There's a lot of overlap of readership between TUHS and 9fans (not
suprising); I'm suprised noone beat me to mentioning this.

But before totally reinventing the wheel, I'd strongly recommend looking
at what Ken et al did with Plan 9.

Arnold

P.S. And yes, we've gotten far off topic, I suppose. :-)

[TUHS] Unix stories

[Dave Horsfall]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

On Sun, 1 Jan 2017, Steve Johnson wrote:

> But how much longer can this go on?  What can we do to fight the 
> exponential bloat of this and other open-souce projects.  All ideas 
> cheerfully entertained...

Line up all the (ir)responsible programmers against a wall?  Computer 
programming is the only discipline I've seen where no formal 
qualifications are required; any idiot can call themselves a "programmer" 
if they've barely learned how to write in BASIC...

Yes, I have a BSc majoring in Computer Science and Mathematics (both pure 
and applied), so I guess I'm a bit elitist :-)

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

[TUHS] Unix stories

[Larry McVoy]

On Tue, Jan 03, 2017 at 09:52:51AM +1100, Dave Horsfall wrote:
> On Sun, 1 Jan 2017, Steve Johnson wrote:
> 
> > But how much longer can this go on??? What can we do to fight the 
> > exponential bloat of this and other open-souce projects.?? All ideas 
> > cheerfully entertained...
> 
> Line up all the (ir)responsible programmers against a wall?  

A buddy of mine at RedHat, in the early days, gaveth me this quote:

	"Average programmers should be rounded up and placed in 
	internment camps to keep their fingers away from keyboards."

That was ~20 years ago.  Not looking like anything has changed.

--lm

[TUHS] Unix stories

[Ronald Natalie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

Johns Hopkins had no Computer Science department at the time I was a student there.   You came out with either a degree in EE or Math if you wanted to do computers.   Of course, due to circumstances, I came out of there with nearly four years of UNIX internals experience.    I also learned in the school of hard knocks about decent code design.

I’ve generally found that having a degree in computer science didn’t indicate that the applicant had any qualification.    I look for prior experience (or at least some sort of indication of a programming experience).    Within 90 days I could generally tell if people were worth retaining (reasonable skills or ability to learn at least).

_Ron

[TUHS] Unix stories

[Ronald Natalie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

I once threatened to break one of my programmers fingers if he continued to make the same mistake.

Another programmer I told him I was going to remove the copy/paste facility from his text editor if he didn’t stop just COPYING the program over and over, rather than  using some common code.

[TUHS] Unix stories

[Tim Bradshaw]

If you think open source is bad you haven't seen much closed-source software, because a lot of it is deeply terrible.  I claim that all large software systems which are not designed to be used by naive users are shit (and most systems which are are also shit).

> On 2 Jan 2017, at 02:03, Steve Johnson <scj at yaccman.com> wrote:
> 
> But how much longer can this go on?  What can we do to fight the exponential bloat of this and other open-souce projects.  All ideas cheerfully entertained...

[TUHS] Unix stories

[Larry McVoy]

It's simply a lack of craftsmen level thinking.  Managers think that people
are fungable, whatever that means, I think it means anyone can do this work.

That's simply not the case, some people get the big picture and the details
and some people don't.

There is also a culture of the cool kids vs the not cool kids.  For example,
at Sun, the kernel group was the top of the heap.  When I was doing nselite
which begat Teamware then BitKeeper then Git et al, I was in the kernel 
group.  They wanted me to go over to the tools group.  I looked over there
and saw people that weren't as good as the kernel people and passed.

Same thing with testing.  So many bad test harnesses.  Because testing
isn't respected so they get the crappy programmers.  One of the best
things I did at BitKeeper was to put testing on the same level as the 
code and the documentation.  As a result, we have a kick ass testing
harness.  Here's a sample t.file in our test harness:

    echo $N Check that component rename is blocked ......................$NL
    nested product
    cd "$HERE/product"
    bk mv gcc gccN 2>ERR && fail -f ERR
    cat <<EOF >WANT
    mvdir: gcc is a component
    Component renaming is unsupported in this release
    EOF
    cmpfiles WANT ERR
    echo OK

The harness, which is all open source under the apache license at
bitkeeper.org, let's you write tiny shell scripts, they either echo
OK (pass), (bug) (pass but not really), or anything else and they 
fail.

The harness allows for parallel runs.  In the early days we ran the tests
in alpha order, later we got a huge speed up by running them in most recently
modified order (fastest time to failure).

All of that is the result of me (semi good, setting it up) and Wayne Scott
(crazy good, he made it work in parallel and bunch of other improvements)
being the guys who worked on it.  We're $250K/year people on average.



On Mon, Jan 02, 2017 at 11:23:47PM +0000, Tim Bradshaw wrote:
> If you think open source is bad you haven't seen much closed-source software, because a lot of it is deeply terrible.  I claim that all large software systems which are not designed to be used by naive users are shit (and most systems which are are also shit).
> 
> > On 2 Jan 2017, at 02:03, Steve Johnson <scj at yaccman.com> wrote:
> > 
> > But how much longer can this go on?  What can we do to fight the exponential bloat of this and other open-souce projects.  All ideas cheerfully entertained...

-- 
---
Larry McVoy            	     lm at mcvoy.com             http://www.mcvoy.com/lm 

[TUHS] Unix stories

[Joerg Schilling]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Larry McVoy <lm at mcvoy.com> wrote:

> There is also a culture of the cool kids vs the not cool kids.  For example,
> at Sun, the kernel group was the top of the heap.  When I was doing nselite
> which begat Teamware then BitKeeper then Git et al, I was in the kernel 
> group.  They wanted me to go over to the tools group.  I looked over there
> and saw people that weren't as good as the kernel people and passed.

From looking at various changes, it is obvious that the tools group also had 
brilliant people, but it is also obvious that the tools in general do not have
that overall quality level as seen in the kernel.

From looking at projects or features, it seems that the kernel people did add 
new ideas whenever they had the idea. From looking at the progress made in the 
tools, it seems that this mostly happened when a manager decided that it would 
be time to put effort in a tool.

Jörg

-- 
 EMail:joerg at schily.net                  (home) Jörg Schilling D-13353 Berlin
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/

[TUHS] Unix stories

[Steve Nickolas]

On Mon, 2 Jan 2017, Nick Downing wrote:

> Yes, I agree, I want to do exactly that. And, I know my ideas are
> probably also perceived as daft. But it's OK :)
>
> Unfortunately a C interpreter does not really work in principle,
> because of the fact you can jump into loops and other kinds of abuse.
> So the correct execution model is a bytecode.

I was mainly looking at UCSD Pascal as a model, which iirc uses a bytecode 
optimized for Pascal.  C isn't too different so the major difference is 
probably its implementation of pseudothreads/multitasking, which I don't 
think UCSD Pascal supported (at least on the Apple ][ where it was 
actually somewhat common).

Well, there's also the issue of UCSD Pascal having a really braindead 
filesystem (even MS-DOS 1.25 was more advanced).

I guess the question would be, what precisely would be needed for a 
reasonably simple Unix-alike (I suppose V7 would be a good initial 
target?) to be run on such, and then I'd have to figure out how to come up 
with the system from that.  Not exactly my strong point =P

That said, it would perhaps result in being able to run a (slow) 
Unix-alike on smaller systems which might not otherwise be able to run 
such an OS because of CPU limitations - like the Apple //e I have 
collecting dust off in the back room. ;)

-uso.

[TUHS] Unix stories

[Steffen Nurpmeso]

Larry McVoy <lm at mcvoy.com> wrote:
 |It's simply a lack of craftsmen level thinking.  Managers think that people
 |are fungable, whatever that means, I think it means anyone can do this \
 |work.
 |
 |That's simply not the case, some people get the big picture and the details
 |and some people don't.
 |
 |There is also a culture of the cool kids vs the not cool kids.  For \
 |example,
 |at Sun, the kernel group was the top of the heap.  When I was doing nselite
 |which begat Teamware then BitKeeper then Git et al, I was in the kernel 
 |group.  They wanted me to go over to the tools group.  I looked over there
 |and saw people that weren't as good as the kernel people and passed.
 |
 |Same thing with testing.  So many bad test harnesses.  Because testing
 |isn't respected so they get the crappy programmers.  One of the best
 |things I did at BitKeeper was to put testing on the same level as the 
 |code and the documentation.  As a result, we have a kick ass testing

I had the thought this being standardized, as part of ISO 9001?
Groups of three which iterate in a rotation over the three parts
of a codebase, documentation / implementation / tests, also
rotating internally, inside the group?  And having some student
satellites.  Or atoms and electrons that form molecules, to be
more bionic.

I think much of the grief is owned to money, a.k.a. short-living
interests.  I think it is crucial, and very likely even essential
for survival, that there are Universities where people have the
possibility to linger badly paid in dark caves for years and
years, to finally come to a spiral conclusion, so to say.
If you doom a young human or student to spend a month doing
nothing but writing tests, you surely do no good, and gain the
same in return, and that only probably not in short-term.
Donald E. Knuth really got that right in his TeXbook, with those
funny homework excercises for over the weekend, say, "write an
operating system".

Laziness is another problem.  I for one dislike a lot what C now
is, where one has to write functions like explicit_bzero()
/ _memset() in order to avoid that compilers optimize away
something that seems to "go away anyway" at the end of a scope.
Or code blow due to automatic inlining and loop unroll.  Or
terrible aliasing and "sequence point" rules, where i think it is
clear what i mean when i write "i = j + ++i" (i believe this is
undefined behaviour).

Explicit instrumentation via new language constructs would require
more man power than paying a standard gremium to define semantics
which effectively allow more compiler optimization and gain more
performance and thus a sellable catchphrase, but on the long term
this surely soils the ground on which we stand.

I for one maintain a codebase that has grown over now almost four
decades and i still cannot say i stand on grounds of pureness,
beauty and elegance; attributes which, were possible, brighten up
everydays work and make a day.

--steffen

[TUHS] Unix stories

[Random832]

On Wed, Jan 4, 2017, at 08:04, Steffen Nurpmeso wrote:
> terrible aliasing and "sequence point" rules, where i think it is
> clear what i mean when i write "i = j + ++i" (i believe this is
> undefined behaviour).

I assume you're imagining it as being equivalent to i = j + i + 1, with
a redundant store operation.

But why couldn't it equally well mean
i = 0; i += j; i+= ++i
i = 0; i += j; i += (i += 1)

If an architecture were to the most efficient way to assign an additive
expression to a variable to zero it out and add each successive operand
to it.

The example seems contrived, because it's honestly hard to make a
reasonable-sounding case for the prefix operators, and my usual go-to
examples require postfix operators and/or pointers. But to be fair, your
example is contrived too; why wouldn't you just do i += j + 1? But for a
better example, I was in a discussion a couple weeks ago with someone
who thought it was clear what they meant by an expression like this:

*a++ = *a

And not only was I easily able to come up with two reasonable-looking
implementations where it means different things, I guessed wrong on
which one they thought it should mean. My examples were stack-based
architectures with a "store to address" instruction taking operands in
each of the two possible orders, making it natural to evaluate either
the RHS or the address-of-LHS first. A more realistic register-based
architecture with pipelining might make it more efficient to evaluate
one or the other first, or parts of each [assuming more complex
expressions than in the example] mixed together, depending on the exact
data dependencies in both expressions.

> Explicit instrumentation via new language constructs would require
> more man power than paying a standard gremium to define semantics
> which effectively allow more compiler optimization and gain more
> performance and thus a sellable catchphrase, but on the long term
> this surely soils the ground on which we stand.
> 
> I for one maintain a codebase that has grown over now almost four
> decades and i still cannot say i stand on grounds of pureness,
> beauty and elegance; attributes which, were possible, brighten up
> everydays work and make a day.
> 
> --steffen

[TUHS] Unix stories

[Ron Natalie]

> I assume you're imagining it as being equivalent to i = j + i + 1, with a redundant store operation.

It's what the language standard specifies, not imagination.  C and C++ state that modifying twice between sequence points or using the value other than to compute the value for a store is undefined behavior.
The languages put no constraint on what may happen when you do this.

[TUHS] Unix stories

[Random832]

On Wed, Jan 4, 2017, at 09:54, Ron Natalie wrote:
> > I assume you're imagining it as being equivalent to i = j + i + 1, with a redundant store operation.
> 
> It's what the language standard specifies, not imagination.  C and C++
> state that modifying twice between sequence points or using the value
> other than to compute the value for a store is undefined behavior.
> The languages put no constraint on what may happen when you do this.

But I'm talking about the alternate universe in which the person I was
replying to is justified in thinking that it's clear what he means, vs a
'plausible' implementation that could arise from methods of translating
expressions into machine operations (since people don't tend to respond
to "it's undefined because it is, and the compiler can arbitrarily mess
things up because it's allowed to by the fact that it's undefined"
without a plausible theory of why something might ever behave in a way
other than the obvious way)

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 08:04, Steffen Nurpmeso wrote:
 |> terrible aliasing and "sequence point" rules, where i think it is
 |> clear what i mean when i write "i = j + ++i" (i believe this is
 |> undefined behaviour).
 |
 |I assume you're imagining it as being equivalent to i = j + i + 1, with
 |a redundant store operation.
 |
 |But why couldn't it equally well mean

No i don't, and the thing is that it could definetely not equally
mean anything.  That is exactly the point.

I skip quite a lot here.
  ...
 |example is contrived too; why wouldn't you just do i += j + 1? But for a
 |better example, I was in a discussion a couple weeks ago with someone
 |who thought it was clear what they meant by an expression like this:
 |
 |*a++ = *a
 |
 |And not only was I easily able to come up with two reasonable-looking
 |implementations where it means different things, I guessed wrong on
 |which one they thought it should mean. My examples were stack-based
 |architectures with a "store to address" instruction taking operands in

So if we agree that a high level language should abstract such
problems from the programmer, with a right hand side that is
evaluated and stored in the target of the left hand side, then all
is fine.

 |each of the two possible orders, making it natural to evaluate either
 |the RHS or the address-of-LHS first. A more realistic register-based
 |architecture with pipelining might make it more efficient to evaluate
 |one or the other first, or parts of each [assuming more complex
 |expressions than in the example] mixed together, depending on the exact
 |data dependencies in both expressions.

It is all right by me.  "Can't say that on the radio."

--steffen

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 09:54, Ron Natalie wrote:
 |>> I assume you're imagining it as being equivalent to i = j + i + \
 |>> 1, with a redundant store operation.
 |> 
 |> It's what the language standard specifies, not imagination.  C and C++
 |> state that modifying twice between sequence points or using the value
 |> other than to compute the value for a store is undefined behavior.
 |> The languages put no constraint on what may happen when you do this.
 |
 |But I'm talking about the alternate universe in which the person I was
 |replying to is justified in thinking that it's clear what he means, vs a
 |'plausible' implementation that could arise from methods of translating
 |expressions into machine operations (since people don't tend to respond
 |to "it's undefined because it is, and the compiler can arbitrarily mess
 |things up because it's allowed to by the fact that it's undefined"
 |without a plausible theory of why something might ever behave in a way
 |other than the obvious way)

It is clear in assembler, and C was ment, as i understand it, as
a higher-level portable abstraction of assembler.  Which alternate
universe do you refer to?

--steffen

[TUHS] Unix stories

[Random832]

On Wed, Jan 4, 2017, at 11:30, Steffen Nurpmeso wrote:
> It is clear in assembler

The operation you described does not exist as a single-statement
construct in assembler for any architecture I'm familiar with.

> , and C was ment, as i understand it, as
> a higher-level portable abstraction of assembler.  Which alternate
> universe do you refer to?
> 
> --steffen

[TUHS] Unix stories

[Random832]

On Wed, Jan 4, 2017, at 11:22, Steffen Nurpmeso wrote:
> Random832 <random832 at fastmail.com> wrote:
>  |On Wed, Jan 4, 2017, at 08:04, Steffen Nurpmeso wrote:
>  |> terrible aliasing and "sequence point" rules, where i think it is
>  |> clear what i mean when i write "i = j + ++i" (i believe this is
>  |> undefined behaviour).
>  |
>  |I assume you're imagining it as being equivalent to i = j + i + 1, with
>  |a redundant store operation.
>  |
>  |But why couldn't it equally well mean
> 
> No i don't,

Then I guessed wrong. Again. (So much for "clear", I suppose). But
you're the one who "think[s] it's clear what [you] mean by it"; so you
simply *must* have a meaning in mind. Why not explain what it is?

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 11:30, Steffen Nurpmeso wrote:
 |> It is clear in assembler
 |
 |The operation you described does not exist as a single-statement
 |construct in assembler for any architecture I'm familiar with.

Ok, but that quite clearly was not what i have meant.  I meant
that if you program in assembler, well, all those newer assembler
languages that i have seen, the target of an operation is the
target of a store, and say if it is a register that is also one of
the sources, it means nothing, from the language side.  ARM has
even predicates that perform operations on that value before the
store, even if the source is the same as the destination.  It
simply strives me absurd that i, in C, cannot simply say what
i want and let the C compiler with all its knowledge of the target
system decide what to do about it.

--steffen

[TUHS] Unix stories

[Random832]

On Wed, Jan 4, 2017, at 11:51, Steffen Nurpmeso wrote:
> Ok, but that quite clearly was not what i have meant.  I meant
> that if you program in assembler, well, all those newer assembler
> languages that i have seen, the target of an operation is the
> target of a store, and say if it is a register that is also one of
> the sources, it means nothing, from the language side.

Yes but you are storing *twice*, two different values, to the same
variable, in the same statement. There's no operation in any assembler
language that does that, and at this point I honestly don't know what
value you expect to 'win'.

>  ARM has
> even predicates that perform operations on that value before the
> store, even if the source is the same as the destination.  It
> simply strives me absurd that i, in C, cannot simply say what
> i want

Why do you think that "i = ... + ++i" is a reasonable way to say what
you want?

> and let the C compiler with all its knowledge of the target
> system decide what to do about it.
> 
> --steffen

[TUHS] Unix stories

[Ron Natalie]

There's a trademark between allowing the compiler to reorder things and having a defined order of operations.
Steps like that are well-defined in Java for instance.   C lets the compiler do what it sees fit.

Note that it's not necessarily any better in assembler.    There are RISC architectures where load-followed-by-store and vice versa may not always be valid if done in quick succession.    Requiring the compiler to insert sequence points typically wastes a lot of cycles.    Assembler programmers tend to think about what they are doing, the C compiler tries to do some of this on its own and its not clairvoyant.

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 11:22, Steffen Nurpmeso wrote:
 |> Random832 <random832 at fastmail.com> wrote:
 |>|On Wed, Jan 4, 2017, at 08:04, Steffen Nurpmeso wrote:
 |>|> terrible aliasing and "sequence point" rules, where i think it is
 |>|> clear what i mean when i write "i = j + ++i" (i believe this is
 |>|> undefined behaviour).
 |>|
 |>|I assume you're imagining it as being equivalent to i = j + i + 1, with
 |>|a redundant store operation.
 |>|
 |>|But why couldn't it equally well mean
 |> 
 |> No i don't,
 |
 |Then I guessed wrong. Again. (So much for "clear", I suppose). But
 |you're the one who "think[s] it's clear what [you] mean by it"; so you
 |simply *must* have a meaning in mind. Why not explain what it is?

Hey.  I was a football (we play it with the feet) goalkeeper, and
i can assure you i can trump louder than many alike.

--steffen

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 11:51, Steffen Nurpmeso wrote:
 |> Ok, but that quite clearly was not what i have meant.  I meant
 |> that if you program in assembler, well, all those newer assembler
 |> languages that i have seen, the target of an operation is the
 |> target of a store, and say if it is a register that is also one of
 |> the sources, it means nothing, from the language side.
 |
 |Yes but you are storing *twice*, two different values, to the same
 |variable, in the same statement. There's no operation in any assembler
 |language that does that, and at this point I honestly don't know what
 |value you expect to 'win'.

Hm.  Yet this is exactly what i want?  (Hihi.  Don't be offended,
i really have already forgotten the example.  It was something
like "*i = j + *i++" or the like..)

 |>  ARM has
 |> even predicates that perform operations on that value before the
 |> store, even if the source is the same as the destination.  It
 |> simply strives me absurd that i, in C, cannot simply say what
 |> i want
 |
 |Why do you think that "i = ... + ++i" is a reasonable way to say what
 |you want?

Man, i write it down, and it even stands several code iterations?
That must be it, then!

 |> and let the C compiler with all its knowledge of the target
 |> system decide what to do about it.

Ciao.

--steffen

[TUHS] Unix stories

[Steffen Nurpmeso]

"Ron Natalie" <ron at ronnatalie.com> wrote:
 |There's a trademark between allowing the compiler to reorder things \
 |and having a defined order of operations.
 |Steps like that are well-defined in Java for instance.   C lets the \
 |compiler do what it sees fit.
 |
 |Note that it's not necessarily any better in assembler.    There are \
 |RISC architectures where load-followed-by-store and vice versa may \
 |not always be valid if done in quick succession.    Requiring the compiler \
 |to insert sequence points typically wastes a lot of cycles.    Assembler \
 |programmers tend to think about what they are doing, the C compiler \
 |tries to do some of this on its own and its not clairvoyant.

I have just read again Clive Feather's ISO/IEC JTC1/SC22/WG14 N925
draft on sequence points, and i seem to be wrong about especially
the shown exampl,e and Random knew that earlier.  I first read
that document in the context of aliasing issues a few years back,
when i saw some BSD changesets fly by, and i remember a thread on
a FreeBSD list, too, where objects backing pointers could no
longer be accessed directly, but first need to be copied over to
some -- then newly introduced -- local scope storage before being
used, because of new aliasing rules of the C language.  It seems
i hyperventilated in the sequence point document back then.

--steffen

[TUHS] Unix stories

[Steffen Nurpmeso]

And Random was sorted out of Cc:, i'm digging into it ;)

--steffen

[TUHS] Unix stories

[Steve Johnson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2813 bytes --]

Let me contaminate this philosophical discussion with some history.

Long long ago, computers were slow and didn't have much memory. 
Because C was targeting system code, it was important to make things
run efficiently.  And the PDP-11 had autoincrement and
autodecrement  hardware.
Early machines also had some kinds of memory management, but most
specified a base and limit.  DEC allowed you to protect the end of a
block of memory, which made it possible to grow the stack backwards
and still be able to add more stack space if you ran out.  But many
other machines required that the stack grow upwards.
The problem this caused was when you had
    foo( f(), g() )
In backward-growing stacks, the most efficient thing was to call g
first, then f.
In forward-growing stacks, the most efficient thing was to call f
first, then g.
For whatever reason, Dennis decided that efficiency on a particular
architecture was more important than consistency, so when f() and g()
had side effects, their order was undefined.
Autoincrement and Autodecrement also got tarred by the same brush:
     foo( *p++, *p++ )
had a slew of "correct" implementations, including ones where p was
incremented twice AFTER the call of foo had returned.  The situation
became critical when getc() was implemented as a macro that pulled
bytes out of an I/O buffer and used autoincrement to do so.  After
some discussion, what I implemented in PCC was that all side effects
of an argument must be carried out before the next argument was
evaluated.  This still didn't solve the argument order problem, but
it did cut down the space of astonishing surprises.

These rules provided rich fodder for Lint, when it came along,
although the function side effect issue was beyond its ken.

Steve

----- Original Message -----
From: "Ron Natalie" <ron@ronnatalie.com>
To:"Random832" <random832 at fastmail.com>, "Steffen Nurpmeso"
<steffen at sdaoden.eu>
Cc:<tuhs at minnie.tuhs.org>
Sent:Wed, 4 Jan 2017 11:58:41 -0500
Subject:Re: [TUHS] Unix stories

 There's a trademark between allowing the compiler to reorder things
and having a defined order of operations.
 Steps like that are well-defined in Java for instance. C lets the
compiler do what it sees fit.

 Note that it's not necessarily any better in assembler. There are
RISC architectures where load-followed-by-store and vice versa may not
always be valid if done in quick succession. Requiring the compiler to
insert sequence points typically wastes a lot of cycles. Assembler
programmers tend to think about what they are doing, the C compiler
tries to do some of this on its own and its not clairvoyant.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20170104/06487f69/attachment.html>

[TUHS] Unix stories

[Steffen Nurpmeso]

Hello,

Random832 <random832 at fastmail.com> wrote:
 |On Wed, Jan 4, 2017, at 11:22, Steffen Nurpmeso wrote:
 |> Random832 <random832 at fastmail.com> wrote:
 |>|On Wed, Jan 4, 2017, at 08:04, Steffen Nurpmeso wrote:
 |>|> terrible aliasing and "sequence point" rules, where i think it is
 |>|> clear what i mean when i write "i = j + ++i" (i believe this is
 |>|> undefined behaviour).
 |>|
 |>|I assume you're imagining it as being equivalent to i = j + i + 1, with
 |>|a redundant store operation.
 |>|
 |>|But why couldn't it equally well mean
 |> 
 |> No i don't,
 |
 |Then I guessed wrong. Again. (So much for "clear", I suppose). But
 |you're the one who "think[s] it's clear what [you] mean by it"; so you
 |simply *must* have a meaning in mind. Why not explain what it is?

so now i really got this a few minute ago after adding negative
history number support (to count from history top):

  tty.c: In function 'c_history':
  tty.c:4157:13: warning: operation on 'entry' may be undefined [-Wsequence-point]
         entry = isneg ? --entry : (siz_t)a_tty.tg_hist_size - entry;
         ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

And in my opinion this is just plain terrible?  I think it is
absolutely clear what i intend, there is not even a dereferenced
pointer involved, the lhv is either a register or if all fails
a stack location.  I don't understand why i have to write

      if(isneg)
         --entry;
      else
         entry = (siz_t)a_tty.tg_hist_size - entry;

to get over this, it is exactly the same?  And it is not that easy
as blaming the compiler (except that gcc often comes over with
terrible warnings, but this is a different issue, and of course,
these are huge codebases and i guess you stumble along and fix the
test warnings that occur when a step turned on some red lights
somewhere else; ok, but while i am in rant mode, i recently had to
introduce a useless enum to turn

 #ifdef HAVE_DEVEL /* Avoid gcc warn cascade since n_ignore is defined locally */
 -n_CTAV(-(uintptr_t)n_IGNORE_TYPE - n__IGNORE_ADJUST == 0);

into

  +n_CTAV(-n__IGNORE_TYPE - n__IGNORE_ADJUST == 0);

where n_IGNORE_TYPE was '#define n_IGNORE_TYPE ((struct
n_ignore*)-3)', and if that is not an integer then, what).  
No, in my opinion this is because of overly construed standard
text of an increasingly unclean standard.  Digging in the dirt is
my whole life anyway, even i don't read newspapers, and i really
would like to see C not becoming just an equal soup of muddy
waters.  Just my one cent.

--steffen

[TUHS] Unix stories

[Random832]

On Thu, Feb 9, 2017, at 08:46, Steffen Nurpmeso wrote:
> so now i really got this a few minute ago after adding negative
> history number support (to count from history top):
> 
>   tty.c: In function 'c_history':
>   tty.c:4157:13: warning: operation on 'entry' may be undefined
>   [-Wsequence-point]
>          entry = isneg ? --entry : (siz_t)a_tty.tg_hist_size - entry;
>          ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> And in my opinion this is just plain terrible?  I think it is
> absolutely clear what i intend, there is not even a dereferenced
> pointer involved, the lhv is either a register or if all fails
> a stack location.  I don't understand why i have to write
> 
>       if(isneg)
>          --entry;
>       else
>          entry = (siz_t)a_tty.tg_hist_size - entry;
> 
> to get over this, it is exactly the same?

What's wrong with entry = isneg ? entry-1 : (siz_t)a_tty.tg_hist_size -
entry;

Same number of characters (two more if you put spaces around the minus
sign, but hardly a huge burden in any case).

You're basically asking for the standard to carve out an exception for
the cases, and precisely only those cases, where the meaning can be seen
to be 100% unambiguous (i.e. that the two values being assigned to a
variable are provably the same value, and there are no other reads) -
which would limit it exclusively to the prefix operator (and assignment
operators, I suppose, "x = x += 1" is as unambiguous as it is
pointless), and only when there is no other expression involved except
for the conditionals (you couldn't have "--entry + x", for example).

[TUHS] Unix stories

[Steffen Nurpmeso]

Random832 <random832 at fastmail.com> wrote:
 |On Thu, Feb 9, 2017, at 08:46, Steffen Nurpmeso wrote:
 |> so now i really got this a few minute ago after adding negative
 |> history number support (to count from history top):
 |> 
 |>   tty.c: In function 'c_history':
 |>   tty.c:4157:13: warning: operation on 'entry' may be undefined
 |>   [-Wsequence-point]
 |>          entry = isneg ? --entry : (siz_t)a_tty.tg_hist_size - entry;
 |>          ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 |> 
 |> And in my opinion this is just plain terrible?  I think it is
 ...
 |What's wrong with entry = isneg ? entry-1 : (siz_t)a_tty.tg_hist_size -
 |entry;

Another example:

  spam.c: In function '_spam_rate2score':
  spam.c:1137:38: warning: to be safe all intermediate pointers in cast from 'char **' to 'const char **' must be 'const' qualified [-Wcast-qual]
      ids = n_idec_ui32_cp(&m, buf, 10, (char const**)&buf);

To satisfy we need in fact a temporary variable, since you cannot
apply cast operators on non-lvalues:

  spam.c: In function '_spam_rate2score':
  spam.c:1137:38: error: lvalue required as unary '&' operand
      ids = n_idec_ui32_cp(&m, buf, 10, &(char const*)buf);

And yeah, you need to do something about:

  spam.c: In function '_spam_rate2score':
  spam.c:1137:38: warning: passing argument 6 of 'n_idec_buf' from incompatible pointer type [-Wincompatible-pointer-types]
      ids = n_idec_ui32_cp(&m, buf, 10, &buf);
                                        ^
  nailfuns.h:319:63: note: in definition of macro 'n_idec_ui32_cp'
      n_idec_buf(RP, CBP, UIZ_MAX, B, (n_IDEC_MODE_LIMIT_32BIT), CLP)
                                                                 ^~~
  nailfuns.h:303:22: note: expected 'const char **' but argument is of type 'char **'
   FL enum n_idec_state n_idec_buf(void *resp, char const *cbuf, uiz_t clen,

I would buy that the other way around, i.e., if "buf" would be
constant and the function expects something non-constant.

 |Same number of characters (two more if you put spaces around the minus
 |sign, but hardly a huge burden in any case).

Ok you're right, this special case is not a huge burden.
With a good font l-1-i may also be no problem, but could.  I mean,
some people even use garbage collectors because they are afraid of
memory holes and such, and then such l-1-i problems which are
known to outperform human visual capabilities except at eleven
thirty in the morning are good to go?  This!  Come on.

 |You're basically asking for the standard to carve out an exception for
 |the cases, and precisely only those cases, where the meaning can be seen
 |to be 100% unambiguous (i.e. that the two values being assigned to a
 |variable are provably the same value, and there are no other reads) -
 |which would limit it exclusively to the prefix operator (and assignment
 |operators, I suppose, "x = x += 1" is as unambiguous as it is
 |pointless), and only when there is no other expression involved except
 |for the conditionals (you couldn't have "--entry + x", for example).

Oh yes, i do.  To me the above basically looks like
a reassignment, a creation of a completely new variable.  The
value is assigned after it has been computed.  The compiler may be
clever enough to realize that in certain cases this can end up as
something like "inc eax" or the like.
But all this surely off-topic.

--steffen

[TUHS] Unix stories

[Dan Cross]

On Mon, Jan 2, 2017 at 9:30 AM, Doug McIlroy <doug at cs.dartmouth.edu> wrote:

> > In one of his books, Wirth laments about programmers proudly
> > showing him terrible code written in Pascal
>
> For your amusement, here's Wirth himself committing that sin:
>
> http://www.cs.dartmouth.edu/~doug/wirth.pdf
>

Doug, I'm trying to understand the provenance of that paper. It appears to
be a variant of a part of CSTR #155, but it is not the same as "Ellipses
Not Yet Made Easy" from that report. It also appears that this particular
document is a scan; is it perhaps a pre-print version of the paper? Or
perhaps it was expanded from what was published in #155?

        - Dan C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20170102/fdc8c8e3/attachment.html>

[TUHS] Unix stories

[Doug McIlroy]

> In one of his books, Wirth laments about programmers proudly
> showing him terrible code written in Pascal

For your amusement, here's Wirth himself committing that sin:

http://www.cs.dartmouth.edu/~doug/wirth.pdf

[TUHS] Unix stories

[Wesley Parish]

Don't we usually express that as "You can write FORTRAN in any language"?

Wesley Parish

Quoting Nemo <cym224 at gmail.com>:

> On 1 January 2017 at 11:50, Noel Chiappa <jnc at mercury.lcs.mit.edu>
> wrote (in part):
> > Which just goes to provide support for my long-term contention, that
> language
> > features can't help a bad programmer, or prevent them from writing
> garbage.
> 
> Indeed. In one of his books, Wirth laments about programmers proudly
> showing him terrible code written in Pascal and goes on to say the
> same thing.
> 
> N.
>  



"I have supposed that he who buys a Method means to learn it." - Ferdinand Sor,
Method for Guitar

"A verbal contract isn't worth the paper it's written on." -- Samuel Goldwyn

[TUHS] Unix stories

[Nemo]

On 1 January 2017 at 11:50, Noel Chiappa <jnc at mercury.lcs.mit.edu>
wrote (in part):
> Which just goes to provide support for my long-term contention, that language
> features can't help a bad programmer, or prevent them from writing garbage.

Indeed. In one of his books, Wirth laments about programmers proudly
showing him terrible code written in Pascal and goes on to say the
same thing.

N.

[TUHS] Unix stories

[Noel Chiappa]

    > From: Nick Downing

    > way overcomplicated and using a very awkward structure of millions of
    > interdependent C++ templates and what-have-you.
    > ...
    > the normal standard use cases that his group have tested and made to
    > work by extensive layers of band-aid fixes, leaving the code in an
    > incomprehensible state.

Which just goes to provide support for my long-term contention, that language
features can't help a bad programmer, or prevent them from writing garbage.

Sure, you can take away 'goto' and other dangerous things, and add a lot of
things that _can_ be used to write good code (e.g. complete typing and type
checking), but that doesn't mean that a user _will_ write good code.

I once did a lot of work with an OS written in a macro assembler, done by
someone really good. (He'd even created macros to do structure declarations!)
It was a joy to work with (very clean and simple), totally bug-free; and very
easy to change/modify, while retaining those characteristics. (I modified the
I/O system to use upcalls to signal asynchronous I/O completion, instead of
IPC messages, and it was like falling off a log.)

Thinking we can provide programming tools/languages which will make good
programmers is like thinking we can provide sculpting equipment which will
make good sculptors.


I don't, alas, have any suggestions for what we _can_ do to make good
programmers. It may be impossible (like making good sculptors - they are born,
not made).

I do recall talking to Jerry Saltzer about system architects, and he said
something to the effect of 'we can run this stuff past students, and some of
them get it, and some don't, and that's about all we can do'.

     Noel