Re: [TUHS] Unix APIs: elegant or not?

Re: [TUHS] Unix APIs: elegant or not?

[Theodore Y. Ts'o]

On Sun, 4 Nov 2018, Chris Hanson wrote:
> Every piece of code that wants to call, say, read(2) needs to handle
> not only real errors but also needs to special-case EINTR and retry
> the read. Thus you should virtually never use read(2), only ever
> something like this:
> ...
> And do this for every classic system call, since virtually no client
> code should ever have to care about EINTR. It was early an
> implementation expediency that became API and that everyone now has
> to just deal with because you can’t expect the system call interface
> you use to do this for you.
>
>This is the sort of wart that should’ve been fixed by System V and/or BSD 4 at latest.

But it *was* fixed in BSD, and it's in POSIX as the SA_RESTART flag to
sigaction (which gives you BSD signal semantics).

POSIX supports both the original V7 and BSD signal semantics, because
by then there were programs which expected system calls to be
interrupted by signals (and to be fair, there are times when that's
the more convenient way of handling an interrupt, as opposed to using
setjump/longjump to break out of a restartable system call).

					- Ted

P.S.  The original implementation of ERESTARTSYS / ERESTARTNOHAND /
ERESTARTNOINTR errno handling in Linux's system call return path was
my fault.  :-)

Re: [TUHS] Unix APIs: elegant or not?

[Donald ODona]

At 4 Nov 2018 22:07:46 +0000 (+00:00) from Chris Hanson <cmhanson@eschatologist.net>:
> And do this for every classic system call, since virtually no client code should ever have to care about EINTR. It was early an implementation expediency that became API and that everyone now has to just deal with because you can’t expect the system call interface you use to do this for you.
> 
> This is the sort of wart that should’ve been fixed by System V and/or BSD 4 at latest.
> 
you are right!

Re: [TUHS] Unix APIs: elegant or not?

[Dave Horsfall]

On Sun, 4 Nov 2018, Noel Chiappa wrote:

> "Virtually". Maybe there are places that want to know if their read call 
> was interrupted; if you don't make this version available to them, how 
> can they tell? Leaving the user as much choice as possible is the only 
> way to go, IMO; why force them to do it the way _you_ think is best?

I certainly don't want the kernel making that decision for me; I might 
want to abandon the entire read() if I was signalled.  I did write some 
code that did that, but I've long since forgotten the details.

> And it makes the OS simpler; any time you can move functionality out of 
> the OS, to the user, that's a Good Thing, IMO. There's nothing stopping 
> people from using the EINTR-hiding wrapper. (Does the Standard I/O 
> library do this, does anyone know?)

Agreed; the kernel should make as few decisions as possible, and I've used 
variations on that wrapper, depending on the signal etc.

> PS: Only system calls that can block can return EINTR; there are quite a 
> few that don't, not sure what the counts are in modern Unix.

Almost invariably serial lines, and I used to do a lot of that code.

-- Dave

Re: [TUHS] Unix APIs: elegant or not?

[Noel Chiappa]

    > From: Chris Hanson

    > you should virtually never use read(2), only ever something like this:
    > ...
    > And do this for every classic system call, since virtually no client
    > code should ever have to care about EINTR.

"Virtually". Maybe there are places that want to know if their read call was
interrupted; if you don't make this version available to them, how can they
tell? Leaving the user as much choice as possible is the only way to go, IMO;
why force them to do it the way _you_ think is best?

And it makes the OS simpler; any time you can move functionality out of the
OS, to the user, that's a Good Thing, IMO. There's nothing stopping people
from using the EINTR-hiding wrapper. (Does the Standard I/O library do this,
does anyone know?)

	Noel

PS: Only system calls that can block can return EINTR; there are quite a few
that don't, not sure what the counts are in modern Unix.

Re: [TUHS] Unix APIs: elegant or not?

[Chris Hanson]

On Nov 4, 2018, at 4:28 AM, arnold@skeeve.com wrote:
> 
> Chris Hanson <cmhanson@eschatologist.net> wrote:
> 
>> If anyone ever asks me about the elegance of UNIX, there’s one word, or really term, that springs to mind: EINTR.
>> 
>> The fact that man pages on modern systems still describe calls as returning -1 with errno set to EINTR baffles me.
> 
> Can you explain this some more?  This sounds like a claim that UNIX
> isn't elegant.  But I'm not following whatever it is you're saying.

Every piece of code that wants to call, say, read(2) needs to handle not only real errors but also needs to special-case EINTR and retry the read. Thus you should virtually never use read(2), only ever something like this:

    ssize_t safe_read(int fd, void *buf, size_t buf_size)
    {
        ssize_t rc;
        do {
            rc = read(fd, buf, buf_size);
        } while ((rc == -1) && (errno == EINTR));
        return rc;
    }

And do this for every classic system call, since virtually no client code should ever have to care about EINTR. It was early an implementation expediency that became API and that everyone now has to just deal with because you can’t expect the system call interface you use to do this for you.

This is the sort of wart that should’ve been fixed by System V and/or BSD 4 at latest.

  — Chris

Re: [TUHS] Unix APIs: elegant or not?

[ron minnich]

On Sun, Nov 4, 2018 at 9:06 AM Warner Losh <imp@bsdimp.com> wrote:
> Every other snowflake device had it's own config program that you had to run (or ioctls your data collection programs had to do). Setting up the sampling rates for the A2D, or the signal strength for an IRIG generator or whatever.


I'm not saying that having a name for everything solved the world's
problems. I'm saying that in Unix, as of the mid 1970s, things could
be named, including devices. And if you weren't using the other
systems available at the time, it's almost impossible to appreciate
just how important that was. I also used contemporary systems such as
OS/MVS, where even in the mid 70s we were accessing data sets by using
a disk and block #.

Although there is an analogy: do you prefer to name your personal
machine 2130706433 or localhost?

ron

Re: [TUHS] Unix APIs: elegant or not?

[Dave Horsfall]

On Sun, 4 Nov 2018, Chris Hanson wrote:

> The fact that man pages on modern systems still describe calls as 
> returning -1 with errno set to EINTR baffles me.

Err, how else would you do it?  You do understand the purpose of EINTR, 
don't you?

-- Dave

Re: [TUHS] Unix APIs: elegant or not?

[Bakul Shah]

> On Nov 4, 2018, at 2:53 AM, Chris Hanson <cmhanson@eschatologist.net> wrote:
> 
>> On Nov 1, 2018, at 3:20 AM, Dave Horsfall <dave@horsfall.org> wrote:
>> 
>> With Unix, everything looks like a file, but as I said, the sockets API broke that convention, forcing users to work at a much lower level than necessary.
> 
> This was broken from the start though, and always really meant everything looks like a file *descriptor*, not a path in the filesystem.

A “file” descriptor is very much like a capability, a handle.
This is separate from hierarchical naming intended for humans.
They are both needed. An ability to name something doesn’t
give you an automatic right to access it. And you don’t always
need to name something. For instance creating a pipe. The
third feature was capturing IO operations to a rather diverse set
of devices and files in a relatively simple interface. The fourth
feature that first attracted me first to Unix was its simple scheme
of major, minor device numbers. The “elegance” of Unix was that
you could get a lot done using a few abstractions. They were not
perfect but were good enough for the most part.

Re: [TUHS] Unix APIs: elegant or not?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 4825 bytes --]

On Sun, Nov 4, 2018 at 9:31 AM ron minnich <rminnich@gmail.com> wrote:

> On Sun, Nov 4, 2018 at 4:00 AM Chris Hanson <cmhanson@eschatologist.net>
> wrote:
>
> > This was broken from the start though, and always really meant
> everything looks like a file *descriptor*, not a path in the filesystem.
>
> OK, I only got into this game in 1976, and a lot had happened in Unix by
> then,
> and maybe you saw some earlier stuff. But certainly in 1976, as
> compared to the other 4 PDP-11 operating systems I was using,
> the fact that resources had names visible to every program was very
> important  to us. On the competitor systems the naming would be built
> into individual programs,
> e.g. PIP or (non-PDP11) the MPE fcopy program, one of the few programs
> on that system where you could name, e.g., the tape drive on your
> terminal.
>
> Being able to use a path name for resources was a very big deal for us
> at the time. And we didn't say file descriptors, we said names.
>
> Hence, I rate your comment as "interesting if true" but I see no
> evidence to support it.
>

It depends on what you did...

Tapes were always special. You could access them via a named interface, but
it was tricky. Tapes are stateful creatures, and some programs wanted
stateful, others wanted stateless. So you got different devices to do
different behavior. Easy to oops and use the rmt0 instead of the nrmt0
device. So while it was a file, it was a file that had odd state you didn't
have disks.

Serial ports were worse. There you had no way to control the serial port
except with fancy extra programs. There was no /dev/ttyd0.9600.8N1 devnode
to open. If you wanted 9600 baud, you had to either hack the driver to do
that on open, or you had to hack your programs to know they were talking to
a special thing and needed special code to cope (raw vs cooked was another
one). so, sure, it was file based, but not entirely file based as you had
to escape to either hacky system calls (v6 and earlier) or later
generalized ioctls (v7 and later).

Every other snowflake device had it's own config program that you had to
run (or ioctls your data collection programs had to do). Setting up the
sampling rates for the A2D, or the signal strength for an IRIG generator or
whatever.

So sure, you could address these devices by a file name, but you couldn't
"cp config /dev/irig" or "cat /dev/ttyd0.2400.7O2" to get data flowing on
the devices, so their addresses in the filesystem namespace were of limited
value since some files were more special than others.

I'm not sure plan-9 solved this by having a second device side-channel
either in its attempt to try a different approach to side-channel
communications than ioctl(2).

Network sockets are a logical conclusion to this state of affairs, btw. All
the plethera of new system calls for them could have been done as ioctls,
though the generic processing stuff for ioctls would have gotten in the way
of some of the socket API. It's another twist on the baud rate problem: I
need to communicate metadata about how to do things to the driver... And in
this case, a lot of the 'how to do things' is negotiated via a different
side channel (routed, ARP, etc) and so you no longer had a raw physical
network device to open, but a logical, cooked one that you interacted at
the top of the stack to and all those other details were abstracted out /
taken care of in other ways. It's unfortunate that you can't just open
"/dev/net/sri-nic.arpa" and have it do the right thing, but files kinda
want to be stateless and there's a lot of state about what you want to do
with that open command...

And while mounting allowed one to get files from a disk, it wasn't so good
at doing things dynamically (as the original system were quite static) and
required a number of ad-hoc hacks to make work right. block vs character
devices showed early on a crack in the uniformity.

So while I love the ideal, I also see how the practical side of things
forced us down the path we're down, ugly warts and all. I'd love to see
someone start with V7ish level of kernel, but add the modern things like
devfs (so one could pass 'state' strings to drivers on 'open' and have them
accept/reject them to get around some of these issues), as well as provide
all objects in a namespace that's sane and mostly enumerable. For that
you'd need new abstractions (like saying that all drivers don't interact
via cdev-nodes like today, but instead present a filesystem-like interface
so you'd open /dev/tty/d0/9600n1/hw-flow and the tty driver would get a
message for its d0 instance of 9600n1/hw-flow, though this would break
dirent(2)). In this world, you could have most of the crazy encoded into a
path for network operations as well... though I have doubts you could do it
in as a performant manner as sockets).

Warner

[-- Attachment #2: Type: text/html, Size: 5564 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[ron minnich]

On Sun, Nov 4, 2018 at 4:00 AM Chris Hanson <cmhanson@eschatologist.net> wrote:

> This was broken from the start though, and always really meant everything looks like a file *descriptor*, not a path in the filesystem.

OK, I only got into this game in 1976, and a lot had happened in Unix by then,
and maybe you saw some earlier stuff. But certainly in 1976, as
compared to the other 4 PDP-11 operating systems I was using,
the fact that resources had names visible to every program was very
important  to us. On the competitor systems the naming would be built
into individual programs,
e.g. PIP or (non-PDP11) the MPE fcopy program, one of the few programs
on that system where you could name, e.g., the tape drive on your
terminal.

Being able to use a path name for resources was a very big deal for us
at the time. And we didn't say file descriptors, we said names.

Hence, I rate your comment as "interesting if true" but I see no
evidence to support it.

ron

Re: [TUHS] Unix APIs: elegant or not?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

On Sun, Nov 4, 2018, 4:28 AM Chris Hanson <cmhanson@eschatologist.net wrote:

> If anyone ever asks me about the elegance of UNIX, there’s one word, or
> really term, that springs to mind: EINTR.
>
> The fact that man pages on modern systems still describe calls as
> returning -1 with errno set to EINTR baffles me.
>

Why? It's still a thing...

Warner

>

[-- Attachment #2: Type: text/html, Size: 888 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[arnold]

Chris Hanson <cmhanson@eschatologist.net> wrote:

> If anyone ever asks me about the elegance of UNIX, there’s one word, or really term, that springs to mind: EINTR.
>
> The fact that man pages on modern systems still describe calls as returning -1 with errno set to EINTR baffles me.
>
>   -- Chris
>

Can you explain this some more?  This sounds like a claim that UNIX
isn't elegant.  But I'm not following whatever it is you're saying.

Thanks,

Arnold

Re: [TUHS] Unix APIs: elegant or not?

[Chris Hanson]

On Nov 1, 2018, at 3:20 AM, Dave Horsfall <dave@horsfall.org> wrote:
> 
> With Unix, everything looks like a file, but as I said, the sockets API broke that convention, forcing users to work at a much lower level than necessary.

This was broken from the start though, and always really meant everything looks like a file *descriptor*, not a path in the filesystem.

There are tons of non-file descriptor namespaces on UNIX platforms. At least some later systems have tried to create a hierarchical “object” abstraction of which file descriptors are just one subtype.

  -- Chris

Re: [TUHS] Unix APIs: elegant or not?

[Chris Hanson]

If anyone ever asks me about the elegance of UNIX, there’s one word, or really term, that springs to mind: EINTR.

The fact that man pages on modern systems still describe calls as returning -1 with errno set to EINTR baffles me.

  -- Chris

Re: [TUHS] Unix APIs: elegant or not?

[Bakul Shah]

On Thu, 01 Nov 2018 09:48:38 -0700 ron minnich <rminnich@gmail.com> wrote:
>
> On Thu, Nov 1, 2018 at 9:44 AM Warner Losh <imp@bsdimp.com> wrote:
> >
> > There's another school of thought too that says the kernel has no business
> > parsing strings with all the security implications of doing so...

Which is more a function of C. A proper string type might've
helped. I would argue it is *easier* to debug string based
stuff than all those ioctls using magic numbers replaced with
#defines.

> And, further, that network architecture I referenced is much less efficient
> than the BSD model -- 5 system calls per accept! so that was starting to
> hit us here on the Akaros project, since we imported the entire Plan 9
> network stack into akaros, along with the drivers. Linux left us in the
> dust on network connections/second. We were going to change it had Akaros
> continued.

What sort of changes were you contemplating?

I view(ed) 9p more as a prototype.

Re: [TUHS] Unix APIs: elegant or not?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1850 bytes --]

On Thu, Nov 1, 2018 at 10:48 AM ron minnich <rminnich@gmail.com> wrote:

>
>
> On Thu, Nov 1, 2018 at 9:44 AM Warner Losh <imp@bsdimp.com> wrote:
>
>>
>>
>> There's another school of thought too that says the kernel has no
>> business parsing strings with all the security implications of doing so...
>>
>>
> yeah, it's an interesting question. But Unix is based on parsing strings.
> The kernel already parses lots of strings for paths, for all kinds of
> reasons, so adding another element in the kernel that uses paths seems
> acceptable.
>

Well, if the nose of a camel comes in under the tent, the rest of the camel
is sure to follow... Parsing a string path is rock simple, but are there
two arguments or three for this command, and what happens if they are too
long, or negative when you expect and fd, or or or. The CVE-archive is
littered with people who thought parsing in the kernel was simple and
easy...


> The bigger problem for Plan 9 that started to bite was i8n. All the Plan 9
> messages are English, oops. All the control messages are english. And so
> on.
>
> And, further, that network architecture I referenced is much less
> efficient than the BSD model -- 5 system calls per accept! so that was
> starting to hit us here on the Akaros project, since we imported the entire
> Plan 9 network stack into akaros, along with the drivers. Linux left us in
> the dust on network connections/second. We were going to change it had
> Akaros continued.
>

That's the other reason to not do stings in the kernel: parsing is takes
time.

I'll grant there's a balance here: A single int fd is nice so you don't
have to pass a pointer to the device call function and provides a level of
abstraction that blocks many bad data attacks. But even with this interface
there's been issues with improper range checks for some interfaces.

Warner

[-- Attachment #2: Type: text/html, Size: 2759 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Thu, Nov 1, 2018 at 9:44 AM Warner Losh <imp@bsdimp.com> wrote:

>
>
> There's another school of thought too that says the kernel has no business
> parsing strings with all the security implications of doing so...
>
>
yeah, it's an interesting question. But Unix is based on parsing strings.
The kernel already parses lots of strings for paths, for all kinds of
reasons, so adding another element in the kernel that uses paths seems
acceptable.

The bigger problem for Plan 9 that started to bite was i8n. All the Plan 9
messages are English, oops. All the control messages are english. And so
on.

And, further, that network architecture I referenced is much less efficient
than the BSD model -- 5 system calls per accept! so that was starting to
hit us here on the Akaros project, since we imported the entire Plan 9
network stack into akaros, along with the drivers. Linux left us in the
dust on network connections/second. We were going to change it had Akaros
continued.

ron

[-- Attachment #2: Type: text/html, Size: 1421 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Thu, Nov 1, 2018 at 9:38 AM Clem Cole <clemc@ccc.com> wrote:

>
>
> On Thu, Nov 1, 2018 at 10:20 AM ron minnich <rminnich@gmail.com> wrote:
>
>> In my view, what went wrong with Unix networking 40 years ago is that it
>> broke from the Unix model, i.e. that resources are accessed via path
>> names, and went with binary descriptors as paths.
>>
> Agreed.
>
> And I think somthing else where P9 differed from UNIX was dealing with OOB
> (control) information (*i.e.* ioctl(2) was a terrible misstake).   Dennis
> and Ken created ioctl(2) with v7 as a generalization of stty/gtty from the
> TTY handler.  At the time, it seemed like a reasonable way to handle those
> 'small things that need to be tweeked - like baud rate or canonicalization;
> but ioctl(2) quickly got abused as the universal end-around, and those
> things caused also sorts of issues (also being a binary interface only
> made it worse, although on the PDP-11 it made sense for size reasons).
>  Creating a seperate interface from the 'file' to orchestrate/control the
> I/O and controlling that as a set of strings not binaries, seems like a
> more sane idea.
>

There's another school of thought too that says the kernel has no business
parsing strings with all the security implications of doing so...

Warner


> ᐧ
>

[-- Attachment #2: Type: text/html, Size: 3209 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[Noel Chiappa]

    > From: Clem Cole

    > (probably because Larry Allen implemented both UNIX Chaos and Aegis IIRC).

Maybe there are two Larry Allen's - the one who did networking stuff at
MIT-LCS was Larry W. Allen, and I'm pretty sure he didn't do Unix CHAOS code
(he was part of our group at LCS, and we only did TCP/IP stuff; someone over
in EE had a Unix with CHAOS code at the time, so it pre-dated his time with
us).

	Noel

Re: [TUHS] Unix APIs: elegant or not?

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

On Thu, Nov 1, 2018 at 10:20 AM ron minnich <rminnich@gmail.com> wrote:

> In my view, what went wrong with Unix networking 40 years ago is that it
> broke from the Unix model, i.e. that resources are accessed via path
> names, and went with binary descriptors as paths.
>
Agreed.

And I think somthing else where P9 differed from UNIX was dealing with OOB
(control) information (*i.e.* ioctl(2) was a terrible misstake).   Dennis
and Ken created ioctl(2) with v7 as a generalization of stty/gtty from the
TTY handler.  At the time, it seemed like a reasonable way to handle those
'small things that need to be tweeked - like baud rate or canonicalization;
but ioctl(2) quickly got abused as the universal end-around, and those
things caused also sorts of issues (also being a binary interface only made
it worse, although on the PDP-11 it made sense for size reasons).
 Creating a seperate interface from the 'file' to orchestrate/control the
I/O and controlling that as a set of strings not binaries, seems like a more
sane idea.
ᐧ

[-- Attachment #2: Type: text/html, Size: 2579 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[ron minnich]

On Thu, Nov 1, 2018 at 7:05 AM Clem Cole <clemc@ccc.com> wrote:

>
> P9 took a different path still (pun intended).

as in: http://doc.cat-v.org/plan_9/4th_edition/papers/net/

It shows that you can use the pathname model for networks, And it
shows one way to get it right. There are others.

Here's a simple example of what get it right means:
there are *no* commands on Plan 9 that end in 6. No ping6, for
example. Plan 9 did not fall over sideways, as Unix did, when
infiiniband introduced 20 octet addresses, which are
the ip6 address and "more".

Just look at how many sockaddr_* there are Linux nowadays, including
those odd ones that have a name starting with a null byte. In Plan 9
there are 1, since a sockaddr in plan 9 is ... a string.

That's not to say plan 9 got it perfect. it's just to say that one can
use the Unix model of pathnames, open/read/write/close, for networks
and IPC in general (even pipes).

It's just a shame nobody's caught on yet ;-)

And, to repeat, a bunch of us in the 80s tried to implement the idea
of /dev/net/host/port (mind was in AmigaDOS), and it fails badly for
all kinds of reasons.
simple one:
what's the implications of
mv /dev/net/harv/20 /dev/net/prep/35

what happens? If only somebody could only find Rob's talk which delves
into this in such nice detail ...

In my view, what went wrong with Unix networking 40 years ago is that
it broke from the Unix model, i.e. that resources are accessed via
path names, and went with binary descriptors as paths. But what can
you do? Synthetics were yet to be created.

ron

Re: [TUHS] Unix APIs: elegant or not?

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Thu, Nov 1, 2018 at 7:48 AM Dave Horsfall <dave@horsfall.org> wrote:

> I did see a proposal a while back (can't remember where/when) to use
> something like open("/dev/net/host/port", ...) to establish a network
> connection (or a server with appropriate flags), in much the same way that
> device files are created dynamically.
>

Dave - that's how the original UNIX ChaosNet code worked and the original
BBN TCP (before sockets).  Apollo used the same scheme with Aegis/Domain
(probably because Larry Allen implemented both UNIX Chaos and Aegis IIRC).

P9 took a different path still (pun intended).

Clem

[-- Attachment #2: Type: text/html, Size: 1384 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[Dave Horsfall]

On Thu, 1 Nov 2018, Pierre DAVID wrote:

> Do you *really* think that IPC System V primitives are better than 
> sockets?

Nope; they're even worse, which I didn't think was possible.  The one 
saving grace (IMHO) of SysV was the TTY driver and its cleaner interface.

I did see a proposal a while back (can't remember where/when) to use 
something like open("/dev/net/host/port", ...) to establish a network 
connection (or a server with appropriate flags), in much the same way that 
device files are created dynamically.

With Unix, everything looks like a file, but as I said, the sockets API 
broke that convention, forcing users to work at a much lower level than 
necessary.

I suppose that around now someone will say that Plan 9 fixed all that :-)

-- Dave

Re: [TUHS] Unix APIs: elegant or not?

[Pierre DAVID]

On Thu, Nov 01, 2018 at 08:31:25AM +1100, Dave Horsfall wrote:
>On Wed, 31 Oct 2018, Paul Winalski wrote:
>
>>I've always thought that the UNIX file primitives very elegantly 
>>adhere to this principle. [...]
>
>Until Berkeley introduced sockets, which has the worst API that I've 
>ever seen in Unix-land...
>

Do you *really* think that IPC System V primitives are better than 
sockets?

Pierre

Re: [TUHS] Unix APIs: elegant or not?

[Dave Horsfall]

On Wed, 31 Oct 2018, Paul Winalski wrote:

> I've always thought that the UNIX file primitives very elegantly adhere 
> to this principle. [...]

Until Berkeley introduced sockets, which has the worst API that I've ever 
seen in Unix-land...

Last place I worked, they had a library that simplified it; the 
application said "I am a server on port NNN" and the client said "I want 
to talk to port NNN on 1.2.3.4" etc.

I wish I'd "borrowed" that code when I left.

-- Dave

Re: [TUHS] Unix APIs: elegant or not?

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 4459 bytes --]

At 2018-10-31T11:47:20-0400, Paul Winalski wrote:
> On 10/31/18, Warren Toomey <wkt@tuhs.org> wrote:
> >
> >   The POSIX file API is a great example, but not of a deep
> >   interface. Rather, it’s a great example of how code with a very
> >   complicated interface may look deceptively simple when reduced to C-style
> >   function signatures.

Indeed.  It is my hope that in the coming years software engineers will
decline to describe an interface as "simple" until they've seen how much
logic is required to formally verify it.
> 
> For me one of the most important software design principles is that
> the simple and most common use cases should be the simplest for the
> user to code.

Yes.  OpenSSL is an infamous example of design failure in this respect.

However, the term "software design principle" is a bit vague.  More
specifically you're identifying a maxim of good programming interface
design.

What's good for a library API is not necessarily what's good for a
system call interface.  System calls are not really library functions,
and that is why they have had a different section in the manual from day
one.

open/reda/write/(l)seek/close did not primarily constitute, as I think
they were originally conceived, and API; they were there to expose
_primitives_ of the operating system.

The lateness in coming of the C standard I/O library may have been
something of a problem here; an I/O library is exactly where you want to
press your design principle to the maximum.  (On the other hand,
sometimes the design space needs to be explored, and you don't know what
the common cases are going to be because you don't have enough data
points yet.  In that case you expose the primitive operations and study
what bubbles up when programmers apply the DRY principle--they are
future standard library calls in disguise.)

But I suspect another problem, even had stdio.h been around in 1972,
would have been the obsessive false economy of "programming close to the
metal".  Because the C language did that, and because the system call
interface was there and easy to grab a hold of, tons of application
programmers thought they should follow suit, violating Knuth's principle
about optimization with fervor.

> I've always thought that the UNIX file primitives very elegantly
> adhere to this principle.  Reading a file in UNIX is a simple
> open()/read().../close() sequence.  Contrast that with VMS's $QIO or
> IBM OS access methods, where the full complexity is not only exposed
> in the interface, it must be considered, set up, and controlled by the
> user for even the simplest operations.  Multibuffered, asynchronous,
> interrupt-driven I/O is more complicated (if not downright clumsy) in
> UNIX than in VMS or OS/VS, but that's OK, IMO--it shifts the
> complexity burden to those doing complex things.

I haven't programmed on VMS or OS/VS, but again this sounds like a
scenario where a standard I/O library should have existed but didn't (or
was inadequate to the demands placed on it--as stdio itself arguably
still is, with a static global errno variable and an original design
that ignored reentrancy entirely, such that we have to consult manuals
to determine which calls are safe for multi-threaded apps or use in a
signal handler).

Context has to be managed somewhere.  open()/read()/.../close() only
look simple because a lot of context has been pushed down into the
kernel--the list of flags supported by open() has steadily grown over
the years.

The POSIX openat() system call family is a superior design, though still
saddled with a lot of context.  I admit, it is probably more annoying to
use for application programmers who want to use it directly.  That's
because they're trying to solve their problems at the wrong level.  It
irritates them that they have to keep track of their own contexts (in
this case, a file descriptor).  A well-designed library is able to do
this for them, but they want to be close to the metal.  And too many of
them will not do the job adequately themselves, so they demand that the
kernel handle it itself.  Now the system call interface is even less a
set of primitives, and even more something that's "easy to code to" for
sloppy programmers.

That is one way kernel interfaces, memory requirements, and
task-switching times bloat.  How close to the metal does one feel then?

-- 
Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[Lawrence Stewart]

My version of this:

With JCL, it is as easy to read 1000 tapes as it is to read one tape.

For JCL, substitute the all-singing API of your choice.

On the other hand you can spend a very long time writing code that actually responds correctly to all the failure cases in, say, a TCP stream connection.

-L

Re: [TUHS] Unix APIs: elegant or not?

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]

On Wed, Oct 31, 2018 at 12:51 PM Paul Winalski <paul.winalski@gmail.com>
wrote:

> For me one of the most important software design principles is that the
> simple and most common use cases should be the simplest for theuser to
> code.

It's OK if complicated things are complicated to code, but simple things
> should be kept simple.
>
+1 Amen.

>
> I've always thought that the UNIX file primitives very elegantly adhere
> to this principle.  Reading a file in UNIX is a simple open()/read().../close()
> sequence.  Contrast that with VMS's $QIO or IBM OS access methods, where
> the full complexity is not only exposed in the interface, it must be
> considered, set up, and controlled by the
> user for even the simplest operations.

As my friend Tom Texiera once so wisely observered, "*it was not so bad
that QIO had over a 1000 options, it was that each of them had to be test
for on each I/O.*"
I also remember having an argument with Culter about QIO vs UNIX I/O (he
was defending QIO at the time as it was 'better' - as being more
complete).   But I pointed out, that it was interesting that after stream
I/O was added to VMS, most DEC customers (and the internal language
libraries team) had switched to using stream (UNIX style I/O) for most
operations because it was simplier and just as fast.   It was one of the
few times, I saw Dave stop arguing as he really did not have a response.



> Multibuffered, asynchronous, interrupt-driven I/O is more complicated (if
> not downright clumsy) in UNIX than in VMS or OS/VS, but that's OK,
> IMO--it shifts the
> complexity burden to those doing complex things.
>
Exactly, those programs that need to do it, can when they need to.   That
said, because of our VMS experience, we added ASTs (which I tink are
simplier than UNIX signals) and a QIO like call to RTU because the
Real-Time customers coming from VMS needed (wanted) it.  Truth is async
I/O  was really useful for some special cases that we had customers that
really need use it.   But for the most part the simple UNIX 5 I/O calls
were good enough (athough I would still rather AST's).

BTW:  when we added threads we discovered that a lot of the need/use for
async I/O also went away and frankly much of the complexity (clumsiness) of
the async I/O code was not longer needed, as the threading to care of it
and certain was smaller and simplier.

Also, I'll agree it was a tad clumsy, but the old double-dd (ddd) program
(you can find it in the UUNET archives), uses two processes that
communicate via a pipe to do the same trick with a traditional (6th
edition) UNIX I/O system to do some of the same things.  Basically the two
processes, take turns between reading and writing.   Thus the asynchronous
is purely left in the kernel.  The user code is actually very simple.
 There is a pipe that passes a token back and forth as to when the next
read/write can start.  For old UNIX systems lacking async I/O, ddd(8)  was
only way I knew you get a tape drive such as QIC/4 or 8 mm, much less a 1/2
streamer to keep up.

Clem
ᐧ

[-- Attachment #2: Type: text/html, Size: 6495 bytes --]

Re: [TUHS] Unix APIs: elegant or not?

[Paul Winalski]

On 10/31/18, Warren Toomey <wkt@tuhs.org> wrote:
>
>   The POSIX file API is a great example, but not of a deep
>   interface. Rather, it’s a great example of how code with a very
>   complicated interface may look deceptively simple when reduced to C-style
>   function signatures.

For me one of the most important software design principles is that
the simple and most common use cases should be the simplest for the
user to code.  It's OK if complicated things are complicated to code,
but simple things should be kept simple.

I've always thought that the UNIX file primitives very elegantly
adhere to this principle.  Reading a file in UNIX is a simple
open()/read().../close() sequence.  Contrast that with VMS's $QIO or
IBM OS access methods, where the full complexity is not only exposed
in the interface, it must be considered, set up, and controlled by the
user for even the simplest operations.  Multibuffered, asynchronous,
interrupt-driven I/O is more complicated (if not downright clumsy) in
UNIX than in VMS or OS/VS, but that's OK, IMO--it shifts the
complexity burden to those doing complex things.

-Paul W.

[TUHS] Unix APIs: elegant or not?

[Warren Toomey]

I was just reading this book review:
http://www.pathsensitive.com/2018/10/book-review-philosophy-of-software.html
and came across these paragraphs:

    <book quote>
    The mechanism for file IO provided by the Unix operating system
    and its descendants, such as Linux, is a beautiful example of a
    deep interface. There are only five basic system calls for I/O,
    with simple signatures:

    int open(const char* path, int flags, mode_t permissions);
    ssize_t read(int fd, void* buffer, size_t count);
    ssize_t write(int fd, const void* buffer, size_t count);
    off_t lseek(int fd, off_t offset, int referencePosition);
    int close(int fd);
    </book quote>

  The POSIX file API is a great example, but not of a deep
  interface. Rather, it’s a great example of how code with a very
  complicated interface may look deceptively simple when reduced to C-style
  function signatures. It’s a stateful API with interesting orderings
  and interactions between calls. The flags and permissions parameters
  of open hide an enormous amount of complexity, with hidden requirements
  like “exactly one of these five bits should be specified.” open may
  return 20 different error codes, each with their own meaning, and many
  with references to specific implementations.

  The authors of SibylIFS tried to write down an exact description of the
  open interface. Their annotated version[1] of the POSIX standard is over
  3000 words. Not counting basic machinery, it took them over 200 lines[2]
  to write down the properties of open in higher-order logic, and another
  70 to give the interactions between open and close.
  [1]: https://github.com/sibylfs/sibylfs_src/blob/8a7f53ba58654249b0ec0725ce3887840d6a1812/fs_spec/src/posix/open.md
  [2]: https://github.com/sibylfs/sibylfs_src/blob/8a7f53ba58654249b0ec0725ce3887840d6a1812/fs_spec/src/t_fs_spec_fs_command_properties.lem_cppo#L460

I just thought it was a thought-provoking comment on the apparent elegance
of the Unix file API that actually has some subtle complexity.

Cheers, Warren