Re: [9fans] 9fs/9auth for FreeBSD

Re: [9fans] 9fs/9auth for FreeBSD

[Russ Cox]

    O_APPEND probably isn't needed on BSD kernels, either. That was me
    being paranoid based on the comment that writes didn't always get
    treated as appends.

No, it's needed.  The behavior on FreeBSD really is what
I said -- if you do a write with a file offset
that is not the end of the file, the write fails.
It doesn't pretend the offset really is at the 
end (as Plan 9 and apparently Linux do).  It fails.
Opening O_APPEND should, at least on a single system,
insulate you from worrying about the offset.
I hadn't thought of that.  (My Unix instincts
continue to dull.)

Russ

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Russ" == Russ Cox <rsc@plan9.bell-labs.com> writes:

    Russ> No, it's needed.  The behavior on FreeBSD really is what I
    Russ> said -- if you do a write with a file offset that is not the
    Russ> end of the file, the write fails.

This is almost certainly a bug. I'll verify what POSIX has to say
about this and get a bug report filed. What (exact) version of FreeBSD
did you observe this behaviour on?

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Wed, 28 Mar 2001, Lyndon Nerenberg wrote:

> >>>>> "Russ" == Russ Cox <rsc@plan9.bell-labs.com> writes:
> 
>     Russ> No, it's needed.  The behavior on FreeBSD really is what I
>     Russ> said -- if you do a write with a file offset that is not the
>     Russ> end of the file, the write fails.
> 
> This is almost certainly a bug. I'll verify what POSIX has to say
> about this and get a bug report filed. What (exact) version of FreeBSD
> did you observe this behaviour on?

See ufs/ufs/ufs_readwrite.c and grep for APPEND. It's present in -STABLE.

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

[I wrote]
> On Wed, 28 Mar 2001, Lyndon Nerenberg wrote:
> > This is almost certainly a bug. I'll verify what POSIX has to say
> > about this and get a bug report filed. What (exact) version of FreeBSD
> > did you observe this behaviour on?
> 
> See ufs/ufs/ufs_readwrite.c and grep for APPEND. It's present in -STABLE.

PS: POSIX says nothing, since append-only is BSDism and POSIX is a
codification of Missed'em'V bugs, so append-only is out of scope.

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:

    Alexander> PS: POSIX says nothing, since append-only is BSDism and
    Alexander> POSIX is a codification of Missed'em'V bugs, so
    Alexander> append-only is out of scope.

Let's make sure we're talking about the same thing here. The chflags()
behaviour is 4.4BSD-specific. The O_APPEND behaviour has been around
since the dawn of time (or at least SVR1). My copy of POSIX is at
home, however SuS V2 states explicitly (both open(2) and write(2))
that O_APPEND causes all writes to set the file pointer to the end of
the file prior to performing the write I/O, and that the combination
of the implicit seek and the associated write are atomic.

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

Okay, I think I see what's going on here. After reading ufs_readwrite.c
it appears things are working as expected, and it's the expectations
that are wrong.

open with O_APPEND forces all writes to be appended. It sets
program behaviour, and is doing the right thing.

chflags with [US]F_APPEND sets a _policy_, that being that any
write to the file must be an append operation. It doesn't
force that behaviour, though, so you still need to open
the file with O_APPEND.

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Wed, 28 Mar 2001, Lyndon Nerenberg wrote:

> >>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:
> 
>     Alexander> PS: POSIX says nothing, since append-only is BSDism and
>     Alexander> POSIX is a codification of Missed'em'V bugs, so
>     Alexander> append-only is out of scope.
> 
> Let's make sure we're talking about the same thing here. The chflags()
> behaviour is 4.4BSD-specific. The O_APPEND behaviour has been around
> since the dawn of time (or at least SVR1). My copy of POSIX is at
> home, however SuS V2 states explicitly (both open(2) and write(2))
> that O_APPEND causes all writes to set the file pointer to the end of
> the file prior to performing the write I/O, and that the combination
> of the implicit seek and the associated write are atomic.

Wait a second. The question being: what happens if you open append-only
file with fd = open("foo", O_RDWR), then lseek(fd, 0, SEEK_SET), then
write(fd, buf, len)?
	Plan9: append to EOF.
	Linux: -EPERM on open()
	FreeBSD: -EPERM on write() (open() succeeds).

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:

    Alexander> Linux: -EPERM on open() 

Broken. (Iff they are modeling chflags(). Linux likes to make
up its own random braindead behaviours though.)

    Alexander> FreeBSD: -EPERM on write() (open() succeeds).

Which is the correct behaviour (for chflags()).

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Wed, 28 Mar 2001, Lyndon Nerenberg wrote:

> >>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:
> 
>     Alexander> Linux: -EPERM on open() 
> 
> Broken. (Iff they are modeling chflags(). Linux likes to make
> up its own random braindead behaviours though.)
> 
>     Alexander> FreeBSD: -EPERM on write() (open() succeeds).
> 
> Which is the correct behaviour (for chflags()).

== behaves the same way as 4.4BSD (system that introduced chflags()).
Whether said behaviour makes sense or not is a completely different
question, though...

Re: [9fans] 9fs/9auth for FreeBSD

[Douglas A. Gwyn]

Alexander Viro wrote:
> PS: POSIX says nothing, since append-only is BSDism and POSIX is a
> codification of Missed'em'V bugs, so append-only is out of scope.

That is uncalled-for and inaccurate.  We put considerable effort
into accommodating *both* the BSD and USG variants of UNIX when
drawing up the (initial) POSIX specs.

O_APPEND was certainly included in the 1988 IEEE 1003.1 (POSIX)
specification; it's defined in <fcntl.h> and means that the file
offset is to be set to the end of the file prior to each write.

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Mon, 2 Apr 2001, Douglas A. Gwyn wrote:

> Alexander Viro wrote:
> > PS: POSIX says nothing, since append-only is BSDism and POSIX is a
> > codification of Missed'em'V bugs, so append-only is out of scope.
>
> That is uncalled-for and inaccurate.  We put considerable effort
> into accommodating *both* the BSD and USG variants of UNIX when
> drawing up the (initial) POSIX specs.
>
> O_APPEND was certainly included in the 1988 IEEE 1003.1 (POSIX)
> specification; it's defined in <fcntl.h> and means that the file
> offset is to be set to the end of the file prior to each write.

Back in 1988 append-only didn't exist. O_APPEND behaviour is the only
part of the thing where everyone agrees - the problem is in the
treatment of append-only files (thing that was introduced after 1988)
when opened _without_ O_APPEND.

Re: [9fans] 9fs/9auth for FreeBSD

[Boyd Roberts]

"Alexander Viro" <viro@math.psu.edu> a crit dans le message news:
Pine.GSO.4.21.0104020559080.12034-100000@weyl.math.psu.edu...
>
> Back in 1988 append-only didn't exist. O_APPEND behaviour is the only
> part of the thing where everyone agrees - the problem is in the
> treatment of append-only files (thing that was introduced after 1988)
> when opened _without_ O_APPEND.


Back in 1981 autolocking and locking files existed, but you'd
have to be running basser's much hacked 32V.  It had resources
limits too.

By 1985 it had copy-on-write fork an 'fast' [one] context switches.

And kre thought 4.2BSD was fast...

IIRC think basse[rt] were 45% faster.

Of course, this took years of work by several people.

--
Boyd Roberts        http://www.insultant.net        boyd@insultant.net

What do you know about surfing, Major?  You're from goddamn New Jersey.

    -- Lt. Colonel Kilgore

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Wed, 28 Mar 2001, Russ Cox wrote:

>     O_APPEND probably isn't needed on BSD kernels, either. That was me
>     being paranoid based on the comment that writes didn't always get
>     treated as appends.
> 
> No, it's needed.  The behavior on FreeBSD really is what
> I said -- if you do a write with a file offset
> that is not the end of the file, the write fails.
> It doesn't pretend the offset really is at the 
> end (as Plan 9 and apparently Linux do).  It fails.

Umm... Actually, I was wrong - what actually happens with append-only
on Linux looks so:
	* if file is marked append-only you can't open it for write without
	  O_APPEND. open() will fail with -EPERM
	* You can't turn O_APPEND off (with fcntl()) if file is append-only
	* For any O_APPEND opened file write() ignores position and goes
	  to the end of file.
In case of FreeBSD you can open append-only files for write without
O_APPEND, but attempt to write not at the EOF will fail.

We could easily switch to Plan9 behaviour - all it takes is
        if (IS_APPEND(inode)) {
-                if  ((flag & FMODE_WRITE) && !(flag & O_APPEND))
-                        goto exit;
@@
        if (IS_APPEND(inode)) {
+                if  (flag & FMODE_WRITE)
+                       flag |= O_APPEND;
 
in open_namei(). I'm not sure that it's the right thing to do, though.
Failing on write() is definitely too late, but silently adding O_APPEND
may be rather confusing. Hell knows... I'll look at it.

Re: [9fans] 9fs/9auth for FreeBSD

[Douglas A. Gwyn]

Alexander Viro wrote:
> ... I'm not sure that it's the right thing to do, though.

UNIX has always had files that have an innate append-only
characteristic,
although it didn't let you specify that a different kind of file should
be treated as having that characteristic.  The UNIX behavior was that
write to such a file succeeded, appending to the end of course, and
seek attempts (to other than the end) would fail.  If some "modern"
UNIX-like system wants to support user designation of files as having
an append-only characteristic, then files that have that characteristic
innately ought to report that the corresponding flag is always set, and
serve as the model for how I/O is performed.

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Mon, 2 Apr 2001, Douglas A. Gwyn wrote:

> Alexander Viro wrote:
> > ... I'm not sure that it's the right thing to do, though.
>
> UNIX has always had files that have an innate append-only
> characteristic,
> although it didn't let you specify that a different kind of file should
> be treated as having that characteristic.  The UNIX behavior was that
> write to such a file succeeded, appending to the end of course, and
> seek attempts (to other than the end) would fail.  If some "modern"

Erm. There is a little problem with lseek() returning errors -
reading these files is perfectly sane thing. Pipes et.al. have
innate append-only characteristic, indeed, but append-only regular
files are different - think of that as enforcement (silent or
not - that's where the differences are) of O_APPEND.

> UNIX-like system wants to support user designation of files as having

<shrug> "modern" is a relative term - whether you want to consider
4.4BSD as such or not is a matter of taste, but behaviour in question goes
back at least to '93 (I don't have CVS logs at hand; most likely
the thing was added not just before the release, so s/93/early 90s/)

> an append-only characteristic, then files that have that characteristic
> innately ought to report that the corresponding flag is always set, and
> serve as the model for how I/O is performed.

IIRC, rationale for append-only _inode_ flag (not O_APPEND) was handling
of the logs and similar animals. That's definitely the most frequent
use of that thing. The only real question being: what's better -
complaining about echo foo > append_only_file or silently assuming that
>> was implied.

Re: [9fans] 9fs/9auth for FreeBSD

[Russ Cox]

  O_APPEND was certainly included in the 1988 IEEE 1003.1 (POSIX)
  specification; it's defined in <fcntl.h> and means that the file
  offset is to be set to the end of the file prior to each write.

This may be true but O_APPEND != append-only, which
is what Viro was talking about.  O_APPEND came up only
as a workaround to the BSD append-only semantics.

Russ

Re: [9fans] 9fs/9auth for FreeBSD

[Russ Cox]

that was me.  the u9fs is for 9P2000 although it
supports 9P1 for backwards compatibility and testing.
the 9auth is inspired by pace willisson (sp?)'s 
freebsd "il over raw sockets" auth server but not
much of his code remains, if any.

both are tcp only, running out of inetd, so they
don't depend on the freebsd il patches.  the biggest
shortcoming is the lack of a portable way to implement
append-only files.  freebsd (and presumably the other
unix clones) take append-only to mean "only appends
succeed" rather than "all writes are treated as
appends", and there's no standard way to enable even this.

these programs aren't online, but i'll put them up next
week (i'm away from the freebsd machine) for the
interested parties.

russ

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Russ" == Russ Cox <rsc@plan9.bell-labs.com> writes:

    Russ> the biggest shortcoming
    Russ> is the lack of a portable way to implement append-only
    Russ> files.  freebsd (and presumably the other unix clones) take
    Russ> append-only to mean "only appends succeed" rather than "all
    Russ> writes are treated as appends", and there's no standard way
    Russ> to enable even this.

On 4.4BSD-derived systems you should be able to get the behaviour you
want with chflags(path, UF_APPEND) after creating the file.  During
subsequent opens you would have to check for this flag, and open with
O_APPEND if it is present (to force the writes to always be treated as
appends). Did you try this? (I realize it's not very portable.)

Without chflags() you would have to hijack on of the mode bits, which
is downright evil.

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Alexander Viro]

On Wed, 28 Mar 2001, Lyndon Nerenberg wrote:

>     Russ> the biggest shortcoming
>     Russ> is the lack of a portable way to implement append-only
>     Russ> files.  freebsd (and presumably the other unix clones) take
>     Russ> append-only to mean "only appends succeed" rather than "all
>     Russ> writes are treated as appends", and there's no standard way
>     Russ> to enable even this.
 
> On 4.4BSD-derived systems you should be able to get the behaviour you
> want with chflags(path, UF_APPEND) after creating the file.  During
> subsequent opens you would have to check for this flag, and open with
> O_APPEND if it is present (to force the writes to always be treated as
> appends). Did you try this? (I realize it's not very portable.)
 
> Without chflags() you would have to hijack on of the mode bits, which
> is downright evil.

In case of Linux append-only is "all writes are treated as appends"
(see mm/filemap.c::generic_file_write()), so no wanking with O_APPEND
on subsequent open() is needed, but marking file append-only
_is_ downright evil - ioctl(fd, EXT2_IOC_SETFLAGS, EXT2_APPEND_FL).
chflags() had been discussed many times, there are several patches
of varying age that implement it, but nobody cared enough to get
them into the tree.

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:

    Alexander> In case of Linux append-only is "all writes are treated
    Alexander> as appends" (see mm/filemap.c::generic_file_write()),
    Alexander> so no wanking with O_APPEND on subsequent open() is
    Alexander> needed,

O_APPEND probably isn't needed on BSD kernels, either. That was me
being paranoid based on the comment that writes didn't always get
treated as appends.

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Boyd Roberts]

Lyndon Nerenberg <lyndon@orthanc.ab.ca> a crit dans le message :
200103281926.f2SJQnl94482@orthanc.ab.ca...
> >>>>> "Russ" == Russ Cox <rsc@plan9.bell-labs.com> writes:
>
>     Russ> the biggest shortcoming
>     Russ> is the lack of a portable way to implement append-only
>     Russ> files.  freebsd (and presumably the other unix clones) take
>     Russ> append-only to mean "only appends succeed" rather than "all
>     Russ> writes are treated as appends", and there's no standard way
>     Russ> to enable even this.
>
> On 4.4BSD-derived systems you should be able to get the behaviour you
> want with chflags(path, UF_APPEND) after creating the file

Nope, you just don't get it.  An auto-apend file means that it
works in general.   Great for security logs when the file
lies on some sort of write once media:

    echo foo > f

Or should we build O_APPEND into _every tool_?

    cat -v city

--
Boyd Roberts        http://www.insultant.net        boyd@insultant.net

What do you know about surfing, Major?  You're from goddamn New Jersey.

    -- Lt. Colonel Kilgore

[9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

A while back someone mentioned having an updated fileserver,  and an auth
servicer, running on FreeBSD. Was that code put online anywhere, and
did someone save a pointer to it?

--lyndon

Re: [9fans] 9fs/9auth for FreeBSD

[Christopher Nielsen]

On Tue, Mar 27, 2001 at 04:53:22PM -0700, Lyndon Nerenberg wrote:

> A while back someone mentioned having an updated fileserver,  and an auth
> servicer, running on FreeBSD. Was that code put online anywhere, and
> did someone save a pointer to it?

I've been working on updating for FreeBSD -stable and
-current the patches for FreeBSD 3.x that implement IL
and 9P for FreeBSD, but I haven't finished them yet.
There are significant diffs between FreeBSD 3.x, 4.x,
and what will be 5.x, so it's taking longer than I
anticipated. Also, I haven't had a whole lot of time
to work on extra projects lately.

Is this what you're referring to?

-- 
Christopher Nielsen - Metal-wielding pyro techie
cnielsen@pobox.com
"Any technology indistinguishable from magic is 
insufficiently advanced." --unknown

Re: [9fans] 9fs/9auth for FreeBSD

[Lyndon Nerenberg]

>>>>> "Christopher" == Christopher Nielsen <cnielsen@pobox.com> writes:

    Christopher> I've been working on updating for FreeBSD -stable and
    Christopher> -current the patches for FreeBSD 3.x that implement
    Christopher> IL and 9P for FreeBSD, but I haven't finished them
    Christopher> yet.

    Christopher> Is this what you're referring to?

No, it was the code Russ mentioned. (Although I'm looking forward
to seeing your work as well.)

--lyndon