[9fans] Fwd: Reading from FS with inaccurate file sizes?

[9fans] Fwd: Reading from FS with inaccurate file sizes?

[CDeVilbiss]

I started a conversation with Amit Singh on the MacFUSE website about
the problems I was having with zero-sized files, and he brought up
this point.  I have neither a Linux nor a FreeBSD system to try this
on, but I know some people on this list have used 9pfuse on those
systems in the past.

Can someone use 9pfuse -D and try out this test?

That is:

prompt # plumber
prompt # 9pfuse -D `namespace`/plumb /mnt/plumb
prompt # wc /mnt/plumb/rules # verify output is correct (and file non-
empty)
prompt # cat new_rule.txt >> /mnt/plumb/rules # what does the 9pfuse
debug output show here, specifically for the offset of the write?


---------- Forwarded message ----------
From: "Amit Singh" <asi...@gmail.com>
Date: Mar 23, 11:33 am
Subject: Reading from FS with inaccurate file sizes?
To: macfuse-devel


> prompt # cat new_rule.txt >> /mnt/plumb/rules # add a rule

When you do this on Linux for a file with an advertised size of 0,
what offset do you get in the write call in your user-space file
system daemon?

Re: [9fans] Fwd: Reading from FS with inaccurate file sizes?

[Kris Maglione]

[-- Attachment #1: Type: text/plain, Size: 6739 bytes --]

On Fri, Mar 23, 2007 at 08:41:27PM -0000, CDeVilbiss@gmail.com wrote:
>Can someone use 9pfuse -D and try out this test?

% 9pfuse -D unix!`{namespace}^/plumb /mnt/plumb
<- Tversion tag 0 msize 8192 version '9P2000'
-> Rversion tag 65535 msize 8192 version '9P2000'
<- Tattach tag 0 fid 0 afid -1 uname kris aname 
-> Rattach tag 0 qid (0000000000000000 0 d)
FUSE -> len 16 unique 0 uid 0 gid 998 pid 17141 Init major 7 minor 8
FUSE <- unique 0 (Init) major 7 minor 5 max_write 4096
FUSE -> len 0 unique 0 uid 0 gid 998 pid 17141 Statfs
FUSE <- unique 0 (Statfs) blocks 0 bfree 0 bavail 0 files 0 ffree 0 bsize 0 namelen 0 frsize 0
FUSE -> len 0 unique 0 uid 0 gid 998 pid 17141 Getattr nodeid 0x1
/* Above 2 lines repeated many times; elided */
<- Tstat tag 0 fid 0
-> Rstat tag 0  stat '.' 'kris' 'kris' 'kris' q (0000000000000000 0 d) m 020000000500 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Getattr) attr_valid 1 ino 0x8000000000000000 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 040500 nlink 1 uid 1000 gid 998 rdev 0
wc /mnt/plumb/rules
FUSE -> len 6 unique 0 uid 1000 gid 998 pid 17888 Lookup nodeid 0x1 name 'rules'
<- Twalk tag 0 fid 0 newfid 1 nwname 1 0:rules 
-> Rwalk tag 0 nwqid 1 0:(0000000000000001 0 ) 
<- Tstat tag 0 fid 1
-> Rstat tag 0  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Lookup) nodeid 0x2 gen 0x1 entry_valid 1 attr_valid 1  ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 8 unique 0 uid 1000 gid 998 pid 17888 Open nodeid 0x2 flags 0 mode 0
<- Twalk tag 0 fid 1 newfid 2 nwname 0 
-> Rwalk tag 0 nwqid 0 
<- Topen tag 0 fid 2 mode 0
-> Ropen tag 0 qid (0000000000000001 0 ) iounit 0
FUSE <- unique 0 (Open) fh 0x3 flags 0x1
FUSE -> len 0 unique 0 uid 1000 gid 998 pid 17888 Getattr nodeid 0x2
<- Tstat tag 0 fid 1
-> Rstat tag 0  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Getattr) attr_valid 1 ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 24 unique 0 uid 1000 gid 998 pid 17888 Read nodeid 0x2 fh 0x3 offset 0 size 8192
<- Tread tag 0 fid 2 offset 0 count 4096
-> Rread tag 0 count 3178 '706c616e 393d2f75 73722f6c 6f63616c 2f706c61 6e390a0a 65646974 6f723d61 636d650a 0a616464 72656c65 6d3d2728 28233f5b 302d395d 2b297c28 2f5b412d'
FUSE <- unique 0 (Read) size 3178
FUSE -> len 24 unique 0 uid 1000 gid 998 pid 17888 Read nodeid 0x2 fh 0x3 offset 3178 size 8192
<- Tread tag 0 fid 2 offset 3178 count 4096
-> Rread tag 0 count 0 ''
FUSE <- unique 0 (Read) size 0
    130     362    3178 /mnt/plumb/rules
% FUSE -> len 24 unique 0 uid 1000 gid 998 pid 17888 Release nodeid 0x2 fh 0x3 flags 0
<- Tclunk tag 0 fid 2
-> Rclunk tag 0
cat lib/plumbing >/mnt/plumb/rules
FUSE -> len 6 unique 0 uid 1000 gid 998 pid 14894 Lookup nodeid 0x1 name 'rules'
<- Twalk tag 0 fid 0 newfid 2 nwname 1 0:rules 
-> Rwalk tag 0 nwqid 1 0:(0000000000000001 0 ) 
<- Tstat tag 0 fid 2
-> Rstat tag 0  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Lookup) nodeid 0x3 gen 0x2 entry_valid 1 attr_valid 1  ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 8 unique 0 uid 1000 gid 998 pid 14894 Open nodeid 0x3 flags 0x1 mode 0
<- Twalk tag 0 fid 2 newfid 3 nwname 0 
-> Rwalk tag 0 nwqid 0 
<- Topen tag 0 fid 3 mode 1
-> Ropen tag 0 qid (0000000000000001 0 ) iounit 0
FUSE <- unique 0 (Open) fh 0x4 flags 0x1
FUSE -> len 0 unique 0 uid 1000 gid 998 pid 14894 Getattr nodeid 0x3
<- Tstat tag 0 fid 2
-> Rstat tag 0  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Getattr) attr_valid 1 ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 88 unique 0 uid 1000 gid 998 pid 14894 Setattr nodeid 0x3 size 0
<- Twalk tag 0 fid 2 newfid 4 nwname 0 
-> Rwalk tag 0 nwqid 0 
<- Topen tag 0 fid 4 mode 17
-> Rerror tag 0 ename file already open
FUSE <- unique 0 error -34 Result too large
<- Tclunk tag 0 fid 4
/mnt/plumb/rules: rc: can't open: Result too large
% FUSE -> len 24 unique 0 uid 1000 gid 998 pid 14894 Release nodeid 0x3 fh 0x4 flags 0x1
<- Tclunk tag 0 fid 3
-> Rclunk tag 0
-> Rclunk tag 1
FUSE <- unique 0 (Release)
wc /mnt/plumb/rules
FUSE -> len 6 unique 0 uid 1000 gid 998 pid 18634 Lookup nodeid 0x1 name 'rules'
<- Twalk tag 0 fid 0 newfid 3 nwname 1 0:rules 
-> Rwalk tag 1 nwqid 1 0:(0000000000000001 0 ) 
<- Tstat tag 0 fid 3
-> Rstat tag 1  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Lookup) nodeid 0x4 gen 0x2 entry_valid 1 attr_valid 1  ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 8 unique 0 uid 1000 gid 998 pid 18634 Open nodeid 0x4 flags 0 mode 0
<- Twalk tag 0 fid 3 newfid 4 nwname 0 
-> Rwalk tag 1 nwqid 0 
<- Topen tag 0 fid 4 mode 0
-> Ropen tag 1 qid (0000000000000001 0 ) iounit 0
FUSE <- unique 0 (Open) fh 0x5 flags 0x1
FUSE -> len 0 unique 0 uid 1000 gid 998 pid 18634 Getattr nodeid 0x4
<- Tstat tag 0 fid 3
-> Rstat tag 1  stat 'rules' 'kris' 'kris' 'kris' q (0000000000000001 0 ) m 0600 at 1173567833 mt 1173567833 l 0 t 3328 d 134563933
FUSE <- unique 0 (Getattr) attr_valid 1 ino 0x1 size 0 blocks 0 atime 1173567833 mtime 1173567833 ctime 1173567833 mode 0100600 nlink 1 uid 1000 gid 998 rdev 0
FUSE -> len 24 unique 0 uid 1000 gid 998 pid 18634 Read nodeid 0x4 fh 0x5 offset 0 size 8192
<- Tread tag 0 fid 4 offset 0 count 4096
-> Rread tag 1 count 3178 '706c616e 393d2f75 73722f6c 6f63616c 2f706c61 6e390a0a 65646974 6f723d61 636d650a 0a616464 72656c65 6d3d2728 28233f5b 302d395d 2b297c28 2f5b412d'
FUSE <- unique 0 (Read) size 3178
FUSE -> len 24 unique 0 uid 1000 gid 998 pid 18634 Read nodeid 0x4 fh 0x5 offset 3178 size 8192
<- Tread tag 0 fid 4 offset 3178 count 4096
-> Rread tag 1 count 0 ''
FUSE <- unique 0 (Read) size 0
    130     362    3178 /mnt/plumb/rules
% FUSE -> len 24 unique 0 uid 1000 gid 998 pid 18634 Release nodeid 0x4 fh 0x5 flags 0
<- Tclunk tag 0 fid 4
-> Rclunk tag 1

-- 
Kris Maglione

Some of it plus the rest of it is all of it.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [9fans] Fwd: Reading from FS with inaccurate file sizes?

[erik quanstrom]

this doesn't look right at all.  almost all the actions in this trace
seem to be duplicated.  e.g. two Rstats of 'rules' in a row.

and for some reason, fuse is trying to write /mnt/plumb/rules
a second time before closing the first instance.  this gives an
error on the second write and only that is being returned.

<- Topen tag 0 fid 4 mode 17
-> Rerror tag 0 ename file already open
FUSE <- unique 0 error -34 Result too large
<- Tclunk tag 0 fid 4
/mnt/plumb/rules: rc: can't open: Result too large

- erik

Re: [9fans] Fwd: Reading from FS with inaccurate file sizes?

[Rolando Segura]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On 3/23/07, erik quanstrom <quanstro@coraid.com> wrote:
>
> this doesn't look right at all.  almost all the actions in this trace
> seem to be duplicated.  e.g. two Rstats of 'rules' in a row.
>
> and for some reason, fuse is trying to write /mnt/plumb/rules
> a second time before closing the first instance.  this gives an
> error on the second write and only that is being returned.
>
> <- Topen tag 0 fid 4 mode 17
> -> Rerror tag 0 ename file already open
> FUSE <- unique 0 error -34 Result too large
> <- Tclunk tag 0 fid 4
> /mnt/plumb/rules: rc: can't open: Result too large
>
> - erik
>

[-- Attachment #2: Type: text/html, Size: 990 bytes --]

Re: [9fans] Fwd: Reading from FS with inaccurate file sizes?

[Russ Cox]

When you cat >> file and file has zero length in stat,
then the write happens at an offset of zero.
This is perhaps the only time that the stat size should
be used for a purpose other than returning from stat.

Synthetic file systems tend not to care about the
offset on writes anyway.

There is another difference between > and >> on Plan 9:
when the shell opens with > it includes OTRUNC;
when it opens with >> it does not.
This is enough of a hint for file systems that care.

On Linux apparently things happen the other way around:
O_TRUNC is never sent, but O_APPEND is sent for >> opens.
MacFUSE doesn't send either, which is another bug I've filed:
http://code.google.com/p/macfuse/issues/detail?id=132

MacFUSE also seems to employ some subterfuge where fds
do not map one-to-one with FUSE file handles.  Another bug I've filed:
http://code.google.com/p/macfuse/issues/detail?id=133

To be fair, these are the kinds of mistakes I would expect any
Unix-mindset implementation to make, and it surprised me quite
a bit that Linux FUSE got so much of this right from the start
(or at least from when I started using it).  I wonder how many
of these mistakes BSD FUSE makes.

Russ

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 27, 6:20 am, r...@swtch.com (Russ Cox) wrote:
> To be fair, these are the kinds of mistakes I would expect any
> Unix-mindset implementation to make, and it surprised me quite
> a bit that Linux FUSE got so much of this right from the start
> (or at least from when I started using it).  I wonder how many
> of these mistakes BSD FUSE makes.

You're assuming quite a bit here, especially in concluding that these
are "mistakes" that you "expect" because of a "Unix-mindset"
implementation.

BTW, I don't know when you started using FUSE on Linux, but it's been
there on Linux at least since 2001. MacFUSE came out in 2007, so your
surprise is surprising.

> Synthetic file systems tend not to care about the
> offset on writes anyway.

And the Mac OS X VFS kernel extension environment isn't exactly geared
towards synthetic file systems. OS X may have an open source kernel,
*but* it's not practical to write kernel extensions that require
kernel changes. Therefore, things a kernel extension can do is limited
by the interfaces/data that are available to the extension in a stock
kernel. In the case of reads/writes when the advertised size is 0, you
run into the unified buffer cache, which really wants to believe the
file size. To get around this, MacFUSE must explicitly implement
separate read/write paths from the vnode operations to user-space and
back. Release 0.2.2 does this for reads if you use the 'direct_io'
option. In other words, if you add the 'direct_io' option while
mounting, what you are looking for should already work. Note that you
will have no buffer cache (which is what you'd want anyway in this
case).

'direct_io' doesn't do anything for writes in Release 0.2.2. It'd be
straightforward to expand the write implementation. A future release
of MacFUSE might have it.

> MacFUSE also seems to employ somesubterfuge where fds
> do not map one-to-one with FUSE file handles.  Another bug I've filed:http://code.google.com/p/macfuse/issues/detail?id=133

The subterfuge is intentional and necessary in the current design. The
open() and close() vnode operations of MacFUSE *do not* have access to
the file descriptor in question. The data structures involved are
opaque, so it'd be quite ugly and unmaintainable to try to get at the
descriptor by brute force. Given the lack of descriptor, you can't
match opens and closes. Along the same lines, MacFUSE only can look at
the vnode, and *not* at file structures, which are inaccessible. You
can't track connections between file structures and FUSE file handles.
Therefore, as a matter of feasibility and simplicity, MacFUSE shares
file handles when possible, with reference counting. For multiple
opens of a single given file, you won't see every open invocation go
up to user space unless the open flags are different from a previous
invocation.

> On Linux apparently things happen the other way around:
> O_TRUNC is never sent, but O_APPEND is sent for >> opens.
> MacFUSE doesn't send either, which is another bug I've filed:http://code.google.com/p/macfuse/issues/detail?id=132
>

Right now, MacFUSE distinguishes between 3 types of open for a given
file: O_RDONLY, O_WRONLY, and O_RDWR. Since write handles could be
shared, adding O_APPEND to the mix means we essentially have two
additional types of open that MacFUSE must track. This isn't too big
of a deal eventually, but the extra complexity wasn't justified in
MacFUSE's nascent days, even though it meant sacrificing some arguably
contrived semantics. I say contrived because O_APPEND is still handled
correctly by the kernel (if you report the correct file size)--it's
just that the flag is not passed to user space. So, like you said in
your bug report, this matters in cases like "shared append-only files
on the server side".

Hope this clarifies things.

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[jas]

If it's an issue for people, sign in to code.google.com and mark/star
it to hopefully convence the MacFUSE team that it's an important thing
to address.  Otherwise it will just be pushed to the bottom of the
stack or flushed as not an issue.

On Mar 27, 8:20 am, r...@swtch.com (Russ Cox) wrote:
...
> To be fair, these are the kinds of mistakes I would expect any
> Unix-mindset implementation to make, and it surprised me quite
> a bit that Linux FUSE got so much of this right from the start
> (or at least from when I started using it).  I wonder how many
> of these mistakes BSD FUSE makes.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Charles Forsyth]

>And the Mac OS X VFS kernel extension environment isn't exactly geared
>towards synthetic file systems. ...

as far as i know, hardly any of them are.  that's the `unix mindset'!

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Uriel]

> BTW, I don't know when you started using FUSE on Linux, but it's been
> there on Linux at least since 2001. MacFUSE came out in 2007, so your
> surprise is surprising.

9P has been around for 20 years, what I find surprising is why anyone
would bother with FUSE.


> The subterfuge is intentional and necessary in the current design. The
> open() and close() vnode operations of MacFUSE *do not* have access to
> the file descriptor in question. The data structures involved are
> opaque, so it'd be quite ugly and unmaintainable to try to get at the
> descriptor by brute force. Given the lack of descriptor, you can't
> match opens and closes. Along the same lines, MacFUSE only can look at
> the vnode, and *not* at file structures, which are inaccessible. You
> can't track connections between file structures and FUSE file handles.
> Therefore, as a matter of feasibility and simplicity, MacFUSE shares
> file handles when possible, with reference counting. For multiple
> opens of a single given file, you won't see every open invocation go
> up to user space unless the open flags are different from a previous
> invocation.

The more I learn about FUSE, the more broken by design it seems. What
is the point of userspace file systems if they can't even handle
open() and close() in any meaningful way?

Ah, the eternal re-invention of square wheels. How infinitely sad *sigh*

uriel

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Francisco J Ballesteros]

Well, I do bother.
I use (or rather, would like to use) FUSE to mount in MacOS a namespace
from a local inferno. That´s what we do to make all the devices in our octopus
available even for the host system.

I mean, I´d like to get fuse for 9p in macosx working properly.
I´m sorry I did not put the effort for doing it, that´s why I did not
complaint, but
it´s still low in my to-do list.


On 3/29/07, Uriel <uriel99@gmail.com> wrote:
> > BTW, I don't know when you started using FUSE on Linux, but it's been
> > there on Linux at least since 2001. MacFUSE came out in 2007, so your
> > surprise is surprising.
>
> 9P has been around for 20 years, what I find surprising is why anyone
> would bother with FUSE.
>
>
> > The subterfuge is intentional and necessary in the current design. The
> > open() and close() vnode operations of MacFUSE *do not* have access to
> > the file descriptor in question. The data structures involved are
> > opaque, so it'd be quite ugly and unmaintainable to try to get at the
> > descriptor by brute force. Given the lack of descriptor, you can't
> > match opens and closes. Along the same lines, MacFUSE only can look at
> > the vnode, and *not* at file structures, which are inaccessible. You
> > can't track connections between file structures and FUSE file handles.
> > Therefore, as a matter of feasibility and simplicity, MacFUSE shares
> > file handles when possible, with reference counting. For multiple
> > opens of a single given file, you won't see every open invocation go
> > up to user space unless the open flags are different from a previous
> > invocation.
>
> The more I learn about FUSE, the more broken by design it seems. What
> is the point of userspace file systems if they can't even handle
> open() and close() in any meaningful way?
>
> Ah, the eternal re-invention of square wheels. How infinitely sad *sigh*
>
> uriel
>
>

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Boris Maryshev]

Unix or not, it would be much appreciated if different fuse  
implementations were compatible.

On 29 Mar 2007, at 12:59, Francisco J Ballesteros wrote:

> Well, I do bother.
> I use (or rather, would like to use) FUSE to mount in MacOS a  
> namespace
> from a local inferno. That´s what we do to make all the devices in  
> our octopus
> available even for the host system.
>
> I mean, I´d like to get fuse for 9p in macosx working properly.
> I´m sorry I did not put the effort for doing it, that´s why I did not
> complaint, but
> it´s still low in my to-do list.
>
>
> On 3/29/07, Uriel <uriel99@gmail.com> wrote:
>> > BTW, I don't know when you started using FUSE on Linux, but it's  
>> been
>> > there on Linux at least since 2001. MacFUSE came out in 2007, so  
>> your
>> > surprise is surprising.
>>
>> 9P has been around for 20 years, what I find surprising is why anyone
>> would bother with FUSE.
>>
>>
>> > The subterfuge is intentional and necessary in the current  
>> design. The
>> > open() and close() vnode operations of MacFUSE *do not* have  
>> access to
>> > the file descriptor in question. The data structures involved are
>> > opaque, so it'd be quite ugly and unmaintainable to try to get  
>> at the
>> > descriptor by brute force. Given the lack of descriptor, you can't
>> > match opens and closes. Along the same lines, MacFUSE only can  
>> look at
>> > the vnode, and *not* at file structures, which are inaccessible.  
>> You
>> > can't track connections between file structures and FUSE file  
>> handles.
>> > Therefore, as a matter of feasibility and simplicity, MacFUSE  
>> shares
>> > file handles when possible, with reference counting. For multiple
>> > opens of a single given file, you won't see every open  
>> invocation go
>> > up to user space unless the open flags are different from a  
>> previous
>> > invocation.
>>
>> The more I learn about FUSE, the more broken by design it seems. What
>> is the point of userspace file systems if they can't even handle
>> open() and close() in any meaningful way?
>>
>> Ah, the eternal re-invention of square wheels. How infinitely sad  
>> *sigh*
>>
>> uriel
>>
>>

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 29, 2:26 am, fors...@terzarima.net (Charles Forsyth) wrote:
> >And the Mac OS X VFS kernel extension environment isn't exactly geared
> >towards synthetic file systems. ...
>
> as far as i know, hardly any of them are.  that's the `unix mindset'!

First off, as much as I find synthetic file systems to be useful, I
don't have a major problem with the Unix mindset.

The way I read the post to which I replied earlier, "mistakes made
because of a Unix mindset" was referring to the MacFUSE
implementation, not Mac OS X itself.

Remember that Mac OS X doesn't even have a proc file system. Its
architecture is also quite stringent about what third parties can do
in the kernel. Therefore, it "isn't exactly geared" even more than
other common Unix-like systems.

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 29, 3:15 am, boris.marys...@gmail.com (Boris Maryshev) wrote:
> Unix or not, it would be much appreciated if different fuse
> implementations were compatible.

Certainly. However:

FUSE was originally designed/implemented for Linux. Unlike Mac OS X,
which has a FreeBSD-like (but not the same) vnode-centric file system
architecture, the Linux file system layer is file-centric. This causes
several issues, combined with the nature of Mac OS X kernel
interfaces. Therefore, a 100% faithful FUSE API implementation is not
currently feasible on Mac OS X *under realistic circumstances*. That's
all.

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 29, 2:55 am, urie...@gmail.com (Uriel) wrote:
> The more I learn about FUSE, the more broken by design it seems. What
> is the point of userspace file systems if they can't even handle
> open() and close() in any meaningful way?
>
> Ah, the eternal re-invention of square wheels. How infinitely sad *sigh*

MacFUSE (http://code.google.com/p/macfuse) is an implementation of the
FUSE API (http://fuse.sourceforge.net) for Mac OS X. You seem to imply
I was explaining *the FUSE design*--no, I was talking about MacFUSE's
departure from the FUSE/Linux behavior because of the underlying OS
differences.

As for eternal reinvention, that's hardly limited to userspace file
systems, or even software. No, the ubiquity of reinvention doesn't
justify reinvention, but I don't really find it infinitely sad--some
instances of (re)invention just don't succeed the way others might.
What *is* sad (well, disappointing rather than sad) is when people are
ultra-quick in jumping to apparently confident conclusions, especially
when they might not have the necessary context. That's the sole reason
I posted here to begin with: to provide some context so those
interested can understand why something is the way it is, rather than
concluding based on presumptions and stereotypes.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[W B Hacker]

Amit Singh wrote:
> On Mar 29, 3:15 am, boris.marys...@gmail.com (Boris Maryshev) wrote:
>> Unix or not, it would be much appreciated if different fuse
>> implementations were compatible.
> 
> Certainly. However:
> 
> FUSE was originally designed/implemented for Linux. Unlike Mac OS X,
> which has a FreeBSD-like (but not the same) vnode-centric file system
> architecture, the Linux file system layer is file-centric. This causes
> several issues, combined with the nature of Mac OS X kernel
> interfaces. Therefore, a 100% faithful FUSE API implementation is not
> currently feasible on Mac OS X *under realistic circumstances*. That's
> all.

Not to complicate the issue, but Mac is not limited to BSD 'like'.

I run all those I install with UFS-only. Grant, Mac's UFS is a few releases 
behind BSD's, but at least it is not hfs / hfs+.

Which - AFAIK, has sod-all to do with FUSE in any case. But at least gives me 
BSD-compatible filenames as well as a faster fs.

And, JFWIW, Mac's UFS supports Inferno-for-OS X just fine.  So AFAIK, a Mac with 
one or more UFS partitions might not have as great a need for FUSE.

Bill Hacker

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Colin DeVilbiss]

On 3/29/07, Amit Singh <asingh@gmail.com> wrote:

> What *is* sad (well, disappointing rather than sad) is when people are
> ultra-quick in jumping to apparently confident conclusions, especially
> when they might not have the necessary context. That's the sole reason
> I posted here to begin with: to provide some context so those
> interested can understand why something is the way it is, rather than
> concluding based on presumptions and stereotypes.

For what it's worth, thanks for posting.  As the original poster, I appreciate the time you put in to making the project in the first place.

Are patches welcome?  If so, I might work on the two "easy" ones: direct_io for writes and O_APPEND.

I need to reread the part of the post about duplicate FDs and such.

Thanks again;
Colin DeVilbiss

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Francisco J Ballesteros]

> And, JFWIW, Mac's UFS supports Inferno-for-OS X just fine.  So AFAIK, a Mac with
> one or more UFS partitions might not have as great a need for FUSE.

I really think it has the need. The main point is that files from the
inferno are not real
files, they provide services instead. So, to make those services
available for the mac,
I mount in macos inferno file trees.

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 29, 6:40 am, cdevilb...@gmail.com (Colin DeVilbiss) wrote:
> Are patches welcome?  If so, I might work on the two "easy" ones: direct_io for writes and O_APPEND.

Patches are welcome. direct_io for writes is already included in the
next release, so don't bother with that. Feel free to give O_APPEND a
shot, though I might look at it when I get a chance. If you do, it'd
be more appropriate to discuss things on the macfuse-devel mailing
list.

> I need to reread the part of the post about duplicate FDs and such.

One-to-one mapping of file descriptors to FUSE file handles is not a
matter of hard-to-implement or not-yet-implemented functionality. It's
not feasible given the current kernel.

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 29, 6:37 am, w...@conducive.org (W B Hacker) wrote:

> Not to complicate the issue, but Mac is not limited to BSD 'like'.

I'm not sure I understand this. The VFS layer is the same regardless
of which file system you use, so the BSD-like VFS layer is always
going to be just that: BSD-like.

> I run all those I install with UFS-only. Grant, Mac's UFS is a few releases
> behind BSD's, but at least it is not hfs / hfs+.
> Which - AFAIK, has sod-all to do with FUSE in any case. But at least gives me
> BSD-compatible filenames as well as a faster fs.

On Mac OS X, there are many, many reasons to use journaled HFS+ over
UFS, and very few reasons to do otherwise. "Faster fs" is not one of
the latter reasons. I'm curious if you've performed, or are aware of,
some benchmarks that show UFS on Mac OS X to be faster overall than HFS
+, assuming some reasonable definition of "faster".

By the way, beginning with Mac OS X 10.3 (Panther), HFS+ comes in a
case-sensitive flavor too.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[David Leimbach]

On 3/29/07, W B Hacker <wbh@conducive.org> wrote:
> Amit Singh wrote:
> > On Mar 29, 3:15 am, boris.marys...@gmail.com (Boris Maryshev) wrote:
> >> Unix or not, it would be much appreciated if different fuse
> >> implementations were compatible.
> >
> > Certainly. However:
> >
> > FUSE was originally designed/implemented for Linux. Unlike Mac OS X,
> > which has a FreeBSD-like (but not the same) vnode-centric file system
> > architecture, the Linux file system layer is file-centric. This causes
> > several issues, combined with the nature of Mac OS X kernel
> > interfaces. Therefore, a 100% faithful FUSE API implementation is not
> > currently feasible on Mac OS X *under realistic circumstances*. That's
> > all.
>
> Not to complicate the issue, but Mac is not limited to BSD 'like'.
>
> I run all those I install with UFS-only. Grant, Mac's UFS is a few releases
> behind BSD's, but at least it is not hfs / hfs+.
>
> Which - AFAIK, has sod-all to do with FUSE in any case. But at least gives me
> BSD-compatible filenames as well as a faster fs.

Got some numbers to back that up?  Or links?  I'm curious.  Because
HFS's many variations (some of which ARE case sensitive) actually do
things like hot clustering and background defragmentation that should,
in theory, help keep things running nicely for quite some time.

>
> And, JFWIW, Mac's UFS supports Inferno-for-OS X just fine.  So AFAIK, a Mac with
> one or more UFS partitions might not have as great a need for FUSE.

I've never had a problem on Mac OS X using Inferno with HFS+, but I
see very little that makes this invalidate uses for FUSE.

Of course your usage may differ from mine, and likely does :-)

Dave

>
> Bill Hacker
>
>


-- 
- Passage Matthew 5:37:
   But let your communication be, Yea, yea; Nay, nay: for whatsoever
is more than these cometh of evil.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[W B Hacker]

David Leimbach wrote:
> On 3/29/07, W B Hacker <wbh@conducive.org> wrote:

*snip* (ufs on mac vs hfs+)

>> BSD-compatible filenames as well as a faster fs.
> 
> Got some numbers to back that up?

Unfortunately not.

I used hfs+ for several months, and (coming off hpfs-386, jfs, and ufs) was 
convinced it was naught but the dying embers of the 'Woz machine' era.

Odd-man-out in any case.

I use ufs so the the detachable HDD and flash are readable across the rest of my 
environment.

Right after the change, it seemed the lowly 1 GHz G4 was on steroids, and I've 
seen the same 'perceived' speedup on 2 OSX 10.3X and 4 10.4X Mac Mini as well, 
though I've retained a largish hfs+ partition on those to support VPC & such.

 > Or links?

Most of the links I found while researching and planning the change were in the 
10.1 era.

 > I'm curious.  Because
> HFS's many variations

- don't forget that some 'hfs' are the Hewlett-Packard File System, not related, 
AFAIK, while others are not relevant to modern device sizes.

 > (some of which ARE case sensitive)

Yes and no. Case-preserving (finally) yes. Sort of.

Case-agnostic, and - as importantly, since I work in Chinese AND not just UTF-8, 
- *encoding-agnostic*, it is not.

There are some comparisons at:

http://en.wikipedia.org/wiki/Comparison_of_file_systems

Fossil is covered, Venti is absent.

 > actually do
> things like hot clustering and background defragmentation that should,
> in theory, help keep things running nicely for quite some time.
> 

As with cooking, 'clean-as-you-go' is more efective than leaving a mess for 
later, so far better to not fragment in the first place.

Ergo not an issue here (hpfs, jfs2, ufs, ufs2).

>>
>> And, JFWIW, Mac's UFS supports Inferno-for-OS X just fine.  So AFAIK, 
>> a Mac with
>> one or more UFS partitions might not have as great a need for FUSE.
> 
> I've never had a problem on Mac OS X using Inferno with HFS+, but I
> see very little that makes this invalidate uses for FUSE.
> 
> Of course your usage may differ from mine, and likely does :-)
> 
> Dave
> 

My impressions of Inferno are that it seems to not be about the VM or even Limbo 
per se, but those as a means to the end of making the Plan9 concepts more easily 
integrated with, and propagated to, the 'rest of the world'

I respect that (apparent) goal.

But hardware is cheap enough, and x86 ubiquitous enough to JFDI with 'native' Plan9.

At least for now...


;-)

Bill

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[erik quanstrom]

On Sat Mar 31 14:41:49 EDT 2007, wbh@conducive.org wrote:
> There are some comparisons at:
> 
> http://en.wikipedia.org/wiki/Comparison_of_file_systems
> 
> Fossil is covered, Venti is absent.

venti is not a filesystem.  it archives blocks and stores them via their
sha1 hash.  it is discussed on wikipedia under the fossil link.

- erik

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[C H Forsyth]

>My impressions of Inferno are that it seems to not be about the VM or even Limbo 
>per se, but those as a means to the end of making the Plan9 concepts more easily 
>integrated with, and propagated to, the 'rest of the world'

it's a decent modular, concurrent programming language (ie, Limbo)
and the same environment everywhere providing a name space interface with the file service protocol
(the Plan 9 concepts) to things other systems make `special'.
the virtual machine is possibly less important: it's mainly a mechanism for easy portability
and easy run-time code generation but could be replaced by another scheme.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[W B Hacker]

C H Forsyth wrote:
>> My impressions of Inferno are that it seems to not be about the VM or even
>> Limbo per se, but those as a means to the end of making the Plan9 concepts
>> more easily integrated with, and propagated to, the 'rest of the world'
> 
> it's a decent modular, concurrent programming language (ie, Limbo) and the
> same environment everywhere providing a name space interface with the file
> service protocol (the Plan 9 concepts) to things other systems make
> `special'.

No rocks to throw in that direction...

 > the virtual machine is possibly less important: it's mainly a
> mechanism for easy portability and easy run-time code generation but could be
> replaced by another scheme.

Or that either.

Though I think it better for eval to trial Plan9 on dedicated hardware, if only 
to sever Inferno and Plan9 for clarity as to who does what, with which, and to whom.

Guess the 'bigger question'  - taking just the 'symptoms' [1] - is why, 
presuming the concepts (Plan9, and any/all 'sputniks') have value [2], it/they 
is/are not in broader use already.

Bill



[1] Active 9Fans list, many brght minds, depth of experience, countered by Vita 
Nouva major news items / 'design wins' at somewhere around 2+ year intervals?

[2] pick a qualifier:

- enough

- current

- other-than conceptual/experimental

- combination(s) thereof

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Charles Forsyth]

>Guess the 'bigger question'  - taking just the 'symptoms' [1] - is why, 
>presuming the concepts (Plan9, and any/all 'sputniks') have value [2], it/they 
>is/are not in broader use already.

because to be useful, they need to be done at a fundamental level,
there's too much inertia represented in existing things, and
the emphasis for some time has been on languages, not systems,
and XML and w3c, not systems.  you can see that when people quite seriously say
things like ``we won't even need an operating system in future, just a browser [and plug-ins]''
without realising that all they've really done is turn the browser into the operating system.
which still leaves the problem of coming up with a reasonable design.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[erik quanstrom]

ah, a reprise of the emacs os idea.

- erik

On Sun Apr  1 06:35:47 EDT 2007, forsyth@terzarima.net wrote:
> and XML and w3c, not systems.  you can see that when people quite seriously say
> things like ``we won't even need an operating system in future, just a browser [and plug-ins]''
> without realising that all they've really done is turn the browser into the operating system.
> which still leaves the problem of coming up with a reasonable design.

Re: [9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Francisco J Ballesteros]

Only that this time it's not lisp. You get the same complexity yet you
are not able
to trace all the internals of your system, errr, browser.

On 4/1/07, erik quanstrom <quanstro@coraid.com> wrote:
> ah, a reprise of the emacs os idea.
>
> - erik
>
> On Sun Apr  1 06:35:47 EDT 2007, forsyth@terzarima.net wrote:
> > and XML and w3c, not systems.  you can see that when people quite seriously say
> > things like ``we won't even need an operating system in future, just a browser [and plug-ins]''
> > without realising that all they've really done is turn the browser into the operating system.
> > which still leaves the problem of coming up with a reasonable design.
>

[9fans] Re: Fwd: Reading from FS with inaccurate file sizes?

[Amit Singh]

On Mar 31, 11:42 am, w...@conducive.org (W B Hacker) wrote:
>
> Unfortunately not.
>
> I used hfs+ for several months, and (coming off hpfs-386, jfs, and ufs) was
> convinced it was naught but the dying embers of the 'Woz machine' era.

Unfortunate indeed. Your metaphor may be cute, but what you're saying
makes absolutely no sense. Apple's HFS and Apple's HFS+ are different,
you realize that, right?

> Yes and no. Case-preserving (finally) yes. Sort of.

By default, HFS+ is case preserving. There is a *case sensitive*
variant called HFSX.