Re: [9fans] 9p2k, fsync

Re: [9fans] 9p2k, fsync

[jmk]

On Wed Feb  7 13:43:21 EST 2001, schwartz@bio.cse.psu.edu wrote:
> | Fsync etc. is at the wrong level.  The issue is not a system-wide
> | question, but a file-server question.
>
> So how does an application with an open fd assure that the write()s it
> has issued are really on disk?  It's impossible for that program to tell
> which file server you're using.  In fact, you might be using several,
> if they're stacked up.
>

I think you have to believe that when the fileserver replies to your
write that the data is written, just like you do when you say 'fsync';
whether the data is actually on some 'stable storage' is, in both
cases, not known.

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <20010207191447.24C56199E1@mail.cse.psu.edu> you write:
>I think you have to believe that when the fileserver replies to your
>write that the data is written, just like you do when you say 'fsync';
>whether the data is actually on some 'stable storage' is, in both
>cases, not known.

I'm not sure that's a leap of faith that I'm always willing to make.

Also, Scott is asking about the protocol itself, not the current
implementation.  Since one of the main principles of Plan 9 is to
abstract things into filesystems served by file servers all over the
network, which may or may not do various levels of caching, it seems a
mistake to leave something like this out.

To that end, I think that some sort of ``Tsync'' message is a good
idea.

I'll admit that the semantics are murky for the existing file server.
What does ``stable storage'' mean?  The magnetic disk cache?  The
worm?  Something else entirely?

	- Dan C.

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <00dc01c092be$4cc38f20$0ab9c6d4@cybercable.fr> you write:
>exactly.  just what is ftpfs going to do with it?  it has
>no idea of what the ftpd at the other end has done.

ftpfs would ignore it.  :-)

	- Dan C.

Re: [9fans] 9p2k, fsync

[Boyd Roberts]

> Okay, short digression on sports injury.  Those who aren't interested
> in such things, or those of weak stomachs, should skip this message
> now.

> Yeah; it was bad enough going in, and I see no reason to take them
> out.  Plus, to get it out, they have to go through quite a bit of
> muscle mass, and that's a pain to heal.  My radial mobility in that
> arm is already shot, but removing the plate wouldn't help.

yeah, that sounds nasty.

> >i got all of mine taken out on wed: LCDC plate, 10 screws, a pin
> >and some wires.

> Ouch, leg injuries suck (I'm guessing this is a leg injury given the
> apparatus you mention above.

yep tibia and kneecap.  not great bones to break.  good to get the
hardware out because bones are piezo-electric and need compression
forces on them to grow/strengthen.  otherwise they become fragile.

looks like i get to sit around the house for the next month and
annoy 9fans :-)

i haven't advanced any further on azerty support, 'cos a general,
lack of sleep and a head full of morphine sort of mangle your
concentration.  it's trivial to do.  guess i'll create a
/lib/azerty and cat it into /dev/kbmap.  then i'll have a look
at the a 3c574 pcmcia ethernet driver.  being able to type
will help.

Re: [9fans] 9p2k, fsync

[Boyd Roberts]

> fsync was a quick non-fix in the wrong place.

sure was.

Re: [9fans] 9p2k, fsync

[Boyd Roberts]

> It's really the titanium plate in my arm that throws me off
> for using the mouse....

so you're keeping your h/w mods?

i got all of mine taken out on wed: LCDC plate, 10 screws, a pin
and some wires.

Re: [9fans] 9p2k, fsync

[Boyd Roberts]

> I'm sure that if there were a Tsync message in
> the protocol, it would suffer the same problem.
> Nine times out of ten it would be the wrong hammer.

exactly.  just what is ftpfs going to do with it?  it has
no idea of what the ftpd at the other end has done.

Re: [9fans] 9p2k, fsync

[Boyd Roberts]

> You know, the usual thing.  It asks that the file's data and metadata
> be written to nonvolatile memory.

that was always a quagmire.

> A more general purpose way to talk about transactions would be better,
> but that seems like a significant change  instead of a small one.

transactions?  that's a whole new ballgame.

Re: [9fans] 9p2k, fsync

[nemo]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

I was not asking for any T* message, other than the ones in 9P.  I was
asking for a process listening for `innocent' command requests, (in
the way of /srv/kfs*.cmd). I define `innocent' as the subset of console
commands that may not cause harm.


[-- Attachment #2: Type: message/rfc822, Size: 2772 bytes --]

From: "Paul Taysom" <Paul_Taysom@novell.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] 9p2k, fsync
Date: Thu, 08 Feb 2001 10:46:03 -0700
Message-ID: <sa8278f7.052@prv-mail20.provo.novell.com>

Tsync: This way there be dragons.
The next thing you'll be asking for is byte range locks and then
locks that can survive crashes and then something to coordinate
updates to multiple files and then away to back it up . . . . . .

If you want ACID let's look at away of providing for it within the context
of the a simple file protocol.

Ideas for sync:
1. Battery backed static RAM for the more resent requests like NetApp.
2. A volume that does write through or flush on close.

I agree that databases have gotten way out of hand but is there away
within the spirit of Plan 9 to provide ACID?  Sync does not meet the
requirement.

Paul Taysom

>>> nemo@gsyc.escet.urjc.es 02/08/01 07:03AM >>>
:  Hey?  No disagreement that it's a fileserver issue, but I should also
:  be able to ask the file server to sync the data associated with a
:  particular FID.

That's a question I have.  I understand it can be done from its
console, but is there any way to ask 9pcfs for an inmediate sync
through the network?

And another one.  Any plans to make the future fs kernel do software
mirrowing on ide disks?

Re: [9fans] 9p2k, fsync

[forsyth]

>>Okay, but I have no way to indicate to the file server, ``hey, now
>>would be a good time to take care of it....''

>>I don't think that the idea of fsync() is bad, even if the
>>implementation doesn't work as advertised.

>>I guess this just isn't that big of a deal for you guys....

not at all.  you're assuming that there are no
techniques available that provide the properties you seek (possibly
better than that, but never mind), BUT neither request nor require fsync (or Tsync);
in fact, for which fsync/Tsync would anyway be a no-op.

fsync was a quick non-fix in the wrong place.

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <20010208062443.1E903199EC@mail.cse.psu.edu> you write:
>> The issue is wether I, as a programmer, have a way of saying, ``please
>> make sure that this file is committed to stable storage, and not in a
>> RAM buffer somewhere.''  Currently, there is no solution to this
>> problem.
>
>This question has been answered. No, there is no solution.

It wasn't a question; it was a statement of a problem, which as you
say, has no solution.

>Other systems claim to have solutions but they are exaggerating.

I frankly don't care what other systems do or don't do; the issue is
what a system *could* do.

>It's an issue for file servers, not operating systems.

I talk to a file server using a file system protocol, right?

In article <20010208062621.A6F2719A02@mail.cse.psu.edu> you write:
>No one here said, ``the file server will take care of it''.

The statement was something along the lines of, ``you have to trust
that when the file server says it's done writing, it's done.''  (Not
an exact quote.)

>What we're saying is that the file server must take care of it if
>anyone will.

Okay, but I have no way to indicate to the file server, ``hey, now
would be a good time to take care of it....''

I don't think that the idea of fsync() is bad, even if the
implementation doesn't work as advertised.

I guess this just isn't that big of a deal for you guys....

	- Dan C.

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <sa8278f7.052@prv-mail20.provo.novell.com> you write:
>Tsync: This way there be dragons.
>The next thing you'll be asking for is byte range locks and then
>locks that can survive crashes and then something to coordinate
>updates to multiple files and then away to back it up . . . . . .

``My God, will it ever cease?!  Make the bad man stop!''  :-)

>If you want ACID let's look at away of providing for it within the context
>of the a simple file protocol.
>
>Ideas for sync:
>1. Battery backed static RAM for the more resent requests like NetApp.
>2. A volume that does write through or flush on close.
>
>I agree that databases have gotten way out of hand but is there away
>within the spirit of Plan 9 to provide ACID?  Sync does not meet the
>requirement.

Yeah, that seems to be the issue.  I'm going to go out on a limb
and say that Nemo was refering not to acid, though, but to an analogue
of the Unix sync(2) system call, initiated over the network.  My
guess is that could be implemented via a filesystem with a control
message that accepted commands like, ``sync.''  Maybe something similar
could be used to initiate syncs on FID's.

	- Dan C.

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <200102081250.NAA07825@boris.cd.chalmers.se> you write:
>Are you *sure* it is the skateboard that is the problem? :-)

Note that I also said I would use a skateboard truck (the metal part
that connects the wheels to the wooden deck) instead of my hands.  It's
really the titanium plate in my arm that throws me off for using the
mouse....

btw- To be fair and honest, it wasn't so much the skateboard itself as
it was my unfortunate inability to stay on the skateboard at times.

	- Dan C.

Re: [9fans] 9p2k, fsync

[Paul Taysom]

Tsync: This way there be dragons.
The next thing you'll be asking for is byte range locks and then
locks that can survive crashes and then something to coordinate
updates to multiple files and then away to back it up . . . . . .

If you want ACID let's look at away of providing for it within the context
of the a simple file protocol.

Ideas for sync:
1. Battery backed static RAM for the more resent requests like NetApp.
2. A volume that does write through or flush on close.

I agree that databases have gotten way out of hand but is there away
within the spirit of Plan 9 to provide ACID?  Sync does not meet the
requirement.

Paul Taysom

>>> nemo@gsyc.escet.urjc.es 02/08/01 07:03AM >>>
:  Hey?  No disagreement that it's a fileserver issue, but I should also
:  be able to ask the file server to sync the data associated with a
:  particular FID.

That's a question I have.  I understand it can be done from its
console, but is there any way to ask 9pcfs for an inmediate sync
through the network?

And another one.  Any plans to make the future fs kernel do software
mirrowing on ide disks?

Re: [9fans] 9p2k, fsync

[nemo]

:  Hey?  No disagreement that it's a fileserver issue, but I should also
:  be able to ask the file server to sync the data associated with a
:  particular FID.

That's a question I have.  I understand it can be done from its
console, but is there any way to ask 9pcfs for an inmediate sync
through the network?

And another one.  Any plans to make the future fs kernel do software
mirrowing on ide disks?

Re: [9fans] 9p2k, fsync

[Laura Creighton]

Earlier Dan Cross wrote:

>	Yeah.  Also, for those of us who have fallen enough times that our
>	hands don't work all that well anymore, the chording required to make a
>	2 button mouse work like a 3 button mouse is pretty uncomfortable.
>	Man, my skateboard used to be pretty evil to me when I was younger.
>

Now he writes:

>	Yeah, I won't disagree with you there, but that doesn't make it any
>	less useful.  It's a choice between a hammer for both nails and screws,
>	or using my hands for both.

Are you *sure* it is the skateboard that is the problem? :-)

Laura Creighton

Re: [9fans] 9p2k, fsync

[Douglas A. Gwyn]

Dan Cross wrote:
> Multics had this problem; no synchronization primitives for segments.
> It was impossible to build a database on it for many years (maybe they
> fixed it eventually; I don't know), because one couldn't guarantee
> serialization of transactions on persistent media.  It would be a shame
> to send Plan 9 down the same road for the exact same reason.

I think that all that is required is consistency.
The server is the right place to enforce that,
and I'd suggest making it an explicit requirement.
How about this: To ensure that a 9P2k message has
been integrated into the server state, just wait
for an ack for the message, or if it is one-way,
send some innocuous request and wait for it to
return data.  Getting the feedback from the server
guarantees that the server (which is responsible
for maintaining consistency) has successfully
processed the previous request.  Of course, if the
service does a lousy job you have reason to demand
an improved server.

I once encountered a disk drive that apparently
wrote data to disk successfully, except it didn't
modify the sectors that it was supposed to have
written.  In the face of such phenomena it is
pointless to insist on absolute guarantees of
synchronization.  Good design calls for such
secondary matters to be handled as close to the
source of the problem as possible, rather than
intruding such low-level considerations into
higher logical levels.  If something does happen
at a lower level that necessarily affects things
at a higher level, then so be it, but if it
couldn't be solved down there it can't be solved
up here either, and up here dealing with it
distracts from the more strategic concerns.
(For similar reasons I am a fan of nested
exception handling instead of functions returning
failure codes, but that's another thread.)

Re: [9fans] 9p2k, fsync

[Nikolai Saoukh]

> Not really; if my application depends on that data being on stable
> storage, not just a RAM buffer, I could still be in trouble.  The
> scenarios are obvious, but the point is that I might have a real need
> for this sort of thing, and ``the file server will take care of it''
> isn't really good enough.

Why not to have part of file server namespace being on stable storage?
Then with sync. write you would be happy.

Re: [9fans] 9p2k, fsync

[rob pike]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

No one here said, ``the file server will take care of it''. What we're
saying is that the file server must take care of it if anyone will.

-rob


[-- Attachment #2: Type: message/rfc822, Size: 3081 bytes --]

From: Dan Cross <cross@math.psu.edu>
To: 9fans@cse.psu.edu
Cc: 
Subject: Re: [9fans] 9p2k, fsync
Date: Thu, 8 Feb 2001 01:01:27 -0500 (EST)
Message-ID: <200102080601.BAA04455@augusta.math.psu.edu>

In article <200102080115.UAA26741@smtp1.fas.harvard.edu> you write:
>There are no asynchronous writes here.

Define asychronous in this context.  There are no asychronous network
transactions, true, but that's not the same thing as an asychronous
write

>By the time the write(2) system call returns,
>the file server has acknowledged receipt of the data.
>At that point, it's the file server's business.

Not really; if my application depends on that data being on stable
storage, not just a RAM buffer, I could still be in trouble.  The
scenarios are obvious, but the point is that I might have a real need
for this sort of thing, and ``the file server will take care of it''
isn't really good enough.

>Fsync is simultaneously too much and too little.
>People who care often care about making sure
>individual blocks are committed to storage, and fsync
>is the only hammer they have.

When the file is modeled as a stream of bytes, ie, the blocks are
abstracted away from me as the programmer, I'm not sure that hammer
isn't the only one appropriate for the nails I'm given.

>I'm sure that if there were a Tsync message in
>the protocol, it would suffer the same problem.
>Nine times out of ten it would be the wrong hammer.

Yeah, I won't disagree with you there, but that doesn't make it any
less useful.  It's a choice between a hammer for both nails and screws,
or using my hands for both.

	- Dan C.

(Of course, I'd use a skateboard truck instead of my hands, but you get
the idea.  I had to do that once repairing a ramp.  :-)

Re: [9fans] 9p2k, fsync

[rob pike]

> The issue is wether I, as a programmer, have a way of saying, ``please
> make sure that this file is committed to stable storage, and not in a
> RAM buffer somewhere.''  Currently, there is no solution to this
> problem.

This question has been answered. No, there is no solution.  Other
systems claim to have solutions but they are exaggerating. It's an
issue for file servers, not operating systems.

-rob

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <200102080115.UAA26741@smtp1.fas.harvard.edu> you write:
>There are no asynchronous writes here.

Define asychronous in this context.  There are no asychronous network
transactions, true, but that's not the same thing as an asychronous
write

>By the time the write(2) system call returns,
>the file server has acknowledged receipt of the data.
>At that point, it's the file server's business.

Not really; if my application depends on that data being on stable
storage, not just a RAM buffer, I could still be in trouble.  The
scenarios are obvious, but the point is that I might have a real need
for this sort of thing, and ``the file server will take care of it''
isn't really good enough.

>Fsync is simultaneously too much and too little.
>People who care often care about making sure
>individual blocks are committed to storage, and fsync
>is the only hammer they have.

When the file is modeled as a stream of bytes, ie, the blocks are
abstracted away from me as the programmer, I'm not sure that hammer
isn't the only one appropriate for the nails I'm given.

>I'm sure that if there were a Tsync message in
>the protocol, it would suffer the same problem.
>Nine times out of ten it would be the wrong hammer.

Yeah, I won't disagree with you there, but that doesn't make it any
less useful.  It's a choice between a hammer for both nails and screws,
or using my hands for both.

	- Dan C.

(Of course, I'd use a skateboard truck instead of my hands, but you get
the idea.  I had to do that once repairing a ramp.  :-)

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <20010208013257.8F20B199E3@mail.cse.psu.edu> you write:
>Actually, one of Plan 9's earliest commercial customers used it to
>support a massive database-like application on serious hardware.  The
>lack of fsync didn't seem to get in the way.

I have no doubt that I can build a transactional protocol on top of
TCP, for instance, but that's not the issue.  Did this project use 9P
directly?

The issue is wether I, as a programmer, have a way of saying, ``please
make sure that this file is committed to stable storage, and not in a
RAM buffer somewhere.''  Currently, there is no solution to this
problem.

	- Dan C.

Re: [9fans] 9p2k, fsync

[rob pike]

Actually, one of Plan 9's earliest commercial customers used it to
support a massive database-like application on serious hardware.  The
lack of fsync didn't seem to get in the way.

-rob

Re: [9fans] 9p2k, fsync

[Dan Cross]

In article <20010207152343.C1B91199E1@mail.cse.psu.edu> you write:
>Fsync etc. is at the wrong level.  The issue is not a system-wide
>question, but a file-server question.  As a trivial example of how
>to think about this problem,
>	disk/kfscmd sync
>strikes closer to the way the problem should be handled than
>a 'sync' system call.  The details of how a file server backs up
>and stabilizes its storage are specific to the server, and should
>be implemented by it.  Plan 9 conventions show how to provide
>such an interface; the file protocol is not the place.

Hey?  No disagreement that it's a fileserver issue, but I should also
be able to ask the file server to sync the data associated with a
particular FID.

Multics had this problem; no synchronization primitives for segments.
It was impossible to build a database on it for many years (maybe they
fixed it eventually; I don't know), because one couldn't guarantee
serialization of transactions on persistent media.  It would be a shame
to send Plan 9 down the same road for the exact same reason.

	- Dan C.

Re: [9fans] 9p2k, fsync

[Russ Cox]

There are no asynchronous writes here.
By the time the write(2) system call returns,
the file server has acknowledged receipt of the data.
At that point, it's the file server's business.

Fsync is simultaneously too much and too little.
People who care often care about making sure
individual blocks are committed to storage, and fsync
is the only hammer they have.

I'm sure that if there were a Tsync message in
the protocol, it would suffer the same problem.
Nine times out of ten it would be the wrong hammer.

Russ

Re: [9fans] 9p2k, fsync

[Scott Schwartz]

| Fsync etc. is at the wrong level.  The issue is not a system-wide
| question, but a file-server question.

So how does an application with an open fd assure that the write()s it
has issued are really on disk?  It's impossible for that program to tell
which file server you're using.  In fact, you might be using several,
if they're stacked up.

Re: [9fans] 9p2k, fsync

[Scott Schwartz]

> What do you mean by 'stable storage' and how does 'fsync' assure that?

You know, the usual thing.  It asks that the file's data and metadata
be written to nonvolatile memory.  Flush OS and network level buffers,
and then ask the disk to sync its cache if possible.  (If not then at
least there's an api to ask for it.)  In the case of a file server backed
by something more abstract, the semantics might be more abstract.

A more general purpose way to talk about transactions would be better,
but that seems like a significant change  instead of a small one.

Re: [9fans] 9p2k, fsync

[rob pike]

Fsync etc. is at the wrong level.  The issue is not a system-wide
question, but a file-server question.  As a trivial example of how
to think about this problem,
	disk/kfscmd sync
strikes closer to the way the problem should be handled than
a 'sync' system call.  The details of how a file server backs up
and stabilizes its storage are specific to the server, and should
be implemented by it.  Plan 9 conventions show how to provide
such an interface; the file protocol is not the place.

-rob

Re: [9fans] 9p2k, fsync

[jmk]

Scott Schwartz <schwartz@bio.cse.psu.edu>:
> What sort of reliability guarantees will the new 9p offer?  In the past,
> Plan 9 had no equivalent of fsync, and no assurance that any particular
> operation was really committed to stable storage.

What do you mean by 'stable storage' and how does 'fsync' assure that?

[9fans] 9p2k, fsync

[Scott Schwartz]

What sort of reliability guarantees will the new 9p offer?  In the past,
Plan 9 had no equivalent of fsync, and no assurance that any particular
operation was really committed to stable storage.