[9fans] page fault with lock held

[9fans] page fault with lock held

[Sam]

Hello 9fans,

I'm testing my AoE driver and have hit something curious.

On devaoe.c read, after getting the sectors from the
remote disk and when trying to copy the data to the
user process using memmove, I get a write page fault.
At the time, I happen to be holding a spinlock over
the device structure, which causes both
port/fault.c:/^fault/ and port/fault.c/^seg/ (via qlock)
to squawk at me about up->nlocks.ref > 0.

The user address in question is 7fffaedc.  Firstly, does
this address seem appropriate and secondly, is dropping
the spinlock during the memmove appropriate?

All signs point to yes, but I'd like some reassurance.

Cheers,

Sam

Re: [9fans] page fault with lock held

[jmk]

no, you should not be holding a spinlock when trying that.
why is your driver trying to copy directly to user space,
isn't it a subdevice of devsd? (i'm not saying that's the way
to do it, just curious).

On Tue Apr 12 13:20:53 EDT 2005, sah@softcardsystems.com wrote:
> Hello 9fans,
> 
> I'm testing my AoE driver and have hit something curious.
> 
> On devaoe.c read, after getting the sectors from the
> remote disk and when trying to copy the data to the
> user process using memmove, I get a write page fault.
> At the time, I happen to be holding a spinlock over
> the device structure, which causes both
> port/fault.c:/^fault/ and port/fault.c/^seg/ (via qlock)
> to squawk at me about up->nlocks.ref > 0.
> 
> The user address in question is 7fffaedc.  Firstly, does
> this address seem appropriate and secondly, is dropping
> the spinlock during the memmove appropriate?
> 
> All signs point to yes, but I'd like some reassurance.
> 
> Cheers,
> 
> Sam

Re: [9fans] page fault with lock held

[Russ Cox]

> no, you should not be holding a spinlock when trying that.
> why is your driver trying to copy directly to user space,
> isn't it a subdevice of devsd? (i'm not saying that's the way
> to do it, just curious).

i'll say that's probably the way to do it.

russ

Re: [9fans] page fault with lock held

[Brantley Coile]

>> no, you should not be holding a spinlock when trying that.
>> why is your driver trying to copy directly to user space,
>> isn't it a subdevice of devsd? (i'm not saying that's the way
>> to do it, just curious).
> 
> i'll say that's probably the way to do it.
> 
> russ

Why?

 Brantley

Re: [9fans] page fault with lock held

[Russ Cox]

> Why?

beacuse it's a disk, and the grand vision was that
sd would cover all the disks.  we never converted
the floppy driver.  it matters more for aoe because
then you don't have to duplicate all the details about
disk partitions and so on.

russ

Re: [9fans] page fault with lock held

[Charles Forsyth]

data structures that must be locked over
long-running operations, and certainly those that might sleep (directly or indirectly)
or invoke i/o elsewhere, should be protected by QLock, not Lock.

Re: [9fans] page fault with lock held

[Sam]

I didn't fit into devsd because the device names (sdXn) weren't
helpful and I need the ability to do extra control processing
I couldn't see how to get from devsd.

e.g.,

I currently have devices named aoe{major}.{minor};
the blade in shelf 10, slot 9 is aoe10.9/.  This
addressing mechanism is much easier to understand
than any sd controller to ethernet controller
mapping I could think up.

Telling devaoe to start using interface ether0 is
done with:

echo bind /net/ether0 >aoectl

There's also an aoeevents file our raid/lvm
software can monitor (for device retransmits, eg).

The interface is still rather pliable as I'm trying to
figure out what will work best.

For each system I've written an aoe driver someone has
asked why I didn't fit into the current code.  It always
turns out that it's easier to cut a new one than bend the
existing code to make it work.  Disk drivers don't typically
have to worry about multiple outstanding requests,
retransmission timeouts, or anytime device discovery.
This isn't to say they don't do command queueing or
support hot plugging, it's just not usually something
I can fit into.

Thanks for the lock tip -

Sam

On Tue, 12 Apr 2005 jmk@plan9.bell-labs.com wrote:

> no, you should not be holding a spinlock when trying that.
> why is your driver trying to copy directly to user space,
> isn't it a subdevice of devsd? (i'm not saying that's the way
> to do it, just curious).
>
> On Tue Apr 12 13:20:53 EDT 2005, sah@softcardsystems.com wrote:
>> Hello 9fans,
>>
>> I'm testing my AoE driver and have hit something curious.
>>
>> On devaoe.c read, after getting the sectors from the
>> remote disk and when trying to copy the data to the
>> user process using memmove, I get a write page fault.
>> At the time, I happen to be holding a spinlock over
>> the device structure, which causes both
>> port/fault.c:/^fault/ and port/fault.c/^seg/ (via qlock)
>> to squawk at me about up->nlocks.ref > 0.
>>
>> The user address in question is 7fffaedc.  Firstly, does
>> this address seem appropriate and secondly, is dropping
>> the spinlock during the memmove appropriate?
>>
>> All signs point to yes, but I'd like some reassurance.
>>
>> Cheers,
>>
>> Sam
>

Re: [9fans] page fault with lock held

[geoff]

I'd encourage fitting aoe into the sd machinery.  Admittedly I haven't
got my sdaoe.c working yet, mostly due to lack of time, but I saw
nothing that would make it difficult to fit aoe into sd.  A little
unorthodox, but not difficult.

Re: [9fans] page fault with lock held

[Dan Cross]

I had wanted to see an sdloopback device (devsdloop was I suppose
what I called it) that could speak SCSI to a file descriptor that
could come from anywhere.  Then things like this might not need to
be in the kernel at all.

	- Dan C.

Re: [9fans] page fault with lock held

[Russ Cox]

> I had wanted to see an sdloopback device (devsdloop was I suppose
> what I called it) that could speak SCSI to a file descriptor that
> could come from anywhere.  Then things like this might not need to
> be in the kernel at all.

sd has little to do with scsi.
in particular, these devices don't speak scsi.

russ

Re: [9fans] page fault with lock held

[Christopher Nielsen]

i agree with geoff here. i don't like the idea of having
a separate interface; everything storage should be connected
into the sd framework.

i have a bit of free time, right now, and if you'd like to
brainstorm off-list about how to integrate aoe into the sd
machinery, i would certainly like to help.

On Tue, Apr 12, 2005 at 01:33:51PM -0700, geoff@collyer.net wrote:
> I'd encourage fitting aoe into the sd machinery.  Admittedly I haven't
> got my sdaoe.c working yet, mostly due to lack of time, but I saw
> nothing that would make it difficult to fit aoe into sd.  A little
> unorthodox, but not difficult.

-- 
Christopher Nielsen
"They who can give up essential liberty for temporary
safety, deserve neither liberty nor safety." --Benjamin Franklin

Re: [9fans] page fault with lock held

[Brantley Coile]

> i agree with geoff here. i don't like the idea of having
> a separate interface; everything storage should be connected
> into the sd framework.
> 
> i have a bit of free time, right now, and if you'd like to
> brainstorm off-list about how to integrate aoe into the sd
> machinery, i would certainly like to help.


We agree as well--except for the reasons Sam mentioned in his earlier
email.  EtherDrives come and go, they have names that don't fit in the
sd way of doing things, and we wanted to get something working quickly
so we can learn how best to marry Plan 9 and AoE.  I think we will
wait until we have some experience under our belt.  I think we may
need to change sd a little to fit things like AoE into it.  It makes
sense to do that after we have something working.

And it's almost working now.


One other thought.  I'm working on some raid software.  It may make
more sense for those devices to appear as sd(3) and the AoE to be
somewhere else.  This will avoid cluttering /dev with devices that you
won't be using as individual blades anyway.  Maybe I need to look into
hooking my raid stuff into that.

  Brantley

Re: [9fans] page fault with lock held

[Charles Forsyth]

>>I think we may need to change sd a little to fit things like AoE into it.  ...

i was going to observe that on some other systems
it can be hard to have changes made to interfaces to suit your purposes,
and sometimes the interfaces are just putrid anyhow, but neither
should be a problem here

Re: [9fans] page fault with lock held

[Dan Cross]

Russ Cox <russcox@gmail.com> writes:
> 
> > I had wanted to see an sdloopback device (devsdloop was I suppose
> > what I called it) that could speak SCSI to a file descriptor that
> > could come from anywhere.  Then things like this might not need to
> > be in the kernel at all.
> 
> sd has little to do with scsi.
> in particular, these devices don't speak scsi.

Maybe I'm confused; it's been a while.  I thought that AoE was ATA over
Ethernet or something like that, and that ATA was basically the SCSI
protocol over different electronics.  I was originally looking at this
in terms of a digital camera that connected via USB, which did speak
SCSI.  Regardless, the salient point was that there should some way to
hook into the devsd framework from userland.

	- Dan C.

Re: [9fans] page fault with lock held

[Brantley Coile]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

ATA has a common and squalid past.
ATAPI is a packet interface layed on top of ATA to pass
SCSI commands to things like CDROMs, but ATA as such
has it's own protocol.

ATA started out as a chip from Western Digitial, the WD1010.  This
ST506 disk controller was used in XT and AT PC. When the silicon could
support it, the functions were moved into the devices and all that was
left was the interface to the 1010.

Logical block addresses have replaced the cyl, head, sector, but much
of the interface is the same as the one I was programming in 1983.
DMA speeds are faster, things like S.M.A.R.T. allow watching for errors
on drives, and now we even have command queueing, but mostly
it's the same stuff.

I like the idea of having a way to hook into the bottom of sd from user
land!

  Brantley

[-- Attachment #2: Type: message/rfc822, Size: 3275 bytes --]

From: Dan Cross <cross@math.psu.edu>
To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu>
Subject: Re: [9fans] page fault with lock held
Date: Wed, 13 Apr 2005 09:13:43 -0400
Message-ID: <200504131313.j3DDDhJ16603@augusta.math.psu.edu>

Russ Cox <russcox@gmail.com> writes:
> 
> > I had wanted to see an sdloopback device (devsdloop was I suppose
> > what I called it) that could speak SCSI to a file descriptor that
> > could come from anywhere.  Then things like this might not need to
> > be in the kernel at all.
> 
> sd has little to do with scsi.
> in particular, these devices don't speak scsi.

Maybe I'm confused; it's been a while.  I thought that AoE was ATA over
Ethernet or something like that, and that ATA was basically the SCSI
protocol over different electronics.  I was originally looking at this
in terms of a digital camera that connected via USB, which did speak
SCSI.  Regardless, the salient point was that there should some way to
hook into the devsd framework from userland.

	- Dan C.

Re: [9fans] page fault with lock held

[Tim Newsham]

> I like the idea of having a way to hook into the bottom of sd from user
> land!

What about the /dev/sd??/raw file?  For scsi devices at least (sdscsi.c)
this calls through to scsirio which directly performs a scsi operation.

>  Brantley

Tim Newsham

Re: [9fans] page fault with lock held

[Dan Cross]

Tim Newsham <newsham@lava.net> writes:
> > I like the idea of having a way to hook into the bottom of sd from user
> > land!
> 
> What about the /dev/sd??/raw file?  For scsi devices at least (sdscsi.c)
> this calls through to scsirio which directly performs a scsi operation.

Isn't that pushing the protocol the opposite direction?

You want a way to tell the kernel, ``here's a file descriptor to a SCSI
device; talk to it and hook it into devsd.''  Not to send a SCSI
command to an existing device that already appears under
/dev/sdwhatever.

	- Dan C.