[9fans] RDMA: DMA over TCP/IP at IETF

[9fans] RDMA: DMA over TCP/IP at IETF

[George Michaelson]

simple abstraction or inevitable nightmare?

	http://www.ietf.org/ietf/01dec/roi.txt

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[Douglas A. Gwyn]

George Michaelson wrote:
> simple abstraction or inevitable nightmare?

I don't think it's really either of those,
but rather an adjustment of implementation
to avoid wasting so many cycles on simply
copying data around when it should already
be within reach of the application.  This
kind of thing has been done within a host
many times, e.g. on some variants of Unix
there were hacks so that a user-mode buffer
that an application arranged to be page-
aligned would be directly mapped onto the
kernel address space so DMA (for example)
would directly get/put the data where the
application wanted it.  It would of course
be much better if this didn't have to be
done as a hack.

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[Boyd Roberts]

"Douglas A. Gwyn" wrote:
> many times, e.g. on some variants of Unix
> there were hacks so that a user-mode buffer
> that an application arranged to be page-
> aligned would be directly mapped onto the
> kernel address space so DMA (for example)
> would directly get/put the data where the
> application wanted it.  It would of course
> be much better if this didn't have to be
> done as a hack.

Some versions for the VAX mapped the UNIBUS address space
smack on top on the clists so if you had tty h/w that could
DMA (the DZ-11? or did you need KMC's for that?) you wouldn't
have to map the UNIBUS for each DMA.  This was a grody hack,
but the alternatives were just too slow, particularily if
you wanted to support 128 students on a '780.

Mapping it into user mode is a real can of worms;  not only is
it highly unportable it really messes up the memory management
because pages have to be locked.  Locking is a nightmare;  don't
lock and you lose, have too many locks and horrible bugs appear
either immediately [bad] or months/years later [worse].

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[Douglas A. Gwyn]

Boyd Roberts wrote:
> Some versions for the VAX mapped the UNIBUS address space
> smack on top on the clists so if you had tty h/w that could
> DMA (the DZ-11? or did you need KMC's for that?) you wouldn't
> have to map the UNIBUS for each DMA.  This was a grody hack,
> but the alternatives were just too slow, particularily if
> you wanted to support 128 students on a '780.

My bet would be the KMC11-B; that was a nice hardware hack.

The DZ11 didn't do DMA (but some other multiplexers did);
I think one of the available DZ11 drivers had "pseudo DMA"
to unify it with the other drivers.

So far as I know the only DZ11 driver that had decent
throughput was the one I wrote (originally for 6th Ed.);
as a reminder the DZ11 had a small (32 char?) silo and
interrupted when silo service was requested, BUT after
moving data to/from the silo the "silo service request"
flag was not set again even if it logically should be,
for something like 30us, and typical drivers even though
they were testing the silo before returning from interrupt
were just not waiting long enough, with the effect that
there was a separate interrupt service for *every* char.
My solution was to burn up those 30us before testing,
which might seem wasteful but actually paid off big time.

> Mapping it into user mode is a real can of worms;  not only is
> it highly unportable it really messes up the memory management
> because pages have to be locked.  Locking is a nightmare;  don't
> lock and you lose, have too many locks and horrible bugs appear
> either immediately [bad] or months/years later [worse].

I don't think there are any inherent problems.  We've been
locking disk inodes for eons.

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[Boyd Roberts]

"Douglas A. Gwyn" wrote:
> My bet would be the KMC11-B; that was a nice hardware hack.

Yeah, we had DZ/KMC's.  I think I ran into this mess when we
[john, piers, me] 'ported' 8th Ed from the common 1127 '750
[comet] to our '780.

It was trivial, but there was the odd glitch between the '750
and the '780; some silly compiler issued 16 bit extract field
on the UNIBUS T[EU]-16 CSR -- the 780 did not like that.

> I don't think there are any inherent problems.  We've been
> locking disk inodes for eons.

I know that.  I was referring to locking in general.

IIRC the 'Multiprocessor sleep/wakeup' paper refers to a problem
which was discovered when a new fast device was added to Plan 9.

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[rob pike]

> IIRC the 'Multiprocessor sleep/wakeup' paper refers to a problem
> which was discovered when a new fast device was added to Plan 9.

If I recall correctly, "IIRC" is a shorthand for "if I recall correctly",
and either you don't or I don't.  The problem the paper addressed
was a general one introduced by, er, doing sleep/wakeup on a
multiprocessor.  The paper is also incomplete; when notes were
added to the system shortly after, it took years to get the code
to its current approximation of correctness.  The general approach
of using a tool to help was not abandoned, however; the system
just kept changing under the model.

Nowadays, Holzmann's stuff is applied directly to the kernel
sources, which helps a lot.

-rob

Re: [9fans] RDMA: DMA over TCP/IP at IETF

[Boyd Roberts]

rob pike wrote:
>
> If I recall correctly, "IIRC" is a shorthand for "if I recall correctly",
> and either you don't or I don't.

http://plan9.bell-labs.com/sys/doc/sleep.html:

The mistake is that the reading of r->p may occur just as the other process
calls sleep, so when the interrupt examines the structure it sees no one
to wake up, and the sleeping process misses its wakeup. We wrote the code
this way because we reasoned that the fetch p = r->p was inherently atomic
and need not be interlocked. The bug was found by examination when a
new, very fast device was added to the system and sleeps and interrupts
were closely overlapped. However, it was in the system for a couple of
months without causing an error.

> Nowadays, Holzmann's stuff is applied directly to the kernel
> sources, which helps a lot.

spin [Holzmann's stuff] is a really neat idea.