[9fans] 9p pipelining experiment

[9fans] 9p pipelining experiment

[cinap_lenrek]

i would like to share a little experiment:

http://felloff.net/usr/cinap_lenrek/stream/

the idea is todo pipelining over 9p only with small
changes to devmnt. that is, instead of:

T1 *stall* R1 T2 *stall* R2 T3 *stall* R3 ...

we want todo:

T1 T2 R1 T3 R2 R3

so i made devstream thats like devdup (/fd) so
when you open /fd/0stream, you get a little stream back
that will do i/o ontop of the original fd channel in
parallel. you cannot seek in a stream, but it will
read or write stuff in this pipelined manner in the
background.

this has similar semantics like the "ftp like steaming 9p"
stuff but doesnt need protocol changes or an out of band
channel.

there are two kinds of files that are important to
distinguish when we do read pipelining:

- files honoring the offset
- files ignoring the offset

in the first case, we have to reorder the responses by file offset
and and need to keep track of holes when a read returns less data
than requested.

in the second case, we need to reorder them in response order.

in the lack of having qid bits for this, i came up with the following
hack to detect the mode:

read from offset 0x7fffffffffffff first. when we get bytes back, we
can assume the file doesnt honor the offset, and otherwise it does.
this is just a hack for now to play with this idea. it would be
better to know beforehand with some mode bits but this appears to
work fine.

for pipelining, we need to know when a request is commited (written
to the 9p channel and on its way to the server), and for reads on
files not honoring the offset, we need to know the order in which
the responses arrived.

for this, the following fields where added to the Proc structure:

+	void	*iocomarg;	/* I/O completion callback for pipelining */
+	void	(*iocomfun)(void*, int);

devmnt just calls (*up->iocomfun)(up->iocomarg, 0); when a 9p
request was commited and (*up->iocomfun)(up->iocomarg, 1) when
it received a response (up is the original processes in this
case, but doesnt call it like that obviously as mountmux() can
run on a different proc).

this is enougth to implement the pipelining. streamwrite() is
implemented by first acquiering a qlock, then we take a buffer
from a queue, then write it. when the file is devmnt, iocom is
called from within devmnt and will release the qlock when the
write is commited. which allows another writer process to take
another buffer from the queue and write it...

note we do not rely on iocom to be called within devmnt, in
case the file doent give us any hints, we just call it ourselfs
after the read or write returns.

things that are not worked out yet completely:

- zero length reads currently terminate the stream
- only way to drain/flush a write stream is to close it (want
some barrier (zero length write??))
- how many procs should we use? currently using 4 readers and 2 writers
- bugs: most likely

any questions?

--
cinap

Re: [9fans] 9p pipelining experiment

[cinap_lenrek]

ah, and it works:

sequential reading remote /dev/zero to /dev/null: 17MB/s
pipelined reading remote /dev/zero to /dev/null: 58MB/s
sequential writing local /dev/zero to remote /dev/null: 13MB/s
pipelined writing local /dev/zero to remove /dev/null: 44MB/s

this is with 4 readers and 2 writers which might not be
optimal, but it shows that pipelining can speed stuff up
at least.

--
cinap