[9front] devfs performance

[9front] devfs performance

[Martijn Heil]

Hello,

Lately I've been playing around with devfs, doing block interleaving
on two identical USB 3.0 HDD's. I was expecting to have a bit more
throughput compared to using one of those disks, but when I measured
read throughput of the block-interleaved virtual device with tput, the
throughput was exactly the same as measured on a single disk.
The read throughput of a single disk as measured with tput is about 60
MiB/s, so I don't think this would saturate the USB 3.0 SuperSpeed
bus.

Taking a look at /sys/src/9/port/devfs.c, and especially interio() at
/sys/src/9/port/devfs.c:1134,
it seems that the internal read and write operations to the files that
make up the virtual device are done synchronously instead of
asynchronously like I (perhaps naively) expected.
As I understand it, it reads one block from the first file, and only
when that block is done reading it starts reading a block from the
other file, and so on.

I think my interpretation here is correct, but I'm not that familiar
with the kernel source, so is what I'm saying here indeed correct?

It might be worthwhile to optimize this so that it reads
asynchronously from the files that the virtual device is built from.
Is this possible? And how could we achieve this?

Thank you for your time
Martijn Heil
aka
ninjoh

[9front] Re: devfs performance

[Martijn Heil]

In the meantime I've started writing an (experimental) patch for
devfs, that should at least work for my use if I succeed. I'll gladly
share it if I consider it anywhere near worthy, but considering my
inexperience with this kernel development I have no expectations.
Nonetheless I'll try my best.

In case anyone is interested in the technical details;
Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134.
This in turn synchronously calls the blocking io() located at
/sys/src/9/port/devfs:1031 for each sequential block to read from the
inner devices.
My idea so far is to have one kproc worker per inner device which will
communicate with the interio() function by the use of two Qio queue's:
one for sending read/write requests to the worker and the other for
sending results back from the worker to interio(). The workers will
then call io() as requested, which of course, will block the worker.
I'm debating whether I should have those live for the duration of the
life of the fsdev device, or whether I should spin them up on-demand.
I think the best option is to have them live for the duration of the
devfs device, so that the overhead of spinning up several kprocs for
every little read and write can be avoided, in turn for some slight(?)
extra memory usage that the idle kprocs will take up.

Again, I have no experience in this and no real expectations either.
Maybe I'm trying something really stupid.
Please feel free to correct.
It's certainly fun and educational.

I'm really interested in any other suggestions on how to do this properly.

Regards,
Martijn Heil
aka
ninjoh

Re: [9front] Re: devfs performance

[ori]

Quoth Martijn Heil <m.heil375@gmail.com>:
> In the meantime I've started writing an (experimental) patch for
> devfs, that should at least work for my use if I succeed. I'll gladly
> share it if I consider it anywhere near worthy, but considering my
> inexperience with this kernel development I have no expectations.
> Nonetheless I'll try my best.
> 
> In case anyone is interested in the technical details;
> Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134.
> This in turn synchronously calls the blocking io() located at
> /sys/src/9/port/devfs:1031 for each sequential block to read from the
> inner devices.
> My idea so far is to have one kproc worker per inner device which will
> communicate with the interio() function by the use of two Qio queue's:
> one for sending read/write requests to the worker and the other for
> sending results back from the worker to interio(). The workers will
> then call io() as requested, which of course, will block the worker.
> I'm debating whether I should have those live for the duration of the
> life of the fsdev device, or whether I should spin them up on-demand.
> I think the best option is to have them live for the duration of the
> devfs device, so that the overhead of spinning up several kprocs for
> every little read and write can be avoided, in turn for some slight(?)
> extra memory usage that the idle kprocs will take up.
> 
> Again, I have no experience in this and no real expectations either.
> Maybe I'm trying something really stupid.
> Please feel free to correct.
> It's certainly fun and educational.
> 
> I'm really interested in any other suggestions on how to do this properly.
> 
> Regards,
> Martijn Heil
> aka
> ninjoh
> 

Yes. this sounds pretty plausible -- though
I suspect we can get a bigger speedup when
doing writes, since we can enqueue the write,
and return immediately (if we are willing to
delay error reporting to the next interaction
with the device.

For reads, we may need to do work elsewhere
in the system to handle larger, or at least
more, reads, though doing them in parallel
should get a nice speedup (eg, using fcp or
very parallel builds)

Re: [9front] Re: devfs performance

[Roberto E. Vargas Caballero]

Hi,

On Mon, Jan 17, 2022 at 04:27:51PM +0100, Martijn Heil wrote:
> In the meantime I've started writing an (experimental) patch for
> devfs, that should at least work for my use if I succeed. I'll gladly
> share it if I consider it anywhere near worthy, but considering my
> inexperience with this kernel development I have no expectations.
> Nonetheless I'll try my best.
> 
> In case anyone is interested in the technical details;
> Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134.
> This in turn synchronously calls the blocking io() located at
> /sys/src/9/port/devfs:1031 for each sequential block to read from the
> inner devices.

I have interleaving in my disk configuration, and I thought that
the access was done in parallel. I  think we should take a deep look
to your patch because otherwise the interleaving configuration
does not make sense.

Regards,