Hello, Lately I've been playing around with devfs, doing block interleaving on two identical USB 3.0 HDD's. I was expecting to have a bit more throughput compared to using one of those disks, but when I measured read throughput of the block-interleaved virtual device with tput, the throughput was exactly the same as measured on a single disk. The read throughput of a single disk as measured with tput is about 60 MiB/s, so I don't think this would saturate the USB 3.0 SuperSpeed bus. Taking a look at /sys/src/9/port/devfs.c, and especially interio() at /sys/src/9/port/devfs.c:1134, it seems that the internal read and write operations to the files that make up the virtual device are done synchronously instead of asynchronously like I (perhaps naively) expected. As I understand it, it reads one block from the first file, and only when that block is done reading it starts reading a block from the other file, and so on. I think my interpretation here is correct, but I'm not that familiar with the kernel source, so is what I'm saying here indeed correct? It might be worthwhile to optimize this so that it reads asynchronously from the files that the virtual device is built from. Is this possible? And how could we achieve this? Thank you for your time Martijn Heil aka ninjoh
In the meantime I've started writing an (experimental) patch for devfs, that should at least work for my use if I succeed. I'll gladly share it if I consider it anywhere near worthy, but considering my inexperience with this kernel development I have no expectations. Nonetheless I'll try my best. In case anyone is interested in the technical details; Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134. This in turn synchronously calls the blocking io() located at /sys/src/9/port/devfs:1031 for each sequential block to read from the inner devices. My idea so far is to have one kproc worker per inner device which will communicate with the interio() function by the use of two Qio queue's: one for sending read/write requests to the worker and the other for sending results back from the worker to interio(). The workers will then call io() as requested, which of course, will block the worker. I'm debating whether I should have those live for the duration of the life of the fsdev device, or whether I should spin them up on-demand. I think the best option is to have them live for the duration of the devfs device, so that the overhead of spinning up several kprocs for every little read and write can be avoided, in turn for some slight(?) extra memory usage that the idle kprocs will take up. Again, I have no experience in this and no real expectations either. Maybe I'm trying something really stupid. Please feel free to correct. It's certainly fun and educational. I'm really interested in any other suggestions on how to do this properly. Regards, Martijn Heil aka ninjoh
Quoth Martijn Heil <m.heil375@gmail.com>:
> In the meantime I've started writing an (experimental) patch for
> devfs, that should at least work for my use if I succeed. I'll gladly
> share it if I consider it anywhere near worthy, but considering my
> inexperience with this kernel development I have no expectations.
> Nonetheless I'll try my best.
>
> In case anyone is interested in the technical details;
> Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134.
> This in turn synchronously calls the blocking io() located at
> /sys/src/9/port/devfs:1031 for each sequential block to read from the
> inner devices.
> My idea so far is to have one kproc worker per inner device which will
> communicate with the interio() function by the use of two Qio queue's:
> one for sending read/write requests to the worker and the other for
> sending results back from the worker to interio(). The workers will
> then call io() as requested, which of course, will block the worker.
> I'm debating whether I should have those live for the duration of the
> life of the fsdev device, or whether I should spin them up on-demand.
> I think the best option is to have them live for the duration of the
> devfs device, so that the overhead of spinning up several kprocs for
> every little read and write can be avoided, in turn for some slight(?)
> extra memory usage that the idle kprocs will take up.
>
> Again, I have no experience in this and no real expectations either.
> Maybe I'm trying something really stupid.
> Please feel free to correct.
> It's certainly fun and educational.
>
> I'm really interested in any other suggestions on how to do this properly.
>
> Regards,
> Martijn Heil
> aka
> ninjoh
>
Yes. this sounds pretty plausible -- though
I suspect we can get a bigger speedup when
doing writes, since we can enqueue the write,
and return immediately (if we are willing to
delay error reporting to the next interaction
with the device.
For reads, we may need to do work elsewhere
in the system to handle larger, or at least
more, reads, though doing them in parallel
should get a nice speedup (eg, using fcp or
very parallel builds)
Hi,
On Mon, Jan 17, 2022 at 04:27:51PM +0100, Martijn Heil wrote:
> In the meantime I've started writing an (experimental) patch for
> devfs, that should at least work for my use if I succeed. I'll gladly
> share it if I consider it anywhere near worthy, but considering my
> inexperience with this kernel development I have no expectations.
> Nonetheless I'll try my best.
>
> In case anyone is interested in the technical details;
> Block-interleaved I/O is handled by interio() at /sys/src/9/port/devfs:1134.
> This in turn synchronously calls the blocking io() located at
> /sys/src/9/port/devfs:1031 for each sequential block to read from the
> inner devices.
I have interleaving in my disk configuration, and I thought that
the access was done in parallel. I think we should take a deep look
to your patch because otherwise the interleaving configuration
does not make sense.
Regards,