[9fans] Load spikes on intervals in qemu

[9fans] Load spikes on intervals in qemu

[Brad Frank]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

Hi, I recently did a clean install of plan9 on qemu on linux. I've noticed
that the load is spiking on an interval every 30 seconds or something like
that. I looked at suggestions that it might be venti and timesync. But it
couldn't be venti because I didn't install venti, I have a fossil only
install. I looked at timesync, and killed it, and the load was still
spiking. It was suggested I try zwansch's gtop, which I did do, and I found
that fossil was using the most utime/stime. But it seemed like two different
threads? Fossil [disk] and Fossil [flush]. When I killed fossil [disk] my
load dropped, but obviously I lost disk access. What could possibly be
wrong, and why is it doing this? Another interesting thing, is that when the
load spikes like that, the emulator seems to temporarily lag or not respond,
until the load drops again, so this also effects the performance of plan9.
Any suggestions would be quite helpful.

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

Re: [9fans] Load spikes on intervals in qemu

[Venkatesh Srinivas]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

On Tue, Aug 17, 2010 at 9:50 AM, Brad Frank <brad.frank@gmail.com> wrote:

> Hi, I recently did a clean install of plan9 on qemu on linux. I've noticed
> that the load is spiking on an interval every 30 seconds or something like
> that. I looked at suggestions that it might be venti and timesync. But it
> couldn't be venti because I didn't install venti, I have a fossil only
> install. I looked at timesync, and killed it, and the load was still
> spiking. It was suggested I try zwansch's gtop, which I did do, and I found
> that fossil was using the most utime/stime. But it seemed like two different
> threads? Fossil [disk] and Fossil [flush]. When I killed fossil [disk] my
> load dropped, but obviously I lost disk access. What could possibly be
> wrong, and why is it doing this? Another interesting thing, is that when the
> load spikes like that, the emulator seems to temporarily lag or not respond,
> until the load drops again, so this also effects the performance of plan9.
> Any suggestions would be quite helpful.
>

I run Plan 9 in qemu, but I run neither fossil nor any other (major) disk
file server in qemu.
Instead, I have Inferno on my host serve files to Plan 9.

To accomplish this:
1) I installed Plan 9, as normal, into a qemu disk image, with fossil as my
fileserver
2) I tarred up my root and copied it to my host
3) I patched the 9 boot program to accept a file server on port 6666 rather
than 564 (styx instead of 9fs)
4) I untarred the root fs and had Inferno: 'styxlisten -A tcp!*!6666 export
/plan9'
5) Start the both (qemu and inferno) together.
6) Profit!

-- vs

[-- Attachment #2: Type: text/html, Size: 1972 bytes --]

Re: [9fans] Load spikes on intervals in qemu

[Eric Van Hensbergen]

On Tue, Aug 17, 2010 at 9:07 AM, Venkatesh Srinivas <me@acm.jhu.edu> wrote:
>
> I run Plan 9 in qemu, but I run neither fossil nor any other (major) disk
> file server in qemu.
> Instead, I have Inferno on my host serve files to Plan 9.
> To accomplish this:
> 1) I installed Plan 9, as normal, into a qemu disk image, with fossil as my
> fileserver
> 2) I tarred up my root and copied it to my host
> 3) I patched the 9 boot program to accept a file server on port 6666 rather
> than 564 (styx instead of 9fs)
> 4) I untarred the root fs and had Inferno: 'styxlisten -A tcp!*!6666 export
> /plan9'
> 5) Start the both (qemu and inferno) together.
> 6) Profit!
>

Its worth noting that you can do the same thing with the native 9p
servers for linux (u9fs, spfs, or npfs).  Alternatively you could run
a AOE vblade server, or a p9p venti (although you'd still need a
solution for the fossil disk, but a ramdisk might be the best solution
here for speed).  It might be a nice project to bundle such an
environment with scripts for folks to use as an easy qemu/kvm start
environment with either Linux based file server or the
venti/ramdisk-fossil solution.

         -eric

Re: [9fans] Load spikes on intervals in qemu

[erik quanstrom]

> Its worth noting that you can do the same thing with the native 9p
> servers for linux (u9fs, spfs, or npfs).  Alternatively you could run
> a AOE vblade server, or a p9p venti (although you'd still need a
> solution for the fossil disk, but a ramdisk might be the best solution
> here for speed).  It might be a nice project to bundle such an
> environment with scripts for folks to use as an easy qemu/kvm start
> environment with either Linux based file server or the
> venti/ramdisk-fossil solution.

why cant fossil also use vblade?

- erik

Re: [9fans] Load spikes on intervals in qemu

[Eric Van Hensbergen]

On Wed, Aug 18, 2010 at 7:42 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>
> why cant fossil also use vblade?
>

I thought I covered that by saying you could run an AOE vblade server
(by which I meant to imply you could run any of the available file
systems over AOE or venti), but perhaps I should have been more
specific.

     -eric

[9fans] Load spikes on intervals in qemu

[Brad Frank]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

Yes, doing those things would be an alternative. But the bigger question is
why is fossil hitting the load like that while running in Qemu. And also,
another question is whether this would be happening on physical hardware as
well... Is there anything I can do to figure out why it is doing that, and
how to resolve it?

[-- Attachment #2: Type: text/html, Size: 321 bytes --]

Re: [9fans] Load spikes on intervals in qemu

[Eric Van Hensbergen]

qemu disk emulation isn't exactly speedy, and fossil probably bangs on
the disk pretty hard and from the sounds of things its treating the
disk access as synchronous (which is why everything else freezes up).
The two combined together is not something you want to deal with.
FWIW, on my system I don't see this problem, but its a 4 processor
3GHZ+ Xeon with 16GB of memory which is probably always hitting the
disk access in Linux's cache.

       -eric


On Tue, Aug 17, 2010 at 11:55 AM, Brad Frank <brad.frank@gmail.com> wrote:
> Yes, doing those things would be an alternative. But the bigger question is
> why is fossil hitting the load like that while running in Qemu. And also,
> another question is whether this would be happening on physical hardware as
> well... Is there anything I can do to figure out why it is doing that, and
> how to resolve it?

Re: [9fans] Load spikes on intervals in qemu

[erik quanstrom]

On Tue Aug 17 14:32:01 EDT 2010, ericvh@gmail.com wrote:
> qemu disk emulation isn't exactly speedy, and fossil probably bangs on
> the disk pretty hard and from the sounds of things its treating the
> disk access as synchronous (which is why everything else freezes up).
> The two combined together is not something you want to deal with.
> FWIW, on my system I don't see this problem, but its a 4 processor
> 3GHZ+ Xeon with 16GB of memory which is probably always hitting the
> disk access in Linux's cache.
>
>        -eric
>
>
> On Tue, Aug 17, 2010 at 11:55 AM, Brad Frank <brad.frank@gmail.com> wrote:
> > Yes, doing those things would be an alternative. But the bigger question is
> > why is fossil hitting the load like that while running in Qemu. And also,
> > another question is whether this would be happening on physical hardware as
> > well... Is there anything I can do to figure out why it is doing that, and
> > how to resolve it?

has anyone tried aoe with qemu?  vblade running in the host
might be faster.  devaoe(3)/sdaoe(3) from 9atom allow
128 outstanding requests), which should help with latency
a bit more.

- erik

Re: [9fans] Load spikes on intervals in qemu

[Eric Van Hensbergen]

On Wed, Aug 18, 2010 at 7:37 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> On Tue, Aug 17, 2010 at 11:55 AM, Brad Frank <brad.frank@gmail.com> wrote:
>> > Yes, doing those things would be an alternative. But the bigger question is
>> > why is fossil hitting the load like that while running in Qemu. And also,
>> > another question is whether this would be happening on physical hardware as
>> > well... Is there anything I can do to figure out why it is doing that, and
>> > how to resolve it?
>
> has anyone tried aoe with qemu?  vblade running in the host
> might be faster.  devaoe(3)/sdaoe(3) from 9atom allow
> 128 outstanding requests), which should help with latency
> a bit more.
>

I haven't tried it, but I agree that it is likely faster.  An even
better solution would be use of the virtio disk or 9p drivers, but I'm
not sure they are available in vanilla qemu.  I don't recall seeing
these delays with qemu-kvm on my laptop when I was running off a
virtio disk -- same thing goes for lguest for those not running on a
KVM-able processor.

        -eric

Re: [9fans] Load spikes on intervals in qemu

[Venkatesh Srinivas]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

virtio is available in vanilla qemu nowadays.

Why do you want to run your file server in qemu anyway, though?

For snapshots, I use dump9660 on my host, coupled with inferno; u9fs would
work just as well.

-- vs

[-- Attachment #2: Type: text/html, Size: 241 bytes --]

Re: [9fans] Load spikes on intervals in qemu

[erik quanstrom]

> virtio is available in vanilla qemu nowadays.
>
> Why do you want to run your file server in qemu anyway, though?
>
> For snapshots, I use dump9660 on my host, coupled with inferno; u9fs would
> work just as well.

that depends on what your goals are.  if your goal is
to have a plan 9 environment and tight integration with
unix or inferno is not important, your solution sounds like a lot of
moving parts relative to a plan 9 file server in qemu,
imho.

- erik

[9fans] Load spikes on intervals in qemu

[Brad Frank]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

Well, it is also important to note that I am getting these spikes on
qemu while the emulator is essentially idling.
There really should be no reason it should be hitting the virtual disk
like that repeatedly on intervals. Is there no
other explanation for why it would be doing that?

[-- Attachment #2: Type: text/html, Size: 309 bytes --]