From mboxrd@z Thu Jan 1 00:00:00 1970 MIME-Version: 1.0 In-Reply-To: <389ed08237f11e30c2f310f5847e34c1@brasstown.quanstro.net> References: <9d4d16071350d120a79044a6c0c1604f@felloff.net> <0A651ACE-ADD2-4C9F-9491-0802B20923B9@gmail.com> <389ed08237f11e30c2f310f5847e34c1@brasstown.quanstro.net> Date: Thu, 19 Dec 2013 16:57:01 +0100 Message-ID: From: Gorka Guardiola To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Content-Type: multipart/alternative; boundary=001a11c1de8e5c6e2b04ede5363d Subject: Re: [9fans] 9front pegs CPU on VMware Topicbox-Message-UUID: a0efbbfa-ead8-11e9-9d60-3106f5b1d025 --001a11c1de8e5c6e2b04ede5363d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Thu, Dec 19, 2013 at 4:19 PM, erik quanstrom wrot= e: > for those without much mwait experience, mwait is a kernel-only primitive > (as per the instructions) that pauses the processor until a change has be= en > made in some range of memory. the size is determined by probing the h/w, > but think cacheline. so the discussion of locking is kernel specific as > well. > > The original discussion started about the runq spin lock, but I think the scope of the problem is more general and the solution can be applied in user and kernel space both. While in user space you would do sleep(0) in the kernel you would sched() or if you are in the scheduler you would loop doing mwait (see my last email). The manual I have available says: "The MWAIT instruction can be executed at any privilege level. The MONITOR CPUID feature flag (ECX[bit 3] when CPUID is executed with EAX =3D 1) indicates the availability of the MONITOR and MWAIT instruction in a processor. When set, the unconditional execution of MWAIT is supported at privilege level 0 and conditional execution is supported at privilege levels 1 through 3 (software should test for the appropriate support of these instructions before unconditional use)." There are also other extensions, which I have not tried. I think the ideas can be used in the kernel or in user space, though I have only tried it the kernel and the implementation is only in the kernel right now. > > > On 17 Dec 2013, at 12:00, cinap_lenrek@felloff.net wrote: > > > > > > thats a surprising result. by dog pile lock you mean the runq spinloc= k > no? > > > > > > > I guess it depends on the HW, but I don=B4t find that so surprising. Yo= u > are looping > > sending messages to the coherency fabric, which gets congested as a > result. > > I have seen that happen. > > i assume you mean that there is contention on the cacheline holding the > runq lock? > i don't think there's classical congestion. as i believe cachelines not > involved in the > mwait would experience no hold up. > I mean congestion in the classical network sense. There are switches and links to exchange messages for the coherency protocol and some them get congested. What I was seeing is the counter of messages growing very very fast and the performance degrading which I interpret as something getting congested. I think when the lock possession is pingponged around (not necessarily contented, but many changes in who is holding the lock or maybe contention) many messages are generated and then the problem occurs. I certainly saw the HW counters for messages go up orders of magnitude when I was not using mwait. > > mwait() does improve things and one would expect the latency to always be > better > than spining*. but as it turns out the current scheduler is pretty > hopeless in its locking > anyway. simply grabbing the lock with lock rather than canlock makes mor= e > sense to me. > These kind of things are subtle, I spent a lot of time measuring and it is difficult to know what is happening always for sure and some of the results are counterintuitive (at least to me) and depend on the concrete hardware/benchmark/test. So take my conclusions with a pinch of salt :-). I think the latency of mwait (this is what I remember for the opterons I was measuring, probably different in intel and in other amd models) is actually worse (bigger) than with spinning, but if you have enough processors doing the spinning (not necessarily on the same locks, but generating traffic) then at some point it reverses because of the traffic in the coherency fabric (or thermal effects, I do remember that without the mwait all the fans would be up and with the mwait they would turn off and the machine would be noticeably cooler). Measure it in your hardware anyway which will a) probably be different with a better monitor/mway. b) you can be sure it works for the loads you are interested in. > also, using ticket locks (see 9atom nix kernel) will provide automatic > backoff within the lock. > ticket locks are a poor solution as they're not really scalable but they > will scale to 24 cpus > much better than tas locks. > > mcs locks or some other queueing-style lock is clearly the long-term > solution. but as > charles points out one would really perfer to figure out a way to fit the= m > to the lock > api. i have some test code, but testing queueing locks in user space is > ... interesting. > i need a new approach. > > Let us know what your conclusions are after you implement and measure them :-). G. --001a11c1de8e5c6e2b04ede5363d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Thu, Dec 19, 2013 at 4:19 PM, erik quanstrom &= lt;quanstro@quan= stro.net> wrote:
for those without much mwait experience, mwait is a kernel= -only primitive
(as per the instructions) that pauses the processor until a change has been=
made in some range of memory. =A0the size is determined by probing the h/w,=
but think cacheline. =A0so the discussion of locking is kernel specific as = well.

=A0
The original discussion started about the runq spin lock, but I thin= k the scope of the
problem is more general = and the solution can be applied in user and kernel space both.
While in user space you would do sleep(0) in the= kernel you would sched() or if you
are in = the scheduler you would loop doing mwait (see my last email).

The manual I have available says:

"The MWAIT instruction can be executed at any privilege level. Th= e MONITOR CPUID feature flag (ECX[bit 3] when CPUID is executed with EAX = =3D 1) indicates the availability of the MONITOR and MWAIT instruction in a= processor. When set, the unconditional execution of MWAIT is supported at = privilege level 0 and conditional execution is supported at privilege level= s 1 through 3 (software should test for the appropriate support of these in= structions before unconditional use)."

There are a= lso other extensions, which I have not tried.
I think the ideas can be used in the kernel or in user space, though I h= ave only tried
it the kernel and the implementation is only in = the kernel right now.
=A0
> > On 17 Dec 2013, at 12:00, cinap_lenrek@felloff.net wrote:
> >
> > thats a surprising result. by dog pile lock you mean the runq spi= nlock no?
> >
>
> I guess it depends on the HW, but I don=B4t find that so surprising. Y= ou are looping
> sending messages to the coherency fabric, which gets congested as a re= sult.
> I have seen that happen.

i assume you mean that there is contention on the cacheline holding t= he runq lock?
i don't think there's classical congestion. =A0as i believe cacheli= nes not involved in the
mwait would experience no hold up.

I me= an congestion in the classical network sense. There are switches and links = to
exchange messages for the coherency protocol and some them get= congested.
What I was seeing is the counter of messages growing very very fast an= d the performance
degrading which I interpret as something gettin= g congested.
I think when the lock possession is pingponged aroun= d (not necessarily contented,
but many changes in who is holding the lock or maybe contention) many = messages are
generated and then the problem occurs. I certainly s= aw the HW counters for messages go up
orders of magnitude when I = was not using mwait.

mwait() does improve things and one would expect the latency to alway= s be better
than spining*. =A0but as it turns out the current scheduler is pretty hopel= ess in its locking
anyway. =A0simply grabbing the lock with lock rather than canlock makes mor= e sense to me.

These kind of things are= subtle, I spent a lot of time measuring and it is difficult to know
what is happening always for sure and some of the results are counteri= ntuitive (at least to me)
and depend on the concrete hardware/ben= chmark/test. So take my conclusions with a pinch of
salt :-).

I think the latency of mwait (this is what I remember
for the opterons I was measuring, probably different in intel and = in other amd models)
is actually worse (bigger) than with spinnin= g,
but if you have enough processors doing the spinning (not necessarily = on the same locks, but
generating traffic) then at some point it = reverses because
of the traffic in the coherency fabric (or therm= al effects, I do remember that
without the mwait all the fans would be up and with the mwait they wou= ld turn off and the
machine would be noticeably cooler). Measure = it in your hardware anyway which will
a) probably be different wi= th a better monitor/mway.
b) you can be sure it works for the loads you are interested in.
=


also, using ticket locks (see 9atom nix kernel) will provide automatic back= off within the lock.
ticket locks are a poor solution as they're not really scalable but the= y will scale to 24 cpus
much better than tas locks.

mcs locks or some other queueing-style lock is clearly the long-term soluti= on. =A0but as
charles points out one would really perfer to figure out a way to fit them = to the lock
api. =A0i have some test code, but testing queueing locks in user space is = ... interesting.
i need a new approach.


Let us know = what your conclusions are after you implement and measure them :-).

G.

=A0
--001a11c1de8e5c6e2b04ede5363d--