From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx0.topicbox.com (localhost.local [127.0.0.1]) by tb-mx0.topicbox.com (Postfix) with ESMTP id 9FBD420A6CDE for ; Sun, 28 Jul 2024 13:50:45 -0400 (EDT) (envelope-from rm@fingolfin.org) Received: from tb-mx0.topicbox.com (localhost [127.0.0.1]) by tb-mx0.topicbox.com (Authentication Milter) with ESMTP id 86D266C102F; Sun, 28 Jul 2024 13:50:45 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1722189045; b=tmYQMlnRB7AMlXVB5fUxQCi88KnYDvp73uLaQdLS1qm+ApSVZX pGtxSZAPIGiJi8HWMQ/lxJrpPDEBrkFputUWsPUujB7ha1D34VhOFyeNwqu6DoJi R659UzyRhoN+ta7TwYQttIVqKq+ffnMfD0zcmfDaAeDiZ2P1PlUpgnve4c3I1oCm JD20PQSfK0YyP0TmKt7zjyQn9tkPlzAnYDfSS1olA6BIs4ZD5wBIgUzWtsEJ5xQN FUQPOEybspxtiZ9w8qY8Twf3RuGsVCPg2G6IFI+UG8TJKbHmFzm1vSgg0EyjHbJn m5kDeimqT/59RGO7ItXP8PgWKU/XbzQxVbbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=message-id:date:mime-version:subject:from:to :reply-to:references:in-reply-to:content-type :content-transfer-encoding; s=arcseal; t=1722189045; bh=ZvNEd6I4 BokuelQZzuXAc2+c9w6l/nHVYVw/a2Lf7rw=; b=teRdkJ8HwxrgZwFaqTibUK6n KgYSdJcxmZrbC+TSLZFnJd2Hd9XhJw1ZMe+cztG9nTfJAuyTOEvw4LW7v5zbIgAi 1ZtJYSHmqsPEgju/0sRtoL7HeVfDkVzgHTtjwExzOwAVs13hInyNF6eCE/iBR69q PTQ+PUIAWUTlZFOjNH7lzNKH9HCmzbsPl40IvVhbOEvMV0vVeC7sO43P/pwDZ6FX +FuJcEbX6X5ncoCaRkszNj0Icf3xnSi4ghBI9g0/nnXPTZ3pJZLWKWFTRjtlC5Hr iTLuXTKQQJ3peosmWKnirlwen2PnXqhcLN8ERCGTdED3+MHj5/7Dk/rK0+WORQ== ARC-Authentication-Results: i=1; tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=fingolfin.org header.i=@fingolfin.org header.b=sQeDh05L header.a=rsa-sha256 header.s=fm1 x-bits=2048; dkim=pass (2048-bit rsa key sha256) header.d=messagingengine.com header.i=@messagingengine.com header.b=mY1HiyCx header.a=rsa-sha256 header.s=fm3 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=fingolfin.org; iprev=pass smtp.remote-ip=103.168.172.153 (fhigh2-smtp.messagingengine.com); spf=pass smtp.mailfrom=rm@fingolfin.org smtp.helo=fhigh2-smtp.messagingengine.com; x-aligned-from=pass (Address match); x-me-sender=pass policy.xms= 9YSmZp8j8eWweRUC6CuXCuRPFAwMtwKtNu0xsmch9hzdKrel6k89CfZq4BJTC9N-YUAqhIaShgePpkkE4XimsCh43tbtZ3Q0XDTsWnqtz0YGVaUiC8yfKH9PF5A7vf-LesP8jTFeBw; x-ptr=pass smtp.helo=fhigh2-smtp.messagingengine.com policy.ptr=fhigh2-smtp.messagingengine.com; x-return-mx=pass header.domain=fingolfin.org policy.is_org=yes (MX Records found: in1-smtp.messagingengine.com,in2-smtp.messagingengine.com); x-return-mx=pass smtp.domain=fingolfin.org policy.is_org=yes (MX Records found: in1-smtp.messagingengine.com,in2-smtp.messagingengine.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 Authentication-Results: tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=fingolfin.org header.i=@fingolfin.org header.b=sQeDh05L header.a=rsa-sha256 header.s=fm1 x-bits=2048; dkim=pass (2048-bit rsa key sha256) header.d=messagingengine.com header.i=@messagingengine.com header.b=mY1HiyCx header.a=rsa-sha256 header.s=fm3 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=fingolfin.org; iprev=pass smtp.remote-ip=103.168.172.153 (fhigh2-smtp.messagingengine.com); spf=pass smtp.mailfrom=rm@fingolfin.org smtp.helo=fhigh2-smtp.messagingengine.com; x-aligned-from=pass (Address match); x-me-sender=pass policy.xms= 9YSmZp8j8eWweRUC6CuXCuRPFAwMtwKtNu0xsmch9hzdKrel6k89CfZq4BJTC9N-YUAqhIaShgePpkkE4XimsCh43tbtZ3Q0XDTsWnqtz0YGVaUiC8yfKH9PF5A7vf-LesP8jTFeBw; x-ptr=pass smtp.helo=fhigh2-smtp.messagingengine.com policy.ptr=fhigh2-smtp.messagingengine.com; x-return-mx=pass header.domain=fingolfin.org policy.is_org=yes (MX Records found: in1-smtp.messagingengine.com,in2-smtp.messagingengine.com); x-return-mx=pass smtp.domain=fingolfin.org policy.is_org=yes (MX Records found: in1-smtp.messagingengine.com,in2-smtp.messagingengine.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrjedtgdduudelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhepkfffgggfuffhvfhrfhgjtgfgsehtjeertddtvdej necuhfhrohhmpeftohgsvghrthcuofhushhtrggttghhihcuoehrmhesfhhinhhgohhlfh hinhdrohhrgheqnecuggftrfgrthhtvghrnhepjeeuvdevuefgvdethfeifedvveekiefh geetjedtheelffevieeluedujeefgffgnecuffhomhgrihhnpehilhhluhhmohhsrdhorh hgnecukfhppedutdefrdduieekrddujedvrdduheefnecuvehluhhsthgvrhfuihiivgep tdenucfrrghrrghmpehinhgvthepuddtfedrudeikedrudejvddrudehfedphhgvlhhope hfhhhighhhvddqshhmthhprdhmvghsshgrghhinhhgvghnghhinhgvrdgtohhmpdhmrghi lhhfrhhomhepoehrmhesfhhinhhgohhlfhhinhdrohhrgheqpdhnsggprhgtphhtthhope dupdhrtghpthhtohepoeguvghvvghlohhpvghrsehlihhsthhsrdhilhhluhhmohhsrdho rhhgqe X-ME-VSScore: -100 X-ME-VSCategory: clean Received-SPF: pass (fingolfin.org: Sender is authorized to use 'rm@fingolfin.org' in 'mfrom' identity (mechanism 'include:spf.messagingengine.com' matched)) receiver=tb-mx0.topicbox.com; identity=mailfrom; envelope-from="rm@fingolfin.org"; helo=fhigh2-smtp.messagingengine.com; client-ip=103.168.172.153 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx0.topicbox.com (Postfix) with ESMTPS for ; Sun, 28 Jul 2024 13:50:45 -0400 (EDT) (envelope-from rm@fingolfin.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id C5AFD114011B; Sun, 28 Jul 2024 13:50:44 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Sun, 28 Jul 2024 13:50:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fingolfin.org; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1722189044; x=1722275444; bh=ZvNEd6I4BokuelQZzuXAc2+c9w6l/nHVYVw /a2Lf7rw=; b=sQeDh05LAYt1XwemlrW4XMvP5z8jYKgxGqbY4TkFrg5/MxxevVs 8BKi4j+fitC6c8HicvdmTQLWhDR2pevNGgVWNmKNWVk0GpZMxC4PzVP2x5IdlhRu HU9QyW1Kmku9b3HShgkSYzPgfleMDH0o+NRa9V1Ipl+WR2fKeDXzzqNbfjPxj0wQ s9BMCHkO7cSRuEOIfOzyE95d5YPelx2MKFdMrpof55ZlLtb5Y8NyHOQZ610L+26f 9mhf1UEBOii56uCC4bepap4V6YbPQOvv9Pt6eNwfJJaxo5yXOQFrLM2kxPOFxDTz VSyBnPLjgfFkqSK4a89Ni0hHF+G3xTLV2IA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1722189044; x= 1722275444; bh=ZvNEd6I4BokuelQZzuXAc2+c9w6l/nHVYVw/a2Lf7rw=; b=m Y1HiyCxpUBnoUDrccrTZLB1NS7GNfpXn1UYpSX5g3zYIP7e8ESrUgpmnQ5bpA0xb HcrH3uQaKH9lDZhZ2jB76bNzy1xju3cY0VbKursImM3iOeyA++HftciJEFXbgcAZ ZPc3xVkVVaGkRAZ2pj9Oe7IRDu9pKRrrGx6wCtN6rwKt17RhfksDJcDIVi5jAU1P xEwfjHv8X+Zve0GyBhcmPCGXW+ejtjYWJ0cbRAu7nqdW9nXjh0ft78LGIhWj3CvS HmtiaE+gEvmODwQfQZ8cqcvFsWKduCZx/zzw7wySRKwKJBuStlB11nZgpxA22gl+ a1u6NoS4vmmpucoo3KI4Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrjedtgdduudelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffhvfhrfhgjtgfgsehtjeertddtvdejnecuhfhrohhmpeftohgs vghrthcuofhushhtrggttghhihcuoehrmhesfhhinhhgohhlfhhinhdrohhrgheqnecugg ftrfgrthhtvghrnhepjeeuvdevuefgvdethfeifedvveekiefhgeetjedtheelffevieel uedujeefgffgnecuffhomhgrihhnpehilhhluhhmohhsrdhorhhgnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheprhhmsehfihhnghholhhfihhn rdhorhhgpdhnsggprhgtphhtthhopedt X-ME-Proxy: Feedback-ID: i4ca146eb:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 28 Jul 2024 13:50:43 -0400 (EDT) Message-ID: Date: Sun, 28 Jul 2024 10:50:40 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [developer] CPU usage - clock related issue From: Robert Mustacchi To: illumos-developer , Marcel Telka Reply-To: illumos-developer References: <4d9797a5-b68c-4c50-9a13-66a9d0d63bd0@fingolfin.org> Content-Language: en-US In-Reply-To: <4d9797a5-b68c-4c50-9a13-66a9d0d63bd0@fingolfin.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Topicbox-Policy-Reasoning: allow: sender is an admin Topicbox-Message-UUID: ed223a14-4d09-11ef-9b3a-1c3a098c7b06 On 7/28/24 09:37, Robert Mustacchi wrote: > On 7/27/24 22:56, Marcel Telka wrote: >> Hi, >> >> It looks like something went wrong between changesets 6e0c6e37fb and >> 8b913f79fc in the illumos-gate. >> >> After upgrade of OpenIndiana from >> osnet-incorporation@0.5.11-2024.0.0.22264 (illumos-6e0c6e37fb) >> to >> osnet-incorporation@0.5.11-2024.0.0.22271 (illumos-8b913f79fc) >> >> I see two processes eating full CPU: nwamd and mariadbd. The machine is >> a qemu/kvm quest (host is Rocky 9). >> >> # dtrace -n 'profile-101 /pid == $target/ { @[ustack()] = count() } tick-10s{exit(0)}' -p $(pgrep -x nwamd) | tail -n 40 >> dtrace: description 'profile-101 ' matched 2 probes >> nwamd`in_past+0x23 >> nwamd`nwamd_event_dequeue+0x1bf >> nwamd`nwamd_event_handler+0x162 >> nwamd`main+0x1b0 >> nwamd`_start_crt+0x9a >> nwamd`_start+0x1a >> 24 >> >> nwamd`in_past+0x26 >> nwamd`nwamd_event_dequeue+0x1bf >> nwamd`nwamd_event_handler+0x162 >> nwamd`main+0x1b0 >> nwamd`_start_crt+0x9a >> nwamd`_start+0x1a >> 25 >> >> libc.so.1`__cp_gethrtime+0x5e >> libc.so.1`__cp_clock_gettime_realtime+0x77 >> libc.so.1`__clock_gettime+0x72 >> libc.so.1`clock_gettime+0x26 >> nwamd`in_past+0x23 >> nwamd`nwamd_event_dequeue+0x1bf >> nwamd`nwamd_event_handler+0x162 >> nwamd`main+0x1b0 >> nwamd`_start_crt+0x9a >> nwamd`_start+0x1a >> 27 >> >> libc.so.1`__cp_tsc_read+0x19 >> libc.so.1`__cp_gethrtime+0x39 >> libc.so.1`__cp_clock_gettime_realtime+0x77 >> libc.so.1`__clock_gettime+0x72 >> libc.so.1`clock_gettime+0x26 >> nwamd`in_past+0x23 >> nwamd`nwamd_event_dequeue+0x1bf >> nwamd`nwamd_event_handler+0x162 >> nwamd`main+0x1b0 >> nwamd`_start_crt+0x9a >> nwamd`_start+0x1a >> 403 >> # dtrace -n 'profile-101 /pid == $target/ { @[ustack()] = count() } tick-10s{exit(0)}' -p $(pgrep -x mariadbd) | tail -n 40 >> dtrace: description 'profile-101 ' matched 2 probes >> mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xb8 >> mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a >> mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 >> libstdc++.so.6.0.32`execute_native_thread_routine+0x10 >> libc.so.1`_thrp_setup+0x77 >> libc.so.1`_lwp_start >> 126 >> >> libc.so.1`__cp_tsc_read+0xf >> libc.so.1`clock_gettime+0x15 >> libstdc++.so.6.0.32`_ZNSt6chrono3_V212steady_clock3nowEv+0x16 >> mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xb0 >> mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a >> mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 >> libstdc++.so.6.0.32`execute_native_thread_routine+0x10 >> libc.so.1`_thrp_setup+0x77 >> libc.so.1`_lwp_start >> 130 >> >> libc.so.1`__cp_tsc_read+0xf >> libc.so.1`clock_gettime+0x15 >> libstdc++.so.6.0.32`_ZNSt6chrono3_V212system_clock3nowEv+0x16 >> mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0x103 >> mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a >> mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 >> libstdc++.so.6.0.32`execute_native_thread_routine+0x10 >> libc.so.1`_thrp_setup+0x77 >> libc.so.1`_lwp_start >> 135 >> >> libc.so.1`__cp_tsc_read+0xf >> libc.so.1`clock_gettime+0x15 >> libstdc++.so.6.0.32`_ZNSt6chrono3_V212steady_clock3nowEv+0x16 >> mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xa8 >> mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a >> mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 >> libstdc++.so.6.0.32`execute_native_thread_routine+0x10 >> libc.so.1`_thrp_setup+0x77 >> libc.so.1`_lwp_start >> 137 >> >> >> The obvious suspect is: >> >> commit 8b6b46dcb073dba71917d6a7309f0df7bad798a2 >> Author: Robert Mustacchi >> Date: Tue Jul 23 14:44:22 2024 +0000 >> >> 14237 Want support for pthread_cond_clockwait() and friends >> Reviewed by: Andy Fiddaman >> Approved by: Gordon Ross >> >> >> but I didn't bisect yet. > > Thanks for the report, Marcel. I will dig in and see what I can find. > Apologies for the trouble. I've root caused this and written it up in https://www.illumos.org/issues/16683. The short form is that I incorrectly handled how the default initializer set the internal clock id in the cond_t. I have verified that nwam no longer is in a 100% loop with the fix in place and confirmed that if I used static initializers that my tests properly caught the issue before the fix and it is working afterwards. I'll be sending a review request with additional regression tests in a separate thread. Again, I'm sorry for the trouble and inconvenience that this caused. Thank you for reporting this Marcel. Robert