From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id 5E1C71E3F405 for ; Sun, 28 Jul 2024 01:56:08 -0400 (EDT) (envelope-from marcel@telka.sk) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id A83E739B80D; Sun, 28 Jul 2024 01:56:08 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1722146168; b=pwlfLawfRr+s3+5uHowXUNWBTZmfLO+0mwoAQDPN7x8TFyUDjv SLcep/P1W+3wG/sdglgZ35aqlO0nisrc13pGwd7NMkxWFKDF6oEY+D6RaenAooQX xF+Pe5AjnOyraC3vdxl5tnPG+howticFg75GonlftL87eEihUQ/vK2C2gqo/MwuJ Mn9rrhNu9h06eqFwFe5VTYfBnfOns48csII0lgIOaOvovNbD9EMdpekSpyeK5U24 CHM/Ha3EoRTfj2jGA0ACW/cxFgfUc1cRWwi2Wk/DG0c+6J9nKxdR/fQBs4EF+Xww r02VmOc3D5j0cDxAUJxveQ6FP4noWRXnzRuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=date:from:to:subject:message-id:mime-version :content-type; s=arcseal; t=1722146168; bh=07EU4BEb9JLN6R1nuYMUo /CPCFkvikiEfYEM3JjnW94=; b=dGdFrJKHQVyFnHxFtGlC45pLUUyY8MAt5poA+ sNhivVScRe+NAvzoCdsJSmEg7O7Yygb+iAPK701ay+NqNGqwOqejUfm5Eh6Pxh53 HpV+NWT0J9Nm+R7EZNml3VoL3ZN1lwjPQ7KW/nP+zfx1uyFcNEU5LuOLXfTLq/Hr AtXpmGJ22fyDDi9pHL1vta8W+e3kJIr0vTq74+XeEHKxB2qrQt1caFHca6w+JBSw LYPgRnQCmB9lmgrMQX+4vwg2giDgwpPSk4F98hyRpyhgVVETknylwmy+U1oVPaKl mso0rGEJ2LqkTPZBIxLN1n+FH6hxIVyGnqryLeOVszp1n1qzQ== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=none (no signatures found); dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=telka.sk; iprev=pass smtp.remote-ip=185.14.234.118 (tortuga.telka.sk); spf=pass smtp.mailfrom=marcel@telka.sk smtp.helo=tortuga.telka.sk; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=tortuga.telka.sk policy.ptr=tortuga.telka.sk; x-return-mx=pass header.domain=telka.sk policy.is_org=yes (MX Records found: tortuga.telka.sk); x-return-mx=pass smtp.domain=telka.sk policy.is_org=yes (MX Records found: tortuga.telka.sk); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=none (no signatures found); dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=telka.sk; iprev=pass smtp.remote-ip=185.14.234.118 (tortuga.telka.sk); spf=pass smtp.mailfrom=marcel@telka.sk smtp.helo=tortuga.telka.sk; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=tortuga.telka.sk policy.ptr=tortuga.telka.sk; x-return-mx=pass header.domain=telka.sk policy.is_org=yes (MX Records found: tortuga.telka.sk); x-return-mx=pass smtp.domain=telka.sk policy.is_org=yes (MX Records found: tortuga.telka.sk); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrieekgddutdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuf fkgggtugesthdtredttddtvdenucfhrhhomhepofgrrhgtvghlucfvvghlkhgruceomhgr rhgtvghlsehtvghlkhgrrdhskheqnecuggftrfgrthhtvghrnhepgfeujeeuieeghefgie eivdeggedugfdttdejheegjeekvddtffeutedtheduhedtnecuffhomhgrihhnpehtvghl khgrrdhskhenucfkphepudekhedrudegrddvfeegrdduudeknecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehinhgvthepudekhedrudegrddvfeegrdduudekpdhhvghl ohepthhorhhtuhhgrgdrthgvlhhkrgdrshhkpdhmrghilhhfrhhomhepoehmrghrtggvlh esthgvlhhkrgdrshhkqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeouggvvhgv lhhophgvrheslhhishhtshdrihhllhhumhhoshdrohhrgheq X-ME-VSScore: 0 X-ME-VSCategory: clean Received-SPF: pass (telka.sk: 185.14.234.118 is authorized to use 'marcel@telka.sk' in 'mfrom' identity (mechanism 'ip4:185.14.234.118' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="marcel@telka.sk"; helo=tortuga.telka.sk; client-ip=185.14.234.118 Received: from tortuga.telka.sk (tortuga.telka.sk [185.14.234.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for ; Sun, 28 Jul 2024 01:56:06 -0400 (EDT) (envelope-from marcel@telka.sk) Received: (qmail 6469 invoked from network); 28 Jul 2024 05:56:04 -0000 Received: from telcontar.in.telka.sk (HELO telcontar) (marcel@10.0.0.10) by tortuga.telka.sk with ESMTPSA (DHE-RSA-AES256-GCM-SHA384 encrypted); 28 Jul 2024 05:56:04 -0000 Date: Sun, 28 Jul 2024 07:56:03 +0200 From: Marcel Telka To: developer@lists.illumos.org Subject: CPU usage - clock related issue Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 18b0de1a-4ca6-11ef-9770-f90e51104628 Hi, It looks like something went wrong between changesets 6e0c6e37fb and 8b913f79fc in the illumos-gate. After upgrade of OpenIndiana from osnet-incorporation@0.5.11-2024.0.0.22264 (illumos-6e0c6e37fb) to osnet-incorporation@0.5.11-2024.0.0.22271 (illumos-8b913f79fc) I see two processes eating full CPU: nwamd and mariadbd. The machine is a qemu/kvm quest (host is Rocky 9). # dtrace -n 'profile-101 /pid == $target/ { @[ustack()] = count() } tick-10s{exit(0)}' -p $(pgrep -x nwamd) | tail -n 40 dtrace: description 'profile-101 ' matched 2 probes nwamd`in_past+0x23 nwamd`nwamd_event_dequeue+0x1bf nwamd`nwamd_event_handler+0x162 nwamd`main+0x1b0 nwamd`_start_crt+0x9a nwamd`_start+0x1a 24 nwamd`in_past+0x26 nwamd`nwamd_event_dequeue+0x1bf nwamd`nwamd_event_handler+0x162 nwamd`main+0x1b0 nwamd`_start_crt+0x9a nwamd`_start+0x1a 25 libc.so.1`__cp_gethrtime+0x5e libc.so.1`__cp_clock_gettime_realtime+0x77 libc.so.1`__clock_gettime+0x72 libc.so.1`clock_gettime+0x26 nwamd`in_past+0x23 nwamd`nwamd_event_dequeue+0x1bf nwamd`nwamd_event_handler+0x162 nwamd`main+0x1b0 nwamd`_start_crt+0x9a nwamd`_start+0x1a 27 libc.so.1`__cp_tsc_read+0x19 libc.so.1`__cp_gethrtime+0x39 libc.so.1`__cp_clock_gettime_realtime+0x77 libc.so.1`__clock_gettime+0x72 libc.so.1`clock_gettime+0x26 nwamd`in_past+0x23 nwamd`nwamd_event_dequeue+0x1bf nwamd`nwamd_event_handler+0x162 nwamd`main+0x1b0 nwamd`_start_crt+0x9a nwamd`_start+0x1a 403 # dtrace -n 'profile-101 /pid == $target/ { @[ustack()] = count() } tick-10s{exit(0)}' -p $(pgrep -x mariadbd) | tail -n 40 dtrace: description 'profile-101 ' matched 2 probes mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xb8 mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 libstdc++.so.6.0.32`execute_native_thread_routine+0x10 libc.so.1`_thrp_setup+0x77 libc.so.1`_lwp_start 126 libc.so.1`__cp_tsc_read+0xf libc.so.1`clock_gettime+0x15 libstdc++.so.6.0.32`_ZNSt6chrono3_V212steady_clock3nowEv+0x16 mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xb0 mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 libstdc++.so.6.0.32`execute_native_thread_routine+0x10 libc.so.1`_thrp_setup+0x77 libc.so.1`_lwp_start 130 libc.so.1`__cp_tsc_read+0xf libc.so.1`clock_gettime+0x15 libstdc++.so.6.0.32`_ZNSt6chrono3_V212system_clock3nowEv+0x16 mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0x103 mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 libstdc++.so.6.0.32`execute_native_thread_routine+0x10 libc.so.1`_thrp_setup+0x77 libc.so.1`_lwp_start 135 libc.so.1`__cp_tsc_read+0xf libc.so.1`clock_gettime+0x15 libstdc++.so.6.0.32`_ZNSt6chrono3_V212steady_clock3nowEv+0x16 mariadbd`_ZN5tpool19thread_pool_generic14wait_for_tasksERSt11unique_lockISt5mutexEPNS_11worker_dataE+0xa8 mariadbd`_ZN5tpool19thread_pool_generic8get_taskEPNS_11worker_dataEPPNS_4taskE+0x8a mariadbd`_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x65 libstdc++.so.6.0.32`execute_native_thread_routine+0x10 libc.so.1`_thrp_setup+0x77 libc.so.1`_lwp_start 137 # The obvious suspect is: commit 8b6b46dcb073dba71917d6a7309f0df7bad798a2 Author: Robert Mustacchi Date: Tue Jul 23 14:44:22 2024 +0000 14237 Want support for pthread_cond_clockwait() and friends Reviewed by: Andy Fiddaman Approved by: Gordon Ross but I didn't bisect yet. Regards. -- +-------------------------------------------+ | Marcel Telka e-mail: marcel@telka.sk | | homepage: http://telka.sk/ | +-------------------------------------------+