From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on starla X-Spam-Level: X-Spam-Status: No, score=-1.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=4.0.1 Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=m88iZHec; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=hNkJU6Ba; dkim-atps=neutral Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [94.130.110.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id C5FA91F47D for ; Thu, 11 Sep 2025 21:07:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1757624794; bh=iQOKChKXGkQsLXjD4qwBrhl/n4WfJq288KG6/KE1URc=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=m88iZHecPkb8iYpRDxUlkHZr4sikuVfTcfap8lM6glTrEb31dj2bokaeC/BBXRHCO oLR2dyBrfDVjaXaWtjUWycZbG6BjxpFc3yLbg8fMWAXC7seuEnZkWu4oArBTHOBewz cx83vaKGeNFbMxlAn7hOLXFgn4Hfeet4z196cE8A= Received: from nue.mailmanlists.eu (localhost [IPv6:::1]) by nue.mailmanlists.eu (Postfix) with ESMTP id 3BD8242E0E for ; Thu, 11 Sep 2025 21:06:34 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=hNkJU6Ba; dkim-atps=neutral Received: from s.wrqvtzvf.outbound-mail.sendgrid.net (s.wrqvtzvf.outbound-mail.sendgrid.net [149.72.126.143]) by nue.mailmanlists.eu (Postfix) with ESMTPS id 9FBC642D10 for ; Thu, 11 Sep 2025 21:05:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=date:from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:date:from:subject:to; s=s1; bh=6p6gKp+Un7zq2ooHzYgdTTabw/VvHvE0C+uhwBycMOY=; b=hNkJU6BaYXJjyK3O50148xxX1SFXoeLMyRvxlf7K73HV+4u8xkeCBjarhtRZqnjqpJHM rcou0vwAwt8DKia90JC0X+aNgYReSQWXpIOJJpWjb/Uy0GGtHuhGld4XXMzmWs+suEgzwo cRwmDMvXTswKU9sNQXDi0dbHuNTjx6qey6mPu2IxZ+PoZ8j5qGZYVX7e0eHmkfe0M5biZy 2ABg1qHTZK6IZJ2tatlzKpvNuCLiesZJ0QjK4GZGsYcaG2dl9Vc/rCMbeT0SPoYIgkXCH1 bBG/CT4K/bmA57Rkr0Q0irbMRz7y5NJpCq3YXWZdZ8zvPbQ4FM1Hrod58DDXSCng== Received: by recvd-6f5cbff5dc-6rwjf with SMTP id recvd-6f5cbff5dc-6rwjf-1-68C33994-28 2025-09-11 21:05:24.479876054 +0000 UTC m=+786275.589840868 Received: from herokuapp.com (unknown) by geopod-ismtpd-14 (SG) with ESMTP id hP6Nrl4CSWa2GPQSAZrc2w for ; Thu, 11 Sep 2025 21:05:24.436 +0000 (UTC) Date: Thu, 11 Sep 2025 21:05:24 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 21571 X-Redmine-Issue-Author: dmorner X-Redmine-Issue-Priority: Normal X-Redmine-Sender: dmorner X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 100022 X-SG-EID: =?us-ascii?Q?u001=2EO3G=2Foho8mwLEY7Kt=2FHzf7Ij1DDvQ+IOwgNts24uqmQJ31icbQWgfOAFV9?= =?us-ascii?Q?burujJjiONuoxQ2Nz+rpQa4hnzW08MwQLENB0bl?= =?us-ascii?Q?NrmuFcX4zL03N8S4WNY=2F3x8YE2VvIRANbKV0aYa?= =?us-ascii?Q?EPKYtaE917zGWj=2Fz8M+oved97iqXqn5VROOXUks?= =?us-ascii?Q?5h9A06eD3yhjD0Jo0=2F+ZBTMJu7Hwvwy+gJgEeoh?= =?us-ascii?Q?Iv64sjY+b=2FbL0vyEPlXYGGrdsRkUvMcm0abt6xl?= =?us-ascii?Q?JK3iaHV3Z5Wslt4jiQLimWXdgw=3D=3D?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: ATVWIVS7BEUC53A6A44GJMO44VIWNEKR X-Message-ID-Hash: ATVWIVS7BEUC53A6A44GJMO44VIWNEKR X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:123224] [Ruby Bug#21571] Ruby sporadically hanging on exit List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "dmorner (Daniel Orner) via ruby-core" Cc: "dmorner (Daniel Orner)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #21571 has been reported by dmorner (Daniel Orner). ---------------------------------------- Bug #21571: Ruby sporadically hanging on exit https://bugs.ruby-lang.org/issues/21571 * Author: dmorner (Daniel Orner) * Status: Open * ruby -v: ruby 3.4.5 (2025-07-16 revision 20cda200d3) +YJIT +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- This is my first bug report, so please let me know if there's anything I can do to improve it. We have a production-grade Rails app that's been running for many years. We recently moved to EKS and upgraded it to the latest Ruby and Rails. We have a number of delayed_job processes that fork on every job that comes in so that the OS can reclaim the memory used in executing it (we implemented this a long time ago because Ruby never gives up any memory that it takes, and some jobs use way more memory than others). In the last couple of weeks, we've noticed a rare occurrence where the delayed job hangs when exiting. The code looks like this: 
    Process.fork do
      ActiveRecord::Base.establish_connection
      execute_job
    end
    Process.wait
The forked child process doesn't exit when this bug occurs, it's just stuck forever, doing nothing. Obviously I don't have a way to reproduce this because it happens maybe once every few thousand jobs, and it happens across all job types. If I run gdb on the child process, I always see something that looks like this (note: I am a total gdb newbie): 
#0  __futex_abstimed_wait_common
    (futex_word=futex_word@entry=0x7fb6af41400c, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=, cancel=cancel@entry=false) at ./nptl/futex-internal.c:103
#1  0x00007fb6d5677f68 in __GI___futex_abstimed_wait64
    (futex_word=futex_word@entry=0x7fb6af41400c, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=) at ./nptl/futex-internal.c:128
#2  0x00007fb6d568138c in __pthread_rwlock_wrlock_full64 (abstime=0x0, clockid=0, rwlock=0x7fb6af414000) at ./nptl/pthread_rwlock_common.c:730
#3  ___pthread_rwlock_wrlock (rwlock=0x7fb6af414000) at ./nptl/pthread_rwlock_wrlock.c:26
#4  0x00007fb6aee22989 in CRYPTO_THREAD_write_lock () at /lib/x86_64-linux-gnu/libcrypto.so.3
#5  0x00007fb6aee15c6a in  () at /lib/x86_64-linux-gnu/libcrypto.so.3
#6  0x00007fb6aee15fa9 in OPENSSL_thread_stop () at /lib/x86_64-linux-gnu/libcrypto.so.3
#7  0x00007fb6aee153b5 in OPENSSL_cleanup () at /lib/x86_64-linux-gnu/libcrypto.so.3
#8  0x00007fb6d563055d in __run_exit_handlers
    (status=0, listp=0x7fb6d57c5820 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at ./stdlib/exit.c:116
#9  0x00007fb6d563069a in __GI_exit (status=) at ./stdlib/exit.c:146
#10 0x00007fb6d5ad3a80 in ruby_stop (ex=) at eval.c:290
#11 0x00007fb6d5bc47b4 in rb_f_fork (obj=) at process.c:4388
#12 rb_f_fork (obj=) at process.c:4378
#13 0x00007fb6d5cad5cc in vm_call_cfunc_with_frame_
    (stack_bottom=, argv=, argc=0, calling=, reg_cfp=0x7fb6d4f68280, ec=0x7fb6d4e4d550)
    at /usr/src/ruby/vm_insnhelper.c:3794
#14 vm_call_cfunc_with_frame (ec=0x7fb6d4e4d550, reg_cfp=0x7fb6d4f68280, calling=) at /usr/src/ruby/vm_insnhelper.c:3840
#15 0x00007fb6d5cb3fef in vm_sendish
    (ec=0x7fb6d4e4d550, reg_cfp=0x7fb6d4f68280, cd=0x7fb69fb17650, block_handler=, method_explorer=mexp_search_method)
    at /usr/src/ruby/vm_callinfo.h:415
#16 0x00007fb6d5cc1e59 in vm_exec_core (ec=0x7fb6af41400c, ec@entry=0x7fb6d4e4d550) at /usr/src/ruby/insns.def:851
#17 0x00007fb6d5cc7ba9 in rb_vm_exec (ec=0x7fb6d4e4d550) at vm.c:2595
#18 0x00007fb6b13e73b9 in  ()
#19 0x00007fb6d4f68328 in  ()
...etc, I can paste more if needed
I can't seem to get `call rb_backtrace()` working in gdb, it never prints anything. This seems to indicate that there's some kind of thread lock when OpenSSL is shutting down. The crazy thing is that **there is only one thread** for most of the processes I inspect. Any help would be greatly appreciated! -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/