From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [94.130.110.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 532581F508 for ; Mon, 27 Jan 2025 08:18:32 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=x3qg3cy6; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=jrwrL1HM; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1737965878; bh=AX7pL16lGm/qN6dJs2JnY6ODDS/y9oTMHkRKn5LtTuo=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=x3qg3cy6YdCWRMxzK6G4d7u+qXTnhbaRltX+rFMpzv3Vc0FQaZq0SHSO326yyr5Kd LRFGMwulWnTBx2UfTepGxuD0Sm6NxlmRQVH5/EchQwjYm6eYrB0XjMfnxfEWRIaU/A lUa537W7mn6VzDaiYu5Ig1Y01T6Qg4P8RdwXWPNg= Received: from nue.mailmanlists.eu (localhost [IPv6:::1]) by nue.mailmanlists.eu (Postfix) with ESMTP id EA0FF46860 for ; Mon, 27 Jan 2025 08:17:58 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=jrwrL1HM; dkim-atps=neutral Received: from s.wrqvtbkv.outbound-mail.sendgrid.net (s.wrqvtbkv.outbound-mail.sendgrid.net [149.72.123.24]) by nue.mailmanlists.eu (Postfix) with ESMTPS id 2F3DC46820 for ; Mon, 27 Jan 2025 08:17:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:from:subject:to; s=s1; bh=0dkuGRaUXhoIf+AIlTlLK3YKW2u1wbeblvrjEejsSZA=; b=jrwrL1HMYWqkAlINJJMMplEvcTe4SsLSll4ipquiMsiz2VLWf25MQEu0gN05SnDl/IOq E65aA+fBe82k3Wp6Q0OigbJPFjwSFw3jLVOCPtJ5WN0l+uCaIBr3kHl9uAOtOoyf2ZFZXQ a5X2xrKNnZsS63DYYg/f6GWnTnlYS/xxt9TuvirnKYxNue81yh0l3ylb3MHsn7YwrjvvMx AbLq+YS9VmlhQlj+NBVt+WcO5A5TO1DKYJeWDpNOhcrs0HoRcA+v3f9ovjbwP3i3ZhzQLU H4j3GtKDJEGQyNeGTCbRGbvnrxUiMH92V6fvnNvDt38qA+qAkl3N2xuPhLDtbaYQ== Received: by recvd-5f9ffdf494-tt7kt with SMTP id recvd-5f9ffdf494-tt7kt-1-6797412F-5 2025-01-27 08:17:51.88343938 +0000 UTC m=+6346871.938780413 Received: from herokuapp.com (unknown) by geopod-ismtpd-26 (SG) with ESMTP id AHGC4HOiSP6fly6VHf_0CA for ; Mon, 27 Jan 2025 08:17:51.811 +0000 (UTC) Date: Mon, 27 Jan 2025 08:17:51 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 20682 X-Redmine-Issue-Author: ono-max X-Redmine-Issue-Priority: Normal X-Redmine-Sender: nobu X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 97516 X-SG-EID: =?us-ascii?Q?u001=2E5PtzXJ23KrYzgM1nrOIr+EQ222PyrDaWSg0Er8CZ8tP86xyXmBM81zBKD?= =?us-ascii?Q?HreavdFYMbHjxXOR6UPMkt=2Fu9CyBIp6y52n8D2y?= =?us-ascii?Q?qA9zqurrgUf3XIM7xqZS=2FNWBOBZlQtetE9K+wwk?= =?us-ascii?Q?uCthwdlQ1StyO1fxe1eKQIF=2FoZTqINWenguEaCr?= =?us-ascii?Q?KEjtbvIoTtdFHpQa9rwfoKXBEdtO4pTrwzIbY3U?= =?us-ascii?Q?R=2FfAwsE3rwfh4nIXcRE5UJ13XixUUMjjA37n=2FBQ?= =?us-ascii?Q?Ejdd21AypasQd=2FfnGSZRMsRjxw=3D=3D?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: XGEH5FQNOJVQT5VZG4QRSUDPL3L4RNXY X-Message-ID-Hash: XGEH5FQNOJVQT5VZG4QRSUDPL3L4RNXY X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.9 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:120798] [Ruby master Bug#20682] Slave PTY output is lost after a child process exits in macOS List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "nobu (Nobuyoshi Nakada) via ruby-core" Cc: "nobu (Nobuyoshi Nakada)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #20682 has been updated by nobu (Nobuyoshi Nakada). ono-max (Naoto Ono) wrote in #note-4: > This program hangs in `Process.wait(pid)`. > > ```ruby > require 'pty' > _, _, pid = PTY.spawn('ruby', '-e', 'puts "a"; puts "b"') > Process.waitpid(pid) > ``` Now I tried this, and it exited immediately, on arm64 macOS 14.7.2 23H311. However, the code in the description still returns `nil`. > ```ruby > require 'pty' > > r, w, pid = PTY.spawn('ruby', '-e', 'puts "a"') > sleep 3 > puts r.gets #=> Returns nil > ``` ---------------------------------------- Bug #20682: Slave PTY output is lost after a child process exits in macOS https://bugs.ruby-lang.org/issues/20682#change-111665 * Author: ono-max (Naoto Ono) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- According to Launchable, the following PTY tests are flaky only on macOS. https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_without_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_with_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_commandline https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_argv0 It's because the slave PTY output is lost after a child process exits in macOS. Here is the code to reproduce the problem. When I remove `sleep 3` from the code, "a" is returned. ``` require 'pty' r, w, pid = PTY.spawn('ruby', '-e', 'puts "a"') sleep 3 puts r.gets #=> Returns nil ``` Based on my investigation, this issue happens in the macOS side and it's almost same as https://github.com/pexpect/pexpect/issues/662. The cause is described as follows in the ticket: > // NOTE[macOS-S_CTTYREF]: On macOS, after a forkpty(), if the pty slave (child) // is closed before the pty master (parent) reads, the pty's buffer is cleared // thus the master (parent) reads nothing. This can happen if the child exits // before the parent has a chance to call master.read(). // // This issue has been reported to Apple, but has not been resolved: // https://developer.apple.com/forums/thread/663632 // // Work around this issue by opening /dev/tty then closing it. This ultimately // causes the child's exit() to flush the slave pty's output buffer in a // blocking way. This fixes the problem on macOS 13.2 in my testing. // // Here's how the workaround works in detail: // // If we open /dev/tty, it sets the S_CTTYREF flag on the process. This flag // remains set if we close the /dev/tty file descriptor. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L128 // Additionally, opening /dev/tty retains a reference to the pty slave. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L147 // // When the child process exits: // // 1. All open file descriptors (including stdin/stdout/stderr which are the pty // slave) are closed. This does *not* drain unread pty slave output. // * If S_CTTYREF was set, closing the file descriptors does not close the // last reference to the pty slave, so no cleanup happens yet. // * NOTE[macOS-pty-close-loss]: If S_CTTYREF was not set, closing the file // descriptors drops the last reference to the pty slave. Unread data is // dropped. // // 2. If the S_CTTYREF flag is set on the child process, the controlling // terminal (pty slave) is closed. XNU's ptsclose() ultimately calls // ttywait(). // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/kern_exit.c#L2272 // * ttywait() is the same as ioctl(slave, TIOCDRAIN); it blocks waiting for // output to be received. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty.c#L1129-L1130 // * NOTE[macOS-pty-waitpid-hang]: Because of the blocking ttywait(), the // process is in an exiting (but not zombie) state. waitpid() will hang. // // * NOTE[macOS-pty-close-loss]: If the S_CTTYREF flag is not set on the // child process, ttywait() is not called, thus the pty slave does not // block waiting for the output to be received, and the output is dropped. // A well-behaving parent will use a poll() loop anyway, so this isn't a // problem. (It does make quick tests annoying to write though.) // // Demonstration of NOTE[macOS-pty-close-loss] (S_CTTYREF is not set before // exit): // // // On macOS, this program should report 'data = ""', demonstrating that // // writes are lost. // // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // // Cause the child to write() then exit(). exit() will drop written // // data. // sleep(1); // // char buffer[10]; // ssize_t rc = read(tty_fd, buffer, sizeof(buffer)); // if (rc < 0) { perror("read"); abort(); } // fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // } // // return 0; // } // // Demonstration of NOTE[macOS-pty-waitpid-hang] (S_CTTYREF is set before exit): // // // On macOS, this program should hang, demonstrating that the child // // process doesn't finish exiting. // // // // During the hang, observe that the child is in an exiting state ("E"): // // // // $ ps -e -o pid,stat | grep 20125 // // 20125 ?Es // // #include // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // close(open("/dev/tty", O_WRONLY)); // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // fprintf(stderr, "child PID: %d\n", pid); // // // This will hang because, despite the child being is an exiting // // state, the child is waiting for us to read(). // pid_t rc = waitpid(pid, NULL, 0); // if (rc < 0) { perror("waitpid"); abort(); } // } // // return 0; // } In Ruby, PTY is implemented with [fork()](https://github.com/ruby/ruby/blob/master/process.c#L1706) and [posix_openpt()](https://github.com/ruby/ruby/blob/master/ext/pty/pty.c#L329) in macOS. I could reproduce the problem in the following script. ``` #include #include #include #include #include int main() { int master_fd, slave_fd; pid_t child_pid; char *slave_name; // Open a master pseudo-terminal master_fd = posix_openpt(O_RDWR | O_NOCTTY); if (master_fd == -1) { perror("posix_openpt"); exit(1); } // Grant access to the slave pseudo-terminal if (grantpt(master_fd) == -1) { perror("grantpt"); exit(1); } // Unlock the slave pseudo-terminal if (unlockpt(master_fd) == -1) { perror("unlockpt"); exit(1); } // Get the name of the slave pseudo-terminal slave_name = ptsname(master_fd); if (slave_name == NULL) { perror("ptsname"); exit(1); } // Fork a child process child_pid = fork(); if (child_pid == -1) { perror("fork"); exit(1); } else if (child_pid == 0) { // Child process // Open the slave pseudo-terminal slave_fd = open(slave_name, O_RDWR); if (slave_fd == -1) { perror("open"); exit(1); } // Create a new session and process group if (setsid() == -1) { perror("setsid"); exit(1); } // Set the controlling terminal for the child process if (ioctl(slave_fd, TIOCSCTTY, NULL) == -1) { perror("ioctl"); exit(1); } // Duplicate the slave file descriptor to stdin, stdout, and stderr if (dup2(slave_fd, STDIN_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDOUT_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDERR_FILENO) == -1) { perror("dup2"); exit(1); } // close(open("/dev/tty", O_WRONLY)); // Close the original slave file descriptor close(slave_fd); // Execute a shell or other program (void)write(STDOUT_FILENO, "y", 1); exit(1); } else { sleep(5); char buffer[10]; ssize_t rc = read(master_fd, buffer, sizeof(buffer)); if (rc < 0) { perror("read"); abort(); } fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // Clean up close(master_fd); } return 0; } ``` -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/