From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [94.130.110.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id CFA871F47A for ; Mon, 19 Aug 2024 01:17:43 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=B8XkOh8O; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=QqLPR8y6; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1724030230; bh=j1TbVLtZRAZnu27DVZPY78bLc3MVHDO4Jr5aivdoRdA=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=B8XkOh8OrticY4Mgv9XIQSPyXBZgvMEzJM/iaKXOzohqeLaXr5dPdmay0Z5PWsCfO QFSvy/gfOsSQmwcNmz46z6hwtRBsl5cArmuBmQLoTt6EQJIm/HBkh8sBWuukXNFTOD JWr6PLsWB2AU5y4PoJdkO+OEfQDF84fO7YNAj1Yw= Received: from nue.mailmanlists.eu (localhost [IPv6:::1]) by nue.mailmanlists.eu (Postfix) with ESMTP id 44B9543C83 for ; Mon, 19 Aug 2024 01:17:10 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=QqLPR8y6; dkim-atps=neutral Received: from s.wfbtzhsw.outbound-mail.sendgrid.net (s.wfbtzhsw.outbound-mail.sendgrid.net [159.183.224.105]) by nue.mailmanlists.eu (Postfix) with ESMTPS id 2179841631 for ; Mon, 19 Aug 2024 01:16:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:from:subject:to; s=s1; bh=lnL4RuZDxTNy61Zj05h2AQGLyPVDasmoOgoB0bUojes=; b=QqLPR8y6lByF4m7XIswSHfhtzvGJ85TfZxMcGN7WaKuvqHSRQ0hMCu28B5DHnR9v4T4U Lcqn6NILXDdXsr5H0kIGqrsV3PN0A7HZtbTrCDpGGLUdL/w8TnMSkQdvhsckiyZ9oHxeL5 pFt2+RlOlJgyHqL83yTxaX4QDQRSTWzuvYL7/cYkRElGWkZq7u8GUJn8NiKDoNz7L1n6hP TbOpLFXBH8FUjwUei7/IjtXdDBd6JCshDGWwVP4hzvxceAtEgSIEghw4hmIbtoqjop2h/9 bJM6/AXkpHibJBExBDBNbpFk6H7tuitYPTqVQxhcCdlxo3hbAYda8Q2H1/uNHCzw== Received: by recvd-b887ddd65-kwhgp with SMTP id recvd-b887ddd65-kwhgp-1-66C29D09-1D 2024-08-19 01:16:57.951202471 +0000 UTC m=+1742270.555991558 Received: from herokuapp.com (unknown) by geopod-ismtpd-26 (SG) with ESMTP id 4c0f4caySBSvTqJvDeTHNA for ; Mon, 19 Aug 2024 01:16:57.932 +0000 (UTC) Date: Mon, 19 Aug 2024 01:16:57 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 20682 X-Redmine-Issue-Author: ono-max X-Redmine-Issue-Priority: Normal X-Redmine-Sender: ono-max X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 95515 X-SG-EID: =?us-ascii?Q?u001=2EZDJU3MvRcQoADZu0w21uLDEuPIX6K8yXc=2F4ZLkljuGHdekcm0ImiA7H+T?= =?us-ascii?Q?OgVabcqI8RNFFndJh625xL8LO=2FhTiKsXz5uvoVH?= =?us-ascii?Q?lK7r9GmBPr5D3PW29+zRjFb4v=2FlQqQWvX4yFZEl?= =?us-ascii?Q?qbrpcBNxA=2FGnavNBIlOtfSvKwY1Lcg=2FtxHIFu8g?= =?us-ascii?Q?Fib1wFxnOEIbIMRBONQeCBwRUS0LAVNBSqVWTwM?= =?us-ascii?Q?ouXkaz+Rx1RXvbgUhJ2XbCMNZPQt4qUGXEXEjAl?= =?us-ascii?Q?QcBe?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: G3U5ABOB2F5DQEDXQIQR5HURJ5GGLYC6 X-Message-ID-Hash: G3U5ABOB2F5DQEDXQIQR5HURJ5GGLYC6 X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.9 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:118879] [Ruby master Bug#20682] Slave PTY output is lost after a child process exits in macOS List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "ono-max (Naoto Ono) via ruby-core" Cc: "ono-max (Naoto Ono)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #20682 has been reported by ono-max (Naoto Ono). ---------------------------------------- Bug #20682: Slave PTY output is lost after a child process exits in macOS https://bugs.ruby-lang.org/issues/20682 * Author: ono-max (Naoto Ono) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- According to Launchable, the following PTY tests are flaky only on macOS. https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_without_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_with_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_commandline https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_argv0 It's because the slave PTY output is lost after a child process exits in macOS. Here is the code to reproduce the problem. When I remove `sleep 3` from the code, "a" is returned. ``` require 'pty' r, w, pid = PTY.spawn('ruby', '-e', 'puts "a"') sleep 3 puts r.gets #=> Returns nil ``` Based on my investigation, this issue happens in the macOS side and it's almost same as https://github.com/pexpect/pexpect/issues/662. The cause is described as follows in the ticket: > // NOTE[macOS-S_CTTYREF]: On macOS, after a forkpty(), if the pty slave (child) // is closed before the pty master (parent) reads, the pty's buffer is cleared // thus the master (parent) reads nothing. This can happen if the child exits // before the parent has a chance to call master.read(). // // This issue has been reported to Apple, but has not been resolved: // https://developer.apple.com/forums/thread/663632 // // Work around this issue by opening /dev/tty then closing it. This ultimately // causes the child's exit() to flush the slave pty's output buffer in a // blocking way. This fixes the problem on macOS 13.2 in my testing. // // Here's how the workaround works in detail: // // If we open /dev/tty, it sets the S_CTTYREF flag on the process. This flag // remains set if we close the /dev/tty file descriptor. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L128 // Additionally, opening /dev/tty retains a reference to the pty slave. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L147 // // When the child process exits: // // 1. All open file descriptors (including stdin/stdout/stderr which are the pty // slave) are closed. This does *not* drain unread pty slave output. // * If S_CTTYREF was set, closing the file descriptors does not close the // last reference to the pty slave, so no cleanup happens yet. // * NOTE[macOS-pty-close-loss]: If S_CTTYREF was not set, closing the file // descriptors drops the last reference to the pty slave. Unread data is // dropped. // // 2. If the S_CTTYREF flag is set on the child process, the controlling // terminal (pty slave) is closed. XNU's ptsclose() ultimately calls // ttywait(). // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/kern_exit.c#L2272 // * ttywait() is the same as ioctl(slave, TIOCDRAIN); it blocks waiting for // output to be received. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty.c#L1129-L1130 // * NOTE[macOS-pty-waitpid-hang]: Because of the blocking ttywait(), the // process is in an exiting (but not zombie) state. waitpid() will hang. // // * NOTE[macOS-pty-close-loss]: If the S_CTTYREF flag is not set on the // child process, ttywait() is not called, thus the pty slave does not // block waiting for the output to be received, and the output is dropped. // A well-behaving parent will use a poll() loop anyway, so this isn't a // problem. (It does make quick tests annoying to write though.) // // Demonstration of NOTE[macOS-pty-close-loss] (S_CTTYREF is not set before // exit): // // // On macOS, this program should report 'data = ""', demonstrating that // // writes are lost. // // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // // Cause the child to write() then exit(). exit() will drop written // // data. // sleep(1); // // char buffer[10]; // ssize_t rc = read(tty_fd, buffer, sizeof(buffer)); // if (rc < 0) { perror("read"); abort(); } // fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // } // // return 0; // } // // Demonstration of NOTE[macOS-pty-waitpid-hang] (S_CTTYREF is set before exit): // // // On macOS, this program should hang, demonstrating that the child // // process doesn't finish exiting. // // // // During the hang, observe that the child is in an exiting state ("E"): // // // // $ ps -e -o pid,stat | grep 20125 // // 20125 ?Es // // #include // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // close(open("/dev/tty", O_WRONLY)); // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // fprintf(stderr, "child PID: %d\n", pid); // // // This will hang because, despite the child being is an exiting // // state, the child is waiting for us to read(). // pid_t rc = waitpid(pid, NULL, 0); // if (rc < 0) { perror("waitpid"); abort(); } // } // // return 0; // } In Ruby, PTY is implemented with [fork()](https://github.com/ruby/ruby/blob/master/process.c#L1706) and [posix_openpt()](https://github.com/ruby/ruby/blob/master/ext/pty/pty.c#L329) in macOS. I could reproduce the problem in the following script. ``` #include #include #include #include #include int main() { int master_fd, slave_fd; pid_t child_pid; char *slave_name; // Open a master pseudo-terminal master_fd = posix_openpt(O_RDWR | O_NOCTTY); if (master_fd == -1) { perror("posix_openpt"); exit(1); } // Grant access to the slave pseudo-terminal if (grantpt(master_fd) == -1) { perror("grantpt"); exit(1); } // Unlock the slave pseudo-terminal if (unlockpt(master_fd) == -1) { perror("unlockpt"); exit(1); } // Get the name of the slave pseudo-terminal slave_name = ptsname(master_fd); if (slave_name == NULL) { perror("ptsname"); exit(1); } // Fork a child process child_pid = fork(); if (child_pid == -1) { perror("fork"); exit(1); } else if (child_pid == 0) { // Child process // Open the slave pseudo-terminal slave_fd = open(slave_name, O_RDWR); if (slave_fd == -1) { perror("open"); exit(1); } // Create a new session and process group if (setsid() == -1) { perror("setsid"); exit(1); } // Set the controlling terminal for the child process if (ioctl(slave_fd, TIOCSCTTY, NULL) == -1) { perror("ioctl"); exit(1); } // Duplicate the slave file descriptor to stdin, stdout, and stderr if (dup2(slave_fd, STDIN_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDOUT_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDERR_FILENO) == -1) { perror("dup2"); exit(1); } // close(open("/dev/tty", O_WRONLY)); // Close the original slave file descriptor close(slave_fd); // Execute a shell or other program (void)write(STDOUT_FILENO, "y", 1); exit(1); } else { sleep(5); char buffer[10]; ssize_t rc = read(master_fd, buffer, sizeof(buffer)); if (rc < 0) { perror("read"); abort(); } fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // Clean up close(master_fd); } return 0; } ``` -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/