From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=0.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_BL_SPAMCOP_NET,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [94.130.110.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 3BAD01F47A for ; Thu, 22 Aug 2024 07:55:14 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=wCJpQacF; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=HgCysLsy; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1724313311; bh=gFDiZKn3OXOilToKA1DmatQtWXD6iaXBQLvwkZ1i308=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=wCJpQacFF2enCjfuCTob/QhNajnjIxcAjiDr40FMfCRK5d/F3RR2f2Z0zKufiDyY2 Y2BDRXcmOxs2G7EiWXnTOboW4J/FkIQrj5NaKJC86JfikgDyH6v8wVbqvXt9qO1ejj gw16M4//qal47NCKlkSQ3cbEl1oH0RkJW7P3c3YE= Received: from nue.mailmanlists.eu (localhost [IPv6:::1]) by nue.mailmanlists.eu (Postfix) with ESMTP id 779B643CBC for ; Thu, 22 Aug 2024 07:55:11 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=HgCysLsy; dkim-atps=neutral Received: from s.wrqvtbkv.outbound-mail.sendgrid.net (s.wrqvtbkv.outbound-mail.sendgrid.net [149.72.123.24]) by nue.mailmanlists.eu (Postfix) with ESMTPS id 30F9B43BF7 for ; Thu, 22 Aug 2024 07:55:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:from:subject:to; s=s1; bh=cL7papb8ZDwZEfj/5SpLKFiCV2Q23hgAVASrlJZN0B0=; b=HgCysLsyOtrKnFvxhGQS1ZhRNSKAhZ/9m10qMOEcDTvv8hYnZ/cgzBvNY90otnPRVou9 TIU7Lj1LavXDUM3pPoeCiTgzAJJbh1lp/BgkwH3AS6aF7ZqpQ2Ry+eNfWXvHcFYosvdZj4 WQwYxsI2vBDigkGrNSm7qDdkt8B8yE2nN9Ls5xy6sN60h9DVXXcimrEuY56IFyF1Z6sl2h hdvz/sSsOvO+WZVtmjaiftDTmwqDW8V5CnP0DjAIiEGmEKNCeL2vV05BVoccIrkOBK83cq qQvif5LnJrqOOBbDx0Q4xV/3hXFMnNpL5NbiqtxPCFbFRnCHCuHthwsGCFdgeUkg== Received: by recvd-6b556c7f5c-jjv9p with SMTP id recvd-6b556c7f5c-jjv9p-1-66C6EED2-11 2024-08-22 07:54:58.838066589 +0000 UTC m=+34750.471306909 Received: from herokuapp.com (unknown) by geopod-ismtpd-39 (SG) with ESMTP id f19bTQS9SaeXTlS4yQLYmA for ; Thu, 22 Aug 2024 07:54:58.767 +0000 (UTC) Date: Thu, 22 Aug 2024 07:54:58 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 20682 X-Redmine-Issue-Author: ono-max X-Redmine-Issue-Priority: Normal X-Redmine-Sender: ono-max X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 95553 X-SG-EID: =?us-ascii?Q?u001=2EZDJU3MvRcQoADZu0w21uLDEuPIX6K8yXc=2F4ZLkljuGHdekcm0ImiA7H+T?= =?us-ascii?Q?OgVabcqI8RNFFndJh625xL8LO=2FhTnAqeELREWBa?= =?us-ascii?Q?P7pnP2yviloT6ciP12UG2mCee=2FrBbZ8v4iKLvTg?= =?us-ascii?Q?fd7nOmKNj9nZpJuQJWyhhFSQrc3BT50C5k=2F2Wad?= =?us-ascii?Q?FFKN6TtSVXj+kuT8u3=2FWsWEb8YOgPK1TjyTVUnd?= =?us-ascii?Q?JXuZTh4nYu7aQ6wxV9bKg0mTUCkgRzt78i9kzcL?= =?us-ascii?Q?RgHG?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: IOTXKK6Y55EPPNGVYPNWQZIMSH2MXPSZ X-Message-ID-Hash: IOTXKK6Y55EPPNGVYPNWQZIMSH2MXPSZ X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.9 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:118917] [Ruby master Bug#20682] Slave PTY output is lost after a child process exits in macOS List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "ono-max (Naoto Ono) via ruby-core" Cc: "ono-max (Naoto Ono)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #20682 has been updated by ono-max (Naoto Ono). @mame and I have noticed that there is a problem in the workaround solution: https://github.com/ruby/ruby/pull/11404 This program hangs in `Process.wait(pid)`. ```ruby require 'pty' r, _, pid = PTY.spawn('ruby', '-e', 'puts "a"; puts "b"') Process.waitpid(pid) ``` On the other hand, this program does not hang and exits successfully. ```ruby require 'pty' r, _, pid = PTY.spawn('ruby', '-e', 'puts "a"; puts "b"') puts r.gets #=> "a\n" Process.waitpid(pid) ``` >>From the above result, at least a user who uses PTY has to read from the IO at least once. ---------------------------------------- Bug #20682: Slave PTY output is lost after a child process exits in macOS https://bugs.ruby-lang.org/issues/20682#change-109487 * Author: ono-max (Naoto Ono) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- According to Launchable, the following PTY tests are flaky only on macOS. https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_without_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_spawn_with_block https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_commandline https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-paths/file%3Dtest%2Ftest_pty.rb%23%23%23class%3DTestPTY%23%23%23testcase%3Dtest_argv0 It's because the slave PTY output is lost after a child process exits in macOS. Here is the code to reproduce the problem. When I remove `sleep 3` from the code, "a" is returned. ``` require 'pty' r, w, pid = PTY.spawn('ruby', '-e', 'puts "a"') sleep 3 puts r.gets #=> Returns nil ``` Based on my investigation, this issue happens in the macOS side and it's almost same as https://github.com/pexpect/pexpect/issues/662. The cause is described as follows in the ticket: > // NOTE[macOS-S_CTTYREF]: On macOS, after a forkpty(), if the pty slave (child) // is closed before the pty master (parent) reads, the pty's buffer is cleared // thus the master (parent) reads nothing. This can happen if the child exits // before the parent has a chance to call master.read(). // // This issue has been reported to Apple, but has not been resolved: // https://developer.apple.com/forums/thread/663632 // // Work around this issue by opening /dev/tty then closing it. This ultimately // causes the child's exit() to flush the slave pty's output buffer in a // blocking way. This fixes the problem on macOS 13.2 in my testing. // // Here's how the workaround works in detail: // // If we open /dev/tty, it sets the S_CTTYREF flag on the process. This flag // remains set if we close the /dev/tty file descriptor. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L128 // Additionally, opening /dev/tty retains a reference to the pty slave. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty_tty.c#L147 // // When the child process exits: // // 1. All open file descriptors (including stdin/stdout/stderr which are the pty // slave) are closed. This does *not* drain unread pty slave output. // * If S_CTTYREF was set, closing the file descriptors does not close the // last reference to the pty slave, so no cleanup happens yet. // * NOTE[macOS-pty-close-loss]: If S_CTTYREF was not set, closing the file // descriptors drops the last reference to the pty slave. Unread data is // dropped. // // 2. If the S_CTTYREF flag is set on the child process, the controlling // terminal (pty slave) is closed. XNU's ptsclose() ultimately calls // ttywait(). // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/kern_exit.c#L2272 // * ttywait() is the same as ioctl(slave, TIOCDRAIN); it blocks waiting for // output to be received. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5e5ce20a938e6554e5/bsd/kern/tty.c#L1129-L1130 // * NOTE[macOS-pty-waitpid-hang]: Because of the blocking ttywait(), the // process is in an exiting (but not zombie) state. waitpid() will hang. // // * NOTE[macOS-pty-close-loss]: If the S_CTTYREF flag is not set on the // child process, ttywait() is not called, thus the pty slave does not // block waiting for the output to be received, and the output is dropped. // A well-behaving parent will use a poll() loop anyway, so this isn't a // problem. (It does make quick tests annoying to write though.) // // Demonstration of NOTE[macOS-pty-close-loss] (S_CTTYREF is not set before // exit): // // // On macOS, this program should report 'data = ""', demonstrating that // // writes are lost. // // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // // Cause the child to write() then exit(). exit() will drop written // // data. // sleep(1); // // char buffer[10]; // ssize_t rc = read(tty_fd, buffer, sizeof(buffer)); // if (rc < 0) { perror("read"); abort(); } // fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // } // // return 0; // } // // Demonstration of NOTE[macOS-pty-waitpid-hang] (S_CTTYREF is set before exit): // // // On macOS, this program should hang, demonstrating that the child // // process doesn't finish exiting. // // // // During the hang, observe that the child is in an exiting state ("E"): // // // // $ ps -e -o pid,stat | grep 20125 // // 20125 ?Es // // #include // #include // #include // #include // #include // #include // #include // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // close(open("/dev/tty", O_WRONLY)); // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // fprintf(stderr, "child PID: %d\n", pid); // // // This will hang because, despite the child being is an exiting // // state, the child is waiting for us to read(). // pid_t rc = waitpid(pid, NULL, 0); // if (rc < 0) { perror("waitpid"); abort(); } // } // // return 0; // } In Ruby, PTY is implemented with [fork()](https://github.com/ruby/ruby/blob/master/process.c#L1706) and [posix_openpt()](https://github.com/ruby/ruby/blob/master/ext/pty/pty.c#L329) in macOS. I could reproduce the problem in the following script. ``` #include #include #include #include #include int main() { int master_fd, slave_fd; pid_t child_pid; char *slave_name; // Open a master pseudo-terminal master_fd = posix_openpt(O_RDWR | O_NOCTTY); if (master_fd == -1) { perror("posix_openpt"); exit(1); } // Grant access to the slave pseudo-terminal if (grantpt(master_fd) == -1) { perror("grantpt"); exit(1); } // Unlock the slave pseudo-terminal if (unlockpt(master_fd) == -1) { perror("unlockpt"); exit(1); } // Get the name of the slave pseudo-terminal slave_name = ptsname(master_fd); if (slave_name == NULL) { perror("ptsname"); exit(1); } // Fork a child process child_pid = fork(); if (child_pid == -1) { perror("fork"); exit(1); } else if (child_pid == 0) { // Child process // Open the slave pseudo-terminal slave_fd = open(slave_name, O_RDWR); if (slave_fd == -1) { perror("open"); exit(1); } // Create a new session and process group if (setsid() == -1) { perror("setsid"); exit(1); } // Set the controlling terminal for the child process if (ioctl(slave_fd, TIOCSCTTY, NULL) == -1) { perror("ioctl"); exit(1); } // Duplicate the slave file descriptor to stdin, stdout, and stderr if (dup2(slave_fd, STDIN_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDOUT_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDERR_FILENO) == -1) { perror("dup2"); exit(1); } // close(open("/dev/tty", O_WRONLY)); // Close the original slave file descriptor close(slave_fd); // Execute a shell or other program (void)write(STDOUT_FILENO, "y", 1); exit(1); } else { sleep(5); char buffer[10]; ssize_t rc = read(master_fd, buffer, sizeof(buffer)); if (rc < 0) { perror("read"); abort(); } fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // Clean up close(master_fd); } return 0; } ``` -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/