From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS,URIBL_SBL_A autolearn=ham autolearn_force=no version=3.4.6 Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [94.130.110.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 09AB41F4CC for ; Tue, 24 Dec 2024 14:41:14 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=aOKvqwre; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=rrRokbGT; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1735051271; bh=3GqdZkmRAhT0o92n5RCCO0Hwkj4Qh5wEzU+NqC2cIm4=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=aOKvqwreHOnLcY963iOkAL4ZVdah97lZ6YwYHld+WVs0A9VKwxHwFaVdepJ/RC4DL g+kTBq0Y4uXic/X9lI5zi6zkFHRwI4+3B37zpPESndDLJN7RGX3ZJUxqyiPmLJofnX krACWIU2n+h8h3MUMNNpRHfP40en/29lyoIOQ+40= Received: from nue.mailmanlists.eu (localhost [IPv6:::1]) by nue.mailmanlists.eu (Postfix) with ESMTP id E8B2745F6D for ; Tue, 24 Dec 2024 14:41:11 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=rrRokbGT; dkim-atps=neutral Received: from s.wfbtzhsv.outbound-mail.sendgrid.net (s.wfbtzhsv.outbound-mail.sendgrid.net [159.183.224.104]) by nue.mailmanlists.eu (Postfix) with ESMTPS id 0233445F3E for ; Tue, 24 Dec 2024 14:41:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:from:subject:to; s=s1; bh=GqVewuIwXFK7fFI2ney2H0OJRAKC5xGzVKrpeeRGD9s=; b=rrRokbGT4AIIZzBbESWPjfpNqmjKw/qUtldNTrzgLklTMYsvjI2i6VHckMoPG/CYm7Vo NFOQnoEa1vUWD5+5kCTVs3YAY1sHc7y5/9KM4I6tW9/f44sQxq8vh1P/AFRryY2nBNJJOm zhDZyKMjlenHjyDBxTd8kLjSrRuCD9IKz9O/mxQ2QekYVh05AZ6KFXb+mGAe/FMBorQGGj C2atTllMAVREPCpqW7gfX4N4z37X0ITVfAYqz99lx/cEg9Qx9iit3+xE/sj66Hxhcrgl2Y e/kDOqkuJsJdN4XIq1K/GkXJMdn/Cl0zIhj8XU5S1Y25uCGWzquIYn/7itqO/y6g== Received: by recvd-84b546689d-rqv5n with SMTP id recvd-84b546689d-rqv5n-1-676AC802-1A 2024-12-24 14:41:06.734586287 +0000 UTC m=+3431840.408940876 Received: from herokuapp.com (unknown) by geopod-ismtpd-8 (SG) with ESMTP id e3M2Wi5UTSiwAcjkOFvQLQ for ; Tue, 24 Dec 2024 14:41:06.646 +0000 (UTC) Date: Tue, 24 Dec 2024 14:41:06 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Feature X-Redmine-Issue-Id: 20564 X-Redmine-Issue-Author: kddnewton X-Redmine-Issue-Priority: Normal X-Redmine-Sender: kddnewton X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 97073 X-SG-EID: =?us-ascii?Q?u001=2E41nWPij=2FD1kdGmpgWeHScbb9X8tZePZrQQx3bl8Vy2odBuMhLMhId2kIq?= =?us-ascii?Q?I4MmMJ43lNHfKPNNds=2Fb3JsTYpYoWyQb3+sPV4d?= =?us-ascii?Q?vscaSKwkBzVkTVdqg7tSmp5EIedtS+zsLmRJ8nG?= =?us-ascii?Q?L9jhNa=2FFua8jt1jHgfW0e2yeb4h492pQOxS9GqE?= =?us-ascii?Q?hbhqV1VMbkZ5MLoIOfcoIKLE1A73vdHSYEFG9fN?= =?us-ascii?Q?J+km=2F6Vx7AfFecTJ7SoWax=2F3Edd62+=2FHPRaInVF?= =?us-ascii?Q?excF7I5+r6Usq8s6ZY7oA4dpqw=3D=3D?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: CXWO2HMEW3JGRY73HMI64EQVJDDLKODU X-Message-ID-Hash: CXWO2HMEW3JGRY73HMI64EQVJDDLKODU X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.9 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:120396] [Ruby master Feature#20564] Switch default parser to Prism List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "kddnewton (Kevin Newton) via ruby-core" Cc: "kddnewton (Kevin Newton)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #20564 has been updated by kddnewton (Kevin Newton). While I no longer work at Shopify for my day job, I am still actively contributing to all of the Prism-related components and intend to continue doing so. In addition to me, Aaron and Eileen are both listed as maintainers, and Peter and Matt both contribute regularly. I am not worried about the continued development of Prism. What would you like to know about the strategy of the Prism gem? ---------------------------------------- Feature #20564: Switch default parser to Prism https://bugs.ruby-lang.org/issues/20564#change-111177 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- This issue is to propose switching the default parser for Ruby to Prism on the master branch. The main goal is to get this default into both master and the next preview so that the community has plenty of time to test before the eventual 3.4 release. Matz has indicated that the final decision about the default parser for Ruby 3.4 will be made later in the year. Decisions about the default parser for Ruby 3.5+ and the relationship between Prism, lrama, parse.y, etc. will also be made at a later time. ## Background For some background on the Prism project itself, here are a couple of links in chronological order that you can read: * https://bugs.ruby-lang.org/issues/19741 * https://railsatscale.com/2023-06-12-rewriting-the-ruby-parser/ * https://kddnewton.com/2024/01/23/prism.html * https://railsatscale.com/2024-04-16-prism-in-2024/ ## Status Prism now passes all Ruby tests and specs. Internally at Shopify, it also passes our Core monolith's CI, which is a large and complex Rails application. We are confident that it is ready for production use. External to CRuby, Prism has already been adopted extensively around the ecosystem, including but not limited to: * [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java) * [TruffleRuby](https://github.com/oracle/truffleruby/issues/3117) (via Java) * [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby) * [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM) * [repl_type_completor](https://github.com/ruby/repl_type_completor) * [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine) (via parser translator) * [ruby-lsp](https://github.com/Shopify/ruby-lsp) * [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator) Fortunately with the community adoption, we have been able to find many edge cases and bugs that have been fixed. Thank you so much to everyone who has tried, used, reported problems with, and improved Prism in the last couple of years. We would not have hit this milestone without this community involvement. ## Caveats While Prism correctly accepts and parses all valid Ruby syntax, there are still a number of cases where it does not reject invalid syntax as it should. We are currently working on these cases, and we are confident that we can eliminate them before the next preview release. You can find them listed here: https://github.com/ruby/prism/issues?q=is%3Aopen+is%3Aissue+label%3Ainvalid-syntax. This is our top priority right now, and I anticipate these being resolved quickly. The other caveat is unknown unknowns: bugs or syntax issues that we have yet to discover. At Shopify we are currently taking numerous steps to mitigate this risk, including: * Prism currently passes all Ruby tests and specs, and will continue to going forward. The tests and specs are extensive, and we are confident this gets us almost all of the way there. * Grammar-based fuzzing to ensure we handle all possible syntax variations (both valid and invalid). As an example, that recently led to the discovery that the current compiler was segfaulting on safe navigation in for loops (https://bugs.ruby-lang.org/issues/20468). It has also found several memory leaks that have been fixed. * Mutation testing of existing source files (by adding and deleting random bytes) to find additional edge cases. Our goal is 100% compatibility, and we believe this is achievable through the above techniques and others. The goal of this proposal is to allow more companies to test their codebase against Prism, and to make this easier. We believe that we can sort out the remaining problems much more quickly in that manner. ## Implementation The implementation will be relatively straightforward, as there is already the `--parser` flag that can be used to switch between the current parser and Prism. We will change the default to be `"prism"` instead of `"parse.y"`. There is no plan to remove or change any code related to `parse.y`, `lrama`, or `compile.c` at this time, as we want to create the minimal amount of change possible to ensure stability. -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/