From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 30905 invoked from network); 30 Nov 2021 08:19:52 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 30 Nov 2021 08:19:52 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1638260392; b=lS9B5gVPlpU2BOk3zHY9abtf/AZnFIM3T/0abkUrSV/cjZdJMllNEVKIO0+dDU6PdRu7vSQXSf SMm8CSzUFXAfd+aKrWmSRK2fzX2vSjI/y1PfG972htEM51WMT1OlB87oYlecs4Ow5FlDZnLu3g Xsko/STCutzWohMNJiphjzVPZ+jnyE5HN7PdSpH35keE/fu7k7UwuqZwUk62KLOd9C+WJxD8Ns WIdVuDcXqR1DaX5C74mk6oNAycGCxEd+H5TABtN+XaNIyoL4RKDBM5YxPVcykMzcbdLn3/slCs gJ2XdTFzV59awWm5TSeqzHraTYIQB5FPeCZXGPXQB4FUBQ==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mail-wm1-f43.google.com) smtp.remote-ip=209.85.128.43; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1638260392; bh=sXfkeOnKf9RhtNxubDeSWg9AcmbgMJcYx0QPL2yovpg=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:DKIM-Signature:DKIM-Signature; b=ZfV2XuS6bEhMwGtQcZZNoXiqlfktjMPCSWaEN1/OTcdAVqmkWFoFJg2G72YWyMkrT6e6z/DY2P bDrVYejhtp95JJRLuWCJC2FKpD7gyPFu/9Z132kPq3W1NXQyiBxGWyDj59E1+duH40i+HtCXmB uWx3PO3tnYCoIKBOZIkVW28VCZgdQlV3j8I3vY1Zf7YHy4ppzOl2MdC7d4LgoxvniyUA8247tF RrjB5Mq13SXcRHfF770lvlPuoUFEpyAx4eQpKnqRgnkJCSSxPc23SDJm7Lzng5fkzV2SXmaABB QT07s3QeCEYyJjKDMUqgpV0f6yh5cs0JVCmyFq6PX0ZVDw==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Type:Cc:To:Subject:Message-ID :Date:From:In-Reply-To:References:MIME-Version:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=1V3Wl7jk2KfFQOPOIVaZu2JgnljlLuushtDz1ox7H4g=; b=QJEQHPFEAUNnQuT02wiwNc8zVx SpQD5C3XMS9vlcjmKgV23rbsjcI+dbMN9IhOL55J/TWqHiPbGsKGKP/MM1IuRoZjXujXxTQWHo6oV K4whz8522ZkXpMol2cqsNFdorzFjB4lS0gKEGsCJvde47iSt/SuHJh05UjC6GLABlDpBPU1eQ5f5W pNg5Ebw7b1yLRB8+tl3Q7fp3Oz3CuXMzkPFaJ6Qa2IogEEhINq2jLuhCiwb6cPuwBfhba3QevLCS0 SXiT8paB8/Hxp5n4KI8I2ULv2YVF5CLZL+exVqpNewTDGbjNPNLUhoH0kAA4Jey/AQfjP2Srgn4cn Y+rA4BNg==; Received: from authenticated user by zero.zsh.org with local id 1mryMg-000D4T-L6; Tue, 30 Nov 2021 08:19:50 +0000 Authentication-Results: zsh.org; iprev=pass (mail-wm1-f43.google.com) smtp.remote-ip=209.85.128.43; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none Received: from mail-wm1-f43.google.com ([209.85.128.43]:40833) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) id 1mryM0-000CLF-TS; Tue, 30 Nov 2021 08:19:09 +0000 Received: by mail-wm1-f43.google.com with SMTP id j140-20020a1c2392000000b003399ae48f58so18824065wmj.5 for ; Tue, 30 Nov 2021 00:19:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1V3Wl7jk2KfFQOPOIVaZu2JgnljlLuushtDz1ox7H4g=; b=NoCPc55/l2uwAd7ML2tNtOq3CA5sa5vFpToeR3K9GyatD48SQqKoI8nXmtI58k2qaw u5xae7XUR9RmfLOzbrNimWJbJ9nmZX9YgM7cqay3+iNEzrG4wcIibURSNB8DyqlHhIBe 5Ii4/wyHBSg79y/I/QZHm8QeNiwI/ZgaZbML+dCwS1vb/KZI2v6F0EmQ3cYTQxjfAe8C 0PMsB1gFzWFR23fLiLm1nIABWGSgjvVDacf6v375lfbgbAdmacPaOu+tqebiOwuSkda0 bN4Wxg0dk7D8JXM2yJMak6Ym+xTOeswqkTqykuL0Y1oL7bCeZ8uSgJT7aYT5nDWErMuj SZjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1V3Wl7jk2KfFQOPOIVaZu2JgnljlLuushtDz1ox7H4g=; b=qFv6jwMy0l7tDWXiao5F3Vb6piT+qvL2xrrbFx8LKWIaQQ2V7aPAREhgTwHx48UEDx XBaJ1P61BeHWUJHK6+Pyuu5Uz96oo3404bp00EZTCjNwxC5Giw1pLaIvb+btypUGA910 YjwpMEa0s469ERB4eDjyXllUAxh2resDFx7bu5RCbWGyYMUQQAN/8QriglLWQR5v9kmv +i+oUGwEKiMDBZ4xKgprurRA9+5m2MqgEG8iKm2kEmQFEnl8qi6VFfSxnqJ1SWE0hI+S 9gBZzAhrsfmxZlyY8SMZfRqjI2ox4XU5BNgZKap11ge6Zr5fP4QDNLxHolgnC1QhkNC/ aR/Q== X-Gm-Message-State: AOAM530ODj5cVqWKAuJyInujjOKRV/ggMib1r3QdAQ5yTWBh5yZl1/Bl kg0FUY7QYkFrOIgj40L5sCNgtYM1ewFBKZ9HY8cx91smyFHaJA== X-Google-Smtp-Source: ABdhPJyc11mrLODPIh5rUF6J+ZuWwMMrUTQN8J1daTNF6HTa0SA5/KVSSTOujgUhl2ut/RLpXXd+vxME286PDWA7PhU= X-Received: by 2002:a7b:c256:: with SMTP id b22mr3189542wmj.176.1638260348393; Tue, 30 Nov 2021 00:19:08 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Zach Riggle Date: Tue, 30 Nov 2021 02:18:57 -0600 Message-ID: Subject: Re: Shell startup, aliases vs. functions vs. autoloadable functions, and zcompile benefits To: Bart Schaefer Cc: Zsh Users Content-Type: multipart/alternative; boundary="0000000000001d9bfe05d1fd361e" X-Seq: 27396 Archived-At: X-Loop: zsh-users@zsh.org Errors-To: zsh-users-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-users-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: --0000000000001d9bfe05d1fd361e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > n=3D($^fpath(e^'n=3D($REPLY/*(N.)); reply=3D("$#n $REPLY")'^)) > print -l ${${(On)n}[1,3]} And this continues to demonstrate that Zsh is the only language that the more I learn, the less readable my code becomes. I really do appreciate you demonstrating the most-Zsh way to achieve the desired result. There really ought to be an explainshell.com equivalent for Zsh expressions / expansion / modifiers / etc. The information is already nicely codified in Zsh autocompletion (e.g. ${( ), it would be nice to feed in expressions like the above and get a sane explanation. I actually intend to use this goal for a babys-first-Rust project, so we'll see how far along I get. The MVP of the project is to take a glob / path expansion expression (e.g. foo/**/bar(^/.) ) and convert it into a BSD find expression. >> for d in $fpath; do n=3D$(ls $d/* | wc -l); echo "$n $d"; done | sort -n= r | head -3 > Good heavens, so many processes and pipes. Pipes are nicely composable, and maintainable by others not intimately familiar with Zsh section 14. The Unix philosophy still applies -- do one thing and do it well. Shells are good at connecting inputs and outputs and modifying them. Sure, $#fpath + 4 processes is heavy vs. an array construct and a builtin -- but I don't think anybody is writing shell scripts for performance. This is more apparent when using e.g. the Rust package 'fd' via `fd --type f .` instead of extglob with **/*(.). 8.2 seconds vs 328 seconds (sample size is 1,641,649 files). The only trade-off is that "fd" does not guarantee any ordering. Skipping fd's internal sorting flags, and piping directly to sort(1) gives a runtime of 30 seconds -- still ten times faster than extglob (which is close to the number of CPU cores I have). $ hyperfine --runs 2 "zsh -il -c 'echo **/*(.)'" "zsh -il -c 'fd --type f .'" Benchmark 1: zsh -il -c 'echo **/*(.)' Time (mean =C2=B1 =CF=83): 328.973 s =C2=B1 2.153 s [User: 198.74= 6 s, System: 86.629 s] Range (min =E2=80=A6 max): 327.451 s =E2=80=A6 330.496 s 2 runs Benchmark 2: zsh -il -c 'fd --type f .' Time (mean =C2=B1 =CF=83): 8.281 s =C2=B1 0.703 s [User: 17.441 = s, System: 47.829 s] Range (min =E2=80=A6 max): 7.784 s =E2=80=A6 8.778 s 2 runs Shells ultimately exist to spawn processes and create pipes. I'd wager that (A) below is more maintainable than (B). A. | sort | head -3 B. {$(${(On)n}[1,3] If there's a sufficient performance benefit to in-shell-process computation, I would love to see some standard library expansions of zsh to reimplement common GNU/BSD utilities as functions *Zach Riggle* On Mon, Nov 29, 2021 at 10:12 PM Bart Schaefer wrote: > On Mon, Nov 29, 2021 at 6:30 PM Zach Riggle wrote: > > > > I would expect that the md5sum of a file is reasonably fast, and could > be stored in the .zwc for sanity checking, instead of just the "newer tha= n" > check. > > To what are you comparing that checksum? It could tell you if the > .zwc file were corrupted, but not whether the file differs from all > the component files that were compiled into it. Even if you could > somehow tell they were different, that doesn't answer the question of > whether the .zwc contains newer versions of any of those functions. > The .zwc does contain a check that it matches the parser version of > the shell that's trying to read it. > > > I expect that I have more $fpath entries than usual, but the total > number of autoloadable functions is much more. > > That's exactly the point: You're unlikely to ever execute most of > those functions, so storing an autoload entry for them is much more > space-efficient (and startup-time faster) than actually parsing and > storing the function definitions themselves. > > > $ for d in $fpath; do n=3D$(ls $d/* | wc -l); echo "$n $d"; done | sort > -nr | head -3 > > Good heavens, so many processes and pipes. > > n=3D($^fpath(e^'n=3D($REPLY/*(N.)); reply=3D("$#n $REPLY")'^)) > print -l ${${(On)n}[1,3]} > --0000000000001d9bfe05d1fd361e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> n=3D($^fpath(e^'n=3D($REPLY/*(N.)); reply=3D(&quo= t;$#n $REPLY")'^))
> print -l ${${(On)n}[1,3]}

And this continues to demonstrate that Zsh is the on= ly language that the more I learn, the less readable my code becomes.=C2=A0= I really do appreciate you demonstrating the most-Zsh way to achieve the d= esired result.

There really ought to be an explainshell.com equivalent for Zsh expressions = / expansion / modifiers / etc.=C2=A0 The information is already nicely codi= fied in Zsh autocompletion (e.g. ${(<TAB> ), it would be nice to feed= in expressions like the above and get a sane explanation.

I actually intend to use this goal for a babys-first-Rust=C2=A0pro= ject, so we'll see how far along I get.=C2=A0 The MVP of the project is= to take a glob / path expansion expression (e.g. foo/**/bar(^/.) ) and con= vert it into a BSD find expression.

>> for d= in $fpath; do n=3D$(ls $d/* | wc -l); echo "$n $d"; done | sort = -nr | head -3
> Good heavens, so many processes and pipes.

Pipes are nicely composable, and maintainable by othe= rs not intimately familiar with Zsh section 14.=C2=A0 The Unix philosophy s= till applies -- do one thing and do it well.=C2=A0 Shells are good at conne= cting inputs and outputs and modifying them.

Sure,= $#fpath=C2=A0+ 4 processes is heavy vs. an array construct and a builtin -= - but I don't think anybody is writing shell scripts for performance.

This is more apparent when using e.g. the Rust pack= age 'fd' via `fd --type f .` instead of extglob with **/*(.).=C2=A0= 8.2 seconds vs 328 seconds (sample size is 1,641,649 files).=C2=A0 The onl= y trade-off is that "fd" does not guarantee any ordering.=C2=A0 S= kipping fd's internal sorting flags, and piping directly to sort(1) giv= es a runtime of 30 seconds -- still ten times faster than extglob (which is= close to the number of CPU cores I have).

$=C2=A0hyperfine --runs 2 "zsh -il -c 'echo **/*(.)'&quo= t; "zsh -il -c 'fd --type f .'"
Benchmark 1: zs= h -il -c 'echo **/*(.)'
=C2=A0 Time (mean =C2=B1 =CF=83): =C2=A0= =C2=A0 328.973 s =C2=B1 =C2=A02.153 s =C2=A0 =C2=A0[User: 198.746 s, Syste= m: 86.629 s]
=C2=A0 Range (min =E2=80=A6 max): =C2=A0 327.451 s =E2=80= =A6 330.496 s =C2=A0 =C2=A02 runs

Benchmark 2: zsh -il -c 'fd --= type f .'
=C2=A0 Time (mean =C2=B1 =CF=83): =C2=A0 =C2=A0 =C2=A08.= 281 s =C2=B1 =C2=A00.703 s =C2=A0 =C2=A0[User: 17.441 s, System: 47.829 s]<= br>
=C2=A0 Range (min =E2=80=A6 max): =C2=A0 =C2=A07.784 s =E2=80=A6 = =C2=A08.778 s =C2=A0 =C2=A02 runs

Shells ultimately exist to spawn processes and create pipes.=C2=A0 I'= d wager that (A) below is more maintainable than (B).

A. | sor= t | head -3
B. {$(${(On)n}[1,3]

If there's a sufficient performance benefit to in-shell-process compu= tation, I would love to see some standard library expansions of zsh to reim= plement common GNU/BSD utilities as functions

Zach Riggle<= /b>


On Mon, Nov 29, 2021 at 10:12 PM B= art Schaefer <schaefer@bras= slantern.com> wrote:
On Mon, Nov 29, 2021 at 6:30 PM Zach Riggle <zachriggle@gmail.com> wro= te:
>
> I would expect that the md5sum of a file is reasonably fast, and could= be stored in the .zwc for sanity checking, instead of just the "newer= than" check.

To what are you comparing that checksum?=C2=A0 It could tell you if the
.zwc file were corrupted, but not whether the file differs from all
the component files that were compiled into it.=C2=A0 Even if you could
somehow tell they were different, that doesn't answer the question of whether the .zwc contains newer versions of any of those functions.
The .zwc does contain a check that it matches the parser version of
the shell that's trying to read it.

> I expect that I have more $fpath entries than usual, but the total num= ber of autoloadable functions is much more.

That's exactly the point:=C2=A0 You're unlikely to ever execute mos= t of
those functions, so storing an autoload entry for them is much more
space-efficient (and startup-time faster) than actually parsing and
storing the function definitions themselves.

> $ for d in $fpath; do n=3D$(ls $d/* | wc -l); echo "$n $d"; = done | sort -nr | head -3

Good heavens, so many processes and pipes.

n=3D($^fpath(e^'n=3D($REPLY/*(N.)); reply=3D("$#n $REPLY")= 9;^))
print -l ${${(On)n}[1,3]}
--0000000000001d9bfe05d1fd361e--