From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 15202 invoked from network); 8 Jul 2022 23:19:26 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 8 Jul 2022 23:19:26 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1657322366; b=HQWfmEMlGqPcxrasmWI0JVFyJ5NrHCV0cgy3t9Z3YBc32lUpVpoulx2DeSlzZpPw7OEFVLlb4y T5go0nZ0jA/c9oDHwX7mQHtBoOeMfwQvrJy/YM5ZUeSNpt1qKVcx8YUX7qvQ6NifVKaczZvUoT zE1Exxgt53dwbZPEfs5TdcbBsMvObdY/PKetjWkBnbI0CNpAIUisgYz1H7sbdp1pjnc2yrfiFx viQ1EJG/R18SojJai6EjrjwMy/+AvQ1fRZwT5KYMpHsnkYzHfRH91OZDwv5g3h/qrAUwVCMn+q Swn12ZSLWm+Wx59E/gXtX0lI7eLiYgfYw38ZvlMaCcHpiw==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mout.gmx.net) smtp.remote-ip=212.227.15.18; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.a=rsa-sha256; dmarc=none header.from=gmx.de; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1657322366; bh=MGN/dhuJuN6no1GXl37FdwUpTwBY8b0qzv35RaN2kN4=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Transfer-Encoding:In-Reply-To:Content-Type: MIME-Version:References:Reply-To:Message-ID:Subject:To:From:Date: DKIM-Signature:DKIM-Signature; b=KJGEi6GTdtNHRmq2+tx5YWj4AUN2cPv44KB38aGdCmnaN8fGCbALwqeIznXWFszZWVibz9e8jm BK2r2H4idg8Pjz9YzxJ4ZDQsBzQxUM96G/xuE05JS6MABsgeIGNQIxVQ6QsxovkSoIUo8cawEb 8r9u+G6OPedf3FtodzMN44rG7jvAbwXtdsmbrZmL5DHzw/TYXuSCy5btX7ogOrYDj8xrojfp0I m/4P8m9zlYsw9KhaOfnKluduDDQ8DLKUVyEqnNUow0mWpmXL6a/VS2TAJQu8j2BvD8zG87I8kE 6kEiPa7I+BHgWnnuQZDV31uE1grHNqBscfmcn7Icl8IIDg==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Transfer-Encoding:In-Reply-To :Content-Type:MIME-Version:References:Reply-To:Message-ID:Subject:To:From: Date:Cc:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID; bh=pmPB03cGrb/pp1LnXYWWyG1M+blcff5Om5UvyLvlzl0=; b=lZHSSudDi+uRr4xaH2G/JRf2ww HX+vzdO7jry6MvjAkiRHdu8JFer/7iBmE16GNNC/FnPsMeIUXgirX4RruYL2gxYRZjLjAE34iwlcb gs2GxoiteaayYF9/8PDrYmt2ej2kjMEGJ0gFx8nT59f88MWCCnPUU3KEHFcIgNkBGtywVsm41dQZp QgUZRJMaVEJYHbm0KiqgTDaa1znp4mEhnDnzp9OSIaGCdS4WaUOT5NxFbNFf0Wkox2Ls6d48dzkbm PVEcD5WwJhpxzN13TY0tyu8K51jkO+LomF0qNdmqd5XQS9OOruE5wSYmwK/g0njDqu+t9NIknWwLJ YumOzskg==; Received: from authenticated user by zero.zsh.org with local id 1o9xFs-000L8K-Ro; Fri, 08 Jul 2022 23:19:24 +0000 Authentication-Results: zsh.org; iprev=pass (mout.gmx.net) smtp.remote-ip=212.227.15.18; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.a=rsa-sha256; dmarc=none header.from=gmx.de; arc=none Received: from mout.gmx.net ([212.227.15.18]:59229) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_256_GCM_SHA384:256) id 1o9xEY-000KML-O6; Fri, 08 Jul 2022 23:18:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1657322282; bh=MGN/dhuJuN6no1GXl37FdwUpTwBY8b0qzv35RaN2kN4=; h=X-UI-Sender-Class:Date:From:To:Subject:Reply-To:References: In-Reply-To; b=bEit8IgENaeYN3JbJCnMUFVOvliF6TdWELeEcXUxn0PP+msBlz/LMv4UulXG/vzVN jkJk0JKGJanPDoxTLeMICOEoVnjM3qWE5LIDj4oWW8TgbQ2aHXkGzecwWETgqUvBx/ 2GePeQWeyP2b8RDiEoCyhTFTwfEPC9IpQqtkN/Ys= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from zitrone.obstwiese ([46.223.163.206]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MfpOd-1nd4GB016X-00gHsb for ; Sat, 09 Jul 2022 01:18:02 +0200 Received: from luthien by zitrone.obstwiese with local (Exim 4.94.2) (envelope-from ) id 1o9xET-0002Os-VU; Sat, 09 Jul 2022 00:17:57 +0100 Date: Sat, 9 Jul 2022 00:17:57 +0100 From: Dominik Vogt To: Zsh Users Subject: Re: Using file lines as "input files" Message-ID: Reply-To: dominik.vogt@gmx.de Mail-Followup-To: Zsh Users References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Provags-ID: V03:K1:XSrtc+aAd2VQJT1pmEGICq/IZQYtgz1FuQXgZ8/B11yfpmpGR7V ZULWdlELbqADcVbNTXEqvullQs0wzyRuUG/XHqgBUntBK3OBGKsYMEF4cbgDsiFp9ULkYrF nmGDeEN1DBLoKigNdQkiBZVX48/Tc4qsL5f3fIM2gwTyMxBin/IKBBf/AQ4vTtnbf8ocl+w 3LFADD+Cwrwu55VwbS0Aw== X-UI-Out-Filterresults: notjunk:1;V03:K0:ydGsvLBjEWQ=:+kSyDXNd7aAZKAaV8+nQsn IVyxXe8U20TEtjovNSbxbR57kEWIjkhuMj8cCVroXerIc+pCx5XK+Y8N4OmOUGLahDXm6YiQy 4SFIulDh5yw8p1qLw41J30fHe9bDUV5aSZ1B8P2QNBdmUmw9frqywleabz+AUfqXnKqE2M+ce aKbh/fSLkyNfqjA9xzQacDeG/cb02RjqG+bSkDq+BlK0zznaNraYKjdGBcpjzMxgMyD64FXM/ eMst0ibr7uW7b0X6clNW8e5MPCePHN9uNL3LZabXKwQMk+pq17cKqSVH8IUP5JXuDENNPh8+4 Knrny7jwfRMoGM//xL8OSEV74YbeUTIe6K16FJyAcgFHBFdC6V2yEp1HoRHx7FyZZwXud4Rws eL/SSV7RfSPUSVsF/RS2svj21+nGrHKp2QaAMuaYSIAv90a0QNsiuKC2NMHG9gO4GKvq0DR0M M0O3sNcKBALMuua4KeBTBhw8zJQMM8z+rSksy/Rq2LuZaqSynqIsvquQsW+yYz+yRawJeVHmx teJ6heGPD/b4F1ulXH/zCjyXJK2DEdtX5y+2oDinY6RwUOGc4ZOxRMYR7GV8lFrahWjI7J39r 9EijtX1tn1H+ev/m2vd0EeGM+nkrhiqBs8m5SDENlR+gzzQW+xaVua9P+ZsDNzOfpjlHjk+R/ OZJwF8a1V263Nxf5wfKgy7y9DKOiPwjhQlp4roVXVRUD/PBgvSpK1yPBVk8kh8MyV+mwuQXgs 5KLzoR9k6iPDaR5RPL6TecRvjGqFjlSJomJGeroMOl0ORX+1woobKFDYUokVjwyG6JEK0BtDL JbDGKNhEf7d5UsNkoF/Ona2UocYphapuBDq4soEn0bQCnwdr0qcH0Kao1fyPjWfK6xjf7Y2nj VNV1I9bq9J+3INl7frswNd4HC9rFoOtOxH6r4o2GDzhGgBnnFc2WH3RCDlGiH1f8r0u8/233g h5afUMC1sRQui90wDjKX0gCK7rJSeLYvhtL6jmXlg3GRfzs8tuQw5/3E8A16jwMe5xM+h/Vrn ETbV+UR77CN6XjFnnZVesqfqE1NECjLmpVyhxlYU1nuitBfmNrcrzBFyGDF0Js4XObY/kfBf2 EaDO6yI+W9nvkKHzbcADKSixRV9Eetn/6cyUaZtkexLjhWfP9aqjjnpqw== Content-Transfer-Encoding: quoted-printable X-Seq: 27878 Archived-At: X-Loop: zsh-users@zsh.org Errors-To: zsh-users-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-users-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: On Fri, Jul 08, 2022 at 03:04:31PM -0700, Bart Schaefer wrote: > On Fri, Jul 8, 2022 at 1:58 PM Dominik Vogt wrote: > > > > Disclaimer: I _know_ this can be done in seconds with perl / > > python, but I like to not rely on scripting languages when the > > shell can do the job. > > This is sort of like saying "I like to not rely on hiking boots when > shoes can do the job." Actually, for me, scripting languages are the "shoes" because they don't interact very well with the command pipeline, unless you spend an absurd amount of work to make them do so. Calling commands for everything can be slower, but most of the time it's just a symptom of bad scripting. GNU coreutils are faster than anything I'll ever be willing to code (or any perl or python script or C or C++ library for that matter). The trick is keeping the process spawning overhead low. > > $ chksum Fline1 Fline2 Fline3 ... Fline265000 > > > > (Of course without actually splitting the input file > > If "not actually splitting" means what it seems to mean, and you > literally want to run cksum, the answer is no. Right. This does the job pretty well, relying entirely on existing Unix tools: ulimit -s 100000 split -l 1 "$INPUTF" ff cksum ff* rm ff* That cuts runtime down to seven seconds instead of four minutes, at the cost of a fem hunred MB on the RAM disk. Splitting the source file and removing the fragments takes about three to four seconds. Thanks for the comments which put me on the right track. =2D- (I prefer to have a huge stack size anyway to be able to do things like "grep foobar **/*(.)".) Ciao Dominik ^_^ ^_^ =2D- Dominik Vogt