From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 4965 invoked from network); 9 Jul 2022 02:22:37 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 9 Jul 2022 02:22:37 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1657333357; b=K7EXAMBCMnHhQbn0ckK3zKLuvG1LRh+KLAS043WGysoFyBMXUMFeq3zVRd6dTwHwKdTMu+YNdW t3tkC5BYU8wIzG8wB0aM2MmKBQhEjsV4IS29z797dNAgw5gPJhvZNuh34MCV7DnlCwRxBWBR8x 4iTWBw/Izxf94SjMAlu/9kTKPoKiTmKqgtu6TzvEFBCNTW184fxaFYb481quEoLG31FYtzJU5m 6xzEwCnFe0EtRZkVPYIpLd+2eJbrUkqmhIQ3mHIcIi2w7abKuXRbihdYU1s73vTMY2Ql8XQYh3 4XBQuvELb7NkLXRDljaIfVyIE7AUwCpOG5lD7HGLAEZjSw==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mail-pl1-f170.google.com) smtp.remote-ip=209.85.214.170; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1657333357; bh=wbBlQ6JCcNoH7elp6P5M3xLhvzjwPCQV6uDtV5Mi5KY=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Type:To:Subject:Message-ID:Date:From:References: In-Reply-To:MIME-Version:DKIM-Signature:DKIM-Signature; b=L0lJ3beLTdnKxki+PZ0zc5VY3BbYttzbSFPN5OLt1mF54UXvVep8wu5kEFnHqaUqq3MCASRFeb /tkiuV7MCQplNl+GTXi8dCLDMTT4zlFE4kTID4oB95kl9hlgqr3cxPzEjZCJK/zidBwZrJ+A3a rqUrmPvzCOpIsrtRGqXTAkt6d9Khn2KSruB1SfHVzoI/7Dy7ho/vevYkIc0YTxWcYTRRuEflZu vTEBz/Xpuabb/ljbC1M6cu30EH/SSu5zyZVe64m05yAwcehJ5sa5MbSGJEwYaWMd+m6QZHYZ6g 5UuHAGMFgWomqKJns9eYmV7Y6wU+IVANqLdH5VR0Bz82eQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Type:To:Subject:Message-ID: Date:From:References:In-Reply-To:MIME-Version:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=VL3E5P2ta9N7v+x1Sv9qjO81Ye9JbLWPAKbOIoPQTOM=; b=HlyWtysSCH5XeDXrGS24+z32zZ rOpCU9BCrq0c4z8rp1Hw4bpiaIcomKvXDcFKtzS0/RhgERWXhMm9ZJtqJ60mMS0ebCdatoTbjnMlJ gYvcoM/PSW3c8bVSTMQx+1iBtIWi6+kWNHvi2KedpcvEhPn4LgwoxXLch00xxwzMAAWRoLLcfTut1 wq3MPrG1OvqKOHQFCJ5KJntR+fbhiPdd+8gti4tS2QBia/Usbd/rTdh/Mosyx4YvKuqyLzXTTfOJS 7U/Tzk683PKubjHsln2ZbYBckXwE5G3YPiPMoNqopLUUlWq2oQ8XOUn2HVZPAxIXLc8pHtFfR3uJE lg408kNQ==; Received: from authenticated user by zero.zsh.org with local id 1oA07A-0000AW-5d; Sat, 09 Jul 2022 02:22:36 +0000 Authentication-Results: zsh.org; iprev=pass (mail-pl1-f170.google.com) smtp.remote-ip=209.85.214.170; dkim=pass header.d=gmail.com header.s=20210112 header.a=rsa-sha256; dmarc=pass header.from=gmail.com; arc=none Received: from mail-pl1-f170.google.com ([209.85.214.170]:34541) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) id 1oA06F-000PS3-Nr; Sat, 09 Jul 2022 02:21:40 +0000 Received: by mail-pl1-f170.google.com with SMTP id z1so258897plb.1 for ; Fri, 08 Jul 2022 19:21:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=VL3E5P2ta9N7v+x1Sv9qjO81Ye9JbLWPAKbOIoPQTOM=; b=bF/w8PaOmif6Xc2IYn26fpkuc4wPBnCi3bvU+iXFDtIJ4s4NC3A4SK2DrD/o6QkWro +fGaxHCHOB0ksL2f+W2mFEL9crgkJvBfCEpYPVpqOmjkRYEu6THWeXFLaczZjQNS5L4F 511//8BipUQBnN0a2Wl3FrVuFSkQaFnF0DFF+28FRg2moQLxW7e7PmL5zLC+uA6bY6Lp gqpa6va4rG2A+QzAhl0V3r6X5a+zxUTHMiMG4HAtQ2qI7OE7W9XBYT8ec+CYutVsKVBn 0T6ugcG/MtvhjnA0DAVHOlrAJBnX7VH95wldtxoo1ffuzHPm8YuQIE/jhk6MRsKR183+ 0TVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=VL3E5P2ta9N7v+x1Sv9qjO81Ye9JbLWPAKbOIoPQTOM=; b=CocIEXeddGfC91bf6vjkfQurWMbxQe30DIvP6wl4UIZXFeMi2tP/dQrKwGPPx4QHjk mI67lKESl1UWnAoS/uRK4OHrI85364ZnypMSzI/XGR4rIUPyGMXbAdYT/MW4BrgtXt9b ho112oibVpYn95Ciq8HE4ifu6OzbguNe3f4T+FYmZO9hYkILNYdZn8hi24j04tMiheZ9 n8zvCY/9IAbtzToqat03sjNqM9ZZ98xYUifKUHKJP0F8j+8oZLkUtEw3Youb4gmltiB0 +k9QnQdY/m5SEZcf2OPbGlS8JM5mmQJ1htCRLFI8++jBFhF7s/47+p5L/ik9Yhd05SkK ZQWA== X-Gm-Message-State: AJIora97yEq3OB4DUcmy3i3v2/Owdhezsa1+EQ4tLqBbSrt8+QLBYMGm G7L+SE0kYCkKFSHC0PUscYI1fKcbyapcyUbQN4I= X-Google-Smtp-Source: AGRyM1sQIQqgVy7Wj7/DSykTVQ9SwBeZy9P9taRz9NSoCYBm5zkvTiHeCdfo5eKExTyKU2+/MjBWNCy16yq0Y2gzARw= X-Received: by 2002:a17:90b:1650:b0:1ec:b5e7:42ae with SMTP id il16-20020a17090b165000b001ecb5e742aemr3259846pjb.15.1657333298233; Fri, 08 Jul 2022 19:21:38 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a05:6a11:af24:0:0:0:0 with HTTP; Fri, 8 Jul 2022 19:21:37 -0700 (PDT) In-Reply-To: References: From: Mikael Magnusson Date: Sat, 9 Jul 2022 04:21:37 +0200 Message-ID: Subject: Re: Using file lines as "input files" To: dominik.vogt@gmx.de, Zsh Users Content-Type: text/plain; charset="UTF-8" X-Seq: 27879 Archived-At: X-Loop: zsh-users@zsh.org Errors-To: zsh-users-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-users-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: On 7/9/22, Dominik Vogt wrote: > On Fri, Jul 08, 2022 at 03:04:31PM -0700, Bart Schaefer wrote: >> On Fri, Jul 8, 2022 at 1:58 PM Dominik Vogt wrote: >> > >> > Disclaimer: I _know_ this can be done in seconds with perl / >> > python, but I like to not rely on scripting languages when the >> > shell can do the job. >> >> This is sort of like saying "I like to not rely on hiking boots when >> shoes can do the job." > > Actually, for me, scripting languages are the "shoes" because they > don't interact very well with the command pipeline, unless you > spend an absurd amount of work to make them do so. Calling > commands for everything can be slower, but most of the time it's > just a symptom of bad scripting. GNU coreutils are faster than > anything I'll ever be willing to code (or any perl or python > script or C or C++ library for that matter). The trick is keeping > the process spawning overhead low. > >> > $ chksum Fline1 Fline2 Fline3 ... Fline265000 >> > >> > (Of course without actually splitting the input file >> >> If "not actually splitting" means what it seems to mean, and you >> literally want to run cksum, the answer is no. > > Right. > > This does the job pretty well, relying entirely on existing Unix > tools: > > ulimit -s 100000 > split -l 1 "$INPUTF" ff > cksum ff* > rm ff* > > That cuts runtime down to seven seconds instead of four minutes, > at the cost of a fem hunred MB on the RAM disk. Splitting the > source file and removing the fragments takes about three to four > seconds. > > Thanks for the comments which put me on the right track. > > -- > > (I prefer to have a huge stack size anyway to be able to do things > like "grep foobar **/*(.)".) I realized I misinterpreted the question originally, and the following doesn't seem to work 100% but it was a fun idea: % mkfifo apipe % foo[265000]='' # number of lines in the file % cksum apipe$^foo # pass "apipe" to cksum 265000 times (in another terminal or job control etc) % while read; do echo $REPLY > apipe; done < infile When I tried the above on some test data, I got about 10 broken pipes. Also several lines sometimes get passed through the pipe without an intervening EOF, I'll admit I don't know the finer points of pipe/fifo behavior when you open and close them rapidly. That said, this also seems to take around 4-5 seconds to run. -- Mikael Magnusson