From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 8963 invoked by alias); 6 Dec 2017 19:51:07 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: List-Unsubscribe: X-Seq: 42089 Received: (qmail 25197 invoked by uid 1010); 6 Dec 2017 19:51:07 -0000 X-Qmail-Scanner-Diagnostics: from mail-qt0-f195.google.com by f.primenet.com.au (envelope-from , uid 7791) with qmail-scanner-2.11 (clamdscan: 0.99.2/21882. spamassassin: 3.4.1. Clear:RC:0(209.85.216.195):SA:0(-1.9/5.0):. Processed in 1.309437 secs); 06 Dec 2017 19:51:07 -0000 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS,T_DKIM_INVALID autolearn=ham autolearn_force=no version=3.4.1 X-Envelope-From: schaefer@brasslantern.com X-Qmail-Scanner-Mime-Attachments: | X-Qmail-Scanner-Zip-Files: | DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brasslantern-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=BfeEq6qG6Z6ZmOM78YmDYg688xyF5K65nZJrl20fWyM=; b=SUfnPPH6n6MAmOIACDjVcKGlLTuB2Sf/uBz7WO1IISXRFc775EhJwMKbl9FNTKgK7W CAZjlsz9H3u9mQr9xLqy2+LGUzTIS7xuILvrDDCovZ1BqZUxGyhF6drKJ11tAyCn4By3 HA79vb7h2ndYao5IFgqvW8yKU63WKSdOq7wHQ/423AbHZIucnRUI08jaMGS12RbVg9Of H7qDelu5JZaBYJLLg1PrgCej0Qj/gEJfENtUjWT0m50eOGWzL2NSCaRWlBpiJgSltiX1 LJLTAN9FeAOmHyG5/tjinTQXVq/RtvKiYAbwU9NAE5yz93EndwZPf+Bfjqd5bOp6HH6V 5YrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=BfeEq6qG6Z6ZmOM78YmDYg688xyF5K65nZJrl20fWyM=; b=pmgKq/WzLvbXRcKaYxms13BzMl/6fqSWTZrRRVYMpdeCH/ByRHP0xx+veqzrwxeTnK eNNMK0o1AD75riMNDa3JUZPUBJE+Ndkcf4LNAzoWmSKblDqdfw0r4pYqkKy0cwbbvw1/ rOvypq2mff9f2ITH7ohK0pXkCTHOeoPuT+AQ3uWd292E/bE4xgEwjkOuyunfid6YhEkD HKZnJVtodIOvHOK9r7uK27dRYlNfhWkHkMPkOtbK1WzPRtj37/bdWD7TrNZMzr1NSinM IG7Gtx9AW9ExM0DGC/Gm/kg+Z7xXr7kvIC9wMHSynL0jXgV8FtKXiet+uqKL5FFw9Sax QCjA== X-Gm-Message-State: AKGB3mLtXL3Fb6Otx2nB1oGUIKmyJFUFZUNg5WL5PKAGYJ5P3nZG900u 6GJiQylIUX57yoPUs0bg5HoPbkN3JRkVD9E8Nbvfug== X-Google-Smtp-Source: AGs4zMY+qenFIBQkGo452DsBR7MRyKH4P0XqWui2PlnbLuCliGBA5dz6aOJ3tgrC9OHsPXBuQ8/JlFqaKYPpGhke5ss= X-Received: by 10.55.68.82 with SMTP id r79mr26039935qka.125.1512589862823; Wed, 06 Dec 2017 11:51:02 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20171206193122.w4r3dhkotlhdtrah@local> References: <20171206193122.w4r3dhkotlhdtrah@local> From: Bart Schaefer Date: Wed, 6 Dec 2017 11:51:02 -0800 Message-ID: Subject: Re: [BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile To: "zsh-workers@zsh.org" Content-Type: text/plain; charset="UTF-8" On Wed, Dec 6, 2017 at 11:31 AM, Zeshy wrote: > Hi all! > > In search of the reason why zsh (5.4.2 and fairly recent git) is > so much slower reading/writing larger history files (13 MB, 330k lines) > than bash (4.4.12), I think I found a bug. If I missed something (like > some option), and this is not a bug, but just misconfiguration, or if > you need more information, please let me know. This happens because of HISTSIZE and the various HIST_*DUP* options. The only way to do duplicate filtering is to examine the entire past history, and there's no setting of HISTSIZE that is interpreted as "unlimited" so rewriting to output only the most recent $HISTSIZE lines is always performed. It would probably be possible to optimize for the case of "$HISTSIZE is larger than the current number of lines and no duplicate removal is enabled" but right now there is just one rewriting algorithm that applies (or not) those conditions internal to the loop. So not technically a bug but not optimized for people who can "afford" the memory cost to have every shell preserve huge amounts of history.