From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <zsh-workers-return-40886-mason-zsh=primenet.com.au@zsh.org>
Received: (qmail 5681 invoked by alias); 23 Mar 2017 10:17:37 -0000
Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm
Precedence: bulk
X-No-Archive: yes
List-Id: Zsh Workers List <zsh-workers.zsh.org>
List-Post: <mailto:zsh-workers@zsh.org>
List-Help: <mailto:zsh-workers-help@zsh.org>
X-Seq: 40886
Received: (qmail 9893 invoked from network); 23 Mar 2017 10:17:37 -0000
X-Qmail-Scanner-Diagnostics: from aok120.rev.netart.pl by f.primenet.com.au (envelope-from <psprint@zdharma.org>, uid 7791) with qmail-scanner-2.11 
 (clamdscan: 0.99.2/21882. spamassassin: 3.4.1.  
 Clear:RC:0(85.128.245.120):SA:0(0.0/5.0):. 
 Processed in 2.479379 secs); 23 Mar 2017 10:17:37 -0000
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on f.primenet.com.au
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=unavailable
	autolearn_force=no version=3.4.1
X-Envelope-From: psprint@zdharma.org
X-Qmail-Scanner-Mime-Attachments: |
X-Qmail-Scanner-Zip-Files: |
Received-SPF: none (ns1.primenet.com.au: domain at zdharma.org does not designate permitted sender hosts)
X-Virus-Scanned: by amavisd-new using ClamAV (7)
Date: Thu, 23 Mar 2017 11:17:26 +0100
From: Sebastian Gniazdowski <psprint@zdharma.org>
To: zsh-workers@zsh.org
Message-ID: <etPan.58d3a0b6.19495cff.10ab3@MacMini.local>
In-Reply-To: <etPan.58d39baa.74b0dc51.10ab3@MacMini.local>
References: <1490076282.321781.918060736.22D2C432@webmail.messagingengine.com>
 <170322205304.ZM5853@torch.brasslantern.com>
 <170322205304.ZM5853@torch.brasslantern.com>
 <etPan.58d39baa.74b0dc51.10ab3@MacMini.local>
Subject: Re: An idea for fast "last-N-lines" read
X-Mailer: Airmail (231)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

W=C5=82=C4=85cz 23 marca 2017 at 05:05:33, Bart Schaefer (schaefer=40bras=
slantern.com) napisano:=C2=A0
> This could, however, be made a lot better, e.g. by introducing a cache=C2=
=A0
> of mapped files into mapfile.c and causing get=5Fcontents() to first us=
e=C2=A0
> the cache (and setpmmapfile to update it, unsetpmmapfile to erase an=C2=
=A0
> entry from it) before resorting to remapping the actual file.=C2=A0

I think this should be done (I might get to it too at some point). As for=
 the use case =22last-N-lines=22, one could do =24=7B=24=7Bmapfile=5Bname=
=5D=7D=5B-250000,-1=5D=7D=7D, providing that average line length would be=
 computed to be 250.. So the 250000 is expected to hold needed 1000 lines=
. Tried this on some log that I aggregate:=C2=A0

% lines=3D(); typeset -=46 SECONDS=3D0; lines=3D( =22=24=7B(=40f)=24=7B=24=
=7Bmapfile=5Binput.db=5D=7D=5B-250000,-1=5D=7D=7D=22 ); echo =24SECONDS, =
=24=7B=23lines=7D=C2=A0
0.3925410000, 1536=C2=A0

Doing this in traditional way:=C2=A0

% lines=3D(); typeset -=46 SECONDS=3D0; lines=3D( =24=7B=22=24=7B(=40f)=22=
=24(<input.db)=22=7D=22=5B-1000,-1=5D=7D ); echo =24SECONDS, =24=7B=23lin=
es=7D=C2=A0
0.8828770000, 1000=C2=A0

Without slicing:=C2=A0

% lines=3D(); typeset -=46 SECONDS=3D0; lines=3D( =22=24=7B(=40f)=22=24(<=
input.db)=22=7D=22 ); echo =24SECONDS, =24=7B=23lines=7D=C2=A0
0.8219170000, 38707=C2=A0

I would expect mapfile to perform little better. =46or input file with 12=
500 lines, it's 0.1625320000, 1151. So the time quite raises from 162 ms =
to 400 ms when input file is larger. This could be just constant. It look=
s like =5B-250000,-1=5D does typical multibyte string iteration over buff=
er, to establish offset. If one would just look for last 250000 bytes and=
 do this at mapfile level, it would be constant. Not sure if unicode can =
be fully broken for whole buffer this way, telling from the way how Zsh h=
andles unicode, it would be just few first characters that could be broke=
n. =46or looking for last-N-lines, I wonder how should be '=5Cn' handled =
in reverse-read unicode buffer.=C2=A0

> =7D BTW, (=40f) skips trailing =5Cn=5Cn... That's quite problematic and=
 there's=C2=A0
> =7D probably no workaround=3F=C2=A0
>=C2=A0
> In double quotes, (=40f) retains empty elements, which includes making=C2=
=A0
> empty elements out of trailing newlines. However, there is no way to=C2=
=A0
> get =24(<file) =5Bor =24(<<<string) etc.=5D to retain trailing newlines=
, which=C2=A0
> is most likely what's misleading you. =C2=A0

Ah, thanks. Wonder how would sysread perform, and what about metafication=
 when using it.=C2=A0

--=C2=A0
Sebastian Gniazdowski=C2=A0
psprint =5Bat=5D zdharma.org