Re: Using file lines as "input files"

[Dominik Vogt <dominik.vogt@gmx.de>]

On Fri, Jul 08, 2022 at 03:04:31PM -0700, Bart Schaefer wrote:
> On Fri, Jul 8, 2022 at 1:58 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> >
> > Disclaimer: I _know_ this can be done in seconds with perl /
> > python, but I like to not rely on scripting languages when the
> > shell can do the job.
>
> This is sort of like saying "I like to not rely on hiking boots when
> shoes can do the job."

Actually, for me, scripting languages are the "shoes" because they
don't interact very well with the command pipeline, unless you
spend an absurd amount of work to make them do so.  Calling
commands for everything can be slower, but most of the time it's
just a symptom of bad scripting.  GNU coreutils are faster than
anything I'll ever be willing to code (or any perl or python
script or C or C++ library for that matter).  The trick is keeping
the process spawning overhead low.

> >   $ chksum Fline1 Fline2 Fline3 ... Fline265000
> >
> > (Of course without actually splitting the input file
>
> If "not actually splitting" means what it seems to mean, and you
> literally want to run cksum, the answer is no.

Right.

This does the job pretty well, relying entirely on existing Unix
tools:

  ulimit -s 100000
  split -l 1 "$INPUTF" ff
  cksum ff*
  rm ff*

That cuts runtime down to seven seconds instead of four minutes,
at the cost of a fem hunred MB on the RAM disk.  Splitting the
source file and removing the fragments takes about three to four
seconds.

Thanks for the comments which put me on the right track.

--

(I prefer to have a huge stack size anyway to be able to do things
like "grep foobar **/*(.)".)

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt