Append newline to many files

Append newline to many files

[Dominik Vogt]

Assume there are thousands of text files that are not terminated
with a newline.  I want to concatenate them all, but add the
missing newline between files.

This works but takes ten times as much time as "cat foo.*".

  for i in foo.*; do cat "$i"; echo; done > out

I can't really think of a fast yet simple solution.

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Append newline to many files

[Roman Perepelitsa]

On Sun, Sep 25, 2022 at 3:21 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> Assume there are thousands of text files that are not terminated
> with a newline.  I want to concatenate them all, but add the
> missing newline between files.
>
> This works but takes ten times as much time as "cat foo.*".
>
>   for i in foo.*; do cat "$i"; echo; done > out
>
> I can't really think of a fast yet simple solution.

This should work:

    print >lf
    files=(foo.*)
    lf=(lf)
    cat -- ${files:^^lf}
    rm lf

Roman.

Re: Append newline to many files

[zzapper]

On 25/09/2022 14:30, Roman Perepelitsa wrote:
> On Sun, Sep 25, 2022 at 3:21 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>> Assume there are thousands of text files that are not terminated
>> with a newline.  I want to concatenate them all, but add the
>> missing newline between files.
>>
>> This works but takes ten times as much time as "cat foo.*".
>>
>>    for i in foo.*; do cat "$i"; echo; done > out
>>
>> I can't really think of a fast yet simple solution.
> This should work:
>
>      print >lf
>      files=(foo.*)
>      lf=(lf)
>      cat -- ${files:^^lf}
>      rm lf
>
> Roman.
>
This looks interesting but any chance of an explanation of how it works 
for us lesser mortals please?

zzapper

Re: Append newline to many files

[Dominik Vogt]

On Mon, Sep 26, 2022 at 09:11:04AM +0100, zzapper wrote:
>
> On 25/09/2022 14:30, Roman Perepelitsa wrote:
> > On Sun, Sep 25, 2022 at 3:21 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> > > Assume there are thousands of text files that are not terminated
> > > with a newline.  I want to concatenate them all, but add the
> > > missing newline between files.
> > >
> > > This works but takes ten times as much time as "cat foo.*".
> > >
> > >    for i in foo.*; do cat "$i"; echo; done > out
> > >
> > > I can't really think of a fast yet simple solution.
> > This should work:
> >
> >      print >lf
> >      files=(foo.*)
> >      lf=(lf)
> >      cat -- ${files:^^lf}
> >      rm lf
> >
> This looks interesting but any chance of an explanation of how it works for
> us lesser mortals please?

It first generates a file "lf" that contains only a newline and
stores the filename in an array "lf", and the names of the files
in question in an array "files".

"${files:^^lf}" merges the two arrays in an interleaving fashion.
The result is file.1 lf file.2 lf file.3 lf ... (The shorter array
is repeated up to the other one's length).

Quite clever solution, but a bit complicated.

--

This one works too and is simple and fast:

  $ sed -e '$s/$/\n/' -e <somereplacement> foo.*
           ^^^
            |
       Replacement is done only on the last line

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Append newline to many files

[Mikael Magnusson]

On 9/26/22, Dominik Vogt <dominik.vogt@gmx.de> wrote:
> On Mon, Sep 26, 2022 at 09:11:04AM +0100, zzapper wrote:
>>
>> On 25/09/2022 14:30, Roman Perepelitsa wrote:
>> > On Sun, Sep 25, 2022 at 3:21 PM Dominik Vogt <dominik.vogt@gmx.de>
>> > wrote:
>> > > Assume there are thousands of text files that are not terminated
>> > > with a newline.  I want to concatenate them all, but add the
>> > > missing newline between files.
>> > >
>> > > This works but takes ten times as much time as "cat foo.*".
>> > >
>> > >    for i in foo.*; do cat "$i"; echo; done > out
>> > >
>> > > I can't really think of a fast yet simple solution.
>> > This should work:
>> >
>> >      print >lf
>> >      files=(foo.*)
>> >      lf=(lf)
>> >      cat -- ${files:^^lf}
>> >      rm lf
>> >
>> This looks interesting but any chance of an explanation of how it works
>> for
>> us lesser mortals please?
>
> It first generates a file "lf" that contains only a newline and
> stores the filename in an array "lf", and the names of the files
> in question in an array "files".
>
> "${files:^^lf}" merges the two arrays in an interleaving fashion.
> The result is file.1 lf file.2 lf file.3 lf ... (The shorter array
> is repeated up to the other one's length).
>
> Quite clever solution, but a bit complicated.

And here's a oneliner version,
() { cat -- ${@[2,-1]:^^1} } =(echo) foo.*

-- 
Mikael Magnusson