Parsing CVS files

Parsing CVS files

[Dominik Vogt]

Hi folks,

I'm looking for an easy way to split the lines of a .csv file into
the fields of an array variable.  There's a script that does that
somewhore on the net.  But that script parses lines character by
character and just manages to parse about 100 (long) lines per
second.

Fields in a .csv file are separated by commas, *but* commas
between a pair of quotes do not split.  Or phrased differently:
Commas that have an even number of double quotes left of them do
split, but commas with an uneven number left of then don't split.

Any ideas for a quick implementation?

Ciao

Dominik ^_^  ^_^

-- 

Dominik Vogt

Re: Parsing CVS files

[Peter Stephenson]

> I'm looking for an easy way to split the lines of a .csv file into
> the fields of an array variable.  There's a script that does that
> somewhore on the net.  But that script parses lines character by
> character and just manages to parse about 100 (long) lines per
> second.
> 
> Fields in a .csv file are separated by commas, *but* commas
> between a pair of quotes do not split.  Or phrased differently:
> Commas that have an even number of double quotes left of them do
> split, but commas with an uneven number left of then don't split.
> 
> Any ideas for a quick implementation?

Sebastian has done similar things so may have better ideas.

If you're happy to use shell syntax --- in other words, the other forms
of quoting are active, not just double quotes, so backslashes and single
quotes might do inconvenient things --- and you're not too bothered
about unquoted spaces, which will add extra lines of splitting, you can
use this trick:

% line='This,"is, quite possibly, a",line,"of,stuff","with,commas"'
% print -rl ${(Q)${${(z)${line//,/, }}%%,}//, /,}
This
is, quite possibly, a
line
of,stuff
with,commas

Each comma gets a space added, then the line is split on syntactically
active spaces; any comma at the end of a field is removed; the remaining
commas are restored.

To strip the quotes, add the (Q) flag to the outermost step.

If you need to be careful about unquoted spaces, you need to be
cleverer:  e.g. backslash quote them and then remove the bacslashes
later.  E.g. up to subtle effects associated with backslashes

print -rl ${${${${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }

will retain existing spaces.

Also, if you want to keep empty fields, you'll need the final result
to use "${(@}this}".  Probably easiest to assign to an array as otherwise
the quotes will affect the substitution.

If you're worried about subtle effects with backslashes, I don't think
you're ever going to be satisfied with a quick and dirty hack like this,
so you'll have to decide how sophisticated you need to be.

pws

Re: Parsing CVS files

[Sebastian Gniazdowski]

On Sat, 2 Feb 2019 at 21:27, Peter Stephenson
> Also, if you want to keep empty fields, you'll need the final result
> to use "${(@}this}".  Probably easiest to assign to an array as otherwise
> the quotes will affect the substitution.

I've had to use "(@)" for every segment, i.e.:

% line='abc,"efg, hehe,yeah",c,,d'
% print -rl "${(@)${(@)${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
abc
"efg, hehe,yeah"
c

d

And with a random "(@)" missing:
% print -rl "${(@)${${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
abc "efg, hehe,yeah" c  d


-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Parsing CVS files

[Dominik Vogt]

On Sun, Feb 03, 2019 at 05:19:08AM +0100, Sebastian Gniazdowski wrote:
> On Sat, 2 Feb 2019 at 21:27, Peter Stephenson
> > Also, if you want to keep empty fields, you'll need the final result
> > to use "${(@}this}".  Probably easiest to assign to an array as otherwise
> > the quotes will affect the substitution.
> 
> I've had to use "(@)" for every segment, i.e.:
> 
> % line='abc,"efg, hehe,yeah",c,,d'
> % print -rl "${(@)${(@)${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
> abc
> "efg, hehe,yeah"
> c
> 
> d
> 
> And with a random "(@)" missing:
> % print -rl "${(@)${${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
> abc "efg, hehe,yeah" c  d

Uff, looks complicated but should fix my problem.  Thanks!

Spaces and backslashes are no real problem beacause they can be
replaced with some unique character sequence before splitting.

Ciao

Dominik ^_^  ^_^

-- 

Dominik Vogt