Hi folks, I'm looking for an easy way to split the lines of a .csv file into the fields of an array variable. There's a script that does that somewhore on the net. But that script parses lines character by character and just manages to parse about 100 (long) lines per second. Fields in a .csv file are separated by commas, *but* commas between a pair of quotes do not split. Or phrased differently: Commas that have an even number of double quotes left of them do split, but commas with an uneven number left of then don't split. Any ideas for a quick implementation? Ciao Dominik ^_^ ^_^ -- Dominik Vogt
> I'm looking for an easy way to split the lines of a .csv file into
> the fields of an array variable. There's a script that does that
> somewhore on the net. But that script parses lines character by
> character and just manages to parse about 100 (long) lines per
> second.
>
> Fields in a .csv file are separated by commas, *but* commas
> between a pair of quotes do not split. Or phrased differently:
> Commas that have an even number of double quotes left of them do
> split, but commas with an uneven number left of then don't split.
>
> Any ideas for a quick implementation?
Sebastian has done similar things so may have better ideas.
If you're happy to use shell syntax --- in other words, the other forms
of quoting are active, not just double quotes, so backslashes and single
quotes might do inconvenient things --- and you're not too bothered
about unquoted spaces, which will add extra lines of splitting, you can
use this trick:
% line='This,"is, quite possibly, a",line,"of,stuff","with,commas"'
% print -rl ${(Q)${${(z)${line//,/, }}%%,}//, /,}
This
is, quite possibly, a
line
of,stuff
with,commas
Each comma gets a space added, then the line is split on syntactically
active spaces; any comma at the end of a field is removed; the remaining
commas are restored.
To strip the quotes, add the (Q) flag to the outermost step.
If you need to be careful about unquoted spaces, you need to be
cleverer: e.g. backslash quote them and then remove the bacslashes
later. E.g. up to subtle effects associated with backslashes
print -rl ${${${${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }
will retain existing spaces.
Also, if you want to keep empty fields, you'll need the final result
to use "${(@}this}". Probably easiest to assign to an array as otherwise
the quotes will affect the substitution.
If you're worried about subtle effects with backslashes, I don't think
you're ever going to be satisfied with a quick and dirty hack like this,
so you'll have to decide how sophisticated you need to be.
pws
On Sat, 2 Feb 2019 at 21:27, Peter Stephenson > Also, if you want to keep empty fields, you'll need the final result > to use "${(@}this}". Probably easiest to assign to an array as otherwise > the quotes will affect the substitution. I've had to use "(@)" for every segment, i.e.: % line='abc,"efg, hehe,yeah",c,,d' % print -rl "${(@)${(@)${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }" abc "efg, hehe,yeah" c d And with a random "(@)" missing: % print -rl "${(@)${${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }" abc "efg, hehe,yeah" c d -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org
On Sun, Feb 03, 2019 at 05:19:08AM +0100, Sebastian Gniazdowski wrote:
> On Sat, 2 Feb 2019 at 21:27, Peter Stephenson
> > Also, if you want to keep empty fields, you'll need the final result
> > to use "${(@}this}". Probably easiest to assign to an array as otherwise
> > the quotes will affect the substitution.
>
> I've had to use "(@)" for every segment, i.e.:
>
> % line='abc,"efg, hehe,yeah",c,,d'
> % print -rl "${(@)${(@)${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
> abc
> "efg, hehe,yeah"
> c
>
> d
>
> And with a random "(@)" missing:
> % print -rl "${(@)${${(@)${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }"
> abc "efg, hehe,yeah" c d
Uff, looks complicated but should fix my problem. Thanks!
Spaces and backslashes are no real problem beacause they can be
replaced with some unique character sequence before splitting.
Ciao
Dominik ^_^ ^_^
--
Dominik Vogt