[9fans] ndb and tokenize

[Saroj Mahapatra <saroj@optonline.net>]

Rob had written in an earlier posting about how he is pushing to use
%q and tokenize consistently everywhere. Ndb now uses double quotes
("avalue" style) and if I remember correctly, can not handle embedded
quotes. Imagine if ndb used rc-style quotes for attribute values
(aname='avalue'). tokenize in its current form can not be used to
parse an arbitrary number of pairs. So I think 'tokenize' should
return the rest of the string (in an out argument) after setting the
last argument in the argument vector. Note that 'getfields' does not
alter a token (such as, compressing quotes); so there is no difficulty
in calling 'getfields' repeatedly to get an arbitrary number of
fields. But 'tokenize' must leave the rest of the string untouched, so
that one can call 'tokenize' again. I have been thinking about
'tokenize' for some time and I'd like to hear others' opinions
regarding this.


Thank you,
Saroj Mahapatra