Trimming strings and Lua string.trim question

Trimming strings and Lua string.trim question

[Rik Kabel]

[-- Attachment #1.1: Type: text/plain, Size: 1437 bytes --]

I am trying to strip all leading and trailing whitespace from a string. 
I have tried two methods, neither of which is satisfactory.

The first method uses \ignorespaces and \removeunwantedspaces. This 
fails to strip leading and trailing newlines.

The second method uses the Lua string.strip function. The wiki description:

    Yields string with leading and trailing whitespace (spaces,
    horizontal and vertical tabs, newlines) removed

suggests that this should do what I want, but I find that it removes 
internal whitespace as well as the leading and trailing whitespace.

Can somebody suggest a better way of doing this?

The following code demonstrates the problem:

    % macros=mkvi
    \starttexdefinition StringStrip #STRING
       \startluacode
         context(string.strip([==[#STRING]==]))
       \stopluacode
    \stoptexdefinition
    \long\def\test{

    This is a test. \quad

    And it has an unexpected result.

    With \tex{ignorespaces} and \tex{removeunwantedspaces}, the
    newlines remain.

    With Lua {\tt string.strip}, the \tex{quad} at the end of a
    paragraph is preserved,    but the other internal whitespace
    (including newlines) is gone!

    }
    \starttext
    \subject{ignorespaces and removeunwantedspaces}
    ¦\ignorespaces\test\removeunwantedspaces¦

    \subject{Lua string.strip}
    ¦\StringStrip{\test}¦
    \stoptext

-- 
Rik

[-- Attachment #1.2: Type: text/html, Size: 2441 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Hans Hagen]

On 3/5/2014 5:45 AM, Rik Kabel wrote:
> I am trying to strip all leading and trailing whitespace from a string.
> I have tried two methods, neither of which is satisfactory.
>
> The first method uses \ignorespaces and \removeunwantedspaces. This
> fails to strip leading and trailing newlines.
>
> The second method uses the Lua string.strip function. The wiki description:
>
>     Yields string with leading and trailing whitespace (spaces,
>     horizontal and vertical tabs, newlines) removed
>
> suggests that this should do what I want, but I find that it removes
> internal whitespace as well as the leading and trailing whitespace.
>
> Can somebody suggest a better way of doing this?
>
> The following code demonstrates the problem:
>
>     % macros=mkvi
>     \starttexdefinition StringStrip #STRING
>        \startluacode
>          context(string.strip([==[#STRING]==]))
>        \stopluacode
>     \stoptexdefinition
>     \long\def\test{
>
>     This is a test. \quad
>
>     And it has an unexpected result.
>
>     With \tex{ignorespaces} and \tex{removeunwantedspaces}, the
>     newlines remain.
>
>     With Lua {\tt string.strip}, the \tex{quad} at the end of a
>     paragraph is preserved,    but the other internal whitespace
>     (including newlines) is gone!
>
>     }
>     \starttext
>     \subject{ignorespaces and removeunwantedspaces}
>     ¦\ignorespaces\test\removeunwantedspaces¦
>
>     \subject{Lua string.strip}
>     ¦\StringStrip{\test}¦
>     \stoptext

striplong

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Rik Kabel]

[-- Attachment #1.1: Type: text/plain, Size: 2108 bytes --]

On 2014-03-05 03:24, Hans Hagen wrote:
> On 3/5/2014 5:45 AM, Rik Kabel wrote:
>> I am trying to strip all leading and trailing whitespace from a string.
>> I have tried two methods, neither of which is satisfactory.
>>
>> The first method uses \ignorespaces and \removeunwantedspaces. This
>> fails to strip leading and trailing newlines.
>>
>> The second method uses the Lua string.strip function. The wiki 
>> description:
>>
>>     Yields string with leading and trailing whitespace (spaces,
>>     horizontal and vertical tabs, newlines) removed
>>
>> suggests that this should do what I want, but I find that it removes
>> internal whitespace as well as the leading and trailing whitespace.
>>
>> Can somebody suggest a better way of doing this?
>>
>> The following code demonstrates the problem:
>>
>>     % macros=mkvi
>>     \starttexdefinition StringStrip #STRING
>>        \startluacode
>>          context(string.strip([==[#STRING]==]))
>>        \stopluacode
>>     \stoptexdefinition
>>     \long\def\test{
>>
>>     This is a test. \quad
>>
>>     And it has an unexpected result.
>>
>>     With \tex{ignorespaces} and \tex{removeunwantedspaces}, the
>>     newlines remain.
>>
>>     With Lua {\tt string.strip}, the \tex{quad} at the end of a
>>     paragraph is preserved,    but the other internal whitespace
>>     (including newlines) is gone!
>>
>>     }
>>     \starttext
>>     \subject{ignorespaces and removeunwantedspaces}
>>     ¦\ignorespaces\test\removeunwantedspaces¦
>>
>>     \subject{Lua string.strip}
>>     ¦\StringStrip{\test}¦
>>     \stoptext
>
> striplong
>

It would be nice if striplong worked, but the example fails when 
string.striplong is used in place of string.strip. The log says:

    ! LuaTeX error [string "\directlua "]:1: attempt to call field
    'striplong' (a nil value)
    stack traceback:
         [string "\directlua "]:1: in main chunk.

Is the format of the call different than shown in the wiki, or is a 
different mechanism required to quote a potentially long string?

-- 
Rik

[-- Attachment #1.2: Type: text/html, Size: 3768 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Wolfgang Schuster]

[-- Attachment #1.1: Type: text/plain, Size: 600 bytes --]


Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com>:

>> striplong 
>> 
> 
> It would be nice if striplong worked, but the example fails when string.striplong is used in place of string.strip. The log says:
> 
> ! LuaTeX error [string "\directlua "]:1: attempt to call field 'striplong' (a nil value)
> stack traceback:
>     [string "\directlua "]:1: in main chunk.
> Is the format of the call different than shown in the wiki, or is a different mechanism required to quote a potentially long string?

The name of the function is “strings.striplong”.

Wolfgang


[-- Attachment #1.2: Type: text/html, Size: 1362 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Wolfgang Schuster]

[-- Attachment #1.1: Type: text/plain, Size: 882 bytes --]


Am 05.03.2014 um 16:18 schrieb Wolfgang Schuster <schuster.wolfgang@gmail.com>:

> 
> Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com>:
> 
>>> striplong 
>>> 
>> 
>> It would be nice if striplong worked, but the example fails when string.striplong is used in place of string.strip. The log says:
>> 
>> ! LuaTeX error [string "\directlua "]:1: attempt to call field 'striplong' (a nil value)
>> stack traceback:
>>     [string "\directlua "]:1: in main chunk.
>> Is the format of the call different than shown in the wiki, or is a different mechanism required to quote a potentially long string?
> 
> The name of the function is “strings.striplong”.

This should be “utilities.strings.striplong” but it has no effect because when take a look
at the saved string in Lua you can see that blank lines are removed from the text.

Wolfgang


[-- Attachment #1.2: Type: text/html, Size: 1975 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Rik Kabel]

[-- Attachment #1.1: Type: text/plain, Size: 1002 bytes --]

On 2014-03-05 10:18, Wolfgang Schuster wrote:
>
> Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com 
> <mailto:context@rik.users.panix.com>>:
>
>>> striplong
>>>
>>
>> It would be nice if striplong worked, but the example fails when 
>> string.striplong is used in place of string.strip. The log says:
>>
>>     ! LuaTeX error [string "\directlua "]:1: attempt to call field
>>     'striplong' (a nil value)
>>     stack traceback:
>>         [string "\directlua "]:1: in main chunk.
>>
>> Is the format of the call different than shown in the wiki, or is a 
>> different mechanism required to quote a potentially long string?
>
> The name of the function is "strings.striplong".
>
> Wolfgang

Thank you, Wolfgang. I have updated the wiki entry for this function. I 
have not been able to verify that other functions are similarly 
misidentified in the wiki, and have therefore not updated any  others. 
Those who are more familiar with the functions may wish to review it.

--
Rik

[-- Attachment #1.2: Type: text/html, Size: 2280 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Rik Kabel]

[-- Attachment #1.1: Type: text/plain, Size: 1295 bytes --]

On 2014-03-05 10:54, Wolfgang Schuster wrote:
>
> Am 05.03.2014 um 16:18 schrieb Wolfgang Schuster 
> <schuster.wolfgang@gmail.com <mailto:schuster.wolfgang@gmail.com>>:
>
>>
>> Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com 
>> <mailto:context@rik.users.panix.com>>:
>>
>>>> striplong
>>>>
>>>
>>> It would be nice if striplong worked, but the example fails when 
>>> string.striplong is used in place of string.strip. The log says:
>>>
>>>     ! LuaTeX error [string "\directlua "]:1: attempt to call field
>>>     'striplong' (a nil value)
>>>     stack traceback:
>>>         [string "\directlua "]:1: in main chunk.
>>>
>>> Is the format of the call different than shown in the wiki, or is a 
>>> different mechanism required to quote a potentially long string?
>>
>> The name of the function is "strings.striplong".
>
> This should be "utilities.strings.striplong" but it has no effect 
> because when take a look
> at the saved string in Lua you can see that blank lines are removed 
> from the text.
>
> Wolfgang

Indeed. This does build without error, but the 
utilities.strings.striplong result is worse than the string.strip 
result, with some additional whitespace at the end of the test string 
from the earlier example.

So, is this a confirmed bug?
-- 
Rik

[-- Attachment #1.2: Type: text/html, Size: 3265 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Hans Hagen]

On 3/6/2014 12:00 AM, Rik Kabel wrote:
> On 2014-03-05 10:54, Wolfgang Schuster wrote:
>>
>> Am 05.03.2014 um 16:18 schrieb Wolfgang Schuster
>> <schuster.wolfgang@gmail.com <mailto:schuster.wolfgang@gmail.com>>:
>>
>>>
>>> Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com
>>> <mailto:context@rik.users.panix.com>>:
>>>
>>>>> striplong
>>>>>
>>>>
>>>> It would be nice if striplong worked, but the example fails when
>>>> string.striplong is used in place of string.strip. The log says:
>>>>
>>>>     ! LuaTeX error [string "\directlua "]:1: attempt to call field
>>>>     'striplong' (a nil value)
>>>>     stack traceback:
>>>>         [string "\directlua "]:1: in main chunk.
>>>>
>>>> Is the format of the call different than shown in the wiki, or is a
>>>> different mechanism required to quote a potentially long string?
>>>
>>> The name of the function is “strings.striplong”.
>>
>> This should be “utilities.strings.striplong” but it has no effect
>> because when take a look
>> at the saved string in Lua you can see that blank lines are removed
>> from the text.
>>
>> Wolfgang
>
> Indeed. This does build without error, but the
> utilities.strings.striplong result is worse than the string.strip
> result, with some additional whitespace at the end of the test string
> from the earlier example.

the definition of 'worse' depends on expectations and given the fact 
that in tex a newline or two newlines in a row have a different meaning 
it is just a guess what you want to achieve

anyway, the next beta will have

local str = table.concat( {
"  ",
"    aap",
"  noot mies",
"  ",
"    ",
" zus    wim jet",
"zus    wim jet",
"       zus    wim jet",
"    ",
}, "\n")

utilities.strings.striplines(str,'prune'):

[[aap
noot mies


zus    wim jet
zus    wim jet
zus    wim jet]]

utilities.strings.striplines(str,'prune and collapse'):

[[aap
noot mies

zus    wim jet
zus    wim jet
zus    wim jet]]

utilities.strings.striplines(str,'prune and no empty'):

[[aap
noot mies
zus    wim jet
zus    wim jet
zus    wim jet]]

utilities.strings.striplines(str,'retain'):

[[
aap
noot mies


zus    wim jet
zus    wim jet
zus    wim jet
]]

utilities.strings.striplines(str,'retain and collapse'):

[[
aap
noot mies

zus    wim jet
zus    wim jet
zus    wim jet
]]

utilities.strings.striplines(str,'retain and no empty'):

[[
aap
noot mies
zus    wim jet
zus    wim jet
zus    wim jet
]]

so you can choose what you like

Hans




-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Rik Kabel]

[-- Attachment #1.1: Type: text/plain, Size: 4207 bytes --]

On 2014-03-06 05:38, Hans Hagen wrote:
> On 3/6/2014 12:00 AM, Rik Kabel wrote:
>> On 2014-03-05 10:54, Wolfgang Schuster wrote:
>>>
>>> Am 05.03.2014 um 16:18 schrieb Wolfgang Schuster
>>> <schuster.wolfgang@gmail.com <mailto:schuster.wolfgang@gmail.com>>:
>>>
>>>>
>>>> Am 05.03.2014 um 16:07 schrieb Rik Kabel <context@rik.users.panix.com
>>>> <mailto:context@rik.users.panix.com>>:
>>>>
>>>>>> striplong
>>>>>>
>>>>>
>>>>> It would be nice if striplong worked, but the example fails when
>>>>> string.striplong is used in place of string.strip. The log says:
>>>>>
>>>>>     ! LuaTeX error [string "\directlua "]:1: attempt to call field
>>>>>     'striplong' (a nil value)
>>>>>     stack traceback:
>>>>>         [string "\directlua "]:1: in main chunk.
>>>>>
>>>>> Is the format of the call different than shown in the wiki, or is a
>>>>> different mechanism required to quote a potentially long string?
>>>>
>>>> The name of the function is “strings.striplong”.
>>>
>>> This should be “utilities.strings.striplong” but it has no effect
>>> because when take a look
>>> at the saved string in Lua you can see that blank lines are removed
>>> from the text.
>>>
>>> Wolfgang
>>
>> Indeed. This does build without error, but the
>> utilities.strings.striplong result is worse than the string.strip
>> result, with some additional whitespace at the end of the test string
>> from the earlier example.
>
> the definition of 'worse' depends on expectations and given the fact 
> that in tex a newline or two newlines in a row have a different 
> meaning it is just a guess what you want to achieve
>
> anyway, the next beta will have
>
> local str = table.concat( {
> "  ",
> "    aap",
> "  noot mies",
> "  ",
> "    ",
> " zus    wim jet",
> "zus    wim jet",
> "       zus    wim jet",
> "    ",
> }, "\n")
>
> utilities.strings.striplines(str,'prune'):
>
> [[aap
> noot mies
>
>
> zus    wim jet
> zus    wim jet
> zus    wim jet]]
>
> utilities.strings.striplines(str,'prune and collapse'):
>
> [[aap
> noot mies
>
> zus    wim jet
> zus    wim jet
> zus    wim jet]]
>
> utilities.strings.striplines(str,'prune and no empty'):
>
> [[aap
> noot mies
> zus    wim jet
> zus    wim jet
> zus    wim jet]]
>
> utilities.strings.striplines(str,'retain'):
>
> [[
> aap
> noot mies
>
>
> zus    wim jet
> zus    wim jet
> zus    wim jet
> ]]
>
> utilities.strings.striplines(str,'retain and collapse'):
>
> [[
> aap
> noot mies
>
> zus    wim jet
> zus    wim jet
> zus    wim jet
> ]]
>
> utilities.strings.striplines(str,'retain and no empty'):
>
> [[
> aap
> noot mies
> zus    wim jet
> zus    wim jet
> zus    wim jet
> ]]
>
> so you can choose what you like
>
> Hans

indeed, beauty is in the eye of the beholder.

Thank you for the flexible solution that allows each of us to find beauty.

Could you please ensure that this handles text in buffers the same way 
that it handles other strings? I noticed that this is handled a bit 
differently, as the following shows,but perhaps it is my clumsy coding. 
The buffer result is closer to what I expect in terms of internal 
whitespace, but still does not show expected trimming of leading and 
trailing whitespace. That trimming, of course, is the point of the exercise.


    \def\StringsStripLong#1%
       {\ctxlua{context(utilities.strings.striplong([==[#1]==]))}}

    \def\StringStrip#1%
       {\ctxlua{context(string.strip([==[#1]==]))}}

    \startbuffer[testbuffer]

    B This is a test.

    And it has an unexpected result.


    \stopbuffer

    \long\def\testmacro{

    M This is a test.

    And it has an unexpected result.


    }

    \starttext

    \subject{Lua string.strip passed TeX buffer}
    ¦\StringStrip{\getbuffer[testbuffer]}¦

    \subject{Lua string.strip passed TeX macro}
    ¦\StringStrip{\testmacro}¦

    \subject{Lua utilities.strings.striplong passed TeX buffer}
    ¦\StringsStripLong{\getbuffer[testbuffer]}¦

    \subject{Lua utilities.strings.striplong passed TeX macro}
    ¦\StringsStripLong{\testmacro}¦

    \stoptext

-- 
Rik

[-- Attachment #1.2: Type: text/html, Size: 7699 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Trimming strings and Lua string.trim question

[Wolfgang Schuster]

Am 06.03.2014 um 16:04 schrieb Rik Kabel <context@rik.users.panix.com>:

> Could you please ensure that this handles text in buffers the same way that it handles other strings? I noticed that this is handled a bit differently, as the following shows,but perhaps it is my clumsy coding. The buffer result is closer to what I expect in terms of internal whitespace, but still does not show expected trimming of leading and trailing whitespace. That trimming, of course, is the point of the exercise.

This can’t work because \getbuffer is a unexpandable command and what you pass to Lua is the string

  \getbuffer[testbuffer]

and not the content of the buffer. When you want to manipulate the content of a buffer you have to access the content at the Lua side with the buffers.getcontent(…) function.

BTW: You don’t need \long in MkIV because definitions use it by default.

Wolfgang

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________