[NTG-context] Bug? \xmlflushspacewise gobbles blanks

[NTG-context] Bug? \xmlflushspacewise gobbles blanks

[Thangalin]

[-- Attachment #1.1: Type: text/plain, Size: 2159 bytes --]

Hi,

Max Chernoff helped fix a possible issue:

\startbuffer[test]
<?xml version="1.0"?>
<document><pre>
|   aaa|  bbb  |ccc   |
|------|-------|------|
|    1 |   2   | 3    |
|    4 |   5   | 6    |
|    7 |   8   | 9    |
</pre></document>
\stopbuffer

\startxmlsetups xml:setup
  \xmlsetsetup{\xmldocument}{*}{-}
  \xmlsetsetup{\xmldocument}{document|pre}{xml:*}
\stopxmlsetups

\xmlregistersetup{xml:setup}

\startxmlsetups xml:document
  \startdocument \xmlflush{#1} \stopdocument
\stopxmlsetups

\startxmlsetups xml:pre
  \startxmldisplayverbatim
    \setupbodyfont[TextFontIN]
    \spaceskip\zeropoint
    \xmlflushspacewise{#1}
  \stopxmldisplayverbatim
\stopxmlsetups

\definefontfamily [TextFontIN] [tt] [Inconsolata]

\usetypescript[TextFont]

\starttext
  \xmlprocessbuffer{main}{test}{}
\stoptext

The patch:

diff --git a/tex/context/base/mkiv/lxml-tex.lua
b/tex/context/base/mkiv/lxml-tex.lua
index 1e57890..90cb993 100644
--- a/tex/context/base/mkiv/lxml-tex.lua
+++ b/tex/context/base/mkiv/lxml-tex.lua
@@ -29,6 +29,7 @@ local lxml = lxml
 local catcodenumbers     = catcodes.numbers
 local ctxcatcodes        = catcodenumbers.ctxcatcodes -- todo: use
different method
 local notcatcodes        = catcodenumbers.notcatcodes -- todo: use
different method
+local tpacatcodes        = catcodenumbers.tpacatcodes

 local commands           = commands
 local context            = context
@@ -245,7 +246,7 @@ local _, xmlspacecapture_yes = context.newtexthandler {
     emptyline  = context.xmlcdataobeyedline,
     simpleline = context.xmlcdataobeyedline,
     space      = context.xmlcdataobeyedspace,
-    catcodes   = notcatcodes,
+    catcodes   = tpacatcodes,
     exception  = entity,
 }
 local _, xmlspacecapture_nop = context.newtexthandler {
@@ -253,7 +254,7 @@ local _, xmlspacecapture_nop = context.newtexthandler {
     emptyline  = context.xmlcdataobeyedline,
     simpleline = context.xmlcdataobeyedline,
     space      = context.xmlcdataobeyedspace,
-    catcodes   = notcatcodes,
+    catcodes   = tpacatcodes,
 }

 local _, xmllinecapture_yes = context.newtexthandler {

See https://tex.stackexchange.com/q/750584/2148

Cheers!

[-- Attachment #1.2: Type: text/html, Size: 2719 bytes --]

[-- Attachment #2: Type: text/plain, Size: 511 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Fwd: Bug? \xmlflushspacewise gobbles blanks

[Hans Hagen via ntg-context]

On 9/4/2025 6:51 AM, Thangalin wrote:
> Hi Hans,
> 
> In case you missed it, Max Chernoff has a patch for ConTeXt below.

i don't track SE to i missed it and if you have such issues best take it 
to the mailing list
> I'm about to release a new version of KeenWrite that has scores of 
> fixes. I'd like to upgrade the container with a version of ConTeXt that 
> has the fix in place. I could patch the container, but would prefer 
> pulling from official sources.
> 
> Thank you!
> 
> Cheers!
> Dave
> 
> ---------- Forwarded message ---------
> From: *Thangalin* <thangalin@gmail.com <mailto:thangalin@gmail.com>>
> Date: Wed, Sep 3, 2025 at 4:13 PM
> Subject: Bug? \xmlflushspacewise gobbles blanks
> To: mailing list for ConTeXt users <ntg-context@ntg.nl <mailto:ntg- 
> context@ntg.nl>>
> 
> 
> Hi,
> 
> Max Chernoff helped fix a possible issue:
> 
> \startbuffer[test]
> <?xml version="1.0"?>
> <document><pre>
> |   aaa|  bbb  |ccc   |
> |------|-------|------|
> |    1 |   2   | 3    |
> |    4 |   5   | 6    |
> |    7 |   8   | 9    |
> </pre></document>
> \stopbuffer
> 
> \startxmlsetups xml:setup
>    \xmlsetsetup{\xmldocument}{*}{-}
>    \xmlsetsetup{\xmldocument}{document|pre}{xml:*}
> \stopxmlsetups
> 
> \xmlregistersetup{xml:setup}
> 
> \startxmlsetups xml:document
>    \startdocument \xmlflush{#1} \stopdocument
> \stopxmlsetups
> 
> \startxmlsetups xml:pre
>    \startxmldisplayverbatim
>      \setupbodyfont[TextFontIN]
>      \spaceskip\zeropoint
>      \xmlflushspacewise{#1}
>    \stopxmldisplayverbatim
> \stopxmlsetups
> 
> \definefontfamily [TextFontIN] [tt] [Inconsolata]
> 
> \usetypescript[TextFont]
> 
> \starttext
>    \xmlprocessbuffer{main}{test}{}
> \stoptext
> 
> The patch:
> 
> diff --git a/tex/context/base/mkiv/lxml-tex.lua b/tex/context/base/mkiv/ 
> lxml-tex.lua
> index 1e57890..90cb993 100644
> --- a/tex/context/base/mkiv/lxml-tex.lua
> +++ b/tex/context/base/mkiv/lxml-tex.lua
> @@ -29,6 +29,7 @@ local lxml = lxml
>   local catcodenumbers     = catcodes.numbers
>   local ctxcatcodes        = catcodenumbers.ctxcatcodes -- todo: use 
> different method
>   local notcatcodes        = catcodenumbers.notcatcodes -- todo: use 
> different method
> +local tpacatcodes        = catcodenumbers.tpacatcodes
> 
>   local commands           = commands
>   local context            = context
> @@ -245,7 +246,7 @@ local _, xmlspacecapture_yes = context.newtexthandler {
>       emptyline  = context.xmlcdataobeyedline,
>       simpleline = context.xmlcdataobeyedline,
>       space      = context.xmlcdataobeyedspace,
> -    catcodes   = notcatcodes,
> +    catcodes   = tpacatcodes,
>       exception  = entity,
>   }
>   local _, xmlspacecapture_nop = context.newtexthandler {
> @@ -253,7 +254,7 @@ local _, xmlspacecapture_nop = context.newtexthandler {
>       emptyline  = context.xmlcdataobeyedline,
>       simpleline = context.xmlcdataobeyedline,
>       space      = context.xmlcdataobeyedspace,
> -    catcodes   = notcatcodes,
> +    catcodes   = tpacatcodes,
>   }
> 
>   local _, xmllinecapture_yes = context.newtexthandler {

although it fixes the issue it's not the reason; as we want roundtrip 
xml and also need to deal with these special characters in tex the | is 
made into an entity

the exception option in the above deals with it

in cldf-ini we need to change this

         if p_exception then
          -- local content = lpegC((1-spacing-p_exception)^1)
             local content = lpegC((1-space-endofline-p_exception)^1)
             pattern =

i'll ponder the catcode change but wonder if it will have other side 
effects (maybe wolfgang see it)
> See https://tex.stackexchange.com/q/750584/2148 <https:// 
> tex.stackexchange.com/q/750584/2148>

that's not the best place to discuss / fix this

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Fwd: Bug? \xmlflushspacewise gobbles blanks

[Wolfgang Schuster]

Am 04.09.2025 um 12:50 schrieb Hans Hagen via ntg-context:
> On 9/4/2025 6:51 AM, Thangalin wrote:
>> Hi Hans,
>>
>> In case you missed it, Max Chernoff has a patch for ConTeXt below.
> 
> i don't track SE to i missed it and if you have such issues best take it 
> to the mailing list
>> I'm about to release a new version of KeenWrite that has scores of 
>> fixes. I'd like to upgrade the container with a version of ConTeXt 
>> that has the fix in place. I could patch the container, but would 
>> prefer pulling from official sources.
>>
>> Thank you!
>>
>> Cheers!
>> Dave
>>
>> ---------- Forwarded message ---------
>> From: *Thangalin* <thangalin@gmail.com <mailto:thangalin@gmail.com>>
>> Date: Wed, Sep 3, 2025 at 4:13 PM
>> Subject: Bug? \xmlflushspacewise gobbles blanks
>> To: mailing list for ConTeXt users <ntg-context@ntg.nl <mailto:ntg- 
>> context@ntg.nl>>
>>
>>
>> Hi,
>>
>> Max Chernoff helped fix a possible issue:
>>
>> \startbuffer[test]
>> <?xml version="1.0"?>
>> <document><pre>
>> |   aaa|  bbb  |ccc   |
>> |------|-------|------|
>> |    1 |   2   | 3    |
>> |    4 |   5   | 6    |
>> |    7 |   8   | 9    |
>> </pre></document>
>> \stopbuffer
>>
>> \startxmlsetups xml:setup
>>    \xmlsetsetup{\xmldocument}{*}{-}
>>    \xmlsetsetup{\xmldocument}{document|pre}{xml:*}
>> \stopxmlsetups
>>
>> \xmlregistersetup{xml:setup}
>>
>> \startxmlsetups xml:document
>>    \startdocument \xmlflush{#1} \stopdocument
>> \stopxmlsetups
>>
>> \startxmlsetups xml:pre
>>    \startxmldisplayverbatim
>>      \setupbodyfont[TextFontIN]
>>      \spaceskip\zeropoint
>>      \xmlflushspacewise{#1}
>>    \stopxmldisplayverbatim
>> \stopxmlsetups
>>
>> \definefontfamily [TextFontIN] [tt] [Inconsolata]
>>
>> \usetypescript[TextFont]
>>
>> \starttext
>>    \xmlprocessbuffer{main}{test}{}
>> \stoptext
>>
>> The patch:
>>
>> diff --git a/tex/context/base/mkiv/lxml-tex.lua b/tex/context/base/ 
>> mkiv/ lxml-tex.lua
>> index 1e57890..90cb993 100644
>> --- a/tex/context/base/mkiv/lxml-tex.lua
>> +++ b/tex/context/base/mkiv/lxml-tex.lua
>> @@ -29,6 +29,7 @@ local lxml = lxml
>>   local catcodenumbers     = catcodes.numbers
>>   local ctxcatcodes        = catcodenumbers.ctxcatcodes -- todo: use 
>> different method
>>   local notcatcodes        = catcodenumbers.notcatcodes -- todo: use 
>> different method
>> +local tpacatcodes        = catcodenumbers.tpacatcodes
>>
>>   local commands           = commands
>>   local context            = context
>> @@ -245,7 +246,7 @@ local _, xmlspacecapture_yes = 
>> context.newtexthandler {
>>       emptyline  = context.xmlcdataobeyedline,
>>       simpleline = context.xmlcdataobeyedline,
>>       space      = context.xmlcdataobeyedspace,
>> -    catcodes   = notcatcodes,
>> +    catcodes   = tpacatcodes,
>>       exception  = entity,
>>   }
>>   local _, xmlspacecapture_nop = context.newtexthandler {
>> @@ -253,7 +254,7 @@ local _, xmlspacecapture_nop = 
>> context.newtexthandler {
>>       emptyline  = context.xmlcdataobeyedline,
>>       simpleline = context.xmlcdataobeyedline,
>>       space      = context.xmlcdataobeyedspace,
>> -    catcodes   = notcatcodes,
>> +    catcodes   = tpacatcodes,
>>   }
>>
>>   local _, xmllinecapture_yes = context.newtexthandler {
> 
> although it fixes the issue it's not the reason; as we want roundtrip 
> xml and also need to deal with these special characters in tex the | is 
> made into an entity
> 
> the exception option in the above deals with it
> 
> in cldf-ini we need to change this
> 
>          if p_exception then
>           -- local content = lpegC((1-spacing-p_exception)^1)
>              local content = lpegC((1-space-endofline-p_exception)^1)
>              pattern =


This doesn't change anything in the output.


The solution from Max keep the spaces but it's still wrong, when you run 
the following example you notice only the spaces before the numbers in 
the column are active but not the once after it.


\startbuffer[test]
<pre>
|---|---|---|
|  1| 2 |3  |
</pre>
\stopbuffer

\startxmlsetups xml:setup
   \xmlsetsetup{\xmldocument}{pre}{xml:pre}
\stopxmlsetups

\xmlregistersetup{xml:setup}

\startxmlsetups xml:pre
   \tttf
   \def\obeyedspace{?}
   \xmlflushspacewise{#1}
\stopxmlsetups

\starttext
\xmlprocessbuffer{main}{test}{}
\stoptext


Wolfgang
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________