[NTG-context] In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[NTG-context] In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[witiko]

Dear list,

I noticed that in LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text.

Consider the following two example files:

#### example.tex

```
\starttext
\newread\example
\openin\example=example.txt
\read\example to\hello
\read\example to\world
\closein\example
\hello\ \world
\stoptext
```

#### example.txt

```
{%
hello}
world
```

Running ConTeXt MkIV with the command `context --luatex example.tex` produces a PDF file with the text "hello world", as expected.

In contrast, running ConTeXt LMTX with the command `context example.tex` produces a PDF file with just the text "world", and a warning that "\end occurred inside a group at level 1". Adding the lines `\show\hello` and `\show\world` before `\stoptext` and rerunning the command `context example.tex` reveals that `\read` only assigned the opening brace `{` rather than the whole balanced text `{hello}` to `\hello`:

```
> \hello=global macro:
{
> \world=global macro:
world
```

Best,
Vit
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Hans Hagen via ntg-context]

On 9/19/2025 11:46 PM, witiko@gmail.com wrote:
> Dear list,
> 
> I noticed that in LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text.

i'll check it but as we just updated and because we don't use those 
reads in context it will not be an immediate fix

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Joseph Wright]

On 20/09/2025 11:15, Hans Hagen via ntg-context wrote:
> On 9/19/2025 11:46 PM, witiko@gmail.com wrote:
>> Dear list,
>>
>> I noticed that in LuaMetaTeX, `\read` sometimes doesn't grab the 
>> entire balanced text.
> 
> i'll check it but as we just updated and because we don't use those 
> reads in context it will not be an immediate fix
> 
> Hans

I'd hope most people wanting to read unbalanced lines are using 
\readline anyway!

Joseph
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Hans Hagen via ntg-context]

On 9/20/2025 12:51 PM, Joseph Wright wrote:
> On 20/09/2025 11:15, Hans Hagen via ntg-context wrote:
>> On 9/19/2025 11:46 PM, witiko@gmail.com wrote:
>>> Dear list,
>>>
>>> I noticed that in LuaMetaTeX, `\read` sometimes doesn't grab the 
>>> entire balanced text.
>>
>> i'll check it but as we just updated and because we don't use those 
>> reads in context it will not be an immediate fix
>>
>> Hans
> 
> I'd hope most people wanting to read unbalanced lines are using 
> \readline anyway!

i wonder how many people that are, unlikely context users; i remember 
that i added it because some tikz picture needed it

btw, the issue had nothign to do with balancing, it was just a comment 
ending up in the macro body

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[witiko]

I am seeing some other issues with `\read` as well. For example, `\def\first{hello  world}` (notice the two spaces) will store 11 tokens "hello␣world" in `\first` but storing `hello  world` in a text file named `example.txt` and reading that file with `\newread\example \openin\example=example.txt \read\example to\second` in LuaMetaTeX stores 13 tokens "hello␣␣world\n" in `\second`. LuaTeX and other TeX engines would store just 12 tokens "hello␣world\n". This is less of an issue than the one described in the original post and can likely be fixed by using `\scantokens` on `\second`.
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Hans Hagen via ntg-context]

On 9/23/2025 11:14 AM, witiko@gmail.com wrote:
> I am seeing some other issues with `\read` as well. For example, `\def\first{hello  world}` (notice the two spaces) will store 11 tokens "hello␣world" in `\first` but storing `hello  world` in a text file named `example.txt` and reading that file with `\newread\example \openin\example=example.txt \read\example to\second` in LuaMetaTeX stores 13 tokens "hello␣␣world\n" in `\second`. LuaTeX and other TeX engines would store just 12 tokens "hello␣world\n". This is less of an issue than the one described in the original post and can likely be fixed by using `\scantokens` on `\second`.

On my machine from this:

\starttext

\newread\example
\openin\example=example.txt
\read\example to\hello
\read\example to\world
\closein\example

(\meaningfull\hello)

(\meaningfull\world)

[\hello\ \world]

\stoptext

I now get:

(global macro:{hello})
(global macro:world)
[hello world]

For

{%
hello}
world

But I used a patched version of \read that is not yet uploaded.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[witiko]

What do you get with the following:

{%
h  ello}
world

Is it (global macro:{h␣␣ello}) or (global macro:{h␣ello})?
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Hans Hagen via ntg-context]

On 9/23/2025 12:09 PM, witiko@gmail.com wrote:
> What do you get with the following:
> 
> {%
> h  ello}
> world
> 
> Is it (global macro:{h␣␣ello}) or (global macro:{h␣ello})?

single space as spaces collapse (and \endlinechar outside 0..127 range 
suppresses spaces at the end)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[witiko]

I am asking because the spaces don't collapse for me with LuaMetaTeX 2.11.08 (level 20250917):

    (global macro:{h␣␣ello} )
    (global macro:world␣)
    [h␣␣ello␣␣world␣]

For

{h  ello}
world

I am assuming that's part of the unreleased fix, i.e. your "patched version of \read" then?

Best,
Vit
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: In LuaMetaTeX, `\read` sometimes doesn't grab the entire balanced text

[Hans Hagen via ntg-context]

On 9/23/2025 1:40 PM, witiko@gmail.com wrote:
> I am asking because the spaces don't collapse for me with LuaMetaTeX 2.11.08 (level 20250917):
> 
>      (global macro:{h␣␣ello} )
>      (global macro:world␣)
>      [h␣␣ello␣␣world␣]
> 
> For
> 
> {h  ello}
> world
> 
> I am assuming that's part of the unreleased fix, i.e. your "patched version of \read" then?

indeed


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________