* Multiple cases of unexpected behaviour in luametatex
@ 2020-07-03 12:55 Marcel Fabian Krüger
2020-07-03 19:56 ` Hans Hagen
0 siblings, 1 reply; 8+ messages in thread
From: Marcel Fabian Krüger @ 2020-07-03 12:55 UTC (permalink / raw)
To: ConTeXt users list
Hi,
I recently noticed some cases where luametatex behaved in unexpected
ways:
- The "Extra \fi" error isn't triggered, instead an extra `\fi`
freezes luametatex. (Can be reproduced by compiling a document which
only consists of a single \fi)
- token.new can only create some `data` tokens, but it doesn't apply
bound checking on it's arguments:
Take
```
\directlua{
t = token.new(0x200000, token.command_id'data')
print(t.cmdname, t.command, t.mode)
}
```
which prints
register 102 0
The issue does not seem to be that such tokens do not exists because
\letdatacode\somedata="200000
\directlua{
local t = token.create'somedata'
print(t.cmdname, t.command, t.index)
}
does print
data 101 2097152
Also for all other commands LuaTeX seems to apply range-checks to
ensure that such overflows don't happen, even if invalid values are
passed as firstargument.
- There is token.primitives(). My assumption is that the returned
table is meant to indicate the command is, mode and name
corresponding to every primitive. (I think it is awesome that such a
table is made available in luametatex) But especially the mode
field sometimes has values which do not correspond to the mode of
the actual primitives:
I tried running the following in an almost iniTeX setting where all
primitives aside from \shipout and \Umathcodenum have their default
definitions:
```
\catcode`\%=12
\catcode`\~=12
\directlua{
local sorted = token.primitives()
table.sort(sorted, function(a,b) return a[1]<b[1] or a[1]==b[1] and a[2]<b[2]end)
for _,info in ipairs(sorted) do
local t = token.create(info[3])
local rc, rm = t.command, t.mode
if rc==info[1] and rm ~= info[2] then
if info[2] == 0 then
print(string.format('MODE MISMATCH, expected zero: \string\\%s: real: %i, command: %i', info[3], rm, rc))
else
print(string.format('MODE MISMATCH: \string\\%s: offset: %i, command: %i', info[3], rm-info[2], rc))
end
elseif rc~=info[1] then print(t.csname)
end
end
}
```
This indicates that there are two kinds of differences:
For some command codes, there are multiple primitives whose second
entry in the token.primitives table is zero even though their mode
is not zero. This especially affects the commands `above`,
`after_something`, `make_box`, `un_vbox`, `set_specification` and
`car_ret`.
E.g. for after_something, all of \atendofgrouped, \afterassigned and
\aftergrouped have a zero as second entry in token.primitives.
The other difference is that all the internal_... commands have a
fixed offset which differes between commands in their mode field.
IMO the difference for the internal_... commands make sense because
they make for easier to use numbers, but having multiple primitives
indicating mode 0 for the other commands seems to make this table
significantly less useful because it can't be used to get a unique
description of a primitive.
(I may have completely misinterpreted the table of course, but given
that for other primitives the values match I do not think so)
Best regards,
Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Multiple cases of unexpected behaviour in luametatex
2020-07-03 12:55 Multiple cases of unexpected behaviour in luametatex Marcel Fabian Krüger
@ 2020-07-03 19:56 ` Hans Hagen
2020-07-04 5:22 ` Marcel Fabian Krüger
0 siblings, 1 reply; 8+ messages in thread
From: Hans Hagen @ 2020-07-03 19:56 UTC (permalink / raw)
To: mailing list for ConTeXt users, Marcel Fabian Krüger
On 7/3/2020 2:55 PM, Marcel Fabian Krüger wrote:
> Hi,
>
> I recently noticed some cases where luametatex behaved in unexpected
> ways:
>
> - The "Extra \fi" error isn't triggered, instead an extra `\fi`
> freezes luametatex. (Can be reproduced by compiling a document which
> only consists of a single \fi)
i already fixed here (noticed it when documenting some conditionals)
> - token.new can only create some `data` tokens, but it doesn't apply
> bound checking on it's arguments:
there is no checking yet, there is an upper limit of 0x1FFFFF, so i'll
add a check for that
> Also for all other commands LuaTeX seems to apply range-checks to
> ensure that such overflows don't happen, even if invalid values are
> passed as firstargument.
indeed, but hadn't yet done that for data, it also need a more strict
check at the tex end (i'm still not sure if i make a slightly different
implementation of it but i can add the test anyway)
> - There is token.primitives(). My assumption is that the returned
> table is meant to indicate the command is, mode and name
> corresponding to every primitive. (I think it is awesome that such a
> table is made available in luametatex) But especially the mode
> field sometimes has values which do not correspond to the mode of
> the actual primitives:
indeed.
> I tried running the following in an almost iniTeX setting where all
> primitives aside from \shipout and \Umathcodenum have their default
> definitions:
>
> ```
> \catcode`\%=12
> \catcode`\~=12
> \directlua{
> local sorted = token.primitives()
> table.sort(sorted, function(a,b) return a[1]<b[1] or a[1]==b[1] and a[2]<b[2]end)
> for _,info in ipairs(sorted) do
> local t = token.create(info[3])
> local rc, rm = t.command, t.mode
> if rc==info[1] and rm ~= info[2] then
> if info[2] == 0 then
> print(string.format('MODE MISMATCH, expected zero: \string\\%s: real: %i, command: %i', info[3], rm, rc))
> else
> print(string.format('MODE MISMATCH: \string\\%s: offset: %i, command: %i', info[3], rm-info[2], rc))
> end
> elseif rc~=info[1] then print(t.csname)
> end
> end
> }
> ```
>
> This indicates that there are two kinds of differences:
> For some command codes, there are multiple primitives whose second
> entry in the token.primitives table is zero even though their mode
> is not zero. This especially affects the commands `above`,
> `after_something`, `make_box`, `un_vbox`, `set_specification` and
> `car_ret`.
> E.g. for after_something, all of \atendofgrouped, \afterassigned and
> \aftergrouped have a zero as second entry in token.primitives.
some tokens are more complex in the sense that they are combinations
(have a follow up) and i'm not sure to what extedn i want to block that
... all a matter of experimenting and time, so
the 'mode' field will be dropped but for now i kept it
some like after_something i need to check (i just didn't update their
ranges yet after adding some more primitives that use them) (maybe some
otheres need an offset added but i'll check it)
> The other difference is that all the internal_... commands have a
> fixed offset which differes between commands in their mode field.
>
> IMO the difference for the internal_... commands make sense because
> they make for easier to use numbers, but having multiple primitives
> indicating mode 0 for the other commands seems to make this table
> significantly less useful because it can't be used to get a unique
> description of a primitive.
>
> (I may have completely misinterpreted the table of course, but given
> that for other primitives the values match I do not think so)
it's a it work in progress as there are some exceptions that use special
chr codes (for instance in conditionals several cmd codes need to have
exclusive codes, so adapting it is a stepwise process; one decision i
need to make there is how close to stay to the original tex codes
eventually i want all to have reasonable ranges in the token interface
(not per se the same as in the engine itself but that's a black box
anyway) which involves some offsetting .. i do that stepwise in order to
keep a working engine (the token interface is not used in context that
much)
Hans
hans
-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Multiple cases of unexpected behaviour in luametatex
2020-07-03 19:56 ` Hans Hagen
@ 2020-07-04 5:22 ` Marcel Fabian Krüger
2020-07-04 8:33 ` ConTeXt installation on Windows 10 Jean-Pierre Delange
2020-07-04 17:18 ` Multiple cases of unexpected behaviour in luametatex Hans Hagen
0 siblings, 2 replies; 8+ messages in thread
From: Marcel Fabian Krüger @ 2020-07-04 5:22 UTC (permalink / raw)
To: Hans Hagen; +Cc: mailing list for ConTeXt users
On Fri, Jul 03, 2020 at 09:56:28PM +0200, Hans Hagen wrote:
> On 7/3/2020 2:55 PM, Marcel Fabian Krüger wrote:
> > Hi,
> >
> > I recently noticed some cases where luametatex behaved in unexpected
> > ways:
> >
> > - The "Extra \fi" error isn't triggered, instead an extra `\fi`
> > freezes luametatex. (Can be reproduced by compiling a document which
> > only consists of a single \fi)
>
> i already fixed here (noticed it when documenting some conditionals)
Great.
>
> > - token.new can only create some `data` tokens, but it doesn't apply
> > bound checking on it's arguments:
>
> there is no checking yet, there is an upper limit of 0x1FFFFF, so i'll add a
> check for that
OK, that explains it.
[...]
>
> eventually i want all to have reasonable ranges in the token interface (not
> per se the same as in the engine itself but that's a black box anyway) which
> involves some offsetting .. i do that stepwise in order to keep a working
> engine (the token interface is not used in context that much)
Please let me know if there is anything I can do to help with that.
Anyway, tehre is another thing I am wondering about: In the latest
luametatex versions, error help strings do not seem to work like they used to.
Especially after \errhelp{some help}\errmessage{error} or
tex.error('fdfd', {'some help'}) in the 'handle_error_hook'
callback, tex.gethelptext returns nil. gethelptext still works for
errors triggered by TeX itself.
One completely different question:
When running luametatex (of course with appropriate format/lua init
script options) but without a TeX file or command, luametatex does not
expect interactive input but instead tries to read the input file
'luametatex.tex'. (Where luametatex does not seem to be hardcoded but
derived from argv[0]) Is there a way to overwrite this default input
file name?
Marcel
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* ConTeXt installation on Windows 10
2020-07-04 5:22 ` Marcel Fabian Krüger
@ 2020-07-04 8:33 ` Jean-Pierre Delange
2020-07-04 8:47 ` Wolfgang Schuster
2020-07-04 17:18 ` Multiple cases of unexpected behaviour in luametatex Hans Hagen
1 sibling, 1 reply; 8+ messages in thread
From: Jean-Pierre Delange @ 2020-07-04 8:33 UTC (permalink / raw)
To: ntg-context
Hello List !
Quite a dummy question : usually I use to work with CTX on Linux, but I
want to install it on Windows 10 x64. Unfortunately, while I download
CTX packages from there http://standalone.contextgarden.net/setup2/ ,
the next step fails : C:\Users\adeim\Documents\context>first-setup.bat
The command returns this : "'rsync' is unknown as an internal or
external command, an executable program or a command file. 'mtxrun' is
not known as an internal or external command, an executable program or a
command file."
The same thing occurs while context is as a file at the very root on
c:\, or within c:\windows, or in C:\Users\adeim\Documents\context>
Many thanks for your help !
JP
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ConTeXt installation on Windows 10
2020-07-04 8:33 ` ConTeXt installation on Windows 10 Jean-Pierre Delange
@ 2020-07-04 8:47 ` Wolfgang Schuster
2020-07-04 10:18 ` Jean-Pierre Delange
0 siblings, 1 reply; 8+ messages in thread
From: Wolfgang Schuster @ 2020-07-04 8:47 UTC (permalink / raw)
To: mailing list for ConTeXt users
Jean-Pierre Delange schrieb am 04.07.2020 um 10:33:
> Hello List !
>
> Quite a dummy question : usually I use to work with CTX on Linux, but
> I want to install it on Windows 10 x64. Unfortunately, while I
> download CTX packages from there
> http://standalone.contextgarden.net/setup2/ , the next step fails :
> C:\Users\adeim\Documents\context>first-setup.bat
>
> The command returns this : "'rsync' is unknown as an internal or
> external command, an executable program or a command file. 'mtxrun' is
> not known as an internal or external command, an executable program or
> a command file."
>
> The same thing occurs while context is as a file at the very root on
> c:\, or within c:\windows, or in C:\Users\adeim\Documents\context>
I suggest to use the installer from Hans website when you don't need pdfTeX.
http://www.pragma-ade.nl/install.htm
Wolfgang
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ConTeXt installation on Windows 10
2020-07-04 8:47 ` Wolfgang Schuster
@ 2020-07-04 10:18 ` Jean-Pierre Delange
2020-07-04 13:09 ` Hans Hagen
0 siblings, 1 reply; 8+ messages in thread
From: Jean-Pierre Delange @ 2020-07-04 10:18 UTC (permalink / raw)
To: ntg-context
Thank you Wolfgang ! It works nicely.
Maybe it would be useful to change the wiki on ConteXt Garden (and other
places where there is CTX help stuff ...). Such pages as here :
https://wiki.contextgarden.net/ConTeXt_Standalone#Installation
JP
Le 04/07/2020 à 10:47, Wolfgang Schuster a écrit :
> Jean-Pierre Delange schrieb am 04.07.2020 um 10:33:
>> Hello List !
>>
>> Quite a dummy question : usually I use to work with CTX on Linux, but
>> I want to install it on Windows 10 x64. Unfortunately, while I
>> download CTX packages from there
>> http://standalone.contextgarden.net/setup2/ , the next step fails :
>> C:\Users\adeim\Documents\context>first-setup.bat
>>
>> The command returns this : "'rsync' is unknown as an internal or
>> external command, an executable program or a command file. 'mtxrun'
>> is not known as an internal or external command, an executable
>> program or a command file."
>>
>> The same thing occurs while context is as a file at the very root on
>> c:\, or within c:\windows, or in C:\Users\adeim\Documents\context>
>
> I suggest to use the installer from Hans website when you don't need
> pdfTeX.
>
> http://www.pragma-ade.nl/install.htm
>
> Wolfgang
>
> ___________________________________________________________________________________
>
> If your question is of interest to others as well, please add an entry
> to the Wiki!
>
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage : http://www.pragma-ade.nl / http://context.aanhet.net
> archive : https://bitbucket.org/phg/context-mirror/commits/
> wiki : http://contextgarden.net
> ___________________________________________________________________________________
>
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: ConTeXt installation on Windows 10
2020-07-04 10:18 ` Jean-Pierre Delange
@ 2020-07-04 13:09 ` Hans Hagen
0 siblings, 0 replies; 8+ messages in thread
From: Hans Hagen @ 2020-07-04 13:09 UTC (permalink / raw)
To: mailing list for ConTeXt users, Jean-Pierre Delange
On 7/4/2020 12:18 PM, Jean-Pierre Delange wrote:
> Thank you Wolfgang ! It works nicely.
>
> Maybe it would be useful to change the wiki on ConteXt Garden (and other
> places where there is CTX help stuff ...). Such pages as here :
> https://wiki.contextgarden.net/ConTeXt_Standalone#Installation
At some point the installation will happen from the garden but we need
to set it up (something being discussed).
Hans
-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Multiple cases of unexpected behaviour in luametatex
2020-07-04 5:22 ` Marcel Fabian Krüger
2020-07-04 8:33 ` ConTeXt installation on Windows 10 Jean-Pierre Delange
@ 2020-07-04 17:18 ` Hans Hagen
1 sibling, 0 replies; 8+ messages in thread
From: Hans Hagen @ 2020-07-04 17:18 UTC (permalink / raw)
To: mailing list for ConTeXt users
On 7/4/2020 7:22 AM, Marcel Fabian Krüger wrote:
> Anyway, tehre is another thing I am wondering about: In the latest
> luametatex versions, error help strings do not seem to work like they used to.
> Especially after \errhelp{some help}\errmessage{error} or
> tex.error('fdfd', {'some help'}) in the 'handle_error_hook'
> callback, tex.gethelptext returns nil. gethelptext still works for
> errors triggered by TeX itself.
The error code has been rewritten but that already happened long ago,
and i never came to hooking the errhelp into that (rather trivial but we
never use that in context anyway), I'll look at it.
> One completely different question:
> When running luametatex (of course with appropriate format/lua init
> script options) but without a TeX file or command, luametatex does not
> expect interactive input but instead tries to read the input file
> 'luametatex.tex'. (Where luametatex does not seem to be hardcoded but
> derived from argv[0]) Is there a way to overwrite this default input
> file name?
No, as that is a side effect of the 'rest of the command line being read
after an injected \input' and that is not the case in luametatex. There
are a few command line arguments and --jobname is one of then. That bit
won't change but one can deal with that in Lua (there is an input line
callback so i might even drop the left over bits of those terminal
related features).
Hans
-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-07-04 17:18 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-03 12:55 Multiple cases of unexpected behaviour in luametatex Marcel Fabian Krüger
2020-07-03 19:56 ` Hans Hagen
2020-07-04 5:22 ` Marcel Fabian Krüger
2020-07-04 8:33 ` ConTeXt installation on Windows 10 Jean-Pierre Delange
2020-07-04 8:47 ` Wolfgang Schuster
2020-07-04 10:18 ` Jean-Pierre Delange
2020-07-04 13:09 ` Hans Hagen
2020-07-04 17:18 ` Multiple cases of unexpected behaviour in luametatex Hans Hagen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).