randomizer line-breaking

randomizer line-breaking

[Steffen Wolfrum]

Hi Hans,

is there some kind of randomizer involved in line-breaking?

I have a 458 pages project, a book of humanities.

When I immediately typeset (engine=luatex) it a second time, the resulting PDF differs on 14 pages, diffpdf-compared to the original PDF
when I typeset it a third time, the resulting PDF differs on 7 pages,
when I typeset it a forth time, the resulting PDF differs on 12 pages,
when I typeset it a fifth time, the resulting PDF differs on 0 pages.

"differs" means that there a paragraphs (mostly footnotes, rarely in the main text) that show different line-breaking: This doesn't mean wrong hyphenation, just *different* hyphenation, different each time.

As you can imagine, I can't provide a minimal example for this, only the entire project-folder and the entire context-standalone.

Steffen


___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/7/2021 4:32 PM, Steffen Wolfrum wrote:
> 
> Hi Hans,
> 
> is there some kind of randomizer involved in line-breaking?
> 
> I have a 458 pages project, a book of humanities.
> 
> When I immediately typeset (engine=luatex) it a second time, the resulting PDF differs on 14 pages, diffpdf-compared to the original PDF
> when I typeset it a third time, the resulting PDF differs on 7 pages,
> when I typeset it a forth time, the resulting PDF differs on 12 pages,
> when I typeset it a fifth time, the resulting PDF differs on 0 pages.
> 
> "differs" means that there a paragraphs (mostly footnotes, rarely in the main text) that show different line-breaking: This doesn't mean wrong hyphenation, just *different* hyphenation, different each time.
> 
> As you can imagine, I can't provide a minimal example for this, only the entire project-folder and the entire context-standalone.
Not that I know of ... but you can run wiht --keeptuc and see if there 
are differences in the tuc file that create some oscillation.

Also, as you mention luatex: there have been no (nor will be) 
fundamental changes to the engine because that would harm 'long term 
stability' as well as not play well with macro packages that depend 'no 
changes' (otherwise an update in engine for an existing setup e.g. older 
texlive, could fail).

So, a change in hyphenation only can happen when there's is something 
different in the node lists (something that influences for instance what 
is seen as boundary of a character). When lua is involved a difference 
can be in hashing (which is random per run) but that seems unlikely here.

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Steffen Wolfrum]

> Am 07.06.2021 um 16:39 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> 
> Not that I know of ... but you can run wiht --keeptuc and see if there are differences in the tuc file that create some oscillation.


Ok. Each run now creates 4 additional files:

file_name-tuc-02.tmp
file_name-tuc-03.tmp
file_name-tuc-04.tmp
file_name-tuc-05.tmp

Is this right?

Steffen
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/7/2021 5:44 PM, Steffen Wolfrum wrote:
> 
> 
>> Am 07.06.2021 um 16:39 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>
>> Not that I know of ... but you can run wiht --keeptuc and see if there are differences in the tuc file that create some oscillation.
> 
> 
> Ok. Each run now creates 4 additional files:
> 
> file_name-tuc-02.tmp
> file_name-tuc-03.tmp
> file_name-tuc-04.tmp
> file_name-tuc-05.tmp
> 
> Is this right?
indeed and you can compare them to get some clue about what changes

normally you get one so your style somehow triggers differences

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Steffen Wolfrum]

> Am 07.06.2021 um 17:59 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> 
> On 6/7/2021 5:44 PM, Steffen Wolfrum wrote:
>>> Am 07.06.2021 um 16:39 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>> 
>>> Not that I know of ... but you can run wiht --keeptuc and see if there are differences in the tuc file that create some oscillation.
>> Ok. Each run now creates 4 additional files:
>> file_name-tuc-02.tmp
>> file_name-tuc-03.tmp
>> file_name-tuc-04.tmp
>> file_name-tuc-05.tmp
>> Is this right?
> indeed and you can compare them to get some clue about what changes
> 
> normally you get one so your style somehow triggers differences


Besides plenty of numbers that differ, there are also entire lines that differ:

One frequent difference for examples is line 2 (macros) and 3 (randomseed) in this:

utilitydata.job.variables.collected={
["macros"]={},
["randomseed"]=0x1.c14cd54cp-1,
["sectionblockorder"]={ "frontpart", "bodypart", "backpart" },


Another difference that pops up 1000 times is this line
["used"]="fit",

in situation like this:
utilitydata.structures.references.collected={
["02"]={
 ["*101"]={
  ["metadata"]=678,
  ["references"]={
   ["block"]="bodypart",
   ["internal"]=102,
   ["prefix"]="02",
   ["realpage"]=16,
   ["reference"]="*101",
   ["section"]=3,
   ["used"]="fit",
   ["view"]="fit",
   ["x"]=25526152,
   ["y"]=36448045,
  },
 },



Does this help?

PS: The entire problem began when I started using hyphenation-exceptions...

Steffen
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/8/2021 12:18 PM, Steffen Wolfrum wrote:
> 
> 
>> Am 07.06.2021 um 17:59 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>
>> On 6/7/2021 5:44 PM, Steffen Wolfrum wrote:
>>>> Am 07.06.2021 um 16:39 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>>>
>>>> Not that I know of ... but you can run wiht --keeptuc and see if there are differences in the tuc file that create some oscillation.
>>> Ok. Each run now creates 4 additional files:
>>> file_name-tuc-02.tmp
>>> file_name-tuc-03.tmp
>>> file_name-tuc-04.tmp
>>> file_name-tuc-05.tmp
>>> Is this right?
>> indeed and you can compare them to get some clue about what changes
>>
>> normally you get one so your style somehow triggers differences
> 
> 
> Besides plenty of numbers that differ, there are also entire lines that differ:
> 
> One frequent difference for examples is line 2 (macros) and 3 (randomseed) in this:
> 
> utilitydata.job.variables.collected={
> ["macros"]={},
> ["randomseed"]=0x1.c14cd54cp-1,
> ["sectionblockorder"]={ "frontpart", "bodypart", "backpart" },
that randomseed should be the same between runs (unless one wipes the 
tuc file) so are you doing something with random?

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Steffen Wolfrum]

> Am 08.06.2021 um 15:01 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> 
> On 6/8/2021 12:18 PM, Steffen Wolfrum wrote:
>>> Am 07.06.2021 um 17:59 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>> 
>>> On 6/7/2021 5:44 PM, Steffen Wolfrum wrote:
>>>>> Am 07.06.2021 um 16:39 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>>>> 
>>>>> Not that I know of ... but you can run wiht --keeptuc and see if there are differences in the tuc file that create some oscillation.
>>>> Ok. Each run now creates 4 additional files:
>>>> file_name-tuc-02.tmp
>>>> file_name-tuc-03.tmp
>>>> file_name-tuc-04.tmp
>>>> file_name-tuc-05.tmp
>>>> Is this right?
>>> indeed and you can compare them to get some clue about what changes
>>> 
>>> normally you get one so your style somehow triggers differences
>> Besides plenty of numbers that differ, there are also entire lines that differ:
>> One frequent difference for examples is line 2 (macros) and 3 (randomseed) in this:
>> utilitydata.job.variables.collected={
>> ["macros"]={},
>> ["randomseed"]=0x1.c14cd54cp-1,
>> ["sectionblockorder"]={ "frontpart", "bodypart", "backpart" },
> that randomseed should be the same between runs (unless one wipes the tuc file) so are you doing something with random?


not that I know of. 

I searched through both, the project folder and my modules (in context standalone), both «random» nowhere was to be found.

perhaps I incidently use a command that utilizes random... what could this be?

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:

> perhaps I incidently use a command that utilizes random... what could this be?
mwe

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Steffen Wolfrum]

> Am 08.06.2021 um 15:30 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> 
> On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:
> 
>> perhaps I incidently use a command that utilizes random... what could this be?
> mwe

pardon?
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/8/2021 3:32 PM, Steffen Wolfrum wrote:
> 
> 
>> Am 08.06.2021 um 15:30 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>
>> On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:
>>
>>> perhaps I incidently use a command that utilizes random... what could this be?
>> mwe
> 
> pardon?
we need a mwe

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Aditya Mahajan]

On Tue, 8 Jun 2021, Steffen Wolfrum wrote:

> 
> 
> > Am 08.06.2021 um 15:30 schrieb Hans Hagen <j.hagen@xs4all.nl>:
> > 
> > On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:
> > 
> >> perhaps I incidently use a command that utilizes random... what could this be?
> > mwe

http://texfaq.org/FAQ-minxampl
https://tex.meta.stackexchange.com/questions/228/ive-just-been-asked-to-write-a-minimal-working-example-mwe-what-is-that

These are latex centric advice on creating a minimal working example but broadly applicable for context as well.

Aditya

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Steffen Wolfrum]

> Am 08.06.2021 um 16:16 schrieb Aditya Mahajan <adityam@umich.edu>:
> 
> On Tue, 8 Jun 2021, Steffen Wolfrum wrote:
> 
>> 
>> 
>>> Am 08.06.2021 um 15:30 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>> 
>>> On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:
>>> 
>>>> perhaps I incidently use a command that utilizes random... what could this be?
>>> mwe
> 
> http://texfaq.org/FAQ-minxampl
> https://tex.meta.stackexchange.com/questions/228/ive-just-been-asked-to-write-a-minimal-working-example-mwe-what-is-that
> 
> These are latex centric advice on creating a minimal working example but broadly applicable for context as well.



Before I posted my initial posting «randomizer line-breaking» I already spent days and weeks boiling down this phenomenon. Unfortunately the random output made it impossible  for me to make a smaller, but *working* (i.e. showing the bug) example. It just appears sometimes, sometimes it doesn't – in between two runs I changed nothing: «As you can imagine, I can't provide a minimal example for this, only the entire project-folder and the entire context-standalone.»

So my hope was that Hans (and Wolfgang et al.) know how things are chained together.
For example: Which ConTeXt command uses randomizer? I have no idea what goes on behind my user interface. Others maybe do.
Another example: The problem only appears when using start-/stopexceptions. Is there a connection between exceptions and random output on the level of low level code, maybe?

Steffen

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: randomizer line-breaking

[Hans Hagen]

On 6/8/2021 4:35 PM, Steffen Wolfrum wrote:
> 
> 
>> Am 08.06.2021 um 16:16 schrieb Aditya Mahajan <adityam@umich.edu>:
>>
>> On Tue, 8 Jun 2021, Steffen Wolfrum wrote:
>>
>>>
>>>
>>>> Am 08.06.2021 um 15:30 schrieb Hans Hagen <j.hagen@xs4all.nl>:
>>>>
>>>> On 6/8/2021 3:15 PM, Steffen Wolfrum wrote:
>>>>
>>>>> perhaps I incidently use a command that utilizes random... what could this be?
>>>> mwe
>>
>> http://texfaq.org/FAQ-minxampl
>> https://tex.meta.stackexchange.com/questions/228/ive-just-been-asked-to-write-a-minimal-working-example-mwe-what-is-that
>>
>> These are latex centric advice on creating a minimal working example but broadly applicable for context as well.
> 
> 
> 
> Before I posted my initial posting «randomizer line-breaking» I already spent days and weeks boiling down this phenomenon. Unfortunately the random output made it impossible  for me to make a smaller, but *working* (i.e. showing the bug) example. It just appears sometimes, sometimes it doesn't – in between two runs I changed nothing: «As you can imagine, I can't provide a minimal example for this, only the entire project-folder and the entire context-standalone.»
> 
> So my hope was that Hans (and Wolfgang et al.) know how things are chained together.
> For example: Which ConTeXt command uses randomizer? I have no idea what goes on behind my user interface. Others maybe do.
> Another example: The problem only appears when using start-/stopexceptions. Is there a connection between exceptions and random output on the level of low level code, maybe?
maybe you have two similar exceptions (hyphens removed), comment 
exceptions and see which one does the harm

all the rest is guesswork

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________