underbraces get progressively more misaligned (w/ minimal example)

underbraces get progressively more misaligned (w/ minimal example)

[Akira Kakuto]

> It does also depend on the width of the text because when I use 48pt in the example below
> the height of the brace is always the same but with 49pt the height increases after the first page.

An experiment:

width  = 48.0719985pt
height = 4.968pt (constant)

width  = 48.0719986pt
height = unpredictable various values

Best regards,
Akira

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Hans Hagen]

On 7/4/2014 9:12 AM, Akira Kakuto wrote:
>> It does also depend on the width of the text because when I use 48pt
>> in the example below
>> the height of the brace is always the same but with 49pt the height
>> increases after the first page.
>
> An experiment:
>
> width  = 48.0719985pt
> height = 4.968pt (constant)
>
> width  = 48.0719986pt
> height = unpredictable various values

So now we need to figure out where it comes from; maybe some math 
parameter get messed up ... it has to do with the mechanism that builds 
extensibles from snippets (begin - connector - middle - connector - end 
glyphs). Depending on the width this goes in steps: first fixed prebuild 
glyphs, later extensibles.

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

Dear list,

As I copyedited my book, I noticed examples where the underbraces in
math formulas were not aligned vertically, especially toward the end of
the book.  Eventually I realized it was not due to my growing
copyediting skill, and I have made the following minimal example showing
the problem (I am using 2014.05.17 MkIV):

\setupformulas[indentnext=auto, spacebefore=none, spaceafter=none]

\starttext

\dorecurse{3000}{
\placeformula\startformula
\underbrace{\hbox{yyyy}}_{\rho} =
\underbrace{\hbox{yyyy yyyy yyyy}}_{\rho}
\stopformula
}

\stoptext

It'll produce a 162-page file.  On the formulas on page 1, the two
underbraces in each equation are aligned.  But there seems to be some
kind of slow leak, because the underbraces become ever more misaligned.
By page 162, they are visibly misaligned by maybe 3pt.  If I increase
the recursion count to 30000, on page 2129 the misalignment is about 30pt.

Taking out the \placeformula makes the misalignment go away.

I tried to reproduce the problem with plain LuaTeX, using the tex file
below, but the underbraces came out aligned.

\newcount\foo
\foo=3000
\loop
$$
\underbrace{\hbox{yyyy}}_{\rho} =
\underbrace{\hbox{yyyy yyyy yyyy}}_{\rho}
$$
\advance \foo -1
\ifnum \foo>0
\repeat

\end

Any thoughts on this very strange issue?

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

Here is a 72dpi rendering of the last page (with recursion count=30000).

-Sanjoy


[-- Attachment #2: page with large misalignments --]
[-- Type: image/png, Size: 19406 bytes --]

[-- Attachment #3: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

I thought the problem might lie with the general
\definemathunderextensible (math-stc.mkvi), because underbrace is
defined as

\definemathunderextensible  [vfenced] [underbrace]    ["FE3DF]          % ["023DF]

Other underextensibles include \underbar:

\definemathunderextensible  [vfenced] [underbar]      ["FE33F]
% ["203E]

So I tried my example replacing \underbrace with \underbar but the
alignment comes out fine (and \overbrace works fine too).  So that
debugging idea didn't work, except to indicate that something special
about underbrace isn't working right.

Does anyone have suggestions for further debugging ideas?

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Wolfgang Schuster]

Am 03.07.2014 um 12:49 schrieb Sanjoy Mahajan <sanjoy@MIT.EDU>:

> I thought the problem might lie with the general
> \definemathunderextensible (math-stc.mkvi), because underbrace is
> defined as
> 
> \definemathunderextensible  [vfenced] [underbrace]    ["FE3DF]          % ["023DF]
> 
> Other underextensibles include \underbar:
> 
> \definemathunderextensible  [vfenced] [underbar]      ["FE33F]
> % ["203E]
> 
> So I tried my example replacing \underbrace with \underbar but the
> alignment comes out fine (and \overbrace works fine too).  So that
> debugging idea didn't work, except to indicate that something special
> about underbrace isn't working right.
> 
> Does anyone have suggestions for further debugging ideas?

math-stc.mkvi:

\unexpanded\def\math_stackers_make_double#top#bottom#category#codepoint#codeextra#text%
  {\begingroup
   \edef\currentmathstackers{#category}%
   \mathstackersparameter\c!left\relax
   \ifmmode\math_class_by_parameter\mathstackersparameter\else\dontleavehmode\fi
     {\edef\currentmathstackers{#category}%
      \edef\m_math_stackers_text_middle {#text}%
      %
      ...
      %
-     \ifx\currentmathtext\empty
+     \ifx\m_math_stackers_text_middle\empty
        \setbox\scratchboxthree\emptyhbox
      \else
        \setmathtextbox\scratchboxthree\hbox{\math_stackers_middletext}%
      \fi
      ...
      \fi}%
  \mathstackersparameter\c!right\relax
  \edef\p_limits{\mathstackersparameter\c!mathlimits}%
  \ifx\p_limits\v!yes
    \expandafter\endgroup\expandafter\limits
  \else
    \expandafter\endgroup
  \fi}

Wolfgang
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

> math-stc.mkvi:
>
> \unexpanded\def\math_stackers_make_double#top#bottom#category#codepoint#codeextra#text%
>       ...
> -     \ifx\currentmathtext\empty
> +     \ifx\m_math_stackers_text_middle\empty
>         \setbox\scratchboxthree\emptyhbox
>       \else

Thanks for looking into this, Wolfgang.

I made this change to math-stc.mkvi in the 2014.05.17 beta, then remade
the MkIV format with

  mtxrun --selfupdate
  mtxrun --generate
  luatools --generate
  context --make

Then I ran context on my test file (with 30000 as the \dorecurse count).

However, the offset underbraces remain.  I'm attaching a image from
p. 2128, by which point the offset is 20 or 30pt.  I am fairly sure I am
running the newly created version, because remaking the format included
these lines:

  open source     > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi
  loading         > ConTeXt Math Macros / Stackers
  close source    > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi

and the log output from running the minimal.tex shows that the the fmt
is indeed dated 2014.7.3:

  mtx-context     | run 1: luatex --fmt="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en" --jobname="minimal" --lua="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en.lui" --no-parse-first-line --c:currentrun=1 --c:fulljobname="./minimal.tex" --c:input="./minimal.tex" --c:kindofrun=1 --c:maxnofruns=8 "cont-yes.mkiv"
  This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/dev) (rev 4972) 
   \write18 enabled.
  open source     > 1 > 1 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/cont-yes.mkiv

  ConTeXt  ver: 2014.05.17 23:46 MKIV beta  fmt: 2014.7.3  int: english/english


-Sanjoy


[-- Attachment #2: page 2128 (PNG) --]
[-- Type: image/png, Size: 13724 bytes --]

[-- Attachment #3: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Hans Hagen]

On 7/3/2014 11:03 PM, Sanjoy Mahajan wrote:
>> math-stc.mkvi:
>>
>> \unexpanded\def\math_stackers_make_double#top#bottom#category#codepoint#codeextra#text%
>>        ...
>> -     \ifx\currentmathtext\empty
>> +     \ifx\m_math_stackers_text_middle\empty
>>          \setbox\scratchboxthree\emptyhbox
>>        \else
>
> Thanks for looking into this, Wolfgang.
>
> I made this change to math-stc.mkvi in the 2014.05.17 beta, then remade
> the MkIV format with
>
>    mtxrun --selfupdate
>    mtxrun --generate
>    luatools --generate
>    context --make

context --make does a --generate so you can skip that and the luatools 
run is not needed either (selfupdate is seldom needed too)

> Then I ran context on my test file (with 30000 as the \dorecurse count).
>
> However, the offset underbraces remain.  I'm attaching a image from
> p. 2128, by which point the offset is 20 or 30pt.  I am fairly sure I am
> running the newly created version, because remaking the format included
> these lines:
>
>    open source     > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi
>    loading         > ConTeXt Math Macros / Stackers
>    close source    > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi
>
> and the log output from running the minimal.tex shows that the the fmt
> is indeed dated 2014.7.3:
>
>    mtx-context     | run 1: luatex --fmt="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en" --jobname="minimal" --lua="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en.lui" --no-parse-first-line --c:currentrun=1 --c:fulljobname="./minimal.tex" --c:input="./minimal.tex" --c:kindofrun=1 --c:maxnofruns=8 "cont-yes.mkiv"
>    This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/dev) (rev 4972)
>     \write18 enabled.
>    open source     > 1 > 1 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/cont-yes.mkiv
>
>    ConTeXt  ver: 2014.05.17 23:46 MKIV beta  fmt: 2014.7.3  int: english/english

we get a .5pt height difference somewhere (nothing else different in a 
trace) ... smells like a luatex bug (testing that now)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

> context --make does a --generate so you can skip that and the luatools 
> run is not needed either (selfupdate is seldom needed too)

So 'context --make' is all one needs?  I'll update the wiki.

> we get a .5pt height difference somewhere (nothing else different in a 
> trace) ... smells like a luatex bug (testing that now)

That is my suspicion as well.  But I couldn't reproduce it with plain
luatex.  Though maybe I needed to use equation numbers, however that is
done in plain tex (the context run doesn't show the problem without the
\placeformula).

I suspect some internal state is getting slightly corrupted and getting
progressively worse with the 30,000 repetitions.

I've been finding many unexplained 0.5pt or so differences, which may be
related to the underbrace problem.  The context, so to speak, is that I
am putting in the last edits to the book, after having gone through and
checked every page many times.  So I want to ensure that none of the
small changes has a big effect, but I don't want to check every page
again (or I'll vomit).

Thus, I run my compare-pdfs.sh script on the previous and the latest
pdf. It renders each page to a B/W png (using mupdf) and then compares
corresponding pages using 'compare' (one of the ImageMagick tools).  Any
differing pixels are in red, and the shared pixels are in very light
gray.  Then all the pages with enough differences are viewed with 'feh',
in order of most-to-least differences.  And I check that nothing major
has changed.

What I keep finding is that pages long after where I make a small change
somehow get changed, with roughly one-pixel shifts to parts of the page.
I'm attaching an example pixel diff.  It shows changes on p.72 of the
mss.  However, the only change to the source code was on p.58 (and was
designed not to change any page breaks, which it didn't).  That same
page (p.72) often gets similar changes with all kinds of small changes.

[Note to me: In case I need to recreate the source for debugging
purposes, it's the change from rev 41ff54 to 2467d5]

In case the script is useful to others (earlier versions are in the
contexttest repository somewhere), I am attaching it.  I use it as
follows with a bash command line.  (The awk in the pipeline makes sure
that only pages with some difference make it to the viewer.)

DPI=72 compare-pdfs.sh old.pdf new.pdf | tee book-print.compare72dpi | awk '{if (($3 > 0.02)) print;}' | sort -nr -k3 | awk '{print $1}' | xargs -r feh

-Sanjoy


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: compare script --]
[-- Type: text/x-sh, Size: 1210 bytes --]

#! /bin/bash

# Usage: $0 file1.pdf file2.pdf
#
#   compares file1.pdf and file2.pdf by rendering each page and using
#   the 'compare' ImageMagick utility
#
# Copyright 2007-2014 Sanjoy Mahajan.  Licensed under the GNU GPL version 3
# or (at your option) any later version.
#
# HISTORY
#   2014-06-22: Use mudraw instead of pdftoppm.  GPL v3+
#   2009-09-30: Fix capture of dB output; don't use a viewer; use pdftoppm
#   2007-01-15: First version
#

if [ -z "$DPI" ]; then
    DPI=72
fi
ext=png

if [ -z "$1" -o -z "$2" ]; then
  echo "Usage: $0 file1.pdf file2.pdf"
  exit 3
fi

# generate the many page images in a temporary directory
d=`mktemp -d`
mkdir -p $d/a $d/b
mudraw -r $DPI -g -o $d/a/%03d.$ext $1
mudraw -r $DPI -g -o $d/b/%03d.$ext $2
wait

# find the union of the page numbers (in case one pdf has more pages)
pages=`ls $d/{a,b}/*.$ext | sed "s%.*/\([0-9][0-9]*\).$ext%\1%" | sort -un`
# compare each page
for p in $pages ; do
  if ! [ -e "$d/a/$p.$ext" ] ; then
    echo "$p: missing from $1"
    continue
  fi
  if ! [ -e "$d/b/$p.$ext" ] ; then
    echo "$p: missing from $2"
    continue
  fi
  echo -n "$d/diff-$p.$ext $p "
  compare -metric mae $d/{a,b}/$p.$ext $d/diff-$p.$ext 2>&1
done

[-- Attachment #3: pixel diff --]
[-- Type: image/png, Size: 35769 bytes --]

[-- Attachment #4: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 2968 bytes --]

On Fri, Jul 4, 2014 at 12:37 AM, Sanjoy Mahajan <sanjoy@mit.edu> wrote:

> > context --make does a --generate so you can skip that and the luatools
> > run is not needed either (selfupdate is seldom needed too)
>
> So 'context --make' is all one needs?  I'll update the wiki.
>
> > we get a .5pt height difference somewhere (nothing else different in a
> > trace) ... smells like a luatex bug (testing that now)
>
> That is my suspicion as well.  But I couldn't reproduce it with plain
> luatex.  Though maybe I needed to use equation numbers, however that is
> done in plain tex (the context run doesn't show the problem without the
> \placeformula).
>
> I suspect some internal state is getting slightly corrupted and getting
> progressively worse with the 30,000 repetitions.
>
> I've been finding many unexplained 0.5pt or so differences, which may be
> related to the underbrace problem.  The context, so to speak, is that I
> am putting in the last edits to the book, after having gone through and
> checked every page many times.  So I want to ensure that none of the
> small changes has a big effect, but I don't want to check every page
> again (or I'll vomit).
>
> Thus, I run my compare-pdfs.sh script on the previous and the latest
> pdf. It renders each page to a B/W png (using mupdf) and then compares
> corresponding pages using 'compare' (one of the ImageMagick tools).  Any
> differing pixels are in red, and the shared pixels are in very light
> gray.  Then all the pages with enough differences are viewed with 'feh',
> in order of most-to-least differences.  And I check that nothing major
> has changed.
>
> What I keep finding is that pages long after where I make a small change
> somehow get changed, with roughly one-pixel shifts to parts of the page.
> I'm attaching an example pixel diff.  It shows changes on p.72 of the
> mss.  However, the only change to the source code was on p.58 (and was
> designed not to change any page breaks, which it didn't).  That same
> page (p.72) often gets similar changes with all kinds of small changes.
>
> [Note to me: In case I need to recreate the source for debugging
> purposes, it's the change from rev 41ff54 to 2467d5]
>
> In case the script is useful to others (earlier versions are in the
> contexttest repository somewhere), I am attaching it.  I use it as
> follows with a bash command line.  (The awk in the pipeline makes sure
> that only pages with some difference make it to the viewer.)
>
> DPI=72 compare-pdfs.sh old.pdf new.pdf | tee book-print.compare72dpi | awk
> '{if (($3 > 0.02)) print;}' | sort -nr -k3 | awk '{print $1}' | xargs -r feh
>
> Thank you for the script.
Under linux you can also try diffpdf
$> aptitude search diffpdf
i   diffpdf
                                                        - compare two PDF
files textually or visually

p   diffpdf:i386
                                                         - compare two PDF
files textually or visually


-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 4161 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

luigi scarso <luigi.scarso@gmail.com> writes:

> Under linux you can also try diffpdf

Thanks -- I'd forgotten about that and just tried it.  If you are
looking for pixel-level changes, my script makes it easier to see
exactly what the changes are.  On the other hand, diffpdf is more
polished than my script.

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Hans Hagen]

On 7/4/2014 12:37 AM, Sanjoy Mahajan wrote:
>> context --make does a --generate so you can skip that and the luatools
>> run is not needed either (selfupdate is seldom needed too)
>
> So 'context --make' is all one needs?  I'll update the wiki.
>
>> we get a .5pt height difference somewhere (nothing else different in a
>> trace) ... smells like a luatex bug (testing that now)
>
> That is my suspicion as well.  But I couldn't reproduce it with plain
> luatex.  Though maybe I needed to use equation numbers, however that is
> done in plain tex (the context run doesn't show the problem without the
> \placeformula).
>
> I suspect some internal state is getting slightly corrupted and getting
> progressively worse with the 30,000 repetitions.

here it happens after a pagebreak (a few lines overflow are ok)

> I've been finding many unexplained 0.5pt or so differences, which may be
> related to the underbrace problem.  The context, so to speak, is that I
> am putting in the last edits to the book, after having gone through and
> checked every page many times.  So I want to ensure that none of the
> small changes has a big effect, but I don't want to check every page
> again (or I'll vomit).
>
> Thus, I run my compare-pdfs.sh script on the previous and the latest
> pdf. It renders each page to a B/W png (using mupdf) and then compares
> corresponding pages using 'compare' (one of the ImageMagick tools).  Any
> differing pixels are in red, and the shared pixels are in very light
> gray.  Then all the pages with enough differences are viewed with 'feh',
> in order of most-to-least differences.  And I check that nothing major
> has changed.
>
> What I keep finding is that pages long after where I make a small change
> somehow get changed, with roughly one-pixel shifts to parts of the page.
> I'm attaching an example pixel diff.  It shows changes on p.72 of the
> mss.  However, the only change to the source code was on p.58 (and was
> designed not to change any page breaks, which it didn't).  That same
> page (p.72) often gets similar changes with all kinds of small changes.
>
> [Note to me: In case I need to recreate the source for debugging
> purposes, it's the change from rev 41ff54 to 2467d5]
>
> In case the script is useful to others (earlier versions are in the
> contexttest repository somewhere), I am attaching it.  I use it as
> follows with a bash command line.  (The awk in the pipeline makes sure
> that only pages with some difference make it to the viewer.)
>
> DPI=72 compare-pdfs.sh old.pdf new.pdf | tee book-print.compare72dpi | awk '{if (($3 > 0.02)) print;}' | sort -nr -k3 | awk '{print $1}' | xargs -r feh
>
> -Sanjoy
>


-- 

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Wolfgang Schuster]

[-- Attachment #1.1: Type: text/plain, Size: 1438 bytes --]


Am 04.07.2014 um 00:51 schrieb Hans Hagen <pragma@wxs.nl>:

> On 7/4/2014 12:37 AM, Sanjoy Mahajan wrote:
>>> context --make does a --generate so you can skip that and the luatools
>>> run is not needed either (selfupdate is seldom needed too)
>> 
>> So 'context --make' is all one needs?  I'll update the wiki.
>> 
>>> we get a .5pt height difference somewhere (nothing else different in a
>>> trace) ... smells like a luatex bug (testing that now)
>> 
>> That is my suspicion as well.  But I couldn't reproduce it with plain
>> luatex.  Though maybe I needed to use equation numbers, however that is
>> done in plain tex (the context run doesn't show the problem without the
>> \placeformula).
>> 
>> I suspect some internal state is getting slightly corrupted and getting
>> progressively worse with the 30,000 repetitions.
> 
> here it happens after a pagebreak (a few lines overflow are ok)


It does also depend on the width of the text because when I use 48pt in the example below
the height of the brace is always the same but with 49pt the height increases after the first page.

\starttext

\dorecurse{100}
  {\startformula
   \mathord
     {%\setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip48pt}}%
      \setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip49pt}}%
      \the\userpageno:\the\ht\scratchbox}%
   \stopformula}

\stoptext

Wolfgang

[-- Attachment #1.2: Type: text/html, Size: 2454 bytes --]

[-- Attachment #2: Type: text/plain, Size: 485 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Hans Hagen]

On 7/4/2014 6:58 AM, Wolfgang Schuster wrote:
>
> Am 04.07.2014 um 00:51 schrieb Hans Hagen <pragma@wxs.nl
> <mailto:pragma@wxs.nl>>:
>
>> On 7/4/2014 12:37 AM, Sanjoy Mahajan wrote:
>>>> context --make does a --generate so you can skip that and the luatools
>>>> run is not needed either (selfupdate is seldom needed too)
>>>
>>> So 'context --make' is all one needs?  I'll update the wiki.
>>>
>>>> we get a ..5pt height difference somewhere (nothing else different in a
>>>> trace) .... smells like a luatex bug (testing that now)
>>>
>>> That is my suspicion as well.  But I couldn't reproduce it with plain
>>> luatex.  Though maybe I needed to use equation numbers, however that is
>>> done in plain tex (the context run doesn't show the problem without the
>>> \placeformula).
>>>
>>> I suspect some internal state is getting slightly corrupted and getting
>>> progressively worse with the 30,000 repetitions.
>>
>> here it happens after a pagebreak (a few lines overflow are ok)
>
> It does also depend on the width of the text because when I use 48pt in
> the example below
> the height of the brace is always the same but with 49pt the height
> increases after the first page.
>
> \starttext
>
> \dorecurse{100}
>    {\startformula
>     \mathord
>
>   {%\setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip48pt}}%
>
> \setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip49pt}}%
>        \the\userpageno:\the\ht\scratchbox}%
>     \stopformula}
>
> \stoptext

\starttext

\dorecurse{1000}
   {\setbox\scratchbox\hbox{$\Umathaccent\fam\zerocount"FE3DF{\hskip49pt}$}%
    \the\ht\scratchbox\par}

\stoptext

that was the one i used when seeing some trigger from the page break 
(the two at the top of page 2 are the overshoot before tex triggers the 
page)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

Wolfgang Schuster <schuster.wolfgang@gmail.com> writes:

> It does also depend on the width of the text because when I use 48pt
> in the example below the height of the brace is always the same but
> with 49pt the height increases after the first page.

An interesting bug.  That's also consistent with my experiments, in which

  \underbrace{\hbox{yyyy yyyy}}_{\rho} 

triggers the problem, but 

  \underbrace{\hbox{yyyy}}_{\rho}

does not.

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Hans Hagen]

On 7/4/2014 11:34 AM, Sanjoy Mahajan wrote:
> Wolfgang Schuster <schuster.wolfgang@gmail.com> writes:
>
>> It does also depend on the width of the text because when I use 48pt
>> in the example below the height of the brace is always the same but
>> with 49pt the height increases after the first page.
>
> An interesting bug.  That's also consistent with my experiments, in which
>
>    \underbrace{\hbox{yyyy yyyy}}_{\rho}
>
> triggers the problem, but
>
>    \underbrace{\hbox{yyyy}}_{\rho}
>
> does not.

Hi Sanjoy,

After a day of testing, tracing, wondering etc Luigi and I figured out 
what was the problem it tooks us lots of debug prints -). Indeed there 
is a bug in luatex. An extensible is made up of a sequence

char
skip 0pt- xpt
char
skip 0pt- xpt
char
skip 0pt- xpt
char

And the code that determines the height/depth took the dimensions of 
skips into the equation while in fact glue has no such dimensions 
(leaders that are a special kind of glue have dimensions). Not the first 
place one looks for such an issue. Probably a side effect of copying a 
function and patching part.

So, some random value (not entirely random as it depends on what mem 
area the glue node is allocated from and that is kind of stable in a 
run) was added which results in the effects you noticed.

Because it is stable within a run (same successive errors) it doesn't 
result in oscillating (too many runs due to differences in results) and 
this is probably why it went unnoticed so long. Interesting. (Normally 
the pseudo random ht/dp were small.)

So, once this fix moves upstream you can check your suspicious cases. 
Luigi will push it in our experimental branch and I suppose that Taco 
will then push it into trunk and after that you can compile or wait till 
the garden carries it.

Thanks for noticing!

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

Hans Hagen <pragma@wxs.nl> writes:

> After a day of testing, tracing, wondering etc Luigi and I figured out 
> what was the problem it tooks us lots of debug prints -). Indeed there 
> is a bug in luatex. 

Great detective work!

> So, some random value (not entirely random as it depends on what mem 
> area the glue node is allocated from and that is kind of stable in a 
> run) was added which results in the effects you noticed.

Ah, so running it 30000 times would allocate the glue node from higher
and higher mem, and thereby making the not-so-random value larger and
larger.

> So, once this fix moves upstream you can check your suspicious cases. 
> Luigi will push it in our experimental branch and I suppose that Taco 
> will then push it into trunk and after that you can compile

Once it in the luatex trunk, I will check it right away.  I am also
happy to test the experimental branch if you give me a pointer to it.

> or wait till the garden carries it.

Do you mean in the minimals?

> Thanks for noticing!

For sure.  I am very happy that it is fixed, and fixed before I have to
give in my absolute final PDF file.  

My only worry is that the carefully tuned page breaks, some of which are
sensitive to 1pt insertions of vertical space, might change once the
underbraces return to their homes.  But I'll tweak it with a bit of
vertical space.

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

> Once it in the luatex trunk, I will check it right away.

I just compiled and installed luatex 0.79.1 r5033, remade the formats,
and the result passes my underbrace tests (the 30,000 iterations and the
examples in my book mss).

Thank you for the great debugging.

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

Sanjoy Mahajan <sanjoy@mit.edu> writes:

> I've been finding many unexplained 0.5pt or so differences, which may
> be related to the underbrace problem. ...  What I keep finding is that
> pages long after where I make a small change somehow get changed, with
> roughly one-pixel shifts to parts of the page.

These small shifts were also caused by the underbrace luatex bug (these
pages all had underbraces on them)---many birds with one stone.

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: underbraces get progressively more misaligned (w/ minimal example)

[Sanjoy Mahajan]

Wolfgang Schuster <schuster.wolfgang@gmail.com> writes:

> math-stc.mkvi:
>
>       %
> -     \ifx\currentmathtext\empty
> +     \ifx\m_math_stackers_text_middle\empty
>         \setbox\scratchboxthree\emptyhbox

Is this change still also needed (in addition to the luatex fix)?

-Sanjoy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : http://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________