In comparison to LaTeX with LuaHBTeX, ConTeXt with LuaMetaTeX is
lightning fast.
pdfTeX is 8bit, and Knuth’s plain TeX is very low level – for a fairer
comparison, try the TeXbook on LuaMetaTeX (I don’t know if “plain” works
though).
I understand that why it is slow, but the problem is that the difference is too high.
My book has lots of math, so perhaps that is the reason.