Playing around with it a bit more, those two issues seem to cover the problems I'm finding.
I added two issues: https://github.com/jgm/pandoc/issues/7324 https://github.com/jgm/pandoc/issues/7323