Re: Removing parts of the document with [walk] in Haskell

[Gwern Branwen <gwern-v26ZT+9V8bxeoWH0uzbU5w@public.gmane.org>]

You might find my own struggles with similar issues interesting:
https://groups.google.com/g/pandoc-discuss/c/jgb-Q0F2p1Y/m/1DogIEAtAQAJ
https://github.com/gwern/gwern.net/blob/master/build/Typography.hs

A few things I'll note: your `isThe` only works if you assume that all
strings have been fully split into Str/Space/Str sequences. But it's
entirely valid to have a Str containing spaces, like `Str " the "`;
your rewrite would fail there. So in your example, you could do
something like `isThe (Str s) = if "the" `T.isInfixOf` s then Str
(T.replace "the" "" s) else (Str s)` and then simply `removeThe = walk
isThe`. (I don't know if it meaningfully differs in performance from
filtering but if it does, it's probably faster. May still be slower
than your wrong version.)

> The same question actually applies to any transformation which "changes the number of elements":

This is a general problem I've beaten my head against many times. If
you do the concat trick, you start running into issues with how the
`walk` operates exactly so it'll sometimes repeat operations or omit
substitutions you expected to happen, and bottomUp/topDown sometimes
cause issues like their own different omissions or just infinite loops
(always fun).

The best approach I've found is to *not* change the number of
elements, using instead `Inline -> Inline` by using 'the Span trick':
'Span' is the *only* Inline which can wrap multiple Inlines without
meaningfully affecting them. (If you wrap them in Para or Div, it's a
Block; if you wrap them in Italics, they're italicized; if you wrap
them in Str, you need to compile down to a flat string deleting all
formatting etc.)

This is how I do auto-smallcaps, inserting wordbreak characters after
slashes, automatically turning matching text into hyperlinks, etc: I
walk the tree, check Strs for matches of a regexp or character, either
return a Str unaltered (so still an Inline) or return a Span (still an
Inline) whose '[Inline]' payload can be anything like '[Str "foo",
Span "smallcaps-auto" [Str "SMALLCAPS"], Str "bar"]' or '[Link
nullAttr ["GPU"]
("https://en.wikipedia.org/wiki/Graphics_processing_unit","")]'... The
walk typechecks, the substitution happens consistently as you expect
without infinite loops or missing or redundant substitutions, and it
isn't too hard to understand.

The drawback is that Span wrappers aren't great in terms of
efficiency, but correctness is more important than speed, and you can
always postprocess the HTML to remove the <span> wrappers not doing
anything.

(You can substitute in more powerful rewrites by going to raw HTML but
it gets very difficult when you work with `RawInline`, because now you
have to be very careful about the opacity of the Text blob inside
RawInline and the ordering and accidentally clobbering your HTML
literals or messing them up. Try to stick to Span rewrites if at all
possible.)

-- 
gwern
https://www.gwern.net