I for one agree about that; one of the reasons that I don't like LaTeX is that it forces me to use English-based markup with non-English text. Perhaps if textual labels can't be avoided they can be configurable, like the actually inserted texts need to be configurable with templates, something like
``````yaml
labels: # Icelandic
sec: kafli
fig: mynd
tab: tafla
``````
(These are the actual Icelandic words for 'section, 'figure, 'table'. They are so short that it makes little sense to truncate them.)
I guess templates should better use the same library as Pandoc's document template. pandoc-crossref's coopting of math syntax is an expedient which wouldn't be needed anymore.
FWIW my list-table filter for example has the same problem, but for now I have abstained from making the div attributes it looks for configurable. It would not be too complicated though:
``````yaml
# Swedish
lol2table_class: lista-till-tabell
table2lol_class: tabell-till-lista
list_table_align: justering
list_table_width: bredd
``````