pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 4825 bytes --]

Searched online for similar cases and didn't find anything relevant.

The context is that I recently was made aware that the French insist that a *thin 
space* be inserted immediately before some punctuation characters *',:!?»%*' 
etc.… So in dialogs for instance e.g. … the .md source has: « · bonjour 
mademoiselle · » where the middle dots represent a single U+202f 
non-breaking space.

When I take a look at the intermediate .tex file that pandoc generates the 
thin spaces are correctly converted to '\,' which I believe is the *latex 
way *of coding thin spaces. But when I run xelatex on the latex file and 
look at the resulting PDF I can see that the thin spaces have become 
regular-width spaces. 

I compared the PDF output to another PDF I had created using plain latex 
rather than pandoc and the U+202F's that I typed in my .tex source clearly 
materialize as thin spaces in the PDF.  

What I suspect at this point is that one of the latex packages that pandoc 
sticks in the generated latex file (or the way it is invoked? perhaps a 
combination of packages? …?) is causing this.

As to an *MWE*… I'm not sure it's really appropriate in this particular 
case…

*Just in case… here's what I get from a minimal .md input file:*

`\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
loaded elsewhere
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - added 
the oneside option: so that the text doesn't jump left & right when reading 
on a tablet/ereader
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provides euro and other symbols
\else % if luatex or xelatex
  \usepackage{unicode-math}
  \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
%   \setmainfont[]{EBGaramond-Regular}
    \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
% cjns1989 - 20191129 - old style numbers 
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage[]{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\usepackage{hyperref}
\hypersetup{
            pdftitle={WME},
            pdfborder={0 0 0},
            breaklinks=true}
\urlstyle{same}  % don't use monospace font for urls
\usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{0}
% Redefines (sub)paragraphs to behave more like sections
\ifx\paragraph\undefined\else
\let\oldparagraph\paragraph
\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
\fi
\ifx\subparagraph\undefined\else
\let\oldsubparagraph\subparagraph
\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
\fi
% set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother

\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[shorthands=off,main=french]{babel}
\else
  % load polyglossia as late as possible as it *could* call bidi if RTL 
lang (e.g. Hebrew or Arabic)
  \usepackage{polyglossia}
  \setmainlanguage[]{french}
\fi

\title{WME}
\date{}

\begin{document}
\maketitle

\$ ECM

\hypertarget{wme-title}{%
\chapter{WME (title)}\label{wme-title}}

en lettres capitales, soigneusement imprimées au pochoir\,:

--- «\,Crétins\,!\,» murmura-t-il.

\end{document}`

*Customization* is minimal: old style numbers (proportional) and one-sided 
since the document is not destined for hard-copy printing…

What I have in mind at this point to try and figure out what is happening 
is to work with a one line .md source that has some U+202F's and remove 
default packages until the problem goes away but before I do this I thought 
maybe someone has run into something similar or might suggest a better 
approach than plain trial and error to help determine the cause of the 
problem.

Thoughts?

Thanks,

CJ

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/818817e7-17c7-4bf4-b9fb-e300f6faaf37%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 5677 bytes --]

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 5794 bytes --]

I replaced these lines in the intermediate latex file generated by pandoc:

`\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[shorthands=off,
main=french]{babel}
\else
  % load polyglossia as late as possible as it *could* call bidi if RTL 
lang (e.g. Hebrew or Arabic)
  \usepackage{polyglossia}
  \setmainlanguage[]{french}
\fi`

by…

`\usepackage[french]{babel}`

… and that took care of the problem.

So it looks like it's either a bug in the panglossia package or misuse 
thereof by pandoc?

Not sure where to go from there since I don't understand what the if/else & 
conditionals are meant to do.

Thanks,

CJ

On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>
> Searched online for similar cases and didn't find anything relevant.
>
> The context is that I recently was made aware that the French insist that 
> a *thin space* be inserted immediately before some punctuation characters 
> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
> non-breaking space.
>
> When I take a look at the intermediate .tex file that pandoc generates the 
> thin spaces are correctly converted to '\,' which I believe is the *latex 
> way *of coding thin spaces. But when I run xelatex on the latex file and 
> look at the resulting PDF I can see that the thin spaces have become 
> regular-width spaces. 
>
> I compared the PDF output to another PDF I had created using plain latex 
> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
> materialize as thin spaces in the PDF.  
>
> What I suspect at this point is that one of the latex packages that pandoc 
> sticks in the generated latex file (or the way it is invoked? perhaps a 
> combination of packages? …?) is causing this.
>
> As to an *MWE*… I'm not sure it's really appropriate in this particular 
> case…
>
> *Just in case… here's what I get from a minimal .md input file:*
>
> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
> loaded elsewhere
> \PassOptionsToPackage{hyphens}{url}
> %
> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
> added the oneside option: so that the text doesn't jump left & right when 
> reading on a tablet/ereader
> \usepackage{lmodern}
> \usepackage{amssymb,amsmath}
> \usepackage{ifxetex,ifluatex}
> \usepackage{fixltx2e} % provides \textsubscript
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[T1]{fontenc}
>   \usepackage[utf8]{inputenc}
>   \usepackage{textcomp} % provides euro and other symbols
> \else % if luatex or xelatex
>   \usepackage{unicode-math}
>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
> %   \setmainfont[]{EBGaramond-Regular}
>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
> % cjns1989 - 20191129 - old style numbers 
> \fi
> % use upquote if available, for straight quotes in verbatim environments
> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
> % use microtype if available
> \IfFileExists{microtype.sty}{%
> \usepackage[]{microtype}
> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
> }{}
> \usepackage{hyperref}
> \hypersetup{
>             pdftitle={WME},
>             pdfborder={0 0 0},
>             breaklinks=true}
> \urlstyle{same}  % don't use monospace font for urls
> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
> \setlength{\emergencystretch}{3em}  % prevent overfull lines
> \providecommand{\tightlist}{%
>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
> \setcounter{secnumdepth}{0}
> % Redefines (sub)paragraphs to behave more like sections
> \ifx\paragraph\undefined\else
> \let\oldparagraph\paragraph
> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
> \fi
> \ifx\subparagraph\undefined\else
> \let\oldsubparagraph\subparagraph
> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
> \fi
> % set default figure placement to htbp
> \makeatletter
> \def\fps@figure{htbp}
> \makeatother
>
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[shorthands=off,main=french]{babel}
> \else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage[]{french}
> \fi
>
> \title{WME}
> \date{}
>
> \begin{document}
> \maketitle
>
> \$ ECM
>
> \hypertarget{wme-title}{%
> \chapter{WME (title)}\label{wme-title}}
>
> en lettres capitales, soigneusement imprimées au pochoir\,:
>
> --- «\,Crétins\,!\,» murmura-t-il.
>
> \end{document}`
>
> *Customization* is minimal: old style numbers (proportional) and 
> one-sided since the document is not destined for hard-copy printing…
>
> What I have in mind at this point to try and figure out what is happening 
> is to work with a one line .md source that has some U+202F's and remove 
> default packages until the problem goes away but before I do this I thought 
> maybe someone has run into something similar or might suggest a better 
> approach than plain trial and error to help determine the cause of the 
> problem.
>
> Thoughts?
>
> Thanks,
>
> CJ
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/158fd0ac-89bc-4fb1-9920-386bf325dad6%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 6882 bytes --]

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[John MacFarlane]

The conditionals just mean that babel is used with pdflatex
and polyglossia with xelatex and lualatex.

maybe there's an option we could/should be passing to
polyglossia?  Check the documentation?

Chris Jones <cjns1989-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I replaced these lines in the intermediate latex file generated by pandoc:
>
> `\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[shorthands=off,
> main=french]{babel}
> \else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage[]{french}
> \fi`
>
> by…
>
> `\usepackage[french]{babel}`
>
> … and that took care of the problem.
>
> So it looks like it's either a bug in the panglossia package or misuse 
> thereof by pandoc?
>
> Not sure where to go from there since I don't understand what the if/else & 
> conditionals are meant to do.
>
> Thanks,
>
> CJ
>
> On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>>
>> Searched online for similar cases and didn't find anything relevant.
>>
>> The context is that I recently was made aware that the French insist that 
>> a *thin space* be inserted immediately before some punctuation characters 
>> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
>> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
>> non-breaking space.
>>
>> When I take a look at the intermediate .tex file that pandoc generates the 
>> thin spaces are correctly converted to '\,' which I believe is the *latex 
>> way *of coding thin spaces. But when I run xelatex on the latex file and 
>> look at the resulting PDF I can see that the thin spaces have become 
>> regular-width spaces. 
>>
>> I compared the PDF output to another PDF I had created using plain latex 
>> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
>> materialize as thin spaces in the PDF.  
>>
>> What I suspect at this point is that one of the latex packages that pandoc 
>> sticks in the generated latex file (or the way it is invoked? perhaps a 
>> combination of packages? …?) is causing this.
>>
>> As to an *MWE*… I'm not sure it's really appropriate in this particular 
>> case…
>>
>> *Just in case… here's what I get from a minimal .md input file:*
>>
>> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
>> loaded elsewhere
>> \PassOptionsToPackage{hyphens}{url}
>> %
>> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
>> added the oneside option: so that the text doesn't jump left & right when 
>> reading on a tablet/ereader
>> \usepackage{lmodern}
>> \usepackage{amssymb,amsmath}
>> \usepackage{ifxetex,ifluatex}
>> \usepackage{fixltx2e} % provides \textsubscript
>> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>>   \usepackage[T1]{fontenc}
>>   \usepackage[utf8]{inputenc}
>>   \usepackage{textcomp} % provides euro and other symbols
>> \else % if luatex or xelatex
>>   \usepackage{unicode-math}
>>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
>> %   \setmainfont[]{EBGaramond-Regular}
>>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
>> % cjns1989 - 20191129 - old style numbers 
>> \fi
>> % use upquote if available, for straight quotes in verbatim environments
>> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
>> % use microtype if available
>> \IfFileExists{microtype.sty}{%
>> \usepackage[]{microtype}
>> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
>> }{}
>> \usepackage{hyperref}
>> \hypersetup{
>>             pdftitle={WME},
>>             pdfborder={0 0 0},
>>             breaklinks=true}
>> \urlstyle{same}  % don't use monospace font for urls
>> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
>> \setlength{\emergencystretch}{3em}  % prevent overfull lines
>> \providecommand{\tightlist}{%
>>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
>> \setcounter{secnumdepth}{0}
>> % Redefines (sub)paragraphs to behave more like sections
>> \ifx\paragraph\undefined\else
>> \let\oldparagraph\paragraph
>> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
>> \fi
>> \ifx\subparagraph\undefined\else
>> \let\oldsubparagraph\subparagraph
>> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
>> \fi
>> % set default figure placement to htbp
>> \makeatletter
>> \def\fps@figure{htbp}
>> \makeatother
>>
>> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>>   \usepackage[shorthands=off,main=french]{babel}
>> \else
>>   % load polyglossia as late as possible as it *could* call bidi if RTL 
>> lang (e.g. Hebrew or Arabic)
>>   \usepackage{polyglossia}
>>   \setmainlanguage[]{french}
>> \fi
>>
>> \title{WME}
>> \date{}
>>
>> \begin{document}
>> \maketitle
>>
>> \$ ECM
>>
>> \hypertarget{wme-title}{%
>> \chapter{WME (title)}\label{wme-title}}
>>
>> en lettres capitales, soigneusement imprimées au pochoir\,:
>>
>> --- «\,Crétins\,!\,» murmura-t-il.
>>
>> \end{document}`
>>
>> *Customization* is minimal: old style numbers (proportional) and 
>> one-sided since the document is not destined for hard-copy printing…
>>
>> What I have in mind at this point to try and figure out what is happening 
>> is to work with a one line .md source that has some U+202F's and remove 
>> default packages until the problem goes away but before I do this I thought 
>> maybe someone has run into something similar or might suggest a better 
>> approach than plain trial and error to help determine the cause of the 
>> problem.
>>
>> Thoughts?
>>
>> Thanks,
>>
>> CJ
>>
>>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/158fd0ac-89bc-4fb1-9920-386bf325dad6%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480ky2tkpn21.fsf%40johnmacfarlane.net.

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 7353 bytes --]

Good… was beginning to wonder with all those wicked viruses going the 
rounds… whether I would get some form of reply.

The problem I think turns out to be related to the way I write French 
punctuation. I learned the hard way about this peculiarity of French 
typography where you must stick those non-breaking thin spaces where they 
belong and made a point of doing it by hand (in vim I haved mapped CTRL+K 
to U+202f). Now what happens is that from his name I guess that the 
gentleman who initially came up with the polyglossia package is a 
frenchman. As such he was very focused on this particular typesetting quirk 
and decided to ensure that this thin spaces business would be taken care of 
automatically by his package. I believe this is the way text processors 
handle it when you tell them that the material is written in French (?).

So what I am beginning to suspect is that pandoc invokes the polyglossia 
package correctly but since I do not rely on this being done for me by the 
package… it may very well turn out that polyglossia inserts a* second thin 
space* next to the one that's already there… adding up to what looks 
basically like a regular-width space. Not sure how I could check the 
resulting PDF and verify this hunch of mine. Another approach would 
obviously be to run a few regex's to get rid of all those U+202f's in my 
.md files and see what happens…

I took a look at the polyglossia doc as you suggest and I did find 
indications that polyglossia does add thin spaces automatically (that's the 
kind of feature that make polyglossia more 'modern' than babel I imagine) 
and that you can use the 'autospacing=false' option should you need to 
disable the feature.

Obviously most people would NOT want pandoc to disable the feature…

So it's really up to me to change my usepackage polyglossia invocation to 
make sure my thin spaces are left alone.

What would be the recommend way to do this… hard code something like 
'\usepackage[autospacing=false]{polyglossia}' in my 
~/.pandoc/templates/default.latex I imagine?

Thanks,

CJ

P.S. how can I fix the typo in the issue's title… U+2024 instead of the 
intended U+202f?

On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>
> Searched online for similar cases and didn't find anything relevant.
>
> The context is that I recently was made aware that the French insist that 
> a *thin space* be inserted immediately before some punctuation characters 
> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
> non-breaking space.
>
> When I take a look at the intermediate .tex file that pandoc generates the 
> thin spaces are correctly converted to '\,' which I believe is the *latex 
> way *of coding thin spaces. But when I run xelatex on the latex file and 
> look at the resulting PDF I can see that the thin spaces have become 
> regular-width spaces. 
>
> I compared the PDF output to another PDF I had created using plain latex 
> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
> materialize as thin spaces in the PDF.  
>
> What I suspect at this point is that one of the latex packages that pandoc 
> sticks in the generated latex file (or the way it is invoked? perhaps a 
> combination of packages? …?) is causing this.
>
> As to an *MWE*… I'm not sure it's really appropriate in this particular 
> case…
>
> *Just in case… here's what I get from a minimal .md input file:*
>
> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
> loaded elsewhere
> \PassOptionsToPackage{hyphens}{url}
> %
> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
> added the oneside option: so that the text doesn't jump left & right when 
> reading on a tablet/ereader
> \usepackage{lmodern}
> \usepackage{amssymb,amsmath}
> \usepackage{ifxetex,ifluatex}
> \usepackage{fixltx2e} % provides \textsubscript
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[T1]{fontenc}
>   \usepackage[utf8]{inputenc}
>   \usepackage{textcomp} % provides euro and other symbols
> \else % if luatex or xelatex
>   \usepackage{unicode-math}
>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
> %   \setmainfont[]{EBGaramond-Regular}
>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
> % cjns1989 - 20191129 - old style numbers 
> \fi
> % use upquote if available, for straight quotes in verbatim environments
> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
> % use microtype if available
> \IfFileExists{microtype.sty}{%
> \usepackage[]{microtype}
> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
> }{}
> \usepackage{hyperref}
> \hypersetup{
>             pdftitle={WME},
>             pdfborder={0 0 0},
>             breaklinks=true}
> \urlstyle{same}  % don't use monospace font for urls
> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
> \setlength{\emergencystretch}{3em}  % prevent overfull lines
> \providecommand{\tightlist}{%
>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
> \setcounter{secnumdepth}{0}
> % Redefines (sub)paragraphs to behave more like sections
> \ifx\paragraph\undefined\else
> \let\oldparagraph\paragraph
> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
> \fi
> \ifx\subparagraph\undefined\else
> \let\oldsubparagraph\subparagraph
> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
> \fi
> % set default figure placement to htbp
> \makeatletter
> \def\fps@figure{htbp}
> \makeatother
>
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[shorthands=off,main=french]{babel}
> \else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage[]{french}
> \fi
>
> \title{WME}
> \date{}
>
> \begin{document}
> \maketitle
>
> \$ ECM
>
> \hypertarget{wme-title}{%
> \chapter{WME (title)}\label{wme-title}}
>
> en lettres capitales, soigneusement imprimées au pochoir\,:
>
> --- «\,Crétins\,!\,» murmura-t-il.
>
> \end{document}`
>
> *Customization* is minimal: old style numbers (proportional) and 
> one-sided since the document is not destined for hard-copy printing…
>
> What I have in mind at this point to try and figure out what is happening 
> is to work with a one line .md source that has some U+202F's and remove 
> default packages until the problem goes away but before I do this I thought 
> maybe someone has run into something similar or might suggest a better 
> approach than plain trial and error to help determine the cause of the 
> problem.
>
> Thoughts?
>
> Thanks,
>
> CJ
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5f3b2ff3-b74f-4ba5-858b-b08b13124190%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 8434 bytes --]

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 5996 bytes --]

I'm trying to modify the default.latex template and I see this:

`\else
  % load polyglossia as late as possible as it *could* call bidi if RTL 
lang (e.g. Hebrew or Arabic)
  \usepackage{polyglossia}
  \setmainlanguage*[$polyglossia-lang.options$]*{$polyglossia-lang.name$}
$for(polyglossia-otherlangs)$
  
\setotherlanguage[$polyglossia-otherlangs.options$]{$polyglossia-otherlangs.name$}
$endfor$
\fi`

Now I suspect that '[autospacing=false]' is an option of the 
\setmainlanguage command rather than a 'global' option of the poylglossia 
package.

Therefore I would need to change this:

`  \setmainlanguage[$polyglossia-lang.options$]{$polyglossia-lang.name$}`

to something that has 'autospacing=false' added to the contents of 
'$polyglossia-lang.options$', presumably a pandoc variable (?)

How do I achieve this?

Thanks,

CJ

On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>
> Searched online for similar cases and didn't find anything relevant.
>
> The context is that I recently was made aware that the French insist that 
> a *thin space* be inserted immediately before some punctuation characters 
> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
> non-breaking space.
>
> When I take a look at the intermediate .tex file that pandoc generates the 
> thin spaces are correctly converted to '\,' which I believe is the *latex 
> way *of coding thin spaces. But when I run xelatex on the latex file and 
> look at the resulting PDF I can see that the thin spaces have become 
> regular-width spaces. 
>
> I compared the PDF output to another PDF I had created using plain latex 
> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
> materialize as thin spaces in the PDF.  
>
> What I suspect at this point is that one of the latex packages that pandoc 
> sticks in the generated latex file (or the way it is invoked? perhaps a 
> combination of packages? …?) is causing this.
>
> As to an *MWE*… I'm not sure it's really appropriate in this particular 
> case…
>
> *Just in case… here's what I get from a minimal .md input file:*
>
> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
> loaded elsewhere
> \PassOptionsToPackage{hyphens}{url}
> %
> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
> added the oneside option: so that the text doesn't jump left & right when 
> reading on a tablet/ereader
> \usepackage{lmodern}
> \usepackage{amssymb,amsmath}
> \usepackage{ifxetex,ifluatex}
> \usepackage{fixltx2e} % provides \textsubscript
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[T1]{fontenc}
>   \usepackage[utf8]{inputenc}
>   \usepackage{textcomp} % provides euro and other symbols
> \else % if luatex or xelatex
>   \usepackage{unicode-math}
>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
> %   \setmainfont[]{EBGaramond-Regular}
>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
> % cjns1989 - 20191129 - old style numbers 
> \fi
> % use upquote if available, for straight quotes in verbatim environments
> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
> % use microtype if available
> \IfFileExists{microtype.sty}{%
> \usepackage[]{microtype}
> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
> }{}
> \usepackage{hyperref}
> \hypersetup{
>             pdftitle={WME},
>             pdfborder={0 0 0},
>             breaklinks=true}
> \urlstyle{same}  % don't use monospace font for urls
> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
> \setlength{\emergencystretch}{3em}  % prevent overfull lines
> \providecommand{\tightlist}{%
>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
> \setcounter{secnumdepth}{0}
> % Redefines (sub)paragraphs to behave more like sections
> \ifx\paragraph\undefined\else
> \let\oldparagraph\paragraph
> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
> \fi
> \ifx\subparagraph\undefined\else
> \let\oldsubparagraph\subparagraph
> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
> \fi
> % set default figure placement to htbp
> \makeatletter
> \def\fps@figure{htbp}
> \makeatother
>
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[shorthands=off,main=french]{babel}
> \else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage[]{french}
> \fi
>
> \title{WME}
> \date{}
>
> \begin{document}
> \maketitle
>
> \$ ECM
>
> \hypertarget{wme-title}{%
> \chapter{WME (title)}\label{wme-title}}
>
> en lettres capitales, soigneusement imprimées au pochoir\,:
>
> --- «\,Crétins\,!\,» murmura-t-il.
>
> \end{document}`
>
> *Customization* is minimal: old style numbers (proportional) and 
> one-sided since the document is not destined for hard-copy printing…
>
> What I have in mind at this point to try and figure out what is happening 
> is to work with a one line .md source that has some U+202F's and remove 
> default packages until the problem goes away but before I do this I thought 
> maybe someone has run into something similar or might suggest a better 
> approach than plain trial and error to help determine the cause of the 
> problem.
>
> Thoughts?
>
> Thanks,
>
> CJ
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/561d210b-ceb6-4f9a-98e2-556f8e12e2ca%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7098 bytes --]

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[John MacFarlane]

You should be able to put something like this in your yaml
metadata:

---
polyglossia-lang:
  options: 'autospacing=false'
...

Chris Jones <cjns1989-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I'm trying to modify the default.latex template and I see this:
>
> `\else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage*[$polyglossia-lang.options$]*{$polyglossia-lang.name$}
> $for(polyglossia-otherlangs)$
>   
> \setotherlanguage[$polyglossia-otherlangs.options$]{$polyglossia-otherlangs.name$}
> $endfor$
> \fi`
>
> Now I suspect that '[autospacing=false]' is an option of the 
> \setmainlanguage command rather than a 'global' option of the poylglossia 
> package.
>
> Therefore I would need to change this:
>
> `  \setmainlanguage[$polyglossia-lang.options$]{$polyglossia-lang.name$}`
>
> to something that has 'autospacing=false' added to the contents of 
> '$polyglossia-lang.options$', presumably a pandoc variable (?)
>
> How do I achieve this?
>
> Thanks,
>
> CJ
>
> On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>>
>> Searched online for similar cases and didn't find anything relevant.
>>
>> The context is that I recently was made aware that the French insist that 
>> a *thin space* be inserted immediately before some punctuation characters 
>> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
>> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
>> non-breaking space.
>>
>> When I take a look at the intermediate .tex file that pandoc generates the 
>> thin spaces are correctly converted to '\,' which I believe is the *latex 
>> way *of coding thin spaces. But when I run xelatex on the latex file and 
>> look at the resulting PDF I can see that the thin spaces have become 
>> regular-width spaces. 
>>
>> I compared the PDF output to another PDF I had created using plain latex 
>> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
>> materialize as thin spaces in the PDF.  
>>
>> What I suspect at this point is that one of the latex packages that pandoc 
>> sticks in the generated latex file (or the way it is invoked? perhaps a 
>> combination of packages? …?) is causing this.
>>
>> As to an *MWE*… I'm not sure it's really appropriate in this particular 
>> case…
>>
>> *Just in case… here's what I get from a minimal .md input file:*
>>
>> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
>> loaded elsewhere
>> \PassOptionsToPackage{hyphens}{url}
>> %
>> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
>> added the oneside option: so that the text doesn't jump left & right when 
>> reading on a tablet/ereader
>> \usepackage{lmodern}
>> \usepackage{amssymb,amsmath}
>> \usepackage{ifxetex,ifluatex}
>> \usepackage{fixltx2e} % provides \textsubscript
>> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>>   \usepackage[T1]{fontenc}
>>   \usepackage[utf8]{inputenc}
>>   \usepackage{textcomp} % provides euro and other symbols
>> \else % if luatex or xelatex
>>   \usepackage{unicode-math}
>>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
>> %   \setmainfont[]{EBGaramond-Regular}
>>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
>> % cjns1989 - 20191129 - old style numbers 
>> \fi
>> % use upquote if available, for straight quotes in verbatim environments
>> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
>> % use microtype if available
>> \IfFileExists{microtype.sty}{%
>> \usepackage[]{microtype}
>> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
>> }{}
>> \usepackage{hyperref}
>> \hypersetup{
>>             pdftitle={WME},
>>             pdfborder={0 0 0},
>>             breaklinks=true}
>> \urlstyle{same}  % don't use monospace font for urls
>> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
>> \setlength{\emergencystretch}{3em}  % prevent overfull lines
>> \providecommand{\tightlist}{%
>>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
>> \setcounter{secnumdepth}{0}
>> % Redefines (sub)paragraphs to behave more like sections
>> \ifx\paragraph\undefined\else
>> \let\oldparagraph\paragraph
>> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
>> \fi
>> \ifx\subparagraph\undefined\else
>> \let\oldsubparagraph\subparagraph
>> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
>> \fi
>> % set default figure placement to htbp
>> \makeatletter
>> \def\fps@figure{htbp}
>> \makeatother
>>
>> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>>   \usepackage[shorthands=off,main=french]{babel}
>> \else
>>   % load polyglossia as late as possible as it *could* call bidi if RTL 
>> lang (e.g. Hebrew or Arabic)
>>   \usepackage{polyglossia}
>>   \setmainlanguage[]{french}
>> \fi
>>
>> \title{WME}
>> \date{}
>>
>> \begin{document}
>> \maketitle
>>
>> \$ ECM
>>
>> \hypertarget{wme-title}{%
>> \chapter{WME (title)}\label{wme-title}}
>>
>> en lettres capitales, soigneusement imprimées au pochoir\,:
>>
>> --- «\,Crétins\,!\,» murmura-t-il.
>>
>> \end{document}`
>>
>> *Customization* is minimal: old style numbers (proportional) and 
>> one-sided since the document is not destined for hard-copy printing…
>>
>> What I have in mind at this point to try and figure out what is happening 
>> is to work with a one line .md source that has some U+202F's and remove 
>> default packages until the problem goes away but before I do this I thought 
>> maybe someone has run into something similar or might suggest a better 
>> approach than plain trial and error to help determine the cause of the 
>> problem.
>>
>> Thoughts?
>>
>> Thanks,
>>
>> CJ
>>
>>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/561d210b-ceb6-4f9a-98e2-556f8e12e2ca%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m25zgnmwnm.fsf%40johnmacfarlane.net.

Re: pandoc correctly translates U+2024 thin space to '\,' but the spaces in PDF created by Xelatex are full-width

[Chris Jones]

[-- Attachment #1.1: Type: text/plain, Size: 5863 bytes --]

Unfortunately further testing showed that neither the global 
'[autospacing=false]' nor the '\AutoSpacing' and '\NoAutoSpacing' commands 
that let you switch autospacing on and off inside a document… advertised in 
the polyglossia documentaton failed/were not recognized… the reason is that 
they only appeared in versions 1.45+ of the pack polyglossia… and the 
textlive installed on my current version of debian/linux is version 1.43. 

Rather than trying to figure out if/how I could upgrade polyglossia to 
something more current without breaking anything I decided to take the easy 
way out… i.e. replace polyglossia by babel in my default.latex custom 
template.

Thank you very much for your help.

CJ

On Saturday, February 1, 2020 at 2:18:30 PM UTC-5, Chris Jones wrote:
>
> Searched online for similar cases and didn't find anything relevant.
>
> The context is that I recently was made aware that the French insist that 
> a *thin space* be inserted immediately before some punctuation characters 
> *',:!?»%*' etc.… So in dialogs for instance e.g. … the .md source has: « 
> · bonjour mademoiselle · » where the middle dots represent a single U+202f 
> non-breaking space.
>
> When I take a look at the intermediate .tex file that pandoc generates the 
> thin spaces are correctly converted to '\,' which I believe is the *latex 
> way *of coding thin spaces. But when I run xelatex on the latex file and 
> look at the resulting PDF I can see that the thin spaces have become 
> regular-width spaces. 
>
> I compared the PDF output to another PDF I had created using plain latex 
> rather than pandoc and the U+202F's that I typed in my .tex source clearly 
> materialize as thin spaces in the PDF.  
>
> What I suspect at this point is that one of the latex packages that pandoc 
> sticks in the generated latex file (or the way it is invoked? perhaps a 
> combination of packages? …?) is causing this.
>
> As to an *MWE*… I'm not sure it's really appropriate in this particular 
> case…
>
> *Just in case… here's what I get from a minimal .md input file:*
>
> `\PassOptionsToPackage{unicode=true}{hyperref} % options for packages 
> loaded elsewhere
> \PassOptionsToPackage{hyphens}{url}
> %
> \documentclass[oneside,10pt,french,]{extbook} % cjns1989 - 27112019 - 
> added the oneside option: so that the text doesn't jump left & right when 
> reading on a tablet/ereader
> \usepackage{lmodern}
> \usepackage{amssymb,amsmath}
> \usepackage{ifxetex,ifluatex}
> \usepackage{fixltx2e} % provides \textsubscript
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[T1]{fontenc}
>   \usepackage[utf8]{inputenc}
>   \usepackage{textcomp} % provides euro and other symbols
> \else % if luatex or xelatex
>   \usepackage{unicode-math}
>   \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
> %   \setmainfont[]{EBGaramond-Regular}
>     \setmainfont[Numbers={OldStyle,Proportional}]{EBGaramond-Regular}      
> % cjns1989 - 20191129 - old style numbers 
> \fi
> % use upquote if available, for straight quotes in verbatim environments
> \IfFileExists{upquote.sty}{\usepackage{upquote}}{}
> % use microtype if available
> \IfFileExists{microtype.sty}{%
> \usepackage[]{microtype}
> \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
> }{}
> \usepackage{hyperref}
> \hypersetup{
>             pdftitle={WME},
>             pdfborder={0 0 0},
>             breaklinks=true}
> \urlstyle{same}  % don't use monospace font for urls
> \usepackage[papersize={3.75 in, 6.0 in},left=.3 in,right=.3 in]{geometry}
> \setlength{\emergencystretch}{3em}  % prevent overfull lines
> \providecommand{\tightlist}{%
>   \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
> \setcounter{secnumdepth}{0}
> % Redefines (sub)paragraphs to behave more like sections
> \ifx\paragraph\undefined\else
> \let\oldparagraph\paragraph
> \renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
> \fi
> \ifx\subparagraph\undefined\else
> \let\oldsubparagraph\subparagraph
> \renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
> \fi
> % set default figure placement to htbp
> \makeatletter
> \def\fps@figure{htbp}
> \makeatother
>
> \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
>   \usepackage[shorthands=off,main=french]{babel}
> \else
>   % load polyglossia as late as possible as it *could* call bidi if RTL 
> lang (e.g. Hebrew or Arabic)
>   \usepackage{polyglossia}
>   \setmainlanguage[]{french}
> \fi
>
> \title{WME}
> \date{}
>
> \begin{document}
> \maketitle
>
> \$ ECM
>
> \hypertarget{wme-title}{%
> \chapter{WME (title)}\label{wme-title}}
>
> en lettres capitales, soigneusement imprimées au pochoir\,:
>
> --- «\,Crétins\,!\,» murmura-t-il.
>
> \end{document}`
>
> *Customization* is minimal: old style numbers (proportional) and 
> one-sided since the document is not destined for hard-copy printing…
>
> What I have in mind at this point to try and figure out what is happening 
> is to work with a one line .md source that has some U+202F's and remove 
> default packages until the problem goes away but before I do this I thought 
> maybe someone has run into something similar or might suggest a better 
> approach than plain trial and error to help determine the cause of the 
> problem.
>
> Thoughts?
>
> Thanks,
>
> CJ
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e9c3e6b9-4bb5-46a4-b35f-449198f6c907%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 6821 bytes --]