Escape characters silently truncate other text

Escape characters silently truncate other text

['amindfv-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org' via pandoc-discuss]

I've come across what seems to be an unusual bug; one that caused a very tricky bug elsewhere in a larger program:

    $ cat /tmp/foo.md
    here's a string "with a\nnewline\ncharacter"
    
    what happened to the other stuff?
    $ pandoc /tmp/foo.md -o /tmp/foo.html
    $ cat /tmp/foo.html
    <p>here’s a string “with a”</p>
    <p>what happened to the other stuff?</p>

In other words, after the appearance of a "\n", everything is truncated on the first line - except the closing quote!

Tom

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Yw57XcTYQBSq80ME%40painter.painter.

Re: Escape characters silently truncate other text

[John MacFarlane]

pandoc -t native will tell you.
\nnewline is being interpreted as a raw LaTeX command.

You can disable this by doing `-f markdown-raw_tex`

But a better approach is to put code like this in backticks, which guarantees that it will appear literally.

> On Aug 30, 2022, at 2:04 PM, 'amindfv-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org' via pandoc-discuss <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> wrote:
> 
> I've come across what seems to be an unusual bug; one that caused a very tricky bug elsewhere in a larger program:
> 
>    $ cat /tmp/foo.md
>    here's a string "with a\nnewline\ncharacter"
> 
>    what happened to the other stuff?
>    $ pandoc /tmp/foo.md -o /tmp/foo.html
>    $ cat /tmp/foo.html
>    <p>here’s a string “with a”</p>
>    <p>what happened to the other stuff?</p>
> 
> In other words, after the appearance of a "\n", everything is truncated on the first line - except the closing quote!
> 
> Tom
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Yw57XcTYQBSq80ME%40painter.painter.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/3EC2D4C1-EA11-46E5-A000-363461CD8A29%40gmail.com.

Re: Escape characters silently truncate other text

['amindfv-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org' via pandoc-discuss]

I'm confused about this for a couple reasons:

  1. The silent truncation: I'd understand if I'd written `here's a string "with a\nequator"` and gotten back `here's a string "with a≠uator"`, but it's much more disturbing to have a single backslash to cause any following text to disappear

  2. raw_tex seems not to actually let me use raw Tex... e.g.

    $ cat /tmp/foo.md 
    here's a string "with a\nnewline\ncharacter"
    
    what happened to the other stuff?
    
    \emph{this also disappears}
    
    This is fine, though: $E=mc^2$
    
    \this_isnt_latex
    
    This is: \neq
    
    $ pandoc /tmp/foo.md -o /tmp/foo.html
    $ cat /tmp/foo.html 
    <p>here’s a string “with a”</p>
    <p>what happened to the other stuff?</p>
    <p></p>
    <p>This is fine, though: <span class="math inline"><em>E</em> = <em>m</em><em>c</em><sup>2</sup></span></p>
    <p>_isnt_latex</p>
    <p>This is: </p>

To me it'd be much more "principle of least astonishment" to only have raw_tex handle commands it understands (and then output them), not consume a lot of text and output nothing.

Thanks (and thanks for Pandoc in general!!),
Tom

On Tue, Aug 30, 2022 at 02:10:12PM -0700, John MacFarlane wrote:
> pandoc -t native will tell you.
> \nnewline is being interpreted as a raw LaTeX command.
> 
> You can disable this by doing `-f markdown-raw_tex`
> 
> But a better approach is to put code like this in backticks, which guarantees that it will appear literally.
> 
> > On Aug 30, 2022, at 2:04 PM, 'amindfv-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org' via pandoc-discuss <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> wrote:
> > 
> > I've come across what seems to be an unusual bug; one that caused a very tricky bug elsewhere in a larger program:
> > 
> >    $ cat /tmp/foo.md
> >    here's a string "with a\nnewline\ncharacter"
> > 
> >    what happened to the other stuff?
> >    $ pandoc /tmp/foo.md -o /tmp/foo.html
> >    $ cat /tmp/foo.html
> >    <p>here’s a string “with a”</p>
> >    <p>what happened to the other stuff?</p>
> > 
> > In other words, after the appearance of a "\n", everything is truncated on the first line - except the closing quote!
> > 
> > Tom
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Yw57XcTYQBSq80ME%40painter.painter.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/3EC2D4C1-EA11-46E5-A000-363461CD8A29%40gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Yw6ILcBFMakgvnrd%40painter.painter.

Re: Escape characters silently truncate other text

[John MacFarlane]

> On Aug 30, 2022, at 2:59 PM, 'amindfv-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org' via pandoc-discuss <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> wrote:
> 
> I'm confused about this for a couple reasons:
> 
>  1. The silent truncation: I'd understand if I'd written `here's a string "with a\nequator"` and gotten back `here's a string "with a≠uator"`, but it's much more disturbing to have a single backslash to cause any following text to disappear
> 

Try it with —verbose

>  2. raw_tex seems not to actually let me use raw Tex... e.g.
> 
>    $ cat /tmp/foo.md 
>    here's a string "with a\nnewline\ncharacter"
> 
>    what happened to the other stuff?
> 
>    \emph{this also disappears}

Again, use pandoc -t native and you’ll see how pandoc is parsing this. It doesn’t disappear. It is parsed as raw LaTeX.  But raw LaTeX content is only passed through to formats that allow it (namely LaTeX, ConTeXt, Beamer, markdown + raw_tex, etc.).


>    This is fine, though: $E=mc^2$

Math is separate from raw tex; it is parsed as a Math element and we convert this as appropriate to the output format.

>    \this_isnt_latex
> 
>    This is: \neq
> 
>    $ pandoc /tmp/foo.md -o /tmp/foo.html
>    $ cat /tmp/foo.html 
>    <p>here’s a string “with a”</p>
>    <p>what happened to the other stuff?</p>
>    <p></p>
>    <p>This is fine, though: <span class="math inline"><em>E</em> = <em>m</em><em>c</em><sup>2</sup></span></p>
>    <p>_isnt_latex</p>
>    <p>This is: </p>
> 
> To me it'd be much more "principle of least astonishment" to only have raw_tex handle commands it understands (and then output them), not consume a lot of text and output nothing


The point of raw TeX is really to allow users to put in the sorrt of thing pandoc *wouldn’t* understand, because they know they want this in the output format.


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/61B0FAA5-C982-4E6E-8ECE-A738785BC612%40gmail.com.