ANN: pandoc 1.17

ANN: pandoc 1.17

[John MacFarlane]

I've released pandoc 1.17.  Release notes and
binary packages can be found on the github release page:

https://github.com/jgm/pandoc/releases/tag/1.17

Thanks to all who contributed, and especially to Jesse Rosenthal,
Chris Forster, Mauro Bieg, and Albert Krewinkel.

Highlights of this release:

* A new output format, tei, contributed by Chris Forster.
  TEI Simple is an XML format for archiving books.

* A new --file-scope option contributed by Jesse Rosenthal.
  By default pandoc concatenates input files before parsing;
  with --file-scope, it will parse each file individually,
  and then combine the parsed contents.  This is helpful
  for some applications.

* Many performance improvements and bug fixes.

API changes:

* Added writeTEI
* Added a general ByteStringReader with warnings, used by
  the docx reader
* Added readDocxWithWarnings
* Changed type of Shared.uniqueIdent argument from [String]
  to Set String

Re: ANN: pandoc 1.17

[John Maxwell]

[-- Attachment #1.1: Type: text/plain, Size: 758 bytes --]

A path to TEI *and* --file-scope, in one release! Awesome!

You pandoc developers are fabulous. Thank you!

- John Maxwell
  Simon Fraser University
  jmax-z1KUqvL5UUQ@public.gmane.org


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/52366e29-dd13-473e-9b24-dfe6c4a4384f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 1201 bytes --]

Re: ANN: pandoc 1.17

[BP Jonsson]

Den 2016-03-20 kl. 18:03, skrev John MacFarlane:
> I've released pandoc 1.17.  Release notes and
> binary packages can be found on the github release page:

Some questions/comments (quotes copied from changelog and README):

> pandoc (1.17)

>   * Added `--file-scope` option (Jesse Rosenthal).
>     By default pandoc operates on multiple files by first concatenating
>     them (around extra line breaks) and then processing the joined file. So
>     it only parses a multi-file document at the document scope. This has the
>     benefit that footnotes and links can be in different files, but for
>     some purposes it is useful to parse the individual files first
>     and then combine their outputs (e.g. when the files use footnotes
>     or links with the same labels).  The `--file-scope` option causes
>     pandoc to parse the files first, and then combine the parsed output,
>     instead of combining before parsing. `--file-scope` is selected
>     automatically for binary input files (which cannot be concatenated)
>     and for pandoc json.

This would mean that I can retire part of the functionality of my 
wrapper script, but...

README says:

> `--file-scope`

>     [snip] If this option is set,
>     footnotes and links will not work across files.
>
>

Would it be possible to prefix a suitably modified filename to 
identifiers to make it work? I realize that this may require any 
or all of having a Note element store the original identifier and 
heuristically creating the requisite links even when no 
appropriate target is present in the current file, at least when 
`--file-scope` is in effect, but I think it might be worth it. 
(Not that I had solved that anywhere near satisfactorily in my 
wrapper.)

>   * HTML reader:

>     + Fixed behavior of base tag (#2777).
>       If the base path does not end with slash, the last component
>       will be replaced.  E.g. base = `http://example.com/foo`
>       combines with `bar.html` to give `http://example.com/bar.html`.
>       If the href begins with a slash, the whole path of the base
>       is replaced.  E.g. base = `http://example.com/foo/` combines
>       with `/bar.html` to give `http://example.com/bar.html`.

What is this good for? I'm not rhetorical, but honestly wonder why 
one would want to remove the last directory component (if I 
understand correctly).


>   * LaTeX reader:

>     + `inlineCommand` now gobbles an empty `{}` after any command (#2687).
>       This gives better results when people write e.g. `\TeX{}` in Markdown.

Much appreciated!

>   * Markdown writer: Use hyphens for YAML metadata block bottom line, for
>     better compatibility with other Markdown flavors (Henrik Tramberend).

It also makes it invalid YAML! Which means that you can no longer 
just save the YAML block to a \*.yaml file should you want to. If 
people want invalid YAML (why would they?) it should at least be 
subject to an option. Put otherwise: why should pandoc 
(unconditionally) produce invalid YAML just because some other 
tool does? Isn't that rather a bug/misfeature in those other tools?

>   * LaTeX writer:

>     + Avoid double toprule in headerless table with caption (#2742).

Thanks!

/bpj

Re: ANN: pandoc 1.17

[John MacFarlane]

+++ BP Jonsson [Mar 21 16 14:10 ]:

>>  * Markdown writer: Use hyphens for YAML metadata block bottom line, for
>>    better compatibility with other Markdown flavors (Henrik Tramberend).
>
>It also makes it invalid YAML! Which means that you can no longer just 
>save the YAML block to a \*.yaml file should you want to. If people 
>want invalid YAML (why would they?) it should at least be subject to 
>an option. Put otherwise: why should pandoc (unconditionally) produce 
>invalid YAML just because some other tool does? Isn't that rather a 
>bug/misfeature in those other tools?

Please see the discussion in
https://github.com/jgm/pandoc/pull/2668.  I made the
argument that the other tools should change.  However, I've
been making this argument for a really long time now, and
they haven't changed.

I'm not convinced there's a real practical application where
having the `...` separator at the end matters.  See
monofon's comment on Jan 22, which gives several examples
of how the change would affect existing YAML tools.

Do you have an actual existing workflow that breaks because
of this change?  I can make this configurable if there's a
compelling reason, but I prefer not to add complexity
unless there's a very good reason.

Re: ANN: pandoc 1.17

[Fedor Sheremetyev]

Looks like reading of JSON format is broken in 1.17 (checked on OS X 10.11.3 and Windows 10 x 64).

The following command produces empty output on OS X with Pandoc 1.17 but prints “<p>Test</p>” with Pandoc 1.16.0.2.

$ echo "Test" | pandoc --to json | pandoc --from json

> On 20 Mar 2016, at 17:03, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
> 
> I've released pandoc 1.17.  Release notes and
> binary packages can be found on the github release page:
> 
> https://github.com/jgm/pandoc/releases/tag/1.17

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/30E2EE61-380A-4BBD-99F7-CDED707092D1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: pandoc 1.17

[Jesse Rosenthal]

Ugh -- I see the issue. I need to fix the way stdin is handled with
"--file-scope". That hadn't come up because all of the testing was done
with, well, files.

Apologies. I'll get that posted ASAP.

Fedor Sheremetyev <sheremetyev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> [ text/plain ]
> Looks like reading of JSON format is broken in 1.17 (checked on OS X 10.11.3 and Windows 10 x 64).
>
> The following command produces empty output on OS X with Pandoc 1.17 but prints “<p>Test</p>” with Pandoc 1.16.0.2.
>
> $ echo "Test" | pandoc --to json | pandoc --from json
>
>> On 20 Mar 2016, at 17:03, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>> 
>> I've released pandoc 1.17.  Release notes and
>> binary packages can be found on the github release page:
>> 
>> https://github.com/jgm/pandoc/releases/tag/1.17
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/30E2EE61-380A-4BBD-99F7-CDED707092D1%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87shzjzhom.fsf%40jhu.edu.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: pandoc 1.17

[Jesse Rosenthal]

Okay, it's fixed in a8a2d4f2. Apologies again for that.

John -- do you think this is brown-paper-bag-worthy?

Also, what would you think about splitting out some of the pandoc.hs
interface logic into Text.Pandoc.UI or something, for the sake of
testing? I'd be happy to make that an early-summer project if you
thought it would be useful.

Best,
Jesse

Jesse Rosenthal <jrosenthal-4GNroTWusrE@public.gmane.org> writes:

> [ text/plain ]
> Ugh -- I see the issue. I need to fix the way stdin is handled with
> "--file-scope". That hadn't come up because all of the testing was done
> with, well, files.
>
> Apologies. I'll get that posted ASAP.
>
> Fedor Sheremetyev <sheremetyev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> [ text/plain ]
>> Looks like reading of JSON format is broken in 1.17 (checked on OS X 10.11.3 and Windows 10 x 64).
>>
>> The following command produces empty output on OS X with Pandoc 1.17 but prints “<p>Test</p>” with Pandoc 1.16.0.2.
>>
>> $ echo "Test" | pandoc --to json | pandoc --from json
>>
>>> On 20 Mar 2016, at 17:03, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>>> 
>>> I've released pandoc 1.17.  Release notes and
>>> binary packages can be found on the github release page:
>>> 
>>> https://github.com/jgm/pandoc/releases/tag/1.17
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/30E2EE61-380A-4BBD-99F7-CDED707092D1%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87oaa7zgj8.fsf%40jhu.edu.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: pandoc 1.17

[Mario Valle]

[-- Attachment #1.1: Type: text/plain, Size: 1776 bytes --]

Just a very minor thing: `pandoc --version` shows copyright 2006-2015 but 
we are in 2016
Thanks for this tool!
mario

Il giorno domenica 20 marzo 2016 18:04:10 UTC+1, John MacFarlane ha scritto:
>
> I've released pandoc 1.17.  Release notes and 
> binary packages can be found on the github release page: 
>
> https://github.com/jgm/pandoc/releases/tag/1.17 
>
> Thanks to all who contributed, and especially to Jesse Rosenthal, 
> Chris Forster, Mauro Bieg, and Albert Krewinkel. 
>
> Highlights of this release: 
>
> * A new output format, tei, contributed by Chris Forster. 
>   TEI Simple is an XML format for archiving books. 
>
> * A new --file-scope option contributed by Jesse Rosenthal. 
>   By default pandoc concatenates input files before parsing; 
>   with --file-scope, it will parse each file individually, 
>   and then combine the parsed contents.  This is helpful 
>   for some applications. 
>
> * Many performance improvements and bug fixes. 
>
> API changes: 
>
> * Added writeTEI 
> * Added a general ByteStringReader with warnings, used by 
>   the docx reader 
> * Added readDocxWithWarnings 
> * Changed type of Shared.uniqueIdent argument from [String] 
>   to Set String 
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/609d178d-7ecb-4d34-a8e7-74bf127b289b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 2921 bytes --]

Re: ANN: pandoc 1.17

[BP Jonsson]

Den 2016-03-21 kl. 17:46, skrev John MacFarlane:
> +++ BP Jonsson [Mar 21 16 14:10 ]:
>
>>>  * Markdown writer: Use hyphens for YAML metadata block bottom
>>> line, for
>>>    better compatibility with other Markdown flavors (Henrik
>>> Tramberend).
>>
>> It also makes it invalid YAML! Which means that you can no

>
> I'm not convinced there's a real practical application where
> having the `...` separator at the end matters.
>
> Do you have an actual existing workflow that breaks because
> of this change?

As it happens I have an application where, while not breaking 
things entirely, the `---` at the end introduces a gotcha in a 
Perl script which calls on pandoc to mirror Markdown files in one 
directory as HTML files in another directory. As part of the 
process the script retrieves the metadata from each Markdown file 
(in order to get the title and build a navigation sidebar) by 
calling pandoc with a template called "yaml.markdown" which contains

     $if(titleblock)$
     $titleblock$
     $else$
     --- {}
     $endif$

and then parses what it gets with a standard YAML tool. What 
happens is, to the best of my knowledge, that because of the 
trailing `---` the YAML parser now sees two YAML documents, the 
second of which is empty, and in scalar context the YAML loading 
function returns the last, empty, document, which happens to be 
returned as an undefined value. Switching to list context 
(`my($meta) = Load $yaml` instead of `my $meta = Load $yaml`) 
removes the need for removing the second 'ghost' document before 
parsing, but it's definitely a gotcha.

While it is not that big a deal it is irritating to now get 
invalid YAML, or rather valid YAML which returns junk, where I 
formerly got valid YAML which could be loaded in list or scalar 
context without further ado.  It is frustrating that pandoc shall 
have to start producing invalid YAML because the rest of the world 
does!