Thought experiment on Pandoc Markdown's syntax

Thought experiment on Pandoc Markdown's syntax

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1461 bytes --]

Dear all,
I recently found this comment summarizing, accurately, I think, much of the 
philosophy behind Pandoc's markdown syntax:
https://github.com/jgm/pandoc/issues/168#issuecomment-327303614

I noticed that some of the considerations are about (a) "markdown-ness" and 
the "overriding design goal" as laid down by John Gruber; and (b) 
compatibility with Gruber & Swartz's markdown syntax.

If you could re-design Pandoc's Markdown from scratch, with the benefit of 
hindsight, and with the primary purpose of serving as a simple-to-write 
markup for scholarly writing and to serve as Pandoc's native format, what 
would you change?  Do you have a favourite bugbear in the Pandoc Markdown 
syntax?

(For instance: perhaps having three different ways of creating horizontal 
rules may be a design error, since it (a) takes away characters from 
fulfilling other functions in the markup; and (b) horizontal rules aren't 
as important today as they perhaps seemed in 2004?)

I felt this would be an interesting thought experiment.

Cheers,
Pranesh


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c04ebc30-7376-4399-bcbe-0889ec5f507dn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1943 bytes --]

Re: Thought experiment on Pandoc Markdown's syntax

[John MacFarlane]

I've written about this here:
https://www.johnmacfarlane.net/beyond-markdown.html


Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Dear all,
> I recently found this comment summarizing, accurately, I think, much of the 
> philosophy behind Pandoc's markdown syntax:
> https://github.com/jgm/pandoc/issues/168#issuecomment-327303614
>
> I noticed that some of the considerations are about (a) "markdown-ness" and 
> the "overriding design goal" as laid down by John Gruber; and (b) 
> compatibility with Gruber & Swartz's markdown syntax.
>
> If you could re-design Pandoc's Markdown from scratch, with the benefit of 
> hindsight, and with the primary purpose of serving as a simple-to-write 
> markup for scholarly writing and to serve as Pandoc's native format, what 
> would you change?  Do you have a favourite bugbear in the Pandoc Markdown 
> syntax?
>
> (For instance: perhaps having three different ways of creating horizontal 
> rules may be a design error, since it (a) takes away characters from 
> fulfilling other functions in the markup; and (b) horizontal rules aren't 
> as important today as they perhaps seemed in 2004?)
>
> I felt this would be an interesting thought experiment.
>
> Cheers,
> Pranesh
>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c04ebc30-7376-4399-bcbe-0889ec5f507dn%40googlegroups.com.

Re: Thought experiment on Pandoc Markdown's syntax

[Lloyd R. Prentice]

Ideally markdown syntax should be as simple as possible to encourage wide adoption yet sufficient expressive to support academic, technical, and literary work. With this in mind, John’s comments with regard to ease of parsing are worthy of considerable discussion and research. 

If the syntax is easy to parse, then we can expect parsers written in many computer languages (as is essentially the case now). Language-specific parsers simplify embedding markdown into applications, make more efficient use of compute resources, and reduce the cognitive load on programmers by simplifying the interface between parser and application.

The biggest problem I see, however, is in the parser-to-output formatIng link.

Pandoc handles this quite beautifully, but it in a brute-force, Swiss Army Knife fashion at the cost of large memory footprint. And, in my experience so far, styling PDF print output is a murky dark art.

The looming challenge then is this—the near infinitely diverse population of potential markdown adopters demands that the ecosystem of parsers and writers support all needs of all users. This is a formidable burden to place on developers and maintainers.

Much can be learned, in my view, by adopting the UNIX philosophy of small single-purposes tools piped together through a simple and consistent interface protocol.

Best,

LRP


Sent from my iPad

> On Oct 25, 2020, at 1:16 PM, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
> 
> 
> I've written about this here:
> https://www.johnmacfarlane.net/beyond-markdown.html
> 
> 
> Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> 
>> Dear all,
>> I recently found this comment summarizing, accurately, I think, much of the 
>> philosophy behind Pandoc's markdown syntax:
>> https://github.com/jgm/pandoc/issues/168#issuecomment-327303614
>> 
>> I noticed that some of the considerations are about (a) "markdown-ness" and 
>> the "overriding design goal" as laid down by John Gruber; and (b) 
>> compatibility with Gruber & Swartz's markdown syntax.
>> 
>> If you could re-design Pandoc's Markdown from scratch, with the benefit of 
>> hindsight, and with the primary purpose of serving as a simple-to-write 
>> markup for scholarly writing and to serve as Pandoc's native format, what 
>> would you change?  Do you have a favourite bugbear in the Pandoc Markdown 
>> syntax?
>> 
>> (For instance: perhaps having three different ways of creating horizontal 
>> rules may be a design error, since it (a) takes away characters from 
>> fulfilling other functions in the markup; and (b) horizontal rules aren't 
>> as important today as they perhaps seemed in 2004?)
>> 
>> I felt this would be an interesting thought experiment.
>> 
>> Cheers,
>> Pranesh
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c04ebc30-7376-4399-bcbe-0889ec5f507dn%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m28sbup6oi.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/55D3E71D-ABA0-4D7A-8E64-7809FEF327FC%40writersglen.com.

Re: Thought experiment on Pandoc Markdown's syntax

[Nils Carlson]

Adding to this discussion - we use asciidoc - for precisely the reasons 
stated below.

Our pipeline is asciidoc --asciidoctor--> docbook --pandoc--> odt 
(opendocument).

We looked at using markdown, but markdown has serious problems with 
compatibility - essentially you need to know which markdown tool will be 
used for the generation and use the dialect of markdown associated with 
that tool. Asciidoc has two implementations - one stale one - the 
original asciidoc - and one living - asciidoctor. With only two 
implementations and one derived and supporting the syntax of the first, 
things become much easier for those writing documents.

Asciidoc also started off as a tool to generate docbook, a structured 
document format, and was therefore designed from the beginning of a 
professional use-case. This shows a lot when doing things like tables, 
or side-notes or similar.

I can't say that there is anything wrong with pandocs markdown - the 
problem is that markdown was never standardized - and I think it's too 
late at this point. It reminds me a lot of shell coding in the days of 
older and more diverse unix systems - you end up searching for common 
denominators, or you give up and do a lot of verification. Today most 
people with a large shell script just run /bin/bash - and avoid /bin/sh. 
Or move to another language all together such as python.

Cheers,

Nils

On 10/25/20 6:32 PM, Lloyd R. Prentice wrote:
> Ideally markdown syntax should be as simple as possible to encourage wide adoption yet sufficient expressive to support academic, technical, and literary work. With this in mind, John’s comments with regard to ease of parsing are worthy of considerable discussion and research.
>
> If the syntax is easy to parse, then we can expect parsers written in many computer languages (as is essentially the case now). Language-specific parsers simplify embedding markdown into applications, make more efficient use of compute resources, and reduce the cognitive load on programmers by simplifying the interface between parser and application.
>
> The biggest problem I see, however, is in the parser-to-output formatIng link.
>
> Pandoc handles this quite beautifully, but it in a brute-force, Swiss Army Knife fashion at the cost of large memory footprint. And, in my experience so far, styling PDF print output is a murky dark art.
>
> The looming challenge then is this—the near infinitely diverse population of potential markdown adopters demands that the ecosystem of parsers and writers support all needs of all users. This is a formidable burden to place on developers and maintainers.
>
> Much can be learned, in my view, by adopting the UNIX philosophy of small single-purposes tools piped together through a simple and consistent interface protocol.
>
> Best,
>
> LRP
>
>
> Sent from my iPad
>
>> On Oct 25, 2020, at 1:16 PM, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>>
>> 
>> I've written about this here:
>> https://www.johnmacfarlane.net/beyond-markdown.html
>>
>>
>> Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>
>>> Dear all,
>>> I recently found this comment summarizing, accurately, I think, much of the
>>> philosophy behind Pandoc's markdown syntax:
>>> https://github.com/jgm/pandoc/issues/168#issuecomment-327303614
>>>
>>> I noticed that some of the considerations are about (a) "markdown-ness" and
>>> the "overriding design goal" as laid down by John Gruber; and (b)
>>> compatibility with Gruber & Swartz's markdown syntax.
>>>
>>> If you could re-design Pandoc's Markdown from scratch, with the benefit of
>>> hindsight, and with the primary purpose of serving as a simple-to-write
>>> markup for scholarly writing and to serve as Pandoc's native format, what
>>> would you change?  Do you have a favourite bugbear in the Pandoc Markdown
>>> syntax?
>>>
>>> (For instance: perhaps having three different ways of creating horizontal
>>> rules may be a design error, since it (a) takes away characters from
>>> fulfilling other functions in the markup; and (b) horizontal rules aren't
>>> as important today as they perhaps seemed in 2004?)
>>>
>>> I felt this would be an interesting thought experiment.
>>>
>>> Cheers,
>>> Pranesh
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c04ebc30-7376-4399-bcbe-0889ec5f507dn%40googlegroups.com.
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m28sbup6oi.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/32f43b23-d4bc-d865-a7cf-c7dc75ad513e%40nilscarlson.se.

Re: Thought experiment on Pandoc Markdown's syntax

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 6138 bytes --]

Arguably Pandoc *does* follow the Unix philosophy as nearly as can
reasonably be done in this problem space with its multiple readers and
writers and intermediary AST representation. Sure you could have each
reader or writer be a separate process and pipe the JSON representation of
the AST between them, but then you also lose something since each
reader/writer would have to duplicate everything which is now common to all
readers and all writers, not least parsing configuration, whether given on
the command line or in YAML files, and in the YAML case that parsing would
need to be done twice, once by the reader and once by the writer. Last but
not least you would need to convince everybody to accept the JSON
representation *as is* as intermediate format, or soon you will soon have
umphteen dialects of the JSON format and will have to write tools to
translate between them too. Sometimes having a black box is an advantage!

In principle you can now reduce duplication of work by saving the JSON
representation of the AST to a file and use that as input for producing
different output formats, which probably would be faster than parsing the
Markdown once for each output format, but that is arguably
micro-optimization.




-- 
Better --help|less than helpless

Den sön 25 okt. 2020 19:33Lloyd R. Prentice <lloyd-l7gIAb2iU4jcM+WK4BI3xw@public.gmane.org> skrev:

> Ideally markdown syntax should be as simple as possible to encourage wide
> adoption yet sufficient expressive to support academic, technical, and
> literary work. With this in mind, John’s comments with regard to ease of
> parsing are worthy of considerable discussion and research.
>
> If the syntax is easy to parse, then we can expect parsers written in many
> computer languages (as is essentially the case now). Language-specific
> parsers simplify embedding markdown into applications, make more efficient
> use of compute resources, and reduce the cognitive load on programmers by
> simplifying the interface between parser and application.
>
> The biggest problem I see, however, is in the parser-to-output formatIng
> link.
>
> Pandoc handles this quite beautifully, but it in a brute-force, Swiss Army
> Knife fashion at the cost of large memory footprint. And, in my experience
> so far, styling PDF print output is a murky dark art.
>
> The looming challenge then is this—the near infinitely diverse population
> of potential markdown adopters demands that the ecosystem of parsers and
> writers support all needs of all users. This is a formidable burden to
> place on developers and maintainers.
>
> Much can be learned, in my view, by adopting the UNIX philosophy of small
> single-purposes tools piped together through a simple and consistent
> interface protocol.
>
> Best,
>
> LRP
>
>
> Sent from my iPad
>
> > On Oct 25, 2020, at 1:16 PM, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
> >
> > 
> > I've written about this here:
> > https://www.johnmacfarlane.net/beyond-markdown.html
> >
> >
> > Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> >
> >> Dear all,
> >> I recently found this comment summarizing, accurately, I think, much of
> the
> >> philosophy behind Pandoc's markdown syntax:
> >> https://github.com/jgm/pandoc/issues/168#issuecomment-327303614
> >>
> >> I noticed that some of the considerations are about (a) "markdown-ness"
> and
> >> the "overriding design goal" as laid down by John Gruber; and (b)
> >> compatibility with Gruber & Swartz's markdown syntax.
> >>
> >> If you could re-design Pandoc's Markdown from scratch, with the benefit
> of
> >> hindsight, and with the primary purpose of serving as a simple-to-write
> >> markup for scholarly writing and to serve as Pandoc's native format,
> what
> >> would you change?  Do you have a favourite bugbear in the Pandoc
> Markdown
> >> syntax?
> >>
> >> (For instance: perhaps having three different ways of creating
> horizontal
> >> rules may be a design error, since it (a) takes away characters from
> >> fulfilling other functions in the markup; and (b) horizontal rules
> aren't
> >> as important today as they perhaps seemed in 2004?)
> >>
> >> I felt this would be an interesting thought experiment.
> >>
> >> Cheers,
> >> Pranesh
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/c04ebc30-7376-4399-bcbe-0889ec5f507dn%40googlegroups.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/m28sbup6oi.fsf%40MacBook-Pro.hsd1.ca.comcast.net
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/55D3E71D-ABA0-4D7A-8E64-7809FEF327FC%40writersglen.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhARCP-YVx6zwqiEUpnE3OjUXFM50-uNE_iq6--PBisjjA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 8378 bytes --]

Re: Thought experiment on Pandoc Markdown's syntax

[John MacFarlane]

Nils Carlson <nils-tjEGpdrBazrmlNawlTRb+A@public.gmane.org> writes:

> I can't say that there is anything wrong with pandocs markdown - the 
> problem is that markdown was never standardized - and I think it's too 
> late at this point.

This is a strange criticism to make in the context of preferring
asciidoc.  After all, there *is* a spec for commonmark, which is
the basis of a steadily increasing percentage of the Markdown
parsers around (including now GitHub, GitLab, stackoverflow,
reddit, Swift, QT, and in the not too distant future pandoc).
There's nothing comparable for asciidoc (a year or two ago I did
hear an announcement of a project to create a formal spec for
asciidoc, but nothing has come of it as far as I know).

Asciidoc just has the (good? bad?) fortune of having only two
implementations (which differ from each other in parsing, in
both documented and undocumented ways), so there is less
variation.

By your logic, pandoc's markdown would instantly become
preferable to asciidoc if I just gave it another name, such as
"panmark," since then there would be only one implementation
that deals with it!  (Or we could go the other way, and add
an asciidoc reader to pandoc -- then there would be three
implementations of asciidoc.)

Re: Thought experiment on Pandoc Markdown's syntax

[Nils Carlson]

> On 26 Oct 2020, at 21:32, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
> 
> Asciidoc just has the (good? bad?) fortune of having only two
> implementations (which differ from each other in parsing, in
> both documented and undocumented ways), so there is less
> variation.

Yes - this is pretty much the point - it’s that way with most single implementation languages. Rust or Go for example have this same advantage - while I mostly code C and C++ which are standardised but with heavy use of dialects dependent on whether you are on Unix or Windows.

That’s not to say having only one implementation is better than having many - nobody would argue that the Clang and LLVM projects haven’t had a positive impact on the quality of GCC. I’m just saying that it means there is less to worry about when it comes to interoperability.

> 
> By your logic, pandoc's markdown would instantly become
> preferable to asciidoc if I just gave it another name, such as
> "panmark," since then there would be only one implementation
> that deals with it!  (Or we could go the other way, and add
> an asciidoc reader to pandoc -- then there would be three
> implementations of asciidoc.)
> 

In a way yes - this is true - if I saw “panmark" I would know exactly what to expect. With markdown I don’t - but as you say the situation is getting better and better. Of course, using the name “panmark” may not be a good marketing strategy, it would just bring clarity to users who recognise the term.

I would love for pandoc to have a first-rate asciidoc implementation - it would definitely make a valuable addition. I think common ground could be found with the author of asciidoctor - I would say that today this is the “reference” implementation, the original asciidoc project is pretty dead.

Cheers,
Nils

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/91035854-536C-4E8D-B2D5-D2A5BC7FB793%40nilscarlson.se.