Syntactically extensible markdown - would something like this work well with pandoc?

Syntactically extensible markdown - would something like this work well with pandoc?

[Daniel Littlewood]

[-- Attachment #1.1: Type: text/plain, Size: 2135 bytes --]

I use pandoc for generating my personal website from Markdown. I really 
like this statement from the "philosophy" section of the manual 
<https://pandoc.org/MANUAL.html#philosophy>:

> ... while pandoc allows the embedding of raw HTML, it discourages it, and 
provides other, non-HTMLish ways of representing important document 
elements like definition lists, tables, mathematics, and footnotes.

At the moment, some of my pages are in HTML because the appropriate feature 
does not exist in markdown. (for example, I use the HTML `details` 
element). I could include HTML in the page but there are constraints there 
(I can't get nice syntax highlighting, and I can't indent properly). I want 
to keep the simple style of markdown, but to "add syntax" that converts to 
the markup I want. Obviously you'd have to come up with syntax that doesn't 
clash with the existing one.

I notice that there are various extensions to Markdown added to the Pandoc 
parser. I wonder how easy it is to write and add new ones? For comparison, 
there is another project, markdown-it 
<https://github.com/markdown-it/markdown-it> which provides facility for 
arbitrary extension of markdown, but is tightly scoped only to HTML. 

I do not understand the workings of pandoc in detail, but I do understand 
the basic idea - that there is a central "document" model, which defines 
things like tables, lists and so on abstractly - and each format has a 
reader and writer in terms of this model. It's possible that the features I 
want to add are not general enough to align with the goals of pandoc.

Finally, if you know any other projects that are working in a similar 
direction, I'd be very interested to hear about them! 

Thanks,
Dan

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8645831f-c1c6-4dba-b1af-f63f1e830821n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2623 bytes --]

Re: Syntactically extensible markdown - would something like this work well with pandoc?

[John MacFarlane]

> On Aug 9, 2022, at 1:15 PM, Daniel Littlewood <danielittlewood-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> I use pandoc for generating my personal website from Markdown. I really like this statement from the "philosophy" section of the manual:
> 
> > ... while pandoc allows the embedding of raw HTML, it discourages it, and provides other, non-HTMLish ways of representing important document elements like definition lists, tables, mathematics, and footnotes.
> 
> At the moment, some of my pages are in HTML because the appropriate feature does not exist in markdown. (for example, I use the HTML `details` element). I could include HTML in the page but there are constraints there (I can't get nice syntax highlighting, and I can't indent properly). I want to keep the simple style of markdown, but to "add syntax" that converts to the markup I want. Obviously you'd have to come up with syntax that doesn't clash with the existing one.
> 
> I notice that there are various extensions to Markdown added to the Pandoc parser. I wonder how easy it is to write and add new ones? For comparison, there is another project, markdown-it which provides facility for arbitrary extension of markdown, but is tightly scoped only to HTML. 
> 
> I do not understand the workings of pandoc in detail, but I do understand the basic idea - that there is a central "document" model, which defines things like tables, lists and so on abstractly - and each format has a reader and writer in terms of this model. It's possible that the features I want to add are not general enough to align with the goals of pandoc.

The easiest approach is to learn how to use Lua filters; this gives you a ton of flexibility. For example, you could write a simple filter that turned markdown

::: {.details}
Summary here.

Rest of it here.

Continued.
:::

into

<details>
<summary>
Summary here.
</summary>
<p>Rest of it here.</p>
<p>Continued</p>
</details>

in HTML, and fell back to something else in other formats.

See the online documentation on lua filters; for more examples, see the github repo pandoc/lua-filters and this mailing list.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/D0F90C99-14A6-42D8-A422-7AF5C9FAB8D7%40gmail.com.

Re: Syntactically extensible markdown - would something like this work well with pandoc?

[Jiří Wolker]

> The easiest approach is to learn how to use Lua filters; this gives you a ton of flexibility. For example, you could write a simple filter that turned markdown
> 
> ::: {.details}
> Summary here.
> 
> Rest of it here.
> 
> Continued.
> :::
> 
> into
> 
> <details>
> <summary>
> Summary here.
> </summary>
> <p>Rest of it here.</p>
> <p>Continued</p>
> </details>

You can even write this:

     ::: details
     Lorem isum dolor sit amet.
     :::

That should look identical to the Lua filter.

If you feel better with using Haskell, you can also write filters in it. 
Its pattern matching feature is quite comfortable when writing Pandoc 
filters.

See https://pandoc.org/filters.html (incl. Haskell filters).

If you want some specialties, you can even write your own reader or writer.

Re: Syntactically extensible markdown - would something like this work well with pandoc?

[Daniel Littlewood]

[-- Attachment #1.1: Type: text/plain, Size: 2152 bytes --]

Hi both, thank you for the tips! I am more familiar with Haskell so I took 
a look at that approach first.
I cribbed the ruby example on the filter docs - but I'm a little stuck.
The attached filter works on an example like the following:

```
$ pandoc -f markdown -t html -F details
::: details
hello

goodbye
:::
<details><summary>hello</summary>goodbye</details>
```

but pattern matching will fail if the inner details content is anything but 
the simplest markup.
Ideally I would do something like "the first block goes in the summary and 
the rest gets rendered as usual outside". But I couldn't figure out how to 
do that - could you give me any ideas?
(by the way, I'm a very newbie haskeller, so any advice on code style is 
also welcome, but not required)

All the best,
Dan

On Wednesday, August 10, 2022 at 10:56:21 AM UTC+1 wol...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:

> > The easiest approach is to learn how to use Lua filters; this gives you 
> a ton of flexibility. For example, you could write a simple filter that 
> turned markdown
> > 
> > ::: {.details}
> > Summary here.
> > 
> > Rest of it here.
> > 
> > Continued.
> > :::
> > 
> > into
> > 
> > <details>
> > <summary>
> > Summary here.
> > </summary>
> > <p>Rest of it here.</p>
> > <p>Continued</p>
> > </details>
>
> You can even write this:
>
> ::: details
> Lorem isum dolor sit amet.
> :::
>
> That should look identical to the Lua filter.
>
> If you feel better with using Haskell, you can also write filters in it. 
> Its pattern matching feature is quite comfortable when writing Pandoc 
> filters.
>
> See https://pandoc.org/filters.html (incl. Haskell filters).
>
> If you want some specialties, you can even write your own reader or writer.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9910f586-ee3f-4719-afc4-dd95ffe6e46an%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3298 bytes --]

[-- Attachment #2: details.hs --]
[-- Type: text/x-haskell, Size: 635 bytes --]

{-# LANGUAGE OverloadedStrings #-}
-- ^ this allows "string literals" to be type-inferred as, say, Text

-- details.hs
import Text.Pandoc.JSON
import System.Environment (getArgs)
import qualified Data.Text as T

details :: Maybe Format -> Block -> Block
details (Just format) x@(Div (_id, classes, _attributes) [Para [Str summary], Para [Str spoiler]]) = 
  case "details" `elem` classes of
    True
      | format == Format "html" -> RawBlock format $
        "<details>" <> "<summary>" <> summary <> "</summary>" <> spoiler <> "</details>"
      | otherwise -> x
    _ -> x
details _ x = x

main :: IO ()
main = toJSONFilter details

Re: Syntactically extensible markdown - would something like this work well with pandoc?

['Saku Laesvuori' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

> but pattern matching will fail if the inner details content is anything but 
> the simplest markup.
> Ideally I would do something like "the first block goes in the summary and 
> the rest gets rendered as usual outside". But I couldn't figure out how to 
> do that - could you give me any ideas?

> details :: Maybe Format -> Block -> Block
> details (Just format) x@(Div (_id, classes, _attributes) [Para [Str summary], Para [Str spoiler]]) = 
>   case "details" `elem` classes of
>     True
>       | format == Format "html" -> RawBlock format $
>         "<details>" <> "<summary>" <> summary <> "</summary>" <> spoiler <> "</details>"
>       | otherwise -> x
>     _ -> x
> details _ x = x

Instead of pattern matching on a list of Para elements with a single
Str element, try matching the head and tail of the list `x@(Div (_id,
classes, _attributes) summary:details)`. Then you can render the summary
and details blocks to html (Text.Pandoc.Writers.HTML).

> (by the way, I'm a very newbie haskeller, so any advice on code style is 
> also welcome, but not required)

You could use guards instead of the case ... of block and test for the
format in pattern matching:

```
details (Just format@(Format "html")) x@(Div ...)
  | "details" `elem` classes = RawBlock format $ ...
  | otherwise = x
details _ x = x
```

Re: Syntactically extensible markdown - would something like this work well with pandoc?

[Daniel Littlewood]

[-- Attachment #1.1: Type: text/plain, Size: 2561 bytes --]

I figured it out - first of all I was assuming RawBlocks have to be closed 
(i.e. you can't have "<details>" without "</details>") and I didn't know 
you could return a list of them. The attached file works perfectly!

On Sunday, August 14, 2022 at 10:24:31 PM UTC+1 Daniel Littlewood wrote:

> Hi both, thank you for the tips! I am more familiar with Haskell so I took 
> a look at that approach first.
> I cribbed the ruby example on the filter docs - but I'm a little stuck.
> The attached filter works on an example like the following:
>
> ```
> $ pandoc -f markdown -t html -F details
> ::: details
> hello
>
> goodbye
> :::
> <details><summary>hello</summary>goodbye</details>
> ```
>
> but pattern matching will fail if the inner details content is anything 
> but the simplest markup.
> Ideally I would do something like "the first block goes in the summary and 
> the rest gets rendered as usual outside". But I couldn't figure out how to 
> do that - could you give me any ideas?
> (by the way, I'm a very newbie haskeller, so any advice on code style is 
> also welcome, but not required)
>
> All the best,
> Dan
>
> On Wednesday, August 10, 2022 at 10:56:21 AM UTC+1 wol...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
>
>> > The easiest approach is to learn how to use Lua filters; this gives you 
>> a ton of flexibility. For example, you could write a simple filter that 
>> turned markdown 
>> > 
>> > ::: {.details} 
>> > Summary here. 
>> > 
>> > Rest of it here. 
>> > 
>> > Continued. 
>> > ::: 
>> > 
>> > into 
>> > 
>> > <details> 
>> > <summary> 
>> > Summary here. 
>> > </summary> 
>> > <p>Rest of it here.</p> 
>> > <p>Continued</p> 
>> > </details> 
>>
>> You can even write this: 
>>
>> ::: details 
>> Lorem isum dolor sit amet. 
>> ::: 
>>
>> That should look identical to the Lua filter. 
>>
>> If you feel better with using Haskell, you can also write filters in it. 
>> Its pattern matching feature is quite comfortable when writing Pandoc 
>> filters. 
>>
>> See https://pandoc.org/filters.html (incl. Haskell filters). 
>>
>> If you want some specialties, you can even write your own reader or 
>> writer. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/71e804e9-be81-43fc-8c40-4e7a1ce77e1fn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3876 bytes --]

[-- Attachment #2: details.hs --]
[-- Type: text/x-haskell, Size: 831 bytes --]

{-# LANGUAGE OverloadedStrings #-}
-- ^ this allows "string literals" to be type-inferred as, say, Text

-- details.hs
import Text.Pandoc.JSON
import System.Environment (getArgs)

details :: Maybe Format -> Block -> [Block]
details (Just (Format "html")) original@(Div (_id, classes, _attributes) (summary:rest))
  | "details" `elem` classes = [ detailOpen]
                            ++ [ summaryOpen, summary, summaryClose ] 
                            ++ rest
                            ++ [ detailClose ]
  | otherwise = [original]
    where [detailOpen, detailClose, 
           summaryOpen, summaryClose] = map (RawBlock "html") ["<details>", "</details>",
                                                               "<summary>", "</summary>"]
details _ original = [original]

main :: IO ()
main = toJSONFilter details

Re: Syntactically extensible markdown - would something like this work well with pandoc?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 4097 bytes --]

A RawBlock or RawInline is just an arbitrary string marked for inclusion in
some output format, or in the case of markdown to be included in a verbatim
block or inline with the special `{=FORMATNAME}` attribute (unless of
course it is `{=markdown}` in which case it will be included verbatim
without any delimiters). It can contain anything or nothing, and it is
entirely up to you to ensure that the result constitutes valid markup
and/or text for the given format, and indeed to make sure that the given
format is one recognised by Pandoc. You can even abuse it for things like
`` `This is a comment`{=comment} `` where the text won't be included in any
output format but will act as a comment which persists if you reformat the
Markdown with Pandoc (use `-f markdown -t markdown` simultaneously.)

Den ons 17 aug. 2022 08:55Daniel Littlewood <danielittlewood-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
skrev:

> I figured it out - first of all I was assuming RawBlocks have to be closed
> (i.e. you can't have "<details>" without "</details>") and I didn't know
> you could return a list of them. The attached file works perfectly!
>
> On Sunday, August 14, 2022 at 10:24:31 PM UTC+1 Daniel Littlewood wrote:
>
>> Hi both, thank you for the tips! I am more familiar with Haskell so I
>> took a look at that approach first.
>> I cribbed the ruby example on the filter docs - but I'm a little stuck.
>> The attached filter works on an example like the following:
>>
>> ```
>> $ pandoc -f markdown -t html -F details
>> ::: details
>> hello
>>
>> goodbye
>> :::
>> <details><summary>hello</summary>goodbye</details>
>> ```
>>
>> but pattern matching will fail if the inner details content is anything
>> but the simplest markup.
>> Ideally I would do something like "the first block goes in the summary
>> and the rest gets rendered as usual outside". But I couldn't figure out how
>> to do that - could you give me any ideas?
>> (by the way, I'm a very newbie haskeller, so any advice on code style is
>> also welcome, but not required)
>>
>> All the best,
>> Dan
>>
>> On Wednesday, August 10, 2022 at 10:56:21 AM UTC+1 wol...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
>> wrote:
>>
>>> > The easiest approach is to learn how to use Lua filters; this gives
>>> you a ton of flexibility. For example, you could write a simple filter that
>>> turned markdown
>>> >
>>> > ::: {.details}
>>> > Summary here.
>>> >
>>> > Rest of it here.
>>> >
>>> > Continued.
>>> > :::
>>> >
>>> > into
>>> >
>>> > <details>
>>> > <summary>
>>> > Summary here.
>>> > </summary>
>>> > <p>Rest of it here.</p>
>>> > <p>Continued</p>
>>> > </details>
>>>
>>> You can even write this:
>>>
>>> ::: details
>>> Lorem isum dolor sit amet.
>>> :::
>>>
>>> That should look identical to the Lua filter.
>>>
>>> If you feel better with using Haskell, you can also write filters in it.
>>> Its pattern matching feature is quite comfortable when writing Pandoc
>>> filters.
>>>
>>> See https://pandoc.org/filters.html (incl. Haskell filters).
>>>
>>> If you want some specialties, you can even write your own reader or
>>> writer.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/71e804e9-be81-43fc-8c40-4e7a1ce77e1fn%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/71e804e9-be81-43fc-8c40-4e7a1ce77e1fn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhAy7e4f__5-uHyoPK74kFbSA9wZJ2rSSFDuHKLvGDSaMQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5715 bytes --]