Can pandoc break up a Markdown by sections when generating HTML pages?

Can pandoc break up a Markdown by sections when generating HTML pages?

[Szabolcs Horvát]

[-- Attachment #1.1: Type: text/plain, Size: 880 bytes --]

I am working on putting some documentation online, and I am generating the 
final pages with pandoc.

Currently I have a gigantic markdown file. Does pandoc have any features to 
generate separate HTML pages for each section? I hope to do this in a way 
that links still work between pages, and that the table of contents is 
still usable.

For reference, this is the current state of the final page (work in 
progress): http://szhorvat.net/mathematica/IGDocumentation/

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1292 bytes --]

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[John MacFarlane]

Not currently.

You could generate docbook from pandoc and then use docbook tools
to generate split HTML.

This gets requested occasionally and maybe you could put up
an issue requesting it.

Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I am working on putting some documentation online, and I am generating the 
> final pages with pandoc.
>
> Currently I have a gigantic markdown file. Does pandoc have any features to 
> generate separate HTML pages for each section? I hope to do this in a way 
> that links still work between pages, and that the table of contents is 
> still usable.
>
> For reference, this is the current state of the final page (work in 
> progress): http://szhorvat.net/mathematica/IGDocumentation/
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480kftfpx7ty.fsf%40johnmacfarlane.net.

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[BP Jonsson]

[-- Attachment #1: Type: text/plain, Size: 2628 bytes --]

I don't think so but you can use a Lua filter to insert a certain string
before each section heading and then use the csplit command to split the
file on that separator.


https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation


If you are on Windows have a look here:


https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html

E.g.

``````lua
local split_level = 2

-- Define once, use repeatedly
local sep = pandoc.RawBlock(
  'markdown', 'XXXsplitXXX'
 )

function Header (elem)
  if split_level ~= elem.level then
    return nil
  else
    return { sep, elem }
  end
end
``````

then

``````sh
pandoc -L sep.lua my-project.md \
 | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
``````

Of course the separator may be anything, but it must not otherwise occur in
your document and should better not contain any regular expression
metacharacters; stick to ASCII alphanumerics and you should be good.


Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:

> I am working on putting some documentation online, and I am generating the
> final pages with pandoc.
>
> Currently I have a gigantic markdown file. Does pandoc have any features
> to generate separate HTML pages for each section? I hope to do this in a
> way that links still work between pages, and that the table of contents is
> still usable.
>
> For reference, this is the current state of the final page (work in
> progress): http://szhorvat.net/mathematica/IGDocumentation/
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4541 bytes --]

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[John MacFarlane]

Your filter would also need to rewrite internal links
so they don't break when the document is split across files.

BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I don't think so but you can use a Lua filter to insert a certain string
> before each section heading and then use the csplit command to split the
> file on that separator.
>
>
> https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
>
>
> If you are on Windows have a look here:
>
>
> https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>
> E.g.
>
> ``````lua
> local split_level = 2
>
> -- Define once, use repeatedly
> local sep = pandoc.RawBlock(
>   'markdown', 'XXXsplitXXX'
>  )
>
> function Header (elem)
>   if split_level ~= elem.level then
>     return nil
>   else
>     return { sep, elem }
>   end
> end
> ``````
>
> then
>
> ``````sh
> pandoc -L sep.lua my-project.md \
>  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
> ``````
>
> Of course the separator may be anything, but it must not otherwise occur in
> your document and should better not contain any regular expression
> metacharacters; stick to ASCII alphanumerics and you should be good.
>
>
> Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:
>
>> I am working on putting some documentation online, and I am generating the
>> final pages with pandoc.
>>
>> Currently I have a gigantic markdown file. Does pandoc have any features
>> to generate separate HTML pages for each section? I hope to do this in a
>> way that links still work between pages, and that the table of contents is
>> still usable.
>>
>> For reference, this is the current state of the final page (work in
>> progress): http://szhorvat.net/mathematica/IGDocumentation/
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
>> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480k1rr8ygaz.fsf%40johnmacfarlane.net.

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[BPJ]

[-- Attachment #1.1: Type: text/plain, Size: 4239 bytes --]

I *think* the attached filter will do the trick. It should make one pass
where it associates each main section id with a sequential filename like
the ones you would get from csplit need every element id with the current
section, then in a second pass it should visit all links and prepend the
appropriate file name to all internal links, unless the link appears to be
in the same section as the target.

BTW if you use csplit you will probably want to use a custom "suffix
format" which includes the right file extension. Just pass an option `-b
'%03d.md'` for example.

Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:

>
> Your filter would also need to rewrite internal links
> so they don't break when the document is split across files.
>
> BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > I don't think so but you can use a Lua filter to insert a certain string
> > before each section heading and then use the csplit command to split the
> > file on that separator.
> >
> >
> >
> https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
> >
> >
> > If you are on Windows have a look here:
> >
> >
> > https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
> >
> > E.g.
> >
> > ``````lua
> > local split_level = 2
> >
> > -- Define once, use repeatedly
> > local sep = pandoc.RawBlock(
> >   'markdown', 'XXXsplitXXX'
> >  )
> >
> > function Header (elem)
> >   if split_level ~= elem.level then
> >     return nil
> >   else
> >     return { sep, elem }
> >   end
> > end
> > ``````
> >
> > then
> >
> > ``````sh
> > pandoc -L sep.lua my-project.md \
> >  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
> > ``````
> >
> > Of course the separator may be anything, but it must not otherwise occur
> in
> > your document and should better not contain any regular expression
> > metacharacters; stick to ASCII alphanumerics and you should be good.
> >
> >
> > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:
> >
> >> I am working on putting some documentation online, and I am generating
> the
> >> final pages with pandoc.
> >>
> >> Currently I have a gigantic markdown file. Does pandoc have any features
> >> to generate separate HTML pages for each section? I hope to do this in a
> >> way that links still work between pages, and that the table of contents
> is
> >> still usable.
> >>
> >> For reference, this is the current state of the final page (work in
> >> progress): http://szhorvat.net/mathematica/IGDocumentation/
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "pandoc-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> >> To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
> >> <
> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> >> .
> >>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhA%3D-uGCjH2Vy6pEM2u%2BtYYSa75Gmf%2BoNUYkYLzEVxtjzA%40mail.gmail.com.

[-- Attachment #1.2: Type: text/html, Size: 6650 bytes --]

[-- Attachment #2: splitlinks.lua --]
[-- Type: application/octet-stream, Size: 1521 bytes --]

local file_fmt = 'my-project%03d'
local sec_level = 2
local sec_count = 0
local cur_sec = {}
local secs = {}
local id_to_sec = {}

local function new_sec ( id )
  sec_count = sec_count + 1
  assert(
    (not secs[id]),
    "Duplicate section id: #" .. id
  )
  local sec = {
    id = id,
    num = sec_count,
    file = string.format(file_fmt, sec_count),
  }
  secs[id] = sec
  return sec
end

cur_sec = new_sec"before-any-sec"

local function collect_ids (elem)
  local id = elem.identifier
  -- safe to assume a heading has an id?
  if 'Header' == elem.tag then
    if elem.level == sec_level then
      cur_sec = new_sec(id)
    end
  elseif 'Link' == elem.tag then
    elem.attributes.sec = cur_sec.id
  end
  if "" == id then return nil end
  assert(
    (not id_to_sec[id]),
    "Duplicate id: #" .. id
  )
  id_to_sec[id] = cur_sec
  return nil
end

local function link_to_sec (link)
  local id = match( link.target, '^%#(.*)') 
  local link_sec = link.attributes.sec
  link.attributes.sec = nil
  if id then
    local sec = assert(
      id_to_sec[id],
      "Link to unknown id: #" .. id
    )
    if link_sec ~= sec.id then
      -- the target is in another sec
      link.target = sec.file .. link.target
    end
    return link
  end
  return nil
end

local collector = {}

local elems_with_ids = {
  'CodeBlock',
  'Div',
  'Header',
  'Code',
  'Image',
  'Link',
  'Span',
}

for _,name in ipairs(elems_with_ids) do
  collector[name] = collect_ids
end

return {
  collector,
  { Link = link_to_sec },
}

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 4634 bytes --]

And of course I forgot to add the extension to the file name format in the
filter.  You will have to adjust the variables at the top to reflect your
reality anyway of course!



Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:

> I *think* the attached filter will do the trick. It should make one pass
> where it associates each main section id with a sequential filename like
> the ones you would get from csplit need every element id with the current
> section, then in a second pass it should visit all links and prepend the
> appropriate file name to all internal links, unless the link appears to be
> in the same section as the target.
>
> BTW if you use csplit you will probably want to use a custom "suffix
> format" which includes the right file extension. Just pass an option `-b
> '%03d.md'` for example.
>
> Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:
>
>>
>> Your filter would also need to rewrite internal links
>> so they don't break when the document is split across files.
>>
>> BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>
>> > I don't think so but you can use a Lua filter to insert a certain string
>> > before each section heading and then use the csplit command to split the
>> > file on that separator.
>> >
>> >
>> >
>> https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
>> >
>> >
>> > If you are on Windows have a look here:
>> >
>> >
>> > https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>> >
>> > E.g.
>> >
>> > ``````lua
>> > local split_level = 2
>> >
>> > -- Define once, use repeatedly
>> > local sep = pandoc.RawBlock(
>> >   'markdown', 'XXXsplitXXX'
>> >  )
>> >
>> > function Header (elem)
>> >   if split_level ~= elem.level then
>> >     return nil
>> >   else
>> >     return { sep, elem }
>> >   end
>> > end
>> > ``````
>> >
>> > then
>> >
>> > ``````sh
>> > pandoc -L sep.lua my-project.md \
>> >  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
>> > ``````
>> >
>> > Of course the separator may be anything, but it must not otherwise
>> occur in
>> > your document and should better not contain any regular expression
>> > metacharacters; stick to ASCII alphanumerics and you should be good.
>> >
>> >
>> > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:
>> >
>> >> I am working on putting some documentation online, and I am generating
>> the
>> >> final pages with pandoc.
>> >>
>> >> Currently I have a gigantic markdown file. Does pandoc have any
>> features
>> >> to generate separate HTML pages for each section? I hope to do this in
>> a
>> >> way that links still work between pages, and that the table of
>> contents is
>> >> still usable.
>> >>
>> >> For reference, this is the current state of the final page (work in
>> >> progress): http://szhorvat.net/mathematica/IGDocumentation/
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> Groups
>> >> "pandoc-discuss" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> an
>> >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> >> To view this discussion on the web visit
>> >>
>> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
>> >> <
>> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer
>> >
>> >> .
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "pandoc-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 7406 bytes --]

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[ChrisD]

[-- Attachment #1: Type: text/plain, Size: 6767 bytes --]

Wouldn't you have to further post-process the output files to make each file a stand alone document? For example each output file needs to include or reference the css file.

This subject does come up on the list with some regularity, although it can be called different things. Use cases I can think of off the top of my head:
- Chunked HTML (as in this thread)
- HTMLHelp output (I would *love* to have this)
- docx output separated by chapters

I think the basic functionality must exist already in the epub writer. Could that be moved somehow to at least support chunked html and htmlhelp?

+1  Please consider adding this to core Pandoc. Thanks,
Chris



On 2/5/2020 4:00 PM, BPJ wrote:
> And of course I forgot to add the extension to the file name format in the filter.  You will have to adjust the variables at the top to reflect your reality anyway of course!
>
>
>
> Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>
>     I *think* the attached filter will do the trick. It should make one pass where it associates each main section id with a sequential filename like the ones you would get from csplit need every element id with the current section, then in a second pass it should visit all links and prepend the appropriate file name to all internal links, unless the link appears to be in the same section as the target.
>
>     BTW if you use csplit you will probably want to use a custom "suffix format" which includes the right file extension. Just pass an option `-b '%03d.md'` for example.
>
>     Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org <mailto:jgm@berkeley.edu>> skrev:
>
>
>         Your filter would also need to rewrite internal links
>         so they don't break when the document is split across files.
>
>         BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> writes:
>
>         > I don't think so but you can use a Lua filter to insert a certain string
>         > before each section heading and then use the csplit command to split the
>         > file on that separator.
>         >
>         >
>         > https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
>         >
>         >
>         > If you are on Windows have a look here:
>         >
>         >
>         > https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>         >
>         > E.g.
>         >
>         > ``````lua
>         > local split_level = 2
>         >
>         > -- Define once, use repeatedly
>         > local sep = pandoc.RawBlock(
>         >   'markdown', 'XXXsplitXXX'
>         >  )
>         >
>         > function Header (elem)
>         >   if split_level ~= elem.level then
>         >     return nil
>         >   else
>         >     return { sep, elem }
>         >   end
>         > end
>         > ``````
>         >
>         > then
>         >
>         > ``````sh
>         > pandoc -L sep.lua my-project.md \
>         >  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
>         > ``````
>         >
>         > Of course the separator may be anything, but it must not otherwise occur in
>         > your document and should better not contain any regular expression
>         > metacharacters; stick to ASCII alphanumerics and you should be good.
>         >
>         >
>         > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat@gmail.com <mailto:szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>         >
>         >> I am working on putting some documentation online, and I am generating the
>         >> final pages with pandoc.
>         >>
>         >> Currently I have a gigantic markdown file. Does pandoc have any features
>         >> to generate separate HTML pages for each section? I hope to do this in a
>         >> way that links still work between pages, and that the table of contents is
>         >> still usable.
>         >>
>         >> For reference, this is the current state of the final page (work in
>         >> progress): http://szhorvat.net/mathematica/IGDocumentation/
>         >>
>         >> --
>         >> You received this message because you are subscribed to the Google Groups
>         >> "pandoc-discuss" group.
>         >> To unsubscribe from this group and stop receiving emails from it, send an
>         >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>         >> To view this discussion on the web visit
>         >> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
>         >> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>         >> .
>         >>
>         >
>         > --
>         > You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>         > To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>         > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com <https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0ea619eb-11ae-16cd-6861-de6fb465b92d%40intielectronics.com.

[-- Attachment #2: Type: text/html, Size: 12542 bytes --]

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[John MacFarlane]

Someone should put up an issue on the tracker if there isn't one
already.

ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> writes:

> Wouldn't you have to further post-process the output files to make each file a stand alone document? For example each output file needs to include or reference the css file.
>
> This subject does come up on the list with some regularity, although it can be called different things. Use cases I can think of off the top of my head:
> - Chunked HTML (as in this thread)
> - HTMLHelp output (I would *love* to have this)
> - docx output separated by chapters
>
> I think the basic functionality must exist already in the epub writer. Could that be moved somehow to at least support chunked html and htmlhelp?
>
> +1  Please consider adding this to core Pandoc. Thanks,
> Chris
>
>
>
> On 2/5/2020 4:00 PM, BPJ wrote:
>> And of course I forgot to add the extension to the file name format in the filter.  You will have to adjust the variables at the top to reflect your reality anyway of course!
>>
>>
>>
>> Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:melroch-Re5JQEeQqe8@public.gmane.orgm>> skrev:
>>
>>     I *think* the attached filter will do the trick. It should make one pass where it associates each main section id with a sequential filename like the ones you would get from csplit need every element id with the current section, then in a second pass it should visit all links and prepend the appropriate file name to all internal links, unless the link appears to be in the same section as the target.
>>
>>     BTW if you use csplit you will probably want to use a custom "suffix format" which includes the right file extension. Just pass an option `-b '%03d.md'` for example.
>>
>>     Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org <mailto:jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>> skrev:
>>
>>
>>         Your filter would also need to rewrite internal links
>>         so they don't break when the document is split across files.
>>
>>         BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> writes:
>>
>>         > I don't think so but you can use a Lua filter to insert a certain string
>>         > before each section heading and then use the csplit command to split the
>>         > file on that separator.
>>         >
>>         >
>>         > https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
>>         >
>>         >
>>         > If you are on Windows have a look here:
>>         >
>>         >
>>         > https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>>         >
>>         > E.g.
>>         >
>>         > ``````lua
>>         > local split_level = 2
>>         >
>>         > -- Define once, use repeatedly
>>         > local sep = pandoc.RawBlock(
>>         >   'markdown', 'XXXsplitXXX'
>>         >  )
>>         >
>>         > function Header (elem)
>>         >   if split_level ~= elem.level then
>>         >     return nil
>>         >   else
>>         >     return { sep, elem }
>>         >   end
>>         > end
>>         > ``````
>>         >
>>         > then
>>         >
>>         > ``````sh
>>         > pandoc -L sep.lua my-project.md \
>>         >  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
>>         > ``````
>>         >
>>         > Of course the separator may be anything, but it must not otherwise occur in
>>         > your document and should better not contain any regular expression
>>         > metacharacters; stick to ASCII alphanumerics and you should be good.
>>         >
>>         >
>>         > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat@gmail.com <mailto:szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>>         >
>>         >> I am working on putting some documentation online, and I am generating the
>>         >> final pages with pandoc.
>>         >>
>>         >> Currently I have a gigantic markdown file. Does pandoc have any features
>>         >> to generate separate HTML pages for each section? I hope to do this in a
>>         >> way that links still work between pages, and that the table of contents is
>>         >> still usable.
>>         >>
>>         >> For reference, this is the current state of the final page (work in
>>         >> progress): http://szhorvat.net/mathematica/IGDocumentation/
>>         >>
>>         >> --
>>         >> You received this message because you are subscribed to the Google Groups
>>         >> "pandoc-discuss" group.
>>         >> To unsubscribe from this group and stop receiving emails from it, send an
>>         >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>         >> To view this discussion on the web visit
>>         >> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
>>         >> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>>         >> .
>>         >>
>>         >
>>         > --
>>         > You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>         > To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>         > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com <https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0ea619eb-11ae-16cd-6861-de6fb465b92d%40intielectronics.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480kzhdwv8x4.fsf%40johnmacfarlane.net.

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[ChrisD]

[-- Attachment #1: Type: text/plain, Size: 8557 bytes --]

I could have sworn there was already at least one issue about this, but I don't find anything when I search. So I added #6122 <https://github.com/jgm/pandoc/issues/6122>.

It's a pretty general feature request, but I tried to add the thoughts from this thread and also a thread "Multiple HTML files (split at H1 or H2 level) from one source?" from 4/13/2019. If anyone has other requirements or suggestions for this feature perhaps you could add them to the issue. Also if you know of related issues please link them up.

HtmlHelp format is windows specific, but it's similar to chunked html. I don't know Haskell and can't really help with code, but I am familiar with HtmlHelp files and how to generate them "by hand". If you need help with what files need to be generated or with testing I'm more than willing to help.

Thanks John,



On 2/5/2020 5:47 PM, John MacFarlane wrote:
> Someone should put up an issue on the tracker if there isn't one
> already.
>
> ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> writes:
>
>> Wouldn't you have to further post-process the output files to make each file a stand alone document? For example each output file needs to include or reference the css file.
>>
>> This subject does come up on the list with some regularity, although it can be called different things. Use cases I can think of off the top of my head:
>> - Chunked HTML (as in this thread)
>> - HTMLHelp output (I would *love* to have this)
>> - docx output separated by chapters
>>
>> I think the basic functionality must exist already in the epub writer. Could that be moved somehow to at least support chunked html and htmlhelp?
>>
>> +1  Please consider adding this to core Pandoc. Thanks,
>> Chris
>>
>>
>>
>> On 2/5/2020 4:00 PM, BPJ wrote:
>>> And of course I forgot to add the extension to the file name format in the filter.  You will have to adjust the variables at the top to reflect your reality anyway of course!
>>>
>>>
>>>
>>> Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:melroch@gmail.com>> skrev:
>>>
>>>      I *think* the attached filter will do the trick. It should make one pass where it associates each main section id with a sequential filename like the ones you would get from csplit need every element id with the current section, then in a second pass it should visit all links and prepend the appropriate file name to all internal links, unless the link appears to be in the same section as the target.
>>>
>>>      BTW if you use csplit you will probably want to use a custom "suffix format" which includes the right file extension. Just pass an option `-b '%03d.md'` for example.
>>>
>>>      Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org <mailto:jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>> skrev:
>>>
>>>
>>>          Your filter would also need to rewrite internal links
>>>          so they don't break when the document is split across files.
>>>
>>>          BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> writes:
>>>
>>>          > I don't think so but you can use a Lua filter to insert a certain string
>>>          > before each section heading and then use the csplit command to split the
>>>          > file on that separator.
>>>          >
>>>          >
>>>          > https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
>>>          >
>>>          >
>>>          > If you are on Windows have a look here:
>>>          >
>>>          >
>>>          > https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>>>          >
>>>          > E.g.
>>>          >
>>>          > ``````lua
>>>          > local split_level = 2
>>>          >
>>>          > -- Define once, use repeatedly
>>>          > local sep = pandoc.RawBlock(
>>>          >   'markdown', 'XXXsplitXXX'
>>>          >  )
>>>          >
>>>          > function Header (elem)
>>>          >   if split_level ~= elem.level then
>>>          >     return nil
>>>          >   else
>>>          >     return { sep, elem }
>>>          >   end
>>>          > end
>>>          > ``````
>>>          >
>>>          > then
>>>          >
>>>          > ``````sh
>>>          > pandoc -L sep.lua my-project.md \
>>>          >  | csplit -f my-project -n 3 --suppress-matched - /XXXsplitXXX/ {*}
>>>          > ``````
>>>          >
>>>          > Of course the separator may be anything, but it must not otherwise occur in
>>>          > your document and should better not contain any regular expression
>>>          > metacharacters; stick to ASCII alphanumerics and you should be good.
>>>          >
>>>          >
>>>          > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat@gmail.com <mailto:szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>>>          >
>>>          >> I am working on putting some documentation online, and I am generating the
>>>          >> final pages with pandoc.
>>>          >>
>>>          >> Currently I have a gigantic markdown file. Does pandoc have any features
>>>          >> to generate separate HTML pages for each section? I hope to do this in a
>>>          >> way that links still work between pages, and that the table of contents is
>>>          >> still usable.
>>>          >>
>>>          >> For reference, this is the current state of the final page (work in
>>>          >> progress): http://szhorvat.net/mathematica/IGDocumentation/
>>>          >>
>>>          >> --
>>>          >> You received this message because you are subscribed to the Google Groups
>>>          >> "pandoc-discuss" group.
>>>          >> To unsubscribe from this group and stop receiving emails from it, send an
>>>          >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>>          >> To view this discussion on the web visit
>>>          >> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
>>>          >> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>          >> .
>>>          >>
>>>          >
>>>          > --
>>>          > You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>>          > To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>>          > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com <https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0ea619eb-11ae-16cd-6861-de6fb465b92d%40intielectronics.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9f81ad9d-3de1-e5a2-1d48-6213b3793d44%40intielectronics.com.

[-- Attachment #2: Type: text/html, Size: 12646 bytes --]

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 10407 bytes --]

On 2020-02-06 00:49, ChrisD wrote:
> Wouldn't you have to further post-process the output files to make each 
> file a stand alone document? For example each output file needs to 
> include or reference the css file.

Yes adding metadata to each file would probably be needed.
I guess that could be done with a smallish script, maybe even a Makefile 
using either Pandoc itself (--metadata-file or just list the common 
metadata as an input file), or even cat.

Another thing I thought of is that you probably would want to rename the 
files from csplit to reflect the id of the section heading (so that the 
file containing the source for the section whose heading id is 
`#introduction` is called `001-introduction.md` and the HTML file 
ultimately `001-introduction.html`.  I have updated my link-fixing 
script so that it writes the csplit-style file name and the section id 
to a file separated by a tab, and then a Perl script which loops over 
that file and renames the files. Perl is my go-to language; obviously a 
script in any language which can loop over a tsv file and rename/copy 
files would do!
Note that this Perl script uses the CPAN module Path::Tiny.  If you have 
Perl installed but aren't used to installing Perl modules run these 
commands:

``````sh
cpan App::cpanminus
cpanm Path::Tiny
``````

Both cpan and cpanm are Perl module installers, but cpanm is much newer 
and much smarter, although it doesn't get installed automagically when 
perl is installed.

Also I have updated the link-fixing script so that it actually works :-) 
including fixing links to inline elements under each heading.
It turned out I had to use pandoc.utils.make_sections and run the 
id-collecting filter on each section div separately.
Anyway those section divs and the cruft they leave around in the heading 
elements are removed once they have done their job.

The work flow for using these scripts is something like this:

``````sh
mkdir split
mkdir new
cd split/
pandoc -L ../splitlinks.lua -L ../insert-split-marker.lua -w markdown 
--atx-headers ../test.md \
  | csplit -f 'my-project' -b '%03d.md' -z --suppress-matched - 
/XXXsplitXXX/ {*}
mv filenames.tsv ..
cd ..
perl rename-split.pl -p meta.yaml -t filenames.tsv -o split/ -n new/
``````

This can probably be automatized with a Makefile.

You may want to adjust the separator marker in insert-split-marker.lua 
and the variables at the top of splitlinks.lua to reflect your needs.

I've assumed that you would want to split the original Markdown files 
into several Markdown files and then generate HTML from each of them 
separately.  It certainly makes things a lot easier when doing the 
splitting (no risk that HTML start and end tags get split between files! 
:-) and when inserting metadata.  If using a Makefile you can obviously 
remove all intermediate files should you want to.



> 
> This subject does come up on the list with some regularity, although it 
> can be called different things. Use cases I can think of off the top of 
> my head:
> - Chunked HTML (as in this thread)
> - HTMLHelp output (I would *love* to have this)
> - docx output separated by chapters
> 
> I think the basic functionality must exist already in the epub writer. 
> Could that be moved somehow to at least support chunked html and htmlhelp?
> 
> +1  Please consider adding this to core Pandoc. Thanks,
> Chris
> 
> 
> 
> On 2/5/2020 4:00 PM, BPJ wrote:
>> And of course I forgot to add the extension to the file name format in 
>> the filter.  You will have to adjust the variables at the top to 
>> reflect your reality anyway of course!
>>
>>
>>
>> Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
>> <mailto:melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>>
>>     I *think* the attached filter will do the trick. It should make 
>> one pass where it associates each main section id with a sequential 
>> filename like the ones you would get from csplit need every element id 
>> with the current section, then in a second pass it should visit all 
>> links and prepend the appropriate file name to all internal links, 
>> unless the link appears to be in the same section as the target.
>>
>>     BTW if you use csplit you will probably want to use a custom 
>> "suffix format" which includes the right file extension. Just pass an 
>> option `-b '%03d.md'` for example.
>>
>>     Den ons 5 feb. 2020 20:40John MacFarlane <jgm@berkeley.edu 
>> <mailto:jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>> skrev:
>>
>>
>>         Your filter would also need to rewrite internal links
>>         so they don't break when the document is split across files.
>>
>>         BP Jonsson <bpjonsson@gmail.com <mailto:bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> 
>> writes:
>>
>>         > I don't think so but you can use a Lua filter to insert a 
>> certain string
>>         > before each section heading and then use the csplit command 
>> to split the
>>         > file on that separator.
>>         >
>>         >
>>         > 
>> https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation 
>>
>>         >
>>         >
>>         > If you are on Windows have a look here:
>>         >
>>         >
>>         > 
>> https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
>>         >
>>         > E.g.
>>         >
>>         > ``````lua
>>         > local split_level = 2
>>         >
>>         > -- Define once, use repeatedly
>>         > local sep = pandoc.RawBlock(
>>         >   'markdown', 'XXXsplitXXX'
>>         >  )
>>         >
>>         > function Header (elem)
>>         >   if split_level ~= elem.level then
>>         >     return nil
>>         >   else
>>         >     return { sep, elem }
>>         >   end
>>         > end
>>         > ``````
>>         >
>>         > then
>>         >
>>         > ``````sh
>>         > pandoc -L sep.lua my-project.md \
>>         >  | csplit -f my-project -n 3 --suppress-matched - 
>> /XXXsplitXXX/ {*}
>>         > ``````
>>         >
>>         > Of course the separator may be anything, but it must not 
>> otherwise occur in
>>         > your document and should better not contain any regular 
>> expression
>>         > metacharacters; stick to ASCII alphanumerics and you should 
>> be good.
>>         >
>>         >
>>         > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
>> <mailto:szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
>>         >
>>         >> I am working on putting some documentation online, and I am 
>> generating the
>>         >> final pages with pandoc.
>>         >>
>>         >> Currently I have a gigantic markdown file. Does pandoc have 
>> any features
>>         >> to generate separate HTML pages for each section? I hope to 
>> do this in a
>>         >> way that links still work between pages, and that the table 
>> of contents is
>>         >> still usable.
>>         >>
>>         >> For reference, this is the current state of the final page 
>> (work in
>>         >> progress): http://szhorvat.net/mathematica/IGDocumentation/
>>         >>
>>         >> --
>>         >> You received this message because you are subscribed to the 
>> Google Groups
>>         >> "pandoc-discuss" group.
>>         >> To unsubscribe from this group and stop receiving emails 
>> from it, send an
>>         >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org 
>> <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>         >> To view this discussion on the web visit
>>         >> 
>> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com 
>>
>>         >> 
>> <https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer> 
>>
>>         >> .
>>         >>
>>         >
>>         > --
>>         > You received this message because you are subscribed to the 
>> Google Groups "pandoc-discuss" group.
>>         > To unsubscribe from this group and stop receiving emails 
>> from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org 
>> <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>>         > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com. 
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org 
>> <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com?utm_medium=email&utm_source=footer>. 
>>
> 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ad1759f3-f4cc-d5d8-34b7-e9e18bffb1a9%40gmail.com.

[-- Attachment #2: insert-split-marker.lua --]
[-- Type: text/x-lua, Size: 228 bytes --]

local split_level = 2

-- Define once, use repeatedly
local sep = pandoc.Para(
  { pandoc.Str'XXXsplitXXX' }
)

function Header (elem)
  if split_level == elem.level then
    return { sep, elem }
  else
    return nil
  end
end

[-- Attachment #3: rename-split.pl --]
[-- Type: application/x-perl, Size: 2827 bytes --]

[-- Attachment #4: splitlinks.lua --]
[-- Type: text/x-lua, Size: 2801 bytes --]

local split_fmt = 'my-project%03d.md'
local md_fmt  = '%03d-%s.md'
local html_fmt  = '%03d-%s.html'
local tsv_name = 'filenames.tsv'
local sec_level = 2
local sec_count = -1
local cur_sec = {}
local secs = {}
local id_to_sec = {}

local function assertf (v, ...)
  if v then return v end
  error( string.format(...), 2 )
end

local tsv_fh = assertf(
  io.open(tsv_name, 'w+'),
  "Couldn't open file '%s' for writing", tsv_name
)

local function new_sec ( id )
  sec_count = sec_count + 1
  assert( (not secs[id]), "Duplicate section id: #" .. id)
  local sec = {
    id = id,
    num = sec_count,
    split = string.format(split_fmt, sec_count),
    md = string.format(md_fmt, sec_count, id),
    html = string.format(html_fmt, sec_count, id),
  }
  tsv_fh:write(sec.split, "\t", sec.md, "\n")
  secs[id] = sec
  return sec
end

local function get_number (elem)
  return assertf(
    (elem.attributes and elem.attributes.number),
    "No 'number' attribute in %s element", elem.tag
  )
end

cur_sec = new_sec"before-any-sec"

local function collect_ids (elem)
  local id = elem.identifier
  if 'Link' == elem.tag then
    elem.attributes.sec = cur_sec.id
  end
  if "" ~= id then
    assert( (not id_to_sec[id]), "Duplicate id: #" .. id)
    id_to_sec[id] = cur_sec
  end
  return elem
end

local id_collector = {}

local elems_with_ids = {
  'CodeBlock',
  'Div',
  'Header',
  'Code',
  'Image',
  'Link',
  'Span',
}

for _,name in ipairs(elems_with_ids) do
  id_collector[name] = collect_ids
end

local function collect_secs (div)
  if not div.classes:includes('section') then return nil end
  local id = div.identifier
  local number = get_number(div)
  local head = div.content[1]
  assertf(
    ('Header' == head.tag),
    "Expected Header at top of of section div %s #%s", number, id
  )
  assertf(
    (get_number(head) == number),
    "Expected section div #%s and top Header to both have number %s", id, number
  )
  head.identifier = id
  head.attributes.number = nil
  if head.level == sec_level then
    cur_sec = new_sec(id)
    return pandoc.walk_block( div, id_collector ).content
  else
    return div.content
  end
end

local function link_to_sec (link)
  local id = string.match( link.target, '^%#(.*)')
  local link_sec = link.attributes.sec
  link.attributes.sec = nil
  if id then
    local sec = assert( id_to_sec[id], "Link to unknown id: #" .. id)
    if link_sec ~= sec.id then
      -- the target is in another sec
      link.target = sec.html .. link.target
    end
  end
  return link
end

function Pandoc (doc)
  local blocks = pandoc.Div(
    pandoc.utils.make_sections(true, 1, doc.blocks)
  )
  blocks = pandoc.walk_block( blocks, { Div = collect_secs } )
  blocks = pandoc.walk_block( blocks, { Link = link_to_sec } )
  doc.blocks = blocks.content
  return doc
end

Re: Can pandoc break up a Markdown by sections when generating HTML pages?

[Szabolcs Horvát]

[-- Attachment #1: Type: text/plain, Size: 11231 bytes --]

Thank you for the responses to everyone, especially BPJ for providing
detailed scripts for this question, as well as for one of my previous ones.

I wanted to indicate that I read the messages. I'll respond in more detail
on the weekend once I get the time to try to put everything together.

On Thu, 6 Feb 2020 at 17:58, BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On 2020-02-06 00:49, ChrisD wrote:
> > Wouldn't you have to further post-process the output files to make each
> > file a stand alone document? For example each output file needs to
> > include or reference the css file.
>
> Yes adding metadata to each file would probably be needed.
> I guess that could be done with a smallish script, maybe even a Makefile
> using either Pandoc itself (--metadata-file or just list the common
> metadata as an input file), or even cat.
>
> Another thing I thought of is that you probably would want to rename the
> files from csplit to reflect the id of the section heading (so that the
> file containing the source for the section whose heading id is
> `#introduction` is called `001-introduction.md` and the HTML file
> ultimately `001-introduction.html`.  I have updated my link-fixing
> script so that it writes the csplit-style file name and the section id
> to a file separated by a tab, and then a Perl script which loops over
> that file and renames the files. Perl is my go-to language; obviously a
> script in any language which can loop over a tsv file and rename/copy
> files would do!
> Note that this Perl script uses the CPAN module Path::Tiny.  If you have
> Perl installed but aren't used to installing Perl modules run these
> commands:
>
> ``````sh
> cpan App::cpanminus
> cpanm Path::Tiny
> ``````
>
> Both cpan and cpanm are Perl module installers, but cpanm is much newer
> and much smarter, although it doesn't get installed automagically when
> perl is installed.
>
> Also I have updated the link-fixing script so that it actually works :-)
> including fixing links to inline elements under each heading.
> It turned out I had to use pandoc.utils.make_sections and run the
> id-collecting filter on each section div separately.
> Anyway those section divs and the cruft they leave around in the heading
> elements are removed once they have done their job.
>
> The work flow for using these scripts is something like this:
>
> ``````sh
> mkdir split
> mkdir new
> cd split/
> pandoc -L ../splitlinks.lua -L ../insert-split-marker.lua -w markdown
> --atx-headers ../test.md \
>   | csplit -f 'my-project' -b '%03d.md' -z --suppress-matched -
> /XXXsplitXXX/ {*}
> mv filenames.tsv ..
> cd ..
> perl rename-split.pl -p meta.yaml -t filenames.tsv -o split/ -n new/
> ``````
>
> This can probably be automatized with a Makefile.
>
> You may want to adjust the separator marker in insert-split-marker.lua
> and the variables at the top of splitlinks.lua to reflect your needs.
>
> I've assumed that you would want to split the original Markdown files
> into several Markdown files and then generate HTML from each of them
> separately.  It certainly makes things a lot easier when doing the
> splitting (no risk that HTML start and end tags get split between files!
> :-) and when inserting metadata.  If using a Makefile you can obviously
> remove all intermediate files should you want to.
>
>
>
> >
> > This subject does come up on the list with some regularity, although it
> > can be called different things. Use cases I can think of off the top of
> > my head:
> > - Chunked HTML (as in this thread)
> > - HTMLHelp output (I would *love* to have this)
> > - docx output separated by chapters
> >
> > I think the basic functionality must exist already in the epub writer.
> > Could that be moved somehow to at least support chunked html and
> htmlhelp?
> >
> > +1  Please consider adding this to core Pandoc. Thanks,
> > Chris
> >
> >
> >
> > On 2/5/2020 4:00 PM, BPJ wrote:
> >> And of course I forgot to add the extension to the file name format in
> >> the filter.  You will have to adjust the variables at the top to
> >> reflect your reality anyway of course!
> >>
> >>
> >>
> >> Den ons 5 feb. 2020 23:50BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> >> <mailto:melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
> >>
> >>     I *think* the attached filter will do the trick. It should make
> >> one pass where it associates each main section id with a sequential
> >> filename like the ones you would get from csplit need every element id
> >> with the current section, then in a second pass it should visit all
> >> links and prepend the appropriate file name to all internal links,
> >> unless the link appears to be in the same section as the target.
> >>
> >>     BTW if you use csplit you will probably want to use a custom
> >> "suffix format" which includes the right file extension. Just pass an
> >> option `-b '%03d.md'` for example.
> >>
> >>     Den ons 5 feb. 2020 20:40John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org
> >> <mailto:jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>> skrev:
> >>
> >>
> >>         Your filter would also need to rewrite internal links
> >>         so they don't break when the document is split across files.
> >>
> >>         BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <mailto:bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
> >> writes:
> >>
> >>         > I don't think so but you can use a Lua filter to insert a
> >> certain string
> >>         > before each section heading and then use the csplit command
> >> to split the
> >>         > file on that separator.
> >>         >
> >>         >
> >>         >
> >>
> https://www.gnu.org/software/coreutils/manual/html_node/csplit-invocation.html#csplit-invocation
> >>
> >>         >
> >>         >
> >>         > If you are on Windows have a look here:
> >>         >
> >>         >
> >>         >
> >> https://tinyapps.org/blog/201606040700_tiny_unix_tools_windows.html
> >>         >
> >>         > E.g.
> >>         >
> >>         > ``````lua
> >>         > local split_level = 2
> >>         >
> >>         > -- Define once, use repeatedly
> >>         > local sep = pandoc.RawBlock(
> >>         >   'markdown', 'XXXsplitXXX'
> >>         >  )
> >>         >
> >>         > function Header (elem)
> >>         >   if split_level ~= elem.level then
> >>         >     return nil
> >>         >   else
> >>         >     return { sep, elem }
> >>         >   end
> >>         > end
> >>         > ``````
> >>         >
> >>         > then
> >>         >
> >>         > ``````sh
> >>         > pandoc -L sep.lua my-project.md \
> >>         >  | csplit -f my-project -n 3 --suppress-matched -
> >> /XXXsplitXXX/ {*}
> >>         > ``````
> >>         >
> >>         > Of course the separator may be anything, but it must not
> >> otherwise occur in
> >>         > your document and should better not contain any regular
> >> expression
> >>         > metacharacters; stick to ASCII alphanumerics and you should
> >> be good.
> >>         >
> >>         >
> >>         > Den ons 5 feb. 2020 15:08Szabolcs Horvát <szhorvat@gmail.com
> >> <mailto:szhorvat-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> skrev:
> >>         >
> >>         >> I am working on putting some documentation online, and I am
> >> generating the
> >>         >> final pages with pandoc.
> >>         >>
> >>         >> Currently I have a gigantic markdown file. Does pandoc have
> >> any features
> >>         >> to generate separate HTML pages for each section? I hope to
> >> do this in a
> >>         >> way that links still work between pages, and that the table
> >> of contents is
> >>         >> still usable.
> >>         >>
> >>         >> For reference, this is the current state of the final page
> >> (work in
> >>         >> progress): http://szhorvat.net/mathematica/IGDocumentation/
> >>         >>
> >>         >> --
> >>         >> You received this message because you are subscribed to the
> >> Google Groups
> >>         >> "pandoc-discuss" group.
> >>         >> To unsubscribe from this group and stop receiving emails
> >> from it, send an
> >>         >> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
> >> <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> >>         >> To view this discussion on the web visit
> >>         >>
> >>
> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com
> >>
> >>         >>
> >> <
> https://groups.google.com/d/msgid/pandoc-discuss/ad056ff9-2db6-40ae-97a9-b552c85d99b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >>
> >>         >> .
> >>         >>
> >>         >
> >>         > --
> >>         > You received this message because you are subscribed to the
> >> Google Groups "pandoc-discuss" group.
> >>         > To unsubscribe from this group and stop receiving emails
> >> from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
> >> <mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> >>         > To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuQEpq3qLAJYQA9Tjo9zeJO0NnGS3Q2nzLmxYJK8GaNbPA%40mail.gmail.com.
>
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "pandoc-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
> >> <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> >> To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com
> >> <
> https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhCxAM6VmKdQtpMo%2Bz1k9ZFJoochWGqRFvkZZ4E3MK-Uiw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> >>
> >
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pandoc-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pandoc-discuss/vH_PY66ti1U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/ad1759f3-f4cc-d5d8-34b7-e9e18bffb1a9%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAJDnrnot4Zxm0TdxkZ5ka-mUU0XPSN%3DaMmqbVnQ5dwW9ZJ8Hdg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 17145 bytes --]