[RFC PATCH] Makefile: work around parallel make issues in docs

[RFC PATCH] Makefile: work around parallel make issues in docs

[tmz]

When make is run with multiple jobs, doc-man and doc-html fail.  The a2x
command tries to write %.5.xml for each invocation, overwriting each
other.

Work around this by copying %.5 to %.5+ in doc-man.  This is a rather
gross hack, but nothing better came to mind.  Using --asciidoc-opts to
pass the '--out-file' did not affect the temporary .xml file at issue.

Signed-off-by: Todd Zullinger <tmz at pobox.com>
---
I ran into this in the Fedora/EPEL package builds long ago and set -j1 for the
doc build at the time. I finally got around to looking a little deeper.

I'm not happy with this patch, but as I said above, nothing better came to
mind.  I'm hoping that submitting this will spur some discussion about a better
way to handle this.  Though maybe the best way to handle it will be "don't
build the docs in parallel, you're not saving any time for the single file that
is built." :)

That's effectively what I've done for ages, a separate 'make -j1 doc-man
doc-html' invocation.

Reproduction should be simple: make -j2 doc-man doc-html

Each a2x process tries to create cgitrc.5.xml in the process of generating the
man and html files, clobbering each other and causing xmllint to fail.

This is still broken when calling 'make -j2 doc' as I only added the rename
work-around to doc-man, so if doc-man, doc-html, and doc-pdf are all run, the
two without the work-around will cause the same issue.

I consider this more of a demonstration of the issue than a fix.  I hope that
by sending it, someone will respond with the obvious, elegant, and correct
solution.  Or maybe moments after I send it, that solution will come to me.
Things often work out that way. :)

Thanks,

Todd

 Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 687069f..da36872 100644
--- a/Makefile
+++ b/Makefile
@@ -131,7 +131,9 @@ doc-html: $(DOC_HTML)
 doc-pdf: $(DOC_PDF)
 
 %.5 : %.5.txt
-	a2x -f manpage $<
+	cp -a $@.txt $@+.txt && \
+	a2x -f manpage $@+.txt && \
+	rm -f $@+.txt
 
 $(DOC_HTML): %.html : %.txt
 	a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $<

[RFC PATCH] Makefile: work around parallel make issues in docs

[thomas.moschny]

Todd Zullinger <tmz at pobox.com>:

> When make is run with multiple jobs, doc-man and doc-html fail.  The
> a2x command tries to write %.5.xml for each invocation, overwriting
> each other.
> 
> Work around this by copying %.5 to %.5+ in doc-man.  This is a rather
> gross hack, but nothing better came to mind.  Using --asciidoc-opts to
> pass the '--out-file' did not affect the temporary .xml file at issue.

What about using a --destination-dir (-D) to a2x? It puts the
temporary .xml also into that dir. Something like this:

diff --git a/Makefile b/Makefile
index 687069f..2ccee34 100644
--- a/Makefile
+++ b/Makefile
@@ -131,7 +131,10 @@ doc-html: $(DOC_HTML)
 doc-pdf: $(DOC_PDF)
 
 %.5 : %.5.txt
-       a2x -f manpage $<
+       mkdir $@.dir
+       a2x -f manpage -D $@.dir $<
+       mv $@.dir/$@ .
+       rmdir $@.dir
 
 $(DOC_HTML): %.html : %.txt
        a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $<


One might want to additionally clean up the directory somewhere (as it
is not removed in case something goes wrong with the a2x call).

- Thomas

[RFC PATCH] Makefile: work around parallel make issues in docs

[tmz]

Thomas Moschny wrote:
> Todd Zullinger <tmz at pobox.com>:
> 
>> When make is run with multiple jobs, doc-man and doc-html fail.  The
>> a2x command tries to write %.5.xml for each invocation, overwriting
>> each other.
>> 
>> Work around this by copying %.5 to %.5+ in doc-man.  This is a rather
>> gross hack, but nothing better came to mind.  Using --asciidoc-opts to
>> pass the '--out-file' did not affect the temporary .xml file at issue.
> 
> What about using a --destination-dir (-D) to a2x? It puts the
> temporary .xml also into that dir. Something like this:
> 
> diff --git a/Makefile b/Makefile
> index 687069f..2ccee34 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -131,7 +131,10 @@ doc-html: $(DOC_HTML)
>  doc-pdf: $(DOC_PDF)
>  
>  %.5 : %.5.txt
> -       a2x -f manpage $<
> +       mkdir $@.dir
> +       a2x -f manpage -D $@.dir $<
> +       mv $@.dir/$@ .
> +       rmdir $@.dir
>  
>  $(DOC_HTML): %.html : %.txt
>         a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $<
> 
> 
> One might want to additionally clean up the directory somewhere (as it
> is not removed in case something goes wrong with the a2x call).

Yeah, that's a nice solution too.  Thanks!

It's really a2x's fault for stripping the extension and
replacing it with .xml rather than just appending .xml to
the target name.  (But perhaps someone on Windows doesn't
want to see a temp file named .html.xml or something.)

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A cynic is a man who, when he smells flowers, looks around for a
coffin.
    -- H. L. Mencken

[RFC PATCH] Makefile: work around parallel make issues in docs

[john]

On Sat, Jun 16, 2018 at 01:38:31AM -0400, Todd Zullinger wrote:
> When make is run with multiple jobs, doc-man and doc-html fail.  The a2x
> command tries to write %.5.xml for each invocation, overwriting each
> other.
> 
> Work around this by copying %.5 to %.5+ in doc-man.  This is a rather
> gross hack, but nothing better came to mind.  Using --asciidoc-opts to
> pass the '--out-file' did not affect the temporary .xml file at issue.
> 
> Signed-off-by: Todd Zullinger <tmz at pobox.com>
> ---
> I ran into this in the Fedora/EPEL package builds long ago and set -j1 for the
> doc build at the time. I finally got around to looking a little deeper.
> 
> I'm not happy with this patch, but as I said above, nothing better came to
> mind.  I'm hoping that submitting this will spur some discussion about a better
> way to handle this.  Though maybe the best way to handle it will be "don't
> build the docs in parallel, you're not saving any time for the single file that
> is built." :)
> 
> That's effectively what I've done for ages, a separate 'make -j1 doc-man
> doc-html' invocation.
> 
> Reproduction should be simple: make -j2 doc-man doc-html
> 
> Each a2x process tries to create cgitrc.5.xml in the process of generating the
> man and html files, clobbering each other and causing xmllint to fail.
> 
> This is still broken when calling 'make -j2 doc' as I only added the rename
> work-around to doc-man, so if doc-man, doc-html, and doc-pdf are all run, the
> two without the work-around will cause the same issue.
> 
> I consider this more of a demonstration of the issue than a fix.  I hope that
> by sending it, someone will respond with the obvious, elegant, and correct
> solution.  Or maybe moments after I send it, that solution will come to me.
> Things often work out that way. :)

How about the patch below instead?  It's a bigger change to the output
format for HTML, but as a side effect it fixes the parallel build.

I considered removing a2x for the manpage output as well, but Asciidoc
can't generate that directly so we'd have to build the documentation
pipeline ourselves.  Git does that, but I don't think it's worth the
effort here because we don't need to customise the process.

-- >8 --
Subject: [PATCH] Makefile: drive asciidoc directly for HTML output

This is mostly taken from Git's doc/Makefile, although simplified for
our use.  The output now uses Asciidoc's default CSS which I think looks
a bit nicer than the Docbook formatting; as a result of this we no
longer need our custom .css file.

A side effect of this change is that temporary files generated from the
HTML output no longer conflict with the manpage output format (because
any temporary HTML output files use names derived from the output
filename which includes .html).
---
 Makefile     | 9 ++++++++-
 cgit-doc.css | 3 ---
 2 files changed, 8 insertions(+), 4 deletions(-)
 delete mode 100644 cgit-doc.css

diff --git a/Makefile b/Makefile
index 687069f..70f32a4 100644
--- a/Makefile
+++ b/Makefile
@@ -24,6 +24,12 @@ DOC_MAN5 = $(patsubst %.txt,%,$(MAN5_TXT))
 DOC_HTML = $(patsubst %.txt,%.html,$(MAN_TXT))
 DOC_PDF  = $(patsubst %.txt,%.pdf,$(MAN_TXT))
 
+ASCIIDOC = asciidoc
+ASCIIDOC_EXTRA =
+ASCIIDOC_HTML = xhtml11
+ASCIIDOC_COMMON = $(ASCIIDOC) $(ASCIIDOC_EXTRA)
+TXT_TO_HTML = $(ASCIIDOC_COMMON) -b $(ASCIIDOC_HTML)
+
 # Define NO_C99_FORMAT if your formatted IO functions (printf/scanf et.al.)
 # do not support the 'size specifiers' introduced by C99, namely ll, hh,
 # j, z, t. (representing long long int, char, intmax_t, size_t, ptrdiff_t).
@@ -134,7 +140,8 @@ doc-pdf: $(DOC_PDF)
 	a2x -f manpage $<
 
 $(DOC_HTML): %.html : %.txt
-	a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $<
+	$(TXT_TO_HTML) -o $@+ $< && \
+	mv $@+ $@
 
 $(DOC_PDF): %.pdf : %.txt
 	a2x -f pdf cgitrc.5.txt
diff --git a/cgit-doc.css b/cgit-doc.css
deleted file mode 100644
index 5a399b6..0000000
--- a/cgit-doc.css
+++ /dev/null
@@ -1,3 +0,0 @@
-div.variablelist dt {
-	margin-top: 1em;
-}
-- 
2.17.1

[RFC PATCH] Makefile: work around parallel make issues in docs

[Jason]

Hey John,

That seems like a reasonable approach. Feel free to queue it up in jk/for-jason.

Jason

[RFC PATCH] Makefile: work around parallel make issues in docs

[tmz]

Hi John,

John Keeping wrote:
> How about the patch below instead?  It's a bigger change to the output
> format for HTML, but as a side effect it fixes the parallel build.

It does, but only if the targets are 'doc-man doc-html'.
For the default 'doc' target which includes 'doc-pdf', the
issue is still present.

As a bonus, the output is much nicer, I think.

We could perhaps work around the 'doc-pdf' failure by adding a
dep on '($DOC_MAN5)':

-- >8 --
diff --git i/Makefile w/Makefile
index 70f32a4..4879a5a 100644
--- i/Makefile
+++ w/Makefile
@@ -143,7 +143,7 @@ $(DOC_HTML): %.html : %.txt
        $(TXT_TO_HTML) -o $@+ $< && \
        mv $@+ $@
 
-$(DOC_PDF): %.pdf : %.txt
+$(DOC_PDF): %.pdf : $(DOC_MAN5) %.txt
        a2x -f pdf cgitrc.5.txt
 
 clean: clean-doc
-- >8 --

That's hackish, no doubt.

We might also want to drop 'doc-pdf' from the default 'doc'
target.  The alternative is driving the asciidoc pipeline
for the pdf generation too.  That looks a little more
involved than doing it for html, but perhaps it's not as bad
as I think.

At first I was concerned that this loses the xsltproc option
to generate consistent id's in the html, but the html
generated by asciidoc this way doesn't appear to suffer from
the problem solved by that xsltproc option. :)

Thanks,

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Common sense is the collection of prejudices acquired by age eighteen.
    -- Albert Einstein

[RFC PATCH] Makefile: work around parallel make issues in docs

[john]

Hi Todd,

On Sat, Jun 16, 2018 at 12:32:06PM -0400, Todd Zullinger wrote:
> John Keeping wrote:
> > How about the patch below instead?  It's a bigger change to the output
> > format for HTML, but as a side effect it fixes the parallel build.
> 
> It does, but only if the targets are 'doc-man doc-html'.
> For the default 'doc' target which includes 'doc-pdf', the
> issue is still present.
> 
> As a bonus, the output is much nicer, I think.
> 
> We could perhaps work around the 'doc-pdf' failure by adding a
> dep on '($DOC_MAN5)':
> 
> -- >8 --
> diff --git i/Makefile w/Makefile
> index 70f32a4..4879a5a 100644
> --- i/Makefile
> +++ w/Makefile
> @@ -143,7 +143,7 @@ $(DOC_HTML): %.html : %.txt
>         $(TXT_TO_HTML) -o $@+ $< && \
>         mv $@+ $@
>  
> -$(DOC_PDF): %.pdf : %.txt
> +$(DOC_PDF): %.pdf : $(DOC_MAN5) %.txt
>         a2x -f pdf cgitrc.5.txt
>  
>  clean: clean-doc
> -- >8 --
> 
> That's hackish, no doubt.
> 
> We might also want to drop 'doc-pdf' from the default 'doc'
> target.  The alternative is driving the asciidoc pipeline
> for the pdf generation too.  That looks a little more
> involved than doing it for html, but perhaps it's not as bad
> as I think.

I think we can definitely drop doc-pdf from the default output.

I'm half tempted to say we should just delete the PDF output completely
and see if anyone complains, unless you know of anyone using this?

Otherwise, the dependency on $(DOC_MAN5) seems reasonable to me,
probably accompanied with a comment explaining the clash in a2x's
intermediate files.


Regards,
John

[RFC PATCH] Makefile: work around parallel make issues in docs

[tmz]

Hi,

John Keeping wrote:
> On Sat, Jun 16, 2018 at 12:32:06PM -0400, Todd Zullinger wrote:
>> We might also want to drop 'doc-pdf' from the default 'doc'
>> target.  The alternative is driving the asciidoc pipeline
>> for the pdf generation too.  That looks a little more
>> involved than doing it for html, but perhaps it's not as bad
>> as I think.
> 
> I think we can definitely drop doc-pdf from the default output.
> 
> I'm half tempted to say we should just delete the PDF output completely
> and see if anyone complains, unless you know of anyone using this?

We don't generate the pdf docs for the Fedora/EPEL cgit
builds and no one has yet filed a bug asking for it.  Poking
around, none of Arch, Debian, Gentoo, and Ubuntu build the
pdf docs (in fact, they don't include the html target
either).

Maybe that's a good sign that the pdf target wouldn't be
missed by many people.

> Otherwise, the dependency on $(DOC_MAN5) seems reasonable
> to me, probably accompanied with a comment explaining the
> clash in a2x's intermediate files.

Cool.  I'll wait a bit and see if anyone chimes in with
support for keeping the pdf target.  If not, dropping it is
the easier option.

Thanks,

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I dropped acid on a Saturday night Just to see what the fuss was
about.  Now there goes the neighborhood.