* [RFC PATCH] Makefile: work around parallel make issues in docs @ 2018-06-16 5:38 tmz 2018-06-16 6:55 ` thomas.moschny 2018-06-16 12:20 ` john 0 siblings, 2 replies; 8+ messages in thread From: tmz @ 2018-06-16 5:38 UTC (permalink / raw) When make is run with multiple jobs, doc-man and doc-html fail. The a2x command tries to write %.5.xml for each invocation, overwriting each other. Work around this by copying %.5 to %.5+ in doc-man. This is a rather gross hack, but nothing better came to mind. Using --asciidoc-opts to pass the '--out-file' did not affect the temporary .xml file at issue. Signed-off-by: Todd Zullinger <tmz at pobox.com> --- I ran into this in the Fedora/EPEL package builds long ago and set -j1 for the doc build at the time. I finally got around to looking a little deeper. I'm not happy with this patch, but as I said above, nothing better came to mind. I'm hoping that submitting this will spur some discussion about a better way to handle this. Though maybe the best way to handle it will be "don't build the docs in parallel, you're not saving any time for the single file that is built." :) That's effectively what I've done for ages, a separate 'make -j1 doc-man doc-html' invocation. Reproduction should be simple: make -j2 doc-man doc-html Each a2x process tries to create cgitrc.5.xml in the process of generating the man and html files, clobbering each other and causing xmllint to fail. This is still broken when calling 'make -j2 doc' as I only added the rename work-around to doc-man, so if doc-man, doc-html, and doc-pdf are all run, the two without the work-around will cause the same issue. I consider this more of a demonstration of the issue than a fix. I hope that by sending it, someone will respond with the obvious, elegant, and correct solution. Or maybe moments after I send it, that solution will come to me. Things often work out that way. :) Thanks, Todd Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 687069f..da36872 100644 --- a/Makefile +++ b/Makefile @@ -131,7 +131,9 @@ doc-html: $(DOC_HTML) doc-pdf: $(DOC_PDF) %.5 : %.5.txt - a2x -f manpage $< + cp -a $@.txt $@+.txt && \ + a2x -f manpage $@+.txt && \ + rm -f $@+.txt $(DOC_HTML): %.html : %.txt a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $< ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 5:38 [RFC PATCH] Makefile: work around parallel make issues in docs tmz @ 2018-06-16 6:55 ` thomas.moschny 2018-06-16 16:40 ` tmz 2018-06-16 12:20 ` john 1 sibling, 1 reply; 8+ messages in thread From: thomas.moschny @ 2018-06-16 6:55 UTC (permalink / raw) Todd Zullinger <tmz at pobox.com>: > When make is run with multiple jobs, doc-man and doc-html fail. The > a2x command tries to write %.5.xml for each invocation, overwriting > each other. > > Work around this by copying %.5 to %.5+ in doc-man. This is a rather > gross hack, but nothing better came to mind. Using --asciidoc-opts to > pass the '--out-file' did not affect the temporary .xml file at issue. What about using a --destination-dir (-D) to a2x? It puts the temporary .xml also into that dir. Something like this: diff --git a/Makefile b/Makefile index 687069f..2ccee34 100644 --- a/Makefile +++ b/Makefile @@ -131,7 +131,10 @@ doc-html: $(DOC_HTML) doc-pdf: $(DOC_PDF) %.5 : %.5.txt - a2x -f manpage $< + mkdir $@.dir + a2x -f manpage -D $@.dir $< + mv $@.dir/$@ . + rmdir $@.dir $(DOC_HTML): %.html : %.txt a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $< One might want to additionally clean up the directory somewhere (as it is not removed in case something goes wrong with the a2x call). - Thomas ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 6:55 ` thomas.moschny @ 2018-06-16 16:40 ` tmz 0 siblings, 0 replies; 8+ messages in thread From: tmz @ 2018-06-16 16:40 UTC (permalink / raw) Thomas Moschny wrote: > Todd Zullinger <tmz at pobox.com>: > >> When make is run with multiple jobs, doc-man and doc-html fail. The >> a2x command tries to write %.5.xml for each invocation, overwriting >> each other. >> >> Work around this by copying %.5 to %.5+ in doc-man. This is a rather >> gross hack, but nothing better came to mind. Using --asciidoc-opts to >> pass the '--out-file' did not affect the temporary .xml file at issue. > > What about using a --destination-dir (-D) to a2x? It puts the > temporary .xml also into that dir. Something like this: > > diff --git a/Makefile b/Makefile > index 687069f..2ccee34 100644 > --- a/Makefile > +++ b/Makefile > @@ -131,7 +131,10 @@ doc-html: $(DOC_HTML) > doc-pdf: $(DOC_PDF) > > %.5 : %.5.txt > - a2x -f manpage $< > + mkdir $@.dir > + a2x -f manpage -D $@.dir $< > + mv $@.dir/$@ . > + rmdir $@.dir > > $(DOC_HTML): %.html : %.txt > a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $< > > > One might want to additionally clean up the directory somewhere (as it > is not removed in case something goes wrong with the a2x call). Yeah, that's a nice solution too. Thanks! It's really a2x's fault for stripping the extension and replacing it with .xml rather than just appending .xml to the target name. (But perhaps someone on Windows doesn't want to see a temp file named .html.xml or something.) -- Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A cynic is a man who, when he smells flowers, looks around for a coffin. -- H. L. Mencken ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 5:38 [RFC PATCH] Makefile: work around parallel make issues in docs tmz 2018-06-16 6:55 ` thomas.moschny @ 2018-06-16 12:20 ` john 2018-06-16 12:50 ` Jason 2018-06-16 16:32 ` tmz 1 sibling, 2 replies; 8+ messages in thread From: john @ 2018-06-16 12:20 UTC (permalink / raw) On Sat, Jun 16, 2018 at 01:38:31AM -0400, Todd Zullinger wrote: > When make is run with multiple jobs, doc-man and doc-html fail. The a2x > command tries to write %.5.xml for each invocation, overwriting each > other. > > Work around this by copying %.5 to %.5+ in doc-man. This is a rather > gross hack, but nothing better came to mind. Using --asciidoc-opts to > pass the '--out-file' did not affect the temporary .xml file at issue. > > Signed-off-by: Todd Zullinger <tmz at pobox.com> > --- > I ran into this in the Fedora/EPEL package builds long ago and set -j1 for the > doc build at the time. I finally got around to looking a little deeper. > > I'm not happy with this patch, but as I said above, nothing better came to > mind. I'm hoping that submitting this will spur some discussion about a better > way to handle this. Though maybe the best way to handle it will be "don't > build the docs in parallel, you're not saving any time for the single file that > is built." :) > > That's effectively what I've done for ages, a separate 'make -j1 doc-man > doc-html' invocation. > > Reproduction should be simple: make -j2 doc-man doc-html > > Each a2x process tries to create cgitrc.5.xml in the process of generating the > man and html files, clobbering each other and causing xmllint to fail. > > This is still broken when calling 'make -j2 doc' as I only added the rename > work-around to doc-man, so if doc-man, doc-html, and doc-pdf are all run, the > two without the work-around will cause the same issue. > > I consider this more of a demonstration of the issue than a fix. I hope that > by sending it, someone will respond with the obvious, elegant, and correct > solution. Or maybe moments after I send it, that solution will come to me. > Things often work out that way. :) How about the patch below instead? It's a bigger change to the output format for HTML, but as a side effect it fixes the parallel build. I considered removing a2x for the manpage output as well, but Asciidoc can't generate that directly so we'd have to build the documentation pipeline ourselves. Git does that, but I don't think it's worth the effort here because we don't need to customise the process. -- >8 -- Subject: [PATCH] Makefile: drive asciidoc directly for HTML output This is mostly taken from Git's doc/Makefile, although simplified for our use. The output now uses Asciidoc's default CSS which I think looks a bit nicer than the Docbook formatting; as a result of this we no longer need our custom .css file. A side effect of this change is that temporary files generated from the HTML output no longer conflict with the manpage output format (because any temporary HTML output files use names derived from the output filename which includes .html). --- Makefile | 9 ++++++++- cgit-doc.css | 3 --- 2 files changed, 8 insertions(+), 4 deletions(-) delete mode 100644 cgit-doc.css diff --git a/Makefile b/Makefile index 687069f..70f32a4 100644 --- a/Makefile +++ b/Makefile @@ -24,6 +24,12 @@ DOC_MAN5 = $(patsubst %.txt,%,$(MAN5_TXT)) DOC_HTML = $(patsubst %.txt,%.html,$(MAN_TXT)) DOC_PDF = $(patsubst %.txt,%.pdf,$(MAN_TXT)) +ASCIIDOC = asciidoc +ASCIIDOC_EXTRA = +ASCIIDOC_HTML = xhtml11 +ASCIIDOC_COMMON = $(ASCIIDOC) $(ASCIIDOC_EXTRA) +TXT_TO_HTML = $(ASCIIDOC_COMMON) -b $(ASCIIDOC_HTML) + # Define NO_C99_FORMAT if your formatted IO functions (printf/scanf et.al.) # do not support the 'size specifiers' introduced by C99, namely ll, hh, # j, z, t. (representing long long int, char, intmax_t, size_t, ptrdiff_t). @@ -134,7 +140,8 @@ doc-pdf: $(DOC_PDF) a2x -f manpage $< $(DOC_HTML): %.html : %.txt - a2x -f xhtml --stylesheet=cgit-doc.css --xsltproc-opts="--param generate.consistent.ids 1" $< + $(TXT_TO_HTML) -o $@+ $< && \ + mv $@+ $@ $(DOC_PDF): %.pdf : %.txt a2x -f pdf cgitrc.5.txt diff --git a/cgit-doc.css b/cgit-doc.css deleted file mode 100644 index 5a399b6..0000000 --- a/cgit-doc.css +++ /dev/null @@ -1,3 +0,0 @@ -div.variablelist dt { - margin-top: 1em; -} -- 2.17.1 ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 12:20 ` john @ 2018-06-16 12:50 ` Jason 2018-06-16 16:32 ` tmz 1 sibling, 0 replies; 8+ messages in thread From: Jason @ 2018-06-16 12:50 UTC (permalink / raw) Hey John, That seems like a reasonable approach. Feel free to queue it up in jk/for-jason. Jason ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 12:20 ` john 2018-06-16 12:50 ` Jason @ 2018-06-16 16:32 ` tmz 2018-06-16 17:18 ` john 1 sibling, 1 reply; 8+ messages in thread From: tmz @ 2018-06-16 16:32 UTC (permalink / raw) Hi John, John Keeping wrote: > How about the patch below instead? It's a bigger change to the output > format for HTML, but as a side effect it fixes the parallel build. It does, but only if the targets are 'doc-man doc-html'. For the default 'doc' target which includes 'doc-pdf', the issue is still present. As a bonus, the output is much nicer, I think. We could perhaps work around the 'doc-pdf' failure by adding a dep on '($DOC_MAN5)': -- >8 -- diff --git i/Makefile w/Makefile index 70f32a4..4879a5a 100644 --- i/Makefile +++ w/Makefile @@ -143,7 +143,7 @@ $(DOC_HTML): %.html : %.txt $(TXT_TO_HTML) -o $@+ $< && \ mv $@+ $@ -$(DOC_PDF): %.pdf : %.txt +$(DOC_PDF): %.pdf : $(DOC_MAN5) %.txt a2x -f pdf cgitrc.5.txt clean: clean-doc -- >8 -- That's hackish, no doubt. We might also want to drop 'doc-pdf' from the default 'doc' target. The alternative is driving the asciidoc pipeline for the pdf generation too. That looks a little more involved than doing it for html, but perhaps it's not as bad as I think. At first I was concerned that this loses the xsltproc option to generate consistent id's in the html, but the html generated by asciidoc this way doesn't appear to suffer from the problem solved by that xsltproc option. :) Thanks, -- Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Common sense is the collection of prejudices acquired by age eighteen. -- Albert Einstein ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 16:32 ` tmz @ 2018-06-16 17:18 ` john 2018-06-16 17:52 ` tmz 0 siblings, 1 reply; 8+ messages in thread From: john @ 2018-06-16 17:18 UTC (permalink / raw) Hi Todd, On Sat, Jun 16, 2018 at 12:32:06PM -0400, Todd Zullinger wrote: > John Keeping wrote: > > How about the patch below instead? It's a bigger change to the output > > format for HTML, but as a side effect it fixes the parallel build. > > It does, but only if the targets are 'doc-man doc-html'. > For the default 'doc' target which includes 'doc-pdf', the > issue is still present. > > As a bonus, the output is much nicer, I think. > > We could perhaps work around the 'doc-pdf' failure by adding a > dep on '($DOC_MAN5)': > > -- >8 -- > diff --git i/Makefile w/Makefile > index 70f32a4..4879a5a 100644 > --- i/Makefile > +++ w/Makefile > @@ -143,7 +143,7 @@ $(DOC_HTML): %.html : %.txt > $(TXT_TO_HTML) -o $@+ $< && \ > mv $@+ $@ > > -$(DOC_PDF): %.pdf : %.txt > +$(DOC_PDF): %.pdf : $(DOC_MAN5) %.txt > a2x -f pdf cgitrc.5.txt > > clean: clean-doc > -- >8 -- > > That's hackish, no doubt. > > We might also want to drop 'doc-pdf' from the default 'doc' > target. The alternative is driving the asciidoc pipeline > for the pdf generation too. That looks a little more > involved than doing it for html, but perhaps it's not as bad > as I think. I think we can definitely drop doc-pdf from the default output. I'm half tempted to say we should just delete the PDF output completely and see if anyone complains, unless you know of anyone using this? Otherwise, the dependency on $(DOC_MAN5) seems reasonable to me, probably accompanied with a comment explaining the clash in a2x's intermediate files. Regards, John ^ permalink raw reply [flat|nested] 8+ messages in thread
* [RFC PATCH] Makefile: work around parallel make issues in docs 2018-06-16 17:18 ` john @ 2018-06-16 17:52 ` tmz 0 siblings, 0 replies; 8+ messages in thread From: tmz @ 2018-06-16 17:52 UTC (permalink / raw) Hi, John Keeping wrote: > On Sat, Jun 16, 2018 at 12:32:06PM -0400, Todd Zullinger wrote: >> We might also want to drop 'doc-pdf' from the default 'doc' >> target. The alternative is driving the asciidoc pipeline >> for the pdf generation too. That looks a little more >> involved than doing it for html, but perhaps it's not as bad >> as I think. > > I think we can definitely drop doc-pdf from the default output. > > I'm half tempted to say we should just delete the PDF output completely > and see if anyone complains, unless you know of anyone using this? We don't generate the pdf docs for the Fedora/EPEL cgit builds and no one has yet filed a bug asking for it. Poking around, none of Arch, Debian, Gentoo, and Ubuntu build the pdf docs (in fact, they don't include the html target either). Maybe that's a good sign that the pdf target wouldn't be missed by many people. > Otherwise, the dependency on $(DOC_MAN5) seems reasonable > to me, probably accompanied with a comment explaining the > clash in a2x's intermediate files. Cool. I'll wait a bit and see if anyone chimes in with support for keeping the pdf target. If not, dropping it is the easier option. Thanks, -- Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I dropped acid on a Saturday night Just to see what the fuss was about. Now there goes the neighborhood. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-06-16 17:52 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-16 5:38 [RFC PATCH] Makefile: work around parallel make issues in docs tmz 2018-06-16 6:55 ` thomas.moschny 2018-06-16 16:40 ` tmz 2018-06-16 12:20 ` john 2018-06-16 12:50 ` Jason 2018-06-16 16:32 ` tmz 2018-06-16 17:18 ` john 2018-06-16 17:52 ` tmz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).