Extension for more general documents

Extension for more general documents

[Thomas Klausner]

Hi!

We'd really like to move away the documents in NetBSD's base system
from the groff -ms and -me macro sets to anything else that can be
easily converted to txt, html, and pdf.

We already have mandoc in the base system and it supports these output
formats. So it seems to be a good candidate, and mdoc already provides
the formatting options we need (AFAIK).

We probably would just need to add a mode to mandoc where it doesn't
insist on a man page header and instead supports some article
information like a title.

Would you be willing to add support for this in mandoc?

Am I overlooking stuff that makes this more hard than I currently
imagine it to be?
 Thomas
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Kristaps Dzonsons]

> We'd really like to move away the documents in NetBSD's base system
> from the groff -ms and -me macro sets to anything else that can be
> easily converted to txt, html, and pdf.
>
> We already have mandoc in the base system and it supports these output
> formats. So it seems to be a good candidate, and mdoc already provides
> the formatting options we need (AFAIK).
>
> We probably would just need to add a mode to mandoc where it doesn't
> insist on a man page header and instead supports some article
> information like a title.
>
> Would you be willing to add support for this in mandoc?
>
> Am I overlooking stuff that makes this more hard than I currently
> imagine it to be?

Thomas,

Can you point to an example or two for reference?

Best,

Kristaps

--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Ingo Schwarze]

Hi Thomas,

Kristaps Dzonsons wrote on Fri, Jun 13, 2014 at 10:46:47AM +0200:
> Thomas Klausner wrote:

>> We'd really like to move away the documents in NetBSD's base system
>> from the groff -ms and -me macro sets to anything else that can be
>> easily converted to txt, html, and pdf.
>>
>> We already have mandoc in the base system and it supports these output
>> formats. So it seems to be a good candidate, and mdoc already provides
>> the formatting options we need (AFAIK).
>>
>> We probably would just need to add a mode to mandoc where it doesn't
>> insist on a man page header and instead supports some article
>> information like a title.

Well, -ms and -me are macro sets just like -man and -mdoc,
so it's not just a matter of taking something away, but a matter
of adding something, that is, defining data structures for abstract
syntax trees for these languages and then adding parsers for these
languages and output modules producing ASCII, HTML, and PostScript
output from the syntax trees.

>> Would you be willing to add support for this in mandoc?

Me, right now, no.  That would be too much work for a very small
number of mostly hopelessly outdated documents.

>> Am I overlooking stuff that makes this more hard than I currently
>> imagine it to be?

> Can you point to an example or two for reference?

http://cvsweb.netbsd.org/bsdweb.cgi/src/bin/csh/USD.doc/

  uses -ms,
  specifically AI AB AE AU DE DS EH FE FS I IP KE KF LP ND NH OH PP SH TL UX
  and bp br nf so ta ti

http://cvsweb.netbsd.org/bsdweb.cgi/src/bin/sh/USD.doc/

  uses -ms,
  in addition to the above: RE RS .[ .] ds sp ul

http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.sbin/lpr/SMM.doc/

  uses -ms (not checking in detail)

Also, bc, bib, ed, ex, eqn, fsck, jove, learn, nvi, ratfor, sed, timed,
ufs, uucp had some -ms documentation.

I guess implementing that would be a task of the same order of magnitude
as the complete man(7) implementation, maybe slightly smaller, probably
about 2000-4000 LOC.

http://cvsweb.netbsd.org/bsdweb.cgi/src/games/trek/USD.doc/

  uses -me,
  specifically FF TE TS as bl bp ce de ds eh el ev f fi ft hc ie in
               na ne nf nr oh pp qq sp ti tl tr ul vs wh

Similarly, there was -me documentation for mail, rogue, and maybe more.

That looks rather low-levelish, so it is harder to estimate than -ms
and may be conceptionally more difficult.  It may be the same order
of magnitude of work, it doesn't look easier in any case.

Just as a few examples...

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Thomas Klausner]

Hi Ingo!

On Fri, Jun 13, 2014 at 12:05:50PM +0200, Ingo Schwarze wrote:
> Well, -ms and -me are macro sets just like -man and -mdoc,
> so it's not just a matter of taking something away, but a matter
> of adding something, that is, defining data structures for abstract
> syntax trees for these languages and then adding parsers for these
> languages and output modules producing ASCII, HTML, and PostScript
> output from the syntax trees.

I think it's fine to rewrite these documents to another set of macros,
if it provides the features. I'm not sure yet but my assumption for
now is that mdoc will fit that role nicely, in most cases.

Just to clarify what I'm asking.
 Thomas
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Thomas Klausner]

On Fri, Jun 13, 2014 at 01:06:03PM +0200, Thomas Klausner wrote:
> I think it's fine to rewrite these documents to another set of macros,
> if it provides the features. I'm not sure yet but my assumption for
> now is that mdoc will fit that role nicely, in most cases.
> 
> Just to clarify what I'm asking.

To clarify even further:
A Markdown input mode would also be a great alternative.
 Thomas
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Svyatoslav Mishyn]

just notice
maybe AsciiDoc will be interesting..

(Fri, 13 Jun 13:44) Thomas Klausner:
> On Fri, Jun 13, 2014 at 01:06:03PM +0200, Thomas Klausner wrote:
> > I think it's fine to rewrite these documents to another set of macros,
> > if it provides the features. I'm not sure yet but my assumption for
> > now is that mdoc will fit that role nicely, in most cases.
> > 
> > Just to clarify what I'm asking.
> 
> To clarify even further:
> A Markdown input mode would also be a great alternative.
>  Thomas
> --
>  To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv
> 

-- 
http://www.juef.tk/
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Thomas Klausner]

On Fri, Jun 13, 2014 at 05:04:03PM +0300, Svyatoslav Mishyn wrote:
> just notice
> maybe AsciiDoc will be interesting..

No, because it's in python and we don't want to import python in the
base system. But thanks for the suggestion.
 Thomas
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Ingo Schwarze]

Hi Thomas,

Thomas Klausner wrote on Fri, Jun 13, 2014 at 10:35:36AM +0200:

> We'd really like to move away the documents in NetBSD's base system
> from the groff -ms and -me macro sets to anything else that can be
> easily converted to txt, html, and pdf.
> 
> We already have mandoc in the base system and it supports these output
> formats. So it seems to be a good candidate, and mdoc already provides
> the formatting options we need (AFAIK).
> 
> We probably would just need to add a mode to mandoc where it doesn't
> insist on a man page header and instead supports some article
> information like a title.

Uh, oh.  Indeed i misunderstood this request completely.

So let's look at how the csh document currently looks like,
as an example:

 ----- 8< ----- schnipp ----- >8 ----- 8< ----- schnapp ----- >8 -----

ischwarze@isnote $ groff -ms -Tascii -P-c tabs csh.*

                    An Introduction to the C shell

                             William Joy
                 (revised for 4.3BSD by Mark Seiden)
                      Computer Science Division
      Department of Electrical Engineering and Computer Science
                  University of California, Berkeley
                      Berkeley, California 94720

                               ABSTRACT

               Csh is a new command language interpreter for
          UNIX(R) systems.  It incorporates good features of
                                 [...]

     Introduction

          A shell is a command language interpreter.  Csh is the
     name of one particular command interpreter on UNIX.  The
     [...]
          In addition to this document, you will want to refer to
     a copy of the UNIX User Reference Manual.  The csh


     USD:4-2                       An Introduction to the C shell


     documentation in section 1 of the manual provides a full
     description of all features of the shell and is the
     [...]

 ----- 8< ----- schnipp ----- >8 ----- 8< ----- schnapp ----- >8 -----

I would consider the repeated page headers (USD:4-2  An Introduction
to the C shell) useless, we deliberately didn't implement these in
mandoc(1).  For ABSTRACT (and other headers), .Sh and .Ss could be
used; losing the centering doesn't seem like a grave problem.
So what remains is the title and author.  The title could just be
provided by Nd, and the author information moved to the end into
the AUTHORS section, and no changes to the mdoc(7) language would be
required at all.  For example:

 ----- 8< ----- schnipp ----- >8 ----- 8< ----- schnapp ----- >8 -----

ischwarze@isnote $ cat csh.mdoc

.Dd $Mdocdate$
.Dt CSH-INTRO 1
.Os
.Sh NAME
.Nm CSH-INTRO
.Nd An Introduction to the C shell
.Sh ABSTRACT
.Nm csh
is a new command language interpreter for UNIX systems.
It incorporates good features of other shells and a history
mechanism similar to the
.Sq redo
of INTERLISP.
[...]
.Sh AUTHORS
.An -nosplit
.An William Joy
(revised for 4.3BSD by
.An Mark Seiden )
.br
Computer Science Division,
Department of Electrical Engineering and Computer Science,
University of California, Berkeley
.br
Berkeley, California 94720

schwarze@isnote $ mandoc csh.mdoc

CSH-INTRO(1)               OpenBSD Reference Manual               CSH-INTRO(1)

NAME
     CSH-INTRO - An Introduction to the C shell

ABSTRACT
     csh is a new command language interpreter for UNIX systems.  It
     incorporates good features of other shells and a history mechanism
     similar to the `redo' of INTERLISP.  [...]

AUTHORS
     William Joy (revised for 4.3BSD by Mark Seiden)
     Computer Science Division, Department of Electrical Engineering and
     Computer Science, University of California, Berkeley
     Berkeley, California 94720

OpenBSD 5.5                      June 14, 2014                     OpenBSD 5.5

 ----- 8< ----- schnipp ----- >8 ----- 8< ----- schnapp ----- >8 -----

So that adds the not strictly required header and footer lines,
the word "NAME", and the identifier "CSH-INTRO".  Do these do any
harm?  If you choose to install the document as a man page, they
may even be useful.

Regarding the AUTHORS, you might also move them to the top, right
after NAME, if you are not using any of the standard sections like
SYNOPSIS, DESCRIPTION, and so on.

In case you don't consider this good enough, what exactly do you
think is required, so which changes/options/features would you
consider the minimum that would need to be changed in mandoc
or added to mandoc?  Please try to provide a list that is complete
and as specific as possible, that will make it easier to judge the
effort required (and maybe to provide an implementation that
actually does what you want).

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Ingo Schwarze]

Hi,

Svyatoslav Mishyn wrote on Fri, Jun 13, 2014 at 05:04:03PM +0300:

> just notice
> maybe AsciiDoc will be interesting..

No, it will not.
Never use AsciiDoc or DocBook for anything related to manuals.

The man(7) code generated from the AsciiDoc/DocBook toolchain
is by far the worst quality autogenerated man(7) code you can
find from any autogeneration tool chain i'm aware of.  It is
the worst by such a large margin that DocBook alone is responsible
for the overwhelming majority of bogus bug reports for mandoc, that
is reports that something is presumably broken in mandoc which turn
out to be merely broken mandoc *input* generated from some buggy
autogenerator.

Besides, the AsciiDoc toolchain is incredibly slow.  Generating man(7)
code from AsciiDoc typically takes about hundred (!) times the CPU time
required to format that man(7) code for terminal output with mandoc(1).

If you must autogenerate man(7) code for some reason, even the Perl 
"plain old documentation" (POD) format is a much better starting
point than AsciiDoc.  But i wouldn't recomment that, or generating
man(7) code at all, either.  Just use mdoc(7), Thomas is right in
that respect.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Extension for more general documents

[Ingo Schwarze]

Hi David,

David Holland wrote on Sat, Jul 05, 2014 at 08:50:08PM +0000:
> On Sat, Jun 14, 2014 at 02:16:00PM +0200, Ingo Schwarze wrote:
>> Thomas Klausner wrote on Fri, Jun 13, 2014 at 10:35:36AM +0200:

>>> We'd really like to move away the documents in NetBSD's base system
>>> from the groff -ms and -me macro sets to anything else that can be
>>> easily converted to txt, html, and pdf.
>>> 
>>> We already have mandoc in the base system and it supports these output
>>> formats. So it seems to be a good candidate, and mdoc already provides
>>> the formatting options we need (AFAIK).
>>> 
>>> We probably would just need to add a mode to mandoc where it doesn't
>>> insist on a man page header and instead supports some article
>>> information like a title.

>> what exactly do you think is required, so which
>> changes/options/features would you consider the minimum that would
>> need to be changed in mandoc or added to mandoc?

> A way to produce an article title and author block that looks like a
> title and author block. I think that's all.

Two changes in the upcoming mandoc-1.13.1 release might help with
this project:

 1) mandoc-1.13.1 no longer discards content preceding the first
    .Sh section header, as mandoc-1.12.3 did.

 2) mandoc-1.13.1 has partial support for the
    .Bd -centered
    display.  While that display ought to be set in fill mode,
    mandoc uses no-fill mode for now.  For title & author blocks,
    that's no problem at all.

So, you are welcome to test the document appended below with
mandoc-1.13.1 - i think it looks reasonable both in -Tascii
and in -Tpdf mode.  Note that mandoc doesn't need the .br
requests right now, but you should put them for compatibility,
or other implementations might re-wrap the lines.

Admittedly, this is not completely portable.  With groff, the
output looks less nice than with mandoc.

> It is not clear at the moment what other typesetting features these
> articles might need - many use tbl (but that's not a problem, right?),

Usually, not a problem.  Now and then, some obscure tbl(7) features
crop up and we need to add them to the mandoc(1) implementation,
and sometimes tbl(7) isn't exactly top priority.  But usually such
stuff is not impossible to get supported.

> and at least one each uses eqn

If that's only used for trivial formulas, mandoc(1) might be good
enough.  If there are substantial mathematical equations,
probably not.  Basically, mandoc only has a terminal engine
for eqn(1), even when writing PS/PDF.  While the parser is quite
good, it's just impossible to adequately render equations in
ASCII text output.

> and pic.

No way.

> Some have bibliography and/or references,

That can probably be converted to .Rs, unless it requires some
kind of database data source with autogeneration or it is of
massive size (like many dozens or even hundeds of references).

> and also bottom-of-page footnotes.

That won't work in mandoc, and i wouldn't know where to start
supporting it.

> Also, at least one
> contains a context-free grammar that currently comes out pretty
> ratty-looking. One also uses little marker tags in the left margin,
> but that can probably be edited out.

No idea regarding those two, that would need inspection of the actual
documents.  Margin tags certainly doesn't sound mdocish.

> At least one article contains floating figures; does mandoc handle
> those? I'm guessing not.

No way.  No figures.  No floats.

> There's also one that wants to have block
> headers on subsections, which I think is difficult in -mdoc.

Block headers?  Not quite sure what that is...

> Several documents also have indexes, most of which currently appear to
> be broken. It would be nice to repair/extend these but it isn't really
> a top priority.

The mdoc(7) language currently has no markup for generating an index.

Well, there is the non-standard ms(7) .IX macro, a variant of which
appears as a home-grown user-defined macro in pod2man(1) man(7)
output.  The main problem with all this is to come up with a
reasonable design what indexing in manual pages is even supposed
to do.  I mean, a manual page has no concept of page numbers.
Where do you want to point?  To line numbers?  To section and
subsection headers?  To .Bl -tag entries?  This all seems far
from obvious to me...

Yours,
  Ingo


.Dd July 28, 2014
.Dt DHOLLAND-EXAMPLE 1
.Os
.Bd -centered
Building and Debugging NetBSD Kernels
.br
David A. Holland
.br
August 17, 2009
.Ed
.Sh NAME
.Nm dholland-example
.Nd example of a title and author block for dholland
.Sh DESCRIPTION
some text
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv