Re: Linking in mdoc(7)

[Steffen Nurpmeso <sdaoden@yandex.com>]

[-- Attachment #1: Type: text/plain, Size: 4835 bytes --]

Hello Kristaps, Jörg and Ingo,

Ingo Schwarze <schwarze@usta.de> wrote:
 |Steffen Nurpmeso wrote on Mon, Mar 09, 2015 at 12:09:08PM +0100:
 |> Kristaps Dzonsons <kristaps@bsd.lv> wrote:
 |>> Summary: mdoc(7) has linking, but it's very restrictive.  You can link

 |Then again, we have a tradition of having good designs override
 |bad ones soon afterwards (no insult intended :).
 |
 |> I have proposed something similar back in September 2014 on groff@
 |
 |Exactly what i wanted to say.  :-)
 |
 |The thread is here:
 |
 |  http://lists.gnu.org/archive/html/groff/2014-09/msg00145.html
 |
 |> and implemented a complete package that can do this and much more
 |> with an interface that follows Ingo's suggestions since then.
 |> The macro is .Mx, the documentation, troff(1) awk(1) preprocessor,
 |> mdoc(1) macro extensions etc., and a patch for less(1) v471 that
 |> allows internal (including .Sh and .Ss, but many more) and
 |> external anchors (for .Xr) to be followed are available.  Use it
 |> or leave it.

 |Even though Steffen rightfully says that i influenced the design
 |he finally chose, and even though that design seemed better to me
 |than what was initially proposed, and even though Steffen was
 |positively enthusiastic about it, i wasn't quite satisfied yet.
 |
 |So my hope is that what we finally arrive at will at least be as
 |powerful as the union of your two implementations and at most as
 |complex as the intersection of both of them, or maybe even better...

That would be my hope, too -- moving the current state forward.
And maybe my final words were a bit too strong too, honestly
at least they were.  :-)

I'm still enthusiastic since it is a real improvement to work with
these more interactive manuals.  And i surely would like to know
any criticism you might have in order to be able to address it,
shall that be possible from the troff(1) side of the road.
Please let me attach the latest version of mdocmx.7 (from January
19th) to make sure we talk about the same state of affairs.

I want to point out that this is a mature and usable system,
regarding the entire mdocmx(1) (awk(1) preprocessor) / mdoc(7)
/ grotty(1) / less(1) pipeline.
Let me elaborate a bit on what i personally miss from that
onwards:

(1) Better performance: the last commit i made to mdocmx(7)
(2015-02-05) was a massive performance improvement (excerpt):

  Before:
        ?0[]$ time (mdocmx.sh nail.1|
        groff -Tutf8 -mdoc -dmx-toc-force=1 -dmx-debug=1 >/dev/null)
            0m1.25s real     0m1.59s user     0m0.06s system
        ?0[]$ time (export MDOCMX_ENABLE=1; mdocmx.sh nail.1|
        groff -Tutf8 -mdoc -dmx-toc-force=1 -dmx-debug=1 >MAN.before)
            0m7.36s real     0m7.73s user     0m0.08s system

  After:
        ?0[]$ time (mdocmx.sh nail.1|
        groff -Tutf8 -mdoc -dmx-toc-force=1 -dmx-debug=1 >/dev/null)
            0m1.19s real     0m1.59s user     0m0.06s system
        ?0[]$ time (export MDOCMX_ENABLE=1; mdocmx.sh nail.1|
        groff -Tutf8 -mdoc -dmx-toc-force=1 -dmx-debug=1 >MAN.old)
            0m2.56s real     0m2.97s user     0m0.07s system

  After, with an experimental .hash troff(1) builtin:
        ?0[]$ time (export MDOCMX_ENABLE=1; mdocmx.sh nail.1|
        groff -Tutf8 -mdoc -dmx-toc-force=1 -dmx-debug=1 >MAN.new)
            0m2.19s real     0m2.61s user     0m0.07s system

  For comparison, here is plain mdocml(1):
        ?0[]$ time (mdocmx.sh nail.1|mandoc > /dev/null)
            0m0.26s real     0m0.19s user     0m0.07s system

This is extreme but nail.1 results in ~4500 lines of rendered
text, the difference for NetBSD's bus_dma.9 for example is much
much smaller.

Shall mdocml(1) go that route i wouldn't even expect a measurable
impact, the problem with troff(1) is that we cannot have hashmaps
or trees from within macros.  (And of course the way mdoc(7) is
implemented is a problem; that can be improved, i would know how,
which would then also allow referenced HTML and PDF etc. output.)

(2) An index facility is missing.  I still find myself using
less(1)'s search command in order to get myself going in manuals.
It would be better if beside automatic TOC creation there would be
automatic index creation -- this is possible with mdocmx(7) and it
will happen: like this you scroll to e.g. the end of the manual to
the index and then can jump directly to the desired anchor.

(3) Due to the way mdoc(7) is implemented the reference anchors
can only be suffixes, not prefixes as is known from lynx(1).
I think it wouldn't bother anyone if mdocml(1) would start
generating anchors for less(1) as prefixes right from the start,
over time i will adjust the mdoc(7) macros (or at least plan to do
so) so they can follow.

--steffen

[-- Attachment #2: Original message content --]
[-- Type: message/rfc822, Size: 5528 bytes --]

From: Ingo Schwarze <schwarze@usta.de>
To: discuss@mdocml.bsd.lv
Subject: Re: Linking in mdoc(7)
Date: Mon, 9 Mar 2015 12:46:48 +0100
Message-ID: <20150309114648.GA18431@athene.usta.de>

Hi Kristaps and Steffen,

Steffen Nurpmeso wrote on Mon, Mar 09, 2015 at 12:09:08PM +0100:
> Kristaps Dzonsons <kristaps@bsd.lv> wrote:

>> Summary: mdoc(7) has linking, but it's very restrictive.  You can link
>> to `Sh' and `Ss' macros by using the `Sx' macro.  I propose adding
>> another macro, `Ix', that creates an invisible anchor that `Sx' can
>> jump to.  Thus, `Sh' and `Ss' work as they always did and `Sx' works
>> as it always did, but we can also jump to arbitrary locations from
>> `Sx' as defined with `Ix', which is otherwise silent.  Obviously, this
>> is only useful in -Thtml.  On the console, nothing happens at all.  I
>> also propose an `Lkx', which is like `Lk' but for inter-page links.
>> This is when the argument to `Sx' should be different from the index name.

My initial reaction is:  It is about time to do something like that.
However, introducing new macros is not be taken lightly, and the
design must not be rushed.  Before committing to something specific,
we should make sure that it is as powerful as possible, as extensible
as possible, and as simple as possible.  I'm not yet convinced the
proposed design is the most general and simple one.  In particular,
it isn't obvious to me yet how implicit linking - i.e. internal
linking that doesn't require any new markup - integrates into
Kristaps concept, and lack of an idea how terminal output can profit
is a very serious downside too, give that terminal output is by far
the most important output mode.

Then again, we have a tradition of having good designs override
bad ones soon afterwards (no insult intended :).

> I have proposed something similar back in September 2014 on groff@

Exactly what i wanted to say.  :-)

The thread is here:

  http://lists.gnu.org/archive/html/groff/2014-09/msg00145.html

> and implemented a complete package that can do this and much more
> with an interface that follows Ingo's suggestions since then.
> The macro is .Mx, the documentation, troff(1) awk(1) preprocessor,
> mdoc(1) macro extensions etc., and a patch for less(1) v471 that
> allows internal (including .Sh and .Ss, but many more) and
> external anchors (for .Xr) to be followed are available.  Use it
> or leave it.

I didn't look at either implementation yet, i focussed on different
topics (man(1), eqn(7), UTF-8, messages, afl mopup, pod2mdoc(1)).

Even though Steffen rightfully says that i influenced the design
he finally chose, and even though that design seemed better to me
than what was initially proposed, and even though Steffen was
positively enthusiastic about it, i wasn't quite satisfied yet.

So my hope is that what we finally arrive at will at least be as
powerful as the union of your two implementations and at most as
complex as the intersection of both of them, or maybe even better...

Kristaps, if you want to commit to minimize risk of loss of work,
maybe create a branch for it for now?  I'm planning to prepare
the 1.13.3 release soon, and this is clearly post-release material.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv



[-- Attachment #3: mdocmx.7 --]
[-- Type: text/plain, Size: 11476 bytes --]

'\" m -- preprocess: mdocmx(1)
.\"@ mdocmx.7 - mdocmx(7) reference manual.
.\"@ mdocmx(7) extends the mdoc(7) semantic markup language by references,
.\"@ allowing mdoc(7) to create anchors and table of contents.
.\"
.\" Written 2014 - 2015 by Steffen (Daode) Nurpmeso <sdaoden@users.sf.net>.
.\" Public Domain
.
.Dd January 19, 2015
.Dt MDOCMX 7
.Os
.Mx -enable
.
.
.Sh NAME
.
.Nm .Mx
.Nd Reference extension for the mdoc semantic markup language
.
.
.Sh SYNOPSIS
.
.Nm
.Fl enable
.Op Ar output-formats
.Pp
.Nm
.Fl disable
.Pp
.Nm
.Nm
.Ar macro
.Nm
.Ar macro Ar key
.Pp
.Bk
.Nm
.Fl toc
.Op Ar output-formats
.Op Fl tree Ar output-formats
.Ek
.
.
.Mx -toc -tree html xhtml pdf ps
.
.
.Sh DESCRIPTION
.
.Xr mdocmx 7
introduces referenceable index anchors to the
.Xr mdoc 7
semantic markup language that is used for
.Ux
manual pages by defining a single new multiplexer command:
.Nm .
A precondition for enabling this extension is to set the environment
variable
.Ev MDOCMX_ENABLE
to a non-empty value (the value is necessary to make the variable
perceivable on the macro level, i.e., to overcome a deficiency of the
.Xr troff 1
command set).
.
.Pp
The
.Xr mdocmx 7
reference extension augments the standard
.Xr mdoc 7
document prologue \(en
.Ic \&Dd ,
.Ic \&Dt
and
.Ic \&Os
\(en with the new command
.Nm
.Fl enable .
It can be restricted to specific output formats by adding those
.Xr troff 1
output devices for which expansion is desired as further arguments to
.Fl enable ;
for convenience all typewriter-like devices can be addressed via
.Ar tty .
It is not an error to specify a device for which no special
.Xr mdocmx 7
support is available, but such requests are simply ignored.
.
.Pp
Because macros driven by single-pass troff implementations cannot
create forward references
.Xr mdoc 7
documents which use this extension need to be preprocessed with the
.Xr mdocmx 1
preprocessor, which is a regular part of
.Xr mdocmx 7
and implemented in portable
.Xr sh 1
and
.Xr awk 1 .
Specialized manual formatters and macros driven by multi-pass troff
interpreters may not require a preprocessor to support this extension.
It is also possible to preprocess the manual once and distribute the
resulting document \(en refer to the
.Sx COMPATIBILITY
section for more on that.
.
.Pp
Sometimes it may be desirable to actively suppress any processing
of the
.Xr mdocmx 7
reference extension: this can either be accomplished by using
.Nm
with the
.Fl disable
argument or by defining the string
.Ql Dv mx-disable ,
as in
.
.Pp
.Dl MDOCMX_ENABLE=anyval; export MDOCMX_ENABLE
.Dl < xy.z mdocmx.sh | troff -Tutf8 -mdoc -a -dmx-disable=1
.
.Ss Creating referenceable anchors
.
After the extension was
.Fl enable Ns
d in the document prologue the third group of
.Nm
usage forms can be used to enqueue index anchor requests.
These requests form a stack which will be consumed (popped) by the
later occurrence of a (corresponding)
.Xr mdoc 7
.Ar macro
which supports referenceable index entries.
The indices are managed with distinct namespaces for each supported
.Ar macro ,
meaning that, e.g.,
.Ql .Mx \&Ic sendmail
and
.Ql .Mx \&Va sendmail
will create distinct index anchors.
.
.Pp
Using the plain macro
.Nm
without arguments creates a stack entry for which both,
the name of the
.Ar macro
as well as the
.Ar key
will be taken from the document content.
.Nm
.Ar macro
will create a stack entry that will be consumed by the next occurrence of
.Ar macro
only, then taking the
.Ar key
off the document content, whereas
.Nm
.Ar macro Ar key
creates a stack entry that also has its
.Ar key
defined, and which will be consumed once an exactly matching macro / key
pair is seen in the document only.
(The
.Sx EXAMPLES
section gives a use case for this form.)
.
.Pp
Using the
.Xr mdocmx 1
preprocessor will also create referenceable anchors for the
.Xr mdoc 7
section header commands
.Ic .Sh
and
.Ic .Ss
automatically, so that a
.Xr mdoc 7
macro package which supports the
.Xr mdocmx 7
extension will be enabled to actually create references with the
.Ic .Sx
command, and, dependent on the output device, cross-references defined
via the command
.Ic .Xr
will also be backed with functionality.
The following macros gain support for referenceable anchors via
.Nm :
.
.Pp
.Bl -tag -compact -width ".It Ic ._B"
.Mx
.It Ic .Ar
Command argument.
.Mx
.It Ic .Cm
Command modifier.
.Mx
.It Ic .Dv
Defined variable or preprocessor constant.
.Mx
.It Ic .Er
Error constant.
.Mx
.It Ic .Ev
Environment variable.
.Mx
.It Ic .Fl
Command line option (flag).
.Mx
.It Ic .Fn
Function name.
.Mx
.It Ic .Fo
Function name (in function block syntax).
This is mapped to
.Ic .Fn ,
.Ic .Fo
has no index by itself.
.Mx
.It Ic .Ic
Internal or interactive command.
.Mx
.It Ic .In
An
.Ql include
file for, e.g., the C programming language.
.Mx
.It Ic .Pa
File system path.
.Mx
.It Ic .Va
Variable name.
.Mx
.It Ic .Vt
Variable type, type reference.
.El
.
.Ss Creating table of contents
.
The final usage form of the
.Xr mdocmx 7
reference extension allows the creation of a document table of content,
which is of special interest when converting a
.Xr mdoc 7
document into formats such as HTML, XHTML or PDF.
To restrict the creation of the table of contents to special output
formats, add the names of those
.Xr troff 1
output devices for which expansion is desired as further arguments to
.Fl toc ;
for convenience all typewriter-like devices can be addressed via
.Ar tty .
.
.Pp
By default only
.Ic .Sh
section headers are a vivid part of the TOC; in order to include
.Ic .Ss
subsections also add a
.Fl tree
argument.
Note that if
.Fl tree
is used in conjunction with output-device restrictions it will only
affect those devices that appear later on the line.
.
.Pp
In the first of the following examples a table of content will be
generated for PDF and typewriter-like devices.
In the second example a tree of contents will instead be generated for
the output formats PDF and HTML, whereas typewriter-like devices will
see a flat table of contents with only section headers.
.
.Pp
.Dl .Mx -toc pdf tty
.Dl .Mx -toc tty -tree html pdf
.
.Ss Strings that affect mdocmx
.
.Sy Note
that due to deficiencies in some implementations of
.Xr troff 1
strings given on the command line (via option
.Fl d Ns )
have to be given an argument in order to be perceived on the macro
level; see the example given above in the section
.Sx DESCRIPTION .
.
.Bl -tag -width ".It Dv _docmx-_oc-_merged"
.Mx
.It Dv mx-debug
If defined
.Xr mdocmx 7
macros will offer some verbosity.
In addition not only references will produce visual output, but also
anchors.
.
.Mx
.It Dv mx-disable
Has the same effect as
.Ql .Mx -disable .
.
.Mx
.It Dv mx-toc-emerged
Normally
.Fl compact
display is used for the table of contents, but when this string is set
an emerged display is used for the first level that lists the headings.
.
.Mx
.It Dv mx-toc-force
Can be used to enforce creation of a table of content even if the
documents
.Fl toc
configuration wouldn't create one for the targeted output device.
.
.Mx
.It Dv mx-toc-name
If defined its content is used as the headline of the table of contents,
which can be used for, e.g., localization purposes.
The default is
.Dq TABLE OF CONTENTS .
.
.Mx
.It Dv mx-toc-numbered
If defined the first level of the table of contents will be numbered.
.El
.
.
.Sh IMPLEMENTATION NOTES
.
The
.Nm
request cannot share a line with other macros, neither in the document
prologue nor in its content.
Whereas that is mostly owed to the necessity of ensuring (backward)
compatibility with environments that don't support
.Xr mdocmx 7 ,
it also simplified implementation of the preprocessor.
.
.Ss Internal extended synopsis
.
In addition to those usage forms that have been described above the
.Ic .Mx
multiplexer command also understands further flags and arguments which
are of possible interest for formatter and macro implementors.
These further flags and arguments are only generated by the
.Xr mdocmx 1
preprocessor and are solely ment to communicate the preprocessed state
of the document to the actual consumers.
.
.Pp
For one a
.Fl preprocessed
flag is appended to the single
.Fl enable
command in the document prologue.
And then an additional
.Fl anchor-spass
form is introduced, which takes two or three arguments \(en
the macro (name of the command) for which this defines an anchor as well
as its key, possibly followed by a numeric argument that describes the
relationship in between section headings: for
.Ic .Sh
commands it defines a running one-based index count of section headers,
for
.Ic .Ss
commands it instead specifies the index of the section header they
belong to, therefore creating the possibility to generate TOCs.
.
.
.Sh ENVIRONMENT
.
Only if the environment variable
.Mx
.Ev MDOCMX_ENABLE
is set to a non-empty value will the
.Xr mdocmx 7
macros generate the necessary information that the chosen output
device of
.Xr troff 1
can, sufficient support provided, use to generate table of contents,
internal and external references.
.
.
.Sh EXAMPLES
.
A complete, but completely fanciful
.Xr mdoc 7
document that uses the
.Xr mdocmx 7
extension would for example be:
.
.Bd -literal -offset indent
\&.Dd January 19, 2015
\&.Dt MDOCMX-EXAMPLE 7
\&.Os
\&.Mx -enable tty
\&.
\&.Sh NAME
\&.Nm mdocmx-example
\&.Nd An example for the mdocmx mdoc reference extension
\&.Mx -toc
\&.
\&.Sh DESCRIPTION
Sors salutis et virtutis michi nunc contraria.
\&.
\&.Bl -tag -width ".It Fn _a_e_i_"
\&.Mx
\&.It Ic .Ar
This will create an anchor for a macro `Ic', key `.Ar'.
\&.Mx
\&.It Ic .Cm
Anchor for `Ic', key `.Cm'.
\&.Mx
\&.It Ic .Dv
And an anchor for `Ic', key `.Dv'.
\&.Mx Ic
\&.Mx Ic "final anchor"
\&.Mx Fn _atexit
\&.It Fn exit
No anchor here.
\&.It Fn at_quick_exit , Fn _atexit
Not for the first, but for the second `Fn' there will be an
anchor with the key `_atexit'.
\&.It Ic "no anchor here"
\&.It Ic "final anchor"
Pops the pushed `Ic' / `final anchor' macro / key pair.
\&.It Ic ciao
Pops the `Ic' and assigns the key `Ciao'.
\&.El
.Ed
.
.
.Sh COMPATIBILITY
.
Using the
.Xr mdocmx 7
extension in
.Xr mdoc 7
manual pages should not cause any compatibility problems in sofar as
all tested environments silently ignore the unknown commands by default.
Because of this, and due to the nature of this extension, an
interesting, backward as well as forward compatible approach to use
.Xr mdocmx 7
may be to preprocess manuals with
.Xr mdocmx 1
on developer machines and instead distribute the resulting documents.
.
.
.Sh SEE ALSO
.
.Xr awk 1 ,
.Xr mandoc 1 ,
.Xr mdocmx 1 ,
.Xr sh 1 ,
.Xr troff 1 ,
.Xr mdoc 7
.
.
.Sh HISTORY
.
The
.Nm
environment appeared in 2014.
.
.
.Sh AUTHORS
.
Idea and first implementation by
.An Steffen Nurpmeso Aq Mt sdaoden@users.sf.net .
.An Ingo Schwarze Aq Mt schwarze@openbsd.org
designed the command semantics.
.
.
.Sh CAVEATS
.
Be aware that the content of the
.Fl width
argument to
.Xr mdoc 7
lists etc. is evaluated as if it were normal document content; e.g., in
the following example the
.Ql \&Fn _atexit
will be evaluated and may thus get used by
.Nm :
.
.Pp
.Dl ".Bl -tag -width "".It Fn _atexit"""
.
.Pp
When developing a manual it may be helpful to increase verbosity of the
.Xr mdocmx 1
preprocessor on its standard error I/O channel by using the
.Fl v
command line flag in order to get a notion on what is going on:
.
.Bd -literal -offset indent
$ mdocmx -vv -T Sh < mdocmx.7 2> stderr.txt | \e
  groff -Tutf8 -mdoc | less
$ cat stderr.txt
.Ed
.
.\" s-ts-mode