document delimiter handling in mdoc(7)

document delimiter handling in mdoc(7)

[Ingo Schwarze]

Hi,

some time ago, i did not merge Kristaps' change of "reserved characters"
to "reserved terms" to mdoc.7 in OpenBSD because "reserved characters"
is already too generic as a term in this context, and "reserved terms"
is even worse.  A better term is "delimiters".  It is already used a lot
in the code.

Right before release is a good time to polish manuals, so i had
another look - and noticed more issues:

The text explains how to escape these characters "for general use",
but fails to explain what they are escaped from, i.e. in which way
they receive special handling in the first place.

The whole subsection is located below "LANGUAGE SYNTAX", but it's
only concerned with macro arguments.  So it should be moved after
explaining macro syntax, and into the "MACRO SYNTAX" section.
As a nice side effect, this very complicated subject is no longer
one of the first subjects covered in the manual.

When reading the patch, you will notice that it is not yet perfect.
It explains the special handling of delimiters, but is very
imprecise as to which macros exactly all this applies to.
There are two reasons why i didn't try to address that yet:
The patch is already big, and figuring out the exact lists of
macros showing the various behaviours is a lot of work - yes,
there are many exceptions.

Note that i removed the lie that delimiters terminate any macros.
In block partial-implicit, only *trailing* *closing* delimiters
terminate the macro, so that's right before the macro scope ends
anyway, and that's already covered by the new "Delimiters" subsection.
For in-line macros, the situation is quite complicated:
Some are terminated by trailing closing delimiters just like
block partial-implicit macros, some are only *interrupted*
by delimiters and resume afterwards, and in_line_eoln macros
don't do any special delimiter handling at all.

The following patch is against OpenBSD; in case you like it,
i will of course resolve the (physical and logical) conflicts
and commit to bsd.lv as well.

Yours,
  Ingo


Index: mdoc.7
===================================================================
RCS file: /cvs/src/share/man/man7/mdoc.7,v
retrieving revision 1.75
diff -u -r1.75 mdoc.7
--- mdoc.7	31 Jul 2011 17:12:29 -0000	1.75
+++ mdoc.7	31 Jul 2011 22:06:02 -0000
@@ -65,43 +65,6 @@
 is also ignored.
 Macro lines with only a control character and optional whitespace are
 stripped from input.
-.Ss Reserved Characters
-Within a macro line, the following characters are reserved:
-.Pp
-.Bl -tag -width Ds -offset indent -compact
-.It \&.
-.Pq period
-.It \e.
-.Pq escaped period
-.It \&,
-.Pq comma
-.It \&:
-.Pq colon
-.It \&;
-.Pq semicolon
-.It \&(
-.Pq left-parenthesis
-.It \&)
-.Pq right-parenthesis
-.It \&[
-.Pq left-bracket
-.It \&]
-.Pq right-bracket
-.It \&?
-.Pq question
-.It \&!
-.Pq exclamation
-.It \&|
-.Pq vertical bar
-.It \e*(Ba
-.Pq reserved-word vertical bar
-.El
-.Pp
-For general use in macro lines, these characters can either be escaped
-with a non-breaking space
-.Pq Sq \e&
-or, if applicable, an appropriate escape sequence can be used.
-In text lines, these may be used as normal punctuation.
 .Ss Special Characters
 Special characters may occur in both macro and text lines.
 Sequences begin with the escape character
@@ -729,9 +692,8 @@
 .It Sx \&Xo  Ta    Yes      Ta    Yes      Ta    closed by Sx \&Xc
 .El
 .Ss Block partial-implicit
-Like block full-implicit, but with single-line scope closed by
-.Sx Reserved Characters
-or end of line.
+Like block full-implicit, but with single-line scope closed by the
+end of the line.
 .Bd -literal -offset indent
 \&.Yo \(lB\-arg \(lBval...\(rB\(rB \(lBbody...\(rB \(lBres...\(rB
 .Ed
@@ -777,9 +739,8 @@
 .It Sx \&Ta  Ta    Yes      Ta    Yes    Ta closed by Sx \&Ta , Sx \&It
 .El
 .Ss In-line
-Closed by
-.Sx Reserved Characters ,
-end of line, fixed argument lengths, and/or subsequent macros.
+Closed by the end of the line, fixed argument lengths,
+and/or subsequent macros.
 In-line macros have only text children.
 If a number (or inequality) of arguments is
 .Pq n ,
@@ -869,6 +830,90 @@
 .It Sx \&br  Ta    \&No     Ta    \&No     Ta    0
 .It Sx \&sp  Ta    \&No     Ta    \&No     Ta    1
 .El
+.Ss Delimiters
+When a macro argument consists of one single input character
+considered as a delimiter, the argument gets special handling.
+This does not apply when delimiters appear in arguments containing
+more than one character.
+Consequently, to prevent special handling and just handle it
+like any other argument, a delimiter can be escaped by prepending
+a non-breaking space
+.Pq Sq \e& .
+In text lines, delimiters never need escaping, but may be used
+as normal punctuation.
+.Pp
+For many macros, when the leading arguments are opening delimiters,
+these delimiters are put before the macro scope,
+and when the trailing arguments are closing delimiters,
+these delimiters are put after the macro scope.
+For example,
+.Pp
+.D1 Pf \. \&Aq "( [ word ] ) ."
+.Pp
+renders as:
+.Pp
+.D1 Aq ( [ word ] ) .
+.Pp
+Opening delimiters are:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&(
+.Pq left-parenthesis
+.It \&[
+.Pq left-bracket
+.El
+.Pp
+Closing delimiters are:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&.
+.Pq period
+.It \&,
+.Pq comma
+.It \&:
+.Pq colon
+.It \&;
+.Pq semicolon
+.It \&)
+.Pq right-parenthesis
+.It \&]
+.Pq right-bracket
+.It \&?
+.Pq question
+.It \&!
+.Pq exclamation
+.El
+.Pp
+Note that even a period preceded by a backslash
+.Pq Sq \e.\&
+gets this special handling; use
+.Sq \e&.
+to prevent that.
+.Pp
+Many in-line macros interrupt their scope when they encouter
+delimiters, and resume their scope when more arguments follow that
+are not delimiters.
+For example,
+.Pp
+.D1 Pf \. \&Fl "a ( b | c \e*(Ba d ) e"
+.Pp
+renders as:
+.Pp
+.D1 Fl a ( b | c \*(Ba d ) e
+.Pp
+This applies to both opening and closing delimiters,
+and also to the middle delimiter:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&|
+.Pq vertical bar
+.El
+.Pp
+As a special case, the predefined string \e*(Ba is handled and rendered
+in the same way as a plain
+.Sq \&|
+character.
+Using this predefined string is not recommended in new manuals.
 .Sh REFERENCE
 This section is a canonical reference of all macros, arranged
 alphabetically.
@@ -2883,7 +2928,7 @@
 lists will restart the sequence only for the sub-list.
 .It
 .Sx \&Li
-followed by a reserved character is incorrectly used in some manuals
+followed by a delimiter is incorrectly used in some manuals
 instead of properly quoting that character, which sometimes works with
 historic groff.
 .It
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: document delimiter handling in mdoc(7)

[Ingo Schwarze]

Hi Jason,

Jason McIntyre wrote on Mon, Aug 01, 2011 at 08:25:17AM +0100:

> - (just checking) you call \& a "non-breaking space", right?

Good catch, that's plainly wrong.
It must be "zero-width space".
Non-breaking would be "\~".
Fixed in two places.

> - your list of delimeters should probably not use (). for example:
> 
> 	.	(period)
> would read better as
> 	.	period

Yes, i removed the .Pq.
While here, i also removed a few hyphens
and added "mark" to "question" and "exclamation".

> - you have a typo: encouter

Fixed, and updated patch appended.

> - shouldn;t the middle delimeter list (|) be listed with the other
>   delimiters?

In a way, that would be nice, but i can't think of a good way to do
it without causing other issues:

 * The sentence about \. should come right after the closing
   delimiters, so there is at least one small paragraph in
   between, anyway.

 * Unlike opening and closing delimiters, middle delimiters do
   not fall out of the beginning and end of macros (except at
   the end after closing delimiters, which is not very useful
   and so weird that i'm not planning to document it).
   Putting them together with the others might cause confusion in
   this respect.

 * The only aspect special about middle delimiters is interrupting
   .Fl-style in-line macros, so i would like to put them close
   to the text explaining that.

 * Having the ugly \*(Ba special case at the very end is quite
   nice, i wouldn't want that ugliness somewhere in the middle.
   However, it should come right after introducing the bar.

So, if you have a suggestion for a better ordering, i'm open to
that, but i can't think of any right now...

Yours,
  Ingo


Index: mdoc.7
===================================================================
RCS file: /cvs/src/share/man/man7/mdoc.7,v
retrieving revision 1.76
diff -u -r1.76 mdoc.7
--- mdoc.7	1 Aug 2011 07:45:08 -0000	1.76
+++ mdoc.7	1 Aug 2011 08:07:14 -0000
@@ -65,43 +65,6 @@
 is also ignored.
 Macro lines with only a control character and optional whitespace are
 stripped from input.
-.Ss Reserved Characters
-Within a macro line, the following characters are reserved:
-.Pp
-.Bl -tag -width Ds -offset indent -compact
-.It \&.
-.Pq period
-.It \e.
-.Pq escaped period
-.It \&,
-.Pq comma
-.It \&:
-.Pq colon
-.It \&;
-.Pq semicolon
-.It \&(
-.Pq left-parenthesis
-.It \&)
-.Pq right-parenthesis
-.It \&[
-.Pq left-bracket
-.It \&]
-.Pq right-bracket
-.It \&?
-.Pq question
-.It \&!
-.Pq exclamation
-.It \&|
-.Pq vertical bar
-.It \e*(Ba
-.Pq reserved-word vertical bar
-.El
-.Pp
-For general use in macro lines, these characters can either be escaped
-with a non-breaking space
-.Pq Sq \e&
-or, if applicable, an appropriate escape sequence can be used.
-In text lines, these may be used as normal punctuation.
 .Ss Special Characters
 Special characters may occur in both macro and text lines.
 Sequences begin with the escape character
@@ -629,7 +592,7 @@
 produces
 .Sq Op Fl O Ar file .
 To prevent a macro call and render the macro name literally,
-escape it by prepending a non-breaking space,
+escape it by prepending a zero-width space,
 .Sq \e& .
 For example,
 .Sq \&Op \e&Fl O
@@ -763,9 +726,8 @@
 .It Sx \&Xo  Ta    Yes      Ta    Yes      Ta    closed by Sx \&Xc
 .El
 .Ss Block partial-implicit
-Like block full-implicit, but with single-line scope closed by
-.Sx Reserved Characters
-or end of line.
+Like block full-implicit, but with single-line scope closed by the
+end of the line.
 .Bd -literal -offset indent
 \&.Yo \(lB\-arg \(lBval...\(rB\(rB \(lBbody...\(rB \(lBres...\(rB
 .Ed
@@ -811,9 +773,8 @@
 .It Sx \&Ta  Ta    Yes      Ta    Yes    Ta closed by Sx \&Ta , Sx \&It
 .El
 .Ss In-line
-Closed by
-.Sx Reserved Characters ,
-end of line, fixed argument lengths, and/or subsequent macros.
+Closed by the end of the line, fixed argument lengths,
+and/or subsequent macros.
 In-line macros have only text children.
 If a number (or inequality) of arguments is
 .Pq n ,
@@ -903,6 +864,90 @@
 .It Sx \&br  Ta    \&No     Ta    \&No     Ta    0
 .It Sx \&sp  Ta    \&No     Ta    \&No     Ta    1
 .El
+.Ss Delimiters
+When a macro argument consists of one single input character
+considered as a delimiter, the argument gets special handling.
+This does not apply when delimiters appear in arguments containing
+more than one character.
+Consequently, to prevent special handling and just handle it
+like any other argument, a delimiter can be escaped by prepending
+a zero-width space
+.Pq Sq \e& .
+In text lines, delimiters never need escaping, but may be used
+as normal punctuation.
+.Pp
+For many macros, when the leading arguments are opening delimiters,
+these delimiters are put before the macro scope,
+and when the trailing arguments are closing delimiters,
+these delimiters are put after the macro scope.
+For example,
+.Pp
+.D1 Pf \. \&Aq "( [ word ] ) ."
+.Pp
+renders as:
+.Pp
+.D1 Aq ( [ word ] ) .
+.Pp
+Opening delimiters are:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&(
+left parenthesis
+.It \&[
+left bracket
+.El
+.Pp
+Closing delimiters are:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&.
+period
+.It \&,
+comma
+.It \&:
+colon
+.It \&;
+semicolon
+.It \&)
+right parenthesis
+.It \&]
+right bracket
+.It \&?
+question mark
+.It \&!
+exclamation mark
+.El
+.Pp
+Note that even a period preceded by a backslash
+.Pq Sq \e.\&
+gets this special handling; use
+.Sq \e&.
+to prevent that.
+.Pp
+Many in-line macros interrupt their scope when they encounter
+delimiters, and resume their scope when more arguments follow that
+are not delimiters.
+For example,
+.Pp
+.D1 Pf \. \&Fl "a ( b | c \e*(Ba d ) e"
+.Pp
+renders as:
+.Pp
+.D1 Fl a ( b | c \*(Ba d ) e
+.Pp
+This applies to both opening and closing delimiters,
+and also to the middle delimiter:
+.Pp
+.Bl -tag -width Ds -offset indent -compact
+.It \&|
+vertical bar
+.El
+.Pp
+As a special case, the predefined string \e*(Ba is handled and rendered
+in the same way as a plain
+.Sq \&|
+character.
+Using this predefined string is not recommended in new manuals.
 .Sh REFERENCE
 This section is a canonical reference of all macros, arranged
 alphabetically.
@@ -2917,7 +2962,7 @@
 lists will restart the sequence only for the sub-list.
 .It
 .Sx \&Li
-followed by a reserved character is incorrectly used in some manuals
+followed by a delimiter is incorrectly used in some manuals
 instead of properly quoting that character, which sometimes works with
 historic groff.
 .It
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv