move roff sections to a separate file

move roff sections to a separate file

[Yuri Pankov]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

I'd like a propose a small change which would simplify maintaining a 
downstream version for us (as we need it modified and with a bit changed 
order) - move list of sections (roff_sec) and their descriptions 
(secnames) to separate file which we keep without a need to sync with 
upstream, roffsec.in. Patch is pretty straightforward and simply moves 
both there.

[-- Attachment #2: roffsec.diff --]
[-- Type: text/plain, Size: 3542 bytes --]

commit 369032da926596cc6cec2a5615fe3afd9f6083db
Author: Yuri Pankov <yuri.pankov@nexenta.com>
Date:   Tue May 30 01:02:14 2017 +0300

    move section list to separate file

diff --git a/mdoc_validate.c b/mdoc_validate.c
index 1d3d81e..ede6863 100644
--- a/mdoc_validate.c
+++ b/mdoc_validate.c
@@ -251,33 +251,6 @@ static	const enum roff_tok rsord[RSORD_MAX] = {
 	MDOC__O
 };
 
-static	const char * const secnames[SEC__MAX] = {
-	NULL,
-	"NAME",
-	"LIBRARY",
-	"SYNOPSIS",
-	"DESCRIPTION",
-	"CONTEXT",
-	"IMPLEMENTATION NOTES",
-	"RETURN VALUES",
-	"ENVIRONMENT",
-	"FILES",
-	"EXIT STATUS",
-	"EXAMPLES",
-	"DIAGNOSTICS",
-	"COMPATIBILITY",
-	"ERRORS",
-	"SEE ALSO",
-	"STANDARDS",
-	"HISTORY",
-	"AUTHORS",
-	"CAVEATS",
-	"BUGS",
-	"SECURITY CONSIDERATIONS",
-	NULL
-};
-
-
 void
 mdoc_node_validate(struct roff_man *mdoc)
 {
diff --git a/roff.h b/roff.h
index f0c2bf4..8f90faa 100644
--- a/roff.h
+++ b/roff.h
@@ -16,6 +16,8 @@
  * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  */
 
+#include "roffsec.in"
+
 struct	ohash;
 struct	mdoc_arg;
 union	mdoc_data;
@@ -26,33 +28,6 @@ enum	roff_macroset {
 	MACROSET_MAN
 };
 
-enum	roff_sec {
-	SEC_NONE = 0,
-	SEC_NAME,
-	SEC_LIBRARY,
-	SEC_SYNOPSIS,
-	SEC_DESCRIPTION,
-	SEC_CONTEXT,
-	SEC_IMPLEMENTATION,	/* IMPLEMENTATION NOTES */
-	SEC_RETURN_VALUES,
-	SEC_ENVIRONMENT,
-	SEC_FILES,
-	SEC_EXIT_STATUS,
-	SEC_EXAMPLES,
-	SEC_DIAGNOSTICS,
-	SEC_COMPATIBILITY,
-	SEC_ERRORS,
-	SEC_SEE_ALSO,
-	SEC_STANDARDS,
-	SEC_HISTORY,
-	SEC_AUTHORS,
-	SEC_CAVEATS,
-	SEC_BUGS,
-	SEC_SECURITY,
-	SEC_CUSTOM,
-	SEC__MAX
-};
-
 enum	roff_type {
 	ROFFT_ROOT,
 	ROFFT_BLOCK,
diff --git a/roffsec.in b/roffsec.in
new file mode 100644
index 0000000..6fd4172
--- /dev/null
+++ b/roffsec.in
@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2008-2012 Kristaps Dzonsons <kristaps@bsd.lv>
+ * Copyright (c) 2010-2017 Ingo Schwarze <schwarze@openbsd.org>
+ * Copyright (c) 2010 Joerg Sonnenberger <joerg@netbsd.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHORS DISCLAIM ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+enum    roff_sec {
+	SEC_NONE = 0,
+	SEC_NAME,
+	SEC_LIBRARY,
+	SEC_SYNOPSIS,
+	SEC_DESCRIPTION,
+	SEC_CONTEXT,
+	SEC_IMPLEMENTATION,     /* IMPLEMENTATION NOTES */
+	SEC_RETURN_VALUES,
+	SEC_ENVIRONMENT,
+	SEC_FILES,
+	SEC_EXIT_STATUS,
+	SEC_EXAMPLES,
+	SEC_DIAGNOSTICS,
+	SEC_COMPATIBILITY,
+	SEC_ERRORS,
+	SEC_SEE_ALSO,
+	SEC_STANDARDS,
+	SEC_HISTORY,
+	SEC_AUTHORS,
+	SEC_CAVEATS,
+	SEC_BUGS,
+	SEC_SECURITY,
+	SEC_CUSTOM,
+	SEC__MAX
+};
+
+static const char * const secnames[SEC__MAX] = {
+	NULL,
+	"NAME",
+	"LIBRARY",
+	"SYNOPSIS",
+	"DESCRIPTION",
+	"CONTEXT",
+	"IMPLEMENTATION NOTES",
+	"RETURN VALUES",
+	"ENVIRONMENT",
+	"FILES",
+	"EXIT STATUS",
+	"EXAMPLES",
+	"DIAGNOSTICS",
+	"COMPATIBILITY",
+	"ERRORS",
+	"SEE ALSO",
+	"STANDARDS",
+	"HISTORY",
+	"AUTHORS",
+	"CAVEATS",
+	"BUGS",
+	"SECURITY CONSIDERATIONS",
+	NULL
+};

Re: move roff sections to a separate file

[Ingo Schwarze]

Hi Yuri,

Yuri Pankov wrote on Tue, May 30, 2017 at 01:14:35AM +0300:

> I'd like a propose a small change which would simplify maintaining a
> downstream version for us (as we need it modified and with a bit changed
> order) - move list of sections (roff_sec) and their descriptions (secnames)
> to separate file which we keep without a need to sync with upstream,
> roffsec.in. Patch is pretty straightforward and simply moves both there.

I see why such small differences can be a bother in maintenance,
but i don't like the direction this is going.

 1. I'd like to encourage more consistency in manual page organization
    across different systems, not facilitate divergence.  Section
    naming and ordering is an area where existing differences are
    relatively mild, and making this aggressively configurable
    really seems like pushing into the wrong direction to me.

 2. Besides harming users by promoting gratuitious differences among
    platforms, such configurability also impedes flexibility of
    development because - even keeping the user-visibile functionality
    constant - the implementation cannot be modified when *.in-style
    interfaces are promised to downstream systems.

 3. Technically, i don't like the concept of *.in files at all.
    They make reading the code harder, as they are neither fish nor
    fowl, neither proper, readable C code, nor proper, readable *.h
    headers.  I want to slowly get rid of them.  I have deleted a
    few already and hope to continue in that direction, even if not
    with terribly high priority.  But i certainly don't want to add
    any more of them.

 4. Your specific patch is incorrect.
    "static const char * const secnames[SEC__MAX]"
    is not a declaration, but a definition.
    So if it goes into an *.in file, that *.in file cannot be
    included in any *.h file, but only in one *.c file, because
    otherwise every *.o file using the *.h file gets its own copy
    of the array.
    But enum roff_sec *is* needed in roff.h, so you are stuck.

I don't think you should try to fix the technicality (4) because
that will only make the patch yet uglier, aggravate (3), and not
solve the more fundamental issues (1) and (2) in the least.


To help finding a solution, i should like to understand the scope
of the issue better.

Which sections specificially do you want to order in which
non-standard way?

Is there any other, specific issue except ordering that you would
like to change?  If so, which one?

Note that, if you simply use some additional, non-standard sections
in some pages, i'm not convinced they need to be listed at all.
They should work just fine as SEC_CUSTOM.

Then again, if a section name is sufficiently widespread, we
usually carry it even if it is not used in every system (e.g. LIBRARY).
If you think there is such a case of a section name used by many, but
not by all systems, that ought to be added, which one is it?

Yours,
  Ingo


> commit 369032da926596cc6cec2a5615fe3afd9f6083db
> Author: Yuri Pankov <yuri.pankov@nexenta.com>
> Date:   Tue May 30 01:02:14 2017 +0300
> 
>     move section list to separate file
> 
> diff --git a/mdoc_validate.c b/mdoc_validate.c
> index 1d3d81e..ede6863 100644
> --- a/mdoc_validate.c
> +++ b/mdoc_validate.c
> @@ -251,33 +251,6 @@ static	const enum roff_tok rsord[RSORD_MAX] = {
>  	MDOC__O
>  };
>  
> -static	const char * const secnames[SEC__MAX] = {
> -	NULL,
> -	"NAME",
> -	"LIBRARY",
> -	"SYNOPSIS",
> -	"DESCRIPTION",
> -	"CONTEXT",
> -	"IMPLEMENTATION NOTES",
> -	"RETURN VALUES",
> -	"ENVIRONMENT",
> -	"FILES",
> -	"EXIT STATUS",
> -	"EXAMPLES",
> -	"DIAGNOSTICS",
> -	"COMPATIBILITY",
> -	"ERRORS",
> -	"SEE ALSO",
> -	"STANDARDS",
> -	"HISTORY",
> -	"AUTHORS",
> -	"CAVEATS",
> -	"BUGS",
> -	"SECURITY CONSIDERATIONS",
> -	NULL
> -};
> -
> -
>  void
>  mdoc_node_validate(struct roff_man *mdoc)
>  {
> diff --git a/roff.h b/roff.h
> index f0c2bf4..8f90faa 100644
> --- a/roff.h
> +++ b/roff.h
> @@ -16,6 +16,8 @@
>   * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
>   */
>  
> +#include "roffsec.in"
> +
>  struct	ohash;
>  struct	mdoc_arg;
>  union	mdoc_data;
> @@ -26,33 +28,6 @@ enum	roff_macroset {
>  	MACROSET_MAN
>  };
>  
> -enum	roff_sec {
> -	SEC_NONE = 0,
> -	SEC_NAME,
> -	SEC_LIBRARY,
> -	SEC_SYNOPSIS,
> -	SEC_DESCRIPTION,
> -	SEC_CONTEXT,
> -	SEC_IMPLEMENTATION,	/* IMPLEMENTATION NOTES */
> -	SEC_RETURN_VALUES,
> -	SEC_ENVIRONMENT,
> -	SEC_FILES,
> -	SEC_EXIT_STATUS,
> -	SEC_EXAMPLES,
> -	SEC_DIAGNOSTICS,
> -	SEC_COMPATIBILITY,
> -	SEC_ERRORS,
> -	SEC_SEE_ALSO,
> -	SEC_STANDARDS,
> -	SEC_HISTORY,
> -	SEC_AUTHORS,
> -	SEC_CAVEATS,
> -	SEC_BUGS,
> -	SEC_SECURITY,
> -	SEC_CUSTOM,
> -	SEC__MAX
> -};
> -
>  enum	roff_type {
>  	ROFFT_ROOT,
>  	ROFFT_BLOCK,
> diff --git a/roffsec.in b/roffsec.in
> new file mode 100644
> index 0000000..6fd4172
> --- /dev/null
> +++ b/roffsec.in
> @@ -0,0 +1,70 @@
> +/*
> + * Copyright (c) 2008-2012 Kristaps Dzonsons <kristaps@bsd.lv>
> + * Copyright (c) 2010-2017 Ingo Schwarze <schwarze@openbsd.org>
> + * Copyright (c) 2010 Joerg Sonnenberger <joerg@netbsd.org>
> + *
> + * Permission to use, copy, modify, and distribute this software for any
> + * purpose with or without fee is hereby granted, provided that the above
> + * copyright notice and this permission notice appear in all copies.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHORS DISCLAIM ALL WARRANTIES
> + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
> + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR
> + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
> + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
> + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
> + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
> + */
> +
> +enum    roff_sec {
> +	SEC_NONE = 0,
> +	SEC_NAME,
> +	SEC_LIBRARY,
> +	SEC_SYNOPSIS,
> +	SEC_DESCRIPTION,
> +	SEC_CONTEXT,
> +	SEC_IMPLEMENTATION,     /* IMPLEMENTATION NOTES */
> +	SEC_RETURN_VALUES,
> +	SEC_ENVIRONMENT,
> +	SEC_FILES,
> +	SEC_EXIT_STATUS,
> +	SEC_EXAMPLES,
> +	SEC_DIAGNOSTICS,
> +	SEC_COMPATIBILITY,
> +	SEC_ERRORS,
> +	SEC_SEE_ALSO,
> +	SEC_STANDARDS,
> +	SEC_HISTORY,
> +	SEC_AUTHORS,
> +	SEC_CAVEATS,
> +	SEC_BUGS,
> +	SEC_SECURITY,
> +	SEC_CUSTOM,
> +	SEC__MAX
> +};
> +
> +static const char * const secnames[SEC__MAX] = {
> +	NULL,
> +	"NAME",
> +	"LIBRARY",
> +	"SYNOPSIS",
> +	"DESCRIPTION",
> +	"CONTEXT",
> +	"IMPLEMENTATION NOTES",
> +	"RETURN VALUES",
> +	"ENVIRONMENT",
> +	"FILES",
> +	"EXIT STATUS",
> +	"EXAMPLES",
> +	"DIAGNOSTICS",
> +	"COMPATIBILITY",
> +	"ERRORS",
> +	"SEE ALSO",
> +	"STANDARDS",
> +	"HISTORY",
> +	"AUTHORS",
> +	"CAVEATS",
> +	"BUGS",
> +	"SECURITY CONSIDERATIONS",
> +	NULL
> +};
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: move roff sections to a separate file

[Yuri Pankov]

On Tue, 30 May 2017 01:47:17 +0200, Ingo Schwarze wrote:
> Hi Yuri,
> 
> Yuri Pankov wrote on Tue, May 30, 2017 at 01:14:35AM +0300:
> 
>> I'd like a propose a small change which would simplify maintaining a
>> downstream version for us (as we need it modified and with a bit changed
>> order) - move list of sections (roff_sec) and their descriptions (secnames)
>> to separate file which we keep without a need to sync with upstream,
>> roffsec.in. Patch is pretty straightforward and simply moves both there.
> 
> I see why such small differences can be a bother in maintenance,
> but i don't like the direction this is going.
> 
>   1. I'd like to encourage more consistency in manual page organization
>      across different systems, not facilitate divergence.  Section
>      naming and ordering is an area where existing differences are
>      relatively mild, and making this aggressively configurable
>      really seems like pushing into the wrong direction to me.
> 
>   2. Besides harming users by promoting gratuitious differences among
>      platforms, such configurability also impedes flexibility of
>      development because - even keeping the user-visibile functionality
>      constant - the implementation cannot be modified when *.in-style
>      interfaces are promised to downstream systems.
> 
>   3. Technically, i don't like the concept of *.in files at all.
>      They make reading the code harder, as they are neither fish nor
>      fowl, neither proper, readable C code, nor proper, readable *.h
>      headers.  I want to slowly get rid of them.  I have deleted a
>      few already and hope to continue in that direction, even if not
>      with terribly high priority.  But i certainly don't want to add
>      any more of them.

That's unfortunate as .in files allow us to keep the man sections 
(msec.in) and library (lib.in) descriptions separate of the code 
updates, see:

https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/mandoc/lib.in
https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/mandoc/msec.in

That's where we historically differ from BSD section layout, and it's 
(sadly) not going to change soon.

>   4. Your specific patch is incorrect.
>      "static const char * const secnames[SEC__MAX]"
>      is not a declaration, but a definition.
>      So if it goes into an *.in file, that *.in file cannot be
>      included in any *.h file, but only in one *.c file, because
>      otherwise every *.o file using the *.h file gets its own copy
>      of the array.
>      But enum roff_sec *is* needed in roff.h, so you are stuck.

It works as it's *static*, but I do agree that it's ugly, it was more an 
idea than actual patch.

> I don't think you should try to fix the technicality (4) because
> that will only make the patch yet uglier, aggravate (3), and not
> solve the more fundamental issues (1) and (2) in the least.
> 
> 
> To help finding a solution, i should like to understand the scope
> of the issue better.
> 
> Which sections specificially do you want to order in which
> non-standard way?
> 
> Is there any other, specific issue except ordering that you would
> like to change?  If so, which one?

We have CONTEXT section after RETURN VALUES.

> Note that, if you simply use some additional, non-standard sections
> in some pages, i'm not convinced they need to be listed at all.
> They should work just fine as SEC_CUSTOM.

The sections I listed below are standard for us, so there's a need to 
order them properly.

> Then again, if a section name is sufficiently widespread, we
> usually carry it even if it is not used in every system (e.g. LIBRARY).
> If you think there is such a case of a section name used by many, but
> not by all systems, that ought to be added, which one is it?
We have the following sections in our man pages that aren't present 
currently in validation, and I don't think it'd make sense to add them 
as generic ones:

INTERFACE LEVEL (after SYNOPSIS and before DESCRIPTION)
CODE SET INDEPENDENCE
INTERFACE STABILITY
MT-LEVEL

Having separate .in file would allow us having our own ordering rules 
while making updates easier - we do NOT modify any of the mdocml source 
except for customized .in files.

To sum it all up, the differences are for historic reasons, and not 
something that can be changed easily, hence the request.
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: move roff sections to a separate file

[Ingo Schwarze]

Hi Yuri,

Yuri Pankov wrote on Tue, May 30, 2017 at 03:09:06AM +0300:
> On Tue, 30 May 2017 01:47:17 +0200, Ingo Schwarze wrote:

>>   3. Technically, i don't like the concept of *.in files at all.
>>      They make reading the code harder, as they are neither fish nor
>>      fowl, neither proper, readable C code, nor proper, readable *.h
>>      headers.  I want to slowly get rid of them.  I have deleted a
>>      few already and hope to continue in that direction, even if not
>>      with terribly high priority.  But i certainly don't want to add
>>      any more of them.

> That's unfortunate as .in files allow us to keep the man sections
> (msec.in) and library (lib.in) descriptions separate of the code

Oh.  Yeah.  Don't panic about those two.  Regarding msec.in, that's
the one of these files where i don't see any realistical chance to
get rid of it in any forseeable future.  That indeed varies so
wildly among systems for historical reasons that there is little
hope of convergence.

Regarding lib.in, i had half forgetten about it already because it
was deleted from OpenBSD long ago.  We don't use the .Lb macro at
all.  But lib.in was expressly kept in the portable distribution
because illumos isn't the only system out there that likes grooming
a bewildering zoo of libraries.  Free, Net, and DragonFly rely on
that too, so it's not going away either.

But as soon as i have to touch st.c for some reason, st.in
is likely to die, and one day, predefs.in might get swallowed up
by roff.c, or by a similar file in the neighbourhood.


[...]
> We have CONTEXT section after RETURN VALUES.

Hmmm.  You have a point here.  We added that section recently
in OpenBSD, and maybe it was premature to make it public in this
way.

I'm currently working on a -Wstyle message level.  That is likely
to need some system dependencies anyway.  I haven't worked out
the required concepts of system dependency yet, but hope to
keep them as simple as possible.  Maybe that will yield a natural
solution.

For the next update of your tree (probably to 1.14.1), i suggest
that you simply maintain a local 4-line patch for CONTEXT; it
shouldn't be too burdensome, the enum doesn't change that
frequently upstream.

I'll try that you can get rid of it again.  In case i forget, it
might be useful to remind me in the context of the *next* update
after that.

> The sections I listed below are standard for us, so there's a need
> to order them properly.
[...]
> We have the following sections in our man pages that aren't present
> currently in validation, and I don't think it'd make sense to add
> them as generic ones:
> 
> INTERFACE LEVEL (after SYNOPSIS and before DESCRIPTION)
> CODE SET INDEPENDENCE
> INTERFACE STABILITY
> MT-LEVEL

I see.  NetBSD has a similar need, they use COMPATIBILITY, CODE
REFERENCES, and SECURITY CONSIDERATIONS in some pages and want those
in a specific order.  While migrating functionality from mdoclint(1)
to mandoc(1) -Wall, i have to find a solution for that anyway, so
that will very likely solve the task for you, too.  The peculiar
ordering of CONTEXT in OpenBSD is of the same kind, i think.

While issues like SEE ALSO before DESCRIPTION are WARNINGS (such
an order would be wrong on any system), the system-specific checks
will probably end up in the lower STYLE message level, but that
shouldn't be a problem for you.

So i think i know now what i need to know when doing the (low-intensity)
work on the message system during the next few months.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv