[TUHS] Re: non-Posix input files

[TUHS] Re: non-Posix input files

[Douglas McIlroy]

> "The requirement that awk add a trailing <newline> to the program argument
> text is to simplify the grammar, making it match a text file in form."

This should no more be a *requirement* for awk than globbing should have
been a requirement for MS-DOS apps. A widespread principle deserves a
widespread answer. If it is a requirement on awk, then for interoperability
it should be made a requirement on all programs that handle text files,
especially editors.

The way to do that, of course, would be to redefine text file to allow a
non-newline as the last character. Ugh.

Not warning perpetuates travesties like "awk END{print NR}' " giving a different
answer than "wc -l".

I agree that awk does the kind thing by supplying the final newline. But
it should recognize that this is non-standard behavior and warn in the
interest of discouraging the proliferation of garbage.

Postel's so-called "robustness principle" is in play here. "Be conservative
in what you send, be liberal in what you accept" would better read,
"Send conservatively; receive amply but grudgingly".

Doug

[TUHS] Re: non-Posix input files

[arnold]

Doug,

It turns out there wasn't enough context and we both got
confused.  The "requirement" is that

	awk '{ print }' ...

look to the lexer / grammer as if it had been "{ print }\n".

There is no requirement that input "text" files behave as if a
final newline is always there.

Thanks,

Arnold

Douglas McIlroy <douglas.mcilroy@dartmouth.edu> wrote:

> > "The requirement that awk add a trailing <newline> to the program argument
> > text is to simplify the grammar, making it match a text file in form."
>
> This should no more be a *requirement* for awk than globbing should have
> been a requirement for MS-DOS apps. A widespread principle deserves a
> widespread answer. If it is a requirement on awk, then for interoperability
> it should be made a requirement on all programs that handle text files,
> especially editors.
>
> The way to do that, of course, would be to redefine text file to allow a
> non-newline as the last character. Ugh.
>
> Not warning perpetuates travesties like "awk END{print NR}' " giving a different
> answer than "wc -l".
>
> I agree that awk does the kind thing by supplying the final newline. But
> it should recognize that this is non-standard behavior and warn in the
> interest of discouraging the proliferation of garbage.
>
> Postel's so-called "robustness principle" is in play here. "Be conservative
> in what you send, be liberal in what you accept" would better read,
> "Send conservatively; receive amply but grudgingly".
>
> Doug

[TUHS] Re: the case of the missing new line, Re: non-Posix input files

[John Levine]

It appears that Douglas McIlroy <douglas.mcilroy@dartmouth.edu> said:
>Postel's so-called "robustness principle" is in play here. "Be conservative
>in what you send, be liberal in what you accept" would better read,
>"Send conservatively; receive amply but grudgingly".

I'm reasonably sure he meant it more narrowly than that. "Be liberal" only
applied when the spec was ambiguous, as was often the case in the early Internet
when new corner cases popped up every day. When the spec clearly said to do
something he had no problem rejecting inputs that were wrong.

I haven't been writing code to handle byte stream text files as long as you have
but there's an unfixable problem since Unix lets you put a stream of bytes
not ending with \n into a file and then try to treat it as text.  Whatever you
do will be wrong for some uses.

Perhaps we can segue to the argument about semicolons in Algol and Pascal vs.
semicolons in PL/I and C which is more or less the same issue.

R's,
John

PS: If only the Unix file system had records with two-byte length prefixes, or
better fixed length 80 byte records, you know, like real computers do.

[TUHS] Re: the case of the missing new line, Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 398 bytes --]

On 3/30/25 10:23 AM, John Levine wrote:

> PS: If only the Unix file system had records with two-byte length prefixes, or
> better fixed length 80 byte records, you know, like real computers do.

Luxury.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 950 bytes --]

On 3/30/25 10:00 AM, arnold@skeeve.com wrote:
> Doug,
> 
> It turns out there wasn't enough context and we both got
> confused.  The "requirement" is that
> 
> 	awk '{ print }' ...
> 
> look to the lexer / grammer as if it had been "{ print }\n".
> 
> There is no requirement that input "text" files behave as if a
> final newline is always there.

There's a similar situation with `sh -c', since sh can treat the end of the
string as a command terminator. The rationale notes it:

"Earlier versions of this standard required that input files to the shell
be text files except that line lengths were unlimited. However, that was
overly restrictive in relation to the fact that shells can parse a script
without a trailing newline[.]"

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 659 bytes --]

On 3/30/25 12:12 PM, Steve Simon wrote:
> 
> Re: newlines at the end of files.
> 
> I hesitate to ask this in such exalted company, but isn’t it a question of whether the newline is (or should be) a line terminator, or a statement separator?

I think that's part of the difference between input files (e.g., to `sort')
and script files (the part of the `awk' specification I quoted earlier)
and the distinction that POSIX makes between them, where applicable.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Steve Simon]

Re: newlines at the end of files.

I hesitate to ask this in such exalted company, but isn’t it a question of whether the newline is (or should be) a line terminator, or a statement separator?

-Steve

[TUHS] Re: non-Posix input files

[arnold]

Chet Ramey <chet.ramey@case.edu> wrote:

> Sorry, I should have been more clear. This requirement is for scripts:
>
> "The requirement that awk add a trailing <newline> to the program argument
> text is to simplify the grammar, making it match a text file in form."
>
> which I imagine all awk implementations do.

Ah. Indeed, gawk does this. I don't remember what goes in The One
True Awk.

Thanks,

Arnold

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 1110 bytes --]

On 3/28/25 3:22 AM, arnold@skeeve.com wrote:
> Chet Ramey via TUHS <tuhs@tuhs.org> wrote:
> 
>> It also says "The standard utilities that have such restrictions always
>> specify "text files" in their STDIN or INPUT FILES sections," so you can't
>> avoid it.
>>
>> awk is one such utility (sh is not). This is an application requirement, so
>> awk is required to add a newline at the end of a file that does not have
>> one.
> 
> No awk I know of actually does that. Instead, they all treat end of file
> as terminating the current input record:

Sorry, I should have been more clear. This requirement is for scripts:

"The requirement that awk add a trailing <newline> to the program argument
text is to simplify the grammar, making it match a text file in form."

which I imagine all awk implementations do.

I think awk does the reasonable thing with an input file that doesn't end
in a newline.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[arnold]

Chet Ramey via TUHS <tuhs@tuhs.org> wrote:

> It also says "The standard utilities that have such restrictions always
> specify "text files" in their STDIN or INPUT FILES sections," so you can't
> avoid it.
>
> awk is one such utility (sh is not). This is an application requirement, so
> awk is required to add a newline at the end of a file that does not have
> one.

No awk I know of actually does that. Instead, they all treat end of file
as terminating the current input record:

	$ for awk in gawk nawk mawk mksawk "busybox awk" goawk
	> do echo == $awk
	> echo -n ab:cd | $awk -v RS=: '{ print ">" $0 "<" }'
	> done
	== gawk
	>ab<
	>cd<
	== nawk
	>ab<
	>cd<
	== mawk
	>ab<
	>cd<
	== mksawk
	>ab<
	>cd<
	== busybox awk
	>ab<
	>cd<
	== goawk
	>ab<
	>cd<

If a newline were being added, the output would have been:

	>ab<
	>cd
	<

Nor is this something that I'm about to change, or suggest to any other
awk author to change.

My two cents,

Arnold

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 1087 bytes --]

On 3/27/25 8:19 PM, Rob Pike wrote:
> What is a "character" according to POSIX? I suspect it's just a byte, which 
> is a long-outdated definition, the char keyword notwithstanding.

"3.58 Character

A sequence of one or more bytes representing a member of a character set.

Note:
     This term corresponds to the ISO C standard term multi-byte character,
where a single-byte character is a special case of a multi-byte character.
Unlike the usage in the ISO C standard, character here has no necessary
relationship with storage space, and byte is used when storage space is
discussed.

     See the definition of the portable character set in 6.1 Portable
Character Set for a further explanation of the graphical representations of
(abstract) characters, as opposed to character encodings."

https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_58


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

I should have added that .1 does not talk about characters.  That was left
to ANSI C

Sent from a handheld expect more typos than usual


On Thu, Mar 27, 2025 at 8:37 PM Clem Cole <clemc@ccc.com> wrote:

> POSIX.1 is clear - a regular file is a sequence of bytes and that is the
> only requirement.   See my previous email where I give the section number
> which is very crisp.
>
> Sent from a handheld expect more typos than usual
>
>
> On Thu, Mar 27, 2025 at 8:19 PM Rob Pike <robpike@gmail.com> wrote:
>
>> What is a "character" according to POSIX? I suspect it's just a byte,
>> which is a long-outdated definition, the char keyword notwithstanding.
>>
>> -rob
>>
>>

[-- Attachment #2: Type: text/html, Size: 1834 bytes --]

[TUHS] Re: non-Posix input files

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

POSIX.1 is clear - a regular file is a sequence of bytes and that is the
only requirement.   See my previous email where I give the section number
which is very crisp.

Sent from a handheld expect more typos than usual


On Thu, Mar 27, 2025 at 8:19 PM Rob Pike <robpike@gmail.com> wrote:

> What is a "character" according to POSIX? I suspect it's just a byte,
> which is a long-outdated definition, the char keyword notwithstanding.
>
> -rob
>
>

[-- Attachment #2: Type: text/html, Size: 1210 bytes --]

[TUHS] Re: non-Posix input files

[Rob Pike]

[-- Attachment #1: Type: text/plain, Size: 147 bytes --]

What is a "character" according to POSIX? I suspect it's just a byte, which
is a long-outdated definition, the char keyword notwithstanding.

-rob

[-- Attachment #2: Type: text/html, Size: 469 bytes --]

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 2057 bytes --]

On 3/27/25 3:35 PM, Clem Cole wrote:
> Chet  - as I said, we tried so hard to keep that kind of crap out.

I believe you!

Even the first published version of 1003.2, in 1992, says

"The definition of text file has caused controversy." and discusses how
the definition seemingly outlaws empty files.


> I'll accept that that is what the words say in the >>awk<< specification 
> document, but as one of the original authors of the first UNIX standard and 
> the later POSIX standard I can say we tried hard to mak sure we got it 
> right and follow the idea: A regular file has no structure and never to 
> allow the standard to impose it. 

Sure. I think the point is that there are a bunch of utilities, like awk,
bc, ed, sort, and so on, whose input is required to be a text file as
defined by the standard. troff isn't in the standard, so Branden can do
what he likes, and if he wants to accept text files as the standard defines
them, then ok.


> The issue here is the term POSIX.   Do you mean it to be the kernel (.1) 
> and if a >>specific<< application with .2  (the C compiler itself, awk, ed) 

There is no distinction like that any more. POSIX is all of it: shell,
utilities, library functions, system calls. The standard guarantees that
where an input file is required to be a text file it is explicitly
mentioned, and it does that.


> I understand having the application do it; I wish it did not.

Sorry, jargon. The application is the thing that calls awk; awk is a
utility (expressive shrug).

So whatever calls awk is responsible for making sure the script and any
input files are text files. awk can add a newline to the end of a script
file that doesn't have one, so the grammar works right. It's also entitled
to do whatever it wants, including throw an error, if the input file isn't
a text file.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

Hi Doug,

At 2025-03-27T15:50:50-0400, Douglas McIlroy wrote:
> It seems my reply to Clem went astray. POSIX Part 2, Shell and
> Utilities, is very clear:
> 
> 2..2.2.181 text file: A File that contains characters organized into
> one or more lines.
> The lines shall not contain NUL characters ...
> 
> 2.2.2.95 line: A sequence of zero or more non-<newline> characters
> plus a terminating <newline> character,
> 
> Oddly--and in my opinion wrongly--the standard excludes empty files.
> It would be a shock if an editor or groff refused to process an empty
> file and thereby broke Kernighan's law, "Do nothing gracefully".

I think I can claim that GNU troff has long upheld that principle.

$ printf '' | ~/groff-HEAD/bin/groff | grep . || echo NO OUTPUT
NO OUTPUT
$ printf '' | ~/groff-1.23.0/bin/groff | grep . || echo NO OUTPUT
NO OUTPUT
$ printf '' | ~/groff-1.22.4/bin/groff | grep . || echo NO OUTPUT
NO OUTPUT
$ printf '' | ~/groff-1.22.3/bin/groff | grep . || echo NO OUTPUT
NO OUTPUT

That carries us back through over 10 years of groff development.  I'd
bet it's been the case since 1991.  Maybe earlier.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: non-Posix input files

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 2881 bytes --]

[CC updated]

Hi folks,

This thread started on the groff@gnu list, so I'm looping it back in.

https://lists.gnu.org/archive/html/groff/2025-02/msg00088.html
https://lists.gnu.org/archive/html/groff/2025-03/msg00070.html

The formatter, GNU troff:

* catenates its input into a single stream, disregarding any file
  "boundaries" except the first and the last;
* does not generally require the input to end with a newline;
* throws a warning diagnostic if the input ends with the *roff escape
  character (and ignores that character);
* warns about and discards NUL characters from the input (along with
  several others that are invalid as input);
* always terminates its _output_ files (whether in Kernighan's "trout"
  format [originally seen in CSTR #97], or "approximate" [`troff -a`]
  format) with newlines (ASCII LF);
* and writes only printable ASCII characters and the space in
  "trout"/"grout" or approximate outputs, with one esoteric
  exception.[1]

I'd like to know what, specifically, this program should be doing
differently.

Regards,
Branden

[1] The `cf` request permits the copying of abritrary bytes from the
    input file named by it to "grout" output.  In the forthcoming
    _groff_ 1.24.0, the formatter handles this request only in the
    non-default "unsafe mode" enabled by the `-U` command-line option.

Quoting groff's Texinfo manual:

 -- Request: .trf ["]file
 -- Request: .cf ["]file
     Copy FILE as "throughput" to GNU 'troff' output.  Each line of FILE
     is output as if preceded by '\!', but is not interpreted by the
     formatter.  If FILE does not end with a newline, 'trf' appends one.
     Both requests break the line before reading FILE, unless invoked
     with the no-break control character.

     When used in a diversion, these requests embed a node (*note Gtroff
     Internals::) in it that, when interpolated, causes the contents of
     FILE to be copied to the output.  In AT&T 'troff', 'cf' does so
     even if a diversion is in use; this behavior is so anomalous that
     it must be considered a bug.

     'cf' copies the contents of FILE completely unprocessed; it is
     therefore an error to use this request in safer mode, which is the
     default.  Invoke GNU 'troff' or a front end with the '-U' option to
     enable unsafe mode.

     'trf' discards invalid input characters; recall *note
     Identifiers::.

     For 'cf', within a diversion, "completely unprocessed" means that
     each line of a file to be inserted is handled as if it were
     preceded by '\!\\!'.

     *Caution:* Use of these requests can put syntactically invalid
     content into the formatter's output, which 'groff''s output drivers
     then fail to process.  One application is to pass instructions to a
     postprocessor that interprets FILE's contents and filters it out
     before sending it to the output device.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[TUHS] Re: non-Posix input files

[Douglas McIlroy]

It seems my reply to Clem went astray. POSIX Part 2, Shell and
Utilities, is very clear:

2..2.2.181 text file: A File that contains characters organized into
one or more lines.
The lines shall not contain NUL characters ...

2.2.2.95 line: A sequence of zero or more non-<newline> characters
plus a terminating <newline> character,

Oddly--and in my opinion wrongly--the standard excludes empty files.
It would be a shock
if an editor or groff refused to process an empty file and thereby
broke Kernighan's law, "Do
nothing gracefully".

Doug

On Thu, Mar 27, 2025 at 3:35 PM Clem Cole <clemc@ccc.com> wrote:
>
> Chet  - as I said, we tried so hard to keep that kind of crap out. Dennis was right. FWIW: with UNIX (POSIX), input will end if an EOF  and the ANSI C untilies will obey it or a newline, so you can write the code to work fine either way. But that's a choice of implementation/what subroutines - how you think about the data.
>
> I'll accept that that is what the words say in the >>awk<< specification document, but as one of the original authors of the first UNIX standard and the later POSIX standard I can say we tried hard to mak sure we got it right and follow the idea: A regular file has no structure and never to allow the standard to impose it.  I think the core standard still says that, and the basic idea is unchanged. The actual structure of the input file is an application idea, not a UNIX/POSIX defined idea.
>
> The issue here is the term POSIX.   Do you mean it to be the kernel (.1) and if a >>specific<< application with .2  (the C compiler itself, awk, ed) which might put structure onto the file and that's fine.  The  >>OS<< does not set the structure — it is done by something else.
>
> I understand having the application do it; I wish it did not. Many applications (even text editors) can (and have) been written without needing one specific structure, which is my point.  I also accept that the folks who took over the standard in the name of "progress" changed (relaxed) much of what we worked so hard to avoid, knowing there were dragons - particularly WRT to textual information. We really did not want to repeat the errors of the 1960s.  i.e., as George Santayana originally wrote, “Those who cannot remember the past are condemned to repeat it.”
> ᐧ
>
> On Thu, Mar 27, 2025 at 3:05 PM Chet Ramey <chet.ramey@case.edu> wrote:
>>
>> On 3/27/25 3:00 PM, Clem Cole wrote:
>> > Argh -- I standard corrected.  We worked hard at the beginning to keep that
>> > crap out -- sigh.
>> >
>> > But  at least is does says: POSIX.1-2024 /_does not _//_distinguish between
>> > text files and binary files_/ (see the ISO C standard)
>>
>> It also says "The standard utilities that have such restrictions always
>> specify "text files" in their STDIN or INPUT FILES sections," so you can't
>> avoid it.
>>
>> awk is one such utility (sh is not). This is an application requirement, so
>> awk is required to add a newline at the end of a file that does not have
>> one.
>>
>> --
>> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>>                  ``Ars longa, vita brevis'' - Hippocrates
>> Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[TUHS] Re: non-Posix input files

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2635 bytes --]

Chet  - as I said, we tried so hard to keep that kind of crap out. Dennis
was right. FWIW: with UNIX (POSIX), input will end if an EOF  and the ANSI
C untilies will obey it or a newline, so you can write the code to work
fine either way. But that's a choice of implementation/what subroutines -
how you think about the data.

I'll accept that that is what the words say in the >>awk<< specification
document, but as one of the original authors of the first UNIX standard and
the later POSIX standard I can say we tried hard to mak sure we got it
right and follow the idea: A regular file has no structure and never to
allow the standard to impose it.  I think the core standard still says
that, and the basic idea is unchanged. The actual structure of the input
file is an application idea, not a UNIX/POSIX defined idea.

The issue here is the term POSIX.   Do you mean it to be the kernel (.1)
and if a >>specific<< application with .2  (the C compiler itself, awk, ed)
which might put structure onto the file and that's fine.  The  >>OS<< does
not set the structure — it is done by something else.

I understand having the application do it; I wish it did not. Many
applications (even text editors) can (and have) been written without
needing one specific structure, which is my point.  I also accept that the
folks who took over the standard in the name of "progress" changed
(relaxed) much of what we worked so hard to avoid, knowing there were
dragons - particularly WRT to textual information. We really did not want
to repeat the errors of the 1960s.  *i.e.*, as George Santayana originally
wrote, *“Those who cannot remember the past are condemned to repeat it.” *
ᐧ

On Thu, Mar 27, 2025 at 3:05 PM Chet Ramey <chet.ramey@case.edu> wrote:

> On 3/27/25 3:00 PM, Clem Cole wrote:
> > Argh -- I standard corrected.  We worked hard at the beginning to keep
> that
> > crap out -- sigh.
> >
> > But  at least is does says: POSIX.1-2024 /_does not _//_distinguish
> between
> > text files and binary files_/ (see the ISO C standard)
>
> It also says "The standard utilities that have such restrictions always
> specify "text files" in their STDIN or INPUT FILES sections," so you can't
> avoid it.
>
> awk is one such utility (sh is not). This is an application requirement, so
> awk is required to add a newline at the end of a file that does not have
> one.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>                  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
>

[-- Attachment #2: Type: text/html, Size: 4410 bytes --]

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 792 bytes --]

On 3/27/25 3:00 PM, Clem Cole wrote:
> Argh -- I standard corrected.  We worked hard at the beginning to keep that 
> crap out -- sigh.
> 
> But  at least is does says: POSIX.1-2024 /_does not _//_distinguish between 
> text files and binary files_/ (see the ISO C standard)

It also says "The standard utilities that have such restrictions always
specify "text files" in their STDIN or INPUT FILES sections," so you can't
avoid it.

awk is one such utility (sh is not). This is an application requirement, so
awk is required to add a newline at the end of a file that does not have
one.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1919 bytes --]

Argh -- I standard corrected.  We worked hard at the beginning to keep that
crap out -- sigh.

But  at least is does says: POSIX.1-2024 *does not **distinguish between
text files and binary files* (see the ISO C standard)

So my point is the same: don't blame POSIX- this is an ANSI C issue, and
utilities really should not care.
ᐧ

On Thu, Mar 27, 2025 at 2:53 PM Chet Ramey <chet.ramey@case.edu> wrote:

> On 3/27/25 2:45 PM, Clem Cole wrote:
> >
> > Branden Robin wrote:
> >
> >
> >      >> info groff gives semantics for including nonempty files that
> don't end
> >      >> with newline. Such files violate the Posix definition of text
> file.
> >
> >
> > Not so fast.   POSIX does not define a text file (that's an ANSI C-ism -
> > thank you DOS -- more in minute).
>
> It does.
>
> "3.387 Text File
>
> A file that contains characters organized into zero or more lines. The
> lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in
> length, including the <newline> character. Although POSIX.1-2024 does not
> distinguish between text files and binary files (see the ISO C standard),
> many utilities only produce predictable or meaningful output when
> operating
> on text files. The standard utilities that have such restrictions always
> specify "text files" in their STDIN or INPUT FILES sections."
>
>
> https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_387
>
> A line is defined as:
>
> "3.185 Line
>
> A sequence of zero or more non-<newline> characters plus a terminating
> <newline> character."
>
>
> https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_185
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>                  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
>

[-- Attachment #2: Type: text/html, Size: 3858 bytes --]

[TUHS] Re: non-Posix input files

[Douglas McIlroy]

I think you're looking at the wrong Posix document. Posix Shell and
Utilities (at least my ancient copy) says

2.2.2.181 Text file: A file that contains one or more lines.
The Lines shall not contain NUL characters ...

Doug

On Thu, Mar 27, 2025 at 2:45 PM Clem Cole <clemc@ccc.com> wrote:
>
>
> Branden Robin wrote:
>
>
>> >> info groff gives semantics for including nonempty files that don't end
>> >> with newline. Such files violate the Posix definition of text file.
>
>
> Not so fast.   POSIX does not define a text file (that's an ANSI C-ism - thank you DOS -- more in minute).
>
> Even in the original POSIX definition we were very careful to >>never<< do that abomination.  What the current POSIX specs says is:
>
> 3.164 File
> An object that can be written to, or read from, or both. A file has certain attributes, including access permissions and type. File types include regular file, character special file, block special file, FIFO special file, symbolic link, socket, and directory. Other types of files may be supported by the implementation.
>
> ....
>
> 3.317 Regular File
> A file that is a randomly accessible sequence of bytes, with no further structure imposed by the system.
>
> In fact, nowhere in the term "text file" defined (or used — search for it) in the POSIX standard.   The problem is that at the same time, ANSI C was being defined in the early 1980s  The DOS weenies wanted to support the silly two characters line terminations and old notion of a structured "text file" which PC/MS-DOS had inherited from the DEC and IBM OS's of 1960s and what IBM has called "access methods" in the old days (in UNIX everything is just a stream of bytes thank you or in Multics, everything is segment).  The UNIX folks on the C committee were very much against adding the support — which Lattice C for MS-DOS had added at the time but happened to be the most popular C compiler for the DOS target and the number of user of the compiler  was growing extremely rapidly.  They were also pushing for things like the near and far keywords.  It was quite a fight in the C community. (It was one of the additions that caused Dennis to quip: "When I read commentary about suggestions for where C should go, I often think back and give thanks that it wasn't developed under the advice of a worldwide crowd."
>
> Anyway, the UNIX folk had to decide which eveils the DOS folks were pushes they could live with and a compromise was the creation of the new "rb" "wb" "ab" crud in the fopen(3) call since we you use "r", "w", "b" as because POSIX allows UNIX to open a what ANSI C called a "text file" since there no such thing under POSIX.
>
> But in the ANSI C spec (which, remember, is >>only<< for C source code), it says:
>
> 5.2.1 Character sets
> .... The representation of each member of the source and execution basic character sets shall fit in a byte.
> In both the source and execution basic character sets, the value of each character after 0 in the above
> list of decimal digits shall be one greater than the value of the previous. In source files, there shall
> be some way of indicating the end of each line of text; this International Standard treats such an
> end-of-line indicator as if it were a single new-line character. In the basic execution character set,
> there shall be control characters representing alert, backspace, carriage return, and new line. If any
> other characters are encountered in a source file (except in an identifier, a character constant, a string
> literal, a header name, a comment, or a preprocessing token that is never converted to a token), the
> behavior is undefined.
>
> The key point in the highlighted text in ANSI C is that it does not require it.   A problem in the  ANSI C standard is that does >>use<< the term "text file" although ANSI C never really defines what one is (and probably could not because of the fight - UNIX would have said -- nope -- no structure required.  ANSI C does say in:
>
> 7.21.2 Streams
> ...
> Environmental limits
> An implementation shall support text files with lines containing at least 254 characters, including
> the terminating new-line character. The value of the macro BUFSIZ shall be at least 256.
>
> The whole compromise was that you did not have to use termination of type. If the concept of a 'textfile" is local to the implementation. UNIX folks get to keep doing things as they did before, so if you used fopen with "r" or "w" its up to the program to worry about the format of the file "access modes" from the 1960s were not needed.
> ᐧ

[TUHS] Re: non-Posix input files

[Chet Ramey via TUHS]

[-- Attachment #1.1: Type: text/plain, Size: 1407 bytes --]

On 3/27/25 2:45 PM, Clem Cole wrote:
> 
> Branden Robin wrote:
> 
> 
>      >> info groff gives semantics for including nonempty files that don't end
>      >> with newline. Such files violate the Posix definition of text file.
> 
> 
> Not so fast.   POSIX does not define a text file (that's an ANSI C-ism - 
> thank you DOS -- more in minute).

It does.

"3.387 Text File

A file that contains characters organized into zero or more lines. The
lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in
length, including the <newline> character. Although POSIX.1-2024 does not
distinguish between text files and binary files (see the ISO C standard),
many utilities only produce predictable or meaningful output when operating 
on text files. The standard utilities that have such restrictions always 
specify "text files" in their STDIN or INPUT FILES sections."

https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_387

A line is defined as:

"3.185 Line

A sequence of zero or more non-<newline> characters plus a terminating
<newline> character."

https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_185

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[TUHS] Re: non-Posix input files

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 4282 bytes --]

Branden Robin wrote:


>> info groff gives semantics for including nonempty files that don't end
> >> with newline. Such files violate the Posix definition of text file.


Not so fast.   POSIX does not define a text file (that's an ANSI C-ism -
thank you DOS -- more in minute).

Even in the original POSIX definition we were very careful to >>never<< do
that abomination.  What the current POSIX specs says is:

3.164 File
An object that can be written to, or read from, or both. A file has certain
attributes, including access permissions and type. File types include regular
file, character special file, block special file, FIFO special file,
symbolic link, socket, and directory. Other types of files may be supported
by the implementation.

....

3.317 Regular File
A file that is a randomly accessible sequence of bytes, with no further
structure imposed by the system.

In fact, nowhere in the term "text file" defined (or used — search for it)
in the POSIX standard.   The problem is that at the same time, ANSI C was
being defined in the early 1980s  The DOS weenies wanted to support the
silly two characters line terminations and old notion of a structured "text
file" which PC/MS-DOS had inherited from the DEC and IBM OS's of 1960s and
what IBM has called "access methods" in the old days (in UNIX everything is
just a stream of bytes thank you or in Multics, everything is segment).
The UNIX folks on the C committee were very much against adding the support
— which Lattice C for MS-DOS had added at the time but happened to be the
most popular C compiler for the DOS target and the number of user of the
compiler  was growing extremely rapidly.  They were also pushing for things
like the near and far keywords.  It was quite a fight in the C community. (It
was one of the additions that caused Dennis to quip: *"**When I read
commentary about suggestions for where C should go, I often think back and
give thanks that it wasn't developed under the advice of a worldwide
crowd."*

Anyway, the UNIX folk had to decide which eveils the DOS folks were pushes
they could live with and a compromise was the creation of the new "rb" "wb"
"ab" crud in the fopen(3) call since we you use "r", "w", "b" as because
POSIX allows UNIX to open a what ANSI C called a "text file" since there no
such thing under POSIX.

But in the ANSI C spec (which, remember, is >>only<< for C source code), it
says:

5.2.1 Character sets
.... The representation of each member of the source and execution basic
character sets shall fit in a byte.
In both the source and execution basic character sets, the value of each
character after 0 in the above
list of decimal digits shall be one greater than the value of the previous. In
source files, there shall
be some way of indicating the end of each line of text; this International
Standard treats such an
end-of-line indicator as if it were a single new-line character. In the
basic execution character set,
there shall be control characters representing alert, backspace, carriage
return, and new line. If any
other characters are encountered in a source file (except in an identifier,
a character constant, a string
literal, a header name, a comment, or a preprocessing token that is never
converted to a token), the
behavior is undefined.

The key point in the highlighted text in ANSI C is that it does not *require
it*.   A problem in the  ANSI C standard is that does >>use<< the term
"text file" although ANSI C never really defines what one is (and probably
could not because of the fight - UNIX would have said -- nope -- no
structure required.  ANSI C does say in:

7.21.2 Streams
...
Environmental limits
An implementation shall support text files with lines containing at least
254 characters, including
the terminating new-line character. The value of the macro BUFSIZ shall be
at least 256.

The whole compromise was that you did not have to use termination of type.
If the concept of a 'textfile" is local to the implementation. UNIX folks
get to keep doing things as they did before, so if you used fopen with "r"
or "w" its up to the program to worry about the format of the file "access
modes" from the 1960s were not needed.
ᐧ

[-- Attachment #2: Type: text/html, Size: 12150 bytes --]

[TUHS] Re: non-Posix input files

[Douglas McIlroy]

>> info groff gives semantics for including nonempty files that don't end
>> with newline. Such files violate the Posix definition of text file.
>>
>> Although groff is certainly justified in providing semantics for
>> non-Posix text, I suggest that it should warn when it does so.

> That's true but I'm hesitant to put groff in the business of wagging its
> finger at users feeding it non-strictly-conforming text files when doing
> so doesn't cause it any problems.

Causing groff problems is an odd criterion. The fact that groff will paste
files together unless the first happens to end in a newline is a sign of
groff 's internals, not of the underlying problem.

A newline missing at the end of a file is typically a symptom of either the
incaution of some other program (perhaps an editor) or of a file having
been unexpectedly truncated (as by a program abort). The latter cause
is common enough to justify warning always, not just about cases that
are inconvenient to groff.

Groff is what it is, but if the treatment of absent final newlines were up
for grabs, I'd argue for the more common solution: in all cases insert
a newline and warn.

Doug