Discarding non-ASCII input

[Joerg Sonnenberger <joerg@britannica.bec.de>]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

Hi all,
at the moment we are discarding any non-ASCII characters. This turns a
bunch of syntactically documents into complete garbage, e.g. by removing
the arguments for .SH macros. I think it is more reasonable to replace
them with "safe" garbage like iconv on most platforms does.

What do you think of the attached patch?

Joerg

[-- Attachment #2: read.c.diff --]
[-- Type: text/x-diff, Size: 1073 bytes --]

Index: read.c
===================================================================
RCS file: /home/joerg/cvsroot/mdocml/read.c,v
retrieving revision 1.26
diff -u -p -r1.26 read.c
--- read.c	7 Nov 2011 01:24:40 -0000	1.26
+++ read.c	5 Feb 2012 00:31:33 -0000
@@ -325,9 +325,9 @@ mparse_buf_r(struct mparse *curp, struct
 			 * Warn about bogus characters.  If you're using
 			 * non-ASCII encoding, you're screwing your
 			 * readers.  Since I'd rather this not happen,
-			 * I'll be helpful and drop these characters so
-			 * we don't display gibberish.  Note to manual
-			 * writers: use special characters.
+			 * I'll be helpful and replace these characters
+			 * with "?", so we don't display gibberish.
+			 * Note to manual writers: use special characters.
 			 */
 
 			c = (unsigned char) blk.buf[i];
@@ -337,6 +337,9 @@ mparse_buf_r(struct mparse *curp, struct
 				mandoc_msg(MANDOCERR_BADCHAR, curp,
 						curp->line, pos, "ignoring byte");
 				i++;
+				if (pos >= (int)ln.sz)
+					resize_buf(&ln, 256);
+				ln.buf[pos++] = '?';
 				continue;
 			}