archived messages with "From " get truncated

archived messages with "From " get truncated

[Vincent Lefevre]

Below is a message that was sent by Stephane to workers.
It has been archived here:

  https://www.zsh.org/mla/workers/2021/msg01272.html

but got truncated just before a line starting with "From ".
It seems that the mail archive software is buggy, thinking
that this starts a new mail message.

In the archives, there is a risk that this message gets truncated
in the forwarded message below, for the same reason. For those who
read the archive, one has, with quotes to avoid the truncation:

[...]
> Why would one write print foo >&- in the first place, other than
> to check whether print reports error or not (and in that regard,
> the behaviour is very misleading)
> 
> From looking at the history, it looks more like:
> 
> 1990 (1.0) echo ignores write failures (by design or not).
[...]

----- Forwarded message from Stephane Chazelas <stephane@chazelas.org> -----

Date: Wed, 9 Jun 2021 19:16:17 +0100
From: Stephane Chazelas <stephane@chazelas.org>
To: Bart Schaefer <schaefer@brasslantern.com>
Cc: Zsh hackers list <zsh-workers@zsh.org>
Subject: Re: [BUG] builtin echo doesn't check write error

2021-06-09 09:13:42 -0700, Bart Schaefer:
[...]
> My (possibly poor) recollection is that this was made this way in part
> for ports to environments that don't have a /dev/null device

While that sounds like a very plausible reason for the original
behaviour whereby "print" would not report any write error,
that doesn't really tie up with the current behaviour where
the error is supressed only when print/echo's stdout is
explicitly closed.

Why would one write print foo >&- in the first place, other than
to check whether print reports error or not (and in that regard,
the behaviour is very misleading)

From looking at the history, it looks more like:

1990 (1.0) echo ignores write failures (by design or not).
1999 workers/9129 Peter writes the "print foo >&-" succeeds, no
     error test case, just documenting the actual behaviour of
     print ignoring errors (here using >&- as the easiest way to
     trigger a write error).
2002 workers/16503 Clint adds some error messages upon write
     errors.
2002 workers/16556 we realise it breaks the test case above, so
     Bart removes the error message on EBADF for that test case
     to pass again. The fact that there's still an error message
     in (print)>&- looks like an oversight. The error in that
     case is output by execcmd_exec after print has returned as
     it tries to fflush stdout again,, not bin_print (you'll
     notice the error message doesn't mention "print").
2011 workers/29845 Peter notices the error is displayed in
     (exec >&-; print) and adds a test case for it, but I'm not
     sure he correctly identified why.

It very much looks like an (multiple) accident of
implementation.

POSIX does say that printf/echo should return with a non-zero
exit status upon error, and stderr be used for diagnostix
errors as usual. It's not clear to me if implementations are at
liberty to decide whether a write() error is considered an error
or not.

In any case, it would be useful from an error point of view to
be able to detect when writing fails.

-- 
Stephane

----- End of forwarded message -----

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: archived messages with "From " get truncated

[Daniel Shahaf]

Vincent Lefevre wrote on Thu, Jun 24, 2021 at 10:36:12 +0200:
> Below is a message that was sent by Stephane to workers.
> It has been archived here:
> 
>   https://www.zsh.org/mla/workers/2021/msg01272.html
> 
> but got truncated just before a line starting with "From ".
> It seems that the mail archive software is buggy, thinking
> that this starts a new mail message.

The problem might be either in the mailing list software or in how we
invoke it.

We invoke mhonarc as follows:

   134          /usr/local/bin/mhonarc \
   135                  -quiet \
   136                  -definevar listlocalpartsansprefix=${listlocalpart#zsh-} \
   137                  -title  "${title}" \
   138                  -ttitle "${title}" \
   139                  -tlevels 9999 \
   140                  -rcfile /usr/local/www/mhonarc.zsh \
   141                  -add \
   142                  -- "$munged_tmpfile" \
   143            || exit EX_UNAVAILABLE

Here, ${munged_tmpfile} is a file that was created by
«() { munged_tmpfile=$1; cat > $munged_tmpfile } =(:)», the whole thing
being invoked by Exim using a «"| /path/to/script"» target in
/etc/aliases.  (The "munged" terminoilogy is because there's also
a «perl -pi -E 's/^X-Seq: …/…/ if (1../^$/)'» in there.)

Does anyone happen to see off the top of their heads what we're doing
wrong?

We still have the original message and can regenerate the archives once
we've fixed the intake problem.

mhonarc v2.6.24.

Thanks for the report, Vincent.

Daniel


> In the archives, there is a risk that this message gets truncated
> in the forwarded message below, for the same reason. For those who
> read the archive, one has, with quotes to avoid the truncation:
> 
> [...]
> > Why would one write print foo >&- in the first place, other than
> > to check whether print reports error or not (and in that regard,
> > the behaviour is very misleading)
> > 
> > From looking at the history, it looks more like:
> > 
> > 1990 (1.0) echo ignores write failures (by design or not).
> [...]
> 
> ----- Forwarded message from Stephane Chazelas <stephane@chazelas.org> -----
> 
> Date: Wed, 9 Jun 2021 19:16:17 +0100
> From: Stephane Chazelas <stephane@chazelas.org>
> To: Bart Schaefer <schaefer@brasslantern.com>
> Cc: Zsh hackers list <zsh-workers@zsh.org>
> Subject: Re: [BUG] builtin echo doesn't check write error
> 
> 2021-06-09 09:13:42 -0700, Bart Schaefer:
> [...]
> > My (possibly poor) recollection is that this was made this way in part
> > for ports to environments that don't have a /dev/null device
> 
> While that sounds like a very plausible reason for the original
> behaviour whereby "print" would not report any write error,
> that doesn't really tie up with the current behaviour where
> the error is supressed only when print/echo's stdout is
> explicitly closed.
> 
> Why would one write print foo >&- in the first place, other than
> to check whether print reports error or not (and in that regard,
> the behaviour is very misleading)
> 
> From looking at the history, it looks more like:
> 
> 1990 (1.0) echo ignores write failures (by design or not).
> 1999 workers/9129 Peter writes the "print foo >&-" succeeds, no
>      error test case, just documenting the actual behaviour of
>      print ignoring errors (here using >&- as the easiest way to
⋮
> 
> ----- End of forwarded message -----
> 
> -- 
> Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
> 

Re: archived messages with "From " get truncated

[Vincent Lefevre]

On 2021-06-24 19:11:56 +0000, Daniel Shahaf wrote:
> Vincent Lefevre wrote on Thu, Jun 24, 2021 at 10:36:12 +0200:
> > Below is a message that was sent by Stephane to workers.
> > It has been archived here:
> > 
> >   https://www.zsh.org/mla/workers/2021/msg01272.html
> > 
> > but got truncated just before a line starting with "From ".
> > It seems that the mail archive software is buggy, thinking
> > that this starts a new mail message.
> 
> The problem might be either in the mailing list software or in how we
> invoke it.
> 
> We invoke mhonarc as follows:
> 
>    134          /usr/local/bin/mhonarc \
>    135                  -quiet \
>    136                  -definevar listlocalpartsansprefix=${listlocalpart#zsh-} \
>    137                  -title  "${title}" \
>    138                  -ttitle "${title}" \
>    139                  -tlevels 9999 \
>    140                  -rcfile /usr/local/www/mhonarc.zsh \
>    141                  -add \
>    142                  -- "$munged_tmpfile" \
>    143            || exit EX_UNAVAILABLE
> 
> Here, ${munged_tmpfile} is a file that was created by
> «() { munged_tmpfile=$1; cat > $munged_tmpfile } =(:)», the whole thing
> being invoked by Exim using a «"| /path/to/script"» target in
> /etc/aliases.  (The "munged" terminoilogy is because there's also
> a «perl -pi -E 's/^X-Seq: …/…/ if (1../^$/)'» in there.)
> 
> Does anyone happen to see off the top of their heads what we're doing
> wrong?

If I understand correctly, $munged_tmpfile is a *single* message.
Then, according to the mhonarc documentation

  https://www.mhonarc.org/MHonArc/doc/quickstart.html#adding

you must not use the "-add" option, but "-single". Alternatively,
you may use "-add" for a single message, but the message must be
read from the standard input instead of being a filename argument.

The examples:
  (1) mhonarc -add <path>/mailfolder
  (2) mhonarc -add < single.msg
  (3) cat single.msg | mhonarc -add
  (4) mhonarc -single < messagefile > file.html
  (5) mhonarc -single messagefile > file.html

The syntax above is case (1), i.e. with a folder. And I suppose
that mhonarc regards a line starting with "From " as the beginning
of a new mail message from the folder, even if this line has a
wrong syntax for such a mail separator. Hence the truncation.
Then, I don't know what happens with the remaining part as this
would yield an invalid mail message; perhaps it is just discarded.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: archived messages with "From " get truncated

[Daniel Shahaf]

Vincent Lefevre wrote on Thu, 24 Jun 2021 23:36 +00:00:
> On 2021-06-24 19:11:56 +0000, Daniel Shahaf wrote:
> > Vincent Lefevre wrote on Thu, Jun 24, 2021 at 10:36:12 +0200:
> > > Below is a message that was sent by Stephane to workers.
> > > It has been archived here:
> > > 
> > >   https://www.zsh.org/mla/workers/2021/msg01272.html
> > > 
> > > but got truncated just before a line starting with "From ".
> > > It seems that the mail archive software is buggy, thinking
> > > that this starts a new mail message.
> > 
> > The problem might be either in the mailing list software or in how we
> > invoke it.
> > 
> > We invoke mhonarc as follows:
> > 
> >    134          /usr/local/bin/mhonarc \
> >    135                  -quiet \
> >    136                  -definevar listlocalpartsansprefix=${listlocalpart#zsh-} \
> >    137                  -title  "${title}" \
> >    138                  -ttitle "${title}" \
> >    139                  -tlevels 9999 \
> >    140                  -rcfile /usr/local/www/mhonarc.zsh \
> >    141                  -add \
> >    142                  -- "$munged_tmpfile" \
> >    143            || exit EX_UNAVAILABLE
> > 
> > Here, ${munged_tmpfile} is a file that was created by
> > «() { munged_tmpfile=$1; cat > $munged_tmpfile } =(:)», the whole thing
> > being invoked by Exim using a «"| /path/to/script"» target in
> > /etc/aliases.  (The "munged" terminoilogy is because there's also
> > a «perl -pi -E 's/^X-Seq: …/…/ if (1../^$/)'» in there.)
> > 
> > Does anyone happen to see off the top of their heads what we're doing
> > wrong?
> 
> If I understand correctly, $munged_tmpfile is a *single* message.

Indeed.

> Then, according to the mhonarc documentation
> 
>   https://www.mhonarc.org/MHonArc/doc/quickstart.html#adding
> 
> you must not use the "-add" option, but "-single".

Thanks!  I missed that flag.  I'll have a look.  (If someone can beat me
to it, please do.)

> Alternatively,
> you may use "-add" for a single message, but the message must be
> read from the standard input instead of being a filename argument.

Yup.  We actually already have a pending merge request for this, but it
fell through the cracks ☹

(MR !5, issue #37; not public, sorry).

> The examples:
>   (1) mhonarc -add <path>/mailfolder
>   (2) mhonarc -add < single.msg
>   (3) cat single.msg | mhonarc -add
>   (4) mhonarc -single < messagefile > file.html
>   (5) mhonarc -single messagefile > file.html
> 
> The syntax above is case (1), i.e. with a folder. And I suppose
> that mhonarc regards a line starting with "From " as the beginning
> of a new mail message from the folder, even if this line has a
> wrong syntax for such a mail separator. Hence the truncation.
> Then, I don't know what happens with the remaining part as this
> would yield an invalid mail message; perhaps it is just discarded.

Apparently it's discarded, unless it looks like a valid email message,
as in the case of git-format-patch(1) emails; see workers/48587 (which
is https://www.zsh.org/mla/workers/2021/msg00812.html).

And yes, I had to give the full link, because the numlist file (= the
database for the "X-Seq number to HTML" redirection script) is broken ☹

So, action items:

1. Review «mhonarc -single»
2. Deploy either «mhonarc -single» or «mhonarc -add < $munged_tmpfile»
3. Regenerate broken messages in the archives [for all lists]
4. Fix appends to the numlist file
5. Regenerate the missing part of the numlist file [for all lists]

Anyone interesting in helping, don't be shy ☺

Thanks a lot, Vincent.

Daniel