filling the References header

filling the References header

[Roderick Schertler]

At some point Gnus stopped filling the References header in followups,
but I'm not sure when.  Now the References header is formatted as a
single line no matter how long it is.  Have I screwed it up or did Gnus
stop filling References by default?  Is there a canned method I can use
to get References filled automatically?

I'm using 5.4.40 with 19.34.

-- 
Roderick Schertler
roderick@argon.org

Re: filling the References header

[Hrvoje Niksic]

Roderick Schertler <roderick@argon.org> writes:

> At some point Gnus stopped filling the References header in
> followups, but I'm not sure when.  Now the References header is
> formatted as a single line no matter how long it is.  Have I screwed
> it up or did Gnus stop filling References by default?

Gnus stopped doing it by default -- since 5.4, if I remember
correctly.  If there was an explanation for the change, I missed it.
Any way, it is understandable, as the continuation ``References'
actually caused problems some people, and the gain wasn't that big.

So, no.  You haven't screwed it up. :-)

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it."                                    -- Donald Knuth

Re: filling the References header

[Roderick Schertler]

On 14 May 1997 03:11:34 +0200, Hrvoje Niksic <hniksic@srce.hr> said:
> 
> Any way, it is understandable, as the continuation ``References'
> actually caused problems some people, and the gain wasn't that big.

Heck, if it makes life harder for the broken software while not bothering
properly written software I'm all for it, especially since the result is
more esthetically pleasing.

I'll probably put a hook together to fix this if nobody has one ready
made.

-- 
Roderick Schertler
roderick@argon.org

Re: filling the References header

[Hrvoje Niksic]

Roderick Schertler <roderick@argon.org> writes:

> Heck, if it makes life harder for the broken software while not bothering
> properly written software I'm all for it, especially since the result is
> more esthetically pleasing.
> 
> I'll probably put a hook together to fix this if nobody has one ready
> made.

:-)

Go for it.  Maybe it would be good if you make it an option, and send
it as a patch to Lars -- so that we can turn the behavior back on in
.emacs.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
WWW:  World-Wide-Waste.  Waste management corporation, which
      handles the billions of tons of garbage generated by just
      about everybody these days.

Re: filling the References header

[Steven L Baur]

> Roderick Schertler <roderick@argon.org> writes:
>> Heck, if it makes life harder for the broken software while not bothering
>> properly written software I'm all for it, especially since the result is
>> more esthetically pleasing.
>> 
>> I'll probably put a hook together to fix this if nobody has one ready
>> made.

IIRC the reason it's turned off is because refilling the references 
violates applicable standards.
-- 
steve@calag.com baur
Unsolicited commercial e-mail will be billed at $250/message.

Re: filling the References header

[Jason L Tibbitts III]

>>>>> "sb" == Steven L Baur <steve@xemacs.org> writes:

sb> IIRC the reason it's turned off is because refilling the references
sb> violates applicable standards.

How can that be?  I've been under the impression that headers can be
refilled at will; filling is a defined and reversible process which has no
effect on the content of the headers.

The lack of filling really gets unnerving as you start to get bounces from
gateways that can't handle the incredibly long lines it generates.  This
actually limits my ability to send mail at all to certain people, which I
deem worse than confusing a few clients.

 - J<

Re: filling the References header

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "JLT" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:
>>>>> "sb" == Steven L Baur <steve@xemacs.org> writes:

sb> IIRC the reason it's turned off is because refilling the references
sb> violates applicable standards.

JLT> How can that be?  I've been under the impression that headers can be
JLT> refilled at will; filling is a defined and reversible process which
JLT> has no effect on the content of the headers.

Technically, one is not supposed to alter the contents of existing headers
unless one is performing a canonical rewrite.  Reformatting a header field
qualifies on at least one level as altering that field and is therefore,
under the strictest interpretation of standards, something that should not
be done.

However, one may (and in fact is required to) append to References.  What
Gnus rationally should be doing is "looking" at the length of the last
References line and if it is greater than some arbitrary value less than
1025 characters it should start a new continuation line with new data.
This is perhaps less pretty than it could be, but since References is not
usually read by humans, who cares?

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3
Charset: noconv

iQCVAwUBM3uFqp6VRH7BJMxHAQF4xwP9E2xPW+W0MAWyUbak1DYNKCsUou+p6Bzf
v1LIUPiUkEy07cBeT7IZfzUq33xGif18/vIRXqRe62nTmPbQgxf3bhP340hPywov
mfaa60ST4VKvMpOlzT2p/7aplWh4vm7ujYT0K9kaI6gEBbQy4cTj5XZnaaruFuV/
uwqPTEWLqLQ=
=Bi86
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
PGP Key: at a key server near you! \ accelerate to dangerous speeds.
                                    \ 

Re: filling the References header

[Sudish Joseph]

Jason L Tibbitts writes:
>>>>>> "sb" == Steven L Baur <steve@xemacs.org> writes:
sb> IIRC the reason it's turned off is because refilling the references
sb> violates applicable standards.

> How can that be?  I've been under the impression that headers can be
> refilled at will; filling is a defined and reversible process which has no
> effect on the content of the headers.

References headers can grow to arbitrary lengths and may overflow
client buffer length limitations or the minimum length that needs to
be supported as per 1036.  Son-of-1036 has precise specs on what must
be done in such cases -- off the top of my head, the first two msg-ids
must be preserved and those closest to the end of the list should also
be preserved.

The gap where a list of msg-ids was removed should be indicated by 3
spaces (I think, it's been a long time since I looked at so1036).
This makes spaces in References headers significant and so it's
illegal to fill such headers.

AFAIK, no UA uses this functionality.  IMHO, using spaces to indicate
protocol-significant data is unwise.  Both here and in the signature
separator.  (I'm not saying we shouldn't follow the specs, of course.)

I like the present behaviour and, in fact, requested that we stop
filling References headers well over a year ago (unsuccessfully).

-Sudish

Re: filling the References header

[Sudish Joseph]

Sudish Joseph writes:
> be done in such cases -- off the top of my head, the first two msg-ids
> must be preserved and those closest to the end of the list should also
> be preserved.

Since I've already dug up so1036 once today, I might as correct
myslef.  Here're the relevant bits:

1036bis> Followup agents SHOULD not shorten References  headers.   If
1036bis> it  is absolutely necessary to shorten the header, as a des-
1036bis> perate last resort, a followup agent MAY do this by deleting
1036bis> some  of  the  message IDs.  However, it MUST not delete the
1036bis> first message ID, the last three message IDs (including that
1036bis> of  the immediate precursor), or any message ID mentioned in
1036bis> the body of the followup.

1036bis> [...] 

1036bis> When a References header is shortened, at least three blanks
1036bis> SHOULD be left between adjacent message IDs  at  each  point
1036bis> where  deletions  were  made.  Software preparing new Refer-
1036bis> ences headers SHOULD preserve multiple blanks in older  Ref-
1036bis> erences content.

1036bis>      NOTE:  It's desirable to have some marker of where
1036bis>      deletions occurred, but the restricted  syntax  of
1036bis>      the  header  makes  this  difficult.   Extra white
1036bis>      space is not a very good marker, since it  may  be
1036bis>      deleted  by  software  that ill-advisedly rewrites
1036bis>      headers, but at least it  doesn't  break  existing
1036bis>      software.

See <URL:ftp://zoo.toronto.edu/pub/news.txt.Z>, it's very well written
(gives background and justifications for lots of seemingly weird
decisions), if incomplete.

-Sudish

Re: filling the References header

[Lars Magne Ingebrigtsen]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> How can that be?  I've been under the impression that headers can be
> refilled at will; filling is a defined and reversible process which has no
> effect on the content of the headers.

Yes, continuation headers are perfectly legal, and software that
doesn't handle it is broken.

But.

1) RFC1036bis talks about preserving white space in the References
header.  So filling it might not be nice.  However, we can ignore
RFC1036bis here, since there is no software anywhere that adhere to it
in this area.

2) inn is broken.  When generating NOV headers on the fly, it doesn't
understand continuation headers.  This means that people who use inn
often see broken threads if the References headers is filled.

As there is no Right Thing to do here, I opted for swithing filling of
this header off by default.  It seems to be working better than the
"on" default setting.

If you want to switch it back on again, alter the `(References)' entry
in `message-header-format-alist' to read
`(References . message-fill-header)' instead.

> The lack of filling really gets unnerving as you start to get bounces from
> gateways that can't handle the incredibly long lines it generates.  This
> actually limits my ability to send mail at all to certain people, which I
> deem worse than confusing a few clients.

Yes, this is indeed a problem.

Uhm.  Perhaps I should switch filling on when mailing, and off when
doing posting?  Ugh.  That wouldn't work in the current paradigm,
since the References header is inserted before the article is edited,
and Messages doesn't know for sure at that point whether the message
is to be mailed or posted or both or neither.

Hm.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: filling the References header

[David Moore]

Roderick Schertler <roderick@argon.org> writes:

> On 14 May 1997 03:11:34 +0200, Hrvoje Niksic <hniksic@srce.hr> said:
> > 
> > Any way, it is understandable, as the continuation ``References'
> > actually caused problems some people, and the gain wasn't that big.
> 
> Heck, if it makes life harder for the broken software while not bothering
> properly written software I'm all for it, especially since the result is
> more esthetically pleasing.

If their software is broken it hurts you too.  Say you post something
with wrapped references.  they reply to it, but their reader truncates
it.  when you get that response it's no longer sorted correctly in
_your_ display.


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: filling the References header

[Greg Stark]

> Perhaps I should switch filling on when mailing, and off when doing posting?

You could fill it in message-send-mail just before passing it off to the
message-send-mail-function. Normally i would advocate for letting the user see
the message as nearly exactly as it will be sent before sending it, but this
seems like a reasonable exception.

You might consider just putting a hook there and providing a function that
fills all the headers or a specified list of headers before sending for users
to put in that hook.

greg

Re: filling the References header

[Jason L Tibbitts III]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Perhaps I should switch filling on when mailing, and off when
LMI> doing posting?

How about some way to fold, but with lines longer than 80 characters.  Or
is the problem that any folding at all is going to hose things?  I suppose
in that case you should try to keep as much on the first line as possible.

BTW, the practical length limit is a paltry 254 characters if you don't
want to bounce off of idiotic sites.  I get lots of bounces from my mailing
lists every time someone includes an unfolded X-Face with 255 characters in
it.  And note that the References: line for this message is already over
that limit with only seven Message-IDs in it.

 - J<

Re: filling the References header

[Lars Magne Ingebrigtsen]

Greg Stark <gsstark@mit.edu> writes:

> You could fill it in message-send-mail just before passing it off to
> the message-send-mail-function. Normally i would advocate for
> letting the user see the message as nearly exactly as it will be
> sent before sending it, but this seems like a reasonable exception.
> 
> You might consider just putting a hook there and providing a
> function that fills all the headers or a specified list of headers
> before sending for users to put in that hook.

Yes...  but this feels a bit unclean.  We want the default to be:

1a) No filling of the References header in news articles
1b) Filling of the References header in mails

2) The user should see the header as it will be sent

These are somewhat contradictory.

Adding `message-{mail,news}-filled-headers' variables seems like
overkill.

Hm.

Perhaps `message-send-mail-hook' should just be initialized to
`(message-fill-references)'?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: filling the References header

[Lars Magne Ingebrigtsen]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> BTW, the practical length limit is a paltry 254 characters if you don't
> want to bounce off of idiotic sites. 

I've never seen anything as extreme as that.  Then again, I chose who
I communicate with with care.  

If their mail software is broken, then they're probably not worth
talking to.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: filling the References header

[Roderick Schertler]

On 19 May 1997 02:26:40 +0200, Lars Magne Ingebrigtsen <larsi@gnus.org> said:
> 
> We want the default to be:
> 
> 1a) No filling of the References header in news articles

You could make it a little better by starting a new line in the references
header if necessary (but leaving the existing text as is).  Then again, the
mixed fill effect resulting might be ugly enough to override the value.

-- 
Roderick Schertler
roderick@argon.org

Re: filling the References header

[Andreas Jaeger]

>>>>> Lars Magne Ingebrigtsen writes:

Lars> Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:
>> BTW, the practical length limit is a paltry 254 characters if you don't
>> want to bounce off of idiotic sites. 

Lars> I've never seen anything as extreme as that.  Then again, I chose who
Lars> I communicate with with care.  
You've got some weeks ago a bug report because our (broken :-( ) mail system at
university is MMDF and MMDF in its standard configuration only allows 
254 characters per header line :-(.
Quite a lot of mail coming from xemacs-beta had reference headers
longer than 254 chars and everybody used Gnus.
All mails with more than 254 chars went directly to my postmaster who
forwarded them to me, getting more and more friendly ;-) in his daily
bunch of forwarded mails:-(.

Lars> If their mail software is broken, then they're probably not worth
Lars> talking to.  :-)
Or they should upgrade. I finally convinced our local admin to patch
MMDF to use up to 1024 characters per line.

Andreas
-- 
 Andreas Jaeger   aj@arthur.rhein-neckar.de    jaeger@informatik.uni-kl.de
  Altenwoogstr. 31      67655 Kaiserslautern, Germany
   Phone +49 631 3403051 Fax/Modem +49 631 3403052
    http://www.student.uni-kl.de/~ajaeger/

Re: filling the References header

[Jason L Tibbitts III]

>>>>> "JLT" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:

JLT> BTW, the practical length limit is a paltry 254 characters if you
JLT> don't want to bounce off of idiotic sites.

And just this morning, in case you don't believe me....

UKCC.uky.edu unable to deliver following mail to recipient(s):
    <censored@ms.uky.edu>
UKCC.uky.edu received negative reply:
554 [ No such file or directory ] Input line to submit longer than 254 characters

The problem was a really huge X-Face header.  Several sites decided to
bounce this message; the list has 600 members.

 - J<

Re: filling the References header

[Lars Magne Ingebrigtsen]

Roderick Schertler <roderick@argon.org> writes:

> You could make it a little better by starting a new line in the references
> header if necessary (but leaving the existing text as is). 

That wouldn't really help any; it would just make the thing totally
unpredictable.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen