Have message-ID set by my email server rather than by Message

Have message-ID set by my email server rather than by Message

[rameiko87]

Hello everyone! I'm on Emacs 29.1 and I use Message and sendmail for 
writing emails. I don't load Gnus but I do the easy C-x m. I can't get 
Emacs not to add Message-ID to my outgoing mail! I would be very 
grateful if some of you could help me out a little, I'm kind of a 
beginner. I set https://bpa.st/QLDQ and yet every email I write contains 
a message-ID which is set by emacs, rather than my email server.

If I customize message-user-fqdn then the Message-ID will reflect that. 
Therefore whatever is setting my message ID despite the config telling 
Message not to, is taking it from the variable message-user-fqdn!

Help is very much appreciated!

Re: Have message-ID set by my email server rather than by Message

[Lars-Johan Liman]

Hi!

Message-ID is one of the very few _mandatory_ header lines, so a message
without one is not a valid RFC 5322 (the sucessor of RFC 822) message.
Therefore the message "must not/cannot" leave your computer without one.
It has to be set on the computer if it's being shipped by a mail-related
protocol ((E)SMTP/LMTP/IMAP/POP[23]/...).

What you _can_ do (if I remember my sendmail.cf right, it's been a few
years since I taught that course ... ;-) ) is to _modify_ the Message-ID
header in your mail server. "If it looks like it's coming from the
computer, then modify Message-ID to look like ... (bla)". But I wouldn't
recommend it. Message-ID is supposed to be set by the sending software
(the Mail User Agent (MUA)) and it's supposed to uniquely identify the
message across all messages in the world ... ever. Modifying it would
make it very difficult to trace and debug how the message traverses the
Internet (which includes your computer). You can probably also influence
exactly how Emacs generates the Message-ID, but again, it's probably not
a good idea to mess with that.

Can I ask what problem you see with having your Emacs generate the
Message-ID?

				Best regards,
				  /Lars-Johan Liman


rameiko87@posteo.net 2023-12-08 18:44 [+0000]:
> Hello everyone! I'm on Emacs 29.1 and I use Message and sendmail for
> writing emails. I don't load Gnus but I do the easy C-x m. I can't get
> Emacs not to add Message-ID to my outgoing mail! I would be very
> grateful if some of you could help me out a little, I'm kind of a
> beginner. I set https://bpa.st/QLDQ and yet every email I write
> contains a message-ID which is set by emacs, rather than my email
> server.

> If I customize message-user-fqdn then the Message-ID will reflect
> that. Therefore whatever is setting my message ID despite the config
> telling Message not to, is taking it from the variable
> message-user-fqdn!

> Help is very much appreciated!

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

Dear Lars-Johan Liman,
Thank you very much for your kind and exhaustive reply. I read in the 
Gnus manual:

> If you have no idea what to insert for "yourmachine.yourdomain.tld", 
> you’ve got several choices. You can either ask your provider if he 
> allows you to use something like yourUserName.userfqdn.provider.net, or 
> you can use somethingUnique.yourdomain.tld if you own the domain 
> yourdomain.tld, or you can register at a service which gives private 
> users a FQDN for free.

None of these apply to me. Then I read just below:

> Finally you can tell Gnus not to generate a Message-ID for News at all 
> (and letting the server do the job) [...] you can also tell Gnus not to 
> generate Message-IDs for mail by saying:
> 
> //there's some code here//
> 
> , however some mail servers don’t generate proper Message-IDs, too, so 
> test if your Mail Server behaves correctly by sending yourself a Mail 
> and looking at the Message-ID.

This is perfect! So I customized message-required-mail-headers. Now your 
explanation shows why I should not remove the mail headers, but still I 
wonder about the meaning of the lines cited above from the Gnus manual. 
And I also wonder what is the meaning of the variable 
message-required-mail-headers if it's essentially ignored by Message 
when I remove the Message-ID header from the list.

Anyway now I'm back to the original problem where I don't know what to 
insert for Message-ID because the recommendations from the Gnus manual 
above don't apply, and also I can't understand how Message generates the 
portion which comes before the at-sign...

And by the way, the standard for me would be 
fedora.mail-host-address-is-not-set . Is this even a fully qualified 
domain name, without extension? And how would I set mail host address? 
Because I can only find the variable mail-host-address which is set to 
system-name, which is fedora. Therefore I would expect the Message-ID to 
be fsfxxxxxxx@fedora
(which by the way I think would miss the extension). If I set 
mail-host-address manually to fedora.ext, would Message-ID then be 
fsfxxxxxxx@fedora.ext? Then why don't they say 
mail-host-extension-is-not-set rather than mail-host-address-is-not-set 
in the default Message-ID when the mail-host-address *is* set but has no 
extension? So much confusion.

On 09.12.2023 08:03, Lars-Johan Liman wrote:
> Hi!
> 
> Message-ID is one of the very few _mandatory_ header lines, so a 
> message
> without one is not a valid RFC 5322 (the sucessor of RFC 822) message.
> Therefore the message "must not/cannot" leave your computer without 
> one.
> It has to be set on the computer if it's being shipped by a 
> mail-related
> protocol ((E)SMTP/LMTP/IMAP/POP[23]/...).
> 
> What you _can_ do (if I remember my sendmail.cf right, it's been a few
> years since I taught that course ... ;-) ) is to _modify_ the 
> Message-ID
> header in your mail server. "If it looks like it's coming from the
> computer, then modify Message-ID to look like ... (bla)". But I 
> wouldn't
> recommend it. Message-ID is supposed to be set by the sending software
> (the Mail User Agent (MUA)) and it's supposed to uniquely identify the
> message across all messages in the world ... ever. Modifying it would
> make it very difficult to trace and debug how the message traverses the
> Internet (which includes your computer). You can probably also 
> influence
> exactly how Emacs generates the Message-ID, but again, it's probably 
> not
> a good idea to mess with that.
> 
> Can I ask what problem you see with having your Emacs generate the
> Message-ID?
> 
> 				Best regards,
> 				  /Lars-Johan Liman

Re: Have message-ID set by my email server rather than by Message

[Arash Esbati]

rameiko87@posteo.net writes:

> This is perfect! So I customized message-required-mail-headers. Now
> your explanation shows why I should not remove the mail headers, but
> still I wonder about the meaning of the lines cited above from the
> Gnus manual. And I also wonder what is the meaning of the variable
> message-required-mail-headers if it's essentially ignored by Message
> when I remove the Message-ID header from the list.

I'm not sure if Message is ignoring the value of
`message-required-mail-headers'.

> Anyway now I'm back to the original problem where I don't know what to
> insert for Message-ID because the recommendations from the Gnus manual
> above don't apply, and also I can't understand how Message generates
> the portion which comes before the at-sign...

My guess is that your message-id like this:

  Message-ID: <8b21e2aa979f8c1944fc0b5ca4be1e5c@posteo.net>

is generated by your sendmail program.  I use msmtp which has an option:

‘set_msgid_header [(auto|off)]’
  When to set a Message-ID header: ‘auto’ adds a Message-ID header if
  the mail does not have one (this is the default), and ‘off’ never sets
  a Message-ID header.  Message-IDs have the form ‘hash@domain’, where
  ‘hash’ is a hash over some values that uniquely identify the mail, and
  ‘domain’ is either the value of the domain command (unless that is the
  default value ‘localhost’), or the domain part of the envelope-from
  address (if available), or the host name of the SMTP server.

Maybe you want to check your sendmail program.

Best, Arash

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

> I'm not sure if Message is ignoring the value of 
> `message-required-mail-headers'.

Well, how else do you explain that when I remove Message-ID from 
message-required-mail-headers and send my email, then the Message-ID is 
given by the value of message-user-fqdn or when nil, by system-name?

> My guess is that your message-id like this:
> Message-ID: <xxxxxxxx@posteo.net>
> is generated by your sendmail program.

Nono forget about these messages to info-gnus-english, which I'm sending 
using posteo's webmail.

> Maybe you want to check your sendmail program.

The Message-ID of my email always depends on message-user-fqdn, 
therefore I deduce that it's Message which is setting it.

Re: Have message-ID set by my email server rather than by Message

[Bob Newell]

Not 100% relevant, and I have posted an earlier version
before, but I have fun with the following which completely
randomizes the Message ID and user agent for every email I
send from gnus.  (Sometimes you actually want this; think
about it!)  However some SMTP servers rudely go ahead and
substitute their own Message ID for mine.  Bad manners indeed.
Gmail fortunately does not.

;;; Fully anonymize/randomize FQDN.

(defun munge-message-id (orig-fun &rest args)
  "Make a random FQDN and User Agent"
  (let* (
	 (alpha "abcdefghijklmnopqrstuvwxyz0123456789")
	 (domain '("com" "org" "net" "info" "us"))
	 (part1 "")
	 (part2 "")
	 (part3 "")
	 (part4 "")
	 (part5 "")
	 num0 rnum)
;;; Temporary.  Not in right place at all.
    (munge-user-agent)
;;; FQDN.
    ;;; Front half.
    (setq num0 (+ 8 (random 6)))
    (while (> num0 0)
      (setq rnum (random 35))
      (setq part1 (concat part1 (substring alpha rnum (+ 1 rnum))))
      (setq num0 (- num0 1))
      )
    (setq num0 (+ 8 (random 6)))
    (while (> num0 0)
      (setq rnum (random 35))
      (setq part2 (concat part2 (substring alpha rnum (+ 1 rnum))))
      (setq num0 (- num0 1))
      )
    ;;; Back half.
    (setq num0 (+ 4 (random 6)))
    (while (> num0 0)
      (setq rnum (random 35))
      (setq part3 (concat part3 (substring alpha rnum (+ 1 rnum))))
      (setq num0 (- num0 1))
      )
    (setq num0 (+ 4 (random 6)))
    (while (> num0 0)
      (setq rnum (random 35))
      (setq part4 (concat part4 (substring alpha rnum (+ 1 rnum))))
      (setq num0 (- num0 1))
      )
    ;;; TLD.
    (setq part5 (nth (random 4) domain))
;;; Build the message-id.
    (concat "<" part1 "." part2 "@" part3 "." part4 "." part5 ">")
    )
  )
(advice-add 'message-make-message-id :around #'munge-message-id)

;;; Semi-randomize User-Agent.

(defun munge-user-agent ()
 "Change user-agent to semi-random string"
 (let* ( (agents '("Mozilla" "Chrome" "Safari" "Brave" "Vivaldi" "Edge"))
	 (versions '("5.0" "5.5" "6.0" "6.5" "4.5" "7.0"))) 
    (setq gnus-user-agent (concat (nth (random 5) agents) "/" (nth (random 5) versions)))))

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Have message-ID set by my email server rather than by Message

[Lars-Johan Liman]

Hi rameiko87!

I know more about message standards then I know about the internals of
GNUS. That's why I come from that angle.

I'm beginning to understand your undelying problem, though, and I
realise it's a generic one that more people "ought to have". Am I right
in understanding it as "my host doesn't know its own hostname
(identity), so it's unable to create a useful Message-ID"?

If I "zoom out" and try to rephrase your problem as a generic question,
I come up with this:

"Which entity (i.e., which device) should be responsible for adding the
Message-ID header to a message composed on a device which has little or
no notion of its own host identity, and in those cases what should the
Message-ID be?"

Is that in line with the problem that you're having?

I happen to know the author of the mail specification RFC, and I'm
thinking about asking him for his thoughts. Would you object to that?

In the meantime, I'll provide this link in the hope that you can find
grains of help in this section of the mail message standard. The full
answer is not in there, but maybe you can make a better informed
decision of what to do if you read that.

https://datatracker.ietf.org/doc/html/rfc5322#section-3.6.4

				Best regards,
				  /Liman


rameiko87@posteo.net 2023-12-09 14:16 [+0000]:
> Dear Lars-Johan Liman,
> Thank you very much for your kind and exhaustive reply. I read in the
> Gnus manual:

>> If you have no idea what to insert for "yourmachine.yourdomain.tld",
>> you’ve got several choices. You can either ask your provider if he
>> allows you to use something like yourUserName.userfqdn.provider.net,
>> or you can use somethingUnique.yourdomain.tld if you own the domain
>> yourdomain.tld, or you can register at a service which gives private
>> users a FQDN for free.

> None of these apply to me. Then I read just below:

>> Finally you can tell Gnus not to generate a Message-ID for News at
>> all (and letting the server do the job) [...] you can also tell Gnus
>> not to generate Message-IDs for mail by saying:
>> //there's some code here//
>> , however some mail servers don’t generate proper Message-IDs, too,
>> so test if your Mail Server behaves correctly by sending yourself a
>> Mail and looking at the Message-ID.

> This is perfect! So I customized message-required-mail-headers. Now
> your explanation shows why I should not remove the mail headers, but
> still I wonder about the meaning of the lines cited above from the
> Gnus manual. And I also wonder what is the meaning of the variable
> message-required-mail-headers if it's essentially ignored by Message
> when I remove the Message-ID header from the list.

> Anyway now I'm back to the original problem where I don't know what to
> insert for Message-ID because the recommendations from the Gnus manual
> above don't apply, and also I can't understand how Message generates
> the portion which comes before the at-sign...

> And by the way, the standard for me would be
> fedora.mail-host-address-is-not-set . Is this even a fully qualified
> domain name, without extension? And how would I set mail host address?
> Because I can only find the variable mail-host-address which is set to
> system-name, which is fedora. Therefore I would expect the Message-ID
> to be fsfxxxxxxx@fedora
> (which by the way I think would miss the extension). If I set
> mail-host-address manually to fedora.ext, would Message-ID then be
> fsfxxxxxxx@fedora.ext? Then why don't they say
> mail-host-extension-is-not-set rather than
> mail-host-address-is-not-set in the default Message-ID when the
> mail-host-address *is* set but has no extension? So much confusion.

> On 09.12.2023 08:03, Lars-Johan Liman wrote:
>> Hi!
>> Message-ID is one of the very few _mandatory_ header lines, so a
>> message
>> without one is not a valid RFC 5322 (the sucessor of RFC 822) message.
>> Therefore the message "must not/cannot" leave your computer without
>> one.
>> It has to be set on the computer if it's being shipped by a
>> mail-related
>> protocol ((E)SMTP/LMTP/IMAP/POP[23]/...).
>> What you _can_ do (if I remember my sendmail.cf right, it's been a
>> few
>> years since I taught that course ... ;-) ) is to _modify_ the
>> Message-ID
>> header in your mail server. "If it looks like it's coming from the
>> computer, then modify Message-ID to look like ... (bla)". But I
>> wouldn't
>> recommend it. Message-ID is supposed to be set by the sending software
>> (the Mail User Agent (MUA)) and it's supposed to uniquely identify the
>> message across all messages in the world ... ever. Modifying it would
>> make it very difficult to trace and debug how the message traverses the
>> Internet (which includes your computer). You can probably also
>> influence
>> exactly how Emacs generates the Message-ID, but again, it's probably
>> not
>> a good idea to mess with that.
>> Can I ask what problem you see with having your Emacs generate the
>> Message-ID?
>> 				Best regards,
>> 				  /Lars-Johan Liman

Re: Have message-ID set by my email server rather than by Message

[Arash Esbati]

rameiko87@posteo.net writes:

> Well, how else do you explain that when I remove Message-ID from
> message-required-mail-headers and send my email, then the Message-ID
> is given by the value of message-user-fqdn or when nil, by
> system-name?

As I already said: My guess was that your sendmail program adds a
message-id.  You can set `message-generate-headers-first' to t and see
what headers Message produces when you start editing a message, maybe
that gives you a clue.  Besides that, I have no other idea.

Best, Arash

Re: Have message-ID set by my email server rather than by Message

[Lars-Johan Liman]

FWIW, I agree with Arash's sentiment. This, again, goes back to the
notion that a message without a Message-ID is (more or less) invalid.
Therefore any and all e-mail software (user agents and transport agents
alike) is disciplined to add a Message-ID to any message without one, to
avoid transferring invalid messages.

It boils down to Postel's principle: "Be liberal in what you accept, and
consevative in what you send!". ;-)

				Cheers,
				  /Liman


rameiko87@posteo.net writes:
>> Well, how else do you explain that when I remove Message-ID from
>> message-required-mail-headers and send my email, then the Message-ID
>> is given by the value of message-user-fqdn or when nil, by
>> system-name?

arash@gnu.org 2023-12-11 07:35 [+0100]:
> As I already said: My guess was that your sendmail program adds a
> message-id.  You can set `message-generate-headers-first' to t and see
> what headers Message produces when you start editing a message, maybe
> that gives you a clue.  Besides that, I have no other idea.

> Best, Arash

Re: Have message-ID set by my email server rather than by Message

[Otto J. Makela]

On 12/11/23 11:01, Lars-Johan Liman wrote:
> FWIW, I agree with Arash's sentiment. This, again, goes back to the
> notion that a message without a Message-ID is (more or less) invalid.
> Therefore any and all e-mail software (user agents and transport agents
> alike) is disciplined to add a Message-ID to any message without one, to
> avoid transferring invalid messages.
A quite normal arrangement is to have separate MTA (message transfer agent,
for internet-incoming email) and MSA (message submission agent, for your
local email clients) services.

MSA's (which typically require user authentication) often don't enforce
message rules as strictly as MTA's which have to contend with incoming spam,
and thus can add in missing Date, Message-ID etc fields.

https://en.wikipedia.org/wiki/Message_submission_agent

-- 
    /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
   /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
  /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

Dear Liman,

> Am I right in understanding it as "my host doesn't know its own 
> hostname (identity), so it's unable to create a useful Message-ID"?

Not sure, because my original problem was that Emacs or one of the 
components which intervene *locally* generate and introduce the 
Message-ID no matter the value of message-required-mail-headers. This is 
still unresolved. The underlying intellectual conundrum I think you 
summarize well:

> "Which entity (i.e., which device) should be responsible for adding the 
> Message-ID header to a message composed on a device which has little or 
> no notion of its own host identity, and in those cases what should the 
> Message-ID be?"

My device composes the message before posteo.de transmits it. What is my 
authority to use the formula xxxx@posteo.de in the Message-ID, 
especially because the mix of self-generated xxxx and official-looking 
@posteo.de looks intellectually confusing. Think of physical post: the 
post office will stamp the branch number and timestamp the envelope when 
they accept my parcel. posteo.de are charged with transmitting my email: 
it only sounds reasonable they would add the Message-ID, which is 
essentially hostname plus timestamp. Now one could make the case for the 
added freedom which comes with deciding the Message-ID of your own 
emails: but if the host can overwrite this anyway, then the benefit of 
the potential added privacy which comes from setting Message-ID at the 
MUA level is not guaranteed, but the drawback of having possibly same 
Message-ID's for different emails (sent through the same host or even 
distinct hosts) is real.

> I happen to know the author of the mail specification RFC, and I'm 
> thinking about asking him for his thoughts. Would you object to that?

I don't object.

Re: Have message-ID set by my email server rather than by Message

[Adam Sjøgren]

One advantage of your email program setting the Message-ID is that the
unique identifier of the email is then known by your email program.


  Best regards,

    Adam

-- 
 "It was called Tinderbox so we could make the joke,        Adam Sjøgren
  'the tree is on fire'. Puns are very important in    asjo@koldfront.dk
  naming tools."

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

>> Well, how else do you explain that when I remove Message-ID from 
>> message-required-mail-headers and send my email, then the Message-ID 
>> is given by the value of message-user-fqdn or when nil, by 
>> system-name?
> 
> As I already said: My guess was that your sendmail program adds a 
> message-id.

But then why does such Message-ID always reflect the value of 
message-user-fqdn ? If it really were sendmail adding it and not 
Message, then why would sendmail set the value of the Message-ID header 
according to the value of the variable message-user-fqdn? I think that 
sendmail doesn't even know about the message-user-fqdn variable.

> You can set `message-generate-headers-first' to t and see what headers 
> Message produces when you start editing a message.

Good suggestion: I just tried. Now message-generate-headers-first is 
true, and Message-ID doesn't appear in the 
message-required-mail-headers. Result: my emails all have Message-ID 
header, and it always reflects the value of message-user-fqdn!

The mystery persists.

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

>> You can set `message-generate-headers-first' to t and see what headers 
>> Message produces when you start editing a message.
> 
> Good suggestion: I just tried. Now message-generate-headers-first is 
> true, and Message-ID doesn't appear in the 
> message-required-mail-headers. Result: my emails all have Message-ID 
> header, and it always reflects the value of message-user-fqdn!

Allow me to make a precision here; my emails *as received* have 
Message-ID, but when editing before send, the header is not there.

Re: Have message-ID set by my email server rather than by Message

[rameiko87]

I made some progress.

> My original problem was that Emacs or one of the components which 
> intervene *locally* generate and introduce the Message-ID no matter the 
> value of message-required-mail-headers.

When message-required-mail-headers excludes Message-ID and 
message-generate-headers-first is true: then Message-ID doesn't appear 
when composing the email. Predictably sendmail adds it, but I still 
cannot explain why sendmail makes it correspond to the value of 
message-user-fqdn, which clearly only mentions "Message" in its name and 
id a variable defined in ‘message.el’ . How does sendmail know about 
it?!

>> "Which entity (i.e., which device) should be responsible for adding 
>> the Message-ID header to a message composed on a device which has 
>> little or no notion of its own host identity, and in those cases what 
>> should the Message-ID be?"

> My device composes the message before posteo.de transmits it. What is 
> my authority to use the formula xxxx@posteo.de in the Message-ID, 
> especially because the mix of self-generated xxxx and official-looking 
> @posteo.de looks intellectually confusing. Think of physical post: the 
> post office will stamp the branch number and timestamp the envelope 
> when they accept my parcel. posteo.de are charged with transmitting my 
> email: it only sounds reasonable they would add the Message-ID, which 
> is essentially hostname plus timestamp. Now one could make the case for 
> the added freedom which comes with deciding the Message-ID of your own 
> emails: but if the host can overwrite this anyway, then the benefit of 
> the potential added privacy which comes from setting Message-ID at the 
> MUA level is not guaranteed, but the drawback of having possibly same 
> Message-ID's for different emails (sent through the same host or even 
> distinct hosts) is real.

(A) Ideally the host would then feed back the Message-ID together with 
confirmation that the email was accepted for delivery, and then if the 
email is being saved locally (FCC) the MUA would store the Message-ID in 
the email message. In practice, which is not ideal for the reasons 
explained above and in particular the inconvenience of storing (FCC) an 
email with Message-ID different from the one the host server might 
decide to assign by overwriting the locally-generated one, the first 
system handling the message will create the Message-ID and store the 
email (FCC) with that Message-ID as soon as the email is accepted for 
delivery. In this case the message-ID can be anything because everything 
is done locally, but I suppose would be good practice to make what comes 
before @ algorithmically generated by the MUA and what is after be the 
hostname.

To mitigate the big inconvenience of the recipient having a different 
Message-ID from the one the sender stored with the sent-email, it would 
be good if the server would confirm that it accepts the message for 
delivery AND that it will honor the Message-ID locally generated, and if 
not both conditions are verified then abort sending the email. I don't 
think this happens.

I wish that (A) were the way things worked.