application/octet-stream sent as 7bit

application/octet-stream sent as 7bit

[Dmitri Paduchikh]

Hello!

In some cases Gnus sends application/octet-stream attachments in
unencoded form:

,----
| --=-=-=
| Content-Type: application/octet-stream
| Content-Disposition: attachment; filename=test.sh
| 
| #!/bin/sh
| echo test
`----

I am concerned that contents can change when transferring a message.
Think of EOL conversions, ">From " stuff, etc. Such changes can corrupt
shell script, for example.

In general, RFC 2046 says that application/octet-stream denotes
arbitrary binary data. For arbitrary binary data I would expect that it
is transferred exactly as is, even if it looks like a 7-bit text.

-- 
Dmitri Paduchikh

Re: application/octet-stream sent as 7bit

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

>>>>> Dmitri Paduchikh wrote:
> In some cases Gnus sends application/octet-stream attachments in
> unencoded form:

> ,----
>| --=-=-=
>| Content-Type: application/octet-stream
>| Content-Disposition: attachment; filename=test.sh
>|
>| #!/bin/sh
>| echo test
> `----

> I am concerned that contents can change when transferring a message.
> Think of EOL conversions, ">From " stuff, etc. Such changes can corrupt
> shell script, for example.

> In general, RFC 2046 says that application/octet-stream denotes
> arbitrary binary data. For arbitrary binary data I would expect that it
> is transferred exactly as is, even if it looks like a 7-bit text.

Agreed.  Aren't there any other application/* types that should
not be forced to be encoded by 7bit?  Otherwise, is it better not
to force the 7bit encoding to all application/* types?

Here are two patches; one is for excluding application/octet-stream,
the other is for excluding application/*.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 787 bytes --]

--- mm-encode.el~	2009-01-22 08:09:02 +0000
+++ mm-encode.el	2009-09-03 11:24:11 +0000
@@ -148,9 +148,11 @@
 	      (mm-content-transfer-encoding mime-type)))
 	 (bits (mm-body-7-or-8)))
     ;; We force buffers that are 7bit to be unencoded, no matter
-    ;; what the preferred encoding is.
-    ;; Only if the buffers don't contain lone lines.
-    (when (and (eq bits '7bit) (not (mm-long-lines-p 76)))
+    ;; what the preferred encoding is, except that the MIME-type is
+    ;; application/octet-stream or the buffers contain long lines.
+    (when (and (eq bits '7bit)
+	       (not (string-equal mime-type "application/octet-stream"))
+	       (not (mm-long-lines-p 76)))
       (setq encoding bits))
     (mm-encode-content-transfer-encoding encoding mime-type)
     encoding))

[-- Attachment #3: Type: text/plain, Size: 1 bytes --]



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: Type: text/x-patch, Size: 776 bytes --]

--- mm-encode.el~	2009-01-22 08:09:02 +0000
+++ mm-encode.el	2009-09-03 11:24:11 +0000
@@ -148,9 +148,11 @@
 	      (mm-content-transfer-encoding mime-type)))
 	 (bits (mm-body-7-or-8)))
     ;; We force buffers that are 7bit to be unencoded, no matter
-    ;; what the preferred encoding is.
-    ;; Only if the buffers don't contain lone lines.
-    (when (and (eq bits '7bit) (not (mm-long-lines-p 76)))
+    ;; what the preferred encoding is, except that the MIME-type is
+    ;; any application/* type or the buffers contain long lines.
+    (when (and (eq bits '7bit)
+	       (not (string-match "\\`application/" mime-type))
+	       (not (mm-long-lines-p 76)))
       (setq encoding bits))
     (mm-encode-content-transfer-encoding encoding mime-type)
     encoding))

Re: application/octet-stream sent as 7bit

[Dmitri Paduchikh]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

Katsumi Yamaoka:

>>>>>> Dmitri Paduchikh wrote:
>> In some cases Gnus sends application/octet-stream attachments in
>> unencoded form:

>> ,----
>>| --=-=-=
>>| Content-Type: application/octet-stream
>>| Content-Disposition: attachment; filename=test.sh
>>|
>>| #!/bin/sh
>>| echo test
>> `----

>> I am concerned that contents can change when transferring a message.
>> Think of EOL conversions, ">From " stuff, etc. Such changes can corrupt
>> shell script, for example.

>> In general, RFC 2046 says that application/octet-stream denotes
>> arbitrary binary data. For arbitrary binary data I would expect that it
>> is transferred exactly as is, even if it looks like a 7-bit text.

> Agreed.  Aren't there any other application/* types that should
> not be forced to be encoded by 7bit?  Otherwise, is it better not
> to force the 7bit encoding to all application/* types?

Hmm... It is unclear to me whether other application/* (or audio/*,
image/*, video/*) require protection from this behavior. Although binary
data most likely isn't 7-bit, it may be 7-bit in principle. IMO, in such
case it would be preferable to provide safe default behavior and means
to change it if user wants to. 7bit enforcement breaks both of these
principles (safe default and user control) thus I would rather remove it
completely as in the patch below. Are there any situations when it is
desirable to enforce 7bit?

> Here are two patches; one is for excluding application/octet-stream,
> the other is for excluding application/*.

Thank you. The following patch is a variation of your patches.

-- 
Dmitri Paduchikh


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 717 bytes --]

diff --git a/lisp/mm-encode.el b/lisp/mm-encode.el
index 39b83ff..963936d 100644
--- a/lisp/mm-encode.el
+++ b/lisp/mm-encode.el
@@ -145,13 +145,7 @@ The encoding used is returned."
 	 (encoding
 	  (or (and (listp type)
 		   (cadr (assq 'encoding type)))
-	      (mm-content-transfer-encoding mime-type)))
-	 (bits (mm-body-7-or-8)))
-    ;; We force buffers that are 7bit to be unencoded, no matter
-    ;; what the preferred encoding is.
-    ;; Only if the buffers don't contain lone lines.
-    (when (and (eq bits '7bit) (not (mm-long-lines-p 76)))
-      (setq encoding bits))
+	      (mm-content-transfer-encoding mime-type))))
     (mm-encode-content-transfer-encoding encoding mime-type)
     encoding))
 

Re: application/octet-stream sent as 7bit

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

>>>>> Dmitri Paduchikh wrote:
[...]
>> Agreed.  Aren't there any other application/* types that should
>> not be forced to be encoded by 7bit?  Otherwise, is it better not
>> to force the 7bit encoding to all application/* types?

> Hmm... It is unclear to me whether other application/* (or audio/*,
> image/*, video/*) require protection from this behavior. Although binary
> data most likely isn't 7-bit, it may be 7-bit in principle. IMO, in such
> case it would be preferable to provide safe default behavior and means
> to change it if user wants to. 7bit enforcement breaks both of these
> principles (safe default and user control) thus I would rather remove it
> completely as in the patch below. Are there any situations when it is
> desirable to enforce 7bit?

>> Here are two patches; one is for excluding application/octet-stream,
>> the other is for excluding application/*.

> Thank you. The following patch is a variation of your patches.
[...]

I like it.  I think removing the 7bit enforcement makes it simple,
necessary and sufficient.  Even so, it can be controlled by
customizing `mm-content-transfer-encoding-defaults' if necessary
(though I don't think so).

I'd like to make one more change, which enables a user to change
the encoding by adding the `encoding=' component to a MIME tag
manually like this:

< #part type="application/octet-stream" disposition=inline encoding=7bit>

It will be useful for testing with various encodings.  The patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1991 bytes --]

--- mm-encode.el~	2009-01-22 08:09:02 +0000
+++ mm-encode.el	2009-09-04 00:28:48 +0000
@@ -137,22 +137,19 @@
    (t
     (error "Unknown encoding %s" encoding))))
 
-(defun mm-encode-buffer (type)
-  "Encode the buffer which contains data of MIME type TYPE.
+(defun mm-encode-buffer (type &optional encoding)
+  "Encode the buffer which contains data of MIME type TYPE by ENCODING.
 TYPE is a string or a list of the components.
+The optional ENCODING overrides the encoding determined according to
+TYPE and `mm-content-transfer-encoding-defaults'.
 The encoding used is returned."
-  (let* ((mime-type (if (stringp type) type (car type)))
-	 (encoding
-	  (or (and (listp type)
-		   (cadr (assq 'encoding type)))
-	      (mm-content-transfer-encoding mime-type)))
-	 (bits (mm-body-7-or-8)))
-    ;; We force buffers that are 7bit to be unencoded, no matter
-    ;; what the preferred encoding is.
-    ;; Only if the buffers don't contain lone lines.
-    (when (and (eq bits '7bit) (not (mm-long-lines-p 76)))
-      (setq encoding bits))
-    (mm-encode-content-transfer-encoding encoding mime-type)
+  (let ((mime-type (if (stringp type) type (car type))))
+    (mm-encode-content-transfer-encoding
+     (or encoding
+	 (setq encoding (or (and (listp type)
+				 (cadr (assq 'encoding type)))
+			    (mm-content-transfer-encoding mime-type))))
+     mime-type)
     encoding))
 
 (defun mm-insert-headers (type encoding &optional file)
--- mml.el~	2009-08-31 00:08:11 +0000
+++ mml.el	2009-09-04 00:28:48 +0000
@@ -585,7 +585,9 @@
 			(unless raw
 			  (setq charset	(mm-encode-body charset))))
 		    (insert contents)))))
-	      (setq encoding (mm-encode-buffer type)
+	      (if (setq encoding (cdr (assq 'encoding cont)))
+		  (setq encoding (intern (downcase encoding))))
+	      (setq encoding (mm-encode-buffer type encoding)
 		    coded (mm-string-as-multibyte (buffer-string))))
 	    (mml-insert-mime-headers cont type charset encoding nil)
 	    (insert "\n" coded))))

Re: application/octet-stream sent as 7bit

[Dmitri Paduchikh]

Katsumi Yamaoka:

> I like it. I think removing the 7bit enforcement makes it simple,
> necessary and sufficient. Even so, it can be controlled by customizing
> `mm-content-transfer-encoding-defaults' if necessary (though I don't
> think so).

> I'd like to make one more change, which enables a user to change the
> encoding by adding the `encoding=' component to a MIME tag manually
> like this:

> < #part type="application/octet-stream" disposition=inline encoding=7bit>

Thanks! It seems to work exactly as you say.

-- 
Dmitri Paduchikh

Re: application/octet-stream sent as 7bit

[Katsumi Yamaoka]

>>>>> Dmitri Paduchikh wrote:
> Katsumi Yamaoka:

>> I like it. I think removing the 7bit enforcement makes it simple,
>> necessary and sufficient. Even so, it can be controlled by customizing
>> `mm-content-transfer-encoding-defaults' if necessary (though I don't
>> think so).

>> I'd like to make one more change, which enables a user to change the
>> encoding by adding the `encoding=' component to a MIME tag manually
>> like this:

>> < #part type="application/octet-stream" disposition=inline encoding=7bit>

> Thanks! It seems to work exactly as you say.

Installed.  Thanks.