Re: Incorrect encoding with `O F' or `O f'

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Wed, May 17 2006, Katsumi Yamaoka wrote:

>>>>>> In <albqtxptoh.fsf@quant8.janestcapital.quant>
>>>>>>	Sam Steingold wrote:
>> when I receive a message with non-ascii characters and I try to save it
>> in a file using O F or O f, the "From:", "To:", "Subject:" and body are
>> not written in the encoding dictated by the destination file (in this
>> case, utf-8, because the file starts with "-*- coding: utf-8-unix -*-").
[...]
>> See http://article.gmane.org/gmane.emacs.gnus.general:61352 and its thread.
>
> I've installed the new feature that allows you to save a decoded
> article (i.e. just the text displayed in the Gnus article buffer)
> in a file.  I've done it only in the CVS trunk and disabled it
> by default, since Emacs pretest is approaching and I think it
> doesn't gain the consent widely.

Why do you think so?  The discussion in the thread cited above was
because of misunderstandings.  After I finally understood what Sam
meant with "cooked", it was clear for me that `O f' and friends are
buggy.  The previous behavior is quite useless (especially for `O b')
in the non-ASCII world, isn't it?  I think such a change is a bug fix
and should go to v5-10 as well (maybe after a some testing on the
trunk).

FYI: The patch[2] applies cleanly to v5-10 (Gnus 5.11) as well.

> Is it your taste?  If you try it, you need to set the
> `gnus-summary-save-article-coding-system' variable[1] to a
> certain universal coding system, e.g., `utf-8', `iso-2022-7bit'.
> And you will be able to save a decoded article together with a
> coding cookie in a file by using the `O F' command.
>
> I haven't implemented this feature in the other commands such as
> `O f' and `O b', because the existing file will get broken when
> you append articles to it after setting (or changing) the value
> of the `gnus-summary-save-article-coding-system' variable.

I'd like to propose the following changes:

- `O F' `O f' and `O b' should always save the decoded text.  When
  given a prefix, the should save in raw form (similar to `g'/`C-u
  g').  Or we could ask the user whether to save decoded or raw.

- Couldn't we use the standard Emacs coding system selection mechanism
  for `O f' and `O b'?  I.e. in short (from the top of my head):

  · Open the output file (using `find-file-no-select'?)

  · Append the decoded article or body to the buffer

  · Save the buffer (`save-buffer') and let the user decide the coding
    system if necessary.

Isn't this feasible?

While at it, we might want to add `O B' (overwrite functionality) as
well.

> Note that buttonized MIME parts will be lost in a saved file.

We could ask the user for confirmation in this case.

Bye, Reiner.

Footnotes: 
[2]  Included here for convenience:

--8<---------------cut here---------------start------------->8---
--- gnus-sum.el:7.143	Fri May 12 08:18:16 2006
+++ gnus-sum.el	Wed May 17 09:58:59 2006
@@ -11227,7 +11227,18 @@

 ;; Summary saving commands.

-(defun gnus-summary-save-article (&optional n not-saved)
+(defcustom gnus-summary-save-article-coding-system nil
+  "Coding system used to save a decoded article to a file.
+This is used when the `gnus-summary-write-article-file' command is run.
+The recommended coding systems include `utf-8', `iso-2022-7bit', and so
+forth.  Note that buttonized MIME parts will be lost in a saved file.
+If it is nil, raw articles will be saved."
+  :type '(choice :format "%{%t%}:\n %[Value Menu%] %v"
+		 (const :tag "Save raw articles")
+		 (symbol :tag "Coding system"))
+  :group 'gnus-article-saving)
+
+(defun gnus-summary-save-article (&optional n not-saved decode)
   "Save the current article using the default saver function.
 If N is a positive number, save the N next articles.
 If N is a negative number, save the N previous articles.
@@ -11235,6 +11246,8 @@
 save those articles instead.
 The variable `gnus-default-article-saver' specifies the saver function."
   (interactive "P")
+  (unless gnus-summary-save-article-coding-system
+    (setq decode nil))
   (let* ((articles (gnus-summary-work-articles n))
 	 (save-buffer (save-excursion
 			(nnheader-set-temp-buffer " *Gnus Save*")))
@@ -11249,14 +11262,26 @@
 	    (gnus-message 1 "Article %d is unsaveable" article))
 	;; This is a real article.
 	(save-window-excursion
-	  (let ((gnus-display-mime-function nil)
-		(gnus-article-prepare-hook nil))
+	  (let ((gnus-display-mime-function (when decode
+					      gnus-display-mime-function))
+		(gnus-article-prepare-hook (when decode
+					     gnus-article-prepare-hook)))
 	    (gnus-summary-select-article t nil nil article)))
 	(save-excursion
 	  (set-buffer save-buffer)
 	  (erase-buffer)
-	  (insert-buffer-substring gnus-original-article-buffer))
-	(setq file (gnus-article-save save-buffer file num))
+	  (if decode
+	      (progn
+		(insert "X-Coding-System: -*- coding: "
+			(symbol-name gnus-summary-save-article-coding-system)
+			"; -*-\n")
+		(insert-buffer-substring gnus-article-buffer))
+	    (insert-buffer-substring gnus-original-article-buffer)))
+	(let ((mm-text-coding-system-for-write
+	       (if decode
+		   gnus-summary-save-article-coding-system
+		 mm-text-coding-system-for-write)))
+	  (setq file (gnus-article-save save-buffer file num)))
 	(gnus-summary-remove-process-mark article)
 	(unless not-saved
 	  (gnus-summary-set-saved-mark article))))
@@ -11324,7 +11349,7 @@
   (interactive "P")
   (require 'gnus-art)
   (let ((gnus-default-article-saver 'gnus-summary-write-to-file))
-    (gnus-summary-save-article arg)))
+    (gnus-summary-save-article arg nil t)))

 (defun gnus-summary-save-article-body-file (&optional arg)
   "Append the current article body to a file.
--8<---------------cut here---------------end--------------->8---
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 15:41:25 +0200]:
>
>   · Append the decoded article or body to the buffer

please note that the headers can also contain non-ASCII (Subject and
full name in To/From/Cc)

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://camera.org http://mideasttruth.com http://palestinefacts.org
http://thereligionofpeace.com http://honestreporting.com http://memri.org
I don't like cats! -- Come on, you just don't know how to cook them!

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Thu, May 18 2006, Sam Steingold wrote:

>> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 15:41:25 +0200]:
>>
>>   · Append the decoded article or body to the buffer
>
> please note that the headers can also contain non-ASCII (Subject and
> full name in To/From/Cc)

Katsumi's patch also decodes the headers.

When I wrote "article" it includes headers and body; I mentioned
"body" separately because of `O b' (gnus-summary-save-article-body-file).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 15:41:25 +0200]:
>
> FYI: The patch[2] applies cleanly to v5-10 (Gnus 5.11) as well.

I applied the patch to gnus-sym.el as found in emacs cvs head, compiled,
loaded it, customized gnus-summary-save-article-coding-system to utf-8,
and tried to save an article using C-o and o.
(gnus-default-article-saver is gnus-summary-save-in-file)

I do not observe any changes in the results.

specifically, if headers say

Content-Type: text/plain; charset="utf-8"; format="flowed"
Content-Transfer-Encoding: 8bit

then I see the body correctly (I.e., I can read it, I get what I want)
but the From and Subject headers look like this:
Subject: Re: =?utf-8?Q?=D1=83=D0=B2=D1=8B_=D1=81?=
 =?utf-8?Q?_?=
 =?utf-8?Q?=D0=B0=D0=BD=D0=B3=D0=BB=D0=B8=D0=B9=D1=81=D0=BA=D0=B8=D0=BC?=
 =?utf-8?Q?_=D0=BD=D0=B5_=D0=B7=D0=BD=D0=B0=D0=BA=D0=BE=D0=BC.?=

if the headers are

Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64

then the body is saved as base64 and the headers look like this:
Subject: Re: =?utf-8?B?0YPQstGLINGBINCw0L3Qs9C70LjQudGB0LrQuNC8INC90LUg?=
 =?utf-8?B?0LfQvdCw0LrQvtC8Lg==?=


-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://honestreporting.com http://ffii.org http://pmw.org.il http://dhimmi.com
http://jihadwatch.org http://iris.org.il http://thereligionofpeace.com
All extremists should be taken out and shot.

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Thu, May 18 2006, Sam Steingold wrote:

>> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 15:41:25 +0200]:
>>
>> FYI: The patch[2] applies cleanly to v5-10 (Gnus 5.11) as well.
>
> I applied the patch to gnus-sym.el as found in emacs cvs head, compiled,
> loaded it, customized gnus-summary-save-article-coding-system to utf-8,
> and tried to save an article using C-o and o.
> (gnus-default-article-saver is gnus-summary-save-in-file)
>
> I do not observe any changes in the results.

Try `o' with...
  (setq gnus-default-article-saver 'gnus-summary-write-article-file)
or try `O F' (gnus-summary-write-article-file) directly.

Anything beyond that (e.g. `O f', `O b') isn't implemented in
Katsumi's patch.

As `C-o' is documented to save in mail box format (mbox) it *must not*
decode anything.  I'll try to clarify this in the doc string and in
(info "(gnus)Saving Articles").

(See my proposal in <news:v964k34fi2.fsf@marauder.physik.uni-ulm.de>.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 21:19:19 +0200]:
>
> Try `o' with...
>   (setq gnus-default-article-saver 'gnus-summary-write-article-file)
> or try `O F' (gnus-summary-write-article-file) directly.

this is no good - this overwrites the file, not appends to it.

also, when I select (using '#') several articles and hit O F, I observe
the following absurd behavior:
I am asked 12(!) times where I want to save the articles:
"save 12 articles to ...."
and the message after each save is "appended to ..."
and finally the file contains just one (last) article.

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://iris.org.il http://honestreporting.com http://truepeace.org
http://dhimmi.com http://memri.org http://mideasttruth.com http://pmw.org.il
Bill Gates is not god and Microsoft is not heaven.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <v964k34fi2.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> On Wed, May 17 2006, Katsumi Yamaoka wrote:

>> I've installed the new feature that allows you to save a decoded
>> article (i.e. just the text displayed in the Gnus article buffer)
>> in a file.  I've done it only in the CVS trunk and disabled it
>> by default, since Emacs pretest is approaching and I think it
>> doesn't gain the consent widely.

> Why do you think so?

I assume that such files are used as mails by other maillers or
equivalents and I believe mails can be read correctly by only
mail softwares.  For instance, a file which is saved by the `O F'
command can be incorporated into a MH folder, and a file which
is saved by the `O f' command can be read by mail readers which
support the Unix mbox format.  If those files are decoded and
re-encoded by a certain coding system, I guess that they cannot
be handled by mail softwares generally.  Moreover, there seems
to be no way to have MIME attachments in such files.  From this
point of view, there's no difference between raw mail file and
MS Word document.

However, if the main purpose to use those files is to glance, to
print on papers, or to project on a big screen in a conference
room, I'm completely mistaken.  I made the change in the trunk
assuming such a purpose, though.

> The discussion in the thread cited above was
> because of misunderstandings.  After I finally understood what Sam
> meant with "cooked", it was clear for me that `O f' and friends are
> buggy.  The previous behavior is quite useless (especially for `O b')
> in the non-ASCII world, isn't it?

It is hard to me to imagine the purpose of using files which are
saved by the `O b' command, regardless of whether they contain
non-ASCII text or not. ;-)

> I think such a change is a bug fix
> and should go to v5-10 as well (maybe after a some testing on the
> trunk).

Hmm...  I personally don't oppose to do that (since those commands
are needless so far to me).

> FYI: The patch[2] applies cleanly to v5-10 (Gnus 5.11) as well.

>> Is it your taste?  If you try it, you need to set the
>> `gnus-summary-save-article-coding-system' variable[1] to a
>> certain universal coding system, e.g., `utf-8', `iso-2022-7bit'.
>> And you will be able to save a decoded article together with a
>> coding cookie in a file by using the `O F' command.
>>
>> I haven't implemented this feature in the other commands such as
>> `O f' and `O b', because the existing file will get broken when
>> you append articles to it after setting (or changing) the value
>> of the `gnus-summary-save-article-coding-system' variable.

> I'd like to propose the following changes:

> - `O F' `O f' and `O b' should always save the decoded text.  When
>   given a prefix, the should save in raw form (similar to `g'/`C-u
>   g').  Or we could ask the user whether to save decoded or raw.

> - Couldn't we use the standard Emacs coding system selection mechanism
>   for `O f' and `O b'?  I.e. in short (from the top of my head):

>   · Open the output file (using `find-file-no-select'?)

>   · Append the decoded article or body to the buffer

>   · Save the buffer (`save-buffer') and let the user decide the coding
>     system if necessary.

> Isn't this feasible?

Yes, it seems to be successful if a suitable coding system is
used and Emacs automatically detects it correctly when reading a
file (it is doubtful that XEmacs-Mule does always do the right
thing if there's no coding cookie, though).  The coding system
to use should be a universal one, such as utf-8 and iso-2022-7bit.
Otherwise, Emacs will try to change it if characters that cannot
be encoded by it are appended.

> While at it, we might want to add `O B' (overwrite functionality) as
> well.

>> Note that buttonized MIME parts will be lost in a saved file.

> We could ask the user for confirmation in this case.

That would be nice.

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Fri, May 19 2006, Katsumi Yamaoka wrote:

>>>>>> In <v964k34fi2.fsf@marauder.physik.uni-ulm.de>
>>>>>>	Reiner Steib wrote:
>
>> On Wed, May 17 2006, Katsumi Yamaoka wrote:
[...]
> I assume that such files are used as mails by other maillers or
> equivalents and I believe mails can be read correctly by only mail
> softwares.

I think this is what the other `O ...' commands are for (see below).

> For instance, a file which is saved by the `O F' command can be
> incorporated into a MH folder,

,----[ (info "(gnus)Saving Articles") ]
| `O h'
|      Save the current article in mh folder format
|      (`gnus-summary-save-article-folder').
`----

> and a file which is saved by the `O f' command can be read by mail
> readers which support the Unix mbox format.  

,----[ (info "(gnus)Saving Articles") ]
| `O m'
|      Save the current article in mail format
|      (`gnus-summary-save-article-mail').
`----

Note that `O f'/`O F' don't produce a "From " line which is absolutely
necessary for correct mbox format.

> If those files are decoded and re-encoded by a certain coding
> system, I guess that they cannot be handled by mail softwares
> generally.  

I don't think that this is the purpose of `O f'/`O F'.

> Moreover, there seems to be no way to have MIME attachments in such
> files.
[ quoting reordered ]
>>> Note that buttonized MIME parts will be lost in a saved file.
>> We could ask the user for confirmation in this case.
> That would be nice.

While a warning is a good thing, I think that the user expects kinda
plain-text output for `O f'/`O F'.  But maybe I'm wrong.

> It is hard to me to imagine the purpose of using files which are
> saved by the `O b' command, regardless of whether they contain
> non-ASCII text or not. ;-)

I neither used `O f'/`O F'/`O b' in ages.  Sam, could you give us some
examples how/why you use `O f'?  Do other people use them?  How/why do
you use them?

[...]
>>   · Open the output file (using `find-file-no-select'?)
>
>>   · Append the decoded article or body to the buffer
>
>>   · Save the buffer (`save-buffer') and let the user decide the coding
>>     system if necessary.
>
>> Isn't this feasible?
>
> Yes, it seems to be successful if a suitable coding system is
> used and Emacs automatically detects it correctly when reading a
> file (it is doubtful that XEmacs-Mule does always do the right
> thing if there's no coding cookie, though).  

I don't we should not refrain from implementing a feature just because
of an incapability of XEmacs.  If we can make it work with XEmacs:
Good.  If not, let's disable it for XEmacs (by default).

[ Even XEmacs 21.5 seems to be broken WRT: Open a buffer with some
Latin-9 chars "ä ö ü" (with LC_CTYPE=de_DE@euro), add an UTF-8
character like "↑", save the buffer:  XEmacs 21.5 write some weird
Latin-9/iso-2022-7bit mixture to the file. ]

> The coding system to use should be a universal one, such as utf-8
> and iso-2022-7bit.  Otherwise, Emacs will try to change it if
> characters that cannot be encoded by it are appended.

That happens for plain text files as well or when the user saves some
text from an e-mail by C&P into some text file.  I don't think that we
should be extra-careful here *if* we don't by-pass the usual Emacs
mechanisms.  For single-message output (O F), we should just try the
default coding (from the Emacs settings) unless the user specifies
`gnus-summary-save-article-coding-system'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2006-05-19 14:08:46 +0200]:
>
> Sam, could you give us some examples how/why you use `O f'?

I use it to save e-mail for future reference.
I want to be able to view the mail both using gnus (or /bin/mail) and
vi/emacs for full text search across messages.
it is more important to me to be able to read the old mail using
vi/emacs than using gnus/mail, i.e., if for some unfathomable reason you
deem it necessary to make gnus unable to read utf-8 files, I would still
stick with utf-8 (as opposed to base64).

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://ffii.org http://palestinefacts.org http://honestreporting.com
http://iris.org.il http://truepeace.org http://thereligionofpeace.com
Save your burned out bulbs for me, I'm building my own dark room.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <v93bf65i9d.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> Note that `O f'/`O F' don't produce a "From " line which is absolutely
> necessary for correct mbox format.

Oops.  I overlooked the main point.

>> If those files are decoded and re-encoded by a certain coding
>> system, I guess that they cannot be handled by mail softwares
>> generally.

> I don't think that this is the purpose of `O f'/`O F'.

You're right.

>> Moreover, there seems to be no way to have MIME attachments in such
>> files.
> [ quoting reordered ]
>>>> Note that buttonized MIME parts will be lost in a saved file.
>>> We could ask the user for confirmation in this case.
>> That would be nice.

> While a warning is a good thing, I think that the user expects kinda
> plain-text output for `O f'/`O F'.  But maybe I'm wrong.

Yes, now I don't think they need to warn about MIME parts.

>> It is hard to me to imagine the purpose of using files which are
>> saved by the `O b' command, regardless of whether they contain
>> non-ASCII text or not. ;-)

> I neither used `O f'/`O F'/`O b' in ages.  Sam, could you give us some
> examples how/why you use `O f'?  Do other people use them?  How/why do
> you use them?

>>>>> In <alk68inoem.fsf@quant8.janestcapital.quant>
>>>>>	Sam Steingold wrote:

> I use it to save e-mail for future reference.
[...]

I use some nnml groups to which I copy very important articles
for future reference.  (OT: The tick mark is not very useful to
me because there're many ticked articles, especially in the ding
group.  I wish the tick marks could have level, !, !!, !!!...)

>>>   · Open the output file (using `find-file-no-select'?)
>>>   · Append the decoded article or body to the buffer
>>>   · Save the buffer (`save-buffer') and let the user decide the coding
>>>     system if necessary.
>>
>>> Isn't this feasible?
>>
>> Yes, it seems to be successful if a suitable coding system is
>> used and Emacs automatically detects it correctly when reading a
>> file (it is doubtful that XEmacs-Mule does always do the right
>> thing if there's no coding cookie, though).

> I don't we should not refrain from implementing a feature just because
> of an incapability of XEmacs.  If we can make it work with XEmacs:
> Good.  If not, let's disable it for XEmacs (by default).

> [ Even XEmacs 21.5 seems to be broken WRT: Open a buffer with some
> Latin-9 chars "ä ö ü" (with LC_CTYPE=de_DE@euro), add an UTF-8
> character like "↑", save the buffer:  XEmacs 21.5 write some weird
> Latin-9/iso-2022-7bit mixture to the file. ]

Oh, that's too bad.  I agree not to make Gnus positively help
such a buggy system.

>> The coding system to use should be a universal one, such as utf-8
>> and iso-2022-7bit.  Otherwise, Emacs will try to change it if
>> characters that cannot be encoded by it are appended.

> That happens for plain text files as well or when the user saves some
> text from an e-mail by C&P into some text file.  I don't think that we
> should be extra-careful here *if* we don't by-pass the usual Emacs
> mechanisms.  For single-message output (O F), we should just try the
> default coding (from the Emacs settings) unless the user specifies
> `gnus-summary-save-article-coding-system'.

I agree.  The Japanese locale leads the default coding system to
be `euc-jp' or `shift_jis' in old systems[1] which is not
suitable to some characters (e.g., those of jisx0213 charsets)
that Windows people in Japan sometimes use.  Even so, I realized
it would not be a serious problem that Emacs tries to change it.

[1] Recent systems use `utf-8' by default, though.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

I've started coding to make `O b', `O f', and `O F' always save
decoded articles.

>>>>> In <b4mbqtqnbu3.fsf@jpl.org> Katsumi Yamaoka wrote:

[...]

>>> The coding system to use should be a universal one, such as utf-8
>>> and iso-2022-7bit.  Otherwise, Emacs will try to change it if
>>> characters that cannot be encoded by it are appended.

>> That happens for plain text files as well or when the user saves some
>> text from an e-mail by C&P into some text file.  I don't think that we
>> should be extra-careful here *if* we don't by-pass the usual Emacs
>> mechanisms.  For single-message output (O F), we should just try the
>> default coding (from the Emacs settings) unless the user specifies
>> `gnus-summary-save-article-coding-system'.

> I agree.  The Japanese locale leads the default coding system to
> be `euc-jp' or `shift_jis' in old systems[1] which is not
> suitable to some characters (e.g., those of jisx0213 charsets)
> that Windows people in Japan sometimes use.  Even so, I realized
> it would not be a serious problem that Emacs tries to change it.

> [1] Recent systems use `utf-8' by default, though.

I changed my idea.  After some trials, I came to the conclusion
that it is better and safe to specify a universal coding system
such as `utf-8'.  Even if Emacs always correctly detect the
coding system when reading a file in which there is no coding
cookie (I doubt it, though), it is annoying that Emacs will
change the coding system when text that cannot be encoded by the
present value is appended to a file.  In addition, to ignore
XEmacs(+Mule) users doesn't seem to be a good idea.  Therefore,
I think having the coding cookie in files is necessary.  XEmacs
ignores the coding cookie which is placed as the local variables
section in the bottom of a file if the file size is larger than
3000 bytes (see files.el), so it should be placed in the top of
a file, and I'm planning to add the one like the following even
to files generated by the `O b' command:

"X-Gnus-Coding-System: -*- coding: utf-8; -*-\n\n"

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Thu, May 18 2006, Sam Steingold wrote:

>> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 21:19:19 +0200]:
>>
>> Try `o' with...
>>   (setq gnus-default-article-saver 'gnus-summary-write-article-file)
>> or try `O F' (gnus-summary-write-article-file) directly.
>
> this is no good - this overwrites the file, not appends to it.

Sure, but the new functionality was only implemented for this command.

> also, when I select (using '#') several articles and hit O F, I observe
> the following absurd behavior:
> I am asked 12(!) times where I want to save the articles:
> "save 12 articles to ...."

,----[ (info "(gnus)Saving Articles") ]
|    All these commands use the process/prefix convention (*note
| Process/Prefix::).  If you save bunches of articles using these
| functions, you might get tired of being prompted for files to save each
| and every article in.  The prompting action is controlled by the
| `gnus-prompt-before-saving' variable, which is `always' by default,
| giving you that excessive prompting action you know and loathe.  If you
| set this variable to `t' instead, you'll be prompted just once for each
| series of articles you save.  If you like to really have Gnus do all
| your thinking for you, you can even set this variable to `nil', which
| means that you will never be prompted for files to save articles in.
| Gnus will simply save all the articles in the default files.
`----

> and the message after each save is "appended to ..."
> and finally the file contains just one (last) article.

Yes, we should fix this.  It's because all saving-commands finally use
`mm-append-to-file' where "Appended to" is hard-coded.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <v964jxm12p.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

[...]

>> also, when I select (using '#') several articles and hit O F, I observe
>> the following absurd behavior:
>> I am asked 12(!) times where I want to save the articles:
>> "save 12 articles to ...."

[...]

>> and the message after each save is "appended to ..."
>> and finally the file contains just one (last) article.

> Yes, we should fix this.

Maybe we have to use the `o f' functionality if several articles
are given to `O F'.

> It's because all saving-commands finally use `mm-append-to-file'
> where "Appended to" is hard-coded.

I'm planning not to use `mm-append-to-file'.

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Tue, May 23 2006, Katsumi Yamaoka wrote:

>>>>>> In <v964jxm12p.fsf@marauder.physik.uni-ulm.de>
>>>>>>	Reiner Steib wrote:
>>> and the message after each save is "appended to ..."
>>> and finally the file contains just one (last) article.
>
>> Yes, we should fix this.
>
> Maybe we have to use the `o f' functionality if several articles
> are given to `O F'.

You probably meant "use the `O f' functionality".

>> It's because all saving-commands finally use `mm-append-to-file'
>> where "Appended to" is hard-coded.
>
> I'm planning not to use `mm-append-to-file'.

Good.  I already expected this. :-)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2006-05-23 13:29:18 +0200]:
>
> On Thu, May 18 2006, Sam Steingold wrote:
>
>>> * Reiner Steib <ervarefgrvo+sebz-hpr@vznc.pp> [2006-05-18 21:19:19 +0200]:
>>>
>>> Try `o' with...
>>>   (setq gnus-default-article-saver 'gnus-summary-write-article-file)
>>> or try `O F' (gnus-summary-write-article-file) directly.
>>
>> this is no good - this overwrites the file, not appends to it.
>
> Sure, but the new functionality was only implemented for this command.

any chance it will be implemented for other commands too?

>> also, when I select (using '#') several articles and hit O F, I observe
>> the following absurd behavior:
>> I am asked 12(!) times where I want to save the articles:
>> "save 12 articles to ...."
>
> ,----[ (info "(gnus)Saving Articles") ]
> |    All these commands use the process/prefix convention (*note
> | Process/Prefix::).  If you save bunches of articles using these
> | functions, you might get tired of being prompted for files to save each
> | and every article in.  The prompting action is controlled by the
> | `gnus-prompt-before-saving' variable, which is `always' by default,
> | giving you that excessive prompting action you know and loathe.  If you
> | set this variable to `t' instead, you'll be prompted just once for each
> | series of articles you save.  If you like to really have Gnus do all
> | your thinking for you, you can even set this variable to `nil', which
> | means that you will never be prompted for files to save articles in.
> | Gnus will simply save all the articles in the default files.
> `----

I have
  (custom-set-variables '((gnus-prompt-before-saving t)))
in ~/.emacs


-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://jihadwatch.org http://dhimmi.com http://camera.org
http://honestreporting.com http://mideasttruth.com http://memri.org
God had a deadline, so He wrote it all in Lisp.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <b4mu07h9h8s.fsf@jpl.org> Katsumi Yamaoka wrote:

> I've started coding to make `O b', `O f', and `O F' always save
> decoded articles.

I've done in the trunk.  Could you verify those commands work
ideally?

I've moved the `gnus-summary-save-article-coding-system' variable
to gnus-art.el, renamed to `gnus-article-save-coding-system',
and made it default to `utf-8' (or the others).  Now the commands
to save decoded articles add the coding cookie to a file.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <v9wtcclxfq.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> On Tue, May 23 2006, Katsumi Yamaoka wrote:

>> Maybe we have to use the `o f' functionality if several articles
>> are given to `O F'.

> You probably meant "use the `O f' functionality".

Yup. ;-p

Unless `gnus-prompt-before-saving' is set to `always', now the
`O F' command behaves as is for the first article and behaves
like `O f' thereafter.  I didn't make it do so when
`gnus-prompt-before-saving' is `always', since a user might want
to specify file names separately to each article.  In that case,
every file should be newly created (i.e., existing files should
be deleted in advance), so the `O f' function which appends an
article to an existing file cannot be used.  However, this might
confuse people since Gnus always suggests the same default file
name for all articles.  Isn't there a better way?

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <b4mirnvha9r.fsf@jpl.org> Katsumi Yamaoka wrote:

> Unless `gnus-prompt-before-saving' is set to `always', now the
> `O F' command behaves as is for the first article and behaves
> like `O f' thereafter.  I didn't make it do so when
> `gnus-prompt-before-saving' is `always', since a user might want
> to specify file names separately to each article.  In that case,
> every file should be newly created (i.e., existing files should
> be deleted in advance), so the `O f' function which appends an
> article to an existing file cannot be used.  However, this might
> confuse people since Gnus always suggests the same default file
> name for all articles.  Isn't there a better way?

I seem to have solved it.  Please try the CVS trunk.

In addition, I've added `O B' command and fixed the `o' command
when `gnus-default-article-saver' is set to
`gnus-summary-save-in-file', `gnus-summary-save-body-in-file',
`gnus-summary-write-to-file', or `gnus-summary-write-body-to-file'
so that it might save decoded articles properly.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

I attached a patch for the latest Gnus v5.11 below.

>>>>> In <b4mslmyy1si.fsf@jpl.org> Katsumi Yamaoka wrote:

>> Unless `gnus-prompt-before-saving' is set to `always', now the
>> `O F' command behaves as is for the first article and behaves
>> like `O f' thereafter.

Well, it might be funny that the `O F' command behaves differently
according to the value of `gnus-prompt-before-saving'.  So does
the `O B' command.  WDYT?  When saving two or more articles
which are marked at a time, Gnus does:

always:
  Prompt you for a file name each time an article is saved.
  Each article is saved separately.  (If you enter the file name
  which has been saved previously, it will be overwritten.)

t:
  Prompt you for a file name only once.  All articles are saved
  in that file.

nil:
  Never prompt you for file names.  Each article is saved
  separately with a default name.

Isn't it better to make it work as if `gnus-prompt-before-saving'
is `always' even if it is t?


[-- Attachment #2: v5_11.patch.gz --]
[-- Type: application/x-gzip, Size: 4200 bytes --]

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <b4my7wpgqik.fsf@jpl.org> Katsumi Yamaoka wrote:

>>>>>> In <b4mslmyy1si.fsf@jpl.org> Katsumi Yamaoka wrote:

>>> Unless `gnus-prompt-before-saving' is set to `always', now the
>>> `O F' command behaves as is for the first article and behaves
>>> like `O f' thereafter.

> Well, it might be funny that the `O F' command behaves differently
> according to the value of `gnus-prompt-before-saving'.  So does
> the `O B' command.  WDYT?

I've made it customizable.  Now the default behavior of the `O F'
command and the `O B' command is to save all articles in a single
file when saving many articles at a time, and this is mentioned
in the docstring of `gnus-default-article-saver' as follows:

--8<---------------cut here---------------start------------->8---
gnus-default-article-saver is a variable defined in `gnus-art.elc'.

[...]

Gnus provides the following functions:

[...]

The symbol of each function may have the following properties:

* :decode
The value non-nil means save decoded articles.  This is meaningful
only with `gnus-summary-save-in-file', `gnus-summary-save-body-in-file',
`gnus-summary-write-to-file', and `gnus-summary-write-body-to-file'.

* :function
The value specifies an alternative function which appends, not
overwrites, articles to a file.  This implies that when saving many
articles at a time, `gnus-prompt-before-saving' is bound to t and all
articles are saved in a single file.  This is meaningful only with
`gnus-summary-write-to-file' and `gnus-summary-write-body-to-file'.

* :headers
The value specifies the symbol of a variable of which the value
specifies headers to be saved.  If it is omitted,
`gnus-save-all-headers' and `gnus-saved-headers' control what
headers should be saved.
--8<---------------cut here---------------end--------------->8---

I will add it to the Info manual and merge all those changes to
the v5-10 branch later.

> When saving two or more articles which are marked at a time, Gnus
> does:

> always:
>   Prompt you for a file name each time an article is saved.
>   Each article is saved separately.  (If you enter the file name
>   which has been saved previously, it will be overwritten.)

> t:
>   Prompt you for a file name only once.  All articles are saved
>   in that file.

> nil:
>   Never prompt you for file names.  Each article is saved
>   separately with a default name.

> Isn't it better to make it work as if `gnus-prompt-before-saving'
> is `always' even if it is t?

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

just a clarification: the RFE that stared this thread was to be able to
_append_ the article (with) the headers to a file that already contains
articles in the same encoding.
I use emacs cvs head, not gnus cvs, so I cannot test your changes.

thanks.

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://pmw.org.il http://memri.org http://palestinefacts.org
http://ffii.org http://openvotingconsortium.org http://iris.org.il
There is an exception to every rule, including this one.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

>>>>> In <m3y7wktr1i.fsf@loiso.podval.org> Sam Steingold wrote:

> just a clarification: the RFE that stared this thread was to be able to
> _append_ the article (with) the headers to a file that already contains
> articles in the same encoding.
> I use emacs cvs head, not gnus cvs, so I cannot test your changes.

> thanks.

Here's a patch to the latest Emacs CVS, though the changes I made
in the v5-10 branch will be merged into Emacs sooner or later.
Please see the docstring of `gnus-default-article-saver'.


[-- Attachment #2: v5_11.patch.gz --]
[-- Type: application/x-gzip, Size: 4413 bytes --]

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Katsumi Yamaoka <lnznbxn@wcy.bet> [2006-05-30 07:34:29 +0900]:
>
>>>>>> In <m3y7wktr1i.fsf@loiso.podval.org> Sam Steingold wrote:
>
>> just a clarification: the RFE that stared this thread was to be able to
>> _append_ the article (with) the headers to a file that already contains
>> articles in the same encoding.
>> I use emacs cvs head, not gnus cvs, so I cannot test your changes.
>
>> thanks.
>
> Here's a patch to the latest Emacs CVS, though the changes I made
> in the v5-10 branch will be merged into Emacs sooner or later.
> Please see the docstring of `gnus-default-article-saver'.

yes, it now "works", i.e., "o" saves the decoded article with some headers.
except for the "^From " separator, i.e., my files will not be accessible
with mail readers.  Too bad [1].

alas, this patch breaks followup:

Debugger entered--Lisp error: (error "Unknown symbol: \x5143e")
  signal(error ("Unknown symbol: \x5143e"))
  error("Unknown symbol: %c" 332862)
  mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")
  sds-message-citation-line-function()
  message-cite-original()
  message-yank-original()
  gnus-inews-yank-articles((190))
  gnus-post-news(nil "nnimap+mail.podval.org:misc/osya" [190 "Re: \x51451\x51460\x51450\x51462\x51458\x5145a" "\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>" #("Tue, 30 May 2006 23:53:56 +0400" 0 1 (gnus-time ...) 1 31 nil) "<937285080.1149018836.147101336.7177@mcgi12.rambler.ru>" "<allksjy0h5.fsf@quant8.janestcapital.quant> <126260569.1149016674.173266644.7036@mcgi12.rambler.ru> <alac8zwbqa.fsf@quant8.janestcapital.quant>" 1305 4 "quant8.janestcapital.quant misc/osya:190" ((To . "sds@podval.org"))] "*Article*" (190) nil nil)
  gnus-summary-followup((190) nil)
  gnus-summary-followup-with-original(nil)
  call-interactively(gnus-summary-followup-with-original)






[1] I know you are going to tell me that mail readers are not supposed
    to see non-ASCII &c.  I don't care.  There is no technical reason
    for that in the modern unicode world - so you can quote RFCs all you
    like, you are still wrong.

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://mideasttruth.com http://ffii.org http://camera.org
http://jihadwatch.org http://truepeace.org http://openvotingconsortium.org
Experience comes with debts.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <alwtc3uu2k.fsf@quant8.janestcapital.quant>
>>>>>	Sam Steingold wrote:

>> * Katsumi Yamaoka <yamaoka@jpl.org> [2006-05-30 07:34:29 +0900]:

>> Here's a patch to the latest Emacs CVS, though the changes I made
>> in the v5-10 branch will be merged into Emacs sooner or later.
>> Please see the docstring of `gnus-default-article-saver'.

> yes, it now "works", i.e., "o" saves the decoded article with some headers.

You've set `gnus-default-article-saver' to
`gnus-summary-save-in-file' or `gnus-summary-write-to-file',
haven't you?  They (and also `gnus-summary-save-body-in-file' and
`gnus-summary-write-body-to-file') have been designed for saving
just text in files, as far as I understand.  Since it is more
than probable that there are messages written in many languages
in those files, I made the functions encode decoded text using a
universal coding system (see `gnus-article-save-coding-system'),
which can safely encode any characters in text.  Actually, it is
easy to me to make a file containing messages written in
Chinese, Deutsch, Francais, Japanese, Korean, Russian, Ukrainian...

There is MIME which is for handling those things.  However, a raw
MIME message is not the one that human reads.  For instance, you
will be unable to read such a file using text readers, such as
`cat', `less', `find-file', etc.  The measures by which you can
read such a *mail* are only mail readers.  On the other hand, the
functions `g-s-save-in-file', `g-s-save-body-in-file',
`g-s-write-to-file', and `g-s-write-body-to-file' are for saving
messages in files, which can be read using text readers.  So, if
you want to read files using mail readers, you have to choose a
proper function including `gnus-summary-save-in-rmail',
gnus-summary-save-in-mail', `gnus-summary-save-in-folde',
`gnus-summary-save-in-vm', and a custom function you made.  Note
that files made by those functions are incompatible with *text*
readers.

> except for the "^From " separator, i.e., my files will not be accessible
> with mail readers.  Too bad [1].

The "From " lines are necessary to a certain mail reader, and the
function that adds them is `gnus-summary-save-in-mail'.  Note
that the function should save raw messages since a mail reader
assumes they are.

> alas, this patch breaks followup:

> Debugger entered--Lisp error: (error "Unknown symbol: \x5143e")
>   signal(error ("Unknown symbol: \x5143e"))
>   error("Unknown symbol: %c" 332862)
>   mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")

This error suggests *mail* messages don't allow non-ASCII text in
at least headers.  In other words, files should contain raw mail
messages.

> [1] I know you are going to tell me that mail readers are not supposed
>     to see non-ASCII &c.  I don't care.  There is no technical reason
>     for that in the modern unicode world - so you can quote RFCs all you
>     like, you are still wrong.

Could you please show an alternative plan concretely if you have?

Though Gnus is going pretty good now, we Japanese (and possibly
people in non-ASCII worlds) were troubled with many programs that
didn't work with non-ASCII text.  To tell the truth, I think it
has not essentially been improved yet even in the unicode world.
An ultimate solution might be to make a common standard to
exchange all information.  For instance, if mail messages allow
raw unicode-encoded text in every part, we can read them by any
program.

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Katsumi Yamaoka <lnznbxn@wcy.bet> [2006-05-31 13:47:44 +0900]:
>
>> alas, this patch breaks followup:
>
>> Debugger entered--Lisp error: (error "Unknown symbol: \x5143e")
>>   signal(error ("Unknown symbol: \x5143e"))
>>   error("Unknown symbol: %c" 332862)
>>   mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")
>
> This error suggests *mail* messages don't allow non-ASCII text in
> at least headers.  In other words, files should contain raw mail
> messages.

this is not the case.
without your patch followup worked fine, and there was non-ASCII
characters in the headers.

>> [1] I know you are going to tell me that mail readers are not supposed
>>     to see non-ASCII &c.  I don't care.  There is no technical reason
>>     for that in the modern unicode world - so you can quote RFCs all you
>>     like, you are still wrong.
>
> Could you please show an alternative plan concretely if you have?

A character is just a character - an abstract object.
A file or a message is a sequence of characters (converted from bytes
according to a global variable gnus-article-save-coding-system).
Note that network transmission (which may mistreat non-ASCII) and
interaction with other users (who may use a different
gnus-article-save-coding-system) are completely irrelevant here.
All that matters is that _my_ files on _my_ HD on _my_ computer are
accessible for _me_ in a uniform way, both ASCII and non-ASCII.
So, all that you need to do is write and parse files in the same coding
system and the parse them (for nnfolder) as a sequence of characters,
not bytes.



-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://truepeace.org http://memri.org http://honestreporting.com
http://palestinefacts.org http://dhimmi.com http://camera.org http://ffii.org
Only the mediocre are always at their best.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <alac8yuz9f.fsf@quant8.janestcapital.quant>
>>>>>	Sam Steingold <sds@podval.org> wrote:

>>> Debugger entered--Lisp error: (error "Unknown symbol: \x5143e")
>>>   signal(error ("Unknown symbol: \x5143e"))
>>>   error("Unknown symbol: %c" 332862)
>>>   mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")

Note that there's a name expressed with Cyrillic characters:
"Ося Рыбаков"

>> * Katsumi Yamaoka <yamaoka@jpl.org> [2006-05-31 13:47:44 +0900]:

>> This error suggests *mail* messages don't allow non-ASCII text in
>> at least headers.  In other words, files should contain raw mail
>> messages.

> this is not the case.
> without your patch followup worked fine, and there was non-ASCII
> characters in the headers.

How was the name "Ося Рыбаков" encoded in the original message
(i.e., raw message)?  If it looked like "=?utf-8?B?...?=", I'm
quite sure there was no way to save that data as non-ASCII text
in a file before I made changes.  And it is hard to me to imagine
there was a raw charset-encoded name in the original message.
You or I completely misunderstand it.

>> Could you please show an alternative plan concretely if you have?

> A character is just a character - an abstract object.
> A file or a message is a sequence of characters (converted from bytes
> according to a global variable gnus-article-save-coding-system).
> Note that network transmission (which may mistreat non-ASCII) and
> interaction with other users (who may use a different
> gnus-article-save-coding-system) are completely irrelevant here.
> All that matters is that _my_ files on _my_ HD on _my_ computer are
> accessible for _me_ in a uniform way, both ASCII and non-ASCII.
> So, all that you need to do is write and parse files in the same coding
> system and the parse them (for nnfolder) as a sequence of characters,
> not bytes.

You are right except that there's no viewpoint on mail readers.
Generally they support only raw messages which don't contain raw
non-ASCII characters in headers.  Gnus is not an exception.

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

> * Katsumi Yamaoka <lnznbxn@wcy.bet> [2006-05-31 23:18:00 +0900]:
>
>>>>>> In <alac8yuz9f.fsf@quant8.janestcapital.quant>
>>>>>>	Sam Steingold <sds@podval.org> wrote:
>
>>>> Debugger entered--Lisp error: (error "Unknown symbol: \x5143e")
>>>>   signal(error ("Unknown symbol: \x5143e"))
>>>>   error("Unknown symbol: %c" 332862)
>>>>   mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")
>
> Note that there's a name expressed with Cyrillic characters:
> "Ося Рыбаков"

yes, of course.
mail-header-parse-address should just skip all non-ASCII, see below.

>>> * Katsumi Yamaoka <yamaoka@jpl.org> [2006-05-31 13:47:44 +0900]:
>
>>> This error suggests *mail* messages don't allow non-ASCII text in
>>> at least headers.  In other words, files should contain raw mail
>>> messages.
>
>> this is not the case.
>> without your patch followup worked fine, and there was non-ASCII
>> characters in the headers.
>
> How was the name "Ося Рыбаков" encoded in the original message (i.e.,
> raw message)?  If it looked like "=?utf-8?B?...?=", I'm quite sure

Subject: Re: =?WINDOWS-1251?B?4fDg8ujq?=
From: =?WINDOWS-1251?B?zvH/IND74eDq7uI=?= <pastushki@rambler.ru>
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="windows-1251"; format="flowed"
MIME-Version: 1.0

> there was no way to save that data as non-ASCII text in a file before
> I made changes.  And it is hard to me to imagine there was a raw
> charset-encoded name in the original message.  You or I completely
> misunderstand it.

yes, the " *Original Article*" buffer contains the "=?..." junk.  it
also contains all the information necessary to convert it to the real
characters, and, given a universal mapping
(gnus-article-save-coding-system) from characters to byte sequences, why
can't it be saved in a file?

>>> Could you please show an alternative plan concretely if you have?
>
>> A character is just a character - an abstract object.
>> A file or a message is a sequence of characters (converted from bytes
>> according to a global variable gnus-article-save-coding-system).
>> Note that network transmission (which may mistreat non-ASCII) and
>> interaction with other users (who may use a different
>> gnus-article-save-coding-system) are completely irrelevant here.
>> All that matters is that _my_ files on _my_ HD on _my_ computer are
>> accessible for _me_ in a uniform way, both ASCII and non-ASCII.
>> So, all that you need to do is write and parse files in the same coding
>> system and the parse them (for nnfolder) as a sequence of characters,
>> not bytes.
>
> You are right except that there's no viewpoint on mail readers.
> Generally they support only raw messages which don't contain raw
> non-ASCII characters in headers.  Gnus is not an exception.

yes, Gnus sucks just as much as all the other crap (/bin/mail &c).
are you sure this is really necessary?

what is the problem with

From: Вася Пупкин <vasya@pupkin.ru>

as opposed to

From: "Vasya Pupkin" <vasya@pupkin.ru>

?? since we know that e-mail is limited to ASCII (and moreover is
surrounded with <>) we can parse it as easily.

But, again, this is all secondary _at the moment_.
Right now all I want is a facility to ...

1. ... __APPEND__ the message (headers + body) to a file ...
2. ... so that everything is Unicode, i.e., no "=?..." garbage, and ...
3. ... the "^From " separator is included so that the file is some day
   accessible with /bin/mail, Gnus et al - when they are finally fixed

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://ffii.org http://thereligionofpeace.com http://honestreporting.com
http://palestinefacts.org http://truepeace.org http://pmw.org.il
Stupidity, like virtue, is its own reward.

Re: Incorrect encoding with `O F' or `O f'

[Reiner Steib]

On Wed, May 31 2006, Sam Steingold wrote:

>>>>>   mail-header-parse-address("\x5143e\x51461\x5146f \x51440\x5146b\x51451\x51450\x5145a\x5145e\x51452 <pastushki@rambler.ru>")
>>
>> Note that there's a name expressed with Cyrillic characters:
>> "Ося Рыбаков"
>
> yes, of course.
> mail-header-parse-address should just skip all non-ASCII, see below.

No.  `mail-parse.el' and friends are for parsing mail messages and not
for parsing some fantasy non-standard somewhat-mbox-like files.

> Right now all I want is a facility to ...
>
> 1. ... __APPEND__ the message (headers + body) to a file ...
> 2. ... so that everything is Unicode, i.e., no "=?..." garbage, and ...

1 and 2 has been done by Katsumi, AFAICS.

> 3. ... the "^From " separator is included so that the file is some day
>    accessible with /bin/mail, Gnus et al - when they are finally fixed

I'd suggest you to write a custom article saver based on the current
code.  Alternatively, you may try to convince the ietf-usefor people
that your suggestion is much better than the current RFCs and drafts.
Good luck!

WRT the From_ line, we could provide a hook which is run in the buffer
before saving it if that is feasible.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Incorrect encoding with `O F' or `O f'

[Sam Steingold]

another thing: if the file into which the message is saved with "C-o" is
visited in a buffer, the buffer is reverted, i.e., one can see the new
message at the end of the file right away.
this is not the case with "o".
-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 5 (Bordeaux)
http://iris.org.il http://openvotingconsortium.org http://palestinefacts.org
http://truepeace.org http://memri.org http://mideasttruth.com http://dhimmi.com
You think Oedipus had a problem -- Adam was Eve's mother.

Re: Incorrect encoding with `O F' or `O f'

[Katsumi Yamaoka]

>>>>> In <allksgssst.fsf@quant8.janestcapital.quant>
>>>>>	Sam Steingold wrote:

> another thing: if the file into which the message is saved with "C-o" is
> visited in a buffer, the buffer is reverted, i.e., one can see the new
> message at the end of the file right away.
> this is not the case with "o".

Yes, I know some saving commands modify a buffer which visits a
file and others don't.  There is potentially a problem in either
case.  Even in the later case, if Gnus and another non-Emacs
mail reader is running, that Gnus modifies a file which another
mail reader uses might cause a serious problem.  I have no idea.
If anything, there seems to be no way to solve it.