Re: should mbox be opened in unibyte mode

Re: should mbox be opened in unibyte mode

[Janusz S. Bien]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes

>binary is an alias for no-conversion in FSF Emacs.  XEmacs used not to have
>a no-conversion (that may have changed).  So be careful.

Thanks for drawing my attention to this difference.

Regards

Janusz

---------------------------------------------------------------------
                     ,   
dr hab. Janusz S. Bien, prof. UW
Prof. Janusz S. Bien, Warsaw Uniwersity
---------------------------------------------------------------------
---------------------------------------------------------------------
Na tym koncie czytam i wysylam poczte i wiadomosci offline -
prosze nie oczekiwac szybkiej odpowiedzi!
Data w naglowku to data rozpoczecia pisania listu, a nie jego wyslania.
---------------------------------------------------------------------
On this account I read/post mail/news offline - do not expect 
an immediate answer!
The date in the header refers to the moment when I started to write
the letter, not to the moment when I sent it.
---------------------------------------------------------------------

Re: should mbox be opened in unibyte mode

[Janusz S. Bien]

Vladimir wrote:

>Here at home i only have GNU Emacs 20.4, and describe-coding-system
>says about `binary':
>
[...]
>Perhaps, 20.6 has a more verbose description (which states in
>particular that 'binary forces unibyte).

There is no substantial difference between GNU 20.4 and 20.6 in this
respect (21 will introduce some important changes).

My quotation come from the "Coding systems" section of Emacs manual
--- look at "International character support" chapter or search the
info files for relevant phrases. The Elisp manual contains also
useful information (IHMO, some of it would be better placed in the
user manual).

>(btw, where i can access the current CVS emacs version for testing?)

It's not so simple, but you definitely will be welcomed as a
pretester. Contact Gerd Moellmann <gerd@gnu.org> for details.

>(BTW, when a buffer is multibyte, C-x RET f binary RET is not supposed
>to switch to unibyte mode, right? only opening a file with a binary
>mode (C-x RET c binary RET C-x C-f filename RET) does this?)

Right.

Regards

Janusz

---------------------------------------------------------------------
                     ,   
dr hab. Janusz S. Bien, prof. UW
Prof. Janusz S. Bien, Warsaw Uniwersity
---------------------------------------------------------------------
---------------------------------------------------------------------
Na tym koncie czytam i wysylam poczte i wiadomosci offline -
prosze nie oczekiwac szybkiej odpowiedzi!
Data w naglowku to data rozpoczecia pisania listu, a nie jego wyslania.
---------------------------------------------------------------------
On this account I read/post mail/news offline - do not expect 
an immediate answer!
The date in the header refers to the moment when I started to write
the letter, not to the moment when I sent it.
---------------------------------------------------------------------

Re: should mbox be opened in unibyte mode

[Vladimir Volovich]

"JSB" == Janusz S Bien writes:

 JSB> What I wrote is about GNU Emacs and is based on its manual for
 JSB> version 20.6, the output of `describe-coding-system' and simple
 JSB> experiments with opening e.g. HELLO file with the `binary'
 JSB> coding system.

Here at home i only have GNU Emacs 20.4, and describe-coding-system
says about `binary':

= -- binary (alias of no-conversion)
  Do no conversion
Type: nil (do no conversion)
EOL type: LF

Perhaps, 20.6 has a more verbose description (which states in
particular that 'binary forces unibyte).

(btw, where i can access the current CVS emacs version for testing?)

 >> is the above about XEmacs or GNU Emacs? If it is about GNU Emacs,
 >> then why setting nnmbox-file-coding-system to 'binary does not
 >> switch off multibyte characters?

 JSB> Looks like some tracing should be done to check whether the
 JSB> values of `nnmbox-file-coding-system' and
 JSB> `nnmbox-active-file-coding-system' are really used when opening
 JSB> mbox.

It looks that they are not used when opening mbox. ;-(
(BTW, when a buffer is multibyte, C-x RET f binary RET is not supposed
to switch to unibyte mode, right? only opening a file with a binary
mode (C-x RET c binary RET C-x C-f filename RET) does this?)

	Best regards, -- Vladimir.

Re: should mbox be opened in unibyte mode

[Janusz S. Bien]

Vladimir wrote:

>is the above about XEmacs or GNU Emacs? If it is about GNU Emacs, then
>why setting nnmbox-file-coding-system to 'binary does not switch off
>multibyte characters?


What I wrote is about GNU Emacs and is based on its manual for
version 20.6, the output of `describe-coding-system' and simple
experiments with opening e.g. HELLO file with the `binary' coding
system.

>is the above about XEmacs or GNU Emacs? If it is about GNU Emacs, then
>why setting nnmbox-file-coding-system to 'binary does not switch off
>multibyte characters?

Looks like some tracing should be done to check whether the values of
`nnmbox-file-coding-system' and `nnmbox-active-file-coding-system'
are really used when opening mbox.

Regards

Janusz

---------------------------------------------------------------------
                     ,   
dr hab. Janusz S. Bien, prof. UW
Prof. Janusz S. Bien, Warsaw Uniwersity
---------------------------------------------------------------------
---------------------------------------------------------------------
Na tym koncie czytam i wysylam poczte i wiadomosci offline -
prosze nie oczekiwac szybkiej odpowiedzi!
Data w naglowku to data rozpoczecia pisania listu, a nie jego wyslania.
---------------------------------------------------------------------
On this account I read/post mail/news offline - do not expect 
an immediate answer!
The date in the header refers to the moment when I started to write
the letter, not to the moment when I sent it.
---------------------------------------------------------------------

Re: should mbox be opened in unibyte mode

[Vladimir Volovich]

"JSB" == Janusz S Bien writes:

 JSB> It, too, sets `enable-multibyte-characters' to `nil'.
 JSB> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 >> i've tried the above settings of nnmbox-file-coding-system and
 >> nnmbox-active-file-coding-system but nevertheless mbox is still
 >> opened in multibyte mode (in violation with the above).

 JSB> Do you use XEmacs or GNU Emacs? This appeared to be relevant.

I use GNU Emacs.

 JSB> You can force (in GNU Emacs) the current buffer to unibyte mode
 JSB> with `toggle-enable-multibyte-characters'. Does it help?

Yes, i can manually switch to unibyte mode in mbox buffer using
toggle-enable-multibyte-characters.

But:

1) imho it should be done by gnus (i.e. gnus should open mbox in
   unibyte mode in the first place)

2) you wrote that

 > In contrast, the coding system `no-conversion' specifies no
 > character code conversion at all--none for non-ASCII byte values and
 > none for end of line.  This is useful for reading or writing binary
 > files, tar files, and other files that must be examined verbatim.  It,
 > too, sets `enable-multibyte-characters' to `nil'.

is the above about XEmacs or GNU Emacs? If it is about GNU Emacs, then
why setting nnmbox-file-coding-system to 'binary does not switch off
multibyte characters?

	Best regards, -- Vladimir.

Re: should mbox be opened in unibyte mode

[Janusz S. Bien]

> JSB> You both are right.
>[...]
> JSB> It, too, sets `enable-multibyte-characters' to `nil'.
> JSB> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
>i've tried the above settings of nnmbox-file-coding-system and
>nnmbox-active-file-coding-system but nevertheless mbox is still opened
>in multibyte mode (in violation with the above).

Do you use XEmacs or GNU Emacs? This appeared to be relevant.

You can force (in GNU Emacs) the current buffer to unibyte mode with
`toggle-enable-multibyte-characters'. Does it help?

Regards

Janusz

P.S. I am now too busy to read this list systematically, I only peek
occasionally at some subjects. I am sorry if I missed some other
comments to my postings.

---------------------------------------------------------------------
                     ,   
dr hab. Janusz S. Bien, prof. UW
Prof. Janusz S. Bien, Warsaw Uniwersity
---------------------------------------------------------------------
---------------------------------------------------------------------
Na tym koncie czytam i wysylam poczte i wiadomosci offline -
prosze nie oczekiwac szybkiej odpowiedzi!
Data w naglowku to data rozpoczecia pisania listu, a nie jego wyslania.
---------------------------------------------------------------------
On this account I read/post mail/news offline - do not expect 
an immediate answer!
The date in the header refers to the moment when I started to write
the letter, not to the moment when I sent it.
---------------------------------------------------------------------

Re: should mbox be opened in unibyte mode

[Vladimir Volovich]

"JSB" == Janusz S Bien writes:

 VV> i thought a bit on this, and it seems that mbox MUST be opened in
 VV> unibyte mode, otherwise the results will be broken.
 SZ> I think unibyte mode does not really matter how mbox is opened,
 SZ> but the coding-system does. Have you tried
 SZ> 
 SZ> (setq nnmbox-file-coding-system 'binary) (setq
 SZ> nnmbox-active-file-coding-system 'binary)

 JSB> You both are right.
[...]
 JSB> It, too, sets `enable-multibyte-characters' to `nil'.
 JSB> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

i've tried the above settings of nnmbox-file-coding-system and
nnmbox-active-file-coding-system but nevertheless mbox is still opened
in multibyte mode (in violation with the above).

and why the defaults are not 'binary but 'raw-text (if the former is
considered to be the Right Thing anyway)?

	Best regards, -- Vladimir.

Re: should mbox be opened in unibyte mode

[Hrvoje Niksic]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Hrvoje Niksic <hniksic@iskon.hr> writes:
> 
> > How can one name be "right"?  
> 
> By working on both Emacsen, for example.

Oh, I see.  You're of course correct -- in that sense, `binary' and
`raw-text' are the right (accepted in both implementations) names for
those features.

Re: should mbox be opened in unibyte mode

[Stainless Steel Rat]

* Hrvoje Niksic <hniksic@iskon.hr>  on Fri, 05 May 2000
| As always with Mule, things are even worse.  Under XEmacs/Mule,
| `no-conversion' means "no charset conversion", but it still converts
| newlines.  Only `binary' really means binary.

Ah! That's right.  My ChangeLog mentions XEmacs being stupid, but I had
forgotten why.  And yes, only "binary" means binary, and it works correctly 
on both Emacsen.
-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball may stick to certain types
Minion of Nathan - Nathan says Hi! \ of skin.
PGP Key: at a key server near you!  \ 

Re: should mbox be opened in unibyte mode

[Kai Großjohann]

Hrvoje Niksic <hniksic@iskon.hr> writes:

> How can one name be "right"?  

By working on both Emacsen, for example.  All other things being
equal, something which works on both Emacsen is `more right' than
something which doesn't.  No?

kai
-- 
Beware of flying birch trees.

Re: should mbox be opened in unibyte mode

[Hrvoje Niksic]

Shenghuo ZHU <zsh@cs.rochester.edu> writes:

> Hrvoje> (If you're wondering why this is so, it's not because we're
> Hrvoje> perverse -- this is the way things were setup in the
> Hrvoje> original Mule, and they got inherited by XEmacs/Mule before
> Hrvoje> FSF people made the swap.)
> 
> IIRC, in both FSF Emacs and XEmacs, `raw-text' means "no charset
> conversion", but it still converts newlines.  So, binary and
> raw-text are the right names, but no-conversion is not.

How can one name be "right"?  Under XEmacs, `no-conversion' and
`raw-text' are synonymous.  Under FSF Emacs, `no-conversion' and
`binary' are synonymous.

The FSF choice probably makes more sense, but as I said, code written
for XEmacs would break if we were to change it now.

Re: should mbox be opened in unibyte mode

[Shenghuo ZHU]

>>>>> "Hrvoje" == Hrvoje Niksic <hniksic@iskon.hr> writes:

Hrvoje> Stainless Steel Rat <ratinox@peorth.gweep.net> writes:
>> * jsbien@mimuw.edu.pl (Janusz S. Bien)  on Fri, 05 May 2000
>> | `binary' is just an alias for `no-conversion'. Here's what the manual
>> | says about it:
>> 
>> binary is an alias for no-conversion in FSF Emacs.  XEmacs used not
>> to have a no-conversion (that may have changed).  So be careful.

Hrvoje> As always with Mule, things are even worse.  Under
Hrvoje> XEmacs/Mule, `no-conversion' means "no charset conversion",
Hrvoje> but it still converts newlines.  Only `binary' really means
Hrvoje> binary.

Hrvoje> (If you're wondering why this is so, it's not because we're
Hrvoje> perverse -- this is the way things were setup in the original
Hrvoje> Mule, and they got inherited by XEmacs/Mule before FSF people
Hrvoje> made the swap.)

IIRC, in both FSF Emacs and XEmacs, `raw-text' means "no charset
conversion", but it still converts newlines.  So, binary and raw-text
are the right names, but no-conversion is not.

-- 
Shenghuo

Re: should mbox be opened in unibyte mode

[Hrvoje Niksic]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * jsbien@mimuw.edu.pl (Janusz S. Bien)  on Fri, 05 May 2000
> | `binary' is just an alias for `no-conversion'. Here's what the manual
> | says about it:
> 
> binary is an alias for no-conversion in FSF Emacs.  XEmacs used not
> to have a no-conversion (that may have changed).  So be careful.

As always with Mule, things are even worse.  Under XEmacs/Mule,
`no-conversion' means "no charset conversion", but it still converts
newlines.  Only `binary' really means binary.

(If you're wondering why this is so, it's not because we're perverse
-- this is the way things were setup in the original Mule, and they
got inherited by XEmacs/Mule before FSF people made the swap.)

Re: should mbox be opened in unibyte mode

[Stainless Steel Rat]

* jsbien@mimuw.edu.pl (Janusz S. Bien)  on Fri, 05 May 2000
| `binary' is just an alias for `no-conversion'. Here's what the manual
| says about it:

binary is an alias for no-conversion in FSF Emacs.  XEmacs used not to have
a no-conversion (that may have changed).  So be careful.
-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.

Re: should mbox be opened in unibyte mode

[Janusz S. Bien]

Shenghuo ZHU <zsh@cs.rochester.edu> writes:

>>>>>> "VV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:
[...]
VV> i thought a bit on this, and it seems that mbox MUST be opened in
VV> unibyte mode, otherwise the results will be broken.
[...]
>
>I think unibyte mode does not really matter how mbox is opened, but
>the coding-system does. Have you tried
>
>(setq nnmbox-file-coding-system 'binary)
>(setq nnmbox-active-file-coding-system 'binary)
>

You both are right.

`binary' is just an alias for `no-conversion'. Here's what the manual
says about it:

   In contrast, the coding system `no-conversion' specifies no
character code conversion at all--none for non-ASCII byte values and
none for end of line.  This is useful for reading or writing binary
files, tar files, and other files that must be examined verbatim.  It,
too, sets `enable-multibyte-characters' to `nil'.
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>It might solve the problem.

I think it should.

Best regards

Janusz

---------------------------------------------------------------------
                     ,   
dr hab. Janusz S. Bien, prof. UW
Prof. Janusz S. Bien, Warsaw Uniwersity
---------------------------------------------------------------------
---------------------------------------------------------------------
Na tym koncie czytam i wysylam poczte i wiadomosci offline -
prosze nie oczekiwac szybkiej odpowiedzi!
Data w naglowku to data rozpoczecia pisania listu, a nie jego wyslania.
---------------------------------------------------------------------
On this account I read/post mail/news offline - do not expect 
an immediate answer!
The date in the header refers to the moment when I started to write
the letter, not to the moment when I sent it.
---------------------------------------------------------------------

Re: should mbox be opened in unibyte mode

[Shenghuo ZHU]

>>>>> "VV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VV> i thought a bit on this, and it seems that mbox MUST be opened in
VV> unibyte mode, otherwise the results will be broken.

VV> mbox may contain arbitrary 8-bit data (coming from 8-bit CTT mime
VV> parts e.g. in utf-8). these 8-bit char sequences must not be
VV> interpreted by mule as multibyte chars because this is plain wrong:
VV> these sequences come from random standard registered charsets which
VV> have nothing to do with internal mule encoding. so, the mbox should be
VV> treated as a raw (uni)byte sequence, but not as something that has
VV> relation to internal mule encoding.

You are right. I just found that Emacs mule can not handle arbitrary
8bit data in multibyte buffer.  Most of these happen for 8bit utf-8 or
koi8-r characters, not for iso-8859-?, cn-gb-2312, etc.  Besides the
mbox buffer, the original article buffers, mm buffers, backlog buffer,
and nnfolder buffers should also be unibyte.  But I am afraid it might
cause disaster if I change too much.  Let wait for Mr. Handa.

-- 
Shenghuo

Re: should mbox be opened in unibyte mode

[Shenghuo ZHU]

>>>>> "VV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VV> "ZSH" == Shenghuo ZHU writes:
ZSH> Multibyte is intentionally enabled in the mbox buffer. I don't
ZSH> know the reason.

VV> i thought a bit on this, and it seems that mbox MUST be opened in
VV> unibyte mode, otherwise the results will be broken.

VV> mbox may contain arbitrary 8-bit data (coming from 8-bit CTT mime
VV> parts e.g. in utf-8). these 8-bit char sequences must not be
VV> interpreted by mule as multibyte chars because this is plain wrong:
VV> these sequences come from random standard registered charsets which
VV> have nothing to do with internal mule encoding. so, the mbox should be
VV> treated as a raw (uni)byte sequence, but not as something that has
VV> relation to internal mule encoding.

I think unibyte mode does not really matter how mbox is opened, but
the coding-system does. Have you tried

(setq nnmbox-file-coding-system 'binary)
(setq nnmbox-active-file-coding-system 'binary)

It might solve the problem.

-- 
Shenghuo

Re: should mbox be opened in unibyte mode

[Vladimir Volovich]

"ZSH" == Shenghuo ZHU writes:

 ZSH> Multibyte is intentionally enabled in the mbox buffer. I don't
 ZSH> know the reason.

i thought a bit on this, and it seems that mbox MUST be opened in
unibyte mode, otherwise the results will be broken.

mbox may contain arbitrary 8-bit data (coming from 8-bit CTT mime
parts e.g. in utf-8). these 8-bit char sequences must not be
interpreted by mule as multibyte chars because this is plain wrong:
these sequences come from random standard registered charsets which
have nothing to do with internal mule encoding. so, the mbox should be
treated as a raw (uni)byte sequence, but not as something that has
relation to internal mule encoding.

	Best regards, -- Vladimir.

Re: should mbox be opened in unibyte mode

[Shenghuo ZHU]

>>>>> "VV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VV> Hi!
VV> gnus seems to open mbox (file for nnmbox backend) in multibyte mode.
VV> i just thought that this could be the reason for problems which i
VV> described earlier (certain character combinations in the range
VV> 0x80-0x9f which happen to come with some mail messages choke emacs and
VV> this results in huge delays and incorrect gnus behavior).

VV> i think that a very probably fix to it would be to open mbox file in
VV> unibyte mode.

VV> nevertheless, the original problem should be solved too (i'll try to
VV> prepare a bug report soon and will send it to Mr. Handa).

Multibyte is intentionally enabled in the mbox buffer. I don't know
the reason.

1998-09-16 10:38:21  Lars Magne Ingebrigtsen  <larsi@gnus.org>

	* nnmbox.el (nnmbox-request-group): Change server.
	(nnmbox-possibly-change-newsgroup): Enable multibyte.

-- 
Shenghuo

should mbox be opened in unibyte mode

[Vladimir Volovich]

Hi!

gnus seems to open mbox (file for nnmbox backend) in multibyte mode.
i just thought that this could be the reason for problems which i
described earlier (certain character combinations in the range
0x80-0x9f which happen to come with some mail messages choke emacs and
this results in huge delays and incorrect gnus behavior).

i think that a very probably fix to it would be to open mbox file in
unibyte mode.

nevertheless, the original problem should be solved too (i'll try to
prepare a bug report soon and will send it to Mr. Handa).

Best,
v.