weirdness wrt file-coding

weirdness wrt file-coding

[Fabrice POPINEAU]

Hi,

Using xemacs-21.0b57 and  xemacs-21.2b5 (NT, compiled natively), I can
only    report weirdness  due    to incorrect  file   type guessing or
conversion in   gnus.  I'm using   pgnus-0.65. The file   retrieved by
movemail is put into ~/.gnus-crash-box  and parsed in the buffer named
" *nnmail incoming*". 

But   the parsing  fails   before  the end   of    the file  (function
nnmail-process-babyl-mail-format, re-search-forward   returns     nil,
delete-region is then called with 0 :-( ).

Switching to the  buffer shows that there are  '^M' at the end of each
line.  Toggling the   file   coding to  'noconv' manually,  and   then
rewriting .gnus-crash-box enables to read  the messages correctly next
time gnus is run.

It acts  like if the  file was read in text  mode, but  the conversion
binary -> text  failed, and the specified  file-coding is text  anyway.

At first, I thought it was apel playing with file-coding, and that
just removing 'tm' use from gnus-setup would do the trick. But
unfortunately, there is a deeper problem.

-- 
Fabrice POPINEAU
------------------------
e-mail:         popineau@ese-metz.fr    |  "God is real ...              
	Fabrice.POPINEAU@supelec.fr     |
voice-mail:   +33 (0) 387764715         |          ... unless integer ?"
surface-mail: Supelec, 2 rue E. Belin,  |
	      F-57078 Metz Cedex 3      |

Re: weirdness wrt file-coding

[Shenghuo ZHU]

>>>>> "FP" == Fabrice POPINEAU <popineau@ese-metz.fr> writes:

FP> Using xemacs-21.0b57 and xemacs-21.2b5 (NT, compiled natively), I
FP> can only report weirdness due to incorrect file type guessing or
FP> conversion in gnus.  I'm using pgnus-0.65. The file retrieved by
FP> movemail is put into ~/.gnus-crash-box and parsed in the buffer
FP> named " *nnmail incoming*".

FP> But the parsing fails before the end of the file (function
FP> nnmail-process-babyl-mail-format, re-search-forward returns nil,
FP> delete-region is then called with 0 :-( ).

FP> Switching to the buffer shows that there are '^M' at the end of
FP> each line.  Toggling the file coding to 'noconv' manually, and
FP> then rewriting .gnus-crash-box enables to read the messages
FP> correctly next time gnus is run.

FP> It acts like if the file was read in text mode, but the conversion
FP> binary -> text failed, and the specified file-coding is text
FP> anyway.

FP> At first, I thought it was apel playing with file-coding, and that
FP> just removing 'tm' use from gnus-setup would do the trick. But
FP> unfortunately, there is a deeper problem.

Try this patch. 

A new variable mm-text-coding-system is added. It is no-conversion in
XEmacs (Unix or NT), raw-text in Emacs, raw-text-dos in NTEmacs (I
guess raw-text != raw-text-dos in NTEmacs).

-- 
Shenghuo

ChangeLog:

Wed Dec  9 13:30:29 1998  Shenghuo ZHU  <zsh@cs.rochester.edu>

	* mm-util.el (mm-running-ntemacs): New variable.
	(mm-text-coding-system): Ditto.
	* nnmail.el (nnmail-incoming-coding-system): Ditto.
	(nnmail-split-incoming): Use nnmail-incoming-coding-system.

:- cut -------------
--- mm-util.el	1998/12/09 16:40:43	1.1
+++ mm-util.el	1998/12/09 18:30:18
@@ -26,10 +26,25 @@
 
 (defvar mm-running-xemacs (string-match "XEmacs" emacs-version))
 
+(defvar mm-running-ntemacs 
+  (and (not mm-running-xemacs) 
+       (string-match "nt" system-configuration)))
+
 (defvar mm-binary-coding-system 
     (if mm-running-xemacs
 	'binary 'no-conversion)
     "100% binary coding system.")   
+
+(defvar mm-text-coding-system 
+  (cond 
+   ((not (fboundp 'coding-system-p)) nil)
+   (mm-running-xemacs  ;; XEmacs
+    (and (coding-system-p 'no-conversion) 'no-conversion))
+   (mm-running-ntemacs ;; NTEmacs
+    (and (coding-system-p 'raw-text-dos) 'raw-text-dos))
+   ((coding-system-p 'raw-text) 'raw-text) ;; Emacs
+   (t nil))
+  "100% text coding system, for removing ^M.")
 
 (defvar mm-default-coding-system nil
   "The default coding system to use.")  

--- nnmail.el	1998/12/09 18:31:41	1.1
+++ nnmail.el	1998/12/09 18:36:06
@@ -501,6 +501,10 @@
       'raw-text-dos 'binary)
   "Another coding system used in nnmail.")
 
+(defvar nnmail-incoming-coding-system
+  mm-text-coding-system
+  "Coding system used in reading inbox")
+
 (defun nnmail-find-file (file)
   "Insert FILE in server buffer safely."
   (set-buffer nntp-server-buffer)
@@ -1007,7 +1011,8 @@
       ;; Insert the incoming file.
       (set-buffer (get-buffer-create " *nnmail incoming*"))
       (erase-buffer)
-      (nnheader-insert-file-contents incoming)
+      (let ((nnheader-file-coding-system nnmail-incoming-coding-system))
+	(nnheader-insert-file-contents incoming))
       (unless (zerop (buffer-size))
 	(goto-char (point-min))
 	(save-excursion (run-hooks 'nnmail-prepare-incoming-hook))

Re: weirdness wrt file-coding

[Fabrice POPINEAU]

Shenghuo ZHU <zsh@cs.rochester.edu> writes:

> Try this patch. 
> 
> A new variable mm-text-coding-system is added. It is no-conversion in
> XEmacs (Unix or NT), raw-text in Emacs, raw-text-dos in NTEmacs (I
> guess raw-text != raw-text-dos in NTEmacs).

Great ! I was looking for something along these lines, and it seems to 
work ok. Not extensively tested, but got some hundred of messages
tight as expected.

Thanks,

Fabrice

> ChangeLog:
> 
> Wed Dec  9 13:30:29 1998  Shenghuo ZHU  <zsh@cs.rochester.edu>
> 
> 	* mm-util.el (mm-running-ntemacs): New variable.
> 	(mm-text-coding-system): Ditto.
> 	* nnmail.el (nnmail-incoming-coding-system): Ditto.
> 	(nnmail-split-incoming): Use nnmail-incoming-coding-system.
> 
> :- cut -------------
> --- mm-util.el	1998/12/09 16:40:43	1.1
> +++ mm-util.el	1998/12/09 18:30:18
> @@ -26,10 +26,25 @@
>  
>  (defvar mm-running-xemacs (string-match "XEmacs" emacs-version))
>  
> +(defvar mm-running-ntemacs 
> +  (and (not mm-running-xemacs) 
> +       (string-match "nt" system-configuration)))
> +
>  (defvar mm-binary-coding-system 
>      (if mm-running-xemacs
>  	'binary 'no-conversion)
>      "100% binary coding system.")   
> +
> +(defvar mm-text-coding-system 
> +  (cond 
> +   ((not (fboundp 'coding-system-p)) nil)
> +   (mm-running-xemacs  ;; XEmacs
> +    (and (coding-system-p 'no-conversion) 'no-conversion))
> +   (mm-running-ntemacs ;; NTEmacs
> +    (and (coding-system-p 'raw-text-dos) 'raw-text-dos))
> +   ((coding-system-p 'raw-text) 'raw-text) ;; Emacs
> +   (t nil))
> +  "100% text coding system, for removing ^M.")
>  
>  (defvar mm-default-coding-system nil
>    "The default coding system to use.")  
> 
> --- nnmail.el	1998/12/09 18:31:41	1.1
> +++ nnmail.el	1998/12/09 18:36:06
> @@ -501,6 +501,10 @@
>        'raw-text-dos 'binary)
>    "Another coding system used in nnmail.")
>  
> +(defvar nnmail-incoming-coding-system
> +  mm-text-coding-system
> +  "Coding system used in reading inbox")
> +
>  (defun nnmail-find-file (file)
>    "Insert FILE in server buffer safely."
>    (set-buffer nntp-server-buffer)
> @@ -1007,7 +1011,8 @@
>        ;; Insert the incoming file.
>        (set-buffer (get-buffer-create " *nnmail incoming*"))
>        (erase-buffer)
> -      (nnheader-insert-file-contents incoming)
> +      (let ((nnheader-file-coding-system nnmail-incoming-coding-system))
> +	(nnheader-insert-file-contents incoming))
>        (unless (zerop (buffer-size))
>  	(goto-char (point-min))
>  	(save-excursion (run-hooks 'nnmail-prepare-incoming-hook))
> 

Re: weirdness wrt file-coding

[Adrian Aichner]

>>>>> "Fabrice" == Fabrice POPINEAU <popineau@ese-metz.fr> writes:

    Fabrice> Hi,

    Fabrice> Using xemacs-21.0b57 and xemacs-21.2b5 (NT, compiled
    Fabrice> natively), I can only report weirdness due to incorrect
    Fabrice> file type guessing or conversion in gnus.  I'm using
    Fabrice> pgnus-0.65. The file retrieved by movemail is put into
    Fabrice> ~/.gnus-crash-box and parsed in the buffer named "
    Fabrice> *nnmail incoming*".

    Fabrice> But the parsing fails before the end of the file
    Fabrice> (function nnmail-process-babyl-mail-format,
    Fabrice> re-search-forward returns nil, delete-region is then
    Fabrice> called with 0 :-( ).

    Fabrice> Switching to the buffer shows that there are '^M' at the
    Fabrice> end of each line.  Toggling the file coding to 'noconv'
    Fabrice> manually, and then rewriting .gnus-crash-box enables to
    Fabrice> read the messages correctly next time gnus is run.

    Fabrice> It acts like if the file was read in text mode, but the
    Fabrice> conversion binary -> text failed, and the specified
    Fabrice> file-coding is text anyway.

    Fabrice> At first, I thought it was apel playing with file-coding,
    Fabrice> and that just removing 'tm' use from gnus-setup would do
    Fabrice> the trick. But unfortunately, there is a deeper problem.

This was my first thought too.

Seems like (emacs-version) "XEmacs 21.0 \"Poitou59\" [Lucid]
(i386-pc-win32) of Wed Dec 09 1998 on DARTER2"
does not cooperate with EFS either.

More later...

Adrian