(file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

(file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]

Hi all,

some time ago (message <87k5bqv5bc.fsf@thinkpad.tsdh.de> on ding) I had
problems when sending mails with big attachments (> 1 MiB or so).  To be
precise, the sending via SMTP works just fine, but I get the error from
the subject when Gnus tries to upload the message to my local IMAP
server because of the Gcc header.

In the thread starting with <87k5bqv5bc.fsf@thinkpad.tsdh.de> I said
that turning off imap logging (nnimap-debug, imap-log, imap-debug all
set to nil) solves the issue for me, but sadly that's not true.

I still get that error and for me its perfectly reproducible.  Here's an
example backtrace.

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (file-error "writing to process" "bad address" #<process imap>)
  process-send-region(#<process imap> 1 8337045)
  imap-send-command(("APPEND \"" "INBOX" "\" " #<buffer  *acc*>) nil)
  imap-send-command-wait(("APPEND \"" "INBOX" "\" " #<buffer  *acc*>))
  imap-message-append("INBOX" #<buffer  *acc*> nil nil " *nnimap* Uni")
  nnimap-request-accept-article("INBOX" "Uni" t)
  gnus-request-accept-article("nnimap+Uni:INBOX" "nnimap:Uni" t t)
  gnus-inews-do-gcc()
  gnus-agent-possibly-do-gcc()
  run-hooks(message-sent-hook)
  message-send(nil)
  message-send-and-exit(nil)
  call-interactively(message-send-and-exit nil nil)
--8<---------------cut here---------------end--------------->8---

The local IMAP server hosting the Gcc-ed group is Dovecot v1.1.11.  I'm
not sure if part of the problem is on its side, but at least I can say
that it has no problems when offlineimap synces big messages from my
IMAP accounts to the local dovecot.

When the Gcc-upload fails, dovecot logs a line like this:

  dovecot: Feb 09 17:14:51 Info: IMAP(uni): \
           Disconnected in APPEND bytes=1992955/1144675

Could somebody please help me with this?

Bye,
Tassilo

Gnus v5.13
GNU Emacs 23.0.90.1 (x86_64-pc-linux-gnu, GTK+ Version 2.14.7)
 of 2009-02-08 on thinkpad
200 cache.uni-koblenz.de InterNetNews NNRP server INN 2.4.2 ready (posting ok).


[-- Attachment #2: User settings --]
[-- Type: application/emacs-lisp, Size: 12882 bytes --]

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> The local IMAP server hosting the Gcc-ed group is Dovecot v1.1.11.  I'm
> not sure if part of the problem is on its side, but at least I can say
> that it has no problems when offlineimap synces big messages from my
> IMAP accounts to the local dovecot.
>
> When the Gcc-upload fails, dovecot logs a line like this:
>
>   dovecot: Feb 09 17:14:51 Info: IMAP(uni): \
>            Disconnected in APPEND bytes=1992955/1144675
>
> Could somebody please help me with this?

I'm using Dovecot (on a remote server, though) and don't have any
problems appending large messages using Gnus.

Are you using the Gnus demon, e.g. for regularly checking your INBOX?
Could it be that the demon changes the mailbox while the APPEND is still
running?

If you cannot use IMAP debugging in Emacs, you can use the Dovecot's
"rawlog" utility: http://wiki.dovecot.org/Debugging/Rawlog . Maybe that
can shed some light on the issue.

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

>> The local IMAP server hosting the Gcc-ed group is Dovecot v1.1.11.  I'm
>> not sure if part of the problem is on its side, but at least I can say
>> that it has no problems when offlineimap synces big messages from my
>> IMAP accounts to the local dovecot.
>>
>> When the Gcc-upload fails, dovecot logs a line like this:
>>
>>   dovecot: Feb 09 17:14:51 Info: IMAP(uni): \
>>            Disconnected in APPEND bytes=1992955/1144675
>>
>> Could somebody please help me with this?
>
> I'm using Dovecot (on a remote server, though) and don't have any
> problems appending large messages using Gnus.

Define large.  It works for me till some specific size, but let's say
with attachments > 3 MiB it always fails.

> Are you using the Gnus demon, e.g. for regularly checking your INBOX?
> Could it be that the demon changes the mailbox while the APPEND is
> still running?

No, I only run offlineimap in a cron job, but that shouldn't cause any
harm, right?

> If you cannot use IMAP debugging in Emacs,

I can.  See the logs I posted in that old message.

> you can use the Dovecot's "rawlog" utility:
> http://wiki.dovecot.org/Debugging/Rawlog . Maybe that can shed some
> light on the issue.

I'll try that later this evening.

Thanks for your help so far!

Bye,
Tassilo

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David Engster <deng@randomsample.de> writes:
>>> The local IMAP server hosting the Gcc-ed group is Dovecot v1.1.11.  I'm
>>> not sure if part of the problem is on its side, but at least I can say
>>> that it has no problems when offlineimap synces big messages from my
>>> IMAP accounts to the local dovecot.
>>>
>>> When the Gcc-upload fails, dovecot logs a line like this:
>>>
>>>   dovecot: Feb 09 17:14:51 Info: IMAP(uni): \
>>>            Disconnected in APPEND bytes=1992955/1144675
>>>
>>> Could somebody please help me with this?
>>
>> I'm using Dovecot (on a remote server, though) and don't have any
>> problems appending large messages using Gnus.
>
> Define large.  It works for me till some specific size, but let's say
> with attachments > 3 MiB it always fails.

I just tried a 5MB attachment.

Does it also sometimes fail for smaller attachments? I just wonder if
this is a size or timing issue.

>> Are you using the Gnus demon, e.g. for regularly checking your INBOX?
>> Could it be that the demon changes the mailbox while the APPEND is
>> still running?
>
> No, I only run offlineimap in a cron job, but that shouldn't cause any
> harm, right?

The above error message from Dovecot says that the connection was closed
during the APPEND. Offlineimap should not be able to do that (I stress
"should").

>
>> If you cannot use IMAP debugging in Emacs,
>
> I can.  See the logs I posted in that old message.

Ah, OK. But could you enable imap-log instead of imap-debug? I always
find the latter a bit hard to parse...

If there's nothing to see, the rawlog utility should tell you what
happens before the connection gets closed.

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

> Does it also sometimes fail for smaller attachments?

Hm, at least I don't remember.  For big attachments it always fails.

> I just wonder if this is a size or timing issue.

I cannot tell, but bigger messages would increase the probability of
timing issues, so it's even harder to distinguish.

>>> Are you using the Gnus demon, e.g. for regularly checking your
>>> INBOX?  Could it be that the demon changes the mailbox while the
>>> APPEND is still running?
>>
>> No, I only run offlineimap in a cron job, but that shouldn't cause
>> any harm, right?
>
> The above error message from Dovecot says that the connection was
> closed during the APPEND. Offlineimap should not be able to do that (I
> stress "should").

I disabled the cron job, but still the APPEND fails.

>>> If you cannot use IMAP debugging in Emacs,
>>
>> I can.  See the logs I posted in that old message.
>
> Ah, OK. But could you enable imap-log instead of imap-debug? I always
> find the latter a bit hard to parse...

I can do that later, if the rawlog doesn't help.

> If there's nothing to see, the rawlog utility should tell you what
> happens before the connection gets closed.

See http://www.tsdh.de/~heimdall/dovecot.rawlog.tar.gz for zipped rawlog
directory.  I deleted all files in there and then sent the big mail, so
no irrelevant stuff is included.

In one file you can see the APPEND, then comes the message header and
body, followed by the base64 encoded attachment, or better part of it.
Then there's a NOOP and that's it.

Bye,
Tassilo

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
>> If there's nothing to see, the rawlog utility should tell you what
>> happens before the connection gets closed.
>
> See http://www.tsdh.de/~heimdall/dovecot.rawlog.tar.gz for zipped rawlog
> directory.  I deleted all files in there and then sent the big mail, so
> no irrelevant stuff is included.
>
> In one file you can see the APPEND, then comes the message header and
> body, followed by the base64 encoded attachment, or better part of it.
> Then there's a NOOP and that's it.

I tried it again at home over my slow DSL line, and now the exact same
thing happens here. I enabled all logs and used tcpdump to record the
traffic to the IMAP server, but to no avail. The imap-log doesn't help
since the buffer which is sent to the IMAP server is simply included
there before process-send-region is called.

I have no idea if this is a problem with Dovecot or maybe a bug in
process-send-region or in send-region (I don't see how any other Gnus or
Emacs code could be involved in this). It'd be interesting to know if
the problem also occurs in Emacs 22. Maybe I have time over the weekend
to test this further...

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

> I tried it again at home over my slow DSL line, and now the exact same
> thing happens here.

Oh, I'm sorry for that. ;-)

> I have no idea if this is a problem with Dovecot or maybe a bug in
> process-send-region or in send-region (I don't see how any other Gnus
> or Emacs code could be involved in this).

`send-region' is only an alias for `process-send-region'.  Maybe we
could create an isolated test case which doesn't involve gnus and
dovecot.  I tried to create such a case, but I don't have enough time
right now.  Basically it should be possible to load a big file into a
temp buffer and use it as input to a `cat' process, or something like
that and then compare if the result equals the input.

Bye,
Tassilo

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David Engster <deng@randomsample.de> writes:
>> I have no idea if this is a problem with Dovecot or maybe a bug in
>> process-send-region or in send-region (I don't see how any other Gnus
>> or Emacs code could be involved in this).
>
> `send-region' is only an alias for `process-send-region'.

Sorry, I should have said "send-region in process.c". The file-error is
thrown there, in the C source.

> Maybe we could create an isolated test case which doesn't involve gnus
> and dovecot. I tried to create such a case, but I don't have enough
> time right now.  Basically it should be possible to load a big file
> into a temp buffer and use it as input to a `cat' process, or
> something like that and then compare if the result equals the input.

I'd be surprised if the error is that general; much more people would
have to see it in this case. I just tried again at work, and here it
doesn't cause any problems, even with a 10MB attachment. I know you're
getting the error while posting to a local IMAP server, but in my case
it looks like a speed/timing issue.

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

David Engster <deng@randomsample.de> writes:
> Tassilo Horn <tassilo@member.fsf.org> writes:
>> David Engster <deng@randomsample.de> writes:
>>> I have no idea if this is a problem with Dovecot or maybe a bug in
>>> process-send-region or in send-region (I don't see how any other Gnus
>>> or Emacs code could be involved in this).
>>
>> `send-region' is only an alias for `process-send-region'.
>
> Sorry, I should have said "send-region in process.c". The file-error is
> thrown there, in the C source.

Make that "send_process"...

>> Maybe we could create an isolated test case which doesn't involve gnus
>> and dovecot. I tried to create such a case, but I don't have enough
>> time right now.  Basically it should be possible to load a big file
>> into a temp buffer and use it as input to a `cat' process, or
>> something like that and then compare if the result equals the input.
>
> I'd be surprised if the error is that general; much more people would
> have to see it in this case.

Well, color me surprised. I was just able to provoke the error with
the procedure you suggested:

(setq numlines 500000)

(progn
  (setq proctest 
	(start-process "cat" (get-buffer-create "cat") "/bin/cat"))
  (with-temp-buffer
    (dotimes (dummy numlines)
      (insert 
       (concat (number-to-string (random))
	       (number-to-string (random))
	       (number-to-string (random))
	       (number-to-string (random))
	       "\n")))
    (process-send-region proctest (point-min) (point-max)))
  (delete-process proctest))

And sometimes(!) I get:

Debugger entered--Lisp error: (file-error "writing to process" "bad address" #<process cat>)
  process-send-region(#<process cat> 1 18672117)

using Emacs from CVS (2009-02-04) on Darwin/Mac OS X. 

The error doesn't always happen. I usually have to repeat the procedure
several times until it occurs. You can increase 'numlines' to make it
more likely. It might also be that it depends on the load of the system
or something like that.

Maybe this

http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=471

might also be connected to this issue.

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

> Well, color me surprised. I was just able to provoke the error with
> the procedure you suggested:
>
> (setq numlines 500000)
>
> (progn
>   (setq proctest 
> 	(start-process "cat" (get-buffer-create "cat") "/bin/cat"))
>   (with-temp-buffer
>     (dotimes (dummy numlines)
>       (insert 
>        (concat (number-to-string (random))
> 	       (number-to-string (random))
> 	       (number-to-string (random))
> 	       (number-to-string (random))
> 	       "\n")))
>     (process-send-region proctest (point-min) (point-max)))
>   (delete-process proctest))
>
> And sometimes(!) I get:
>
> Debugger entered--Lisp error: (file-error "writing to process" "bad address" #<process cat>)
>   process-send-region(#<process cat> 1 18672117)
>
> using Emacs from CVS (2009-02-04) on Darwin/Mac OS X. 
>
> The error doesn't always happen. I usually have to repeat the
> procedure several times until it occurs. You can increase 'numlines'
> to make it more likely. It might also be that it depends on the load
> of the system or something like that.

I tried 4 times (using emacs -Q) and always (!) got the error after some
seconds.  System load is low.

GNU Emacs 23.0.90.1 (x86_64-pc-linux-gnu, GTK+ Version 2.14.7)
of 2009-02-10 on thinkpad

> Maybe this
>
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=471
>
> might also be connected to this issue.

Yes, at least it sounds quite similar.

Should I report an emacs bug report for this or do you want to do it?

Bye,
Tassilo

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David Engster <deng@randomsample.de> writes:
>> The error doesn't always happen. I usually have to repeat the
>> procedure several times until it occurs. You can increase 'numlines'
>> to make it more likely. It might also be that it depends on the load
>> of the system or something like that.
>
> I tried 4 times (using emacs -Q) and always (!) got the error after some
> seconds.  System load is low.
> GNU Emacs 23.0.90.1 (x86_64-pc-linux-gnu, GTK+ Version 2.14.7)
> of 2009-02-10 on thinkpad
>
>> Maybe this
>>
>> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=471
>>
>> might also be connected to this issue.
>
> Yes, at least it sounds quite similar.
>
> Should I report an emacs bug report for this or do you want to do it?

Either way is fine. Could you also run a quick test with Emacs 22? If it
doesn't happen there, one could try to find the commit which caused the
regression, which would be very helpful information for a bug report.

-David

Re: (file-error "writing to process" "bad address" #<process imap>) when Gcc-ing a group

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

>> Should I report an emacs bug report for this or do you want to do it?
>
> Either way is fine. Could you also run a quick test with Emacs 22? If
> it doesn't happen there, one could try to find the commit which caused
> the regression, which would be very helpful information for a bug
> report.

I'll do it later this evening.  Thanks a lot for your help.

Bye,
Tassilo