davmail and nnimap issues

davmail and nnimap issues

[Steinar Bang]

When the current gnus HEAD, nnimap with davmail is back to this
behaviour:
 http://permalink.gmane.org/gmane.emacs.gnus.general/83746

In short: the correct number of unread articles is reported, but when I
enter the group, this single line is all that I see in the summary
buffer:
 O  [   ?: nobody                 ] (none)
(the text colour of the line is blue)

Before entering the summary buffer of the group and displaying the above
results, it reads a lot of data, ie. lines like this in the minibuffer.
 nnimap read 900k from localhost

It also write these two lines to the minibuffer:
 No more unseen articles
 No more unread articles

Looking at the IMAP traffic between gnus and davmail with wireshark, the
IMAP code looks valid to me.

If I'm to guess what fails is the parsing of the IMAP results into
something that can be put into the Summary buffer.

If I was going to edebug this, what function should I instrument?

I tried doing C-M-x on
 (deffoo nnimap-request-group (group &optional server dont-check info)

But that didn't make the debugger start when entering an nnimap group on
the davmail server.

Thanks!


- Steinar

 

How to debug a new IMAP server type?

[Steinar Bang]

How do one go about debugging a new IMAP server type?

What buffers do I look in? What do I look for in those buffers? Are
there debug options to switch on?

What I've done so far, is to use WireShark and also trying to step
through entering a group on the server in edebug.  Neither of these has
made me any wiser.

As far as I can tell from the WireShark grab, there is a seemingly OK
IMAP dialog between Gnus and the server.  But the result in Gnus is
rather disappointing,  A single article shows up in the summary, looking
like this:
 O. [   ?: nobody                 ] (none)
Entering that article displays an article that looks OK, and looks like
it could be the article in question, based on the time it was sent.

(The server in question is davmail, running on a debian testing machine)

Re: How to debug a new IMAP server type?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> How do one go about debugging a new IMAP server type?

Set `nnimap-record-commands' to t.

> What buffers do I look in? What do I look for in those buffers? Are
> there debug options to switch on?

" *imap log*" and " *nnimap ... *" buffers.

> As far as I can tell from the WireShark grab, there is a seemingly OK
> IMAP dialog between Gnus and the server.  But the result in Gnus is
> rather disappointing,  A single article shows up in the summary, looking
> like this:
>  O. [   ?: nobody                 ] (none)
> Entering that article displays an article that looks OK, and looks like
> it could be the article in question, based on the time it was sent.
>
> (The server in question is davmail, running on a debian testing machine)

Stepping trough `nnimap-transform-headers' might also help.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: How to debug a new IMAP server type?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

>> What buffers do I look in? What do I look for in those buffers? Are
>> there debug options to switch on?

> " *imap log*"

If I understand it correctly, It seems to be fetching information for
just a single article (UID 2017), this is from entering the davmail
INBOX:
21:06:04 [localhost] 41 SELECT "INBOX"
21:06:06 [localhost] 42 SELECT "INBOX"
21:06:10 [localhost] 43 UID FETCH 2017 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)])

For comparison, here is entering an nnimap group on my dovecot server:
21:09:56 [imap.mydomain.com] 46 SELECT "INBOX"
21:09:56 [imap.mydomain.com] 47 SELECT "INBOX"
21:09:56 [imap.mydomain.com] 48 UID FETCH 1:2,52,551:557,561:562,564:567,570,572,584:585,609,652,655,658:669,686,791,798:801,805,836,1005:1006,1023:1024,1053,1058,1068,1084,1088,1090:1094,1097,1099:1100,1104,1110,1116,1127,1197,1204,1206,1209,1225:1226,1254,1262,1280,1284,1313,1331,1341,1347:1348,1350,1417,1441,1443,1447,1451,1485,1489,1493,1539,1554,1576,1592,1594 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)])

> and " *nnimap ... *" buffers.
  
The " *nnimap localhost 1143  *nntpd**" buffer seems to contain the
email I see when I selet the single article I see in the Summary.

Then it ends with:
* BYE Closing connection

Process *nnimap*<1> connection broken by remote peer

Is that significant?

Then there is also the buffer " *nnimap localhost 1143  *nntpd**-398832"
with the following content:
211 2017 Article retrieved.
Chars: 28785
Lines: 6
From: John Doe <john.doe@somecompany.no>
To: Steinar BANG <steinar.bang@somecompany.no>
Date: Tue, 21 Jan 2014 08:13:04 +0100
Subject: SV: Hjemme med sykt barn
Message-ID: <701DABD47A1A7044999B70D8862952E339DA8A8829@SVVEPOST01.somecompany.no>
References: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
In-Reply-To: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
)
55 OK UID FETCH completed
* BYE Closing connection

Process *nnimap*<1> connection broken by remote peer
.

(in fact there seems to be a new such buffer, with an "-somenumber"
extension, created every time I enter the INBOX, or when I try to
display the single article.

And they are never cleaned up as far as I can tell.

> Stepping trough `nnimap-transform-headers' might also help.

It stepped through the while loop in the function once (which by now
isn't too surprising, there seems to be only one article seen by this
point in time).

It does what it's supposed to do with the single article it has to work
with, and what's left at the end, is this:
211 2017 Article retrieved.
Chars: 28785
Lines: 6
From: John Doe <john.doe@somecompany.no>
To: Steinar BANG <steinar.bang@somecompany.no>
Date: Tue, 21 Jan 2014 08:13:04 +0100
Subject: SV: Hjemme med sykt barn
Message-ID: <701DABD47A1A7044999B70D8862952E339DA8A8829@SVVEPOST01.somecompany.no>
References: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
In-Reply-To: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
)
70 OK UID FETCH completed
* BYE Closing connection

Process *nnimap*<1> connection broken by remote peer
.

And from that it should have been able to at least display meaningful
information from that article, rather than what it displays, which is
this: 
  O. [   ?: nobody                 ] (none)

Here is the summary line for the article two articles earlier in the thread:
? +    [  21: Steinar Bang           ] How to debug a new IMAP server type?

So there should have been a line number (which is present in the headers
left in the " *nnimap..." buffer), and there should have been the real
name of the sender (which is also present in From: header left in the "
*nnimap..." buffer), and there should have been a subject (which is also
present in the leftover headers in the " *nnimap..." buffer).

As far as I can tell the headers all have the correct form.

Is it the "* BYE" and the closing connection message that is confusing
the stuff that is trying to display the summary?

Re: How to debug a new IMAP server type?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Then there is also the buffer " *nnimap localhost 1143  *nntpd**-398832"
> with the following content:

[...]

> (in fact there seems to be a new such buffer, with an "-somenumber"
> extension, created every time I enter the INBOX, or when I try to
> display the single article.

That's kinda odd, but it might be because the IMAP server i closing the
connection all the time.

> And they are never cleaned up as far as I can tell.

The should be killed when you exit Gnus.

>> Stepping trough `nnimap-transform-headers' might also help.
>
> It stepped through the while loop in the function once (which by now
> isn't too surprising, there seems to be only one article seen by this
> point in time).
>
> It does what it's supposed to do with the single article it has to work
> with, and what's left at the end, is this:

Could you post the output from the buffer before
`nnimap-transform-headers' is called?

> Is it the "* BYE" and the closing connection message that is confusing
> the stuff that is trying to display the summary?

Yes, that would be my guess.  `nnimap-transform-headers' should clean up
that stuff before delivering the data to Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: How to debug a new IMAP server type?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

>> And they are never cleaned up as far as I can tell.

> They should be killed when you exit Gnus.

Yes, they are.  But when Gnus was left running, it created quite a
number of these buffers.

>> It does what it's supposed to do with the single article it has to work
>> with, and what's left at the end, is this:

> Could you post the output from the buffer before
> `nnimap-transform-headers' is called?

* 883 FETCH (UID 2017 RFC822.SIZE 28785 BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "QUOTED-PRINTABLE" 546 6)("TEXT" "HTML" ("CHARSET" "ISO-8859-1") NIL NIL "QUOTED-PRINTABLE" 6106 76) "ALTERNATIVE") BODY[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)] {420}
From: John Doe <john.doe@somecompany.no>
To: Steinar BANG <steinar.bang@somecompany.no>
Date: Tue, 21 Jan 2014 08:13:04 +0100
Subject: SV: Hjemme med sykt barn
Message-ID: <701DABD47A1A7044999B70D8862952E339DA8A8829@SVVEPOST01.somecompany.no>
References: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
In-Reply-To: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
)
510 OK UID FETCH completed

Re: How to debug a new IMAP server type?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Yes, they are.  But when Gnus was left running, it created quite a
> number of these buffers.

Yeah, it should perhaps kill them off more aggressively, but having the
IMAP server drop the connection that often shouldn't really happen...

>> Could you post the output from the buffer before
>> `nnimap-transform-headers' is called?
>
> * 883 FETCH (UID 2017 RFC822.SIZE 28785 BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "QUOTED-PRINTABLE" 546 6)("TEXT" "HTML" ("CHARSET" "ISO-8859-1") NIL NIL "QUOTED-PRINTABLE" 6106 76) "ALTERNATIVE") BODY[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)] {420}
> From: John Doe <john.doe@somecompany.no>
> To: Steinar BANG <steinar.bang@somecompany.no>
> Date: Tue, 21 Jan 2014 08:13:04 +0100
> Subject: SV: Hjemme med sykt barn
> Message-ID: <701DABD47A1A7044999B70D8862952E339DA8A8829@SVVEPOST01.somecompany.no>
> References: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
> In-Reply-To: <82a64bae86d240bf9076e8263d6fd2dc@DB3PR07MB092.eurprd07.prod.outlook.com>
> )
> 510 OK UID FETCH completed

Hm.  No GOODBYE this time?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: How to debug a new IMAP server type?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

>> 510 OK UID FETCH completed

> Hm.  No GOODBYE this time?

Not immediately.  This content was grabbed immediately after entering
the group.

The GOODBYE appears after a period of time

Re: How to debug a new IMAP server type?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> The GOODBYE appears after a period of time

Aha.

I think I've found the problem now.  Davmail doesn't include an empty
line after the headers, which all the other IMAP servers do.  (It's not
a bug in Davmail or anything -- nnimap parses this a bit too sloppily.)

Try it now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: How to debug a new IMAP server type?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> I think I've found the problem now.  Davmail doesn't include an empty
> line after the headers, which all the other IMAP servers do.  (It's not
> a bug in Davmail or anything -- nnimap parses this a bit too sloppily.)

> Try it now.

It worked fine now:
 - Entering the INBOX showed the one message as a ticked message (I
   hadn't ticked it intentionally, but I guess it must have been Gnus
   setting a particular mark)
 - I then entered the group after first doing `C-u 100' and I saw the
   last 100 articles in the INBOX in blue, entering the articles worked
   fine 

So now Gnus again works with davmail.  Thanks! :-)

Note to others: Davmail nnimap folders seriously benefits from being
agentized. 

Was the davmail fix synced to emacs bzr? (Was: How to debug a new IMAP server type?)

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

>> I think I've found the problem now.  Davmail doesn't include an empty
>> line after the headers, which all the other IMAP servers do.  (It's
>> not a bug in Davmail or anything -- nnimap parses this a bit too
>> sloppily.)

>> Try it now.

> It worked fine now:
[snip!]
> So now Gnus again works with davmail.  Thanks! :-)

Did the davmail fix make it into emacs bzr?
 http://git.gnus.org/cgit/gnus.git/commit/?id=54c8699609d199dd2388f69581f7b7e1e5850c14

And if it was synced over: will it be in the Gnus of the next emacs
release?

Re: Was the davmail fix synced to emacs bzr?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Did the davmail fix make it into emacs bzr?

Yes, it's in bzr Emacs, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Was the davmail fix synced to emacs bzr?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Did the davmail fix make it into emacs bzr?

> Yes, it's in bzr Emacs, too.

Thanks!

Do you know if it will be part of the next released Emacs?  (whatever
version that is?)

Re: Was the davmail fix synced to emacs bzr?

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Do you know if it will be part of the next released Emacs?  (whatever
> version that is?)

Yes, it will.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Was the davmail fix synced to emacs bzr?

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Do you know if it will be part of the next released Emacs?  (whatever
>> version that is?)

> Yes, it will.

I visited the gmane.emacs.devel tonight to see if I could figure out
which version this was... and then I got stuck in the move-emacs-to-git
thread(s) of 9 to 8 weeks ago... some of it quite funny.  I even LOL'd
at least once.

(I liked your local-blame idea of 5 weeks agoi, though.  Finding the
history of a particular line is what I actually want)

I'm guessing, but not sure, that the version of emacs containing Gnus
with the davmail fix will be 24.4 (since accoring to
https://www.gnu.org/software/emacs/ the current version is 24.3).