More IMAP testing, please

More IMAP testing, please

[Lars Magne Ingebrigtsen]

We got a fair amount of nnimap testing and bug stomping yesterday, but
I'm sure there must be a lot more problems hiding here and there, so
it'd be great if more IMAP users took the plunge and gave the new nnimap
a whirl and report problems (or whether it works fine for your).

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new
> nnimap a whirl and report problems (or whether it works fine for
> your).

To all the fence sitters: it now works for me with a single dovecot and
agent.  And it is a _lot_ faster than the old version.  Especially at
startup and on `g' (it may be that there are other speedups here, in the
way gnus now scans groups over multiple servers.  I have a single nnimap
and 3 or 4 nntp servers, all agentized).

Re: More IMAP testing, please

[Andrew Cohen]

I'm obviously doing something stupid here...

I've downloaded the git repository, compiled the lisp, added it first in
my load-path and set up a gnus-secondary server using ssl. When I start gnus,
sometimes I get server denied and sometimes not. I've tried debugging
nnimap-open-connection when it fails, and I get this:


Unable to open server nnimap+test due to: Symbol's function definition is void:\
 setf


But why sometimes and not others? I would have thought that either setf
is defined at compile time or its not. And why doesn't anyone else have
this problem? My .emacs only has the load path and (require 'gnus-load)

Re: More IMAP testing, please

[Andrew Cohen]

Seems like the recent race condition fix corrects the problem.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> Unable to open server nnimap+test due to: Symbol's function definition is void:\
>  setf
>
> But why sometimes and not others? I would have thought that either setf
> is defined at compile time or its not. And why doesn't anyone else have
> this problem? My .emacs only has the load path and (require 'gnus-load)

That is odd.  nnimap has a

(eval-when-compile
  (require 'cl))

which I thought would do the trick, since setf is from cl-macs, and
isn't needed at run time.

If you do a

(require 'cl)

before starting Gnus, does that help?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Wes Hardaker]

>>>>> On Sun, 19 Sep 2010 17:41:50 +0200, Steinar Bang <sb@dod.no> said:

SB> To all the fence sitters: it now works for me with a single dovecot and
SB> agent.

I'm nervously looking forward to trying it soon.  Mail is my (work) life
and missing reading messages or worse losing messages would be the
death of me.  I'm sure that losing messages on the server isn't likely,
but if I miss seeing one (cause it's not in the summary buffer or, um, or...)
then I'd be in deep trouble.

So, maybe a statement like "the worst that could happen would be ..."
would push me over the fence assuming a decent ending to the statement ;-)
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Wes Hardaker <wes@hardakers.net> writes:

> So, maybe a statement like "the worst that could happen would be ..."
> would push me over the fence assuming a decent ending to the statement ;-)

If you don't do client-side splitting, then the worst that can happen is
that you lose article flags, as reported by Dan Christensen.  I haven't
been able to figure out how that could happen, though...

Well, even worse would be it not working at all.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Wes Hardaker <wes@hardakers.net> writes:
>
>> So, maybe a statement like "the worst that could happen would be ..."
>> would push me over the fence assuming a decent ending to the statement ;-)
>
> If you don't do client-side splitting, then the worst that can happen is
> that you lose article flags, as reported by Dan Christensen.  I haven't
> been able to figure out how that could happen, though...

Well, `B m' loses messages currently.  This seems like the most
important bug at this point.

Dan

Re: More IMAP testing, please

[Wes Hardaker]

>>>>> On Mon, 20 Sep 2010 12:24:10 -0400, Dan Christensen <jdc@uwo.ca> said:

DC> Well, `B m' loses messages currently.  This seems like the most
DC> important bug at this point.

Ahh...  And I use that all the time since that's how I get important
messages out of my "marked as spam" folder into my "no it's not, please
retrain on it" folder.
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: More IMAP testing, please

[Wes Hardaker]

>>>>> On Mon, 20 Sep 2010 18:14:23 +0200, Lars Magne Ingebrigtsen <larsi@gnus.org> said:

LMI> If you don't do client-side splitting, then the worst that can happen is
LMI> that you lose article flags, as reported by Dan Christensen.  I haven't
LMI> been able to figure out how that could happen, though...

Losing flags would suck, but can at least be recovered from.  The
biggest issue is that when I lose flags I have to resort everything.

Now...  if there was just a way to use the new nnimap for some folders,
and the older for some more important ones :-)

LMI> Well, even worse would be it not working at all.  :-)

Nah...  I've actually tagged my current git pull with "works" so that I
can fall back to it if need be after my next (very scary) pull.
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: More IMAP testing, please

[Ralf Angeli]

* Lars Magne Ingebrigtsen (2010-09-19) writes:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

I just tried it with imap.web.de which did not work with the old
nnimap.el.  There should be some messages about web.de in the ding
archives.  IIRC the problem has been due to the article numbers being
very large on this server.

With the new code I could actually see the contents of my INBOX.  It had
about 20-something mails with ads and one legitimate one which was the
newest.  After deleting the ads and reading the legitimate mail I left
the group and tried to enter it again while asking Gnus to show me the
last 11 mails.  That's when it went nuts. (c:  While using 100% of the
CPU it sucked up all my memory in less than a minute.  That's when X
froze and I had to do a hard reboot.  It's been some time since this
happened the last time. (c:

Okay I tried this again and got the following backtrace with
`debug-on-quit' enabled, when I interrupted the loop:

Debugger entered--Lisp error: (quit)
  gnus-uncompress-range((146430175 . 688002009))
  gnus-articles-to-read("nnimap+web.de:INBOX" 11)
  gnus-select-newsgroup("nnimap+web.de:INBOX" 11 nil)
  gnus-summary-read-group-1("nnimap+web.de:INBOX" 11 t nil nil nil)
  gnus-summary-read-group("nnimap+web.de:INBOX" 11 t nil nil nil nil)
  gnus-group-read-group(11 t)
  gnus-group-select-group(11)
  gnus-topic-select-group(11)
  call-interactively(gnus-topic-select-group nil nil)

Let me know what I can do for further debugging.

-- 
Ralf

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ralf Angeli <angeli@caeruleus.net> writes:

> After deleting the ads and reading the legitimate mail I left
> the group and tried to enter it again while asking Gnus to show me the
> last 11 mails.  That's when it went nuts. (c:  While using 100% of the
> CPU it sucked up all my memory in less than a minute.

Oh, dear.

> That's when X froze and I had to do a hard reboot.

Wow. 

> Okay I tried this again and got the following backtrace with
> `debug-on-quit' enabled, when I interrupted the loop:
>
> Debugger entered--Lisp error: (quit)
>   gnus-uncompress-range((146430175 . 688002009))
>   gnus-articles-to-read("nnimap+web.de:INBOX" 11)

Yeah, that's not going to work.  Gnus decompresses some of the ranges
before group selection, and when they are big like that, it's going to
totally kill your Emacs (and your machine).

My longer-term plans involve stopping all range
decompression/compression, and working directly with the ranges.  But
that's a rather big undertaking, so it's not going to happen
immediately.  In the mean time, you still can't use nnimap to read
messages from that server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ralf Angeli]

* Lars Magne Ingebrigtsen (2010-09-19) writes:

> Ralf Angeli <angeli@caeruleus.net> writes:
>
>> Debugger entered--Lisp error: (quit)
>>   gnus-uncompress-range((146430175 . 688002009))
>>   gnus-articles-to-read("nnimap+web.de:INBOX" 11)
>
> Yeah, that's not going to work.  Gnus decompresses some of the ranges
> before group selection, and when they are big like that, it's going to
> totally kill your Emacs (and your machine).
>
> My longer-term plans involve stopping all range
> decompression/compression, and working directly with the ranges.  But
> that's a rather big undertaking, so it's not going to happen
> immediately.  In the mean time, you still can't use nnimap to read
> messages from that server.

Bummer.  But it's just a fall-back account which I can access with a
not-to-be-named alternative MUA, so no problem. (c:

-- 
Ralf

Re: More IMAP testing, please

[Manoj Srivastava]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

        It is working fine for a gmail account and a Courier-IMAP server
 over TLS here.

        manoj
-- 
Dieters live life in the fasting lane.
Manoj Srivastava <srivasta@acm.org> <http://www.golden-gryphon.com/>  
4096R/C5779A1C E37E 5EC5 2A01 DA25 AD20  05B6 CF48 9438 C577 9A1C

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 2754 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

I was reading manoj mail. I pressed ^.
The parent (your (Lars) mail) was not visible in the summary.

This is the error I got:

Debugger entered--Lisp error: (wrong-type-argument listp t)
  gnus-read-header("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  gnus-summary-insert-subject("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  byte-code("\304 \305\x18\211\x19\203-\0	@\x10\306\b!\203&\0\307\n!\211\x13\203&\0\310\305\211\211\v$\210\311\312\313\"\210	A\211\x11\204	\0*\314\315\316\n#\207" [gnus-override-method --cl-dolist-temp-- message-id number gnus-refer-article-methods nil gnus-check-server gnus-summary-insert-subject gnus-summary-select-article throw found t gnus-message 3 "Couldn't fetch article %s"] 6)
  gnus-summary-refer-article("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  gnus-summary-refer-parent-article(1)
  call-interactively(gnus-summary-refer-parent-article nil nil)
  recursive-edit()
  byte-code("\306\x10	@\307=\203!\0\310\311\312\"\210\313\311!\211\x1aA@)\242\314=\203!\0\310\315\312\"\210\316\v!\210\317 \210\320	!\210\f\203d\0\321ed\"\rV\203W\0eb\210\322\r\245y\210`\x1e^[db\210\322\r\245\rZy\210\x0e^[`|\210)\323c\210eb\210\324\325\326 \"\210\327\306!\210\324\330!\210\331\312\x1e\x1c\x1e\x1d\324\330!\210\212\332 \210+\331\207" [unread-command-char debugger-args x debugger-buffer noninteractive debugger-batch-max-lines -1 debug backtrace-debug 4 t backtrace-frame lambda 5 pop-to-buffer debugger-mode debugger-setup-buffer count-lines 2 "...\n" message "%s" buffer-string kill-emacs "" nil recursive-edit middlestart buffer-read-only standard-output] 4)
  debug(error (wrong-type-argument listp t))
  gnus-read-header("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  gnus-summary-insert-subject("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  byte-code("\304 \305\x18\211\x19\203-\0	@\x10\306\b!\203&\0\307\n!\211\x13\203&\0\310\305\211\211\v$\210\311\312\313\"\210	A\211\x11\204	\0*\314\315\316\n#\207" [gnus-override-method --cl-dolist-temp-- message-id number gnus-refer-article-methods nil gnus-check-server gnus-summary-insert-subject gnus-summary-select-article throw found t gnus-message 3 "Couldn't fetch article %s"] 6)
  gnus-summary-refer-article("<m3lj6xx4mn.fsf@quimbies.gnus.org>")
  gnus-summary-refer-parent-article(1)
  call-interactively(gnus-summary-refer-parent-article nil nil)


If I do press `/ o 100', the parent mail become visible and it works as
expected.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I was reading manoj mail. I pressed ^.
> The parent (your (Lars) mail) was not visible in the summary.
>
> This is the error I got:
>
> Debugger entered--Lisp error: (wrong-type-argument listp t)
>   gnus-read-header("<m3lj6xx4mn.fsf@quimbies.gnus.org>")

Fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> Fixed and pushed.

I confirm.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

On Sun, Sep 19 2010, Julien Danjou wrote:

I've got another problem, and I don't know where the hell it come from.

My IMAP server is on localhost.

I hit that while running M-x gnus:

(defun gnus-start-news-server
[…]
       (gnus-y-or-n-p
	(format
	 "%s (%s) open error: '%s'.  Continue? "
	 (car gnus-select-method) (cadr gnus-select-method)
	 (gnus-status-message gnus-select-method)))

I press 'n', and retry, and it works the 2nd or third time. If I
debug-on-entry, everything is fine. I wonder if there's not a race
condition somewhere…

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I hit that while running M-x gnus:
>
> (defun gnus-start-news-server
> […]
>        (gnus-y-or-n-p
> 	(format
> 	 "%s (%s) open error: '%s'.  Continue? "
> 	 (car gnus-select-method) (cadr gnus-select-method)
> 	 (gnus-status-message gnus-select-method)))
>
> I press 'n', and retry, and it works the 2nd or third time. If I
> debug-on-entry, everything is fine. I wonder if there's not a race
> condition somewhere…

That's definitely a possibility.  But it might be something else.  If
you `C-g' out of the question, what's in the "*imap <server> ...*
buffer?  (There might be more of them -- find the most recent one for
the server in question.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> That's definitely a possibility.  But it might be something else.  If
> you `C-g' out of the question, what's in the "*imap <server> ...*
> buffer?  (There might be more of them -- find the most recent one for
> the server in question.)

I don't have such a buffer when hitting C-g here.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

>> That's definitely a possibility.  But it might be something else.  If
>> you `C-g' out of the question, what's in the "*imap <server> ...*
>> buffer?  (There might be more of them -- find the most recent one for
>> the server in question.)
>
> I don't have such a buffer when hitting C-g here.

Sorry, *nnimap <stuff>*...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>>> That's definitely a possibility.  But it might be something else.  If
>>> you `C-g' out of the question, what's in the "*imap <server> ...*
>>> buffer?  (There might be more of them -- find the most recent one for
>>> the server in question.)
>>
>> I don't have such a buffer when hitting C-g here.
>
> Sorry, *nnimap <stuff>*...

Well I'm sorry, I should have wrote: I don't have any buffer matching
anything that look like imap/nntp/gnus or anything else. :-(

So they either never existed or they have been destroyed at the time the
error splash on my face. :-(

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Well I'm sorry, I should have wrote: I don't have any buffer matching
> anything that look like imap/nntp/gnus or anything else. :-(
>
> So they either never existed or they have been destroyed at the time the
> error splash on my face. :-(

Yes, the buffers are killed once you exit Gnus, but they should be
created immediately when trying to talk to an IMAP server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> Yes, the buffers are killed once you exit Gnus, but they should be
> created immediately when trying to talk to an IMAP server.

I've cheated like that:

% git diff
diff --git a/lisp/gnus-int.el b/lisp/gnus-int.el
index f245907..5d72880 100644
--- a/lisp/gnus-int.el
+++ b/lisp/gnus-int.el
@@ -146,6 +146,7 @@ If CONFIRM is non-nil, the user will be asked for an NNTP server."
        (gnus-server-opened gnus-select-method)
        (gnus-open-server gnus-select-method)
        gnus-batch-mode
+       (error "AAAHH")
        (gnus-y-or-n-p
        (format
         "%s (%s) open error: '%s'.  Continue? "

M-x toggle-debug-on-error

I managed to get the *nnimap* buffer! How disapointing:

Resolving 'keller.adm.naquadah.org'...
Connecting to '127.0.1.1:993'...
- Ephemeral Diffie-Hellman parameters
 - Using prime: 1024 bits
 - Secret key: 1021 bits
 - Peer's public key: 1023 bits
- Certificate type: X.509
 - Got a certificate list of 1 certificates.
 - Certificate[0] info:
  - subject `O=Dovecot mail server,OU=keller.adm.naquadah.org,CN=keller.adm.naquadah.org,EMAIL=root@keller.adm.naquadah.org', issuer `O=Dovecot mail server,OU=keller.adm.naquadah.org,CN=keller.adm.naquadah.org,EMAIL=root@keller.adm.naquadah.org', RSA key 1024 bits, signed using RSA-SHA, activated `2010-08-31 13:49:12 UTC', expires `2020-08-30 13:49:12 UTC', SHA-1 fingerprint `6279c93fc1cf5eb740420267f21a09de19e72f1c'
- The hostname in the certificate matches 'keller.adm.naquadah.org'.
- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted
- Version: TLS1.0
- Key Exchange: DHE-RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
- Handshake was completed

- Simple Client Mode:

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.


-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.

Hm.

Well, try evaling this:

(nnimap-open-server "Mail" '((nnimap-address "keller.adm.naquadah.org")))

and see what happens.  If it returns nil, it means that it thinks that
it couldn't contact the server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> (nnimap-open-server "Mail" '((nnimap-address "keller.adm.naquadah.org")))
>
> and see what happens.  If it returns nil, it means that it thinks that
> it couldn't contact the server.

It returns t right now.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

On Sun, Sep 19 2010, Julien Danjou wrote:

> It returns t right now.

Then I quitted Gnus. Pasted the line in *scratch*, and C-x C-e.

First execution: started a process, returned t.
Second execution: returned process imap
Third execution: returned process imap
etc.

Killed the nnimap buffer. Retried. Same result.
Killed the nnimap buffer. Retried. Same result.
…

Then:
First execution: returned nil.

Really really weird. :(
-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Then:
> First execution: returned nil.
>
> Really really weird. :(

The stuff in the *nnimap ...* buffers is the same whether you get a nil
or a t?  If that's the case, then it has to be a race condition.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> Then:
>> First execution: returned nil.
>>
>> Really really weird. :(
>
> The stuff in the *nnimap ...* buffers is the same whether you get a nil
> or a t?  If that's the case, then it has to be a race condition.

No.

If it returns t I got:
* ENABLED QRESYNC
16034 OK Enabled.

If it's nil, I got:
Resolving 'keller.adm.naquadah.org'...
Connecting to '127.0.1.1:993'...
- Ephemeral Diffie-Hellman parameters
 - Using prime: 1024 bits
 - Secret key: 1023 bits
 - Peer's public key: 1023 bits
- Certificate type: X.509
 - Got a certificate list of 1 certificates.
 - Certificate[0] info:
  - subject `O=Dovecot mail server,OU=keller.adm.naquadah.org,CN=keller.adm.naquadah.org,EMAIL=root@keller.adm.naquadah.org', issuer `O=Dovecot mail server,OU=keller.adm.naquadah.org,CN=keller.adm.naquadah.org,EMAIL=root@keller.adm.naquadah.org', RSA key 1024 bits, signed using RSA-SHA, activated `2010-08-31 13:49:12 UTC', expires `2020-08-30 13:49:12 UTC', SHA-1 fingerprint `6279c93fc1cf5eb740420267f21a09de19e72f1c'
- The hostname in the certificate matches 'keller.adm.naquadah.org'.
- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted
- Version: TLS1.0
- Key Exchange: DHE-RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
- Handshake was completed

- Simple Client Mode:

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready..

I think I found it.  There's indeed a race condition there.  I'll test
it a bit more and push it out in a few minutes.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> I think I found it.  There's indeed a race condition there.  I'll test
> it a bit more and push it out in a few minutes.

It seems fixed indeed, thanks Lars!

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready..

Oh, this means that it's contacted the server, but haven't sent any
login credentials.  Looking at the code, it looks like this would be the
result of nnimap can't find any credentials.  But it should prompt you
in that case.  I'll try to reproduce.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, this means that it's contacted the server, but haven't sent any
> login credentials.  Looking at the code, it looks like this would be the
> result of nnimap can't find any credentials.  But it should prompt you
> in that case.  I'll try to reproduce.

Wasn't able to.  But it has to be something in the

                            (nnimap-credentials
			      nnimap-address
			      (if nnimap-server-port
				  (cons (format "%s" nnimap-server-port) ports)
				ports))

bit that returns nil...
                                
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On Sun, Sep 19 2010, Julien Danjou wrote:

> Well I'm sorry, I should have wrote: I don't have any buffer matching
> anything that look like imap/nntp/gnus or anything else. :-(
>
> So they either never existed or they have been destroyed at the time the
> error splash on my face. :-(

gnus-opened-servers is a variable defined in `gnus.el'.
Its value is 
(((nnimap "Mail"
          (nnimap-address "keller.adm.naquadah.org"))
  denied))

So at least I know that the problem is `denied'.

Goin' deeper now…

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> gnus-opened-servers is a variable defined in `gnus.el'.
> Its value is 
> (((nnimap "Mail"
>           (nnimap-address "keller.adm.naquadah.org"))
>   denied))
>
> So at least I know that the problem is `denied'.
>
> Goin' deeper now…

If a server is marked as denied, that means that it failed to open it.
If you go to the server buffer and say `C' on it to close it, a `g' in
the group buffer should re-request it again.  Probably.  Or just restart
Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> If a server is marked as denied, that means that it failed to open it.
> If you go to the server buffer and say `C' on it to close it, a `g' in
> the group buffer should re-request it again.  Probably.  Or just restart
> Gnus.

By the way, `C' does nothing on my IMAP server in the server buffer. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> By the way, `C' does nothing on my IMAP server in the server buffer. :-)

If it said "denied", `C' should change that to "closed"...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

On Sun, Sep 19 2010, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> By the way, `C' does nothing on my IMAP server in the server buffer. :-)
>
> If it said "denied", `C' should change that to "closed"...

No, it's opened right now. But it seems to be a noop:

(deffoo nnimap-close-server (&optional server)
  t)

No?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> No, it's opened right now. But it seems to be a noop:
>
> (deffoo nnimap-close-server (&optional server)
>   t)
>
> No?

Yes, the only thing it does is remove the `denied' thing, if it is
denied.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[David Engster]

Lars Magne Ingebrigtsen writes:
> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

Server: Dovecot Debian stable.

* I'm running my IMAP server on a non-standard port; while this is
  respected for open-tls-stream, you always use default ports for
  querying auth-resources.

* First startup took very long. I guess this is due to 

  19:11:56 14 UID FETCH 59:* FLAGS

  on every mailbox. If I remember correctly, the old code did only
  EXAMINE? Now it's pretty fast, maybe because Dovecot does some
  caching here.

* If I try to get an article's parent, and this parent wasn't requested
  in the first batch, I get the following error"

Debugger entered--Lisp error: (wrong-type-argument listp t)
  gnus-read-header("<m3iq21zzqy.fsf@quimbies.gnus.org>")
  gnus-summary-insert-subject("<m3iq21zzqy.fsf@quimbies.gnus.org>")
  byte-code("\304 \305\211\203-	@\306\b!\203&\307\n!\211\203&\310\305\211\211\v$\210\311\312\313\"\210	A\211\204	*\314\315\316\n#\207" [gnus-override-method --cl-dolist-temp-- message-id number gnus-refer-article-methods nil gnus-check-server gnus-summary-insert-subject gnus-summary-select-article throw found t gnus-message 3 "Couldn't fetch article %s"] 6)
  gnus-summary-refer-article("<m3iq21zzqy.fsf@quimbies.gnus.org>")
  gnus-summary-refer-parent-article(1)
  call-interactively(gnus-summary-refer-parent-article nil nil)

Otherwise - wow, really fast. :-)

-David

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> * I'm running my IMAP server on a non-standard port; while this is
>   respected for open-tls-stream, you always use default ports for
>   querying auth-resources.

Fixed and pushed.

> * First startup took very long. I guess this is due to 
>
>   19:11:56 14 UID FETCH 59:* FLAGS
>
>   on every mailbox. If I remember correctly, the old code did only
>   EXAMINE? Now it's pretty fast, maybe because Dovecot does some
>   caching here.

The first time you run it, it does a total flag rescan.  Subsequent
times it does partial flag rescans, so it'll only be slow the very first
time you use it.

> * If I try to get an article's parent, and this parent wasn't requested
>   in the first batch, I get the following error"
>
> Debugger entered--Lisp error: (wrong-type-argument listp t)
>   gnus-read-header("<m3iq21zzqy.fsf@quimbies.gnus.org>")

Right...  I'll look into it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[David Engster]

Lars Magne Ingebrigtsen writes:
> David Engster <deng@randomsample.de> writes:
>
>> * I'm running my IMAP server on a non-standard port; while this is
>>   respected for open-tls-stream, you always use default ports for
>>   querying auth-resources.
>
> Fixed and pushed.

Thanks, it works now.

>> * First startup took very long. I guess this is due to 
>>
>>   19:11:56 14 UID FETCH 59:* FLAGS
>>
>>   on every mailbox. If I remember correctly, the old code did only
>>   EXAMINE? Now it's pretty fast, maybe because Dovecot does some
>>   caching here.
>
> The first time you run it, it does a total flag rescan.  Subsequent
> times it does partial flag rescans, so it'll only be slow the very first
> time you use it.

Ah, OK. First startup easily took 30 seconds or so. I think a message
like 'Don't hit C-g just yet I'm doing serious work here' would be helpful.

Still, while entering/reading articles is way faster, 'g' is a tad
slower than it used to be, at least for me.

>> * If I try to get an article's parent, and this parent wasn't requested
>>   in the first batch, I get the following error"
>>
>> Debugger entered--Lisp error: (wrong-type-argument listp t)
>>   gnus-read-header("<m3iq21zzqy.fsf@quimbies.gnus.org>")
>
> Right...  I'll look into it.

Oh, and hitting 'A T' yields "gnus-summary-refer-thread: Can't fetch
thread from back ends that don't support NOV". I'm pretty sure this used
to work...?

-David

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Ah, OK. First startup easily took 30 seconds or so. I think a message
> like 'Don't hit C-g just yet I'm doing serious work here' would be helpful.

Yup.

> Still, while entering/reading articles is way faster, 'g' is a tad
> slower than it used to be, at least for me.

What's your setup?  Many different IMAP/nntp servers, or just one?  And
how many IMAP mailboxes?  And what's the IMAP server type?

> Oh, and hitting 'A T' yields "gnus-summary-refer-thread: Can't fetch
> thread from back ends that don't support NOV". I'm pretty sure this used
> to work...?

Heh.  Hadn't thought about that one.  Yes, nnimap no longer serves out
NOV headers, since we're getting normal headers from the server, and
first converting them to NOV and then parsing the NOV seemed like waste
to me.

I think the right fix here is to change `A T' so that it works on
non-NOV backends, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[David Engster]

Lars Magne Ingebrigtsen writes:
> David Engster <deng@randomsample.de> writes:
>> Still, while entering/reading articles is way faster, 'g' is a tad
>> slower than it used to be, at least for me.
>
> What's your setup?  Many different IMAP/nntp servers, or just one?  And
> how many IMAP mailboxes?  And what's the IMAP server type?

'g' queries 15 mailboxes on one IMAP server, which is Dovecot Debian
stable via network/ssl on a virtual server. This takes maybe 1 second,
which is not really a problem, but the old code was faster (since it
only did EXAMINE and not FLAG fetching, as far as I remember).

> I think the right fix here is to change `A T' so that it works on
> non-NOV backends, too.

Yeah well, it's not terribly important at the moment.

Oh, and nnmairix is broken with nnimap now since you haven't implemented
nnimap-request-delete-group (yet). ;-)

-David

Re: More IMAP testing, please

[Dan Christensen]

David Engster <deng@randomsample.de> writes:

> Oh, and nnmairix is broken with nnimap now since you haven't implemented
> nnimap-request-delete-group (yet). ;-)

Is this hard to fix?  For me, the error I get is:

Debugger entered--Lisp error: (void-function nnimap-request-delete-group)
  nnimap-request-delete-group("zz_mairix-nnmairixsearch-1" t "rocky")
  apply(nnimap-request-delete-group ("zz_mairix-nnmairixsearch-1" t "rocky"))
  nnmairix-call-backend("request-delete-group" "zz_mairix-nnmairixsearch-1" t "rocky")
  nnmairix-delete-recreate-group("nnmairix+mairixserver:nnmairixsearch")
  nnmairix-update-and-clear-marks("nnmairix+mairixserver:nnmairixsearch")
  nnmairix-search("test")
  call-interactively(nnmairix-search t nil)
  execute-extended-command(nil)
  call-interactively(execute-extended-command nil nil)

Also, if I try to open an existing search group, I get:

nnmairix-request-group-with-article-number-correction: Search failed: "^Matched.*messages"

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Debugger entered--Lisp error: (void-function nnimap-request-delete-group)
>   nnimap-request-delete-group("zz_mairix-nnmairixsearch-1" t "rocky")

I've now implemented this function.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Debugger entered--Lisp error: (void-function nnimap-request-delete-group)
>>   nnimap-request-delete-group("zz_mairix-nnmairixsearch-1" t "rocky")
>
> I've now implemented this function.

Well, now it seems like nnmairix needs nnimap-request-create-group...

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Well, now it seems like nnmairix needs nnimap-request-create-group...

This is implemented now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[David Engster]

David Engster writes:
> Lars Magne Ingebrigtsen writes:
>> David Engster <deng@randomsample.de> writes:
>>> Still, while entering/reading articles is way faster, 'g' is a tad
>>> slower than it used to be, at least for me.
>>
>> What's your setup?  Many different IMAP/nntp servers, or just one?  And
>> how many IMAP mailboxes?  And what's the IMAP server type?
>
> 'g' queries 15 mailboxes on one IMAP server, which is Dovecot Debian
> stable via network/ssl on a virtual server. This takes maybe 1 second,
> which is not really a problem, but the old code was faster (since it
> only did EXAMINE and not FLAG fetching, as far as I remember).

Here are some numbers to back this up: Running (gnus-group-get-new-news)
50 times takes ~15sec with the old nnimap, ~21secs with the new one. The
interesting bit is that the old code took only ~10 GCs, while the new
one needs ~120, which take ~6 seconds, so that's your time difference
right there.

-David

Re: More IMAP testing, please

[David Engster]

David Engster writes:
> David Engster writes:
>> Lars Magne Ingebrigtsen writes:
>>> David Engster <deng@randomsample.de> writes:
>>>> Still, while entering/reading articles is way faster, 'g' is a tad
>>>> slower than it used to be, at least for me.
>>>
>>> What's your setup?  Many different IMAP/nntp servers, or just one?  And
>>> how many IMAP mailboxes?  And what's the IMAP server type?
>>
>> 'g' queries 15 mailboxes on one IMAP server, which is Dovecot Debian
>> stable via network/ssl on a virtual server. This takes maybe 1 second,
>> which is not really a problem, but the old code was faster (since it
>> only did EXAMINE and not FLAG fetching, as far as I remember).
>
> Here are some numbers to back this up: Running (gnus-group-get-new-news)
> 50 times takes ~15sec with the old nnimap, ~21secs with the new one. The
> interesting bit is that the old code took only ~10 GCs, while the new
> one needs ~120, which take ~6 seconds, so that's your time difference
> right there.

I forgot to mention that I did test the old nnimap with an older git
checkout, so it's not only nnimap but also all the other code that's
changed in the back end which might add to the GCs.

-David

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Here are some numbers to back this up: Running (gnus-group-get-new-news)
> 50 times takes ~15sec with the old nnimap, ~21secs with the new one. The
> interesting bit is that the old code took only ~10 GCs, while the new
> one needs ~120, which take ~6 seconds, so that's your time difference
> right there.

Right.  I've had another look at the flags parsing code, and I think
it's as inefficient as it's possible to be.  :-)  I'll rewrite it to
collect less garbage and stuff later tonight.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

I previously used a test account and only now tried to switch my real
Gnus. However it silently fails:

Opening TLS connection to `ltw.loris.tv'...
Opening TLS connection with `gnutls-cli -p imaps ltw.loris.tv'...failed
Opening TLS connection with `gnutls-cli -p imaps ltw.loris.tv
--protocols ssl3'...failed
Opening TLS connection with `openssl s_client -connect
ltw.loris.tv:imaps -no_ssl2 -ign_eof'...done
Opening TLS connection to `ltw.loris.tv'...done
Couldn't open nnimap:ltw

First: Shouldn't the user get a message that opening the server failed?
I can only see in server buffer that status is denied, but neither
during start up nor in Group buffer is the failed state visible.

How do I debug this? I have a buffer named
*nnimap ltw.loris.tv nil  *nntpd**
With the text

2 BAD Missing '"'^M                                                                                                                          
Process *nnimap* killed

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> First: Shouldn't the user get a message that opening the server failed?
> I can only see in server buffer that status is denied, but neither
> during start up nor in Group buffer is the failed state visible.

Yes, the error should be more noticeable, but I'm not sure how to do
that without annoying users.  Most of us have, I'm sure, a server or two
in Gnus that don't respond. 

> How do I debug this? I have a buffer named
> *nnimap ltw.loris.tv nil  *nntpd**
> With the text
>
> 2 BAD Missing '"'^M

Have a look in the *imap log* buffer to see what it's trying to send.
From the look of the error message, it might be a quoting issue.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> *nnimap ltw.loris.tv nil  *nntpd**
>> With the text
>>
>> 2 BAD Missing '"'^M
>
> Have a look in the *imap log* buffer to see what it's trying to send.
> From the look of the error message, it might be a quoting issue.

19:59:06 1 LOGIN "frank.schmitt@gmail.com" "XXXXXXXXXXXXXXXXXXX^M"^M
19:59:06 2 LOGIN "fschmitt" "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX^M"^M

The ^M after the password looks broken. My ~/.authinfo had DOS line
endings while I run under Unix. After saving ~/.authinfo with Unix
line endings it works.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> 19:59:06 1 LOGIN "frank.schmitt@gmail.com" "XXXXXXXXXXXXXXXXXXX^M"^M
> 19:59:06 2 LOGIN "fschmitt" "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX^M"^M
>
> The ^M after the password looks broken. My ~/.authinfo had DOS line
> endings while I run under Unix. After saving ~/.authinfo with Unix
> line endings it works.

So that means that .netrc is parsing the lines kinda wrong.  I think.
Is there a convention for opening files in Emacs that may possibly have
been saved in DOS mode?  Otherwise the netrc.el parsing function may
just want to strip the ^M before parsing.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is there a convention for opening files in Emacs that may possibly have
> been saved in DOS mode?

Should be detected automatically when line ending is consistent.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Should be detected automatically when line ending is consistent.

netrc just does an `insert-file-contents'.  Does it do it do?  Well, I
guess so, since `insert-file-contents-literally' exists...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> Should be detected automatically when line ending is consistent.
>
> netrc just does an `insert-file-contents'.  Does it do it do?  Well, I
> guess so, since `insert-file-contents-literally' exists...

BTW, this is still broken. I just upgraded to Git gnus on an other
machine with my old .authinfo which has consistent DOS line endings and
it still sends the ^M as part of the password.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: More IMAP testing, please

[Ted Zlatanov]

On Fri, 24 Sep 2010 09:53:47 +0200 Frank Schmitt <ich@frank-schmitt.net> wrote: 

FS> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>> 
>>> Should be detected automatically when line ending is consistent.
>> 
>> netrc just does an `insert-file-contents'.  Does it do it do?  Well, I
>> guess so, since `insert-file-contents-literally' exists...

FS> BTW, this is still broken. I just upgraded to Git gnus on an other
FS> machine with my old .authinfo which has consistent DOS line endings and
FS> it still sends the ^M as part of the password.

Are there ^M characters in this for you?

      (with-temp-buffer
        (insert-file-contents "~/.netrc")
        (buffer-string))  

because that's what netrc.el does.  The decoding and line conversion are
entirely handled by Emacs with all the settings defaulted.  The
`coding-system-for-read' variable, in particular, is not set.  Check
your `file-coding-system-alist' to be sure you don't have any overrides.

Ted

Re: More IMAP testing, please

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

In the server buffer of my gmail account, I can see one group for each
Gmail label as expected, but I can't find the groups [Gmail]/Spam and
[Gmail]/Trash which I can see, e.g., with Sylpheed Claws.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> In the server buffer of my gmail account, I can see one group for each
> Gmail label as expected, but I can't find the groups [Gmail]/Spam and
> [Gmail]/Trash which I can see, e.g., with Sylpheed Claws.

The [] characters were part of the set Gnus ignored in group names.
I've now removed that -- I don't think it'll break anything in Gnus, and
it works for me.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

I've just upgraded from a pre-git gnus (Dec 2009, I think) to the latest
git version, and I'm having some problems with my imap groups.  The
server in question is dovecot on localhost:

   '(nnimap "rocky"
	    (nnimap-address "localhost")
	    (nnimap-server-port 143)
	    (nnimap-stream network)
	    ; Summary generation is faster with this t:
	    (nnimap-nov-is-evil t)

My largest group has about 6500 articles, but after a few minutes of
using gnus, gnus would only show me the newest 71 articles (which seem
to be the ones whose article numbers are within 100 of the latest).
`M-g' on the group fixed this, but then I found I had lost *all*
marks on the older articles (tick, expired, answered definitely
lost, read not lost).

This also happened in my Inbox, a smaller group.

This is a very serious problem for me, and I'm happy to provide
whatever information I can to debug it.

A less important problem: for some reason, two of the groups on this
server show up with "*" instead of the unread article count.  I think
these are on the only two groups that are empty.  If I hit `M g', my
summary buffer shows 1 unseen and 1 unread.  But when I try to enter the
groups, gnus notices that there aren't any articles.  After that, `M g'
shows 0 unread and unseen.

Good news:  haven't noticed problems with two nntp servers or with
a remote imap server using "ssh %s dovecot --exec-mail imap", although I
haven't tested either very much yet.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> My largest group has about 6500 articles, but after a few minutes of
> using gnus, gnus would only show me the newest 71 articles (which seem
> to be the ones whose article numbers are within 100 of the latest).
> `M-g' on the group fixed this, but then I found I had lost *all*
> marks on the older articles (tick, expired, answered definitely
> lost, read not lost).

That's not good.

To take the last problem first, Gnus will only send over updates of
marks when it thinks a mark has changed.  So if Gnus doesn't think that
article 423 has a tick, it won't de-flag the article on the server,
either -- it should only send over updates of articles that has changed
status while in the summary buffer.

I think.

So I don't understand that at all.

The former problem sounds like a parsing problem of some kind, but since
`M-g' is able to parse the data, I'm not sure how to interpret that...

> A less important problem: for some reason, two of the groups on this
> server show up with "*" instead of the unread article count.  I think
> these are on the only two groups that are empty.  If I hit `M g', my
> summary buffer shows 1 unseen and 1 unread.  But when I try to enter the
> groups, gnus notices that there aren't any articles.  After that, `M g'
> shows 0 unread and unseen.

Sounds like an off-by-one error on empty groups.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> My largest group has about 6500 articles, but after a few minutes of
>> using gnus, gnus would only show me the newest 71 articles (which seem
>> to be the ones whose article numbers are within 100 of the latest).
>> `M-g' on the group fixed this, but then I found I had lost *all*
>> marks on the older articles (tick, expired, answered definitely
>> lost, read not lost).
>
> That's not good.
>
> To take the last problem first, Gnus will only send over updates of
> marks when it thinks a mark has changed.  So if Gnus doesn't think that
> article 423 has a tick, it won't de-flag the article on the server,
> either -- it should only send over updates of articles that has changed
> status while in the summary buffer.
>
> I think.
>
> So I don't understand that at all.

I've verified that these flags were removed from the articles on the
server (a total of more than 1000 flags).  And this problem happened
with two different imap groups, at the same time.

All flags except read ("Seen", I think) were removed: Answered, Flagged,
gnus-expire, gnus-save and gnus-forward were the ones that were there.

> The former problem sounds like a parsing problem of some kind, but since
> `M-g' is able to parse the data, I'm not sure how to interpret that...

To me it sounds like a problem involving the strategy of sometimes only
getting mark information for the last 100 articles, and sometimes
getting it for all articles.

So far it hasn't happened again, but I'm still in the same emacs session
so I can provide whatever information you need.  I've included an except
from the *imap log* buffer below, and can send more if that would help.
My INBOX has about 80 articles, but many of them are quite old so their
article numbers aren't in the most recent 100 numbers.

>> A less important problem: for some reason, two of the groups on this
>> server show up with "*" instead of the unread article count.  I think
>> these are on the only two groups that are empty.  If I hit `M g', my
>> summary buffer shows 1 unseen and 1 unread.  But when I try to enter the
>> groups, gnus notices that there aren't any articles.  After that, `M g'
>> shows 0 unread and unseen.
>
> Sounds like an off-by-one error on empty groups.

This is repeatable, so let me know if there's anything I can do to help
debug it.

Dan

16:33:52 9 EXAMINE "INBOX"
16:33:52 10 UID FETCH 5459:* FLAGS
...
16:34:07 35 EXAMINE "INBOX"
16:34:07 36 UID FETCH 5459:* FLAGS
...
16:34:25 59 EXAMINE "INBOX"
16:34:25 60 UID FETCH 5459:* FLAGS
...
16:35:24 84 EXAMINE "INBOX"
16:35:24 85 UID FETCH 5459:* FLAGS
16:35:24 86 SELECT "INBOX"
16:35:24 87 SELECT "INBOX"
16:35:24 88 UID FETCH 5466:5559 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
16:35:34 89 UID STORE 4808,5398 -FLAGS.SILENT (gnus-forward)
16:35:34 90 UID STORE 5355,5358,5361:5362,5365,5372,5377:5378,5380:5381,5398,5400,5404,5409,5411,5416:5419,5421,5423,5425,5434:5437,5443 -FLAGS.SILENT (gnus-expire)
16:35:34 91 UID STORE 4357,4808,5358,5418:5419,5425,5436:5437 -FLAGS.SILENT (\Answered)
16:35:34 92 UID STORE 4781,4808,4813,5420 -FLAGS.SILENT (\Flagged)
...
16:38:41 113 EXAMINE "INBOX"
16:38:41 114 UID FETCH 1:* FLAGS
16:38:41 115 SELECT "INBOX"
16:38:41 116 SELECT "INBOX"
16:38:41 117 UID FETCH 123:5559 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> 16:35:24 84 EXAMINE "INBOX"
> 16:35:24 85 UID FETCH 5459:* FLAGS
> 16:35:24 86 SELECT "INBOX"
> 16:35:24 87 SELECT "INBOX"
> 16:35:24 88 UID FETCH 5466:5559 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
> 16:35:34 89 UID STORE 4808,5398 -FLAGS.SILENT (gnus-forward)
> 16:35:34 90 UID STORE 5355,5358,5361:5362,5365,5372,5377:5378,5380:5381,5398,5400,5404,5409,5411,5416:5419,5421,5423,5425,5434:5437,5443 -FLAGS.SILENT (gnus-expire)
> 16:35:34 91 UID STORE 4357,4808,5358,5418:5419,5425,5436:5437 -FLAGS.SILENT (\Answered)
> 16:35:34 92 UID STORE 4781,4808,4813,5420 -FLAGS.SILENT (\Flagged)

Huh.  It clear all the marks that are below the selected range of
articles.  Where's the code that does this...

Here it is:

	(when (and (gnus-check-backend-function
		    'request-set-mark gnus-newsgroup-name)
		   (not (gnus-article-unpropagatable-p (cdr type))))
	  (let* ((old (cdr (assq (cdr type) (gnus-info-marks info))))
		 (del (gnus-remove-from-range (gnus-copy-sequence old) list))
		 (add (gnus-remove-from-range
		       (gnus-copy-sequence list) old)))
	    (when add
	      (push (list add 'add (list (cdr type))) delta-marks))
	    (when del
	      (push (list del 'del (list (cdr type))) delta-marks))))

      (when delta-marks
	(unless (gnus-check-group gnus-newsgroup-name)
	  (error "Can't open server for %s" gnus-newsgroup-name))
	(gnus-request-set-mark gnus-newsgroup-name delta-marks))

Ok, it finds the old data from the group info (`gnus-info-marks'), and
then...  What's `list', then?  It's the article list of this type of
marks.  It's set upon group entry, and should be the same as the stored
info.  ...

Aha!

`gnus-adjust-marked-articles' looks at the marks lists, and if the
active ranges indicate that some of the marks refer to articles that no
longer exists, then they remove those marks!

So the cause here must be that the active data is wrong.

What does

(gnus-active "nnimap+server:group")

say for that group?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> 16:35:24 84 EXAMINE "INBOX"
>> 16:35:24 85 UID FETCH 5459:* FLAGS
>> 16:35:24 86 SELECT "INBOX"
>> 16:35:24 87 SELECT "INBOX"
>> 16:35:24 88 UID FETCH 5466:5559 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
>> 16:35:34 89 UID STORE 4808,5398 -FLAGS.SILENT (gnus-forward)
>> 16:35:34 90 UID STORE 5355,5358,5361:5362,5365,5372,5377:5378,5380:5381,5398,5400,5404,5409,5411,5416:5419,5421,5423,5425,5434:5437,5443 -FLAGS.SILENT (gnus-expire)
>> 16:35:34 91 UID STORE 4357,4808,5358,5418:5419,5425,5436:5437 -FLAGS.SILENT (\Answered)
>> 16:35:34 92 UID STORE 4781,4808,4813,5420 -FLAGS.SILENT (\Flagged)
>
> Huh.  It clear all the marks that are below the selected range of
> articles.  Where's the code that does this...
...
> So the cause here must be that the active data is wrong.
>
> What does
>
> (gnus-active "nnimap+server:group")
>
> say for that group?

(gnus-active "nnimap+rocky:INBOX")

gives

(5606 . 5645)

(Both with an old copy of Gnus and with a git version from earlier
today.)

It seems like a good safety check would be to never try to adjust flags
on articles outside of the active range.  Then if the active range gets
messed up because of another bug or some corruption, at least the damage
is limited.

Dan

Re: More IMAP testing, please

[Dan Christensen]

Dan Christensen <jdc@uwo.ca> writes:

> (gnus-active "nnimap+rocky:INBOX")
>
> gives
>
> (5606 . 5645)

But that's not going to help you, since I decided to move all of the
articles from that group to itself, to compress the article numbers as a
way of possibly working around this bug.

However, here's a group that has ~6500 articles in it:

(gnus-active "nnimap+rocky:hha")

and yet I get

(36564 . 36664)

a range of just 101 articles.  So there is something fishy going on.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> But that's not going to help you, since I decided to move all of the
> articles from that group to itself, to compress the article numbers as a
> way of possibly working around this bug.
>
> However, here's a group that has ~6500 articles in it:
>
> (gnus-active "nnimap+rocky:hha")
>
> and yet I get
>
> (36564 . 36664)
>
> a range of just 101 articles.  So there is something fishy going on.

Yes.  Very fishy.  And that "101" number points strongly at the partial
updating code.

If you `M-g' the group, I assume that the `gnus-active' turns into more
reasonable numbers?  And then a `g' will turn it back to 101?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> But that's not going to help you, since I decided to move all of the
>> articles from that group to itself, to compress the article numbers as a
>> way of possibly working around this bug.
>>
>> However, here's a group that has ~6500 articles in it:
>>
>> (gnus-active "nnimap+rocky:hha")
>>
>> and yet I get
>>
>> (36564 . 36664)
>>
>> a range of just 101 articles.  So there is something fishy going on.
>
> Yes.  Very fishy.  And that "101" number points strongly at the partial
> updating code.
>
> If you `M-g' the group, I assume that the `gnus-active' turns into more
> reasonable numbers?  And then a `g' will turn it back to 101?

After starting Gnus (git version as of right now):

(36564 . 36664)

After `M-g' on the hha group:

(29919 . 36664)

After `g' in the *Group* buffer:

(29919 . 36664)

So it stays with the extended range.

By the way, it feels to me like there are two mark-related bugs that
I've reported.  The first one is the one where all marks were lost on
articles with numbers more than 100 less than the highest, except Seen
marks.  This happened in two groups (INBOX and hha) at the same time,
and didn't involve me doing anything except entering and exiting the
groups (possibly with prefix args).  I don't know how to reproduce this.

The second one is that whenever I moved an article from a group to
itself, all marks (including Seen marks) were lost.  It's possible
that this only affected articles with numbers more than 100 less than
the highest, but I thought it affected all articles.  This was
reproducible with the version from a day or two ago, but now it appears
to be fixed.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> By the way, it feels to me like there are two mark-related bugs that
> I've reported.  The first one is the one where all marks were lost on
> articles with numbers more than 100 less than the highest, except Seen
> marks.  This happened in two groups (INBOX and hha) at the same time,
> and didn't involve me doing anything except entering and exiting the
> groups (possibly with prefix args).  I don't know how to reproduce this.

This has probably been fixed.  Or "fixed".  It's an ugly hack to not
prune any marks according to the active range on group entry.

Why the active range is wrong is something I'm still poking around to
try to find out...

> The second one is that whenever I moved an article from a group to
> itself, all marks (including Seen marks) were lost.  It's possible
> that this only affected articles with numbers more than 100 less than
> the highest, but I thought it affected all articles.  This was
> reproducible with the version from a day or two ago, but now it appears
> to be fixed.

Yes, Andrew Cohen found a fix for that problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> It seems like a good safety check would be to never try to adjust flags
> on articles outside of the active range.  Then if the active range gets
> messed up because of another bug or some corruption, at least the damage
> is limited.

Yes.  I've now added a quick hack to avoid pruning marks on group entry.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> It seems like a good safety check would be to never try to adjust flags
>> on articles outside of the active range.  Then if the active range gets
>> messed up because of another bug or some corruption, at least the damage
>> is limited.
>
> Yes.  I've now added a quick hack to avoid pruning marks on group entry.

I looked at what you did (the "(setq min 1) thing) and while it might
fix this particular bug, I was actually recommending a more defensive
change:  whenever Gnus is about to sent commands to the imap server,
check whether all article numbers that will be affected are within the
active range.  If not, something has surely gone wrong, and at the very
least those articles should not be touched.

This is defensive in the sense that no matter what bug caused the
problem, at least the damage would be limited.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I looked at what you did (the "(setq min 1) thing) and while it might
> fix this particular bug, I was actually recommending a more defensive
> change:  whenever Gnus is about to sent commands to the imap server,
> check whether all article numbers that will be affected are within the
> active range.  If not, something has surely gone wrong, and at the very
> least those articles should not be touched.

Excellent idea.  I've now done this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[James Cloos]

[Replying to a random art on this thread]

My X server crashed today (the laptop, not the X process) and that
forced me to start up gnus again.

So I installed the current master to try it out.  That was commit
c833671c9d32c73eb7ff7d40e2f99bd9a570bc4e.

It was unable to login to my imap at all; The server buffer said
denied.

The relevant config from gnus-secondary-select-methods is:

 (nnimap "L"
	 (nnimap-address "localhost")
	 (nnimap-need-unselect-to-notice-new-mail nil)
	 (gnus-check-new-newsgroups t)
	 (nnimap-stream shell)
	 (imap-shell-program "/usr/sbin/myimapd -n")
	 (nnimap-authinfo-file "~/.iai.gpg"))

(with some of the names changed to protect both the innocent /and/ the
guilty. ☺)

I had to back down to commit 6e9a3072f961d1003cf14b37fdd4c922cc4e3db1
in order to interact with mail at all.

(Even though imapd is run via imap-shell-program, it is not pre-
authenticated.  I only use the shell because running imap over tcp —
even over loopback — has about twice the latency.  Using the pty
cuts the startup time in half.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> The relevant config from gnus-secondary-select-methods is:
>
>  (nnimap "L"
> 	 (nnimap-address "localhost")
> 	 (nnimap-need-unselect-to-notice-new-mail nil)
> 	 (gnus-check-new-newsgroups t)
> 	 (nnimap-stream shell)
> 	 (imap-shell-program "/usr/sbin/myimapd -n")
> 	 (nnimap-authinfo-file "~/.iai.gpg"))

nnimap uses auth-sources now.  Perhaps the `auth-sources' variable
should be overridden by `nnimap-authinfo-file' if it exists?

The rest looks OK, and you should have been prompted for user
name/password.  That didn't happen?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The rest looks OK, and you should have been prompted for user
> name/password.  That didn't happen?

If you still have access to the git version, could you look into the
"*nnimap ... *" buffer when this happens and post the contents?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> If you still have access to the git version, could you look into the
LMI> "*nnimap ... *" buffer when this happens and post the contents?

I will try it out again later, and do that if it still fails.

(The nice thing about git is that it is *easy* to change versions; only
the cost of gnus startup takes any time at all.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More IMAP testing, please

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> nnimap uses auth-sources now.  Perhaps the `auth-sources' variable
LMI> should be overridden by `nnimap-authinfo-file' if it exists?

OK.  When I tried having an ~/.authinfo.gpg file, I got a prompt from
gpg-agent every time I sent a mail, even though I have gnus post mail
to localhost:smtp, which does not require any auth.  So I went back
to just an .authinfo file.  (Nothing in it matters much, since it is
all localhost.)

LMI> The rest looks OK, and you should have been prompted for user
LMI> name/password.  That didn't happen?

No, it did not.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More IMAP testing, please

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

James Cloos <cloos@jhcloos.com> writes:

> OK.  When I tried having an ~/.authinfo.gpg file, I got a prompt from
> gpg-agent every time I sent a mail, even though I have gnus post mail
> to localhost:smtp, which does not require any auth.  So I went back to
> just an .authinfo file.  (Nothing in it matters much, since it is all
> localhost.)

This doesn't happen to me. However, my "send-mail-function" is set to
using "sendmail-send-it".

Charles

-- 
"...Unix, MS-DOS, and Windows NT (also known as the Good, the Bad, and
the Ugly)."
(By Matt Welsh)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Dave Goldberg]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
LMI> nnimap uses auth-sources now.  Perhaps the `auth-sources' variable
LMI> should be overridden by `nnimap-authinfo-file' if it exists?

> OK.  When I tried having an ~/.authinfo.gpg file, I got a prompt from
> gpg-agent every time I sent a mail, even though I have gnus post mail
> to localhost:smtp, which does not require any auth.  So I went back
> to just an .authinfo file.  (Nothing in it matters much, since it is
> all localhost.)

LMI> The rest looks OK, and you should have been prompted for user
LMI> name/password.  That didn't happen?

> No, it did not.

I am now having the same problem (no prompt for user/password nor even
to decrypt my .authinfo.gpg file) after doing a git pull about ten
minutes ago.  Unfortunately I'm at work so can't spend too much time
debugging and have switched back to a previously working snapshot for
the time being.

My select method setup (both my cyrus and exchange IMAP servers are
secondary) look similar to James' though I removed the reference to
nnimap-authinfo-file some time ago in favor of auth-sources.

One thing that looks odd is I've got several buffer names like this:

   "*nnimap mail.mywork.org nil  *nntpd**<2>" 0 Fundamental

Somehow having *nntpd* in what looks like a buffer intended for IMAP
seems wrong.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: More IMAP testing, please

[Dave Goldberg]

> My select method setup (both my cyrus and exchange IMAP servers are
> secondary) look similar to James' though I removed the reference to
> nnimap-authinfo-file some time ago in favor of auth-sources.

I take it back, my select methods are not really like James' at all in
retrospect.  In case it helps, here they are:

	    (nnimap "cyrus"
		    (nnimap-address
		     "mail.mywork.org")
		    (nnimap-stream starttls)
		    (nnimap-need-unselect-to-notice-new-mail t)
		    (nnimap-nov-is-evil t))
	    (nnimap "exchange"
		    (nnimap-address
		     "exchange.mywork.org")
		    (nnimap-stream starttls)
		    (nnimap-need-unselect-to-notice-new-mail t)
		    (nnimap-nov-is-evil t)))


-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

>> My select method setup (both my cyrus and exchange IMAP servers are
>> secondary) look similar to James' though I removed the reference to
>> nnimap-authinfo-file some time ago in favor of auth-sources.
>
> I take it back, my select methods are not really like James' at all in
> retrospect.  In case it helps, here they are:
>
> 	    (nnimap "cyrus"
> 		    (nnimap-address
> 		     "mail.mywork.org")
> 		    (nnimap-stream starttls)
> 		    (nnimap-need-unselect-to-notice-new-mail t)
> 		    (nnimap-nov-is-evil t))

No prompting for user name/password/anything might mean that it doesn't
get that far.  What's in the *nnimap ...* buffers?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>>> My select method setup (both my cyrus and exchange IMAP servers are
>>> secondary) look similar to James' though I removed the reference to
>>> nnimap-authinfo-file some time ago in favor of auth-sources.
>> 
>> I take it back, my select methods are not really like James' at all in
>> retrospect.  In case it helps, here they are:
>> 
>> (nnimap "cyrus"
>> (nnimap-address
>> "mail.mywork.org")
>> (nnimap-stream starttls)
>> (nnimap-need-unselect-to-notice-new-mail t)
>> (nnimap-nov-is-evil t))

> No prompting for user name/password/anything might mean that it doesn't
> get that far.  What's in the *nnimap ...* buffers?

They were empty, but I think I understand it.  In Message-id:
<87ocbo4e4c.fsf@gmail.com> Eythan Weg noted that starttls support was
missing, which was almost certainly my problem.  I'll be testing the
new starttls support when I get to work tomorrow.

Thanks,

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> One thing that looks odd is I've got several buffer names like this:
>
>    "*nnimap mail.mywork.org nil  *nntpd**<2>" 0 Fundamental
>
> Somehow having *nntpd* in what looks like a buffer intended for IMAP
> seems wrong.

The nntpd buffer is the buffer used for communication between the
backend and Gnus, so it's correct, even though it's confusing.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[James Cloos]

[sending this vai gmane from another box...]

It turned out that the bug is that imap-shell-program override in
gnus-secondary-select-methods is ignoreed, defaulting to the version
specified in imap.el.

Startup might have improved.  the imapd side of things ony needed about
half as much time, since it did an lsub and examines, rahter than status,
examine tuples, followed by an lsub, followed by another round of status,
exmine tuples.  But now emacs is spinning.  Overall startup might end up
similar or even worse that before?

Fixing the imap-shell-program bug, though, it the biggest issue.

-JimC

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

On Thu, Sep 23 2010, James Cloos wrote:

> It turned out that the bug is that imap-shell-program override in
> gnus-secondary-select-methods is ignoreed, defaulting to the version
> specified in imap.el.

It's nnimap-shell-program (now).

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Dan Christensen]

Julien Danjou <julien@danjou.info> writes:

> On Thu, Sep 23 2010, James Cloos wrote:
>
>> It turned out that the bug is that imap-shell-program override in
>> gnus-secondary-select-methods is ignoreed, defaulting to the version
>> specified in imap.el.
>
> It's nnimap-shell-program (now).

It looks to me like there is a server variable nnimap-shell-program
and a global variable imap-shell-program which is used as the default
for the server variable.  And it certainly worked for me to set
imap-shell-program.

Yesterday, I switched to using server variables since I wanted a
different program for each server, and at one point Gnus started
trying to use some default ssl programs.  Restarting Gnus made it
use my value.  Not sure what caused this.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> It looks to me like there is a server variable nnimap-shell-program
> and a global variable imap-shell-program which is used as the default
> for the server variable.  And it certainly worked for me to set
> imap-shell-program.

It'll work if you set imap-shell-program before loading nnimap.el, but
not the other way around.  That's the problem with these backwards
compatibility workarounds.

It's possible to do it dynamically in `nnimap-open-server', but you end
up now knowing whether to trust the value of nnimap-shell-program or
imap-shell-program, and somebody is going to be unhappy either way.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[James Cloos]

>>>>> "JD" == Julien Danjou <julien@danjou.info> writes:

JD> It's nnimap-shell-program (now).

Ah.  I've made that change and'll see whether it works the next time I
start up.

Thanks.

-Jimc
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More IMAP testing, please

[James Cloos]

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:

JC> Overall startup might end up similar or even worse that before?

In fact it did end up the same.  The only difference is that emacs' now
takes a quarter of the time itself, instead of nearly none.

So it is 2:1:1 imapd:emacs:imapd instead of 0:4 emacs:imapd.  (Rounded,
of course.)

I doubt there can be much improvement, unless the (new) middle step,
between the lsub+examine and the emamine+uidfetch, can be improved.

An ltrace on emacs during that point showed lots of mmap/memcpy/free
activity.  But no (or *very* few) syscalls.  Notably I never saw calls
to brk(2), so gc was probably the biggest issue.

I'm happy.  Other aspects seem faster.  And the improvement over tcp
will be significant, from the pipelining.  But startup time over a pty
is a wash.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More IMAP testing, please

[Ted Zlatanov]

On Thu, 23 Sep 2010 01:50:39 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> nnimap uses auth-sources now.  Perhaps the `auth-sources' variable
LMI> should be overridden by `nnimap-authinfo-file' if it exists?

No, please.  Let's obsolete all these extra ways to specify an authinfo
file.

On Wed, 22 Sep 2010 19:58:01 -0400 James Cloos <cloos@jhcloos.com> wrote: 

JC> OK.  When I tried having an ~/.authinfo.gpg file, I got a prompt from
JC> gpg-agent every time I sent a mail, even though I have gnus post mail
JC> to localhost:smtp, which does not require any auth.

smtpmail.el needs to be smarter then.  I guess it currently looks for a
password regardless.  That seems to be an issue with
`smtpmail-try-auth-methods' and not with auth-source.el specifically so
I think it should be treated as a bug.

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> No, please.  Let's obsolete all these extra ways to specify an authinfo
> file.

I tend to agree, because it always gets to confusing to everybody to
have all these legacy variables that sometimes kinda work...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> My largest group has about 6500 articles, but after a few minutes of
>> using gnus, gnus would only show me the newest 71 articles (which seem
>> to be the ones whose article numbers are within 100 of the latest).
>> `M-g' on the group fixed this, but then I found I had lost *all*
>> marks on the older articles (tick, expired, answered definitely
>> lost, read not lost).
>
> That's not good.

I have a backup of my .newsrc.eld from before the marks were lost.  Is
there an easy way to use the backup to make Gnus change the marks to
what they should be, and then propagate those changes to the IMAP
server?  It's not critical in this case, but it would be nice to have
them back.

Dan

Re: More IMAP testing, please

[Ted Zlatanov]

On Mon, 20 Sep 2010 17:10:11 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> I have a backup of my .newsrc.eld from before the marks were lost.  Is
DC> there an easy way to use the backup to make Gnus change the marks to
DC> what they should be, and then propagate those changes to the IMAP
DC> server?  It's not critical in this case, but it would be nice to have
DC> them back.

Lok at what gnus-sync.el does.

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I have a backup of my .newsrc.eld from before the marks were lost.  Is
> there an easy way to use the backup to make Gnus change the marks to
> what they should be, and then propagate those changes to the IMAP
> server?  It's not critical in this case, but it would be nice to have
> them back.

There's no built-in functionality for that in Gnus directly -- it tries
to only update marks that have changed by comparing what marks existed
when you entered the group, and what marks exist when you exit the
group.  Hm...  perhaps there is a way.  If you enter the group, then
`G E' to edit the group info, and then insert the marks/etc from the old
file relevant to that group, and then exit the group?  Then it's
possible that Gnus will push over the marks...  er...  no, it works the
opposite way.

Uhm.

If you set gnus-newsgroup-replied (etc) to the uncompressed lists from
the old .newsrc.eld, then Gnus will push them over.  But that's not, er,
easy to do. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

I have just tried client-side splitting and found one small problem. In 
nnimap-fetch-inbox the case with nnimap-split-download-body-default
should return "[]" rather than "" (the latter results in an imap error
response). 

	    (if nnimap-split-download-body-default
		"[]"
	      "[1]")))

Once I made this change I get mail split, but I haven't yet received
enough mail to see if its working properly.

Re: More IMAP testing, please

[Andrew Cohen]

Now I have received fair number of email messages and can say that
splitting is working fine with the previous fix I sent.

I'm also seeing the previously reported issue with
nnimap-wait-for-response being called with a nil sequence in
nnimap-request-set-mark.

I also saw the problem moving articles (as previously reported) and found
that gnus-find-method-for-group was returning different things for two
groups both on the same imap server. So the nnimap move functions were
not using the internal-move-group code (and I gather that moving between
different servers isn't working yet). I tracked this down to two
different specifications in the .newsrc.eld file. Some groups had

(nnimap "home")

while others had

"nnimap:home"

I changed all instances to the latter (is this the right thing to do?)
and now it works fine.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I'm also seeing the previously reported issue with
> nnimap-wait-for-response being called with a nil sequence in
> nnimap-request-set-mark.

That should be fixed...

> (nnimap "home")
>
> while others had
>
> "nnimap:home"
>
> I changed all instances to the latter (is this the right thing to do?)
> and now it works fine.

I think I've fixed this now.  Could you try the following?

(gnus-server-equal '(nnimap "home") "nnimap:home")

If it says t, then the fixed I pushed out now is correct.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    Lars> Andrew Cohen <cohen@andy.bu.edu> writes:
    >> I'm also seeing the previously reported issue with
    >> nnimap-wait-for-response being called with a nil sequence in
    >> nnimap-request-set-mark.

    Lars> That should be fixed...

Yes it is.

    >> (nnimap "home")
    >> 
    >> while others had
    >> 
    >> "nnimap:home"
    >> 
    >> I changed all instances to the latter (is this the right thing to
    >> do?)  and now it works fine.

    Lars> I think I've fixed this now.  Could you try the following?

    Lars> (gnus-server-equal '(nnimap "home") "nnimap:home")

    Lars> If it says t, then the fixed I pushed out now is correct.

No luck:

(gnus-server-equal '(nnimap "home") "nnimap:home")
nil


I'm trying expiry out (with a nice, safe backup of my mail). 

Seems to be a typo in nnimap.el:

time-minus -> time-subtract

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> (gnus-server-equal '(nnimap "home") "nnimap:home")
> nil

What does

(gnus-server-to-method "nnimap:home")

say?

> I'm trying expiry out (with a nice, safe backup of my mail). 
>
> Seems to be a typo in nnimap.el:
>
> time-minus -> time-subtract

Fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

It says:

(gnus-server-to-method "nnimap:home")
(nnimap "home" (nnimap-address "andy.bu.edu") (nnimap-stream ssl) (nnimap-server-port 993) (nnimap-split-predicate "UNDELETED") (nnir-search-engine imap))



(gnus-server-to-method '(nnimap "home"))
(nnimap "home")

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> (gnus-server-to-method "nnimap:home")
> (nnimap "home" (nnimap-address "andy.bu.edu") (nnimap-stream ssl) (nnimap-server-port 993) (nnimap-split-predicate "UNDELETED") (nnir-search-engine imap))
>
> (gnus-server-to-method '(nnimap "home"))
> (nnimap "home")

Right.  The former is the correct method?  Then (nnimap "home") wasn't a
correct server, but Gnus handled that situation badly.  I'm now
investigating why.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    Lars> Andrew Cohen <cohen@andy.bu.edu> writes:
    >> (gnus-server-to-method "nnimap:home") (nnimap "home"
    >> (nnimap-address "andy.bu.edu") (nnimap-stream ssl)
    >> (nnimap-server-port 993) (nnimap-split-predicate "UNDELETED")
    >> (nnir-search-engine imap))
    >> 
    >> (gnus-server-to-method '(nnimap "home")) (nnimap "home")

    Lars> Right.  The former is the correct method?  Then (nnimap
    Lars> "home") wasn't a correct server, but Gnus handled that
    Lars> situation badly.  I'm now investigating why.


Following the latest changes I'm having a problem with
gnus-group-completing-read. Previously if I typed "Tra" at the prompt
this would complete to

"nnimap+home:Trash"

now it completes to

"nnimap+:Trash"


And on another note---having some problems with expiry which I haven't
been able to track down. I have a group "test" with 900 fairly old
messages. I set an expiry-target of "test.old" and turn on
total-expire. The articles older than 7 days are correctly placed in
"test.old", but then things freeze. I waited half an hour and then gave
up. C-g a few times gets out of it, but now the original group "test"
still has most of the articles that got copied to "test.old" (at least
it seems like a small number have been deleted but most are still
around). 

Andy

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> Following the latest changes I'm having a problem with
> gnus-group-completing-read. Previously if I typed "Tra" at the prompt
> this would complete to
>
> "nnimap+home:Trash"
>
> now it completes to
>
> "nnimap+:Trash"

I haven't changed anything in the completion code the last week...  What
happens if you TAB a few more times?

> And on another note---having some problems with expiry which I haven't
> been able to track down. I have a group "test" with 900 fairly old
> messages. I set an expiry-target of "test.old" and turn on
> total-expire. The articles older than 7 days are correctly placed in
> "test.old", but then things freeze. I waited half an hour and then gave
> up. C-g a few times gets out of it, but now the original group "test"
> still has most of the articles that got copied to "test.old" (at least
> it seems like a small number have been deleted but most are still
> around). 

Did it say it was doing anything before you pressed `C-g'?  Also
`(setq debug-on-quit t)' and repeat and post the backtrace.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:


    Lars> I haven't changed anything in the completion code the last
    Lars> week...  What happens if you TAB a few more times?

Oops, my mistake. I have two nnimap servers, "home" and "search", and
tabbing now allows me to choose. (Don't know why it completed directly
to nnimap+home earlier)


    Lars> Did it say it was doing anything before you pressed `C-g'?

No. It had finished "expiring" and I presumed it was
deleting/expunging. 

    Lars> Also `(setq debug-on-quit t)' and repeat and post the
    Lars> backtrace.

I'll give it a try.

andy

Re: More IMAP testing, please

[Andrew Cohen]

    Lars> Did it say it was doing anything before you pressed `C-g'?

    Andrew> No. It had finished "expiring" and I presumed it was
    Andrew> deleting/expunging.

    Lars> Also `(setq debug-on-quit t)' and repeat and post the
    Lars> backtrace.

    Andrew> I'll give it a try.


A bit more info on expiring---the problem happens only when there is a
large number of articles to expire. debug-on-quit suggests its waiting
for an nnimap response:

Debugger entered--Lisp error: (quit)
  accept-process-output(#<process *nnimap*> 0 10)
  nnheader-accept-process-output(#<process *nnimap*>)
  nnimap-wait-for-response(5108)
  nnimap-request-group("uhc.old.old" "home" nil nil)
  gnus-request-group("nnimap+home:uhc.old.old")
  nnmail-expiry-target-group("nnimap+home:uhc.old.old" "uhc.old")
  nnimap-process-expiry-targets((5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ...) "uhc.old" "home")
  nnimap-request-expire-articles((1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) "uhc.old" "home" nil)
  gnus-request-expire-articles((1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 ...) "nnimap+home:uhc.old")
  gnus-summary-expire-articles()
  run-hooks(gnus-summary-prepare-exit-hook)
  apply(run-hooks gnus-summary-prepare-exit-hook)
  gnus-run-hooks(gnus-summary-prepare-exit-hook)
  gnus-summary-exit()
  call-interactively(gnus-summary-exit nil nil)

I don't understand what is going on very well, but at least a couple of
times when it happened the mini-buffer flashed a message about getting
new news. (I have gnus-demon setup to run every few minutes.) I turned
off the demons and tried again and it worked fine. So maybe its possible
that fetch new mail ends up being run in the middle of expiring and
clobbers the nnimap buffer response?

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I don't understand what is going on very well, but at least a couple of
> times when it happened the mini-buffer flashed a message about getting
> new news. (I have gnus-demon setup to run every few minutes.) I turned
> off the demons and tried again and it worked fine. So maybe its possible
> that fetch new mail ends up being run in the middle of expiring and
> clobbers the nnimap buffer response?

Yes, that sounds likely.  I've now made the command inhibit the daemon.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Andrew Cohen <cohen@andy.bu.edu> writes:
>
>> I don't understand what is going on very well, but at least a couple of
>> times when it happened the mini-buffer flashed a message about getting
>> new news. (I have gnus-demon setup to run every few minutes.) I turned
>> off the demons and tried again and it worked fine. So maybe its possible
>> that fetch new mail ends up being run in the middle of expiring and
>> clobbers the nnimap buffer response?
>
> Yes, that sounds likely.  I've now made the command inhibit the daemon.

Could there still be a problem if the daemon starts fetching new mail,
and while this is happening, Gnus starts expiring articles?  Should 
some data structures be protected by locks?

A related question: when (important) new mail arrives, I have a shell
script that includes:

  emacsclient --no-wait --eval '(jdc-gnus-demon-scan-mail)' --alternate-editor=/dev/null

where jdc-gnus-demon-scan-mail is a function like gnus-demon-scan-mail,
but which scans mail servers only.  (The default function doesn't seem
to restrict to mail servers, despite its name.)

Every now and then I've noticed that Gnus locks up if I'm doing
something exactly when my shell script runs.  This didn't happen with
the old nnimap, but has happened a few times with the new one.

Is there a way to tell emacs to only let an emacsclient instance run
something when the server isn't already running some code?  I had
assumed that this was how things worked by default, since emacs doesn't
have multi-threading.  Ah, the timer node in the elisp manual says that
timers can run when functions like sit-for and read-event are running,
so the same is probably true for emacsclient connections.

Maybe my code should check current-idle-time; or maybe it should just
schedule something to run using run-with-idle-timer...

And I wonder if the gnus-demon-* functions that run should use
with-local-quit; at least, the ones that run external programs
or use network access should do this, I think.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Could there still be a problem if the daemon starts fetching new mail,
> and while this is happening, Gnus starts expiring articles?  Should 
> some data structures be protected by locks?

They should, but there's really no overview of what needs to be
protected, so it's rather chancy.

All the backend functions should set a semaphore to say that they're in
use, though.

> Is there a way to tell emacs to only let an emacsclient instance run
> something when the server isn't already running some code?  I had
> assumed that this was how things worked by default, since emacs doesn't
> have multi-threading.  Ah, the timer node in the elisp manual says that
> timers can run when functions like sit-for and read-event are running,
> so the same is probably true for emacsclient connections.

Yup.

> Maybe my code should check current-idle-time; or maybe it should just
> schedule something to run using run-with-idle-timer...

If gnus-command-method is non-nil, then stuff is being run.  But is it
set globally, or just bound in that thread?  I'm not sure how the Emacs
threading model works...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I have just tried client-side splitting and found one small problem. In 
> nnimap-fetch-inbox the case with nnimap-split-download-body-default
> should return "[]" rather than "" (the latter results in an imap error
> response). 
>
> 	    (if nnimap-split-download-body-default
> 		"[]"
> 	      "[1]")))

Thanks; applied and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

I just tried using `B m' to move an article from an nnimap group
to an nnfolder group, and the result was that the following was
appended to the nnfolder file (without the leading whitespace):

  From nobody Sun Sep 19 20:34:21 2010
  Message-ID: <87sk155k36.fsf@totally-fudged-out-message-id>
  Lines: 0
  Xref: rocky cherrington-dejong:1594
  X-Gnus-Article-Number: 1594   Sun, 19 Sep 2010 20:34:21 -0400
  

That's it, no other headers, and just a single blank line after
those headers.

Luckily, the message was also stored on a remote imap server.
Reading it there, I tried `B c' to the same nnfolder group, 
and that time it worked.

Further tests show that this is repeatable:  `B m' fails and
`B c' works.  This has the potential to cause loss of mail.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I just tried using `B m' to move an article from an nnimap group
> to an nnfolder group, and the result was that the following was
> appended to the nnfolder file (without the leading whitespace):
>
>   From nobody Sun Sep 19 20:34:21 2010
>   Message-ID: <87sk155k36.fsf@totally-fudged-out-message-id>
>   Lines: 0
>   Xref: rocky cherrington-dejong:1594
>   X-Gnus-Article-Number: 1594   Sun, 19 Sep 2010 20:34:21 -0400

Oops.  Fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I just tried using `B m' to move an article from an nnimap group
>> to an nnfolder group, and the result was that the following was
>> appended to the nnfolder file (without the leading whitespace):
>>
>>   From nobody Sun Sep 19 20:34:21 2010
>>   Message-ID: <87sk155k36.fsf@totally-fudged-out-message-id>
>>   Lines: 0
>>   Xref: rocky cherrington-dejong:1594
>>   X-Gnus-Article-Number: 1594   Sun, 19 Sep 2010 20:34:21 -0400
>
> Oops.  Fixed and pushed.

Confirmed working now.  Thanks!

Dan

Re: More IMAP testing, please

[Dan Christensen]

There are still some bugs with `B m'.  The most serious one is that if 
I try to use `B m' to move a message from an IMAP group to that same 
IMAP group, then *all* articles in the IMAP group lose their Seen flags,
gnus-expire flags, and maybe other flags too.  I think this sequence 
of commands are the relevant ones, and commands 330 and 331 look
particularly suspicious to me.

20:32:57 322 EXAMINE "INBOX"
20:32:57 323 UID FETCH 1:* FLAGS
20:32:57 324 SELECT "INBOX"
20:32:57 325 SELECT "INBOX"
20:32:57 326 UID FETCH 5573:5581 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
20:33:02 327 UID COPY 5581 "INBOX"
20:33:02 328 UID STORE 5581 +FLAGS.SILENT (\Deleted)
20:33:02 329 UID EXPUNGE 5581
20:33:02 330 UID SEARCH HEADER Message-Id nil
20:33:02 331 UID STORE 0 +FLAGS.SILENT (\Seen)
20:33:02 332 SELECT "INBOX"
20:33:02 333 UID FETCH 1:* FLAGS
20:33:04 334 UID STORE 1:5586 -FLAGS.SILENT (\Seen)
20:33:04 335 UID STORE 5586 -FLAGS.SILENT (\Flagged)
20:33:04 336 SELECT "INBOX"
20:33:04 337 UID FETCH 1:* FLAGS
20:33:04 338 SELECT "INBOX"
20:33:04 339 UID FETCH 5578:5586 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
20:33:12 340 UID STORE 5586 +FLAGS.SILENT (\Seen)
20:33:12 341 UID STORE 5586 +FLAGS.SILENT (\Flagged)

There's also another bug, which also existed with the previous nnimap.el
implementation, so I'm used to it, but since you have this fresh in your
mind, I'll bet it's easy to fix:  If I change the marks on an article in
an IMAP group and then do an internal move to another group on the same
server, the new article keeps its old marks.

Bug number three with `B m':  if I process mark a bunch of articles
and try to use `B m' on them, every second one gets moved, and the
rest are untouched.

All three of these bugs are reproducible.

Dan

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Dan Christensen <jdc@uwo.ca>:

> There are still some bugs with `B m'.

Hm... that one I use (or "used to use") a lot.

Perhaps I should hold off using `B m' until theres' an all clear on this
group...? 

Re: More IMAP testing, please

[Andrew Cohen]

>>>>> "Dan" == Dan Christensen <jdc@uwo.ca> writes:

    Dan> There are still some bugs with `B m'.  The most serious one is
    Dan> that if I try to use `B m' to move a message from an IMAP group
    Dan> to that same IMAP group, then *all* articles in the IMAP group
    Dan> lose their Seen flags, gnus-expire flags, and maybe other flags
    Dan> too.  I think this sequence of commands are the relevant ones,
    Dan> and commands 330 and 331 look particularly suspicious to me.

The message-id in nnimap-request-move-article is not getting set
properly. There are two issues: 

1. The article headers need to be fetched first, since we may not be
   looking at the article.

2. message-field-value doesn't strip the carriage return at the end.


My elisp is terrible but this seems to work for me. I've changed
message-field-value and I don't know if this has negative consequences
somewhere else.

diff --git a/lisp/message.el b/lisp/message.el
index f3697eb..0e6d6d5 100644
--- a/lisp/message.el
+++ b/lisp/message.el
@@ -1959,6 +1959,8 @@ see `message-narrow-to-headers-or-head'."
     (when value
       (while (string-match "\n[\t ]+" value)
        (setq value (replace-match " " t t value)))
+      (while (string-match "\r$" value)
+       (setq value (replace-match "" t t value)))
       value)))
 
 (defun message-field-value (header &optional not-all)


diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index 434ad01..3196c0f 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -459,7 +459,25 @@ textual parts.")
   (when (nnimap-possibly-change-group group server)
     ;; If the move is internal (on the same server), just do it the easy
     ;; way.
-    (let ((message-id (message-field-value "message-id")))
+;    (let ((message-id (message-field-value "message-id")))
+    (let ((message-id
+          (with-current-buffer (nnimap-buffer)
+            (nnimap-send-command "SELECT %S" (utf7-encode group t))
+            (erase-buffer)
+            (nnimap-wait-for-response
+             (nnimap-send-command
+              "UID FETCH %s %s"
+              article
+              (format "(UID RFC822.SIZE BODYSTRUCTURE %s)"
+                      (format
+                       (if (member "IMAP4REV1"
+                                   (nnimap-capabilities nnimap-object))
+                           "BODY.PEEK[HEADER.FIELDS %s]"
+                         "RFC822.HEADER.LINES %s")
+                       '(Message-Id)
+                       )))
+             t)
+            (message-field-value "message-id"))))
       (if internal-move-group
          (let ((result
                 (with-current-buffer (nnimap-buffer)

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> The message-id in nnimap-request-move-article is not getting set
> properly. There are two issues: 
>
> 1. The article headers need to be fetched first, since we may not be
>    looking at the article.
>
> 2. message-field-value doesn't strip the carriage return at the end.

Thanks for tracking down the bug and suggesting a fix.  I've now fixed
this in a slightly different way and pushed it out.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

All three of the bugs mentioned below appear to be fixed in the latest
git version.

Dan

Dan Christensen <jdc@uwo.ca> writes:

> There are still some bugs with `B m'.  The most serious one is that if 
> I try to use `B m' to move a message from an IMAP group to that same 
> IMAP group, then *all* articles in the IMAP group lose their Seen flags,
> gnus-expire flags, and maybe other flags too.  I think this sequence 
> of commands are the relevant ones, and commands 330 and 331 look
> particularly suspicious to me.
>
> 20:32:57 322 EXAMINE "INBOX"
> 20:32:57 323 UID FETCH 1:* FLAGS
> 20:32:57 324 SELECT "INBOX"
> 20:32:57 325 SELECT "INBOX"
> 20:32:57 326 UID FETCH 5573:5581 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
> 20:33:02 327 UID COPY 5581 "INBOX"
> 20:33:02 328 UID STORE 5581 +FLAGS.SILENT (\Deleted)
> 20:33:02 329 UID EXPUNGE 5581
> 20:33:02 330 UID SEARCH HEADER Message-Id nil
> 20:33:02 331 UID STORE 0 +FLAGS.SILENT (\Seen)
> 20:33:02 332 SELECT "INBOX"
> 20:33:02 333 UID FETCH 1:* FLAGS
> 20:33:04 334 UID STORE 1:5586 -FLAGS.SILENT (\Seen)
> 20:33:04 335 UID STORE 5586 -FLAGS.SILENT (\Flagged)
> 20:33:04 336 SELECT "INBOX"
> 20:33:04 337 UID FETCH 1:* FLAGS
> 20:33:04 338 SELECT "INBOX"
> 20:33:04 339 UID FETCH 5578:5586 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
> 20:33:12 340 UID STORE 5586 +FLAGS.SILENT (\Seen)
> 20:33:12 341 UID STORE 5586 +FLAGS.SILENT (\Flagged)
>
> There's also another bug, which also existed with the previous nnimap.el
> implementation, so I'm used to it, but since you have this fresh in your
> mind, I'll bet it's easy to fix:  If I change the marks on an article in
> an IMAP group and then do an internal move to another group on the same
> server, the new article keeps its old marks.
>
> Bug number three with `B m':  if I process mark a bunch of articles
> and try to use `B m' on them, every second one gets moved, and the
> rest are untouched.
>
> All three of these bugs are reproducible.
>
> Dan

Re: More IMAP testing, please

[Dan Christensen]

I have %k in my gnus-summary-line-format, and in nnimap groups, the
displayed size doesn't seem to include the attachments.  I think the
old behaviour, in which attachments are included in the size, but 
not in the line number count, is better.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I have %k in my gnus-summary-line-format, and in nnimap groups, the
> displayed size doesn't seem to include the attachments.  I think the
> old behaviour, in which attachments are included in the size, but 
> not in the line number count, is better.

Fixed and pushed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I have %k in my gnus-summary-line-format, and in nnimap groups, the
>> displayed size doesn't seem to include the attachments.  I think the
>> old behaviour, in which attachments are included in the size, but 
>> not in the line number count, is better.
>
> Fixed and pushed now.

Confirmed working now.  Thanks!

Dan

Re: More IMAP testing, please

[Gijs Hillenius]

On 19 Sep 2010, Lars Magne Ingebrigtsen wrote:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

I switched last night/this morning. Some remarks and an error:


I have in .gnus

;; Email (Imap)
(setq gnus-secondary-select-methods
      '((nnimap "thuis"
	       (nnimap-address "mail.server.address")
	       (nnimap-stream ssl) ; was tls
	       (nnimap-authinfo-file
		"/home/gijs/.imap-authinfo"))))


and in the Server Buffer, the same list for the server except that
nnimap-stream ssl here mentioned tls, not ssl. (I had changed that
yesterday evening in .gnus) 

This makes gnus do openssl / gnutls for each of my kazillion
mailboxes. That slows the start-up of gnus down significantly....  I
manually killed the openssl stream a few times to get gnus to start so I
could actually edit the server buffer to make these two configurations
the same (replace tls by ssl).

At that point, I noticed the mail.server was in the server list
twice. But that disappeared after the edit, quit and restart.

Now gnus starts *really* fast, and I can get to my mail. 

I enter a mailbox (called INBOX.stilte) , look at some of the mails and
enter q to do something else. I get a "Format specifier doesn't match
argument type. Here is what the debugger says:



Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
  format("^%d .*\n" "nil")
  nnimap-wait-for-response(nil)
  nnimap-request-set-mark("INBOX.stilte" (((...) del (recent))) "thuis")
  gnus-request-set-mark("nnimap+thuis:INBOX.stilte" (((...) del (recent))))
  gnus-update-marks()
  gnus-summary-update-info()
  gnus-summary-exit()
  call-interactively(gnus-summary-exit nil nil)
  recursive-edit()
  byte-code("\306\x10	@\307=\203!

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> I switched last night/this morning. Some remarks and an error:
>
> I have in .gnus
>
> ;; Email (Imap)
> (setq gnus-secondary-select-methods
>       '((nnimap "thuis"
> 	       (nnimap-address "mail.server.address")
> 	       (nnimap-stream ssl) ; was tls
> 	       (nnimap-authinfo-file
> 		"/home/gijs/.imap-authinfo"))))
>
> and in the Server Buffer, the same list for the server except that
> nnimap-stream ssl here mentioned tls, not ssl. (I had changed that
> yesterday evening in .gnus) 
>
> This makes gnus do openssl / gnutls for each of my kazillion
> mailboxes. That slows the start-up of gnus down significantly....  I
> manually killed the openssl stream a few times to get gnus to start so I
> could actually edit the server buffer to make these two configurations
> the same (replace tls by ssl).

I think the thing that tries to make servers unique fails in the
situation where there are two same-named servers with different
parameters.  In that case it should really just open two servers, but
instead it opens one server per group, which is obviously not very
pleasant.  I'll try to reproduce and fix.

> I enter a mailbox (called INBOX.stilte) , look at some of the mails and
> enter q to do something else. I get a "Format specifier doesn't match
> argument type. Here is what the debugger says:
>
> Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
>   format("^%d .*\n" "nil")
>   nnimap-wait-for-response(nil)

This has been fixed, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Gijs Hillenius]

Two more things:

0) gnus at start-up asks from my login name and password. I thought
"nnimap-authinfo-file" would take care of this? So something in my
set-up must be kaputt.

From my .gnus

,----
| ;; Email (Imap)
| (setq gnus-secondary-select-methods
|       '((nnimap "thuis"
| 	       (nnimap-address "mail.server.address")
| 	       (nnimap-stream ssl) ; was tls
| 	       (nnimap-authinfo-file
| 		"/home/gijs/.imap-authinfo"))))
`----


1) Sent mail seems not to be stored on the imap server. I have to spell the
documentation on this yet..

Again from my .gnus

,----
| ;; Sent Mail On Server
| (setq gnus-message-archive-method
|       '(nnimap "thuis"
| 	       (nnimap-address "mail.server.address")))
| 
| ;; E-mail archive, per month & per Gcc-split (see Posting Styles)
| ;;(setq gnus-message-archive-group "INBOX.Sent")  ;; original folder
| (setq gnus-message-archive-group
|       '((if (message-news-p)
| 	    "misc-news"
| 	  (concat "INBOX.sent." (format-time-string "%Y-%m")))))
`----


which in the previous imap mean I would find mail sent today in the
folder 

INBOX.sent.2010.09

but that is no longer the case. 


-- 
Drop in any mailbox.

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> 0) gnus at start-up asks from my login name and password. I thought
> "nnimap-authinfo-file" would take care of this? So something in my
> set-up must be kaputt.
>
> From my .gnus
>
> ,----
> | ;; Email (Imap)
> | (setq gnus-secondary-select-methods
> |       '((nnimap "thuis"
> | 	       (nnimap-address "mail.server.address")
> | 	       (nnimap-stream ssl) ; was tls
> | 	       (nnimap-authinfo-file
> | 		"/home/gijs/.imap-authinfo"))))
> `----

nnimap-authinfo-file isn't used.  Instead auth-sources is.

> 1) Sent mail seems not to be stored on the imap server. I have to spell the
> documentation on this yet..
>
> Again from my .gnus
>
> ,----
> | ;; Sent Mail On Server
> | (setq gnus-message-archive-method
> |       '(nnimap "thuis"
> | 	       (nnimap-address "mail.server.address")))
> | 
> | ;; E-mail archive, per month & per Gcc-split (see Posting Styles)
> | ;;(setq gnus-message-archive-group "INBOX.Sent")  ;; original folder
> | (setq gnus-message-archive-group
> |       '((if (message-news-p)
> | 	    "misc-news"
> | 	  (concat "INBOX.sent." (format-time-string "%Y-%m")))))
> `----

Hm...  Is this still not working?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

GCC not working (Was Re: More IMAP testing, please)

[Gijs Hillenius]

On 21 Sep 2010, Lars Magne Ingebrigtsen wrote:


[...]

>> Again from my .gnus
>>
>> ,----
>> | ;; Sent Mail On Server
>> | (setq gnus-message-archive-method
>> |       '(nnimap "thuis"
>> | 	       (nnimap-address "mail.server.address")))
>> | 
>> | ;; E-mail archive, per month & per Gcc-split (see Posting Styles)
>> | ;;(setq gnus-message-archive-group "INBOX.Sent") ;; original folder
>> | (setq gnus-message-archive-group
>> |       '((if (message-news-p)
>> | 	    "misc-news"
>> | 	  (concat "INBOX.sent." (format-time-string "%Y-%m")))))
>> `----
>
> Hm...  Is this still not working?

Ooo I was all excited, raced home to update Gnus and find out. But,
nope. At least, not here it aint working yet.

Re: GCC not working

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

>>> | (setq gnus-message-archive-method
>>> |       '(nnimap "thuis"
>>> | 	       (nnimap-address "mail.server.address")))

[...]

>> Hm...  Is this still not working?
>
> Ooo I was all excited, raced home to update Gnus and find out. But,
> nope. At least, not here it aint working yet.

I just tried it, and it worked for me:

21:31:40 229 SELECT "sent.2010w38"
21:31:40 230 CREATE "sent.2010w38"
21:31:40 231 APPEND "sent.2010w38" {1490}
21:31:40 232 SELECT "sent.2010w38"
21:31:40 233 UID SEARCH HEADER Message-Id "<m339t1d17n.fsf@quimbies.gnus.org>"

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: GCC not working

[Gijs Hillenius]

On 22 Sep 2010, Lars Magne Ingebrigtsen wrote:

> Gijs Hillenius <gijs@hillenius.net> writes:
>
>>>> | (setq gnus-message-archive-method
>>>> |       '(nnimap "thuis"
>>>> | 	       (nnimap-address "mail.server.address")))
>
> [...]
>
>>> Hm...  Is this still not working?
>>
>> Ooo I was all excited, raced home to update Gnus and find out. But,
>> nope. At least, not here it aint working yet.
>
> I just tried it, and it worked for me:
>
> 21:31:40 229 SELECT "sent.2010w38"
> 21:31:40 230 CREATE "sent.2010w38"
> 21:31:40 231 APPEND "sent.2010w38" {1490}
> 21:31:40 232 SELECT "sent.2010w38"
> 21:31:40 233 UID SEARCH HEADER Message-Id "<m339t1d17n.fsf@quimbies.gnus.org>"

Ah! 

I looked in my imap-log and I get a similar APPEND. However the message
does not appear in the INBOX.sent.2010-09 folder. 

from the imap-log"

06:56:53 1181 APPEND "INBOX.sent.2010-09" {791}
06:56:53 1182 UID SEARCH HEADER Message-Id "<87aan9t5uz.fsf@hillenius.net>"

Re: GCC not working

[Gijs Hillenius]

On 22 Sep 2010, Lars Magne Ingebrigtsen wrote:

> Gijs Hillenius <gijs@hillenius.net> writes:
>
>>>> | (setq gnus-message-archive-method
>>>> |       '(nnimap "thuis"
>>>> | 	       (nnimap-address "mail.server.address")))
>
> [...]
>
>>> Hm...  Is this still not working?
>>
>> Ooo I was all excited, raced home to update Gnus and find out. But,
>> nope. At least, not here it aint working yet.
>
> I just tried it, and it worked for me:
>
> 21:31:40 229 SELECT "sent.2010w38"
> 21:31:40 230 CREATE "sent.2010w38"
> 21:31:40 231 APPEND "sent.2010w38" {1490}
> 21:31:40 232 SELECT "sent.2010w38"
> 21:31:40 233 UID SEARCH HEADER Message-Id "<m339t1d17n.fsf@quimbies.gnus.org>"

Ditto for the Usenet message I just sent: this is from imap-log:

07:02:12 1183 SELECT "misc-news"
07:02:12 1184 CREATE "misc-news"
07:02:12 1185 APPEND "misc-news" {1774}
07:02:12 1186 SELECT "misc-news"

But the message itself is not stored in this folder (which already
exists, so I am not sure why the CREATE is needed).

regarding previous reply: I just searched for 87aan9t5uz.fsf on the
Cyrus server, no results. So either it gets appended to a folder not on
the IMAP server. Or it says it is appended, but it actually is not. 

Re: GCC not working

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> Ditto for the Usenet message I just sent: this is from imap-log:
>
> 07:02:12 1183 SELECT "misc-news"
> 07:02:12 1184 CREATE "misc-news"
> 07:02:12 1185 APPEND "misc-news" {1774}
> 07:02:12 1186 SELECT "misc-news"
>
> But the message itself is not stored in this folder (which already
> exists, so I am not sure why the CREATE is needed).

Could you edebug through `nnimap-request-accept-article' and see what
happens? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Charles Philip Chan]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Gijs Hillenius <gijs@hillenius.net> writes:

Strange, gcc works for me.

> ,----
> | ;; Sent Mail On Server
> | (setq gnus-message-archive-method
> |       '(nnimap "thuis"
> | 	       (nnimap-address "mail.server.address")))
> | 
> | ;; E-mail archive, per month & per Gcc-split (see Posting Styles)
> | ;;(setq gnus-message-archive-group "INBOX.Sent")  ;; original folder
> | (setq gnus-message-archive-group
> |       '((if (message-news-p)
> | 	    "misc-news"
> | 	  (concat "INBOX.sent." (format-time-string "%Y-%m")))))
> `----

My gnus-message-archive-method is set to "archive" (the default). I also
have this set:

,----
| (setq gnus-message-archive-group
|       (concat "nnimap+Default:Sent." (format-time-string "%Y-%b")))
`----

Charles

-- 
A Linux machine! because a 486 is a terrible thing to waste!
(By jjs@wintermute.ucr.edu, Joe Sloan)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Gijs Hillenius]

On 23 Sep 2010, Charles Philip Chan wrote:

> Gijs Hillenius <gijs@hillenius.net> writes:
>
> Strange, gcc works for me.
>
>> ,----
>> | ;; Sent Mail On Server
>> | (setq gnus-message-archive-method
>> |       '(nnimap "thuis"
>> | 	       (nnimap-address "mail.server.address")))
>> | 
>> | ;; E-mail archive, per month & per Gcc-split (see Posting Styles)
>> | ;;(setq gnus-message-archive-group "INBOX.Sent") ;; original folder
>> | (setq gnus-message-archive-group
>> |       '((if (message-news-p)
>> | 	    "misc-news"
>> | 	  (concat "INBOX.sent." (format-time-string "%Y-%m")))))
>> `----
>
> My gnus-message-archive-method is set to "archive" (the default). I also
> have this set:
>
> ,----
> | (setq gnus-message-archive-group
> |       (concat "nnimap+Default:Sent." (format-time-string "%Y-%b")))
> `----

Ah! So the culprit is likely in my .gnus

Earlier this afternoon, I tried fiddling with it, but so far without the
desired result. I have a feeling I will discover something ugly in my
configuration.

If/when I find out, I'll post it here.

Gcc not yet working (Was: Re: More IMAP testing, please)

[Gijs Hillenius]

On 23 Sep 2010, Gijs Hillenius wrote:


[...]

>
> Ah! So the culprit is likely in my .gnus

[...]

> If/when I find out, I'll post it here.

E-debuging this, and thanks to Lars looking at the result and
understanding it, it looks like it is that my stmp-setup makes gcc uses
TLS.

Re: Gcc not yet working

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> E-debuging this, and thanks to Lars looking at the result and
> understanding it, it looks like it is that my stmp-setup makes gcc uses
> TLS.

What's stmp?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gcc not yet working

[Gijs Hillenius]

On 25 Sep 2010, Lars Magne Ingebrigtsen wrote:

> Gijs Hillenius <gijs@hillenius.net> writes:
>
>> E-debuging this, and thanks to Lars looking at the result and
>> understanding it, it looks like it is that my stmp-setup makes gcc uses
>> TLS.
>
> What's stmp?

eh, I meant smtp. Woops. 

Re: More IMAP testing, please

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> We got a fair amount of nnimap testing and bug stomping yesterday, but
> I'm sure there must be a lot more problems hiding here and there, so
> it'd be great if more IMAP users took the plunge and gave the new nnimap
> a whirl and report problems (or whether it works fine for your).

It's working fine for me here to my Exchange server (2003, I
think). Only one issue so far: is article deletion supposed to work? I
mark articles with B DEL, but they're still there marked 'O' when I
re-enter the group. Is there some please-expunge-on-exit variable I need
to set?

Regards

Robert

PS 'G x' fails as well, probably because you haven't implemented
nnimap-expunge yet. :)

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> It's working fine for me here to my Exchange server (2003, I
> think). Only one issue so far: is article deletion supposed to work? I
> mark articles with B DEL, but they're still there marked 'O' when I
> re-enter the group. Is there some please-expunge-on-exit variable I need
> to set?

There's now a new server variable `nnimap-expunge' that defaults to nil,
but I think the general feeling here now is that it should default to t?

> PS 'G x' fails as well, probably because you haven't implemented
> nnimap-expunge yet. :)

It's implemented now.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> It's working fine for me here to my Exchange server (2003, I
>> think). Only one issue so far: is article deletion supposed to work? I
>> mark articles with B DEL, but they're still there marked 'O' when I
>> re-enter the group. Is there some please-expunge-on-exit variable I need
>> to set?
>
> There's now a new server variable `nnimap-expunge' that defaults to nil,
> but I think the general feeling here now is that it should default to t?

Yes.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: More IMAP testing, please

[Eythan Weg]

Lars Magne Ingebrigtsen <larsi@gnus.org>
Sun, 19 Sep 2010 15:08:32 +0200

   We got a fair amount of nnimap testing and bug stomping yesterday, but
   I'm sure there must be a lot more problems hiding here and there, so
   it'd be great if more IMAP users took the plunge and gave the new nnimap
   a whirl and report problems (or whether it works fine for your).


I access two servers: gmail and microsoft exchange 2007 imap server.
The former works and the latter is denied with this message.
  
  
  *** Fatal error: A TLS packet with unexpected length was received.
  *** Handshake has failed
  GNUTLS ERROR: A TLS packet with unexpected length was received.
  Resolving 'owa.ivytech.edu'...
  Connecting to '168.91.17.201:143'...
  
  Process *nnimap* exited abnormally with code 1
  This method of specifying algorithms is deprecated. Please use the
  --priority option.
  *** Fatal error: A TLS packet with unexpected length was received.
  *** Handshake has failed
  GNUTLS ERROR: A TLS packet with unexpected length was received.
  Resolving 'owa.ivytech.edu'...
  Connecting to '168.91.17.201:143'...
  
  Process *nnimap* exited abnormally with code 1
  592:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown 
 protocol:s23_clnt.c:604:
  CONNECTED(00000003)
  
  Process *nnimap* finished


Any ideas what to do?

Thanks, Eythan 

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

eythanweg@gmail.com (Eythan Weg) writes:

> I access two servers: gmail and microsoft exchange 2007 imap server.
> The former works and the latter is denied with this message.

[...]

>   592:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown 
>  protocol:s23_clnt.c:604:
>   CONNECTED(00000003)
>
>   Process *nnimap* finished

Hm.

What happens if you say

gnutls-cli -p 993 host

and

openssl s_client -connect host:993 -no_ssl2 -ign_eof

?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Andrew Cohen]

I have been doing client-side splitting for 24 hours and it seems to be
working without issues. I haven't lost any mail and it seems to end up
in the right groups. 

It looks like expiry is not yet fully implemented? I send mail to backup
groups after certain time periods and it would be nice to have this
happen. 

Andy

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> It looks like expiry is not yet fully implemented? I send mail to backup
> groups after certain time periods and it would be nice to have this
> happen. 

No, expiry (and EXPUNGE) isn't implemented at all.  I'm not quite sure
what the most logical EXPUNGE behaviour would be...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Andrew Cohen <cohen@andy.bu.edu> writes:
>
>> It looks like expiry is not yet fully implemented? I send mail to backup
>> groups after certain time periods and it would be nice to have this happen.
>
> No, expiry (and EXPUNGE) isn't implemented at all.  I'm not quite sure
> what the most logical EXPUNGE behaviour would be...

Well, the two behaviours that might be of interest to me personally are:

1. mark deleted and always expunge on group exit.
2. move to a target folder only.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Daniel Pittman <daniel@rimspace.net>:

> Well, the two behaviours that might be of interest to me personally are:

> 1. mark deleted and always expunge on group exit.
> 2. move to a target folder only.

Coincidentially the ones I use myself.

I `B DEL' out everything I don't want to keep (spam, invoices for paid
bills, etc.), and let the non-ticked articles eventually expire to an
archive group.

Re: More IMAP testing, please

[Daniel Pittman]

Steinar Bang <sb@dod.no> writes:
>>>>>> Daniel Pittman <daniel@rimspace.net>:
>
>> Well, the two behaviours that might be of interest to me personally are:
>
>> 1. mark deleted and always expunge on group exit.
>> 2. move to a target folder only.
>
> Coincidentially the ones I use myself.
>
> I `B DEL' out everything I don't want to keep (spam, invoices for paid
> bills, etc.), and let the non-ticked articles eventually expire to an
> archive group.

I bet they cover the 90 percent case, and probably even the 99 percent, for
what "most users" of IMAP do.  OTOH, Gnus users are not exactly typical...

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: More IMAP testing, please

[Robert Pluim]

Daniel Pittman <daniel@rimspace.net> writes:

> Steinar Bang <sb@dod.no> writes:
>>>>>>> Daniel Pittman <daniel@rimspace.net>:
>>
>>> Well, the two behaviours that might be of interest to me personally are:
>>
>>> 1. mark deleted and always expunge on group exit.
>>> 2. move to a target folder only.
>>
>> Coincidentially the ones I use myself.
>>
>> I `B DEL' out everything I don't want to keep (spam, invoices for paid
>> bills, etc.), and let the non-ticked articles eventually expire to an
>> archive group.
>
> I bet they cover the 90 percent case, and probably even the 99 percent, for
> what "most users" of IMAP do.  OTOH, Gnus users are not exactly typical...

I'm not sure about that in this case. Given a command with the following
docstring:

  Documentation:
  Delete the N next (mail) articles.
  This command actually deletes articles.  This is not a marking
  command.  The article will disappear forever from your life, never to
  return.

I think a typical Gnus user would expect the article to disappear,
dammit. As would a non-typical one.

Robert

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I think a typical Gnus user would expect the article to disappear,
> dammit. As would a non-typical one.

Yes.  But it's not that simple.  :-)

For servers that have the UIDPLUS capability, `B DEL' should (and does,
I think?) first mark the message as deleted, and then expunge the
message in question.

If the server doesn't have UIDPLUS, the only way to make messages go
totally away is to do an EXPUNGE on the mailbox.  That will make all
messages that are marked as \Deleted disappear forever.

So what should the default behaviour on non-UIDPLUS servers be?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ted Zlatanov]

On Tue, 21 Sep 2010 18:29:20 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Robert Pluim <rpluim@gmail.com> writes:
>> I think a typical Gnus user would expect the article to disappear,
>> dammit. As would a non-typical one.

LMI> Yes.  But it's not that simple.  :-)

LMI> For servers that have the UIDPLUS capability, `B DEL' should (and does,
LMI> I think?) first mark the message as deleted, and then expunge the
LMI> message in question.

LMI> If the server doesn't have UIDPLUS, the only way to make messages go
LMI> totally away is to do an EXPUNGE on the mailbox.  That will make all
LMI> messages that are marked as \Deleted disappear forever.

LMI> So what should the default behaviour on non-UIDPLUS servers be?

Same as now: EXPUNGE the mailbox.  This is normal IMAP behavior so I
don't think it will surprise users.

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Same as now: EXPUNGE the mailbox.  This is normal IMAP behavior so I
> don't think it will surprise users.

It's normal to EXPUNGE the mailbox every time the user clicks "delete"
on an article?  I think somebody protested when I had the inbox mail
splitting thing do an EXPUNGE on the inbox, or do I misremember?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ted Zlatanov]

On Tue, 21 Sep 2010 19:06:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Same as now: EXPUNGE the mailbox.  This is normal IMAP behavior so I
>> don't think it will surprise users.

LMI> It's normal to EXPUNGE the mailbox every time the user clicks "delete"
LMI> on an article?

But `B DEL' is not "delete" is it?  It's "delete this truly and forever"
and in order to do that, IMAP without UIDPLUS needs to do an expunge.

I guess you could temporarily set the other `E' articles to a
gnus-keep-around mark, run the expunge, then set them back...  But
that's hacky.

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I guess you could temporarily set the other `E' articles to a
> gnus-keep-around mark, run the expunge, then set them back...  But
> that's hacky.

It is kinda hacky...  but it's possible.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> I guess you could temporarily set the other `E' articles to a
>> gnus-keep-around mark, run the expunge, then set them back...  But
>> that's hacky.
>
> It is kinda hacky...  but it's possible.

I thought expirable articles would be marked with the gnus-expire
flag, not the \Deleted flag, so they wouldn't be affected by an
expunge.  Or am I misunderstanding what Ted meant by `E'?

In general, I'm confused about whether you guys are discussing
total-expire or auto-expire.  I use auto-expire, and I hope that
old articles marked as expirable will be deleted now and then.

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I thought expirable articles would be marked with the gnus-expire
> flag, not the \Deleted flag, so they wouldn't be affected by an
> expunge.  Or am I misunderstanding what Ted meant by `E'?

It's a bit confusing.  :-)

"Expiry" is a Gnus concept, and doesn't touch the IMAP servers much.
(That is, Gnus stores the mark on the server, but it doesn't understand
it.)

What we're discussing is how Gnus is to delete articles -- either
expired articles, or just plain deleted articles.  The only way to do
that is to mark the articles in the server as \Deleted, and then either
say "UID EXPUNGE", which will, er, delete the articles, or just say
"EXPUNGE", which will delete all \Delete-marked articles in the group.

I'm not sure whether the latter is naughty or not.  It's tempting to do,
because it's so easy, but I have no idea what IMAP users expect to
happen.

This only affects users of older versions of Dovecot of the major
server types in use at present, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> What we're discussing is how Gnus is to delete articles -- either
> expired articles, or just plain deleted articles.  The only way to do
> that is to mark the articles in the server as \Deleted, and then either
> say "UID EXPUNGE", which will, er, delete the articles, or just say
> "EXPUNGE", which will delete all \Delete-marked articles in the group.
>
> I'm not sure whether the latter is naughty or not.  It's tempting to do,
> because it's so easy, but I have no idea what IMAP users expect to
> happen.
>

I expect it to happen :)

> This only affects users of older versions of Dovecot of the major
> server types in use at present, I think.

And whatever variant of Exchange I happen to be talking to, which I
think is 2003. (unless we're missing some magic incantation to make
Exchange speak 'modern' IMAP, which is always possible).

Regards

Robert

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Dan Christensen <jdc@uwo.ca>:

> I thought expirable articles would be marked with the gnus-expire
> flag, not the \Deleted flag, so they wouldn't be affected by an
> expunge.  Or am I misunderstanding what Ted meant by `E'?

> In general, I'm confused about whether you guys are discussing
> total-expire or auto-expire. 

The expiry mechanism is the same, whether you use total-expire or
auto-expire, isn't it...?

The way I've understood it, is that if you have
 (total-expire . t)
in the group parameters, then the articles marked with \Deleted will be
affected by the expiry.

Or was that "marked with \Seen"...?  Hm... now I'm confused.
Nah... probably "marked with \Seen".

Anyway, the behaviour I've had with the previous nnimap was that I had
total-expire.  I used `B DEL' to permanently delete stuff I didn't want
to keep, and just let the rest lie there marked with \Seen (I
think...?), and then occasionally expire that over into an archive
group/mailbox. 

For this to work then `B DEL' had to set \Delete, I guess.  And unticked
articles were marked with \Seen.

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Tue, Sep 21 2010, Lars Magne Ingebrigtsen wrote:

> It is kinda hacky...  but it's possible.

And why not be conservative by default and let the user set a variable
to allow Gnus to do EXPUNGE wherever it wants?

   (setq nnimap-stop-being-be-conservative t)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> And why not be conservative by default and let the user set a variable
> to allow Gnus to do EXPUNGE wherever it wants?
>
>    (setq nnimap-stop-being-be-conservative t)

I had hoped not having to have that be customisable.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ted Zlatanov]

On Tue, 21 Sep 2010 19:29:25 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Tue, Sep 21 2010, Lars Magne Ingebrigtsen wrote:
>> It is kinda hacky...  but it's possible.

JD> And why not be conservative by default and let the user set a variable
JD> to allow Gnus to do EXPUNGE wherever it wants?

JD>    (setq nnimap-stop-being-be-conservative t)

It places an extra burden on every user ("what is that?  should I use
it?") for the sake of the few who don't have UIDPLUS.  Articles marked
as `E' (expirable) are clearly meant to be deleted sooner or later so
doing it sooner is not a problem IMO.

We can also simply require UIDPLUS.  It's 2010 after all.

Lars, do you have a list of supported IMAP servers and capability
strings?  I'm guessing Dovecot and Courier at least?

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Lars, do you have a list of supported IMAP servers and capability
> strings?  I'm guessing Dovecot and Courier at least?

I don't know of any servers where the new nnimap is confirmed not to
work, except that server that uses UIDs to compute big prime numbers.
And that's more a Gnus issue than an nnimap issue.

Dovecot on Debian Stale doesn't support UIDPLUS.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It places an extra burden on every user ("what is that?  should I use
> it?") for the sake of the few who don't have UIDPLUS.  Articles marked
> as `E' (expirable) are clearly meant to be deleted sooner or later so
> doing it sooner is not a problem IMO.

Again, I don't see what articles marked with 'E' have to do with the
expunging problem.  (I read Lars' reply to my last message, and it
confirmed how I thought things worked.)  If I mark an article with E,
and Gnus does a full expunge on the IMAP group, that article will be
fine, since it isn't marked as \Deleted.

So I think it's fine for Gnus to simply do a full expunge when UIDPLUS
is not supported.  It should only affect articles that get marked as
\Deleted (but don't get expunged) by another client, since Gnus doesn't
do that.

(But if I'm misunderstanding something, then let me make it clear that I
*don't* think it's fine for articles marked 'E' to be expired early
just because I do `B DEL' on a different article.  But I don't think 
this will happen.)

Dan

Re: More IMAP testing, please

[Ted Zlatanov]

On Tue, 21 Sep 2010 15:33:26 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> It places an extra burden on every user ("what is that?  should I use
>> it?") for the sake of the few who don't have UIDPLUS.  Articles marked
>> as `E' (expirable) are clearly meant to be deleted sooner or later so
>> doing it sooner is not a problem IMO.

DC> Again, I don't see what articles marked with 'E' have to do with the
DC> expunging problem.  (I read Lars' reply to my last message, and it
DC> confirmed how I thought things worked.)  If I mark an article with E,
DC> and Gnus does a full expunge on the IMAP group, that article will be
DC> fine, since it isn't marked as \Deleted.

I mentioned this in my bug followup: it used to be marked as \Deleted
and expunged after `gnus-summary-expire-articles-now'.  So you are right
about the current behavior but it's not consistent with the prior
behavior.

DC> (But if I'm misunderstanding something, then let me make it clear that I
DC> *don't* think it's fine for articles marked 'E' to be expired early
DC> just because I do `B DEL' on a different article.  But I don't think 
DC> this will happen.)

I think so far the votes on that behavior are 2 in favor of "just
expunge," 2 in favor of "do a hack to work around the lack of UIDPLUS,"
and 1 (yours) in favor of "don't expunge in that case."

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think so far the votes on that behavior are 2 in favor of "just
> expunge," 2 in favor of "do a hack to work around the lack of UIDPLUS,"
> and 1 (yours) in favor of "don't expunge in that case."

No, I haven't voted.  :-)

I'm just asking.  If I were to vote, I'd vote for "just expunge".

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think so far the votes on that behavior are 2 in favor of "just
>> expunge," 2 in favor of "do a hack to work around the lack of UIDPLUS,"
>> and 1 (yours) in favor of "don't expunge in that case."

> No, I haven't voted.  :-)

> I'm just asking.  If I were to vote, I'd vote for "just expunge".

Well... I now have UIDPLUS, so my vote goes for "just expunge".

The people who may be bitten by this, are probably the Exchange
users...?  They don't have the option of upgrading to a version that
supports it.

Re: More IMAP testing, please

[Jake Colman]

>>>>> "SB" == Steinar Bang <sb@dod.no> writes:

   SB> The people who may be bitten by this, are probably the Exchange
   SB> users...?  They don't have the option of upgrading to a version
   SB> that supports it.

My office is still on Exchange 2003 and I have no option of changing
that.  I'm sure I'm not the only one in this position.  Since the
current nnimap works with Exchange 2003, it would be a darned shame if
the new and improved nnimap does not.

-- 
Jake Colman | Director, Software Development
Principia Partners LLC
101 West Elm Street | Conshohocken | PA 19428 | +1 (610) 755 9770
t: +1 (610) 755 9786 | c: +1 (610) 348 2788 | f: +1 (201) 221 8929
e: colman@ppllc.com | w: www.principiapartners.com
Credit technology innovation awards winner 2008 and 2009

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Jake Colman <colman@ppllc.com> writes:

> My office is still on Exchange 2003 and I have no option of changing
> that.  I'm sure I'm not the only one in this position.  Since the
> current nnimap works with Exchange 2003, it would be a darned shame if
> the new and improved nnimap does not.

It should work.  We're just discussing whether EXPUNGE should be on by
default or not for servers that don't support UIDPLUS.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ted Zlatanov]

On Wed, 22 Sep 2010 15:26:31 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> No, I haven't voted.  :-)

Oh, but you WANT to.

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Oh, but you WANT to.

Yes I do.  :-)

I'll flip the default of nnimap-expire to t, and remove
nnimap-expire-inbox.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'll flip the default of nnimap-expire to t, and remove
> nnimap-expire-inbox.

I mean nnimap-expunge*.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 21 Sep 2010 15:33:26 -0400 Dan Christensen <jdc@uwo.ca> wrote: 
>
> DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> It places an extra burden on every user ("what is that?  should I use
>>> it?") for the sake of the few who don't have UIDPLUS.  Articles marked
>>> as `E' (expirable) are clearly meant to be deleted sooner or later so
>>> doing it sooner is not a problem IMO.
>
> DC> Again, I don't see what articles marked with 'E' have to do with the
> DC> expunging problem.  (I read Lars' reply to my last message, and it
> DC> confirmed how I thought things worked.)  If I mark an article with E,
> DC> and Gnus does a full expunge on the IMAP group, that article will be
> DC> fine, since it isn't marked as \Deleted.
>
> I mentioned this in my bug followup: it used to be marked as \Deleted
> and expunged after `gnus-summary-expire-articles-now'.  So you are right
> about the current behavior but it's not consistent with the prior
> behavior.

Sorry, but I still don't understand what you mean, and I'm not sure
where to look to see your bug followup.

Can you explain a scenario in which doing expunge when the user hits `B
DEL' will cause other articles to be removed from the IMAP server?

a) If I have an article marked `E', and do `B DEL' on another article,
my article marked `E' should still be around.  I'm not aware of any
version of Gnus that would put the \Deleted flag on the `E' article.

b) If I run `gnus-summary-expire-articles-now', I would expect that all
articles marked with `E' will get removed from the IMAP server (i.e. get
marked \Deleted and then expunged).  I don't use this much, but the
documentation is pretty clear.  Has something changed here?  And what
does this have to do with `B DEL'?

As far as I know, Gnus never puts the \Deleted mark on an article
without also expunging it, so unless the user uses a different client
to do this, running an expunge should be safe.

Ted Zlatanov <tzz@lifelogs.com> writes:

> DC> (But if I'm misunderstanding something, then let me make it clear that I
> DC> *don't* think it's fine for articles marked 'E' to be expired early
> DC> just because I do `B DEL' on a different article.  But I don't think 
> DC> this will happen.)
>
> I think so far the votes on that behavior are 2 in favor of "just
> expunge," 2 in favor of "do a hack to work around the lack of UIDPLUS,"
> and 1 (yours) in favor of "don't expunge in that case."

If "yours" means me, I don't think that represents my vote, since I
don't see how an article marked `E' will be affected by doing `B DEL'.
I still vote for always doing an expunge, unless someone explains how
that could be a problem.

[My server *has* UIDPLUS, so I'm not affected by any of this, but
I'd like to clear up my confusion.]

Dan

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Can you explain a scenario in which doing expunge when the user hits `B
> DEL' will cause other articles to be removed from the IMAP server?

If the user has marked oodles of messages as \Deleted via a different
user agent, and not issued an EXPUNGE, then deleting any message from
Gnus in the mailbox will expunge them all.  (In the non-UIDPLUS case.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Ted Zlatanov]

On Wed, 22 Sep 2010 13:05:11 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Sorry, but I still don't understand what you mean, and I'm not sure
DC> where to look to see your bug followup.

I think I'm confusing everyone.  Sorry for the noise.

Ted

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

JD> And why not be conservative by default and let the user set a variable
JD> to allow Gnus to do EXPUNGE wherever it wants?

JD> (setq nnimap-stop-being-be-conservative t)

> It places an extra burden on every user ("what is that?  should I use
> it?")

AOL.

> for the sake of the few who don't have UIDPLUS.

Well... I still don't... ah what the hell
  http://backports.debian.org/Instructions/

# telnet localhost imap
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN AUTH=CRAM-MD5] Dovecot ready.
a001 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS STARTTLS AUTH=PLAIN AUTH=CRAM-MD5
a001 OK Capability completed.
a002 logout
* BYE Logging out
a002 OK Logout completed.
Connection closed by foreign host.

Now I also support UIDPLUS!

Re: More IMAP testing, please

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]


Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 21 Sep 2010 19:06:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org>
> wrote:
>
> LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
> LMI> It's normal to EXPUNGE the mailbox every time the user clicks "delete"
> LMI> on an article?
>
> But `B DEL' is not "delete" is it?  It's "delete this truly and forever"
> and in order to do that, IMAP without UIDPLUS needs to do an expunge.
>
> I guess you could temporarily set the other `E' articles to a
> gnus-keep-around mark, run the expunge, then set them back...  But
> that's hacky.

I use multiple IMAP clients, and have configured gnus thusly:

  (defun gdt-gnus-move-to-trash ()
    (interactive)
    (gnus-summary-move-article nil gdt-trash-newsgroup))

  (add-hook              
    'gnus-summary-mode-hook
    (lambda ()
      (local-set-key "d" 'gdt-gnus-move-to-trash)))

  (setq gdt-trash-newsgroup "nnimap+work.lexort.com:Trash")

and then 'd' moves an article to Trash, which I have set up as
total-expire.

I suppose there's some notion of in-group-but-marked-deleted, but I
never figured that out, and I've been happy with the above.

Occasionally I do 'B DEL' as well.
Mostly because my macro doesn't play well with seting marks with '#'.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: More IMAP testing, please

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> Same as now: EXPUNGE the mailbox.  This is normal IMAP behavior so I
>> don't think it will surprise users.
>
> It's normal to EXPUNGE the mailbox every time the user clicks "delete"
> on an article?

Nope.  Mostly, "normal" would actually be the "move to trash" case.

OTOH, the IMAP design, *and* implementations, assume that EXPUNGE can be
expensive, and it is usually reserved for things like doing once, not on every
single delete operation.

> I think somebody protested when I had the inbox mail splitting thing do an
> EXPUNGE on the inbox, or do I misremember?

I don't think it is a good idea, but I could live with it. :)

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: More IMAP testing, please

[Robert Pluim]

Daniel Pittman <daniel@rimspace.net> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>
>>> Same as now: EXPUNGE the mailbox.  This is normal IMAP behavior so I
>>> don't think it will surprise users.
>>
>> It's normal to EXPUNGE the mailbox every time the user clicks "delete"
>> on an article?
>
> Nope.  Mostly, "normal" would actually be the "move to trash" case.
>
> OTOH, the IMAP design, *and* implementations, assume that EXPUNGE can
> be expensive, and it is usually reserved for things like doing once,
> not on every single delete operation.
>

Whatever it is that Lars has implemented up to now for B-DEL works fine
for me (I think that's EXPUNGE every time), and I'm one of the
aforementioned Exchange users.  I have no complaints about the
performance, although I haven't tried it in a group with more than 400
articles.

If it ever turns out to be too expensive, I can always bug you to
implement 'Mark deleted & expunge on group exit' ;)

Regards

Robert

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> OTOH, the IMAP design, *and* implementations, assume that EXPUNGE can be
> expensive, and it is usually reserved for things like doing once, not on every
> single delete operation.

When expiring, the EXPUNGE is done at the end, after marking all the
messages as \Deleted.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> I think a typical Gnus user would expect the article to disappear,
>> dammit. As would a non-typical one.
>
> Yes.  But it's not that simple.  :-)
>
> For servers that have the UIDPLUS capability, `B DEL' should (and does,
> I think?) first mark the message as deleted, and then expunge the
> message in question.
>

I don't think mine has UIDPLUS, then. Which log buffer do I need to look
at to figure it out?

> If the server doesn't have UIDPLUS, the only way to make messages go
> totally away is to do an EXPUNGE on the mailbox.  That will make all
> messages that are marked as \Deleted disappear forever.
>
> So what should the default behaviour on non-UIDPLUS servers be?

I vote for expunge-on-exit, because that matches my expectations, but I
would be fine with expunge-manually as default, as long as there was a
configuration variable or a hook I could use to get expunge-on-exit.

Regards

Robert

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I don't think mine has UIDPLUS, then. Which log buffer do I need to look
> at to figure it out?

Go to the *nnimap <server> ...* buffer and say

(nnimap-capabilities nnimap-object)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> I don't think mine has UIDPLUS, then. Which log buffer do I need to look
>> at to figure it out?
>
> Go to the *nnimap <server> ...* buffer and say
>
> (nnimap-capabilities nnimap-object)

("IMAP4" "IMAP4REV1" "AUTH=NTLM" "AUTH=GSSAPI" "AUTH=PLAIN" "IDLE" "NAMESPACE" "LITERAL+")

:(

Robert

Re: More IMAP testing, please

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> For servers that have the UIDPLUS capability, `B DEL' should (and does,
> I think?) first mark the message as deleted, and then expunge the
> message in question.

> If the server doesn't have UIDPLUS, the only way to make messages go
> totally away is to do an EXPUNGE on the mailbox.  That will make all
> messages that are marked as \Deleted disappear forever.

Hm... it sounds like to be safe I should to pull dovecot 1.2.x from
debian backports, rather than wait for a new stable ("squeeze") to pull
it in for me...?

(For new readers: the current debian stable ("lenny") has dovecot
version 1.0.15, which does not support UIDPLUS.  That support came, in
2008 or thereabouts, with version 1.1, back )

Re: More IMAP testing, please

[Ted Zlatanov]

On Mon, 20 Sep 2010 23:37:59 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Andrew Cohen <cohen@andy.bu.edu> writes:
>> It looks like expiry is not yet fully implemented? I send mail to backup
>> groups after certain time periods and it would be nice to have this
>> happen. 

LMI> No, expiry (and EXPUNGE) isn't implemented at all.  I'm not quite sure
LMI> what the most logical EXPUNGE behaviour would be...

I mark articles `E'xpired and then run
`gnus-summary-expire-articles-now' all the time.  Can that work, please?

Ted

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I mark articles `E'xpired and then run
> `gnus-summary-expire-articles-now' all the time.  Can that work, please?

Sure.  I'm looking at how to make expiry work now...

So nnimap needs the article dates of everything so that it can expire
articles.  "UID SEARCH SENTBEFORE" should do the trick, shouldn't it?
Alternatively "UID SEARCH SINCE", and the delete all the articles not
mentioned by that command.

(-expire-articles-now would, of course, just delete the list of articles
without looking at the date, so it should be a much faster operation.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So nnimap needs the article dates of everything so that it can expire
> articles.  "UID SEARCH SENTBEFORE" should do the trick, shouldn't it?
> Alternatively "UID SEARCH SINCE", and the delete all the articles not
> mentioned by that command.

I forgot to ask what I wanted to ask.

Do all the common servers implement "UID SEARCH SENTBEFORE", and is it
usable?  That is, is it fast enough to actually be used?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Tue, Sep 21 2010, Lars Magne Ingebrigtsen wrote:

> Do all the common servers implement "UID SEARCH SENTBEFORE", and is it
> usable?  That is, is it fast enough to actually be used?

I don't know if all server implements it, but Dovecot, my favorite,
implements it and make it fast because it has a good indexing function.

I did not test, I can't find the correct syntax for that IMAP command…

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: More IMAP testing, please

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I don't know if all server implements it, but Dovecot, my favorite,
> implements it and make it fast because it has a good indexing function.

I've tried it with Dovecot and Gmail now, and they both seem fast.
Julien Danjou <julien@danjou.info> writes:

> I did not test, I can't find the correct syntax for that IMAP command…

It's pretty wacky:

UID SEARCH SENTBEFORE 14-SEP-2010

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More IMAP testing, please

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On Tue, Sep 21 2010, Lars Magne Ingebrigtsen wrote:

> It's pretty wacky:
>
> UID SEARCH SENTBEFORE 14-SEP-2010

. UID SEARCH SENTBEFORE 14-SEP-2010
* SEARCH 1 2 […] 11514 11515
. OK Search completed (0.385 secs).

Seems fast here. :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]