nnimap does not create non-existent groups on expiry (lost mail)

nnimap does not create non-existent groups on expiry (lost mail)

[Nevin Kapur]

I recently moved from using nnfolder groups as expiry targets to using
nnimap groups.  In either case an expiry target that depends on the
message's month and year is being used.  When I was using nnfolder
groups as expiry targets, non-existent groups were being automatically
created.  However this is not the case with nnimap groups.  For
example, I set nnmail-expiry-target to "nnimap:mail/ne", a group that
does not exist, and set about expiring some mail.  To my horror I
discovered that the messages were expired and the new group was not
created, resulting in lost mail.

Here's an excerpt from the imap log: the group is not created and Gnus
is silently losing mail.

3510 UID COPY 7 "mail/ne"
* OK [TRYCREATE] Must create mailbox before copy
3510 NO [TRYCREATE] UID COPY failed: No such destination mailbox

Any fixes?

-Nevin

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Nevin Kapur]

I've made the following naive change and now nonexistent groups do get
created on expiry.  Is this the right thing to do?  If so, would
someone apply the patch?

Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 6.56
diff -u -r6.56 nnimap.el
--- lisp/nnimap.el	2002/12/28 18:52:55	6.56
+++ lisp/nnimap.el	2003/01/06 18:31:54
@@ -1422,7 +1422,7 @@
 			  nnimap-current-move-group)
 			 (imap-message-copy (number-to-string
 					     nnimap-current-move-article)
-					    group 'dontcreate nil
+					    group nil nil
 					    nnimap-server-buffer))
 		  (with-current-buffer (current-buffer)
 		    (goto-char (point-min))

2003-01-06  Nevin Kapur  <nevin@jhu.edu>

	* nnimap.el (nnimap-request-accept-article): Create group if it
		  doesn't exist.

-Nevin                  

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Simon Josefsson]

Nevin Kapur <nevin@jhu.edu> writes:

> I've made the following naive change and now nonexistent groups do get
> created on expiry.  Is this the right thing to do?  If so, would
> someone apply the patch?

In theory it is the right thing, but various IMAP server bugs make the
TRYCREATE stuff unreliable, so with your patch I believe nnimap would
have sent a CREATE command for every expired article (which would have
failed in the normal case).  Does the following work instead?  It also
has the feature that no articles are expired if the group could not be
created.  Thanks for debugging this.

Index: nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 6.56
diff -u -p -u -w -r6.56 nnimap.el
--- nnimap.el	2002/12/28 18:52:55	6.56
+++ nnimap.el	2003/01/06 23:08:26
@@ -1341,7 +1344,8 @@ function is generally only called when G
 		imap-current-message))
 
 (defun nnimap-expiry-target (arts group server)
-  (unless (eq nnmail-expiry-target 'delete)
+  (when (and (not (eq nnmail-expiry-target 'delete))
+	     (nnimap-request-create-group group server))
     (with-temp-buffer
       (dolist (art arts)
 	(nnimap-request-article art group server (current-buffer))

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Nevin Kapur]

Simon Josefsson <jas@extundo.com> writes:

> In theory it is the right thing, but various IMAP server bugs make the
> TRYCREATE stuff unreliable, so with your patch I believe nnimap would
> have sent a CREATE command for every expired article (which would have
> failed in the normal case).  Does the following work instead?  It also
> has the feature that no articles are expired if the group could not be
> created.  

I applied your patch, but I don't think it does the right thing.
Isn't it trying to create the (already existing) group from where the
function is being called rather than the target given by
nnmail-expiry-target?

I took a stab at modifying the patch but ran into some difficulties.
I'm not sure if one should check for the existence of the target group
in the function nnimap-expiry-target when it is possible that
nnmail-expiry-target returns a non-nnimap group.

Is it better to move the check inside nnimap-request-accept-article?

Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 6.56
diff -u -r6.56 nnimap.el
--- lisp/nnimap.el	2002/12/28 18:52:55	6.56
+++ lisp/nnimap.el	2003/01/07 03:09:43
@@ -1413,7 +1413,8 @@
 	result))))
 
 (deffoo nnimap-request-accept-article (group &optional server last)
-  (when (nnimap-possibly-change-server server)
+  (when (and (nnimap-possibly-change-server server)
+	     (nnimap-request-create-group group))
     (let (uid)
       (if (setq uid
 		(if (string= nnimap-current-server nnimap-current-move-server)


-Nevin

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Simon Josefsson]

Nevin Kapur <nevin@jhu.edu> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> In theory it is the right thing, but various IMAP server bugs make the
>> TRYCREATE stuff unreliable, so with your patch I believe nnimap would
>> have sent a CREATE command for every expired article (which would have
>> failed in the normal case).  Does the following work instead?  It also
>> has the feature that no articles are expired if the group could not be
>> created.  
>
> I applied your patch, but I don't think it does the right thing.
> Isn't it trying to create the (already existing) group from where the
> function is being called rather than the target given by
> nnmail-expiry-target?

Oops!  Right.

> I took a stab at modifying the patch but ran into some difficulties.
> I'm not sure if one should check for the existence of the target group
> in the function nnimap-expiry-target when it is possible that
> nnmail-expiry-target returns a non-nnimap group.
>
> Is it better to move the check inside nnimap-request-accept-article?

I think it is more appropriate in nnmail-expiry-target-group.  What do
the nnmail experts say?  Perhaps we can document this so it isn't a
problem in the future:

   Is nnfoo-request-accept-article supposed to create the group if it
   does not exists, or is that up to the caller?

The answer should go into the manual, I think.

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> I think it is more appropriate in nnmail-expiry-target-group.  What do
> the nnmail experts say?  Perhaps we can document this so it isn't a
> problem in the future:
>
>    Is nnfoo-request-accept-article supposed to create the group if it
>    does not exists, or is that up to the caller?

It's up to the caller.

> The answer should go into the manual, I think.

Now done.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> I think it is more appropriate in nnmail-expiry-target-group.  What do
>> the nnmail experts say?  Perhaps we can document this so it isn't a
>> problem in the future:
>>
>>    Is nnfoo-request-accept-article supposed to create the group if it
>>    does not exists, or is that up to the caller?
>
> It's up to the caller.

Then n-e-t-g should be fixed too?

2003-01-08  Simon Josefsson  <jas@extundo.com>

	* nnmail.el (nnmail-expiry-target-group): Request group, create it
	not successful.

--- nnmail.el.~6.54.~	Sat Jan  4 00:13:49 2003
+++ nnmail.el	Wed Jan  8 07:09:52 2003
@@ -1748,9 +1748,11 @@
     (when (nnheader-functionp target)
       (setq target (funcall target group)))
     (unless (eq target 'delete)
+      (when (or (gnus-request-group target)
+		(gnus-request-create-group target))
       (let ((group-art (gnus-request-accept-article target nil nil t)))
 	(when (consp group-art)
-	  (gnus-group-mark-article-read target (cdr group-art)))))))
+	    (gnus-group-mark-article-read target (cdr group-art))))))))
 
 (defun nnmail-fancy-expiry-target (group)
   "Returns a target expiry group determined by `nnmail-fancy-expiry-targets'."

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> Then n-e-t-g should be fixed too?
>
> 2003-01-08  Simon Josefsson  <jas@extundo.com>
>
> 	* nnmail.el (nnmail-expiry-target-group): Request group, create it
> 	not successful.

Yes, I think that's correct.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Then n-e-t-g should be fixed too?
>>
>> 2003-01-08  Simon Josefsson  <jas@extundo.com>
>>
>> 	* nnmail.el (nnmail-expiry-target-group): Request group, create it
>> 	not successful.
>
> Yes, I think that's correct.

OK, installed.  Nevin, please try current CVS.

Re: nnimap does not create non-existent groups on expiry (lost mail)

[Nevin Kapur]

Simon Josefsson <jas@extundo.com> writes:

> OK, installed.  Nevin, please try current CVS.

Looks great.  Thanks for fixing this.

-Nevin