latest imap changes break refiling

latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

Hello,

I'm using the master branch of the git version of gnus, and after
a recent update I have some problems when refiling messages. This is the
backtrace that I get:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("^[^:]+:" nil)
  nnimap-retrieve-group-data-early("local" (nil))
  nnimap-request-group("old.lists.formal.2014.01" "local" nil nil)
  gnus-request-group("nnimap+local:old.lists.formal.2014.01")
  nnmail-expiry-target-group("nnimap+local:old.lists.formal.2014.01" "lists.formal")

`nnimap+local' is a local dovecot server (I refile messages that are
older than 1 year old locally):

#+begin_src emacs-lisp
(setq gnus-secondary-select-methods 
      '((nnimap 
         "local"
         (nnimap-address "localhost")
         (nnimap-server-port 143)
         (nnimap-stream network))))
#+end_src

Is this a bug with the recent imap changes?

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 284 bytes --]

On 2015-01-29 09:02, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> I'm using the master branch of the git version of gnus, and after
> a recent update I have some problems when refiling messages.

I meant "expiring".

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Lars Ingebrigtsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Hello,
>
> I'm using the master branch of the git version of gnus, and after
> a recent update I have some problems when refiling messages. This is the
> backtrace that I get:
>
> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>   string-match("^[^:]+:" nil)
>   nnimap-retrieve-group-data-early("local" (nil))
>   nnimap-request-group("old.lists.formal.2014.01" "local" nil nil)
>   gnus-request-group("nnimap+local:old.lists.formal.2014.01")

Hm.  Is "nnimap+local:old.lists.formal.2014.01" a group that Gnus
possibly doesn't know about yet (i.e., it isn't subscribed yet when this
is done)?  If so, I think I understand why this happens...

I'll push a fix in a few minutes.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On 2015-01-29 19:06, Lars Ingebrigtsen <larsi@gnus.org> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> I'm using the master branch of the git version of gnus, and after
>> a recent update I have some problems when refiling messages. This is the
>> backtrace that I get:
>>
>> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>>   string-match("^[^:]+:" nil)
>>   nnimap-retrieve-group-data-early("local" (nil))
>>   nnimap-request-group("old.lists.formal.2014.01" "local" nil nil)
>>   gnus-request-group("nnimap+local:old.lists.formal.2014.01")
>
> Hm.  Is "nnimap+local:old.lists.formal.2014.01" a group that Gnus
> possibly doesn't know about yet (i.e., it isn't subscribed yet when this
> is done)?  If so, I think I understand why this happens...

Well, it shows up in the Group buffer when I show every group, and I can
access it …

> I'll push a fix in a few minutes.

Thanks, I can easily give it a try (I only need to change the expiry
delay).

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

On 2015-01-29 09:51, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

>> Hm.  Is "nnimap+local:old.lists.formal.2014.01" a group that Gnus
>> possibly doesn't know about yet (i.e., it isn't subscribed yet when this
>> is done)?  If so, I think I understand why this happens...
>
> Well, it shows up in the Group buffer when I show every group, and I can
> access it …
>
>> I'll push a fix in a few minutes.
>
> Thanks, I can easily give it a try (I only need to change the expiry
> delay).

I just gave a try to the last push version, and I no longer can enter an
imap group:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  nnimap-request-group("lists.js" "zimbra" t nil)
  gnus-request-group("lists.js" t)
  gnus-select-newsgroup("lists.js" nil nil)
  gnus-summary-read-group-1("lists.js" nil t nil nil nil)
  gnus-summary-read-group("lists.js" nil t nil nil nil nil)

I edebugged it and the issue is the last "insert" command, where
`active' is nil, because it is never set, because `info' is nil (lines
807-821 of nnimap.el).

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Lars Ingebrigtsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> I just gave a try to the last push version, and I no longer can enter an
> imap group:
>
> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>   nnimap-request-group("lists.js" "zimbra" t nil)

Gah.

I've now reverted the previous patch and will take another stab at this
tomorrow, if time allows.  (I'll be on the train to Canberra...)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On 2015-01-29 22:08, Lars Ingebrigtsen <larsi@gnus.org> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> I just gave a try to the last push version, and I no longer can enter an
>> imap group:
>>
>> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>>   nnimap-request-group("lists.js" "zimbra" t nil)
>
> Gah.
>
> I've now reverted the previous patch and will take another stab at this
> tomorrow, if time allows.  (I'll be on the train to Canberra...)

Sure, no hurry. If it helps I can edebug this further.

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Lars Ingebrigtsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Sure, no hurry. If it helps I can edebug this further.

Sorry for the delay.  This should now be fixed in git Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 4607 bytes --]

On 2015-02-05 13:29, Lars Ingebrigtsen <larsi@gnus.org> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> Sure, no hurry. If it helps I can edebug this further.
>
> Sorry for the delay.  This should now be fixed in git Gnus.

I'm sorry, there is still a bug:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("^[^:]+:" nil)
  nnimap-retrieve-group-data-early("local" (nil))
  (let ((sequences (nnimap-retrieve-group-data-early server (list info)))) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 (assoc group nnimap-current-infos))))
  (progn (let ((sequences (nnimap-retrieve-group-data-early server (list info)))) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 (assoc group nnimap-current-infos)))))
  (if (or (not dont-check) (not (setq active (nth 2 (assoc group nnimap-current-infos))))) (progn (let ((sequences (nnimap-retrieve-group-data-early server (list info)))) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 (assoc group nnimap-current-infos))))))
  (progn (if (or (not dont-check) (not (setq active (nth 2 (assoc group nnimap-current-infos))))) (progn (let ((sequences (nnimap-retrieve-group-data-early server (list info)))) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 (assoc group nnimap-current-infos)))))) (erase-buffer) (insert (format "211 %d %d %d %S\n" (- (cdr active) (car active)) (car active) (cdr active) group)) t)
  (if result (progn (if (or (not dont-check) (not (setq active (nth 2 (assoc group nnimap-current-infos))))) (progn (let ((sequences (nnimap-retrieve-group-data-early server ...))) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 (assoc group nnimap-current-infos)))))) (erase-buffer) (insert (format "211 %d %d %d %S\n" (- (cdr active) (car active)) (car active) (cdr active) group)) t))
  (save-current-buffer (set-buffer nntp-server-buffer) (if result (progn (if (or (not dont-check) (not (setq active (nth 2 ...)))) (progn (let ((sequences ...)) (nnimap-finish-retrieve-group-infos server (list info) sequences t) (setq active (nth 2 ...))))) (erase-buffer) (insert (format "211 %d %d %d %S\n" (- (cdr active) (car active)) (car active) (cdr active) group)) t)))
  (let ((result (nnimap-change-group (if (and (not dont-check) (assoc group nnimap-current-infos)) nil group) server)) articles active marks high low) (save-current-buffer (set-buffer nntp-server-buffer) (if result (progn (if (or (not dont-check) (not (setq active ...))) (progn (let (...) (nnimap-finish-retrieve-group-infos server ... sequences t) (setq active ...)))) (erase-buffer) (insert (format "211 %d %d %d %S\n" (- (cdr active) (car active)) (car active) (cdr active) group)) t))))
  nnimap-request-group("old.lists.orgmode.2014.01" "local" nil nil)
  gnus-request-group("nnimap+local:old.lists.orgmode.2014.01")
  nnmail-expiry-target-group("nnimap+local:old.lists.orgmode.2014.01" "lists.orgmode")
  nnimap-process-expiry-targets((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode" "zimbra")
  nnimap-request-expire-articles((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode" "zimbra" nil)
  gnus-request-expire-articles((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode")
  gnus-summary-expire-articles()
  run-hooks(gnus-summary-prepare-exit-hook)
  apply(run-hooks gnus-summary-prepare-exit-hook)
  gnus-run-hooks(gnus-summary-prepare-exit-hook)
  gnus-summary-exit()
  call-interactively(gnus-summary-exit nil nil)
  command-execute(gnus-summary-exit)

It seems that `gnus-request-group' is called twice, the first one with
correct arguments (`dont-check' being t and `info' being nil) but the
second time with both `dont-check' and `info' at nil, which breaks
things.

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Lars Ingebrigtsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> I'm sorry, there is still a bug:
>
> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>   string-match("^[^:]+:" nil)
>   nnimap-retrieve-group-data-early("local" (nil))

This should be fixed in git Gnus now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 3894 bytes --]

On 2015-02-13 17:23, Lars Ingebrigtsen <larsi@gnus.org> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> I'm sorry, there is still a bug:
>>
>> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>>   string-match("^[^:]+:" nil)
>>   nnimap-retrieve-group-data-early("local" (nil))
>
> This should be fixed in git Gnus now.

Unfortunately it still breaks here. Here is what I see:

I quit the group with messages to expire, and it starts doing something,
as seen in the echo area:

Expiring article lists.orgmode:2068 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2069 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2070 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2071 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2072 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2073 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2074 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2075 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2076 to nnimap+local:old.lists.orgmode.2014.01
nnimap read 0k from localhost
Expiring article lists.orgmode:2077 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2078 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2079 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2080 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2081 to nnimap+local:old.lists.orgmode.2014.01
Expiring article lists.orgmode:2082 to nnimap+local:old.lists.orgmode.2014.01

It then starts going slower and slower doing the expiration:

Expiring article lists.orgmode:2248 to nnimap+local:old.lists.orgmode.2014.01
nnimap read 0k from localhost
Expiring article lists.orgmode:2249 to nnimap+local:old.lists.orgmode.2014.01
nnimap read 0k from localhost
Expiring article lists.orgmode:2250 to nnimap+local:old.lists.orgmode.2014.01
nnimap read 0k from localhost

and then after a while (when it's done?) there is an error:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("^[^:]+:" nil)
  nnimap-retrieve-group-data-early("local" (nil))
  nnimap-request-group("old.lists.orgmode.2014.02" "local" t nil)
  gnus-request-group("nnimap+local:old.lists.orgmode.2014.02" t)
  nnimap-process-expiry-targets((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode" "zimbra")
  nnimap-request-expire-articles((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode" "zimbra" nil)
  gnus-request-expire-articles((2030 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 ...) "lists.orgmode")
  gnus-summary-expire-articles()
  run-hooks(gnus-summary-prepare-exit-hook)
  apply(run-hooks gnus-summary-prepare-exit-hook)
  gnus-run-hooks(gnus-summary-prepare-exit-hook)
  gnus-summary-exit()
  call-interactively(gnus-summary-exit nil nil)
  command-execute(gnus-summary-exit)

If I try again it starts from the beginning, but it's already slow and
gets still slower, and fails with the same error.

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

On 2015-02-13 10:34, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> On 2015-02-13 17:23, Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>>
>>> I'm sorry, there is still a bug:
>>>
>>> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>>>   string-match("^[^:]+:" nil)
>>>   nnimap-retrieve-group-data-early("local" (nil))
>>
>> This should be fixed in git Gnus now.
>
> Unfortunately it still breaks here.

As an additional data point, expiring within a server works. Things
break when expiring from one imap server to the other.

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]

Hello,

On 2015-02-21 10:40, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> As an additional data point, expiring within a server works. Things
> break when expiring from one imap server to the other.

Should I report this as an emacs bug so that we keep track of it, or is
there another place to report bugs for the git version of gnus?

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 2485 bytes --]

On 2015-02-13 10:34, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> On 2015-02-13 17:23, Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>>
>>> I'm sorry, there is still a bug:
>>>
>>> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>>>   string-match("^[^:]+:" nil)
>>>   nnimap-retrieve-group-data-early("local" (nil))
>>
>> This should be fixed in git Gnus now.
>
> Unfortunately it still breaks here. Here is what I see:

I've dug a little in this, here are my findings. It seems the problems
only occur in the following setting:
- the target group is in a different server
- the target group is empty.

I'm able to reproduce this without the expiring process: if I move
a message (using "B m") to a group satisfying these conditions I get the
same error.

The problem happens in the last line of that snippet

#+begin_src emacs-lisp
(deffoo nnimap-request-group (group &optional server dont-check info)
  (setq group (nnimap-decode-gnus-group group))
  (let ((result (nnimap-change-group
      	   ;; Don't SELECT the group if we're going to select it
      	   ;; later, anyway.
      	   (if (and (not dont-check)
      		    (assoc group nnimap-current-infos))
      	       nil
      	     group)
      	   server))
        articles active marks high low)
    (with-current-buffer nntp-server-buffer
      (when result
        (when (or (not dont-check)
      	    (not (setq active
      		       (nth 2 (assoc group nnimap-current-infos)))))
          (let ((sequences (nnimap-retrieve-group-data-early
      		      server (list info))))
#+end_src

It happens if the function is called with info being nil, then
`nnimap-retrieve-group-data-early' is called with `(nil)' as second
argument (the singleton list containing the empty list).

Since `(nil)' is not nil, the guard of the `when' in the following
snippet does not prevent the code from being run

#+begin_src emacs-lisp
(deffoo nnimap-retrieve-group-data-early (server infos)
  (when (and (nnimap-change-group nil server)
             infos)
#+end_src

and the rest of the code breaks (when a `dolist' is done on the list
`(nil)').

Unfortunately I don't know what should be done in `nnimap-request-group'
if `info' is nil, so I cannot propose a patch.

I'll gladly take any suggestion as how to fix this.

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: latest imap changes break refiling

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

On 2015-01-29 09:51, Alan Schmitt <alan.schmitt@polytechnique.org> writes:

>> Hm.  Is "nnimap+local:old.lists.formal.2014.01" a group that Gnus
>> possibly doesn't know about yet (i.e., it isn't subscribed yet when this
>> is done)?  If so, I think I understand why this happens...
>
> Well, it shows up in the Group buffer when I show every group, and I can
> access it …
>
>> I'll push a fix in a few minutes.
>
> Thanks, I can easily give it a try (I only need to change the expiry
> delay).

I just gave a try to the last push version, and I no longer can enter an
imap group:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  nnimap-request-group("lists.js" "zimbra" t nil)
  gnus-request-group("lists.js" t)
  gnus-select-newsgroup("lists.js" nil nil)
  gnus-summary-read-group-1("lists.js" nil t nil nil nil)
  gnus-summary-read-group("lists.js" nil t nil nil nil nil)

I edebugged it and the issue is the last "insert" command, where
`active' is nil, because it is never set, because `info' is nil (lines
807-821 of nnimap.el).

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]