Re: gnus-search since: not working for nnimap

Re: gnus-search since: not working for nnimap

[Eric Abrahamsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> You can play with the bits and pieces to see where things might be going
>> wrong:
>>
>> (gnus-search-query-parse-date "1y") -> (28 1 2022)
>> (gnus-search-parse-query "since:1y") - > ((since 28 1 2022))
>>
>> So far so good.
>>
>> (gnus-search-transform
>>  (make-instance 'gnus-search-imap)
>>  (gnus-search-parse-query "since:1y")) -> "SINCE 28-Jan-2022"
>>
>> Still looking correct (right?).
>
> Everything is fine up until this point.
>
>> Try doing "C-u G G" on an nnimap group to shut off gnus-search parsing,
>> and enter "SINCE 28-Jan-2022" as the search string. Does that return
>> anything?
>
> Sill nothing:
>
> Group nnselect:nnselect-87cz6x1zv6.fsf contains no messages

We'll have to get closer to the IMAP server, then. If you don't have TLS
enabled on your local dovecot you can probably log in and test with:

telnet localhost 143

Then:

a login <username> <password>
b select "INBOX"
c UID SEARCH SINCE 28-Jan-2022

To confirm that messages are found.

Then it's a matter of seeing that same conversation between Gnus and
Dovecot, to find where things are going wrong. You can set
`nnimap-record-commands' to t, then conduct a search, and also find
which of your " *nnimap localhost nil *nntpd**-373470" buffers
corresponds to this server. That should allow you to reconstruct the
conversation.

Another option would be to edebug the `gnus-search-imap-search-command'
function, and see what we're getting back from Dovecot.

Hopefully something in there will be revealing.

Re: gnus-search since: not working for nnimap

[Julien Cubizolles]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> We'll have to get closer to the IMAP server, then. If you don't have TLS
> enabled on your local dovecot you can probably log in and test with:
>
> telnet localhost 143
>
> Then:
>
> a login <username> <password>
> b select "INBOX"
> c UID SEARCH SINCE 28-Jan-2022
>
> To confirm that messages are found.


--8<---------------cut here---------------start------------->8---
/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/email/Maildir/Free:LAYOUT=fs
then
a SELECT "General"
b UID SEARCH SINCE 28-Jan-2022
--8<---------------cut here---------------end--------------->8---

It found many messages:

--8<---------------cut here---------------start------------->8---
b UID SEARCH SINCE 28-Jan-2022
* SEARCH 2870 2874 2977 2987 2988 2997 3069 3312 3347 3390 3455 3645 3651 3656 3677 3691 3707 3729 3730 3748 3749 3750 3767 3871 3954 3985 4038 4041 4045 4046 4048 4053 4054 4056 4058 4060 4065 4073 4074 4082 4085 4094 4108 4125 4134 4286 4294 4324 4326 4335 4337 4342 4371 4413 4414 4433 4455 4458 4473 4483 4521 4522 4560 4562 4593 4603 4611 4615 4617 4630 4633 4636 4638 4639 4651 4676 4678 4679 4681 4682 4685 4688 4691 4694 4695 4696 4709 4712 4713 4715 4716 4717 4718 4723 4737 4747 4765 4766 4772 4782 4790 4793 4806 4808 4810 4813 4814 4815 4818 4820 4821 4822 4826 4827 4828 4829 4830 4831 4832 4833 4834 4835 4836 4838 4839 4840 4841 4842 4843 4844 4846 4849 4850
b OK Search completed (0.002 + 0.000 + 0.001 secs)
--8<---------------cut here---------------end--------------->8---


> Then it's a matter of seeing that same conversation between Gnus and
> Dovecot, to find where things are going wrong. You can set
> `nnimap-record-commands' to t, then conduct a search, 

I found a "*imap log*" buffer. When conducting the "since:1y" search,
the only log is:
--8<---------------cut here---------------start------------->8---
07:56:57 [FreeOffline] 10993 SELECT "General"
07:56:57 [FreeOffline] 10994 UID SEARCH SINCE 30-janv.-2022
--8<---------------cut here---------------end--------------->8---


> and also find which of your " *nnimap localhost nil *nntpd**-373470"
> buffers corresponds to this server. 

there isn't any buffer with this name, only the *imap log*

> That should allow you to reconstruct the conversation.
>
> Another option would be to edebug the `gnus-search-imap-search-command'
> function, and see what we're getting back from Dovecot.
>
> Hopefully something in there will be revealing.

Indeed: the nnimap-command used is:

--8<---------------cut here---------------start------------->8---
nnimap-send-command("UID SEARCH %s" "SINCE 30-janv.-2022")
--8<---------------cut here---------------end--------------->8---

which seems to be invalid:
+ because of the french localized version of the date (janv instead of jan)
+ because of the dot

--8<---------------cut here---------------start------------->8---
UID SEARCH SINCE 28-Janv-2022
UID SEARCH SINCE 28-Jan.-2022
--8<---------------cut here---------------end--------------->8---

both return:
--8<---------------cut here---------------start------------->8---
UID BAD Error in IMAP command SEARCH: Invalid search date parameter (0.001 + 0.000 secs).
--8<---------------cut here---------------end--------------->8---

in a direct session with dovecot.


-- 
Julien Cubizolles

Re: gnus-search since: not working for nnimap

[Julien Cubizolles]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> We'll have to get closer to the IMAP server, then.

Actually we don't. I didn't pay enough attention to the results of the
previous test you suggested:

--8<---------------cut here---------------start------------->8---
(gnus-search-transform
 (make-instance 'gnus-search-imap)
 (gnus-search-parse-query "since:1y")) -> "SINCE 28-Jan-2022"
--8<---------------cut here---------------end--------------->8---

returns
--8<---------------cut here---------------start------------->8---
"SINCE 30-janv.-2022"
--8<---------------cut here---------------end--------------->8---

but it should be
--8<---------------cut here---------------start------------->8---
"SINCE 30-jan-2022"
--8<---------------cut here---------------end--------------->8---

'gnus-search-transform seems to be at fault here. It uses
'(format-time-string "%e-%b-%Y") where %b returns the "locale’s
abbreviated month name" when imap commands expects the english abbreviated
month name.

-- 
Julien Cubizolles

Re: gnus-search since: not working for nnimap

[Julien Cubizolles]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> 'gnus-search-transform seems to be at fault here. It uses
> '(format-time-string "%e-%b-%Y") where %b returns the "locale’s
> abbreviated month name" when imap commands expects the english abbreviated
> month name.

It's working with the attached patch. I'm not sure this is the best way
to go but I'm happy with it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-search.patch --]
[-- Type: text/x-diff, Size: 587 bytes --]

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index 27c71fa6c6..0d9a62f911 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1330,9 +1330,10 @@ elements are present."
 		      (1- nyear)
 		    nyear))
 	  (setq dmonth 1))))
-    (format-time-string
-     "%e-%b-%Y"
-     (encode-time 0 0 0 dday dmonth dyear))))
+    (with-locale-environment "C"
+      (format-time-string
+       "%e-%b-%Y"
+       (encode-time 0 0 0 dday dmonth dyear)))))
 
 (cl-defmethod gnus-search-imap-handle-string ((engine gnus-search-imap)
 					      (str string))

[-- Attachment #3: Type: text/plain, Size: 24 bytes --]


-- 
Julien Cubizolles

Re: gnus-search since: not working for nnimap

[Eric Abrahamsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> 'gnus-search-transform seems to be at fault here. It uses
>> '(format-time-string "%e-%b-%Y") where %b returns the "locale’s
>> abbreviated month name" when imap commands expects the english abbreviated
>> month name.
>
> It's working with the attached patch. I'm not sure this is the best way
> to go but I'm happy with it.
>
> diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
> index 27c71fa6c6..0d9a62f911 100644
> --- a/lisp/gnus/gnus-search.el
> +++ b/lisp/gnus/gnus-search.el
> @@ -1330,9 +1330,10 @@ elements are present."
>  		      (1- nyear)
>  		    nyear))
>  	  (setq dmonth 1))))
> -    (format-time-string
> -     "%e-%b-%Y"
> -     (encode-time 0 0 0 dday dmonth dyear))))
> +    (with-locale-environment "C"
> +      (format-time-string
> +       "%e-%b-%Y"
> +       (encode-time 0 0 0 dday dmonth dyear)))))
>  
>  (cl-defmethod gnus-search-imap-handle-string ((engine gnus-search-imap)
>  					      (str string))

Ooh, that's a great catch, thank you very much. I'll look around and see
if there are any other similar issues in this library, and push a fix to
master soon.

Thanks for the report,
Eric

Re: gnus-search since: not working for nnimap

[Björn Bidar]

I suggest to use locale C but C.UTF-8, C is to limited.

Re: gnus-search since: not working for nnimap

[Julien Cubizolles]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> You can play with the bits and pieces to see where things might be going
> wrong:
>
> (gnus-search-query-parse-date "1y") -> (28 1 2022)
> (gnus-search-parse-query "since:1y") - > ((since 28 1 2022))
>
> So far so good.
>
> (gnus-search-transform
>  (make-instance 'gnus-search-imap)
>  (gnus-search-parse-query "since:1y")) -> "SINCE 28-Jan-2022"
>
> Still looking correct (right?).

Everything is fine up until this point.

> Try doing "C-u G G" on an nnimap group to shut off gnus-search parsing,
> and enter "SINCE 28-Jan-2022" as the search string. Does that return
> anything?

Sill nothing:

--8<---------------cut here---------------start------------->8---
Group nnselect:nnselect-87cz6x1zv6.fsf contains no messages
--8<---------------cut here---------------end--------------->8---




-- 
Julien Cubizolles

Re: gnus-search since: not working for nnimap

[Eric Abrahamsen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> I've gnus-search configured to use gnus-search-imap for a nnimap
> method. G G (gnus-group-read-ephemeral-search-group) doesn't return any
> result when using a query like "since:1y" on a nnimap group whereas:
>
> + the same query works in a nnml group
> + a query like "from:user" returns many results in this time range on
>   the same nnimap group.
>
> Isn't "since:1y" a valid query for nnimap ?

It should be!

> The nnimap group is managed by a local dovecot server. Does it
> matter ?

No, I don't think so. I tried it on one of my own nnimap servers, backed
by local dovecot, and it returned several thousand messages.

You can play with the bits and pieces to see where things might be going
wrong:

(gnus-search-query-parse-date "1y") -> (28 1 2022)
(gnus-search-parse-query "since:1y") - > ((since 28 1 2022))

So far so good.

(gnus-search-transform
 (make-instance 'gnus-search-imap)
 (gnus-search-parse-query "since:1y")) -> "SINCE 28-Jan-2022"

Still looking correct (right?).

Try doing "C-u G G" on an nnimap group to shut off gnus-search parsing,
and enter "SINCE 28-Jan-2022" as the search string. Does that return
anything?

Eric

gnus-search since: not working for nnimap

[Julien Cubizolles]

I've gnus-search configured to use gnus-search-imap for a nnimap
method. G G (gnus-group-read-ephemeral-search-group) doesn't return any
result when using a query like "since:1y" on a nnimap group whereas:

+ the same query works in a nnml group
+ a query like "from:user" returns many results in this time range on
  the same nnimap group.

Isn't "since:1y" a valid query for nnimap ?

The nnimap group is managed by a local dovecot server. Does it matter ? 


-- 
Julien Cubizolles