Gnus development mailing list
 help / color / mirror / Atom feed
* gnus-search-imap and quoted queries
@ 2022-04-19 16:10 Sergio Durigan Junior
  2022-04-19 21:30 ` dick
  2022-04-23  5:23 ` Sergio Durigan Junior
  0 siblings, 2 replies; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-04-19 16:10 UTC (permalink / raw)
  To: ding

Hi,

I have GNUS configured to use a local Dovecot IMAP server (a fairly
common setup, I think) and I've been trying to get gnus-search-imap
working.  I set gnus-search-use-parsed-queries to t, by the way.

I noticed that when I specify a quoted query as the search input, the
results don't reflect what I'd expect.  For example, I have emails from
the following authors:

  - Debian FTP Masters
  - Debian Bug Tracking System

If I type G G and write my query as:

  from:"Debian FTP Masters"

I see that the results include both emails from both authors.  The same
applies for other types of search (body, subject, etc.).  If I specify a
single term, things seem to work as expected.

I recently set up Full Text Search on Dovecot using the
dovecot-fts-xapian plugin.

Has anyone experienced this behaviour?

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 16:10 gnus-search-imap and quoted queries Sergio Durigan Junior
@ 2022-04-19 21:30 ` dick
  2022-04-19 21:37   ` Sergio Durigan Junior
  2022-04-23  5:23 ` Sergio Durigan Junior
  1 sibling, 1 reply; 13+ messages in thread
From: dick @ 2022-04-19 21:30 UTC (permalink / raw)
  To: Sergio Durigan Junior; +Cc: ding

C-u G G from "Debian FTP Masters"

The C-u says:
"I'm using Dovecot's query syntax, not Gnus's, so don't mess with it."

Note Dovecot's syntax does not admit a colon after "from".


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 21:30 ` dick
@ 2022-04-19 21:37   ` Sergio Durigan Junior
  2022-04-19 22:15     ` dick
  2022-04-19 22:47     ` Eric Abrahamsen
  0 siblings, 2 replies; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-04-19 21:37 UTC (permalink / raw)
  To: dick; +Cc: ding

On Tuesday, April 19 2022, dick wrote:

> C-u G G from "Debian FTP Masters"

Nothing changed.

> The C-u says:
> "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it."

I'm aware of what C-u does in this case, but IIUC the new gnus-search
backend should translate my query into Dovecot's parlance if
gnus-search-use-parsed-queries if t, which it is.  Or am I
misunderstanding something?

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 21:37   ` Sergio Durigan Junior
@ 2022-04-19 22:15     ` dick
  2022-04-20 18:49       ` Sergio Durigan Junior
  2022-04-19 22:47     ` Eric Abrahamsen
  1 sibling, 1 reply; 13+ messages in thread
From: dick @ 2022-04-19 22:15 UTC (permalink / raw)
  To: Sergio Durigan Junior; +Cc: ding

SDJ> I'm aware of what C-u does in this case, but IIUC the new
SDJ> gnus-search backend should translate my query into Dovecot's
SDJ> parlance

Transpilation, even for a simple grammar, is a lot to ask from an
obscure 25yo mail user agent.  Even if that had been done, it wouldn't
have been done well.

At this point I'd `C-u C-M-x` on

(cl-defmethod gnus-search-run-search ((engine gnus-search-imap)

in gnus-search.el to step through.  I know that's out-of-depth for many
gnusers, which always struck me as odd given the elisp chops implicitly
required by gnus's incomprehensibility.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 21:37   ` Sergio Durigan Junior
  2022-04-19 22:15     ` dick
@ 2022-04-19 22:47     ` Eric Abrahamsen
  2022-04-20 19:12       ` Sergio Durigan Junior
  1 sibling, 1 reply; 13+ messages in thread
From: Eric Abrahamsen @ 2022-04-19 22:47 UTC (permalink / raw)
  To: ding

Sergio Durigan Junior <sergiodj@sergiodj.net> writes:

> On Tuesday, April 19 2022, dick wrote:
>
>> C-u G G from "Debian FTP Masters"
>
> Nothing changed.
>
>> The C-u says:
>> "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it."
>
> I'm aware of what C-u does in this case, but IIUC the new gnus-search
> backend should translate my query into Dovecot's parlance if
> gnus-search-use-parsed-queries if t, which it is.  Or am I
> misunderstanding something?

It's possible that something has gone wrong with the handling of the
quotes. An easy way to tell what's Gnus is looking at is to run:

(let* ((q "from:\"Debian FTP Masters\"")
       (parsed (gnus-search-parse-query q))
       (engine (make-instance 'gnus-search-imap :literal-plus t)))
  (gnus-search-transform engine parsed))

"FROM {20+}
\"Debian FTP Masters\""

I set :literal-plus because I think that's Dovecot's config out of the
box. You can also replace the ad-hoc engine there with your actual
engine, by finding it in `gnus-search-engine-instance-alist'.

If the results look normal to you, the method to edebug is probably
`gnus-search-imap-search-command'.

Eric



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 22:15     ` dick
@ 2022-04-20 18:49       ` Sergio Durigan Junior
  2022-04-20 21:04         ` dick
  0 siblings, 1 reply; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-04-20 18:49 UTC (permalink / raw)
  To: dick; +Cc: ding

On Tuesday, April 19 2022, dick wrote:

> SDJ> I'm aware of what C-u does in this case, but IIUC the new
> SDJ> gnus-search backend should translate my query into Dovecot's
> SDJ> parlance
>
> Transpilation, even for a simple grammar, is a lot to ask from an
> obscure 25yo mail user agent.  Even if that had been done, it wouldn't
> have been done well.

gnus-search is new, but as far as I have seen it works surprisingly fine
for several backends, IMAP included.

> At this point I'd `C-u C-M-x` on
>
> (cl-defmethod gnus-search-run-search ((engine gnus-search-imap)

I did that.

The query I was using got translated correctly into:

  "FROM {20+}\n\"Debian FTP Masters\""

and then passed to Dovecot.  I tried invoking "doveadm search" directly
on the command line passing the query GNUS is using and noticed that the
results are the same, i.e., it didn't do an exact match on "Debian FTP
Masters" and ended up returning emails from "Debian Bug Tracking System"
as well.

At this point, I think it's safe to say that this is an
issue/idiosyncrasy of Dovecot or IMAP itself, and not GNUS.  Apparently
it's not possible to do a search for exact terms using quotes.

I will keep investigating, but I wonder if someone else has experienced
this as well.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 22:47     ` Eric Abrahamsen
@ 2022-04-20 19:12       ` Sergio Durigan Junior
  0 siblings, 0 replies; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-04-20 19:12 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Tuesday, April 19 2022, Eric Abrahamsen wrote:

> Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
>
>> On Tuesday, April 19 2022, dick wrote:
>>
>>> C-u G G from "Debian FTP Masters"
>>
>> Nothing changed.
>>
>>> The C-u says:
>>> "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it."
>>
>> I'm aware of what C-u does in this case, but IIUC the new gnus-search
>> backend should translate my query into Dovecot's parlance if
>> gnus-search-use-parsed-queries if t, which it is.  Or am I
>> misunderstanding something?
>
> It's possible that something has gone wrong with the handling of the
> quotes. An easy way to tell what's Gnus is looking at is to run:
>
> (let* ((q "from:\"Debian FTP Masters\"")
>        (parsed (gnus-search-parse-query q))
>        (engine (make-instance 'gnus-search-imap :literal-plus t)))
>   (gnus-search-transform engine parsed))
>
> "FROM {20+}
> \"Debian FTP Masters\""
>
> I set :literal-plus because I think that's Dovecot's config out of the
> box. You can also replace the ad-hoc engine there with your actual
> engine, by finding it in `gnus-search-engine-instance-alist'.
>
> If the results look normal to you, the method to edebug is probably
> `gnus-search-imap-search-command'.

Thanks, Eric.

As I wrote in my reply to dick, it seems to me that the query is
correct.  At least I was able to execute it via "doveadm search" and
produce the same results as I see on GNUS.  I'm thinking that the
problem is with how Dovecot interprets the query (and how it doesn't
seem to be searching for the exact terms, even though they are quoted).

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-20 18:49       ` Sergio Durigan Junior
@ 2022-04-20 21:04         ` dick
  0 siblings, 0 replies; 13+ messages in thread
From: dick @ 2022-04-20 21:04 UTC (permalink / raw)
  To: Sergio Durigan Junior; +Cc: ding

Out of more than just curiosity, how does one "doveadm" a search query
with the literal+ annotation?

doveadm search -u <yourself> from {20+}^M"Debian FTP Masters"

Note the tricky CRLF between {20+} and "Debian".


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-19 16:10 gnus-search-imap and quoted queries Sergio Durigan Junior
  2022-04-19 21:30 ` dick
@ 2022-04-23  5:23 ` Sergio Durigan Junior
  2022-04-23 16:01   ` Eric Abrahamsen
  1 sibling, 1 reply; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-04-23  5:23 UTC (permalink / raw)
  To: ding

On Tuesday, April 19 2022, I wrote:

> I have GNUS configured to use a local Dovecot IMAP server (a fairly
> common setup, I think) and I've been trying to get gnus-search-imap
> working.  I set gnus-search-use-parsed-queries to t, by the way.
>
> I noticed that when I specify a quoted query as the search input, the
> results don't reflect what I'd expect.

FWIW, I finally had some time to look deeper into this and found that
the culprit was the Dovecot FTS plugin I was using (fts-xapian).
It doesn't follow the IMAP RFC strictly and therefore can't/won't do
substring searches.

I'm now using another Xapian-based plugin called fts-flatcurve which
DTRT.

Thanks for the help, it's been fun investigating this problem.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-23  5:23 ` Sergio Durigan Junior
@ 2022-04-23 16:01   ` Eric Abrahamsen
  2022-05-01  5:14     ` Sergio Durigan Junior
  0 siblings, 1 reply; 13+ messages in thread
From: Eric Abrahamsen @ 2022-04-23 16:01 UTC (permalink / raw)
  To: ding

Sergio Durigan Junior <sergiodj@sergiodj.net> writes:

> On Tuesday, April 19 2022, I wrote:
>
>> I have GNUS configured to use a local Dovecot IMAP server (a fairly
>> common setup, I think) and I've been trying to get gnus-search-imap
>> working.  I set gnus-search-use-parsed-queries to t, by the way.
>>
>> I noticed that when I specify a quoted query as the search input, the
>> results don't reflect what I'd expect.
>
> FWIW, I finally had some time to look deeper into this and found that
> the culprit was the Dovecot FTS plugin I was using (fts-xapian).
> It doesn't follow the IMAP RFC strictly and therefore can't/won't do
> substring searches.
>
> I'm now using another Xapian-based plugin called fts-flatcurve which
> DTRT.
>
> Thanks for the help, it's been fun investigating this problem.

Hey that's pretty interesting. Looks like I'm using lucene, which seems
to work fine. I wonder if it merits a brief mention in the manual that
different backends behave differently.



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-04-23 16:01   ` Eric Abrahamsen
@ 2022-05-01  5:14     ` Sergio Durigan Junior
  2022-05-03 20:01       ` Eric Abrahamsen
  0 siblings, 1 reply; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-05-01  5:14 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Saturday, April 23 2022, Eric Abrahamsen wrote:

> Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
>
>> On Tuesday, April 19 2022, I wrote:
>>
>>> I have GNUS configured to use a local Dovecot IMAP server (a fairly
>>> common setup, I think) and I've been trying to get gnus-search-imap
>>> working.  I set gnus-search-use-parsed-queries to t, by the way.
>>>
>>> I noticed that when I specify a quoted query as the search input, the
>>> results don't reflect what I'd expect.
>>
>> FWIW, I finally had some time to look deeper into this and found that
>> the culprit was the Dovecot FTS plugin I was using (fts-xapian).
>> It doesn't follow the IMAP RFC strictly and therefore can't/won't do
>> substring searches.
>>
>> I'm now using another Xapian-based plugin called fts-flatcurve which
>> DTRT.
>>
>> Thanks for the help, it's been fun investigating this problem.
>
> Hey that's pretty interesting. Looks like I'm using lucene, which seems
> to work fine. I wonder if it merits a brief mention in the manual that
> different backends behave differently.

I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in
the manual.  WDYT about the patch below?

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/

From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001
From: Sergio Durigan Junior <sergiodj@sergiodj.net>
Date: Sun, 1 May 2022 01:11:42 -0400
Subject: [PATCH] Mention different behaviours in IMAP FTS plugins

* doc/misc/gnus.texi (Search Engines): Mention different behaviours in
IMAP FTS plugins.
---
 doc/misc/gnus.texi | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi
index e51ae7d424d..03330224e09 100644
--- a/doc/misc/gnus.texi
+++ b/doc/misc/gnus.texi
@@ -21727,6 +21727,13 @@ Search Engines
 to manage updates yourself, though this will likely change in the
 future.
 
+If you use the IMAP search engine you may want to configure your IMAP
+server to provide Full Text Search (FTS) functionality, which greatly
+improves the performance and results.  Beware though that some FTS
+backends may not implement text search as described by RFC 3501, which
+states that substring search should be performed.  This difference in
+behaviour can ultimately affect the search results.
+
 Lastly, all search engines accept a @code{raw-queries-p} option.  This
 indicates that engines of this type (or this particular engine) should
 always use raw queries, never parsed (@pxref{Search Queries}).
-- 
2.35.1



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-05-01  5:14     ` Sergio Durigan Junior
@ 2022-05-03 20:01       ` Eric Abrahamsen
  2022-05-05  2:06         ` Sergio Durigan Junior
  0 siblings, 1 reply; 13+ messages in thread
From: Eric Abrahamsen @ 2022-05-03 20:01 UTC (permalink / raw)
  To: ding

Sergio Durigan Junior <sergiodj@sergiodj.net> writes:

> On Saturday, April 23 2022, Eric Abrahamsen wrote:
>
>> Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
>>
>>> On Tuesday, April 19 2022, I wrote:
>>>
>>>> I have GNUS configured to use a local Dovecot IMAP server (a fairly
>>>> common setup, I think) and I've been trying to get gnus-search-imap
>>>> working.  I set gnus-search-use-parsed-queries to t, by the way.
>>>>
>>>> I noticed that when I specify a quoted query as the search input, the
>>>> results don't reflect what I'd expect.
>>>
>>> FWIW, I finally had some time to look deeper into this and found that
>>> the culprit was the Dovecot FTS plugin I was using (fts-xapian).
>>> It doesn't follow the IMAP RFC strictly and therefore can't/won't do
>>> substring searches.
>>>
>>> I'm now using another Xapian-based plugin called fts-flatcurve which
>>> DTRT.
>>>
>>> Thanks for the help, it's been fun investigating this problem.
>>
>> Hey that's pretty interesting. Looks like I'm using lucene, which seems
>> to work fine. I wonder if it merits a brief mention in the manual that
>> different backends behave differently.
>
> I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in
> the manual.  WDYT about the patch below?
>
> -- 
> Sergio
> GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
> Please send encrypted e-mail if possible
> https://sergiodj.net/
>
> From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001
> From: Sergio Durigan Junior <sergiodj@sergiodj.net>
> Date: Sun, 1 May 2022 01:11:42 -0400
> Subject: [PATCH] Mention different behaviours in IMAP FTS plugins
>
> * doc/misc/gnus.texi (Search Engines): Mention different behaviours in
> IMAP FTS plugins.
> ---
>  doc/misc/gnus.texi | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi
> index e51ae7d424d..03330224e09 100644
> --- a/doc/misc/gnus.texi
> +++ b/doc/misc/gnus.texi
> @@ -21727,6 +21727,13 @@ Search Engines
>  to manage updates yourself, though this will likely change in the
>  future.
>  
> +If you use the IMAP search engine you may want to configure your IMAP
> +server to provide Full Text Search (FTS) functionality, which greatly
> +improves the performance and results.  Beware though that some FTS
> +backends may not implement text search as described by RFC 3501, which
> +states that substring search should be performed.  This difference in
> +behaviour can ultimately affect the search results.
> +
>  Lastly, all search engines accept a @code{raw-queries-p} option.  This
>  indicates that engines of this type (or this particular engine) should
>  always use raw queries, never parsed (@pxref{Search Queries}).

Looks good to me! Thanks.



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: gnus-search-imap and quoted queries
  2022-05-03 20:01       ` Eric Abrahamsen
@ 2022-05-05  2:06         ` Sergio Durigan Junior
  0 siblings, 0 replies; 13+ messages in thread
From: Sergio Durigan Junior @ 2022-05-05  2:06 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Tuesday, May 03 2022, Eric Abrahamsen wrote:

> Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
>> I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in
>> the manual.  WDYT about the patch below?
>>
>> -- 
>> Sergio
>> GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
>> Please send encrypted e-mail if possible
>> https://sergiodj.net/
>>
>> From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001
>> From: Sergio Durigan Junior <sergiodj@sergiodj.net>
>> Date: Sun, 1 May 2022 01:11:42 -0400
>> Subject: [PATCH] Mention different behaviours in IMAP FTS plugins
>>
>> * doc/misc/gnus.texi (Search Engines): Mention different behaviours in
>> IMAP FTS plugins.
>> ---
>>  doc/misc/gnus.texi | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi
>> index e51ae7d424d..03330224e09 100644
>> --- a/doc/misc/gnus.texi
>> +++ b/doc/misc/gnus.texi
>> @@ -21727,6 +21727,13 @@ Search Engines
>>  to manage updates yourself, though this will likely change in the
>>  future.
>>  
>> +If you use the IMAP search engine you may want to configure your IMAP
>> +server to provide Full Text Search (FTS) functionality, which greatly
>> +improves the performance and results.  Beware though that some FTS
>> +backends may not implement text search as described by RFC 3501, which
>> +states that substring search should be performed.  This difference in
>> +behaviour can ultimately affect the search results.
>> +
>>  Lastly, all search engines accept a @code{raw-queries-p} option.  This
>>  indicates that engines of this type (or this particular engine) should
>>  always use raw queries, never parsed (@pxref{Search Queries}).
>
> Looks good to me! Thanks.

Thanks!

I'll send the patch to emacs-devel soon.

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2022-05-05  2:07 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-19 16:10 gnus-search-imap and quoted queries Sergio Durigan Junior
2022-04-19 21:30 ` dick
2022-04-19 21:37   ` Sergio Durigan Junior
2022-04-19 22:15     ` dick
2022-04-20 18:49       ` Sergio Durigan Junior
2022-04-20 21:04         ` dick
2022-04-19 22:47     ` Eric Abrahamsen
2022-04-20 19:12       ` Sergio Durigan Junior
2022-04-23  5:23 ` Sergio Durigan Junior
2022-04-23 16:01   ` Eric Abrahamsen
2022-05-01  5:14     ` Sergio Durigan Junior
2022-05-03 20:01       ` Eric Abrahamsen
2022-05-05  2:06         ` Sergio Durigan Junior

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).