Hi, I have GNUS configured to use a local Dovecot IMAP server (a fairly common setup, I think) and I've been trying to get gnus-search-imap working. I set gnus-search-use-parsed-queries to t, by the way. I noticed that when I specify a quoted query as the search input, the results don't reflect what I'd expect. For example, I have emails from the following authors: - Debian FTP Masters - Debian Bug Tracking System If I type G G and write my query as: from:"Debian FTP Masters" I see that the results include both emails from both authors. The same applies for other types of search (body, subject, etc.). If I specify a single term, things seem to work as expected. I recently set up Full Text Search on Dovecot using the dovecot-fts-xapian plugin. Has anyone experienced this behaviour? Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/
C-u G G from "Debian FTP Masters" The C-u says: "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it." Note Dovecot's syntax does not admit a colon after "from".
On Tuesday, April 19 2022, dick wrote: > C-u G G from "Debian FTP Masters" Nothing changed. > The C-u says: > "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it." I'm aware of what C-u does in this case, but IIUC the new gnus-search backend should translate my query into Dovecot's parlance if gnus-search-use-parsed-queries if t, which it is. Or am I misunderstanding something? Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/
SDJ> I'm aware of what C-u does in this case, but IIUC the new SDJ> gnus-search backend should translate my query into Dovecot's SDJ> parlance Transpilation, even for a simple grammar, is a lot to ask from an obscure 25yo mail user agent. Even if that had been done, it wouldn't have been done well. At this point I'd `C-u C-M-x` on (cl-defmethod gnus-search-run-search ((engine gnus-search-imap) in gnus-search.el to step through. I know that's out-of-depth for many gnusers, which always struck me as odd given the elisp chops implicitly required by gnus's incomprehensibility.
Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
> On Tuesday, April 19 2022, dick wrote:
>
>> C-u G G from "Debian FTP Masters"
>
> Nothing changed.
>
>> The C-u says:
>> "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it."
>
> I'm aware of what C-u does in this case, but IIUC the new gnus-search
> backend should translate my query into Dovecot's parlance if
> gnus-search-use-parsed-queries if t, which it is. Or am I
> misunderstanding something?
It's possible that something has gone wrong with the handling of the
quotes. An easy way to tell what's Gnus is looking at is to run:
(let* ((q "from:\"Debian FTP Masters\"")
(parsed (gnus-search-parse-query q))
(engine (make-instance 'gnus-search-imap :literal-plus t)))
(gnus-search-transform engine parsed))
"FROM {20+}
\"Debian FTP Masters\""
I set :literal-plus because I think that's Dovecot's config out of the
box. You can also replace the ad-hoc engine there with your actual
engine, by finding it in `gnus-search-engine-instance-alist'.
If the results look normal to you, the method to edebug is probably
`gnus-search-imap-search-command'.
Eric
On Tuesday, April 19 2022, dick wrote: > SDJ> I'm aware of what C-u does in this case, but IIUC the new > SDJ> gnus-search backend should translate my query into Dovecot's > SDJ> parlance > > Transpilation, even for a simple grammar, is a lot to ask from an > obscure 25yo mail user agent. Even if that had been done, it wouldn't > have been done well. gnus-search is new, but as far as I have seen it works surprisingly fine for several backends, IMAP included. > At this point I'd `C-u C-M-x` on > > (cl-defmethod gnus-search-run-search ((engine gnus-search-imap) I did that. The query I was using got translated correctly into: "FROM {20+}\n\"Debian FTP Masters\"" and then passed to Dovecot. I tried invoking "doveadm search" directly on the command line passing the query GNUS is using and noticed that the results are the same, i.e., it didn't do an exact match on "Debian FTP Masters" and ended up returning emails from "Debian Bug Tracking System" as well. At this point, I think it's safe to say that this is an issue/idiosyncrasy of Dovecot or IMAP itself, and not GNUS. Apparently it's not possible to do a search for exact terms using quotes. I will keep investigating, but I wonder if someone else has experienced this as well. Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/
On Tuesday, April 19 2022, Eric Abrahamsen wrote: > Sergio Durigan Junior <sergiodj@sergiodj.net> writes: > >> On Tuesday, April 19 2022, dick wrote: >> >>> C-u G G from "Debian FTP Masters" >> >> Nothing changed. >> >>> The C-u says: >>> "I'm using Dovecot's query syntax, not Gnus's, so don't mess with it." >> >> I'm aware of what C-u does in this case, but IIUC the new gnus-search >> backend should translate my query into Dovecot's parlance if >> gnus-search-use-parsed-queries if t, which it is. Or am I >> misunderstanding something? > > It's possible that something has gone wrong with the handling of the > quotes. An easy way to tell what's Gnus is looking at is to run: > > (let* ((q "from:\"Debian FTP Masters\"") > (parsed (gnus-search-parse-query q)) > (engine (make-instance 'gnus-search-imap :literal-plus t))) > (gnus-search-transform engine parsed)) > > "FROM {20+} > \"Debian FTP Masters\"" > > I set :literal-plus because I think that's Dovecot's config out of the > box. You can also replace the ad-hoc engine there with your actual > engine, by finding it in `gnus-search-engine-instance-alist'. > > If the results look normal to you, the method to edebug is probably > `gnus-search-imap-search-command'. Thanks, Eric. As I wrote in my reply to dick, it seems to me that the query is correct. At least I was able to execute it via "doveadm search" and produce the same results as I see on GNUS. I'm thinking that the problem is with how Dovecot interprets the query (and how it doesn't seem to be searching for the exact terms, even though they are quoted). -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/
Out of more than just curiosity, how does one "doveadm" a search query with the literal+ annotation? doveadm search -u <yourself> from {20+}^M"Debian FTP Masters" Note the tricky CRLF between {20+} and "Debian".
On Tuesday, April 19 2022, I wrote: > I have GNUS configured to use a local Dovecot IMAP server (a fairly > common setup, I think) and I've been trying to get gnus-search-imap > working. I set gnus-search-use-parsed-queries to t, by the way. > > I noticed that when I specify a quoted query as the search input, the > results don't reflect what I'd expect. FWIW, I finally had some time to look deeper into this and found that the culprit was the Dovecot FTS plugin I was using (fts-xapian). It doesn't follow the IMAP RFC strictly and therefore can't/won't do substring searches. I'm now using another Xapian-based plugin called fts-flatcurve which DTRT. Thanks for the help, it's been fun investigating this problem. -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/
Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
> On Tuesday, April 19 2022, I wrote:
>
>> I have GNUS configured to use a local Dovecot IMAP server (a fairly
>> common setup, I think) and I've been trying to get gnus-search-imap
>> working. I set gnus-search-use-parsed-queries to t, by the way.
>>
>> I noticed that when I specify a quoted query as the search input, the
>> results don't reflect what I'd expect.
>
> FWIW, I finally had some time to look deeper into this and found that
> the culprit was the Dovecot FTS plugin I was using (fts-xapian).
> It doesn't follow the IMAP RFC strictly and therefore can't/won't do
> substring searches.
>
> I'm now using another Xapian-based plugin called fts-flatcurve which
> DTRT.
>
> Thanks for the help, it's been fun investigating this problem.
Hey that's pretty interesting. Looks like I'm using lucene, which seems
to work fine. I wonder if it merits a brief mention in the manual that
different backends behave differently.
On Saturday, April 23 2022, Eric Abrahamsen wrote: > Sergio Durigan Junior <sergiodj@sergiodj.net> writes: > >> On Tuesday, April 19 2022, I wrote: >> >>> I have GNUS configured to use a local Dovecot IMAP server (a fairly >>> common setup, I think) and I've been trying to get gnus-search-imap >>> working. I set gnus-search-use-parsed-queries to t, by the way. >>> >>> I noticed that when I specify a quoted query as the search input, the >>> results don't reflect what I'd expect. >> >> FWIW, I finally had some time to look deeper into this and found that >> the culprit was the Dovecot FTS plugin I was using (fts-xapian). >> It doesn't follow the IMAP RFC strictly and therefore can't/won't do >> substring searches. >> >> I'm now using another Xapian-based plugin called fts-flatcurve which >> DTRT. >> >> Thanks for the help, it's been fun investigating this problem. > > Hey that's pretty interesting. Looks like I'm using lucene, which seems > to work fine. I wonder if it merits a brief mention in the manual that > different backends behave differently. I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in the manual. WDYT about the patch below? -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/ From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001 From: Sergio Durigan Junior <sergiodj@sergiodj.net> Date: Sun, 1 May 2022 01:11:42 -0400 Subject: [PATCH] Mention different behaviours in IMAP FTS plugins * doc/misc/gnus.texi (Search Engines): Mention different behaviours in IMAP FTS plugins. --- doc/misc/gnus.texi | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi index e51ae7d424d..03330224e09 100644 --- a/doc/misc/gnus.texi +++ b/doc/misc/gnus.texi @@ -21727,6 +21727,13 @@ Search Engines to manage updates yourself, though this will likely change in the future. +If you use the IMAP search engine you may want to configure your IMAP +server to provide Full Text Search (FTS) functionality, which greatly +improves the performance and results. Beware though that some FTS +backends may not implement text search as described by RFC 3501, which +states that substring search should be performed. This difference in +behaviour can ultimately affect the search results. + Lastly, all search engines accept a @code{raw-queries-p} option. This indicates that engines of this type (or this particular engine) should always use raw queries, never parsed (@pxref{Search Queries}). -- 2.35.1
Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
> On Saturday, April 23 2022, Eric Abrahamsen wrote:
>
>> Sergio Durigan Junior <sergiodj@sergiodj.net> writes:
>>
>>> On Tuesday, April 19 2022, I wrote:
>>>
>>>> I have GNUS configured to use a local Dovecot IMAP server (a fairly
>>>> common setup, I think) and I've been trying to get gnus-search-imap
>>>> working. I set gnus-search-use-parsed-queries to t, by the way.
>>>>
>>>> I noticed that when I specify a quoted query as the search input, the
>>>> results don't reflect what I'd expect.
>>>
>>> FWIW, I finally had some time to look deeper into this and found that
>>> the culprit was the Dovecot FTS plugin I was using (fts-xapian).
>>> It doesn't follow the IMAP RFC strictly and therefore can't/won't do
>>> substring searches.
>>>
>>> I'm now using another Xapian-based plugin called fts-flatcurve which
>>> DTRT.
>>>
>>> Thanks for the help, it's been fun investigating this problem.
>>
>> Hey that's pretty interesting. Looks like I'm using lucene, which seems
>> to work fine. I wonder if it merits a brief mention in the manual that
>> different backends behave differently.
>
> I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in
> the manual. WDYT about the patch below?
>
> --
> Sergio
> GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36
> Please send encrypted e-mail if possible
> https://sergiodj.net/
>
> From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001
> From: Sergio Durigan Junior <sergiodj@sergiodj.net>
> Date: Sun, 1 May 2022 01:11:42 -0400
> Subject: [PATCH] Mention different behaviours in IMAP FTS plugins
>
> * doc/misc/gnus.texi (Search Engines): Mention different behaviours in
> IMAP FTS plugins.
> ---
> doc/misc/gnus.texi | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi
> index e51ae7d424d..03330224e09 100644
> --- a/doc/misc/gnus.texi
> +++ b/doc/misc/gnus.texi
> @@ -21727,6 +21727,13 @@ Search Engines
> to manage updates yourself, though this will likely change in the
> future.
>
> +If you use the IMAP search engine you may want to configure your IMAP
> +server to provide Full Text Search (FTS) functionality, which greatly
> +improves the performance and results. Beware though that some FTS
> +backends may not implement text search as described by RFC 3501, which
> +states that substring search should be performed. This difference in
> +behaviour can ultimately affect the search results.
> +
> Lastly, all search engines accept a @code{raw-queries-p} option. This
> indicates that engines of this type (or this particular engine) should
> always use raw queries, never parsed (@pxref{Search Queries}).
Looks good to me! Thanks.
On Tuesday, May 03 2022, Eric Abrahamsen wrote: > Sergio Durigan Junior <sergiodj@sergiodj.net> writes: >> I certainly wouldn't mind seeing this possible idiosyncrasy mentioned in >> the manual. WDYT about the patch below? >> >> -- >> Sergio >> GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 >> Please send encrypted e-mail if possible >> https://sergiodj.net/ >> >> From ff85d8b3df751342de6b0071b9c17688086fe744 Mon Sep 17 00:00:00 2001 >> From: Sergio Durigan Junior <sergiodj@sergiodj.net> >> Date: Sun, 1 May 2022 01:11:42 -0400 >> Subject: [PATCH] Mention different behaviours in IMAP FTS plugins >> >> * doc/misc/gnus.texi (Search Engines): Mention different behaviours in >> IMAP FTS plugins. >> --- >> doc/misc/gnus.texi | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/doc/misc/gnus.texi b/doc/misc/gnus.texi >> index e51ae7d424d..03330224e09 100644 >> --- a/doc/misc/gnus.texi >> +++ b/doc/misc/gnus.texi >> @@ -21727,6 +21727,13 @@ Search Engines >> to manage updates yourself, though this will likely change in the >> future. >> >> +If you use the IMAP search engine you may want to configure your IMAP >> +server to provide Full Text Search (FTS) functionality, which greatly >> +improves the performance and results. Beware though that some FTS >> +backends may not implement text search as described by RFC 3501, which >> +states that substring search should be performed. This difference in >> +behaviour can ultimately affect the search results. >> + >> Lastly, all search engines accept a @code{raw-queries-p} option. This >> indicates that engines of this type (or this particular engine) should >> always use raw queries, never parsed (@pxref{Search Queries}). > > Looks good to me! Thanks. Thanks! I'll send the patch to emacs-devel soon. Cheers, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible https://sergiodj.net/