Broken IMAP search with nnir.el

Broken IMAP search with nnir.el

[Tassilo Horn]

Hi all,

I'm using nnir.el to search on an IMAP server, but there are some
problems:

1. If I search the group INBOX it finds some messages in subgroups
   thereof, but not all. For example if I search for "Welcome" it finds
   messages in a lot of subfolders, but if I search for "Frank Schmitt",
   it'll say "gnus-group-read-ephemeral-group: Couldn't request group:
   Search produced empty results.", althought there are some messages
   from him in different groups. If I do `G G' on those groups directly,
   his messages are found.

2. If I hit RET on the search results in the *Summary* buffer, sometimes
   a wrong message gets displayed in the *Article* buffer. See

       http://www.tsdh.de/images/nnir-imap-wrong-msg.png

   As you can see, the subject in summary and article buffer are
   different. "Frank Schmitt" is another recipient of the message
   displayed in *Article*, but it's not displayed in *Summary* and the
   linkage is wrong.

Bye,
Tassilo
-- 
          "Emacs is not a development tool but a way of life."
                - David Kastrup in alt.religion.emacs -

Re: Broken IMAP search with nnir.el

[Simon Josefsson]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Hi all,
>
> I'm using nnir.el to search on an IMAP server, but there are some
> problems:
>
> 1. If I search the group INBOX it finds some messages in subgroups
>    thereof, but not all. For example if I search for "Welcome" it finds
>    messages in a lot of subfolders, but if I search for "Frank Schmitt",
>    it'll say "gnus-group-read-ephemeral-group: Couldn't request group:
>    Search produced empty results.", althought there are some messages
>    from him in different groups. If I do `G G' on those groups directly,
>    his messages are found.

Do (setq imap-log t) and repeat the searches and post the *imap-log*
buffer contents.  Unless there is a silly bug in nnir, I'd say it is
your server that isn't returning the correct data.

> 2. If I hit RET on the search results in the *Summary* buffer, sometimes
>    a wrong message gets displayed in the *Article* buffer. See
>
>        http://www.tsdh.de/images/nnir-imap-wrong-msg.png
>
>    As you can see, the subject in summary and article buffer are
>    different. "Frank Schmitt" is another recipient of the message
>    displayed in *Article*, but it's not displayed in *Summary* and the
>    linkage is wrong.

This may be due to caching -- when you search with nnir the second
time, it re-uses the article numbers since the first search, and if
articles from the first search were cached, they will be displayed.
This is just a theory though.

/Simon

Re: Broken IMAP search with nnir.el

[Tassilo Horn]

Simon Josefsson <simon@josefsson.org> writes:

Hi Simon,

>> 1. If I search the group INBOX it finds some messages in subgroups
>>    thereof, but not all. For example if I search for "Welcome" it finds
>>    messages in a lot of subfolders, but if I search for "Frank Schmitt",
>>    it'll say "gnus-group-read-ephemeral-group: Couldn't request group:
>>    Search produced empty results.", althought there are some messages
>>    from him in different groups. If I do `G G' on those groups directly,
>>    his messages are found.
>
> Do (setq imap-log t) and repeat the searches and post the *imap-log*
> buffer contents.  Unless there is a silly bug in nnir, I'd say it is
> your server that isn't returning the correct data.

Damn, I'm dumb! The searches don't search subfolders, but I had some
copies of messages in other folders in my inbox, so I got the impression
it would search subfolders, too.

So point 1 is clearly a bug in my brain, not in nnir.el. Maybe I should
get an update. ;-)

>> 2. If I hit RET on the search results in the *Summary* buffer, sometimes
>>    a wrong message gets displayed in the *Article* buffer. See
>>
>>        http://www.tsdh.de/images/nnir-imap-wrong-msg.png
>>
>>    As you can see, the subject in summary and article buffer are
>>    different. "Frank Schmitt" is another recipient of the message
>>    displayed in *Article*, but it's not displayed in *Summary* and the
>>    linkage is wrong.
>
> This may be due to caching -- when you search with nnir the second
> time, it re-uses the article numbers since the first search, and if
> articles from the first search were cached, they will be displayed.
> This is just a theory though.

Yes, the displayed message was part of a former search result. Is it
possible to disable this caching?

And another thing: Is it possible to jump from a search result message
to the original message, so that the correct Posting Styles and Group
Parameters are used when I want to reply/forward/whatsoever?

Bye,
Tassilo
-- 
      "DRM manages rights in the same way a jail manages freedom"

Re: Broken IMAP search with nnir.el

[Simon Josefsson]

Tassilo Horn <tassilo@member.fsf.org> writes:

>>> 2. If I hit RET on the search results in the *Summary* buffer, sometimes
>>>    a wrong message gets displayed in the *Article* buffer. See
>>>
>>>        http://www.tsdh.de/images/nnir-imap-wrong-msg.png
>>>
>>>    As you can see, the subject in summary and article buffer are
>>>    different. "Frank Schmitt" is another recipient of the message
>>>    displayed in *Article*, but it's not displayed in *Summary* and the
>>>    linkage is wrong.
>>
>> This may be due to caching -- when you search with nnir the second
>> time, it re-uses the article numbers since the first search, and if
>> articles from the first search were cached, they will be displayed.
>> This is just a theory though.
>
> Yes, the displayed message was part of a former search result. Is it
> possible to disable this caching?

I think it might be easier to have nnir never re-use article numbers.
But I'm not familiar enough with nnir to fix this.

> And another thing: Is it possible to jump from a search result message
> to the original message, so that the correct Posting Styles and Group
> Parameters are used when I want to reply/forward/whatsoever?

Not that I know.

/Simon

Re: Broken IMAP search with nnir.el

[Andreas Seltenreich]

Simon Josefsson writes:

> Tassilo Horn <tassilo@member.fsf.org> writes:
>
>> Yes, the displayed message was part of a former search result. Is it
>> possible to disable this caching?

gnus-backlog-shutdown tends to clean up this kind of caching mess for
me.

> I think it might be easier to have nnir never re-use article numbers.
> But I'm not familiar enough with nnir to fix this.

I'm afraid this would break solid groups, which work ok for engines
that support ordering of returned articles by delivery-date or
something comparable instead of "Score".

>> And another thing: Is it possible to jump from a search result message
>> to the original message, so that the correct Posting Styles and Group
>> Parameters are used when I want to reply/forward/whatsoever?

,----[ C-h f gnus-summary-nnir-goto-thread RET ]
| gnus-summary-nnir-goto-thread is an interactive Lisp function in `nnir.el'.
| (gnus-summary-nnir-goto-thread)
| 
| Only applies to nnir groups.  Go to group this article came from
| and show thread that contains this article.
`----

regards,
andreas

Re: Broken IMAP search with nnir.el

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

I wrote:

> Simon Josefsson writes:
>
>> I think it might be easier to have nnir never re-use article numbers.
>> But I'm not familiar enough with nnir to fix this.
>
> I'm afraid this would break solid groups, which work ok for engines
> that support ordering of returned articles by delivery-date or
> something comparable instead of "Score".

Actually, the same trick that nnweb uses to avoid caching anomalies
with ephemeral groups could be applied to nnir as well.  nnweb uses
message-unique-id to generate names for ephemeral groups, and pushing
such an id into nnir's query parameters has the same effect.

Tassilo: Could you try to reproduce the problem with the attached
patch?  If the problem is actually caching related, the symptoms
should disappear.

regards,
andreas


[-- Attachment #2: Type: text/x-patch, Size: 674 bytes --]

*** nnir.el	17 Jul 2006 17:14:52 +0200	7.19
--- nnir.el	25 Mär 2007 22:21:01 +0200	
***************
*** 294,299 ****
--- 294,300 ----
  (require 'nnoo)
  (require 'gnus-group)
  (require 'gnus-sum)
+ (require 'message)
  (eval-and-compile
    (require 'gnus-util))
  (eval-when-compile
***************
*** 610,615 ****
--- 611,617 ----
      (if extra-parms
          (setq parms (nnir-read-parms query))
        (setq parms (list (cons 'query query))))
+     (add-to-list 'parms (cons 'unique-id (message-unique-id)) t)
      (gnus-group-read-ephemeral-group
       (concat "nnir:" (prin1-to-string parms)) '(nnir "") t
       (cons (current-buffer)

Re: Broken IMAP search with nnir.el

[Tassilo Horn]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

Hi Andreas,

>>> I think it might be easier to have nnir never re-use article
>>> numbers.  But I'm not familiar enough with nnir to fix this.
>>
>> I'm afraid this would break solid groups, which work ok for engines
>> that support ordering of returned articles by delivery-date or
>> something comparable instead of "Score".
>
> Actually, the same trick that nnweb uses to avoid caching anomalies
> with ephemeral groups could be applied to nnir as well.  nnweb uses
> message-unique-id to generate names for ephemeral groups, and pushing
> such an id into nnir's query parameters has the same effect.
>
> Tassilo: Could you try to reproduce the problem with the attached
> patch?

I tried, but failed. ;-)

> If the problem is actually caching related, the symptoms should
> disappear.

Indeed, thanks a lot!

Bye,
Tassilo
-- 
Früher haben wir vor dem Essen immer gebetet, aber jetzt hat meine Frau
einen Kochkurs gemacht.

Re: Broken IMAP search with nnir.el

[Andreas Seltenreich]

Tassilo Horn writes:

> Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
>
>> Tassilo: Could you try to reproduce the problem with the attached
>> patch?
>
> I tried, but failed. ;-)
>
>> If the problem is actually caching related, the symptoms should
>> disappear.
>
> Indeed, thanks a lot!

Thanks for testing, I've now installed the patch in No Gnus.  Btw, an
easy way to reproduce the gnus-backlog related caching problem without
the patch is repeating an nnir search with a different set of process
marked groups.

regards,
andreas

Re: Broken IMAP search with nnir.el

[Tassilo Horn]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

Hi Andreas,

> Thanks for testing, I've now installed the patch in No Gnus.

Great.

> Btw, an easy way to reproduce the gnus-backlog related caching problem
> without the patch is repeating an nnir search with a different set of
> process marked groups.

Yay! I wasn't aware oy process marks on groups but wondered if it's
possible to search in all my imap groups. What a joyful coincidence!

Regards,
Tassilo
-- 
The  desire  to  be  rewarded  for one's  creativity  does  not  justify
depriving  the world  in  general of  all  or part  of that  creativity.
(Richard M. Stallman)

Re: Broken IMAP search with nnir.el

[Tassilo Horn]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

Hi Andreas,

> ,----[ C-h f gnus-summary-nnir-goto-thread RET ]
> | gnus-summary-nnir-goto-thread is an interactive Lisp function in `nnir.el'.
> | (gnus-summary-nnir-goto-thread)
> | 
> | Only applies to nnir groups.  Go to group this article came from
> | and show thread that contains this article.
> `----

Great!

Regards,
Tassilo
-- 
No person,  no idea, and no  religion deserves to be  illegal to insult,
not even the Church of Emacs. (Richard M. Stallman)