Searching emails on an IMAP server?

Searching emails on an IMAP server?

[Niklas Morberg]

Hi,

I've been using different methods of managing my emails over
the years and finally settled for a single archive folder
and then search after what I'm looking for.

I'm currently using Zoe <URL:http://www.zoe.nu> which is ok,
but I lack integration with gnus -- typically I want to
reply or forward messages I've found.

I took a look at nnir.el and got it to work with imap, but
it is oh so slow (and I haven't groked the search interface
-- I can't seem to search for two different words; the
search always seems to use a full string match).

Are there any better options out there? I still want to have
all my emails on the IMAP-server, but I don't mind a local
copy (as with Zoe). I use a GNU/Linux system, and the
IMAP-server is running Exchange.

Niklas

Re: Searching emails on an IMAP server?

[James Leifer]

Niklas Morberg <niklas.morberg@axis.com> writes:

> Are there any better options out there? I still want to have
> all my emails on the IMAP-server, but I don't mind a local
> copy (as with Zoe). I use a GNU/Linux system, and the
> IMAP-server is running Exchange.

Hi,

I run a cron job that indexes all my mail, which consists of about 30K
messages stored in maildir or mh format (one file per message)
nightly.  The indexing is performed by swish++ which is available
under Debian woody.

Then when I want to search, I populate a directory with symlinks to
the messages swish++ returned.

Advantages: this gives me nearly instantaneous searching (fraction of
a second) through all my mail.

Disadvantages: the folder of search results does not have the marks
associated with the real messages nor are mark-changes to the search
results propogated to the real messages.

BTW, maybe nnir.el has a swish++ driver?

Cheers,
-James

Re: Searching emails on an IMAP server?

[Niklas Morberg]

James Leifer <James.Leifer@inria.fr> writes:

> Niklas Morberg <niklas.morberg@axis.com> writes:
>
>> Are there any better options out there? I still want to have
>> all my emails on the IMAP-server, but I don't mind a local
>> copy (as with Zoe). I use a GNU/Linux system, and the
>> IMAP-server is running Exchange.
>
> Hi,

Hi James,

> I run a cron job that indexes all my mail, which consists of about 30K
> messages stored in maildir or mh format (one file per message)
> nightly.  The indexing is performed by swish++ which is available
> under Debian woody.

Ah, so you already have all your mails available locally?

> Then when I want to search, I populate a directory with symlinks to
> the messages swish++ returned.
>
> Advantages: this gives me nearly instantaneous searching (fraction of
> a second) through all my mail.

Excellent, just what I'm after.

> Disadvantages: the folder of search results does not have the marks
> associated with the real messages nor are mark-changes to the search
> results propogated to the real messages.

That's not a big problem for me.

> BTW, maybe nnir.el has a swish++ driver?

Actually it does. It also supports glimpse, waissearch,
excite, swish-e, namazu and hyrex (and native IMAP search).

The thing I haven't found yet is how to get any of these
search methods to play nice with IMAP -- they all seem to
want local files to work with.

Maybe I could run a cron job that copies mail off my IMAP
server and stores them locally (just like Zoe)? Or maybe use
Zoe just as an email fetch script and use another search
engine to index them? A Zoe backed to gnus would be another
option, but I don't think I could pull off writing one.

Thanks for your reply James, I'll do some more googling to
see if I can come up with anything else.

Niklas

Re: Searching emails on an IMAP server?

[James Leifer]

Niklas Morberg <niklas.morberg@axis.com> writes

(snipping liberally)

> Ah, so you already have all your mails available locally?

Yes.

> Maybe I could run a cron job that copies mail off my IMAP
> server and stores them locally (just like Zoe)?

Perhaps you could look at offlineimap which is supposed to do this
kind of copying well.

I assume you have no access to the machine that runs the imap server?
If you did have access, you could probably do what I do locally but on
*that* machine.  Then when you wanted to search, you would be able to
do:

  ssh imap-server search-and-populate-folder-script "my search string"

and view the results folder through any imap client back on your local
machine.

> Thanks for your reply James, I'll do some more googling to
> see if I can come up with anything else.

You're welcome.  Let us know what you discover!

-J

Re: Searching emails on an IMAP server?

[Kai Grossjohann]

Niklas Morberg <niklas.morberg@axis.com> writes:

> The thing I haven't found yet is how to get any of these
> search methods to play nice with IMAP -- they all seem to
> want local files to work with.

You could use the Agent to cache mails locally.  Then you can tell
your favorite search engine to pick them up.

> Maybe I could run a cron job that copies mail off my IMAP
> server and stores them locally (just like Zoe)?

That would be possible, though with the Agent, Gnus can do it.

> Or maybe use Zoe just as an email fetch script and use another
> search engine to index them? A Zoe backed to gnus would be another
> option, but I don't think I could pull off writing one.

How does one talk to Zoe?  Writing a backend is not that difficult.
The backend interface is documented well enough, and you don't have
to understand why all the functions are supposed to be present, you
just implement what it says in the documentation and all is well.

The most difficult part for me was to understand the terminology.
But the terminology is explained in the node "Back End Interface".

Kai

Re: Searching emails on an IMAP server?

[Niklas Morberg]

Kai Grossjohann <kai@emptydomain.de> writes:

> Niklas Morberg <niklas.morberg@axis.com> writes:
>
>> Maybe I could run a cron job that copies mail off my IMAP
>> server and stores them locally (just like Zoe)?
>
> That would be possible, though with the Agent, Gnus can do it.

I'll take a look at the agent again. I tried it out for
about a year ago, but got into a some problems with marks
not being preserved when moving messages. Since then I've
stopped using marks so it really doesn't matter.

Is any of the search engines currently supported by Gnus
"better" than the others?

>> Or maybe use Zoe just as an email fetch script and use another
>> search engine to index them? A Zoe backed to gnus would be another
>> option, but I don't think I could pull off writing one.
>
> How does one talk to Zoe?

Zoe has an HTTP interface. See screenshot at
<URL:http://zoe.nu/itstories/story.php?data=stories&num=23&sec=2>.

> Writing a backend is not that difficult. The backend
> interface is documented well enough, and you don't
> have to understand why all the functions are supposed
> to be present, you just implement what it says in the
> documentation and all is well.

Would the best option be to implement a new backend, or
"just" add a new search engine to nnir.el? (I was thinking
about the latter in my original post.)

My biggest gripe with Zoe is that it is very slow if there
are a lot of hits to your search. For now, I'll boot up the
agent and see how it pans out.

> The most difficult part for me was to understand the
> terminology. But the terminology is explained in the node
> "Back End Interface".

Thanks for the pointer, I'll keep it for future reference.

Niklas