regular and adaptive scoring with nnimap

regular and adaptive scoring with nnimap

[Ted Zlatanov]

As I understand it, regular and adaptive scoring use local files to
store their data.  With nnimap, that's not a great idea because mail
can be read from many locations, so the scores will be out of sync.

It would be nice if I could store score files on the IMAP server as
articles in a special group.  I don't think IMAP provides for a
simpler way to do this.

I don't know if this has been brought up before, sorry if it's an old
topic.

Ted

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It would be nice if I could store score files on the IMAP server as
> articles in a special group.  I don't think IMAP provides for a
> simpler way to do this.

IIUC, IMAP goes together with something called ACAP and another kind
of protocol (one is obsoleted by the other, but I don't know which is
which).  ACAP and its friend whose name I've forgotten allow you to
store configuration data on a server.
-- 
Ambibibentists unite!

Re: regular and adaptive scoring with nnimap

[Simon Josefsson]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> It would be nice if I could store score files on the IMAP server as
>> articles in a special group.  I don't think IMAP provides for a
>> simpler way to do this.
>
> IIUC, IMAP goes together with something called ACAP and another kind
> of protocol (one is obsoleted by the other, but I don't know which is
> which).  ACAP and its friend whose name I've forgotten allow you to
> store configuration data on a server.

Perhaps IMSP?  But ACAP/IMSP doesn't seem to fly (not even packaged
for Debian), so I think storing MUA data on the IMAP server as
articles is the best we can do.  I have always wondered if tramp could
be used to implement a IMAP-based file system for emacs...

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Sun, 26 Jan 2003, jas@extundo.com wrote:
> Perhaps IMSP?  But ACAP/IMSP doesn't seem to fly (not even packaged
> for Debian), so I think storing MUA data on the IMAP server as
> articles is the best we can do.  I have always wondered if tramp
> could be used to implement a IMAP-based file system for emacs...

Using tramp over IMAP would be quite a sight.  But assuming most Gnus
users don't have tramp installed, can we provide a generic
functionality of storing and retrieving a buffer to the IMAP server,
given a group name and a logical name for the buffer contents?

The logical name could be stored in the subject of the message, I
suppose, for quick retrieval.

Or should we bite the bullet and add gnus-tramp.el to Gnus to provide
nnimap functionality for tramp?  I wouldn't mind that, since I use
tramp, and I think it's a very clean approach to the problem.

Ted

Re: regular and adaptive scoring with nnimap

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 26 Jan 2003, jas@extundo.com wrote:
>> Perhaps IMSP?  But ACAP/IMSP doesn't seem to fly (not even packaged
>> for Debian), so I think storing MUA data on the IMAP server as
>> articles is the best we can do.  I have always wondered if tramp
>> could be used to implement a IMAP-based file system for emacs...
>
> Using tramp over IMAP would be quite a sight.  But assuming most Gnus
> users don't have tramp installed, can we provide a generic
> functionality of storing and retrieving a buffer to the IMAP server,
> given a group name and a logical name for the buffer contents?
>
> The logical name could be stored in the subject of the message, I
> suppose, for quick retrieval.

Sounds like a plan.   imap-db.el or something.

> Or should we bite the bullet and add gnus-tramp.el to Gnus to provide
> nnimap functionality for tramp?  I wouldn't mind that, since I use
> tramp, and I think it's a very clean approach to the problem.

It wouldn't even have to be nnimap specific.  Files could be organised
as gnus/nnimap+foobar/group/4711 or something.  But this sounds like
real work; imap-db.el would probably be quite short.

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Using tramp over IMAP would be quite a sight.  But assuming most Gnus
> users don't have tramp installed, can we provide a generic
> functionality of storing and retrieving a buffer to the IMAP server,
> given a group name and a logical name for the buffer contents?

On the one hand, Tramp will be available with Emacs 21.4, so its
availability is not that much of a problem.

On the other hand, it's not clear to me that much of the Tramp
infrastructure could be used for a filename handler that uses IMAP.
Probably it could just as easily be implemented as its own filename
handler.

The filename handler doesn't need to be that complete, either: just
enough to load/store the config data.  Filename completion and that
stuff isn't really needed, I think.
-- 
Ambibibentists unite!

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Sun, 26 Jan 2003, kai.grossjohann@uni-duisburg.de wrote:
> On the other hand, it's not clear to me that much of the Tramp
> infrastructure could be used for a filename handler that uses IMAP.
> Probably it could just as easily be implemented as its own filename
> handler.
> 
> The filename handler doesn't need to be that complete, either: just
> enough to load/store the config data.  Filename completion and that
> stuff isn't really needed, I think.

If I knew anything about filename handlers, I'd love to write this.
How hard are they to use?  This needs to be integrated with nnimap.el
pretty tightly, so it should be an interesting module.

Ted

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If I knew anything about filename handlers, I'd love to write this.
> How hard are they to use?  This needs to be integrated with nnimap.el
> pretty tightly, so it should be an interesting module.

When I wrote Tramp, I just looked at the magic filenames node in the
elisp reference and at the documentation of each function referenced
there.  From time to time, I also looked at Ange-FTP to see how it
does things.  (That's how I discovered that (0 0) appears to be the
way to express a don't-know value for the mtime, ctime and atime
values returned from file-attributes.)

It seems intimidating at first, but turns out to be not that
difficult.
-- 
Ambibibentists unite!

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Mon, 27 Jan 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> If I knew anything about filename handlers, I'd love to write this.
>> How hard are they to use?  This needs to be integrated with
>> nnimap.el pretty tightly, so it should be an interesting module.
> 
> When I wrote Tramp, I just looked at the magic filenames node in the
> elisp reference and at the documentation of each function referenced
> there.  From time to time, I also looked at Ange-FTP to see how it
> does things.  (That's how I discovered that (0 0) appears to be the
> way to express a don't-know value for the mtime, ctime and atime
> values returned from file-attributes.)
> 
> It seems intimidating at first, but turns out to be not that
> difficult.

Well, I'm basically trying to figure things out at this point...

I wrote some pieces of code, specifically file creation and listing of
groups on a nnimap server.  I'm very slowly working my way through the
file operations - stuff like make-directory is trivial when dealing
with files, but pretty complex with nnimap.  Fortunately I can just
ignore most file functionality.

groups will be mapped to directories
files will be mapped to articles (filename == article number)
the current nnimap server will always be used (good idea?  bad idea?)

Filenames will have to be numeric.  Retrieving an article by subject
is significantly harder.

I'm currently trying to figure out how to enter the storage nnimap
group to store or retrieve an article without affecting variables like
gnus-newsgroup-name.

Ted

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I wrote some pieces of code, specifically file creation and listing of
> groups on a nnimap server.  I'm very slowly working my way through the
> file operations - stuff like make-directory is trivial when dealing
> with files, but pretty complex with nnimap.  Fortunately I can just
> ignore most file functionality.

Yeah, you could throw an error on make-directory.  Or you do
something similar to `G m'?

> groups will be mapped to directories
> files will be mapped to articles (filename == article number)
> the current nnimap server will always be used (good idea?  bad idea?)

See below.

> Filenames will have to be numeric.  Retrieving an article by subject
> is significantly harder.

Yes.

> I'm currently trying to figure out how to enter the storage nnimap
> group to store or retrieve an article without affecting variables like
> gnus-newsgroup-name.

You could circumvent nnimap and use imap.el.  Then the nnimap
variables will not be affected.

How about allowing the user to type the user and host name into the
filename?
-- 
A turnip curses Elvis

Re: regular and adaptive scoring with nnimap

[Andreas Fuchs]

On 2003-02-03, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Filenames will have to be numeric.  Retrieving an article by subject
> is significantly harder.

Why not just use http://www.isi.edu/in-notes/rfc2192.txt URLs?

> I'm currently trying to figure out how to enter the storage nnimap
> group to store or retrieve an article without affecting variables like
> gnus-newsgroup-name.

These are lexically bound. It should be possible to (let
((gnus-newsgroup-name "name")) <do things>) and have them automatically
restored after <do things>.

hth,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs

Re: regular and adaptive scoring with nnimap

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I wrote some pieces of code, specifically file creation and listing of
> groups on a nnimap server.  I'm very slowly working my way through the
> file operations - stuff like make-directory is trivial when dealing
> with files, but pretty complex with nnimap.  Fortunately I can just
> ignore most file functionality.
>
> groups will be mapped to directories
> files will be mapped to articles (filename == article number)
> the current nnimap server will always be used (good idea?  bad idea?)

As Kai said, using imap.el directly might be better.  nnimap.el is a
hack but I like to believe that imap.el isn't as bad.  I think nnimap
still alters some Gnus internal variables when you enter a group, for
instance.

OTOH using nnimap might make it easy to generalize the code for any
Gnus backend, which could be interesting.

> Filenames will have to be numeric.  Retrieving an article by subject
> is significantly harder.

imap.el's `imap-search' is rather powerful;

(imap-search "HEADER Message-Id \"mydirectory/myfile.txt\"")

should return all UID's for articles with that Subject.

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Tue, 4 Feb 2003, asf@void.at wrote:
> On 2003-02-03, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Filenames will have to be numeric.  Retrieving an article by
>> subject is significantly harder.
> 
> Why not just use http://www.isi.edu/in-notes/rfc2192.txt URLs?

Chiefly, it was my ignorance of that RFC that caused me to overlook it
:) Now that I look at it, it's rather difficult to reconcile it with
Gnus.  The URL has a lot of information built-in that in Gnus is
associated with the server name.  So I should ignore that part of the
RFC and just do

imap://minbari.org/storage?SUBJECT%20filename

passing the filename as a subject string search, and

imap://minbari.org/storage?UID=20

for direct access to articles.

Something like this is probably good enough, and I'd rather let
imap.el worry about the best authenticator.  Does that sound OK?

Supporting the full syntax is way too much work for something that
Gnus already does internally.  If anyone wants to work on supporting
the full URL scheme and create an ephemeral server on the fly to
retrieve a message, they're welcome :)

>> I'm currently trying to figure out how to enter the storage nnimap
>> group to store or retrieve an article without affecting variables
>> like gnus-newsgroup-name.
> 
> These are lexically bound. It should be possible to (let
> ((gnus-newsgroup-name "name")) <do things>) and have them
> automatically restored after <do things>.

Sorry, I should have been clearer.  I understand about local scope, I
just don't know which variables to save before entering another group
temporarily.  But I think this won't be a problem if I use imap.el
instead of nnimap.el.

Ted

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Tue, 04 Feb 2003, jas@extundo.com wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> I wrote some pieces of code, specifically file creation and listing
>> of groups on a nnimap server.  I'm very slowly working my way
>> through the file operations - stuff like make-directory is trivial
>> when dealing with files, but pretty complex with nnimap.
>> Fortunately I can just ignore most file functionality.
>>
>> groups will be mapped to directories files will be mapped to
>> articles (filename == article number) the current nnimap server
>> will always be used (good idea?  bad idea?)
> 
> As Kai said, using imap.el directly might be better.  nnimap.el is a
> hack but I like to believe that imap.el isn't as bad.  I think
> nnimap still alters some Gnus internal variables when you enter a
> group, for instance.

I was hoping to find those variables, but I see that it's a losing
game to play catch-up with the rest of Gnus as new group-level
variables are introduced.  There's no save-all-variables-in-scope, is
there? :)

> OTOH using nnimap might make it easy to generalize the code for any
> Gnus backend, which could be interesting.

I thought about that, but I'm not sure I'm up to doing all that work -
I'm pretty sure it requires modifications to Gnus in order to allow a
temporary switch to another group without repercussions to the current
group.

>> Filenames will have to be numeric.  Retrieving an article by
>> subject is significantly harder.
> 
> imap.el's `imap-search' is rather powerful;
> 
> (imap-search "HEADER Message-Id \"mydirectory/myfile.txt\"")
> 
> should return all UID's for articles with that Subject.

OK, I'll use that and allow for both numeric and by-subject
addressing.  Thanks!

Ted

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Tue, 04 Feb 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I'm currently trying to figure out how to enter the storage nnimap
>> group to store or retrieve an article without affecting variables
>> like gnus-newsgroup-name.
> 
> You could circumvent nnimap and use imap.el.  Then the nnimap
> variables will not be affected.

Great idea.

> How about allowing the user to type the user and host name into the
> filename?

All that (user name, authentication scheme, etc.) is normally
associated with a server by Gnus but it's a part of the URL with
imap:// URLs.  I think I'll only interpret host names, I can't write
the code to handle creating an ephemeral server for arbitrary message
URLs and it's really unnecessary IMO for what I need.

Ted

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> All that (user name, authentication scheme, etc.) is normally
> associated with a server by Gnus but it's a part of the URL with
> imap:// URLs.  I think I'll only interpret host names, I can't write
> the code to handle creating an ephemeral server for arbitrary message
> URLs and it's really unnecessary IMO for what I need.

Why's there an ephemeral server if you avoid nnimap?

Hm.  Maybe I should read that RFC...
-- 
A turnip curses Elvis

Re: regular and adaptive scoring with nnimap

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 4 Feb 2003, asf@void.at wrote:
>> On 2003-02-03, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> Filenames will have to be numeric.  Retrieving an article by
>>> subject is significantly harder.
>> 
>> Why not just use http://www.isi.edu/in-notes/rfc2192.txt URLs?
>
> Chiefly, it was my ignorance of that RFC that caused me to overlook it
> :) Now that I look at it, it's rather difficult to reconcile it with
> Gnus.  The URL has a lot of information built-in that in Gnus is
> associated with the server name.

But you're going to use imap.el?  That has no built-in information,
I'd think -- it's just a wrapper around the IMAP protocol, right?

But I haven't really looked closely, so this might be very wrong.
-- 
A turnip curses Elvis

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

On Tue, 04 Feb 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> On Tue, 4 Feb 2003, asf@void.at wrote:
>>> On 2003-02-03, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>>> Filenames will have to be numeric.  Retrieving an article by
>>>> subject is significantly harder.
>>> 
>>> Why not just use http://www.isi.edu/in-notes/rfc2192.txt URLs?
>>
>> Chiefly, it was my ignorance of that RFC that caused me to overlook
>> it :) Now that I look at it, it's rather difficult to reconcile it
>> with Gnus.  The URL has a lot of information built-in that in Gnus
>> is associated with the server name.
> 
> But you're going to use imap.el?  That has no built-in information,
> I'd think -- it's just a wrapper around the IMAP protocol, right?
> 
> But I haven't really looked closely, so this might be very wrong.

You're right.  But interpreting the full URL syntax is, IMO, not
necessary with Gnus since a server name is already associated with all
the necessary parameters.  To re-interpret the full URL scheme would
be fun, I'm sure, but would not make sense in writing a simple
store/retrieve IMAP file handler for Gnus.

Ted

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

How does this look, for retrieving an article and searching a group by
any parameter?

(defun nnimap-db-get-article-contents (article group &optional server)
  (let ((data))
    (when (numberp article)
      (with-current-buffer (nnimap-get-server-buffer server)
	(let ((saved (imap-current-mailbox)))
	  (with-temp-buffer
	    (nnimap-request-article article group server (buffer-name))
	    (setq data (buffer-string)))
	  (nnimap-possibly-change-group saved server))))
    data))

;; (nnimap-db-get-article-contents 4388 "ding")

(defun nnimap-db-get-uids-for-parameter (group parameter subject &optional server)
  (let ((articles))
    (when (and (stringp subject) (stringp parameter) (stringp group))
      (with-current-buffer (nnimap-get-server-buffer server)
	(let ((saved (imap-current-mailbox)))
	  (nnimap-possibly-change-group group server)
	  (setq articles (imap-search (format "HEADER %s \"%s\"" parameter subject)))
	  (nnimap-possibly-change-group saved server))))
    articles))
	
;; (nnimap-db-get-uids-for-parameter "ding" "Subject" "spam")
;; (nnimap-db-get-uids-for-parameter "ding" "Message-Id" "<ilu4r7k83j5.fsf@latte.josefsson.org>")

I'm assuming two things:

1) (nnimap-possibly-change-group nil) is harmless
2) (nnimap-possibly-change-group nil) returns the current group

Are both true?

I would use imap exclusively, but nnimap-possibly-change-group and the
other functions it offers are too useful :)

Also, do I need to do

(setq gnus-newsgroup-active (gnus-activate-group group))

before each operation in case the nnimap server is not activated?

Thanks
Ted

Re: regular and adaptive scoring with nnimap

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> How does this look, for retrieving an article and searching a group by
> any parameter?
>
> (defun nnimap-db-get-article-contents (article group &optional server)
>   (let ((data))
>     (when (numberp article)
>       (with-current-buffer (nnimap-get-server-buffer server)
> 	(let ((saved (imap-current-mailbox)))
> 	  (with-temp-buffer
> 	    (nnimap-request-article article group server (buffer-name))
> 	    (setq data (buffer-string)))
> 	  (nnimap-possibly-change-group saved server))))
>     data))

Looks good.  You might want to remove the store-and-restore current
mailbox, I don't think it is necessary and on some servers it will be
really slow.

> (defun nnimap-db-get-uids-for-parameter (group parameter subject &optional server)
>   (let ((articles))
>     (when (and (stringp subject) (stringp parameter) (stringp group))
>       (with-current-buffer (nnimap-get-server-buffer server)
> 	(let ((saved (imap-current-mailbox)))
> 	  (nnimap-possibly-change-group group server)
> 	  (setq articles (imap-search (format "HEADER %s \"%s\"" parameter subject)))
> 	  (nnimap-possibly-change-group saved server))))
>     articles))

Looks good, with the same comment as above.

> ;; (nnimap-db-get-uids-for-parameter "ding" "Subject" "spam")
> ;; (nnimap-db-get-uids-for-parameter "ding" "Message-Id" "<ilu4r7k83j5.fsf@latte.josefsson.org>")
>
> I'm assuming two things:
>
> 1) (nnimap-possibly-change-group nil) is harmless
> 2) (nnimap-possibly-change-group nil) returns the current group
>
> Are both true?

n-p-c-g is actually somewhat broken (uidvalidity), so I suggest not
using it.  Instead, either use

      (if (imap-current-mailbox-p group)
	  imap-current-mailbox
	(if (imap-mailbox-select group)
           imap-current-mailbox
         (error "Could not select group: %s" (imap-error-text)))

or something.

> I would use imap exclusively, but nnimap-possibly-change-group and the
> other functions it offers are too useful :)

Maybe the above code should be added as imap-possibly-change-group.  Hm.

> Also, do I need to do
>
> (setq gnus-newsgroup-active (gnus-activate-group group))
>
> before each operation in case the nnimap server is not activated?

You should try to make sure the server is open, yes.  Setting that
variable is probably not the right thing though, I think it is buffer
local to summary buffers. g-a-g calls various hooks and stuff too, so
perhaps it is not perfect to use.

Re: regular and adaptive scoring with nnimap

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Attached is the latest imap-db.el.  It only does file-exists-p right
now, but it's got the functions for file retrieval and file creation
in place.  It can't write over an existing file yet (replace body).
Plus, I think the file retrieval may need to get just the body, not
the body plus the headers.

imap-db-create-article is especially hairy.  I'm not sure I'm doing
the right things in there.  I'm not inserting the new article in the
nnmail cache, for instance.

I added the imap-possibly-change-group function, feel free to move it
to imap.el if it's OK.

I picked a fairly simple format:

nnimap://mail.lifelogs.com/ding?SUBJECT%20spam

that's it, no UIDs, no authentications, no uidvalidity.  I'm happy to
let nnimap.el handle all of that.

Anyhow, let me know if there are obvious bugs or errors in the code.

Ted


[-- Attachment #2: imap-db.el --]
[-- Type: application/emacs-lisp, Size: 7104 bytes --]