message registry for Gnus

message registry for Gnus

[Ted Zlatanov]

So, I've been thinking about a message registry.  What will it do?

- track messages by ID

- associate a Lisp structure (probably a list, hashes are too new?)
  with a message ID

- allow every customer package to hook into the relevant functions

- store the registry in a separate file or in the newsrc.eld?

Before I start coding, I wanted to sort of look around and see what
people think.

In the context of spam.el, the registry will be used to track when a
message changes groups, or is processed by a spam/ham processor.  That
way, we can see if a message that will be registered as spam with
ifile now has been registered as ham with Bogofilter, for instance,
and reverse the Bogofilter registration before registering with ifile.

In Gnus terms, what are the basic message transfer functions?

- message copy

- message move

- new message spooling

- old message respooling

What else?

Is this easily doable?

Thanks
Ted

Re: message registry for Gnus

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> So, I've been thinking about a message registry.  What will it do?
>
> - track messages by ID
>
> - associate a Lisp structure (probably a list, hashes are too new?)
>   with a message ID

Way cool.  I think it's safe to use hashes -- Gnus uses them itself,
I think.  See gnus-make-hashtable, gnus-gethash, gnus-sethash.

-- 
Ambibibentists unite!

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> In the context of spam.el, the registry will be used to track when a
> message changes groups, or is processed by a spam/ham processor.  That
> way, we can see if a message that will be registered as spam with
> ifile now has been registered as ham with Bogofilter, for instance,
> and reverse the Bogofilter registration before registering with ifile.

Sure, sounds like a good idea.

> In Gnus terms, what are the basic message transfer functions?
>
> - message copy
>
> - message move
>
> - new message spooling
>
> - old message respooling

And expiry with expiry-target.

Copy, move and respooling are all done in
`gnus-summary-move-article'. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Sat, 01 Feb 2003, larsi@gnus.org wrote:
>> In Gnus terms, what are the basic message transfer functions?
>>
>> - message copy
>>
>> - message move
>>
>> - new message spooling
>>
>> - old message respooling
> 
> And expiry with expiry-target.
> 
> Copy, move and respooling are all done in
> `gnus-summary-move-article'. 

OK, I added the hooks for copy/move/respool/delete/expire.  Attached
is a patch and an empty gnus-registry.el with an example of how things
are called.  I haven't committed anything to CVS.

The message deletion hook (gnus-summary-article-delete-hook)
invocation in gnus-summary-move-article doesn't seem to do anything,
when is that 'junk condition in gnus-sum.el used?

I think I got expiry and deletion right otherwise, can you check?
Especially expiry is tricky, I'm not sure I understand the whole
function.

Incoming messages seem to be spooled in several places, so I'm not
sure how to deal with that.  I only want to intercept nnmail and
nnimap for now, should I just prepend the hook call to their
respective split-methods?

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: registry.patch --]
[-- Type: text/x-patch, Size: 3554 bytes --]

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 6.153
diff -r6.153 gnus.el
61a62,65
> (defgroup gnus-registry nil
>   "Article Registry."
>   :group 'gnus)
> 
3114,3115c3118,3120
< (defun gnus-group-prefixed-name (group method)
<   "Return the whole name from GROUP and METHOD."
---
> (defun gnus-group-prefixed-name (group method &optional full)
>   "Return the whole name from GROUP and METHOD.  Call with full set to
> get the fully qualified group name (even if the server is native)."
3118c3123
< 	  (gnus-server-equal method "native")
---
> 	  (and (not full) (gnus-server-equal method "native"))
3121a3127,3140
> 
> (defun gnus-group-guess-prefixed-name (group)
>   "Guess the whole name from GROUP and METHOD."
>   (gnus-group-prefixed-name group (gnus-find-method-for-group
> 			       group)))
> 
> (defun gnus-group-full-name (group method)
>   "Return the full name from GROUP and METHOD, even if the method is
> native."
>   (gnus-group-prefixed-name group method t))
> 
> (defun gnus-group-guess-full-name (group)
>   "Guess the full name from GROUP, even if the method is native."
>   (gnus-group-full-name group (gnus-find-method-for-group group)))
Index: gnus-sum.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v
retrieving revision 6.300
diff -r6.300 gnus-sum.el
849a850,864
> (defcustom gnus-summary-article-move-hook nil
>   "*A hook called after an article is moved, copied, respooled, or crossposted."
>   :group 'gnus-summary
>   :type 'hook)
> 
> (defcustom gnus-summary-article-delete-hook nil
>   "*A hook called after an article is deleted."
>   :group 'gnus-summary
>   :type 'hook)
> 
> (defcustom gnus-summary-article-expire-hook nil
>   "*A hook called after an article is expired."
>   :group 'gnus-summary
>   :type 'hook)
> 
8759,8760c8774,8781
< 	  (gnus-summary-mark-article article gnus-canceled-mark)
< 	  (gnus-message 4 "Deleted article %s" article)))
---
> 	  (let ((id (mail-header-id (gnus-data-header 
> 				     (assoc article (gnus-data-list nil))))))
> 	    (gnus-summary-mark-article article gnus-canceled-mark)
> 	    (gnus-message 4 "Deleted article %s" article)
> 	    ;; run the move/copy/crosspost/respool hook
> 	    (run-hook-with-args 'gnus-summary-article-delete-hook 
> 				action id gnus-newsgroup-name nil
> 				select-method))))
8838c8859,8866
< 	       article gnus-newsgroup-name (current-buffer)))))
---
> 	       article gnus-newsgroup-name (current-buffer))))
> 
> 	  ;; run the move/copy/crosspost/respool hook
> 	  (let ((id (mail-header-id (gnus-data-header 
> 				   (assoc article (gnus-data-list nil))))))
> 	  (run-hook-with-args 'gnus-summary-article-move-hook 
> 			      action id gnus-newsgroup-name to-newsgroup
> 			      select-method)))
9059c9087,9093
< 		  (gnus-summary-mark-article article gnus-canceled-mark))))))
---
> 		  (gnus-summary-mark-article article gnus-canceled-mark)
> 		  (let ((id (mail-header-id (gnus-data-header 
> 					     (assoc article 
> 						    (gnus-data-list nil))))))
> 		    (run-hook-with-args 'gnus-summary-article-expire-hook
> 					'delete id gnus-newsgroup-name nil
> 					nil)))))))
9107a9142,9147
> 	(let* ((article (car articles))
> 	       (id (mail-header-id (gnus-data-header 
> 				    (assoc article (gnus-data-list nil))))))
> 	  (run-hook-with-args 'gnus-summary-article-delete-hook
> 			      'delete id gnus-newsgroup-name nil
> 			      nil))

[-- Attachment #3: gnus-registry.el --]
[-- Type: application/emacs-lisp, Size: 1818 bytes --]

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The message deletion hook (gnus-summary-article-delete-hook)
> invocation in gnus-summary-move-article doesn't seem to do anything,
> when is that 'junk condition in gnus-sum.el used?

You have `gnus-move-split-methods', which can return groups and
`junk', which means "delete this".  Mostly useful when respooling.

> I think I got expiry and deletion right otherwise, can you check?
> Especially expiry is tricky, I'm not sure I understand the whole
> function.

Could you do a unified diff?  I have a hard time reading raw diffs... 

> Incoming messages seem to be spooled in several places, so I'm not
> sure how to deal with that.  I only want to intercept nnmail and
> nnimap for now, should I just prepend the hook call to their
> respective split-methods?

I think that would probably be the best way, but I forget how that
stuff works in detail...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

On Sun, 02 Feb 2003, larsi@gnus.org wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> The message deletion hook (gnus-summary-article-delete-hook)
>> invocation in gnus-summary-move-article doesn't seem to do
>> anything, when is that 'junk condition in gnus-sum.el used?
> 
> You have `gnus-move-split-methods', which can return groups and
> `junk', which means "delete this".  Mostly useful when respooling.

OK, as long as I'm right in calling it a delete hook...

>> I think I got expiry and deletion right otherwise, can you check?
>> Especially expiry is tricky, I'm not sure I understand the whole
>> function.
> 
> Could you do a unified diff?  I have a hard time reading raw
> diffs...

Sorry, I forgot the -u.  Unified diff attached, plus gnus-registry.el
for good measure.

>> Incoming messages seem to be spooled in several places, so I'm not
>> sure how to deal with that.  I only want to intercept nnmail and
>> nnimap for now, should I just prepend the hook call to their
>> respective split-methods?
> 
> I think that would probably be the best way, but I forget how that
> stuff works in detail...  :-)

Does anyone know?  I'm just trying to avoid an hour of
source-diving...

Thanks
Ted


[-- Attachment #2: gnus-registry.el --]
[-- Type: application/emacs-lisp, Size: 1818 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: registry.patch --]
[-- Type: text/x-patch, Size: 5235 bytes --]

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 6.153
diff -u -r6.153 gnus.el
--- gnus.el	24 Jan 2003 19:35:31 -0000	6.153
+++ gnus.el	2 Feb 2003 15:07:05 -0000
@@ -59,6 +59,10 @@
   :link '(custom-manual "(gnus)Article Caching")
   :group 'gnus)
 
+(defgroup gnus-registry nil
+  "Article Registry."
+  :group 'gnus)
+
 (defgroup gnus-start nil
   "Starting your favorite newsreader."
   :group 'gnus)
@@ -3111,14 +3115,29 @@
 (defsubst gnus-method-to-full-server-name (method)
   (format "%s+%s" (car method) (nth 1 method)))
 
-(defun gnus-group-prefixed-name (group method)
-  "Return the whole name from GROUP and METHOD."
+(defun gnus-group-prefixed-name (group method &optional full)
+  "Return the whole name from GROUP and METHOD.  Call with full set to
+get the fully qualified group name (even if the server is native)."
   (and (stringp method) (setq method (gnus-server-to-method method)))
   (if (or (not method)
-	  (gnus-server-equal method "native")
+	  (and (not full) (gnus-server-equal method "native"))
 	  (string-match ":" group))
       group
     (concat (gnus-method-to-server-name method) ":" group)))
+
+(defun gnus-group-guess-prefixed-name (group)
+  "Guess the whole name from GROUP and METHOD."
+  (gnus-group-prefixed-name group (gnus-find-method-for-group
+			       group)))
+
+(defun gnus-group-full-name (group method)
+  "Return the full name from GROUP and METHOD, even if the method is
+native."
+  (gnus-group-prefixed-name group method t))
+
+(defun gnus-group-guess-full-name (group)
+  "Guess the full name from GROUP, even if the method is native."
+  (gnus-group-full-name group (gnus-find-method-for-group group)))
 
 (defun gnus-group-real-prefix (group)
   "Return the prefix of the current group name."
Index: gnus-sum.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v
retrieving revision 6.300
diff -u -r6.300 gnus-sum.el
--- gnus-sum.el	27 Jan 2003 02:56:44 -0000	6.300
+++ gnus-sum.el	2 Feb 2003 15:07:12 -0000
@@ -847,6 +847,21 @@
   :group 'gnus-summary
   :type 'hook)
 
+(defcustom gnus-summary-article-move-hook nil
+  "*A hook called after an article is moved, copied, respooled, or crossposted."
+  :group 'gnus-summary
+  :type 'hook)
+
+(defcustom gnus-summary-article-delete-hook nil
+  "*A hook called after an article is deleted."
+  :group 'gnus-summary
+  :type 'hook)
+
+(defcustom gnus-summary-article-expire-hook nil
+  "*A hook called after an article is expired."
+  :group 'gnus-summary
+  :type 'hook)
+
 (defcustom gnus-summary-display-arrow
   (and (fboundp 'display-graphic-p)
        (display-graphic-p))
@@ -8756,8 +8771,14 @@
 		      (nnheader-get-report (car to-method))))
        ((eq art-group 'junk)
 	(when (eq action 'move)
-	  (gnus-summary-mark-article article gnus-canceled-mark)
-	  (gnus-message 4 "Deleted article %s" article)))
+	  (let ((id (mail-header-id (gnus-data-header 
+				     (assoc article (gnus-data-list nil))))))
+	    (gnus-summary-mark-article article gnus-canceled-mark)
+	    (gnus-message 4 "Deleted article %s" article)
+	    ;; run the move/copy/crosspost/respool hook
+	    (run-hook-with-args 'gnus-summary-article-delete-hook 
+				action id gnus-newsgroup-name nil
+				select-method))))
        (t
 	(let* ((pto-group (gnus-group-prefixed-name
 			   (car art-group) to-method))
@@ -8835,7 +8856,14 @@
 	      (gnus-request-article-this-buffer article gnus-newsgroup-name)
 	      (nnheader-replace-header "Xref" new-xref)
 	      (gnus-request-replace-article
-	       article gnus-newsgroup-name (current-buffer)))))
+	       article gnus-newsgroup-name (current-buffer))))
+
+	  ;; run the move/copy/crosspost/respool hook
+	  (let ((id (mail-header-id (gnus-data-header 
+				   (assoc article (gnus-data-list nil))))))
+	  (run-hook-with-args 'gnus-summary-article-move-hook 
+			      action id gnus-newsgroup-name to-newsgroup
+			      select-method)))
 
 	;;;!!!Why is this necessary?
 	(set-buffer gnus-summary-buffer)
@@ -9056,7 +9084,13 @@
 	      (dolist (article expirable)
 		(when (and (not (memq article es))
 			   (gnus-data-find article))
-		  (gnus-summary-mark-article article gnus-canceled-mark))))))
+		  (gnus-summary-mark-article article gnus-canceled-mark)
+		  (let ((id (mail-header-id (gnus-data-header 
+					     (assoc article 
+						    (gnus-data-list nil))))))
+		    (run-hook-with-args 'gnus-summary-article-expire-hook
+					'delete id gnus-newsgroup-name nil
+					nil)))))))
 	(gnus-message 6 "Expiring articles...done")))))
 
 (defun gnus-summary-expire-articles-now ()
@@ -9105,6 +9139,12 @@
 	;; after all.
 	(unless (memq (car articles) not-deleted)
 	  (gnus-summary-mark-article (car articles) gnus-canceled-mark))
+	(let* ((article (car articles))
+	       (id (mail-header-id (gnus-data-header 
+				    (assoc article (gnus-data-list nil))))))
+	  (run-hook-with-args 'gnus-summary-article-delete-hook
+			      'delete id gnus-newsgroup-name nil
+			      nil))
 	(setq articles (cdr articles)))
       (when not-deleted
 	(gnus-message 4 "Couldn't delete articles %s" not-deleted)))

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

>> Could you do a unified diff?  I have a hard time reading raw
>> diffs...
>
> Sorry, I forgot the -u.  Unified diff attached, plus gnus-registry.el
> for good measure.

Ok; looks good to me...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Incoming messages seem to be spooled in several places, so I'm not
> sure how to deal with that.  I only want to intercept nnmail and
> nnimap for now, should I just prepend the hook call to their
> respective split-methods?

Are you saying that you haven't done respooling yet in your patch?
(If your patch does respooling, then I have to go back and read it
again.)

Yes, calling the spool hook just before the split function is called
(or just after) seems to be right.  Just be careful to abstain from
calling the hook on B q or B t.
-- 
A turnip curses Elvis

Re: message registry for Gnus

[Ted Zlatanov]

On Sun, 02 Feb 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Incoming messages seem to be spooled in several places, so I'm not
>> sure how to deal with that.  I only want to intercept nnmail and
>> nnimap for now, should I just prepend the hook call to their
>> respective split-methods?
> 
> Are you saying that you haven't done respooling yet in your patch?
> (If your patch does respooling, then I have to go back and read it
> again.)

I *think* I do respooling.  I may be wrong.

> Yes, calling the spool hook just before the split function is called
> (or just after) seems to be right.  Just be careful to abstain from
> calling the hook on B q or B t.

I really wish someone familiar with nnmail.el and nnimap.el could help
me with this :)  The message insertion functionality is, to say the
least, pretty complicated...

Ted

Re: message registry for Gnus

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

>> Yes, calling the spool hook just before the split function is called
>> (or just after) seems to be right.  Just be careful to abstain from
>> calling the hook on B q or B t.
>
> I really wish someone familiar with nnmail.el and nnimap.el could help
> me with this :)  The message insertion functionality is, to say the
> least, pretty complicated...

I'm not sure exactly what the problem is, but I could try to help with
the nnimap issues.  nnimap-split-articles calls nnmail-cache-insert
for newly splitted articles, but I'm not sure that is related.

Re: message registry for Gnus

[Ted Zlatanov]

On Tue, 04 Feb 2003, jas@extundo.com wrote:
> I'm not sure exactly what the problem is, but I could try to help
> with the nnimap issues.  nnimap-split-articles calls
> nnmail-cache-insert for newly splitted articles, but I'm not sure
> that is related.

I just need a hook added at the right place that would call

(gnus-summary-article-spool-hook 'spool article-id new-group) 

after the message is inserted.  Would nnmail-cache-insert be the right
place for both nnmail and nnimap, since it's used by both?

Ted

Re: message registry for Gnus

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 04 Feb 2003, jas@extundo.com wrote:
>> I'm not sure exactly what the problem is, but I could try to help
>> with the nnimap issues.  nnimap-split-articles calls
>> nnmail-cache-insert for newly splitted articles, but I'm not sure
>> that is related.
>
> I just need a hook added at the right place that would call
>
> (gnus-summary-article-spool-hook 'spool article-id new-group) 
>
> after the message is inserted.  Would nnmail-cache-insert be the right
> place for both nnmail and nnimap, since it's used by both?

Looking at nnimap, it seems like it would work.  n-c-i is called
during splitting and in nnimap-request-accept-article which should
cover most normal methods to insert articles in groups.  (Although
expiry-target's are probably not covered.)

The nnmail-cache-accepted-message-ids variable must be set though, but
maybe with your code that variable could default to t (unless that
changes something else too).

Re: message registry for Gnus

[Ted Zlatanov]

On Wed, 05 Feb 2003, jas@extundo.com wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>> after the message is inserted.  Would nnmail-cache-insert be the
>> right place for both nnmail and nnimap, since it's used by both?
> 
> Looking at nnimap, it seems like it would work.  n-c-i is called
> during splitting and in nnimap-request-accept-article which should
> cover most normal methods to insert articles in groups.  (Although
> expiry-target's are probably not covered.)

Expiry is covered by gnus-summary-article-expire-hook, I think.

> The nnmail-cache-accepted-message-ids variable must be set though,
> but maybe with your code that variable could default to t (unless
> that changes something else too).

I'm not sure how and where to do this.  See my modification of
nnmail-cache-insert: it's just a hook call at the beginning.  Is that
OK?

Ted

Re: message registry for Gnus

[Raja R Harinath]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> In the context of spam.el, the registry will be used to track when a
>> message changes groups, or is processed by a spam/ham processor.  That
>> way, we can see if a message that will be registered as spam with
>> ifile now has been registered as ham with Bogofilter, for instance,
>> and reverse the Bogofilter registration before registering with ifile.
>
> Sure, sounds like a good idea.

Deja vu :-)  See

  http://article.gmane.org/gmane.emacs.gnus.general/8065

for some of the possible uses for the registry.  I'm sure the actual
mechanics suggested there are somewhat bogus.

- Hari
-- 
Raja R Harinath ------------------------------ harinath@cs.umn.edu

Re: message registry for Gnus

[Ted Zlatanov]

On Sat, 01 Feb 2003, harinath@cs.umn.edu wrote:
> See
> 
>   http://article.gmane.org/gmane.emacs.gnus.general/8065
> 
> for some of the possible uses for the registry.  I'm sure the actual
> mechanics suggested there are somewhat bogus.

Wow, you posted that in 1996?  You've been slacking :)

(I'm quoting from the web site, sorry for the weird quoting)

Hari wrote:
| I wonder if using a single message-id hashtb with the other facilities
| being implemented as property get/sets will be a better alternative.         
| This would require a major rearchitecting of Gnus.                           

I think the registry should coexist with everything else.  I want it
to be as flexible as possible.  The properties you mention can be
associated with a message ID through a registry API, sure, but so can
any other properties.

As for the properties themselves (with the understanding that those
would just hook into the registry, without being integral to its
existence):

:seen is not needed as you state, is it?  We only need to store the
actual article number and full group name associated with the message
ID in a property that's a list each time we see the message ID
spooled, copied, or moved.  If there are more than one entries in the
list, the article is duplicated somewhere.  This is sufficient to find
any article's copies by message ID.

:refs-to and :has-refs-from seem to be better off in a separate table
(if articles are nodes in a graph and references are edges - for a
generic graph it's easier to store the edges in a separate table if I
remember my database design)

:where - see :seen above, probably not necessary as we can find where
an article has been copied or moved easily

:gl-prediction and :gl-rating are useful properties in the context of
the GroupLens system, but I think we should move away from the
GroupLens name :)  It's just another kind of scoring, so let's call it
distributed scoring or something like that (the manual talks about a
different kind of distributed scoring with remote score files, but
that's not that different from GroupLans after all).

Ted

Re: message registry for Gnus

[Raja R Harinath]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 01 Feb 2003, harinath@cs.umn.edu wrote:
>> See
>> 
>>   http://article.gmane.org/gmane.emacs.gnus.general/8065
>> 
>> for some of the possible uses for the registry.  I'm sure the actual
>> mechanics suggested there are somewhat bogus.
>
> Wow, you posted that in 1996?  You've been slacking :)

[blush]

But as Lars said then, the set of message-ids covered were somewhat
disjoint.  If we now have a registry of all mail message-ids, then
it's more appropriate now.

> :seen is not needed as you state, is it?  We only need to store the
> actual article number and full group name associated with the message
> ID in a property that's a list each time we see the message ID
> spooled, copied, or moved.  If there are more than one entries in the
> list, the article is duplicated somewhere.  This is sufficient to find
> any article's copies by message ID.

Yep.  However, the idea was for an in-memory data structure only.
So, I wanted a doubly-linked-list to get at all the seen msgids to
save as ~/News/suppression.  Anyway, I didn't say the design was good :-)

> :refs-to and :has-refs-from seem to be better off in a separate table
> (if articles are nodes in a graph and references are edges - for a
> generic graph it's easier to store the edges in a separate table if I
> remember my database design)

This is the 'adjacency list' data structure -- good in-memory, and not
too shabby for saving on-disk, esp. in a custom registry.

- Hari
-- 
Raja R Harinath ------------------------------ harinath@cs.umn.edu

Re: message registry for Gnus

[Ted Zlatanov]

The registry is in place.  Basically, I added some hooks to
gnus-sum.el and nnmail.el and I think they will do the right thing.

The only thing that does not seem to work properly is that in
nnmail-spool-hook, this code for group "mail":

(gnus-group-full-name group (gnus-find-method-for-group group))

will find "nnml+private:mail" even though the spooling is being done
for "nnimap+server.name.com:mail".  This code is used by
gnus-group-guess-full-name in gnus.el, and that's the function I want
to fix.  See gnus-registry.el for an example of what the functions
produce.

Anyone with ideas?  Once the spooling hook works, I can go on to the
actual data registry...

Thanks
Ted

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> (gnus-group-full-name group (gnus-find-method-for-group group))
>
> will find "nnml+private:mail" even though the spooling is being done
> for "nnimap+server.name.com:mail".

`gnus-find-method-for-group' only works (reliably) for Gnus groups
names, and not for server group names.  That is, it works for
prefixed group names, which is what you're trying to find.  :-)

So you can't use that in the back end.  You have to do it the other
way around, and feed `gnus-group-prefixed-name' the select method in
some manner.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

On Fri, 07 Feb 2003, larsi@gnus.org wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> (gnus-group-full-name group (gnus-find-method-for-group group))
>>
>> will find "nnml+private:mail" even though the spooling is being
>> done for "nnimap+server.name.com:mail".
> 
> `gnus-find-method-for-group' only works (reliably) for Gnus groups
> names, and not for server group names.  That is, it works for
> prefixed group names, which is what you're trying to find.  :-)

Argh.

> So you can't use that in the back end.  You have to do it the other
> way around, and feed `gnus-group-prefixed-name' the select method in
> some manner.

I'm not sure how to get the select method though.  I thought it would
be gnus-current-select-method, but that's always the nnml method, even
when nnimap splitting is done!  I think there's something with the
binding of that variable, and my Lisp is hardly good enough to figure
out those closures or whatever is going on.

So instead I brute-forced gnus-request-scan to set the new variable
gnus-internal-registry-spool-current-method to gnus-command-method
before a request-scan operation, and then gnus-registry.el uses that
variable.  It works very nicely, and I'm happy with the registry now,
but please take a look and let me know if I'm missing something
obvious as usual.

I named the variable gnus-internal-registry-spool-current-method, I
hope that's OK with everyone.  It's certainly long enough that you
can't miss it :)

Ted

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I named the variable gnus-internal-registry-spool-current-method, I
> hope that's OK with everyone. 

It's OK by me.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

I'm not sure how to deal with this issue.  The (gnus-data-header)
information about an article from gnus-data-list is available for the
registry when I catch respooling, moving, copying, expiring, and
deletion of articles.  Spooling, however, only gives me the article's ID:

(from nnmail.el)

(defun nnmail-cache-insert (id grp)
  (run-hook-with-args 'nnmail-spool-hook 
		      id grp)
...


Is the data header for the article available somehow at the point of
cache insertion in nnmail.el?  I would really like to have that header
available for message analysis at that moment, not later when the
article is copied/moved/etc, so the registry interface is consistent.

Also, to save a hashtable, it seems like I have to convert it to an
alist and put that in a file, and convert the other way when I want to
load from a file.  Is there an easier way?

Thanks
Ted

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Is the data header for the article available somehow at the point of
> cache insertion in nnmail.el?

No, the header data is being built by Gnus upon group entry.  When
nnmail is snarfing mail, it has no such structure.

> Also, to save a hashtable, it seems like I have to convert it to an
> alist and put that in a file, and convert the other way when I want to
> load from a file.  Is there an easier way?

No, that's the way to do it.  :-)  Hash tables (well, obarrays,
really) don't (to use a Javaish term) serialize very well.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

On Sat, 22 Feb 2003, larsi@gnus.org wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Is the data header for the article available somehow at the point
>> of cache insertion in nnmail.el?
> 
> No, the header data is being built by Gnus upon group entry.  When
> nnmail is snarfing mail, it has no such structure.

Hrm.  Can I put the spool hook at a later point, just after the header
data is available?

>> Also, to save a hashtable, it seems like I have to convert it to an
>> alist and put that in a file, and convert the other way when I want
>> to load from a file.  Is there an easier way?
> 
> No, that's the way to do it.  :-)  Hash tables (well, obarrays,
> really) don't (to use a Javaish term) serialize very well.

Could something like hashtable-to-alist-string and
hashtable-from-alist-string be added to the Gnus core?  It would make
my life easier, and I'm definitely not the person to write reliable
versions of these functions :)  I think these functions could probably
be useful in other places as well; I looked but found no such
functionality already in place.

Serialization would be a wonderful feature for Emacs Lisp in general.

Thanks!
Ted

Re: message registry for Gnus

[Andreas Fuchs]

Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Could something like hashtable-to-alist-string and
> hashtable-from-alist-string be added to the Gnus core?  It would make
> my life easier, and I'm definitely not the person to write reliable
> versions of these functions :)  

Me neither, but will these help:

(defun alist-to-hashtable (alist)
  "Build a hashtable from the values in ALIST."
  (let ((ht (make-hash-table)))
    (mapc
     (lambda (kv-pair)
       (puthash (car kv-pair) (cdr kv-pair) ht))
     alist)
     ht))

(defun hashtable-to-alist (hash)
  "Build an alist from the values in HASH."
  (let ((list nil))
    (maphash
     (lambda (key value)
       (setq list (cons (cons key value) list))))))

;;? You will of course have to use (read) and (format) accordingly. (-:

> Serialization would be a wonderful feature for Emacs Lisp in general.

Absolutely.

Have fun,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

gnus-registry: alist-to-hashtable, hashtable-to-alist (was: message registry for Gnus)

[Reiner Steib]

[ Digging up a very old message.  Therefore Cc-ing involved people.... ]

On Mon, Feb 24 2003, Andreas Fuchs wrote:

> Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Could something like hashtable-to-alist-string and
>> hashtable-from-alist-string be added to the Gnus core?  It would make
>> my life easier, and I'm definitely not the person to write reliable
>> versions of these functions :)  
>
> Me neither, but will these help:
^^^^^^^^^^^^ ???

Who actually wrote these functions?  ChangeLog.2 says:

,----[ ChangeLog.2 ]
| 2003-03-28  Teodor Zlatanov  <tzz@lifelogs.com>
| 
| 	[...]
| 
| 	* gnus-registry.el (gnus-registry-translate-to-alist)
| 	(gnus-registry-translate-from-alist, alist-to-hashtable)
| 	(hashtable-to-alist): new functions
| 	(gnus-register-spool-action): add a spool item to the registry
`----

If Andreas wrote them, it should have been documented it in the
ChangeLog as follows:

,----
| 2003-03-28  Andreas Fuchs  <asf@void.at>
| 
| 	* gnus-registry.el (alist-to-hashtable, hashtable-to-alist): New
| 	functions.
`----

(Andreas has papers on file, so no need to think about "tiny change").
If Andreas didn't write them, we need to know where they came from.

We really must be more careful about these issues!

Later these were moved to gnus-util.el and Romain added the following
comment:

,----[ gnus-util.el ]
| ;; The following two functions are used in gnus-registry.
| ;; They were contributed by Andreas Fuchs <asf@void.at>.
`----

,----[ ChangeLog ]
| 2006-04-13  Romain Francoise  <romain@orebokech.com>
| 
| 	* gnus-util.el (gnus-alist-to-hashtable, gnus-hashtable-to-alist):
| 	Moved here (and renamed) from gnus-registry.el.
| 
| 	* gnus-registry.el: Require gnus-util.
| 	Use `gnus-alist-to-hashtable' and `gnus-hashtable-to-alist'.
`----

[ I'd like to put this change in v5-10 as well, because Gnus functions
  should use the "gnus-" prefix. ]

> (defun alist-to-hashtable (alist)
>   "Build a hashtable from the values in ALIST."
>   (let ((ht (make-hash-table)))
>     (mapc
>      (lambda (kv-pair)
>        (puthash (car kv-pair) (cdr kv-pair) ht))
>      alist)
>      ht))
>
> (defun hashtable-to-alist (hash)
>   "Build an alist from the values in HASH."
>   (let ((list nil))
>     (maphash
>      (lambda (key value)
>        (setq list (cons (cons key value) list))))))

[ The current code in gnus-utils.el looks slightly different (see
  rev. 6.7): ]

--8<---------------cut here---------------start------------->8---
@@ -1,16 +1,21 @@
-(defun alist-to-hashtable (alist)
+;; The following two functions are used in gnus-registry.
+;; They were contributed by Andreas Fuchs <asf@void.at>.
+(defun gnus-alist-to-hashtable (alist)
   "Build a hashtable from the values in ALIST."
-  (let ((ht (make-hash-table)))
+  (let ((ht (make-hash-table
+            :size 4096
+            :test 'equal)))
     (mapc
      (lambda (kv-pair)
        (puthash (car kv-pair) (cdr kv-pair) ht))
      alist)
      ht))
 
-(defun hashtable-to-alist (hash)
+(defun gnus-hashtable-to-alist (hash)
   "Build an alist from the values in HASH."
   (let ((list nil))
     (maphash
      (lambda (key value)
-       (setq list (cons (cons key value) list))))))
-
+       (setq list (cons (cons key value) list)))
+     hash)
+    list))
--8<---------------cut here---------------end--------------->8---

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: gnus-registry: alist-to-hashtable, hashtable-to-alist

[Ted Zlatanov]

On 28 Sep 2006, reinersteib+gmane@imap.cc wrote:

> [ Digging up a very old message.  Therefore Cc-ing involved people.... ]
>
> On Mon, Feb 24 2003, Andreas Fuchs wrote:
>
>> Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> Could something like hashtable-to-alist-string and
>>> hashtable-from-alist-string be added to the Gnus core?  It would make
>>> my life easier, and I'm definitely not the person to write reliable
>>> versions of these functions :)  
>>
>> Me neither, but will these help:
> ^^^^^^^^^^^^ ???
>
> Who actually wrote these functions?  ChangeLog.2 says:
>
> ,----[ ChangeLog.2 ]
> | 2003-03-28  Teodor Zlatanov  <tzz@lifelogs.com>
> | 
> | 	[...]
> | 
> | 	* gnus-registry.el (gnus-registry-translate-to-alist)
> | 	(gnus-registry-translate-from-alist, alist-to-hashtable)
> | 	(hashtable-to-alist): new functions
> | 	(gnus-register-spool-action): add a spool item to the registry
> `----
>
> If Andreas wrote them, it should have been documented it in the
> ChangeLog as follows:
>
> ,----
> | 2003-03-28  Andreas Fuchs  <asf@void.at>
> | 
> | * gnus-registry.el (alist-to-hashtable, hashtable-to-alist): New
> | 	functions.
> `----
>
> (Andreas has papers on file, so no need to think about "tiny change").
> If Andreas didn't write them, we need to know where they came from.
>
> We really must be more careful about these issues!

I guess it was my mistake.  Sorry.

> Later these were moved to gnus-util.el and Romain added the following
> comment:
>
> ,----[ gnus-util.el ]
> | ;; The following two functions are used in gnus-registry.
> | ;; They were contributed by Andreas Fuchs <asf@void.at>.
> `----
>
> ,----[ ChangeLog ]
> | 2006-04-13  Romain Francoise  <romain@orebokech.com>
> | 
> | * gnus-util.el (gnus-alist-to-hashtable, gnus-hashtable-to-alist):
> | 	Moved here (and renamed) from gnus-registry.el.
> |
> | 	* gnus-registry.el: Require gnus-util.
> | 	Use `gnus-alist-to-hashtable' and `gnus-hashtable-to-alist'.
> `----
>
> [ I'd like to put this change in v5-10 as well, because Gnus functions
> should use the "gnus-" prefix. ]
>
>> (defun alist-to-hashtable (alist)
>> "Build a hashtable from the values in ALIST."
>> (let ((ht (make-hash-table)))
>> (mapc
>> (lambda (kv-pair)
>> (puthash (car kv-pair) (cdr kv-pair) ht))
>> alist)
>> ht))
>>
>> (defun hashtable-to-alist (hash)
>> "Build an alist from the values in HASH."
>> (let ((list nil))
>> (maphash
>> (lambda (key value)
>> (setq list (cons (cons key value) list))))))
>
> [ The current code in gnus-utils.el looks slightly different (see
> rev. 6.7): ]
> @@ -1,16 +1,21 @@
> -(defun alist-to-hashtable (alist)
> +;; The following two functions are used in gnus-registry.
> +;; They were contributed by Andreas Fuchs <asf@void.at>.
> +(defun gnus-alist-to-hashtable (alist)
> "Build a hashtable from the values in ALIST."
> -  (let ((ht (make-hash-table)))
> +  (let ((ht (make-hash-table
> +            :size 4096
> +            :test 'equal)))
> (mapc
> (lambda (kv-pair)
> (puthash (car kv-pair) (cdr kv-pair) ht))
> alist)
> ht))
>
> -(defun hashtable-to-alist (hash)
> +(defun gnus-hashtable-to-alist (hash)
> "Build an alist from the values in HASH."
> (let ((list nil))
> (maphash
> (lambda (key value)
> -       (setq list (cons (cons key value) list))))))
> -
> +       (setq list (cons (cons key value) list)))
> +     hash)
> +    list))

The newer version of gnus-hashtable-to-alist is correct.  The older
version is, I think, broken (maphash is called with a nil argument).

gnus-alist-to-hashtable is also better in the newer version.

I'm OK with all these changes, backported into 5.10 as well.  I
intended to revise and document gnus-registry.el in addition to
working on spam.el again, but my second child born 3 months ago has
made my free time very scarce.  If someone wants to pick up
maintenance of either gnus-registry.el or spam.el, they are welcome to
it--otherwise I'll continue to support them as time allows.

Thanks
Ted

Re: gnus-registry: alist-to-hashtable, hashtable-to-alist

[CHENG Gao]

I trust Gnus hashtb<->alist conversion functions have no problem.

I suspected function gnus-gnus-to-quick-newsrc-format is the culprit.

As Handa said, # (blah blah ...) is representation of string with text
properties.
I had a test by removing all # and leave "0 40 (auto-composed nil)", and
gnus registry works.

In gnus-start.el function gnus-gnus-to-quick-newsrc-format line 2841
,----
|            (print-readably t)
`----
I juct checked. At least in my Emacs 23 (06-9-15 cvs), variable
print-readably does not exist.

I nearly damaged my brain by reading code of prin1/princ/print.

At last I think gnus-gnus-to-quick-newsrc-format should be ok. It does
what it should do, even the result is bad.

A silly question: 
If an entry in gnus-reristry-alist carries text-property, prin1 will
print it as string and use # as quoting character. Is it right?

If so, I think we need check all puthash (to gnus-registry-hashtb) in
gnus-registry.el for unremoved text property.

My knowledge is not adquate for real useful finding. Wish what I found
can be a little useful to you.

Re: gnus-registry: alist-to-hashtable, hashtable-to-alist

[Reiner Steib]

On Thu, Sep 28 2006, CHENG Gao wrote:

> I trust Gnus hashtb<->alist conversion functions have no problem.

Yes.  This thread is completely unrelated to your problem.

It would make it easier for people to track the discussion about your
problem, if you stay in the previous threads "gnus-registry.el on
emacs-unicode-2" or "gnus-registry text property removal question".
Thanks.

I will followup to your message there.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: message registry for Gnus

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Hrm.  Can I put the spool hook at a later point, just after the header
> data is available?

It might never be available -- you might just be calling nnmail to
fetch mail without even starting Gnus.  In theory.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: message registry for Gnus

[Ted Zlatanov]

On Mon, 24 Feb 2003, larsi@gnus.org wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Hrm.  Can I put the spool hook at a later point, just after the
>> header data is available?
> 
> It might never be available -- you might just be calling nnmail to
> fetch mail without even starting Gnus.  In theory.  :-)

OK, so where can I insert a hook for the registry, if all I care for
is the first time a message is seen in a group, with the article's
header data available?  I would prefer to avoid the braindead solution
of scanning all articles in a newsgroup and noticing the new ones
every time - although that might not be too hard if I look at the
"unseen" articles.

Ted

Re: message registry for Gnus

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK, so where can I insert a hook for the registry, if all I care for
> is the first time a message is seen in a group, with the article's
> header data available?  I would prefer to avoid the braindead solution
> of scanning all articles in a newsgroup and noticing the new ones
> every time - although that might not be too hard if I look at the
> "unseen" articles.

Do you want to see the article again after it has moved to another
group?  It might not be unseen anymore at that point.

-- 
A preposition is not a good thing to end a sentence with.

Re: message registry for Gnus

[Ted Zlatanov]

On Tue, 25 Feb 2003, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> OK, so where can I insert a hook for the registry, if all I care
>> for is the first time a message is seen in a group, with the
>> article's header data available?  I would prefer to avoid the
>> braindead solution of scanning all articles in a newsgroup and
>> noticing the new ones every time - although that might not be too
>> hard if I look at the "unseen" articles.
> 
> Do you want to see the article again after it has moved to another
> group?  It might not be unseen anymore at that point.

I can track the article at that point with the move hook - it's only
the first time I see it that I need to initialize its registry entry.

In other words, an article's life is like this:

spool -> group1 -> move -> group2 -> delete
          \-> copy -> group3 -> delete

I can track everything except the "spool -> group1" part.

The main reason why I want to register the article when it's in the
Gnus newsgroup data, rather than when it's spooled by nnmail, is that
at spool time the data in the article may be malformed.  For instance,
the message ID (which is crucial for tracking the message) may be
invalid when nnmail spools the article (cf. discussion on article
numbers yesterday on this list).  There's a better chance that I'll
get a "good" message ID if I grab it from the Gnus newsgroup data,
rather than parsing it out of the spooled message.

As far as the user is concerned, registering the article at spool time
vs. registering it at the time of summary entry is the same thing.  I
think.

Ted

Re: message registry for Gnus

[Ted Zlatanov]

I made a commit for the registry that shouldn't, but may impact
loading and saving of the newsrc.eld (a new variable is added, and
without loading gnus-registry.el nothing interesting should happen).
It worked OK for me.  I'll be on vacation for the next week, so if
there are issues, please back out my commit as I won't be able to.

Thanks
Ted

Re: message registry for Gnus

[Ted Zlatanov]

On Fri, 31 Jan 2003, tzz@lifelogs.com wrote:
> - track messages by ID
> 
> - associate a Lisp structure (probably a list, hashes are too new?)
>   with a message ID

This is working, in a first-draft-please-put-me-out-of-my-misery sort
of way.  I store message headers separately from message spool data.
Please take a look and let me know what you think (gnus-registry.el
with the definitions of gnus-registry-alist and
gnus-registry-headers-alist in gnus.el).  I'm pretty sure there's bugs
in the code.

> - store the registry in a separate file or in the newsrc.eld?

Right now, the registry is stored in the newsrc.eld.  This may be a
problem later - it will grow quickly.  We can either compress the
registry or move it to a separate file; removing old registry entries
would probably be OK for deleted messages but we have to be careful
with copies of a message (removal of a copy means we have to keep the
records for the original around).  Ideas welcome.

Ted