Registry is working (first cut)

Registry is working (first cut)

[Ted Zlatanov]

I consider this a bug-fixing patch to the registry, since it didn't
work at all well before.  I'll feel better knowing a semi-working
version is going into Gnus 5.10.

If you load gnus-registry.el, the following will happen:

- all messages will be tracked to the group where they are spooled,
  copied, or moved.  Deletions are also caught.  Multiple groups (for
  a copy) are OK, only the latest one is caught.

- when you enter a group, the message IDs of all the articles will be
  registered, UNLESS the article is already registered.  Thus,
  multiple copies will not be registered when you enter a group.

NOTE: I'd like to limit this to mail groups only, what's the
"canonical" way?  message-this-is-{news|mail} didn't seem to work when
I tested it.  In the meanwhile, you will get a lot of registered
messages from newsgroups...

- only the short version of the group name is saved, for many reasons

- the registry will be saved in your .newsrc.eld.  I can change that
  if people feel it's too large.  I optimized the format a bit, so it
  should be fast and compact now.

- gnus-registry-split-fancy-with-parent works exactly like
  nnmail-split-fancy-with-parent, just use 

(: gnus-registry-split-fancy-with-parent) 

  in your split-fancy rules.

- gnus-registry-clear can be invoked to clear the registry, if you
  need to.

Have fun with it...

Thanks
Ted

Re: Registry is working (first cut)

[Raymond Scholz]

* Ted Zlatanov <tzz@lifelogs.com> wrote:

> NOTE: I'd like to limit this to mail groups only, what's the
> "canonical" way?  message-this-is-{news|mail} didn't seem to work when
> I tested it.  In the meanwhile, you will get a lot of registered
> messages from newsgroups...

Uhm... Lots of, yes :-)

There is code to avoid registering groups matching "^nntp" in there.
But the comparison ist made against `gnus-newsgroup-name' which
doesn't carry this prefix if it is the primary select method.

> - only the short version of the group name is saved, for many reasons

I used to have some nnml and nnfolder groups with the same name...

> - the registry will be saved in your .newsrc.eld.  I can change that
>   if people feel it's too large.  I optimized the format a bit, so it
>   should be fast and compact now.

I think it would be useful to put in a separate file.

> - gnus-registry-split-fancy-with-parent works exactly like
>   nnmail-split-fancy-with-parent, just use 
>
> (: gnus-registry-split-fancy-with-parent) 
>
>   in your split-fancy rules.

Ah, this is cool.  I tried getting `nnmail-split-fancy-with-parent' to
work with nnimap but didn't succeed.

My old mail tool chain was like this: 

fetchmail-->procmail-->directory spool for Gnus (mailing lists)
                    |
                    +->/var/spool/mail-->nnmail-split-fancy-with-parent
                                         and some split rules

Now that I've converted to IMAP I have

fetchmail-->maildrop-->IMAP folders

And a nasty cludge with a custom X-Message-Source header indication
the folder where a message originates, telling maildrop on my desktop
the target folder when sorting mail written from my laptop.

I wan't to reestablish some sort of "splitting with parent".  This is
my current setup, which gives me the backtrace below:

(require 'gnus-registry)
(setq nnimap-split-fancy
      '(|
;	(: nnmail-split-fancy-with-parent)
	(: gnus-registry-split-fancy-with-parent)
	"INBOX"))

(setq nnimap-split-rule 'nnimap-split-fancy)
(setq nnimap-split-inbox '("INBOX"))
(setq nnimap-split-predicate "UNDELETED")


Debugger entered--Lisp error: (wrong-type-argument stringp 108)
  string-match(108 "INBOX.mail.spam")
  #[(x) "Â\b	\"‡" [x word string-match] 3](108)
  mapcar(#[(x) "Â\b	\"‡" [x word string-match] 3] "list\\.")
  gnus-registry-grep-in-list("INBOX.mail.spam" "list\\.")
  #[(x) "Ä\b!†\a\0	\x11Å	\n\"„\x16\0Å	\v\"…\x19\0Ɖ\x11‡" [x res gnus-registry-unfollowed-groups nnmail-split-fancy-with-parent-ignore-groups gnus-registry-fetch-group gnus-registry-grep-in-list nil] 3]("<625501c3101a$5fc96a6a$41526e96@4m02043>")
  mapcar(#[(x) "Ä\b!†\a\0	\x11Å	\n\"„\x16\0Å	\v\"…\x19\0Ɖ\x11‡" [x res gnus-registry-unfollowed-groups nnmail-split-fancy-with-parent-ignore-groups gnus-registry-fetch-group gnus-registry-grep-in-list nil] 3] ("<625501c3101a$5fc96a6a$41526e96@4m02043>"))
  gnus-registry-split-fancy-with-parent()
  eval((gnus-registry-split-fancy-with-parent))
  nnmail-split-it((: gnus-registry-split-fancy-with-parent))
  nnmail-split-it((| (: gnus-registry-split-fancy-with-parent) "INBOX"))
  nnmail-split-fancy()
  nnimap-split-fancy()
  nnimap-split-to-groups(nnimap-split-fancy)
  nnimap-split-articles(nil "")
  nnimap-request-scan(nil "")
  gnus-request-scan(nil (nnimap "" (nnimap-stream network) (nnimap-server-port 143) (nnimap-expunge-on-close always) (nnimap-list-pattern "INBOX*") (nnimap-address "mail")))
  gnus-read-active-file-1((nnimap "" (nnimap-stream network) (nnimap-server-port 143) (nnimap-expunge-on-close always) (nnimap-list-pattern "INBOX*") (nnimap-address "mail")) nil)
  gnus-read-active-file()
  gnus-group-get-new-news(nil)
  call-interactively(gnus-group-get-new-news)

Cheers, Ray
-- 
The only agents that insert X-Priority: high headers are 1) Outlook and
2) spammers.  So X-Priority is a pretty reliable header to use to filter
out spam and other annoyances.       (Lars M. Ingebrigtsen in gnus.ding)

Re: Registry is working (first cut)

[Reiner Steib]

On Sun, May 04 2003, Raymond Scholz wrote:

> * Ted Zlatanov <tzz@lifelogs.com> wrote:
>> NOTE: I'd like to limit this to mail groups only, what's the
>> "canonical" way?  message-this-is-{news|mail} didn't seem to work when
>> I tested it.  

AFAICS, `message-this-is-news' is set explicitely when calling
functions as `message-followup' or `message-news*'.

> There is code to avoid registering groups matching "^nntp" in there.
> But the comparison ist made against `gnus-newsgroup-name' which
> doesn't carry this prefix if it is the primary select method.

I didn't look at the code.  But wouldn't it make more sense to use a
group parameter or a local variable (local to the current summary
buffer)?

And/or make a new function `gnus-newsgroup-news-p' which checks
something like (car (gnus-find-method-for-group gnus-newsgroup-name)).

>> - only the short version of the group name is saved, for many reasons
>
> I used to have some nnml and nnfolder groups with the same name...

Me too!!!1  E.g., I have INBOX and Spam groups on various
select-methods.  And also the same groups names for Gcc'ing and for
`o' (`gnus-default-article-saver') from nntp groups.

>> - the registry will be saved in your .newsrc.eld.  I can change that
>>   if people feel it's too large. [...]
> I think it would be useful to put in a separate file.

ACK.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Registry is working (first cut)

[Ted Zlatanov]

On Sun, 04 May 2003, 4.uce.03.r.s@nurfuerspam.de wrote:
> I didn't look at the code.  But wouldn't it make more sense to use a
> group parameter or a local variable (local to the current summary
> buffer)?

Any suggestions for the group parameter name?  gnus-registry-ignore?

> And/or make a new function `gnus-newsgroup-news-p' which checks
> something like (car (gnus-find-method-for-group
> gnus-newsgroup-name)).

This is a good idea.  But there's backends like nnrss, which are "sort
of" news.  Maybe the backend itself should declare if it's news?

>>> - only the short version of the group name is saved, for many
>>>   reasons
>>
>> I used to have some nnml and nnfolder groups with the same name...
> 
> Me too!!!1  E.g., I have INBOX and Spam groups on various
> select-methods.  And also the same groups names for Gcc'ing and for
> `o' (`gnus-default-article-saver') from nntp groups.

Well, consider what happens when splitting.  Unless you moved a
message from nnimap:Spam to nnml:Spam, returning "Spam" in the
nnimap-split-fancy or nnmail-split-fancy is the right thing to do and
will send the message to the place it was seen last.  I know this
should work properly, but I just can't obtain the long group name
reliably on spool/copy/move/delete/etc. actions.

>>> - the registry will be saved in your .newsrc.eld.  I can change
>>>   that if people feel it's too large. [...]
>> I think it would be useful to put in a separate file.
> 
> ACK.

Er, is that "ack, I hate it" or "ack, but it must be done"?  Please
try to say what you mean, not what you feel :)

The registry being a part of the newsrc.eld is a bit too much IMHO.
It grows quickly, and it needs to be rsynced/stored in nnimap (see
prev. article about this) separately from the newsrc.eld.

Ted

Re: Registry is working (first cut)

[Karl Pflästerer]

On  8 May 2003, Ted Zlatanov <- tzz@lifelogs.com wrote:

> The registry being a part of the newsrc.eld is a bit too much IMHO.
> It grows quickly, and it needs to be rsynced/stored in nnimap (see
> prev. article about this) separately from the newsrc.eld.

IMO it should be separate. There is yet to much stuff in newsrc.eld (I
first make a copy, cross my fingers and then try to edit it if I have to
do some troubleshooting).

bye
   KP

-- 
Männer der Wissenschaft! Man sagt ihr viele nach, 
aber die meisten mit Unrecht.  
                             Karl Kraus 'Aphorismen'

Modification to newsrc saving, please check! (was: Registry is working (first cut))

[Ted Zlatanov]

On Thu, 08 May 2003, sigurd@12move.de wrote:
> On  8 May 2003, Ted Zlatanov <- tzz@lifelogs.com wrote:
> 
>> The registry being a part of the newsrc.eld is a bit too much IMHO.
>> It grows quickly, and it needs to be rsynced/stored in nnimap (see
>> prev. article about this) separately from the newsrc.eld.
> 
> IMO it should be separate. There is yet to much stuff in newsrc.eld
> (I first make a copy, cross my fingers and then try to edit it if I
> have to do some troubleshooting).

The attached patch allows gnus-registry.el to save its
gnus-registry-alist in a separate file.

***

I tried to reuse as much as possible of the gnus-start.el code, but
that meant I had to modify gnus-gnus-to-quick-newsrc-format a bit.
Because I don't want to corrupt everyone's newsrc.eld file, please
check the attached patch and let me know if you see any problems with
it.

***

It would be nice if the gnus-save-newsrc-file function was more
modular, so I could save the registry cache through it instead of
essentially copying it into gnus-registry-cache-save.  The idea of
saving variables to a file in a way that would let us just (load) that
file is very convenient.  We should facilitate it as much as possible
for other areas of Gnus.

I won't commit the patch for a day or so, to make sure everyone is OK
with it.

Ted

Re: Modification to newsrc saving, please check!

[Ted Zlatanov]

On Thu, 08 May 2003, tzz@lifelogs.com wrote:
> The attached patch allows gnus-registry.el to save its
> gnus-registry-alist in a separate file.
> 
> ***
> 
> I tried to reuse as much as possible of the gnus-start.el code, but
> that meant I had to modify gnus-gnus-to-quick-newsrc-format a bit.
> Because I don't want to corrupt everyone's newsrc.eld file, please
> check the attached patch and let me know if you see any problems
> with it.
> 
> ***
> 
> It would be nice if the gnus-save-newsrc-file function was more
> modular, so I could save the registry cache through it instead of
> essentially copying it into gnus-registry-cache-save.  The idea of
> saving variables to a file in a way that would let us just (load)
> that file is very convenient.  We should facilitate it as much as
> possible for other areas of Gnus.
> 
> I won't commit the patch for a day or so, to make sure everyone is
> OK with it.

I'm committing this to CVS, please be aware and back up your
.newsrc.eld files just in case.  It has worked fine for me.

Ted

Re: Modification to newsrc saving, please check!

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It would be nice if the gnus-save-newsrc-file function was more
> modular, so I could save the registry cache through it instead of
> essentially copying it into gnus-registry-cache-save.  The idea of
> saving variables to a file in a way that would let us just (load) that
> file is very convenient.  We should facilitate it as much as possible
> for other areas of Gnus.

Sounds like a plan.  So how about making it more modular?  Obvious
parameters that I see is the way that the contents of the buffer/file
are generated, and also filename and coding and suchlike.

-- 
This line is not blank.

Re: Modification to newsrc saving, please check!

[Ted Zlatanov]

On Sun, 18 May 2003, kai.grossjohann@gmx.net wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> It would be nice if the gnus-save-newsrc-file function was more
>> modular, so I could save the registry cache through it instead of
>> essentially copying it into gnus-registry-cache-save.  The idea of
>> saving variables to a file in a way that would let us just (load)
>> that file is very convenient.  We should facilitate it as much as
>> possible for other areas of Gnus.
> 
> Sounds like a plan.  So how about making it more modular?  Obvious
> parameters that I see is the way that the contents of the
> buffer/file are generated, and also filename and coding and
> suchlike.

I'll think about it, maybe an association table of symbols with file
names and nil means the regular place.

Specifically for the registry, however, I may have to change the
format yet again.  Saving the whole registry with every update is
pretty inefficient.  I'm thinking of your format for the nnmail cache,
with the extra data stored as a Lisp expression at the end of the
line.  Something like:

Message-ID group1 group2 group3 '((creation-time 10000000))

It's much faster to use this, but of course more work is needed to
work around the limitations of the format.  Argh.  If only Emacs had a
built-in file database.

Ted

Re: Modification to newsrc saving, please check!

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's much faster to use this, but of course more work is needed to
> work around the limitations of the format.  Argh.  If only Emacs had a
> built-in file database.

That would be cool indeed!

-- 
This line is not blank.

Re: Registry is working (first cut)

[Ted Zlatanov]

On Sun, 04 May 2003, ray-2003@zonix.de wrote:
> There is code to avoid registering groups matching "^nntp" in there.
> But the comparison ist made against `gnus-newsgroup-name' which
> doesn't carry this prefix if it is the primary select method.

Yes, it's not a good method.  I think I'll use group parameters, as
suggested by Reiner.

>> - only the short version of the group name is saved, for many
>>   reasons
> 
> I used to have some nnml and nnfolder groups with the same name...

Yes, I would like to use the long names, but I haven't found a
reliable way to capture that.  If anyone can help me obtain the
correct long name for a group, I'm all ears.  I kept getting the wrong
backend on things like drafts, article moves/copies, spooling, etc.

>> - the registry will be saved in your .newsrc.eld.  I can change
>>   that if people feel it's too large.  I optimized the format a
>>   bit, so it should be fast and compact now.
> 
> I think it would be useful to put in a separate file.

I can separate into a file, any suggestions for the name?

I will try to get nnimap storage of files to work, so that you can
store the registry in an article on a nnimap server.  That would be
useful.  I sort of left that project on the side for a little bit, but
it's time to get it working so people don't have to use rsync to
synchronize their registry.

> I wan't to reestablish some sort of "splitting with parent".  This
> is my current setup, which gives me the backtrace below:
> 
> (require 'gnus-registry)
> (setq nnimap-split-fancy
>       '(|
> ;	(: nnmail-split-fancy-with-parent)
> 	(: gnus-registry-split-fancy-with-parent)
> 	"INBOX"))
> 
> (setq nnimap-split-rule 'nnimap-split-fancy)
> (setq nnimap-split-inbox '("INBOX"))
> (setq nnimap-split-predicate "UNDELETED")

I see the problem, it's nnmail-split-fancy-with-parent-ignore-groups,
which can be a single regexp.  I only treated it as a list.  Should be
OK now, I put a change in CVS.

Thanks
Ted

Re: Registry is working (first cut)

[Ted Zlatanov]

On Thu, 08 May 2003, tzz@lifelogs.com wrote:
> On Sun, 04 May 2003, ray-2003@zonix.de wrote:
>> There is code to avoid registering groups matching "^nntp" in
>> there.  But the comparison ist made against `gnus-newsgroup-name'
>> which doesn't carry this prefix if it is the primary select method.
> 
> Yes, it's not a good method.  I think I'll use group parameters, as
> suggested by Reiner.

The registry will now use either the registry-ignore group/topic
boolean parameter, or the global gnus-registry-ignored-groups
variable.  Both associate nil (the default) with registry processing,
and anything else with registry ignoring.  The old
gnus-registry-unregistered-group-regex variable is gone.

Ted

Re: Registry is working (first cut)

[Raymond Scholz]

* Ted Zlatanov <tzz@lifelogs.com> wrote:

> Yes, I would like to use the long names, but I haven't found a
> reliable way to capture that.  If anyone can help me obtain the
> correct long name for a group, I'm all ears.  I kept getting the wrong
> backend on things like drafts, article moves/copies, spooling, etc.

Sorry, I've no idea.  There's `gnus-newsgroup-name' and a function
`gnus-real-group-name'.  The goal is to get a "fully" qualified group
name, right?  `gnus-newsgroup-name' looks like that to me.

> I see the problem, it's nnmail-split-fancy-with-parent-ignore-groups,
> which can be a single regexp.  I only treated it as a list.  Should be
> OK now, I put a change in CVS.

Wow, this works.  God Ol' Fancy Splitting with Parent is back. Thanks
a lot!

More questions/ideas:

What will happen, if gnus-registry instructs the splitting function to
write to a non-writeable group like NNTP?  I just tried and the
message went into my INBOX.  This is OK but I didn't find any code in
gnus-registry.el doing this :-)

There will be an expiry process for the registry, right?

AFAICS, there's no way to limit the registry collecting message IDs
from certain groups.  This would help to reduce the size of the
registry.

Some "Gnus synchronisation" should be possible if the fully qualified
groupnames are stored in the registry.  If the registry could store
article marks (read, ticked etc.) one could carry the registry from
one place to another and get his Gnusae in sync.  Without the need of
common servers because syncing is done on base of message IDs.

Cheers, Ray
-- 
I have a local archive newsserver with groups with more than 50k messages in
it. When I try to enter those groups with Gnus I can have a cigarette before
Gnus is ready. -- Oh no, I can see the lawsuits ... "Gnus causes cancer".
                                (Frank Schmitt and Simon Josefsson on (ding))

Re: Registry is working (first cut)

[Raymond Scholz]

* Raymond Scholz <ray-2003@zonix.de> wrote:

> AFAICS, there's no way to limit the registry collecting message IDs
> from certain groups.  This would help to reduce the size of the
> registry.

Sorry, I haven't read you other messages carefully enough.

Putting (registry-ignore t) in the Topic Parameters of my news groups
topic works fine.

Cheers, Ray
-- 
Hmmm... those vi mugs /are/ kinda cool. though... how about a Gnus
barrel with the refcard printed on it. And an XEmacs super tanker to
go with it...                                (Robin S. Socha in geg)

Re: Registry is working (first cut)

[Ted Zlatanov]

On Sun, 11 May 2003, ray-2003@zonix.de wrote:
> * Ted Zlatanov <tzz@lifelogs.com> wrote:
> 
>> Yes, I would like to use the long names, but I haven't found a
>> reliable way to capture that.  If anyone can help me obtain the
>> correct long name for a group, I'm all ears.  I kept getting the
>> wrong backend on things like drafts, article moves/copies,
>> spooling, etc.
> 
> Sorry, I've no idea.  There's `gnus-newsgroup-name' and a function
> `gnus-real-group-name'.  The goal is to get a "fully" qualified
> group name, right?  `gnus-newsgroup-name' looks like that to me.

It doesn't work for moving, for instance.  You don't get the right
group name for either the source or the target group, I forget which.
Also with draft messages you get some really strange behavior.

> What will happen, if gnus-registry instructs the splitting function
> to write to a non-writeable group like NNTP?  I just tried and the
> message went into my INBOX.  This is OK but I didn't find any code
> in gnus-registry.el doing this :-)

Because the group name returned is the short name, the splitting
function will get a group name equivalent to the short name of the
NNTP group.  The nnmail/nnimap backend will then create that group, I
believe.

> There will be an expiry process for the registry, right?

Currently, you can only clear it.  The age of the entry is not
stored.  We can do one or both of:

- limit registry size and delete old entries when we go over the size

- delete entries when they are too old

We just need to store the creation date in the extra data field.

> AFAICS, there's no way to limit the registry collecting message IDs
> from certain groups.  This would help to reduce the size of the
> registry.

registry-ignore only works for the automatic registration that occurs
when you enter a group.  It's not applicable for the registration that
happens when you copy/move/delete/spool a message.  Should it?

> Some "Gnus synchronisation" should be possible if the fully
> qualified groupnames are stored in the registry.  If the registry
> could store article marks (read, ticked etc.) one could carry the
> registry from one place to another and get his Gnusae in sync.
> Without the need of common servers because syncing is done on base
> of message IDs.

Well, consider that external processes can add or delete messages to
the IMAP groups, for instance.  Marks can be set or unset by external
processes also.  It's not an easy problem.  I've been thinking about
it.

Furthermore, marks are per-group+article, while the registry is
per-MessageID.  The format of the registry would have to be modified
so instead of just the group name, we also store the marks for the
MessageID in the group entry, plus maybe the article number.

Ted

Re: Registry is working (first cut)

[Ted Zlatanov]

The registry now stores a modification time parameter in each entry.
It doesn't expire old ones yet, though.

Also, you need to set gnus-registry-install to t to load the registry
now.  This is so it doesn't get loaded automatically.  I think I did
it right as mentioned by Dave Love, unfortunately I'm in a rush and I
will be offline for a bit, so if there are problems feel free to patch
or undo my changes.

Thanks
Ted

Re: Registry is working (first cut)

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes, I would like to use the long names, but I haven't found a
> reliable way to capture that.  If anyone can help me obtain the
> correct long name for a group, I'm all ears.  I kept getting the wrong
> backend on things like drafts, article moves/copies, spooling, etc.

The old fancy-split-with-parent code finds out the backend that the
group belongs to.  Does that help at all?

Maybe the registry is called in other places, so that it has
different information.

-- 
This line is not blank.