nnima-split-with-parent

nnima-split-with-parent

[Ted Zlatanov]

I want nnimap-split-with-parent! :) 

I was looking at the nnmail-split-with-parent code, and I see that the
nnmail cache is used for duplicate detection *and* for splitting with
the parent.  So it's not easy to abstract the nnmail cache as it is to
arbitrary newsgroup names, we might get erroneous warnings about
duplicates.  I think the format for the "new" article-ID cache should
be

backend-name\tarticle-ID\tgroup-name

We can keep the current nnmail cache as well, but this format can
replace the old cache format with very little work.  Maybe we can copy
and fix up the new-cache from the old nnmail-cache if we're upgrading,
and otherwise always use the new-cache file?

We should separate the functionality into gnus-article-cache.el if
it's going to be generalized (or whatever the file is called).

Functions affected (new names are not necessarily final):

nnmail-cache-insert: needs to be generalized to
gnus-article-cache-insert, maybe figuring out the backend name instead
of having it passed in for minimum code changes?

nnmail-cache-fetch-group: needs to be generalized to
gnus-article-cache-fetch-group, and the logic altered a little to
allow for the backend name at beggining of line.  Like the insert
function, maybe it should figure out the current backend name instead
of having it passed in for minimum code changes?

nnmail-cache-id-exists-p: generalize to
gnus-article-cache-id-exists-p, unchanged otherwise

the other nnmail-cache-* functions should be easily migrated to
gnus-article-cache-*

I hope someone is interested in doing this or at least giving me some
feedback...

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I want nnimap-split-with-parent! :) 

Me, too.  A simple way to get it is to frob
nnmail-cache-primary-mail-backend so that it returns your nnimap
backend.  But this invalidates previous lines in .nnmail-cache.

> I was looking at the nnmail-split-with-parent code, and I see that the
> nnmail cache is used for duplicate detection *and* for splitting with
> the parent.  So it's not easy to abstract the nnmail cache as it is to
> arbitrary newsgroup names, we might get erroneous warnings about
> duplicates.  I think the format for the "new" article-ID cache should
> be
>
> backend-name\tarticle-ID\tgroup-name
>
> We can keep the current nnmail cache as well, but this format can
> replace the old cache format with very little work.  Maybe we can copy
> and fix up the new-cache from the old nnmail-cache if we're upgrading,
> and otherwise always use the new-cache file?

This is a very good plan.  I've been thinking hard to change the
.nnmail-cache format in an upwards-compatible way, but I think your
suggestion is better: replace the nnmail-cache implementation with a
new one that just reads and converts the old file, if necessary.

Just for reference, the upwards-compatible format change would be as
follows.  Right now, there are some lines with just msg-id in it.
Some have msg-id, then tab, then group name.  The change would be to
add lines with msg-id, then tab, then group name, then server name.

If the group name is given but the server is missing, then assume
it's the server returned by nnmail-cache-primary-mail-backend.

For new lines, we might want to always write the server name.  Write
"native" for the native server.

But this is just for reference, I think your solution is better.

> We should separate the functionality into gnus-article-cache.el if
> it's going to be generalized (or whatever the file is called).
>
> Functions affected (new names are not necessarily final):
>
> nnmail-cache-insert: needs to be generalized to
> gnus-article-cache-insert, maybe figuring out the backend name instead
> of having it passed in for minimum code changes?

nnmail-cache-insert already knows the group I think.  It can intuit
the server from there.  (It currently does I believe.)

Oh, it uses gnus-command-method instead.

> nnmail-cache-fetch-group: needs to be generalized to
> gnus-article-cache-fetch-group, and the logic altered a little to
> allow for the backend name at beggining of line.  Like the insert
> function, maybe it should figure out the current backend name instead
> of having it passed in for minimum code changes?

Yes.  gnus-command-method.

> nnmail-cache-id-exists-p: generalize to
> gnus-article-cache-id-exists-p, unchanged otherwise
>
> the other nnmail-cache-* functions should be easily migrated to
> gnus-article-cache-*
>
> I hope someone is interested in doing this or at least giving me some
> feedback...

I'm afraid I don't have time, but I'm happy that *you're* interested :-)
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnima-split-with-parent

[Ted Zlatanov]

On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
> For new lines, we might want to always write the server name.  Write
> "native" for the native server.

Why not use the full name given by gnus-command-method,
e.g. "nnml+private"?  What's special about "native"?  Am I
misunderstanding the terminology?

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
>> For new lines, we might want to always write the server name.  Write
>> "native" for the native server.
>
> Why not use the full name given by gnus-command-method,
> e.g. "nnml+private"?  What's special about "native"?  Am I
> misunderstanding the terminology?

Usually, fully qualified group names from the native server do not
have a prefix (see your group buffer).  So we have to have _some_
prefix.

It could be that it is good to use the normal prefixes for the native
server, too.  Good idea.  Then the strange `native' prefix wouldn't
be necessary.
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnimap-split-with-parent

[Ted Zlatanov]

On Sat, 07 Dec 2002, jan@rychter.com wrote:
>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com>:
> [...]
>  Ted> I think the format for the "new" article-ID cache should be
>  Ted> backend-name\tarticle-ID\tgroup-name
> 
> Do I understand correctly that when you write backend-name you mean
> backend+server (to support multiple IMAP servers, for instance)?

Yes.  The fully qualified group name.  This can be a problem if the
user sets up a backend as secondary, then makes it the primary
backend.  It should be documented that the user can't expect the cache
to work in that case unless they edit it themselves.

>  Ted> We can keep the current nnmail cache as well, but this format
>  Ted> can replace the old cache format with very little work.  Maybe
>  Ted> we can copy and fix up the new-cache from the old nnmail-cache
>  Ted> if we're upgrading, and otherwise always use the new-cache
>  Ted> file?
> 
> *sigh* I thought this would be easier :-)
> 
> Unless I'm missing something, there is no immediately obvious way to
> do that. The current nnmail-cache functions store the "short" group
> name plus the message id, and I know of no generally correct way to
> get from there to the full backend+server:group description. We can
> either make assumptions, or just start a new cache.

I think we have to start a "version 2" cache.

> Assumptions *could* be fairly easy, as it seems that
> nnmail-cache-insert works for the "primary mail backend" only,
> "primary" being defined by the nnmail-cache-primary-mail-backend
> function which I'm not sure I understand. It seems it was supposed
> to find the first mail backend that supports respooling and has a
> nnsomething-get-new-mail function. Strange. So I guess we could just
> convert the current .nnmail-cache and use
> (gnus-method-to-full-server-name
> (nnmail-cache-primary-mail-backend)) for the backend description.

That sounds good.  It will work.  Given:

        gnus-select-method (quote (nnml "private" (nnml-marks-is-evil t)))

You can move an article to "group" and it's understood that this means
"nnml+private:group" currently, but you can also explicitly move it to
"nnml+private:group" and it will work correctly (as expected).  So all
backends can be full qualified, whether they are primary or secondary.

(any Gnus gurus, feel free to chime in if this is incorrect)

Since we're creating a new cache without overwriting the old one, it
will be a pretty safe operation.

> [...]
>  Ted> nnmail-cache-insert: needs to be generalized to
>  Ted> gnus-article-cache-insert, maybe figuring out the backend name
>  Ted> instead of having it passed in for minimum code changes?
> [...]
> 
> Is there a canonical way of figuring out what the current backend
> is?
> 
> Can I just do (gnus-method-to-full-server-name
> (gnus-command-method)) and hope to get the right thing?

I don't know.  I can go source-diving if no Gnus expert jumps into the
discussion.

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Jan Rychter <jan@rychter.com> writes:

> Unless I'm missing something, there is no immediately obvious way to do
> that. The current nnmail-cache functions store the "short" group name
> plus the message id, and I know of no generally correct way to get from
> there to the full backend+server:group description. We can either make
> assumptions, or just start a new cache.

Note that nnmail-cache makes sure to only store group names from the
right server, cf. nnmail-cache-primary-mail-backend.

When I did this, I thought only one server would be needed anyway, so
I didn't do more.

Oh, another possibility is to store the fully qualified group name,
instead of devising a file format with three fields.  I guess it's a
matter of taste which is to be preferred.
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnima-split-with-parent

[Ted Zlatanov]

On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
> Jan Rychter <jan@rychter.com> writes:
> 
>> Unless I'm missing something, there is no immediately obvious way
>> to do that. The current nnmail-cache functions store the "short"
>> group name plus the message id, and I know of no generally correct
>> way to get from there to the full backend+server:group
>> description. We can either make assumptions, or just start a new
>> cache.
> 
> Note that nnmail-cache makes sure to only store group names from the
> right server, cf. nnmail-cache-primary-mail-backend.
> 
> When I did this, I thought only one server would be needed anyway,
> so I didn't do more.
> 
> Oh, another possibility is to store the fully qualified group name,
> instead of devising a file format with three fields.  I guess it's a
> matter of taste which is to be preferred.

Storing things as "backend\tmessage-id\tgroupname" is easier to parse
(since we always want to separate the backend from the groupname), and
the field order makes it easier to reuse the old nnmail-cache
functions.  That's why I suggested that format.  I'm OK with any other format.

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Storing things as "backend\tmessage-id\tgroupname" is easier to parse
> (since we always want to separate the backend from the groupname), and
> the field order makes it easier to reuse the old nnmail-cache
> functions.  That's why I suggested that format.  I'm OK with any other format.

gnus-group-name-to-method can convert a fully qualified group name
into a method, and gnus-group-real-name gets the group name without
server.

So we're back to the `matter of taste' thing :-p

But this doesn't deal with the unqualified group names that are
currently in the file.
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnima-split-with-parent

[Ted Zlatanov]

On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
> But this doesn't deal with the unqualified group names that are
> currently in the file.

We convert them to the nnmail-cache-primary-backend:groupname fully
qualified format.  Wouldn't that make sense?

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
>> But this doesn't deal with the unqualified group names that are
>> currently in the file.
>
> We convert them to the nnmail-cache-primary-backend:groupname fully
> qualified format.  Wouldn't that make sense?

Usually, an unqualified group name is assumed to come from the
primary backend (ie, gnus-select-method).  So it's not sufficient to
_just_ call the functions I gave.

Of course, for new lines there isn't really a problem.

But how do you distinguish a new line from an old one?  Oh, you
_always_ add a prefix, even for the primary backend.  Yes, that would
work.
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnima-split-with-parent

[Ted Zlatanov]

On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
>>> But this doesn't deal with the unqualified group names that are
>>> currently in the file.
>>
>> We convert them to the nnmail-cache-primary-backend:groupname fully
>> qualified format.  Wouldn't that make sense?
> 
> Usually, an unqualified group name is assumed to come from the
> primary backend (ie, gnus-select-method).  So it's not sufficient to
> _just_ call the functions I gave.

Here are the upgrade steps when running nnimap-split-with-parent, when
it needs to load the history-cache:

1) does the history-cache (nnmail cache version 2) file exist?  if
   yes, go to (3)

2) upgrade ~/.nnmail-cache to a new history-cache file, or create a
   new history-cache file if ~/.nnmail-cache does not exist

a) creating a new file: just touch the file 

b) upgrading: all lines get prefixed with the return value of
nnmail-cache-primary-backend, then a \t character

3) load the history-cache file with the
   "backend\tmessage-id\tgroupname" format

Does that makes sense?  From (3) on, all work is done with the new
history-cache file.  The user could even go back to using the old
nnmail-split-with-parent function in their nnmail split, since
~/.nnmail-cache would be the same as before.

Ted

Re: nnima-split-with-parent

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> On Sat, 07 Dec 2002, kai.grossjohann@uni-duisburg.de wrote:
>>>> But this doesn't deal with the unqualified group names that are
>>>> currently in the file.
>>>
>>> We convert them to the nnmail-cache-primary-backend:groupname fully
>>> qualified format.  Wouldn't that make sense?
>> 
>> Usually, an unqualified group name is assumed to come from the
>> primary backend (ie, gnus-select-method).  So it's not sufficient to
>> _just_ call the functions I gave.
>
> Here are the upgrade steps when running nnimap-split-with-parent, when
> it needs to load the history-cache:
>
> 1) does the history-cache (nnmail cache version 2) file exist?  if
>    yes, go to (3)
>
> 2) upgrade ~/.nnmail-cache to a new history-cache file, or create a
>    new history-cache file if ~/.nnmail-cache does not exist
>
> a) creating a new file: just touch the file 
>
> b) upgrading: all lines get prefixed with the return value of
> nnmail-cache-primary-backend, then a \t character
>
> 3) load the history-cache file with the
>    "backend\tmessage-id\tgroupname" format
>
> Does that makes sense?  From (3) on, all work is done with the new
> history-cache file.

That's the right approach.

> The user could even go back to using the old
> nnmail-split-with-parent function in their nnmail split, since
> ~/.nnmail-cache would be the same as before.

Well, the file ~/.nnmail-cache isn't updated anymore, so using that
isn't useful for more than a couple of days ;-)

-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: nnima-split-with-parent

[Kai Großjohann]

Jan Rychter <jan@rychter.com> writes:

>>>>>> "Kai" == Kai Großjohann <kai.grossjohann@uni-duisburg.de> writes:
>
> [...]
>
>  Kai> Note that nnmail-cache makes sure to only store group names from
>  Kai> the right server, cf. nnmail-cache-primary-mail-backend.
>
> Yes, I noticed that. I removed that restriction. I hope that doesn't
> break things.

It means that if I move a message from nnml:mail.misc to
nnimap:INBOX.foo, then a reply to that message comes in, the
nnmail-split-fancy-with-parent code will try to put that reply in
nnml:INBOX.foo, which wasn't intended.

>  Kai> When I did this, I thought only one server would be needed anyway,
>  Kai> so I didn't do more.
>
>  Kai> Oh, another possibility is to store the fully qualified group
>  Kai> name, instead of devising a file format with three fields.  I
>  Kai> guess it's a matter of taste which is to be preferred.
>
> I also prefer the fully qualified group name approach. Except that I
> dont't know what a fully qualified group name really is (I think I know,
> but I haven't seen the definition anywhere). And what parts of Gnus are
> capable of using it. But as I'm reading more and more of Gnus code, I
> think FQGNs should be used much more than they are.

Right.
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)