Do nnmail-split-fancy-with-parent for nnimap?

Do nnmail-split-fancy-with-parent for nnimap?

[Kai Großjohann]

In principle, it should be possible to use
nnmail-split-fancy-with-parent for nnimap, too.  But there are some
issues, and I'm looking for ideas about them and about issues I
missed.

nnmail-cache-insert looks for the "primary mail backend" and only
inserts the group name without server name.  My idea was to extend
this mechanism to record which server the group is in.  Now the cache
lines are msg id, then tab, then group.  I could add another tab and
then the server.  That would make the format backward-compatible.  The
code would assume the primary mail backend for lines missing the
server.

I guess that nnmail-split-fancy-with-parent should somehow find out
the current backend and then ignore all the lines with the wrong
backend.  Right?  How would nnmail-split-fancy-with-parent find out
the current backend?  (Especially given that it might be called from
nnmail-split-methods as well as nnimap-split-rule!)

It would be really cool if ~/.nnmail-cache was stored on the server
somehow.  Is there a way to store data like this alongside IMAP mail
on a server?

Hm.  Actually I'm going to be really really really really busy
starting in a couple of days, so I'm afraid this might be no more
than some wishful thinking and day-dreaming :-|

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Ted Zlatanov]

On Sun, 06 Oct 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
> In principle, it should be possible to use
> nnmail-split-fancy-with-parent for nnimap, too.  But there are some
> issues, and I'm looking for ideas about them and about issues I
> missed.

I'd love this functionality!

> nnmail-cache-insert looks for the "primary mail backend" and only
> inserts the group name without server name.  My idea was to extend
> this mechanism to record which server the group is in.  Now the
> cache lines are msg id, then tab, then group.  I could add another
> tab and then the server.  That would make the format
> backward-compatible.  The code would assume the primary mail backend
> for lines missing the server.

Why not store the group name as Gnus shows it, e.g. when moving
articles?  nnxyz+abcdef:groupname is what I'm thinking of.  Then the
unqualified groups are in the primary backend - as they would be if
you manually moved articles around.  I'm talking as a Gnus user, maybe
the implementation is much more complex that way.  I don't know the
internals of mail moving.

> It would be really cool if ~/.nnmail-cache was stored on the server
> somehow.  Is there a way to store data like this alongside IMAP mail
> on a server?

I think the network usage would be too high for this.

> Hm.  Actually I'm going to be really really really really busy
> starting in a couple of days, so I'm afraid this might be no more
> than some wishful thinking and day-dreaming :-|

I hope someone else has interest in this!

Ted

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 06 Oct 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
>> In principle, it should be possible to use
>> nnmail-split-fancy-with-parent for nnimap, too.  But there are some
>> issues, and I'm looking for ideas about them and about issues I
>> missed.
>
> I'd love this functionality!

Me too 8-)

>> nnmail-cache-insert looks for the "primary mail backend" and only
>> inserts the group name without server name.  My idea was to extend
>> this mechanism to record which server the group is in.  Now the
>> cache lines are msg id, then tab, then group.  I could add another
>> tab and then the server.  That would make the format
>> backward-compatible.  The code would assume the primary mail backend
>> for lines missing the server.
>
> Why not store the group name as Gnus shows it, e.g. when moving
> articles?  nnxyz+abcdef:groupname is what I'm thinking of.  Then the
> unqualified groups are in the primary backend - as they would be if
> you manually moved articles around.  I'm talking as a Gnus user, maybe
> the implementation is much more complex that way.  I don't know the
> internals of mail moving.

The problem is backward compatibility.  Suppose people have nntp as
their primary server, and nnml and nnimap as secondary servers.  Then
right now foo.bar means the group nnml:foo.bar (because nnml is the
"primary mail backend"), but your suggestion would change it to mean
a group on the nntp server.

But maybe I'm overlooking something obvious?

(One possibility would be to make all groups be prefixed, maybe
we could use "native" for the primary server.)

>> It would be really cool if ~/.nnmail-cache was stored on the server
>> somehow.  Is there a way to store data like this alongside IMAP mail
>> on a server?
>
> I think the network usage would be too high for this.

You must be kidding.  Right now, nnimap splitting transfers each
message over the net twice (one fetch from nnimap:INBOX, one put to
the target group).  I don't see how it can hurt to add 60 or 80 bytes
to this traffic.

No offense intended.

(It conceivable that .nnmail-cache must always be transferred in
full, maybe via ange-ftp.  This is a bad solution indeed, and for
this your counter argument would be right.  I wasn't even considering
such bad solutions before you mentioned the possibility :-)

>> Hm.  Actually I'm going to be really really really really busy
>> starting in a couple of days, so I'm afraid this might be no more
>> than some wishful thinking and day-dreaming :-|
>
> I hope someone else has interest in this!

Right.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>>> It would be really cool if ~/.nnmail-cache was stored on the server
>>> somehow.  Is there a way to store data like this alongside IMAP mail
>>> on a server?
>>
>> I think the network usage would be too high for this.
>
> You must be kidding.  Right now, nnimap splitting transfers each
> message over the net twice (one fetch from nnimap:INBOX, one put to
> the target group).  

Really?  I thought it only fetched the headers once, and then used
COPY+Delete to move it.  Maybe things work differently when fancy
splitting is used, I don't use it.

Maybe tramp could be extended to support reading/storing files in IMAP
somehow?  I don't know much about tramp so maybe this won't work.

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> You must be kidding.  Right now, nnimap splitting transfers each
>> message over the net twice (one fetch from nnimap:INBOX, one put to
>> the target group).  
>
> Really?  I thought it only fetched the headers once, and then used
> COPY+Delete to move it.  Maybe things work differently when fancy
> splitting is used, I don't use it.

Probably I'm wrong.  But still, the few bytes of data from the
corresponding ~/.nnmail-cache line won't make a difference, I think.

Unless the whole file needs to be transferred at all times.

> Maybe tramp could be extended to support reading/storing files in IMAP
> somehow?  I don't know much about tramp so maybe this won't work.

How come everybody wants to extend Tramp? ;-)  It can be done using
filename handlers, but I think it doesn't make sense to extend Tramp
(except maybe for unifying the filename syntax; and there a new
solution is in the works).

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Ted Zlatanov]

On Sun, 06 Oct 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
>> Why not store the group name as Gnus shows it, e.g. when moving
>> articles?  nnxyz+abcdef:groupname is what I'm thinking of.  Then
>> the unqualified groups are in the primary backend - as they would
>> be if you manually moved articles around.  I'm talking as a Gnus
>> user, maybe the implementation is much more complex that way.  I
>> don't know the internals of mail moving.
> 
> The problem is backward compatibility.  Suppose people have nntp as
> their primary server, and nnml and nnimap as secondary servers.
> Then right now foo.bar means the group nnml:foo.bar (because nnml is
> the "primary mail backend"), but your suggestion would change it to
> mean a group on the nntp server.

Right, so the problem is that right now, the cache assumes nnml.  You
could have a cache version at the top of the file (like BBDB), and
upgrade users to the newer cache version when they run the new
function, rewriting every group name to "nnml:groupname".

Or you could have a file called .nnimap.cache, separate from the nnml
cache.

> (One possibility would be to make all groups be prefixed, maybe
> we could use "native" for the primary server.)

That's what I was thinking, but realize that the "native" prefix or
whatever we use for the primary server is superfluous - if it's an
arbitrary string, you may as well skip it.

>>> It would be really cool if ~/.nnmail-cache was stored on the
>>> server somehow.  Is there a way to store data like this alongside
>>> IMAP mail on a server?
>>
>> I think the network usage would be too high for this.
> 
> You must be kidding.  Right now, nnimap splitting transfers each
> message over the net twice (one fetch from nnimap:INBOX, one put to
> the target group).  I don't see how it can hurt to add 60 or 80
> bytes to this traffic.

> (It conceivable that .nnmail-cache must always be transferred in
> full, maybe via ange-ftp.  This is a bad solution indeed, and for
> this your counter argument would be right.  I wasn't even
> considering such bad solutions before you mentioned the possibility
> :-)

You would have to either break up the cache in some way, or transfer
the entire cache every time.  You can store the cache in a group
(e.g. "gnus-imap-split-cache") that would basically be a mail group
like any other; corruption of its data is the user's problem.  Any
other storage of the cache would need support on the IMAP server,
AFAIK.

Now, if you transfer the entire cache, the chance of data corruption
is smaller, and the code is much simpler.  That's the solution I
thought of, initially.

Breaking up the cache in some way would complicate the code
considerably.  Also, you would have to devise an efficient way to
search all the cache elements for an article ID, or have the subject
of each article in the cache group somehow contain the article ID.

Either way, with a single cache, or with a piecemeal cache, it's a lot
of traffic when your cache grows to 50000 articles.

Am I missing something obvious?

Ted

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Kai Großjohann]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 06 Oct 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
>> 
>> The problem is backward compatibility.  Suppose people have nntp as
>> their primary server, and nnml and nnimap as secondary servers.
>> Then right now foo.bar means the group nnml:foo.bar (because nnml is
>> the "primary mail backend"), but your suggestion would change it to
>> mean a group on the nntp server.
>
> Right, so the problem is that right now, the cache assumes nnml.  You
> could have a cache version at the top of the file (like BBDB), and
> upgrade users to the newer cache version when they run the new
> function, rewriting every group name to "nnml:groupname".

That's a good idea, actually.

> Or you could have a file called .nnimap.cache, separate from the nnml
> cache.

Right.

>> (One possibility would be to make all groups be prefixed, maybe
>> we could use "native" for the primary server.)
>
> That's what I was thinking, but realize that the "native" prefix or
> whatever we use for the primary server is superfluous - if it's an
> arbitrary string, you may as well skip it.

No, with the suggestion "native:foo.bar" would be a new-style line
whereas "foo.bar" would indicate an old-style line.

Anyhow.  All these methods are certainly viable methods.

>>>> It would be really cool if ~/.nnmail-cache was stored on the
>>>> server somehow.  Is there a way to store data like this alongside
>>>> IMAP mail on a server?
>>>
>>> I think the network usage would be too high for this.
>> 
>> You must be kidding.  Right now, nnimap splitting transfers each
>> message over the net twice (one fetch from nnimap:INBOX, one put to
>> the target group).  I don't see how it can hurt to add 60 or 80
>> bytes to this traffic.
>
>> (It conceivable that .nnmail-cache must always be transferred in
>> full, maybe via ange-ftp.  This is a bad solution indeed, and for
>> this your counter argument would be right.  I wasn't even
>> considering such bad solutions before you mentioned the possibility
>> :-)
>
> You would have to either break up the cache in some way, or transfer
> the entire cache every time.

Yeah, I was thinking of just transferring new entries.  The current
implementation only does three operations on the cache: (1) append an
item (at the end), (2) delete items from the front, and (3) search
items in it.

So just posting a message in a special group with the data in the
subject or something like that would do the trick.  But it would be
quite inefficient.

> You can store the cache in a group (e.g. "gnus-imap-split-cache")
> that would basically be a mail group like any other; corruption of
> its data is the user's problem.

That's a good idea.

> Any other storage of the cache would need support on the IMAP
> server, AFAIK.

Yes.  I was hoping for some additional kind of protocols that IMAP
servers are likely to speak.  (Just as they now speak Sieve for
another purpose.)

> Now, if you transfer the entire cache, the chance of data corruption
> is smaller, and the code is much simpler.  That's the solution I
> thought of, initially.

Simplicity of implementation is a good thing, of course.

> Breaking up the cache in some way would complicate the code
> considerably.  Also, you would have to devise an efficient way to
> search all the cache elements for an article ID, or have the subject
> of each article in the cache group somehow contain the article ID.
>
> Either way, with a single cache, or with a piecemeal cache, it's a lot
> of traffic when your cache grows to 50000 articles.
>
> Am I missing something obvious?

I don't think so.  Thanks for your hints.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Paul Jarc]

Ted Zlatanov <tzz@lifelogs.com> wrote:
> Mail-Followup-To: Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai
>  Großjohann), @ns2.beld.net, ding@gnus.org

I wonder where that came from.

> On Sun, 06 Oct 2002, Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:
>> (One possibility would be to make all groups be prefixed, maybe
>> we could use "native" for the primary server.)
>
> That's what I was thinking, but realize that the "native" prefix or
> whatever we use for the primary server is superfluous - if it's an
> arbitrary string, you may as well skip it.

"native" happens to be nonarbitrary; Gnus really recognizes it as the
name of the primary server, so "native:group.name" is a valid FQGN.
But I think it's better to use the "backend+address:group.name"
scheme.  That's the only way to qualify names of groups on secondary
servers, and we don't really need another naming scheme that works on
only some servers.


paul

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I guess that nnmail-split-fancy-with-parent should somehow find out
> the current backend and then ignore all the lines with the wrong
> backend.  Right?  How would nnmail-split-fancy-with-parent find out
> the current backend?  (Especially given that it might be called from
> nnmail-split-methods as well as nnimap-split-rule!)

`gnus-command-method' might work.

> It would be really cool if ~/.nnmail-cache was stored on the server
> somehow.  Is there a way to store data like this alongside IMAP mail
> on a server?

Not really, but you can use either a dummy mailbox and put the file in
a message in it (mark it \Deleted and it won't show up in Gnus), or
store a dummy message in a standard mailbox ("INBOX" maybe) and
somehow mark it so you can find it reliably.

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> I guess that nnmail-split-fancy-with-parent should somehow find out
>> the current backend and then ignore all the lines with the wrong
>> backend.  Right?  How would nnmail-split-fancy-with-parent find out
>> the current backend?  (Especially given that it might be called from
>> nnmail-split-methods as well as nnimap-split-rule!)
>
> `gnus-command-method' might work.

I shall try that.

>> It would be really cool if ~/.nnmail-cache was stored on the server
>> somehow.  Is there a way to store data like this alongside IMAP mail
>> on a server?
>
> Not really, but you can use either a dummy mailbox and put the file in
> a message in it (mark it \Deleted and it won't show up in Gnus), or
> store a dummy message in a standard mailbox ("INBOX" maybe) and
> somehow mark it so you can find it reliably.

Isn't there some kind of Advanced Configuration Management Protocol
or somesuch?  I remember a four-letter acronym starting with A which
was mentioned in conjunction with IMAP.  It was for storing some kind
of config data.

Maybe I should have mentioned that I control the server, so I can
install cool daemons on it.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Andi Hechtbauer]

>>>>> Kai Großjohann writes:

    KG> Isn't there some kind of Advanced Configuration Management
    KG> Protocol or somesuch?  I remember a four-letter acronym
    KG> starting with A which was mentioned in conjunction with IMAP.
    KG> It was for storing some kind of config data.

ITYM ACAP[1] - the Application Configuration Access Protocol, an
internet protocol for accessing client program options,
configurations, and preference information remotely.

Developed around Cyrus IMAP Server.

Not a clue if it's still in active development, of if it's any good.
I just heard of it.

__
[1] http://asg.web.cmu.edu/acap/

Re: Do nnmail-split-fancy-with-parent for nnimap?

[Danny Siu]

Kai Großjohann writes:

  Kai> In principle, it should be possible to use
  Kai> nnmail-split-fancy-with-parent for nnimap, too.  But there are some
  Kai> issues, and I'm looking for ideas about them and about issues I
  Kai> missed.

I have been wanting this for a long time.

-- 
Danny Siu