mail split with multiple backends

mail split with multiple backends

[joules]

Re: mail...

I like using nnfolder but it can be annoying for large lists. So, I
want to have one group set to nnml and the rest in nnfolder. I was
able to do this without problems. (Yes, I changed nnml-directory to be
different than nnfolder-directory.) Now, I want to split some messages
to nnml:foo and others to nnfolder:bar. nnmail-split-methods doesn't
seem to like if I add the nn*: stuff. 

Any thoughts?

-- 
Julian

Re: mail split with multiple backends

[Kai Großjohann]

There are a number of ways to do it.  One possibility is this: suppose
you have nnml and nnfolder backends.  Then you set up
nnmail-split-methods for nnml and include a rule which puts all
nnfolder messages into a special group; nnml:nnfolder, say.

Then you enter that group, `M P a' to process-mark all articles,
temporarily change nnmail-split-methods to a value which is good for
nnfolder, then `B r' to resplit the messages using the new
nnmail-split-methods.  Specify nnfolder as the backend to use.

The second alternative is to have some program outside of Gnus which
splits the mail (such as procmail).  Then you set mail-sources such
that it only picks up the nnml part, get new mail, then set
mail-sources such that it only picks up the nnfolder part, and get new
mail again.  You also have to arrange that the first time you get new
mail, the nnml backend gets it, and the second time you do it, the
nnfolder backend gets it.

Ugly, but ought to work.

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Rob Browning]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Then you enter that group, `M P a' to process-mark all articles,
> temporarily change nnmail-split-methods to a value which is good for
> nnfolder, then `B r' to resplit the messages using the new
> nnmail-split-methods.  Specify nnfolder as the backend to use.

Beware, though, that your marks, if you have any, will be lost when
you do this.  I didn't realize that once, and wished I had later :>

Another thing I forgot to put on my wishlist was persistent (across
respools) marks.

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: mail split with multiple backends

[Harry Putnam]

Rob Browning <rlb@cs.utexas.edu> writes:

> Beware, though, that your marks, if you have any, will be lost when
> you do this.  I didn't realize that once, and wished I had later :>
> 
> Another thing I forgot to put on my wishlist was persistent (across
> respools) marks.

Are you referring here to a respool across different backends?

Respooling from nnml to nnml the marks persist.

Re: mail split with multiple backends

[Rob Browning]

Harry Putnam <reader@newsguy.com> writes:

> Are you referring here to a respool across different backends?
> 
> Respooling from nnml to nnml the marks persist.

Has that changed recently?

If not then I'm surprised, I sure did something a while back that lost
my marks, and I could have sworn it was process marking all the
articles in one of my most important groups and then respooling (B r)
them to the same group.  I did that because I'd changed my split rules
and I wanted to use them to file some of the articles elsewhere, but I
certainly didn't want *any* of them to lose their marks.

In fact, I still haven't finished recovering from that.  I just don't
have the time to deal with that much work.  I now have a group with
8638 messages in it, and I have no idea what in there is
important/pending vs archive material :< It's been long enough now
though that I'm hoping that for anything important that I accidentally
dropped, the relevant people have since contacted me.

(I just recalled another wishlist item -- making
 gnus-use-cross-reference a backend variable so that you can, for
 example, set it separately for mail and news (I'd like t for news and
 nil for mail).  Originally I just set it because I wanted
 cross-references updated for news, but I didn't think carefully
 enough about what the implications for mail were...)

FWIW
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: mail split with multiple backends

[Harry Putnam]

Rob Browning <rlb@cs.utexas.edu> writes:

> Harry Putnam <reader@newsguy.com> writes:
> 
> > Are you referring here to a respool across different backends?
> > 
> > Respooling from nnml to nnml the marks persist.
> 
> Has that changed recently?

I don't think so, but reviewing past changel logs will tell the story.
I respool one nnml group several times daily and see marks persist
into the destination groups.  I haven't tried every mark known to gnus
but `r', `!' and `?' persist.

Just now testing from a different backend, I see marks persist from
nndoc to nnml too.

> If not then I'm surprised, I sure did something a while back that lost
> my marks

Maybe you inadvertantly hit `M[eta]-c' from group buffer...

Re: mail split with multiple backends

[Rob Browning]

Harry Putnam <reader@newsguy.com> writes:

> Just now testing from a different backend, I see marks persist from
> nndoc to nnml too.

OK I'll check it out.

> > If not then I'm surprised, I sure did something a while back that lost
> > my marks
> 
> Maybe you inadvertantly hit `M[eta]-c' from group buffer...

I don't think so, I was in the summary buffer, but it's been a long
time.  If Gnus hasn't changed, then perhaps I did something other than
what I thought I did...

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: mail split with multiple backends

[Kai Großjohann]

On 20 Oct 2000, Rob Browning wrote:

> (I just recalled another wishlist item -- making
>  gnus-use-cross-reference a backend variable so that you can, for
>  example, set it separately for mail and news (I'd like t for news
>  and nil for mail).  Originally I just set it because I wanted
>  cross-references updated for news, but I didn't think carefully
>  enough about what the implications for mail were...)

What are the implications?  Sorry if I appear to be dense.

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Rob Browning]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> What are the implications?  Sorry if I appear to be dense.

No not at all.  I just didn't figure anyone would care, so I didn't
elaborate.  Unless you had psychic powers, you couldn't have known
what I meant :>

My problems, and perhaps they're just mine, were that given
crossposting, cross-referencing, and the various expiry behaviors, it
was really hard for me to know, on any intuitive level, or with any
feeling of confidence, exactly what might happen when I hit, say "B
del" or "E", or "d" on a given article.  Especially since, unless I'm
recalling wrong, marks are *not* handled consistently (i.e. I think I
recall that "E" propagates as "r" if crossreferencing is on in certain
circumstances!)

After I had a couple of surprises, once I sat down, read the docs more
carefully, looked at some of the source, and thought a bit about it, I
realized that it was probably doing more or less what I told it to,
though that wasn't what I expected.

For example, in news, crossreference handling is something I generally
want.  If I read an article (or kill it), I don't want to see it again
in another group.  For mail, though, say I have rules that'll
crosspost anything addressed to me directly and to debian-bugs to my
"inbox" and my "debian-bugs" groups.  I do this so that I won't
accidentally miss something important in the thousands of messages
that go into my debian-bugs group.  However, most of the time, when a
bug pops up in my inbox (and debian-bugs), I just want to delete the
one in my inbox, after seeing it, and then later, when I start working
on the bug, go over to my debian-bugs group and find/use/reply-to the
copy there.

When I first got started, I just figured, "no problem", I'll just
expire/mark-as-read the one in my inbox and that'll do the trick.
Unfortunately, since I had crossreferencing turned on (because I
generally wanted it for news) the other article was deleted as well.
It took me a while to realize what was going on, and by then I had
lost quite a lot of articles.  This was further complicated (and
unfortunately I don't recall the exact details) by the totally
unexpected handling of "E" across groups in the context of
crossreferencing.

Ignoring my misunderstanding for a minute, as a utility issue, I tend
to feel that there should be separate commands for "mark this article
as <FOO>" and "mark this article and all cross references as <FOO>".
It would be nice to be able to hit "d" for mark as deleted, and "C-u
d" to get all the cross-references too.  It might also be nice to have
a per-backend flag so that you could invert the sense of these
commands.  That way I could set the flag oppositely for my mail and
news backends.  In one "d" means mark this article.  In the other,
"d" means mark this article and all crossreferences.

As I said, perhaps all of this is just a still-not-quite-resolved
difference between the way I tend to want to think about/do things
mail related, and what gnus wants, but all these little incidents have
kinda added up -- hence my current disucssion.  Gnus seems to be so
close to exactly what I want out of a MUA -- so I'm just wondering if
I can help fix up those last bits, presuming I can acutally figure out
exactly what I want :>

Just having *really* clear documentation of what's supposed to happen
would help a lot.  Things like the "E"/"r" issue should be documented
-- I had to look at the code -- I suspect, though, that if all the
expiry/read/E/r/crossreference behaviors were documented, we may see
that some configurations produce behaviours that just don't make
sense.  Then we could set about deciding what would make sense,
documenting that, and then fixing up the code.

When I was browsing the source, it seemed like the whole "marks"
infrastructure needs a major overhaul.  As I suggested earlier, it
might be beneficial to change it so that marks are just lists of
symbols attached to articles.  The expire mark, for example, and it's
relation to readedness and deletion in the code made it seem like "E"
isn't quite a full fledged citizen -- it doesn't play nice with
scoring, and as it doesn't seem to work well with crossposting,
behaving better or worse depending on the target group's expiry
policy.  If 'expire was a mark that was completely separate from 'read
and if an article could have each mark independently '(expire read), I
think many of those problems might go away...

Thanks, and hope that wasn't too confusing.
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: mail split with multiple backends

[Kai Großjohann]

On 21 Oct 2000, Rob Browning wrote:

> Ignoring my misunderstanding for a minute, as a utility issue, I
> tend to feel that there should be separate commands for "mark this
> article as <FOO>" and "mark this article and all cross references as
> <FOO>".  It would be nice to be able to hit "d" for mark as deleted,
> and "C-u d" to get all the cross-references too.  It might also be
> nice to have a per-backend flag so that you could invert the sense
> of these commands.  That way I could set the flag oppositely for my
> mail and news backends.  In one "d" means mark this article.  In the
> other, "d" means mark this article and all crossreferences.

I want this too.  In fact, I suggested this in a discussion way back
when; this has come up on the Gnus list a couple of times.

(I didn't suggest these key bindings -- `C-u d' now means mark 4
articles as read, I think -- but that's just a minor matter which
barely justifies mentioning.)

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Didier Verna]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> There are a number of ways to do it.  One possibility is this: suppose
> you have nnml and nnfolder backends.  Then you set up
> nnmail-split-methods for nnml and include a rule which puts all
> nnfolder messages into a special group; nnml:nnfolder, say.
> 
> Then you enter that group, `M P a' to process-mark all articles,
> temporarily change nnmail-split-methods to a value which is good for
> nnfolder, then `B r' to resplit the messages using the new
> nnmail-split-methods.  Specify nnfolder as the backend to use.
> 
> The second alternative is to have some program outside of Gnus which
> splits the mail (such as procmail).  Then you set mail-sources such
> that it only picks up the nnml part, get new mail, then set
> mail-sources such that it only picks up the nnfolder part, and get new
> mail again.  You also have to arrange that the first time you get new
> mail, the nnml backend gets it, and the second time you do it, the
> nnfolder backend gets it.

        There's a thrid possibility that I use in a backend I wrote (I'll tell
you what it is soon on this list, as it might be of some interest): the idea
is that you tell *both* backends to retreive mail, but in one of them, you
override the mail-source and split-methods variable.

        So you can do this for instance:
- As in Kai's second option, you split your mails in two different "source"
  files (with procmail for instance)
- You set *both* <backend>-get-new-mail variables to t
- You define the variables nnml-mail-sources and nnml-split-methods (if you
  decide that nnml is the secondary backend
- And finally, you rewrite the nnml functions that get mails, in order to wrap
  their code in a 'let statement that will allow you to temporarily assign
  different values to mail-sources and split-methods.

        For instance, all calls to nnmail-get-new-mail should be wrapped in
something like:

  (let ((mail-sources nnml-mail-sources)
	(nnml-split-methods nnml-split-methods))
    (nnml-get-new-mail 'nnml nil nnml-directory group)))


This method is very nice because all you have to do is type 'g' in order to
get mails from/for multiple mail backends at once, each one with its own
mail-sources and split-methods. There is one drawback however: you can't use
the respooling facilities across different backends.

-- 
    /     /   _   _       Didier Verna        http://www.inf.enst.fr/~verna/
 - / / - / / /_/ /        EPITA / LRDE         mailto:didier@lrde.epita.fr
/_/ / /_/ / /__ /      14-16 rue Voltaire        Tel. +33 (1) 53 14 59 47
                   94276 Kremlin-Bicêtre cedex   Fax. +33 (1) 44 08 01 99

Re: mail split with multiple backends

[Kai Großjohann]

On 23 Oct 2000, Didier Verna wrote:

> For instance, all calls to nnmail-get-new-mail should be wrapped in
> something like:
> 
>   (let ((mail-sources nnml-mail-sources)
> 	(nnml-split-methods nnml-split-methods))
>     (nnml-get-new-mail 'nnml nil nnml-directory group)))
> 
> 
> This method is very nice because all you have to do is type 'g' in
> order to get mails from/for multiple mail backends at once, each one
> with its own mail-sources and split-methods. There is one drawback
> however: you can't use the respooling facilities across different
> backends.

Wouldn't it be even nicer if it was possible to have mail-sources and
nnmail-split-methods be server parameters?  Then you could go through
all *-get-new-mail functions and bind the two variables appropriately.

How's that sound?

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Didier Verna]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> Wouldn't it be even nicer if it was possible to have mail-sources and
> nnmail-split-methods be server parameters?  Then you could go through
> all *-get-new-mail functions and bind the two variables appropriately.

        Yup. The only thing I'm not very confident about is how we would deal
with interaction between different servers (like respooling across backends).
Making server parameters of these variables don't seem like a sufficiently
abstracted design to me; the same applies to the kludge I'm currently using
BTW.

        The other problem with this solution is that it still requires an
external mail split process, before Gnus actually reads its mail sources. This
should also be ideally avoided. I haven't thought very much about this yet.

-- 
    /     /   _   _       Didier Verna        http://www.inf.enst.fr/~verna/
 - / / - / / /_/ /        EPITA / LRDE         mailto:didier@lrde.epita.fr
/_/ / /_/ / /__ /      14-16 rue Voltaire        Tel. +33 (1) 53 14 59 47
                   94276 Kremlin-Bicêtre cedex   Fax. +33 (1) 44 08 01 99

Re: mail split with multiple backends

[Kai Großjohann]

On 23 Oct 2000, Didier Verna wrote:
> 
> Yup. The only thing I'm not very confident about is how we would
> deal with interaction between different servers (like respooling
> across backends).

Well, the user enters a target server when using `B r', and the target
server parameters decide the split methods.  In the context of `B r',
mail-sources is irrelevant.

> Making server parameters of these variables don't seem like a
> sufficiently abstracted design to me; the same applies to the kludge
> I'm currently using BTW.

Hm.  I thought that server parameters would be a very good place to do
it, but I've been thinking more from an implementation point of view,
I'm afraid.  Hm.

> The other problem with this solution is that it still requires an
> external mail split process, before Gnus actually reads its mail
> sources. This should also be ideally avoided. I haven't thought very
> much about this yet.

You're right, this is bad.  It could be circumvented if there was some
protocol whereby mail splitting could leave messages where they are.

But maybe it would be even better if there was just one variable which
contained rules for splitting, and the target groups could be on any
server and the rules could include conditions based on where the
messages come from.

I think that Lars says that this would be hard to implement, though,
because the (some?) backends circumvent the normal
-request-accept-article machinery, for efficiency.  I wonder how much
efficiency does this buy us?  Maybe the splitting machinery should use
the normal -request-accept-article machinery.

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Toby Speight]

0> In article <vafsnpnzqvb.fsf@lucy.cs.uni-dortmund.de>,
0> Kai Großjohann <URL:mailto:Kai.Grossjohann@CS.Uni-Dortmund.DE> ("Kai") wrote:

Kai> I think that Lars says that this would be hard to implement,
Kai> though, because the (some?) backends circumvent the normal
Kai> -request-accept-article machinery, for efficiency.  I wonder
Kai> how much efficiency does this buy us?

Quite a lot, I think, in the case of IMAP.  AIUI, it uses IMAP's move
command while splitting - if it didn't, it would have to download each
article and upload it again.

I'm not sure how hard it would be to special-case the code for a
same-server split.  I don't expect it would be hard, but I'm not
volunteering ;-)

Re: mail split with multiple backends

[Kai Großjohann]

On 26 Oct 2000, streapadair@gmx.net wrote:
> 
> Quite a lot, I think, in the case of IMAP.  AIUI, it uses IMAP's
> move command while splitting - if it didn't, it would have to
> download each article and upload it again.

That's silly, of course.  I think that all of Gnus could benefit from
a faster move method.  So why don't we add logic to Gnus which allows
a faster move?  The logic would take the source and target groups as
input, then decide whether a fast move is possible, and invoke the
fast move function as applicable.

kai
-- 
I like BOTH kinds of music.

Re: mail split with multiple backends

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> > Quite a lot, I think, in the case of IMAP.  AIUI, it uses IMAP's
> > move command while splitting - if it didn't, it would have to
> > download each article and upload it again.
> 
> That's silly, of course.  I think that all of Gnus could benefit from
> a faster move method.  So why don't we add logic to Gnus which allows
> a faster move?  The logic would take the source and target groups as
> input, then decide whether a fast move is possible, and invoke the
> fast move function as applicable.

This wouldn't help non-IMAP mail splitting since they don't use "move"
or "copy" at all, but rather "append".  Which is good and fast for
nnmail-like backends.  A backend independent splitter that used
backend optimizations where possible, with a user interface something
along the lines of

http://www.gnus.org/list-archives/ding/199901/msg00485.html

would be nice, and not very difficult, I think.

Re: mail split with multiple backends

[Harry Putnam]

joules@writeme.com writes:

> Re: mail...
> 
> I like using nnfolder but it can be annoying for large lists. So, I
> want to have one group set to nnml and the rest in nnfolder. I was
> able to do this without problems. (Yes, I changed nnml-directory to be
> different than nnfolder-directory.) Now, I want to split some messages
> to nnml:foo and others to nnfolder:bar. nnmail-split-methods doesn't
> seem to like if I add the nn*: stuff. 
> 
> Any thoughts?

The short answer is to use nnml for everthing.  Else you get into the
complexities that Kai draws out in some detail.

Do you have a special need for nnfolder groups?  One thing might be to
make them available to other mailers such as 'mutt' that cannot read
nnml groups.

I find it handy to use nnml as the only backend, but still use
`procmail' for some things.  The procmail spool files are snarfed up
by gnus and put in nnml groups *WITHOUT* going through what ever is
set for nnmail split methods.

That is the normal functioning of a gnus/procmail combo.