Moving from nnml to nnimap...

Moving from nnml to nnimap...

[Jinhyok Heo]

Now I'm moving from nnml to nnimap and I want to hear some advice from
you.

First of all, is there an easy way to move an existing nnml group to
nnimap one? I did try to use "B c" to move one to a new nnimap group,
but it was too slow for an large or huge group.

Secondly, what size of one nnimap group is appropriate? I think nnimap
will be slower than nnml and it is good to maintain appropriate size
of nnimap groups. Can you give me a rough size?

Thirdly, is there a way to share gnus marks among xemacses on other
machines? I remember I saw an thread regarding this issue. I want to
know the current status of it.

Lastly, I use UW imap server, which doesn't seem to be recommended for
gnus. Does it have any defect unlike Courier or Cyrus imap?

Many questions. :)

Thanks in advance.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" -- a Ladakhi monk

Re: Moving from nnml to nnimap...

[Daniel Pittman]

On Mon, 24 Nov 2003, Jinhyok Heo wrote:
> First of all, is there an easy way to move an existing nnml group to
> nnimap one? I did try to use "B c" to move one to a new nnimap group,
> but it was too slow for an large or huge group.

I never found one, although you could possibly write a routine if you
knew lisp well enough...

> Secondly, what size of one nnimap group is appropriate? I think nnimap
> will be slower than nnml and it is good to maintain appropriate size
> of nnimap groups. Can you give me a rough size?

With a real IMAP server, performance should be as good as, or better
than, Gnus and nnml, in my experience.

> Thirdly, is there a way to share gnus marks among xemacses on other
> machines? I remember I saw an thread regarding this issue. I want to
> know the current status of it.

If you use IMAP, the marks are stored on the server...

> Lastly, I use UW imap server, which doesn't seem to be recommended for
> gnus. 

That would be because it's ... not very good, to put it mildly. The
upstream policy on security issues ("use something else") was also not
inspiring...

> Does it have any defect unlike Courier or Cyrus imap?

Well, the default use of 'mbox' folders will make your performance suck
completely. :)

I strongly recommend the Cyrus IMAP server. While it is much harder to
get up and running than the competition, the performance is great and
it's extremely powerful and useful.

Courier is fine for small groups, but isn't a well featured IMAP server
for the "shared" or "group" stuff that you can do (public and shared
folders, for example).

         Daniel

-- 
Anyone who doubts the value of [probabilistic] statements sould try to
determine the total assets of insurance companies in America. This astonishing
sum has been accumulated through skillful and successful prediction of the
behaviour of large groups of people.
        -- Roy Weatherford, _Philosophical Foundations of Probability Theory_

Re: Moving from nnml to nnimap...

[Ted Zlatanov]

On Mon, 24 Nov 2003, daniel@rimspace.net wrote:

On Mon, 24 Nov 2003, Jinhyok Heo wrote:
>> First of all, is there an easy way to move an existing nnml group
>> to nnimap one? I did try to use "B c" to move one to a new nnimap
>> group, but it was too slow for an large or huge group.
> 
> I never found one, although you could possibly write a routine if
> you knew lisp well enough...

I think it's easiest to condense the nnml group into a single mbox
file and import that into the IMAP server if you have shell access to
it, and it allows mbox storage.  I've done that with UW IMAP, and it
works nicely.  Unfortunately, marks will be lost - if you need those,
and you have some choice in the IMAP server, use one that allows
Maildir storage.  It will be much faster to move articles into the
IMAP server then.

> I strongly recommend the Cyrus IMAP server. While it is much harder
> to get up and running than the competition, the performance is great
> and it's extremely powerful and useful.

I'm also evaluating dovecot for Gnus, I'll let the group know how
performance compares between dovecot and Cyrus, when I have a few
months of experience.  UW IMAP performance with Gnus is pretty bad.
The only reason I'm using UW IMAP is because that's what my ISP
provides.  It is occasionally convenient to access folders as mbox
files. that's the only benefit to UW IMAP.

Ted

Re: Moving from nnml to nnimap...

[Ken Raeburn]

I'm also interested in possibly making the move to nnimap.

I use crossposting a lot in my incoming mail splitting, though.
I want the messages marked for expiration in all groups at the same
time it's marked for expiration in one of the other groups.  Do marks
get propagated for cross-posted articles in nnimap?

Ken

nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

The silence is deafening....  So, is no one actually using
cross-posting when splitting articles in an nnimap server?

My experimentation so far (playing with imtest, don't have the guest
account and nnimap config set up yet) seems to indicate that the marks
won't be propagated.  If this is the case, I think it would be a good
idea for the documentation to indicate this.

(Actually, seen and expire are the only marks I really care about, if
they're done any differently.)

I wrote, in November:
> I use crossposting a lot in my incoming mail splitting, though.
> I want the messages marked for expiration in all groups at the same
> time it's marked for expiration in one of the other groups.  Do marks
> get propagated for cross-posted articles in nnimap?

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> The silence is deafening....  So, is no one actually using
> cross-posting when splitting articles in an nnimap server?
>
> My experimentation so far (playing with imtest, don't have the guest
> account and nnimap config set up yet) seems to indicate that the marks
> won't be propagated.  If this is the case, I think it would be a good
> idea for the documentation to indicate this.

IMAP doesn't understand the concept of crossposting, I think, so this
sounds normal.  How does crosspost mark propagation work for nntp
groups?  I suspect it might be possible to implement something like it
for nnimap by looking in the nnmail message-id split log when applying
marks to a group (i.e., for each article with new marks, look up the
message id in the split log and find out what other groups it was
crossposted to and apply the same mark to it).

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
>> My experimentation so far (playing with imtest, don't have the guest
>> account and nnimap config set up yet) seems to indicate that the marks
>> won't be propagated.  If this is the case, I think it would be a good
>> idea for the documentation to indicate this.
>
> IMAP doesn't understand the concept of crossposting, I think, so this
> sounds normal.  How does crosspost mark propagation work for nntp
> groups?

Using the xref header, I believe, which as far as I can tell, is
generated on the fly for nnimap, and only lists the current group.

I wonder, is there a way to go through all the groups and ask the
server if a certain set of message-ids are present in each?  It would
probably be painfully slow if there are a lot of groups (I've got over
200 groups in my nnml hierarchy), and would require either staying
connected until all the groups are scanned or storing the data in some
magic place so the scan can be resumed if interrupted.

Worst case, perhaps Gnus could create a variant on the Xref header,
which says which folders the article was going to be copied to, and
leave out the message numbers, but that presumably requires Gnus to
download, edit, and upload each message, and it wouldn't work with
something like Sieve doing the refiling as messages come in, which is
where I'd prefer to get eventually.

>    I suspect it might be possible to implement something like it
> for nnimap by looking in the nnmail message-id split log when applying
> marks to a group (i.e., for each article with new marks, look up the
> message id in the split log and find out what other groups it was
> crossposted to and apply the same mark to it).

That means this split log would have to be permanent, not per-session,
so I could read (and mark expirable across groups) in this session a
message I got during the last session.  That per-server data will get
very big, and slow to scan.

And it would need to be shared between different client systems one
might run Gnus on.  On the one hand, that might not be much worse than
the .newsrc.eld file, but I'd kinda like to be able to browse my mail
on my home server from my Gnus session at work on occasion, and vice
versa, and they really shouldn't have to share all the .newsrc.eld
data.  And isn't it part of the point of IMAP that such data lives on
the server?

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>>> My experimentation so far (playing with imtest, don't have the guest
>>> account and nnimap config set up yet) seems to indicate that the marks
>>> won't be propagated.  If this is the case, I think it would be a good
>>> idea for the documentation to indicate this.
>>
>> IMAP doesn't understand the concept of crossposting, I think, so this
>> sounds normal.  How does crosspost mark propagation work for nntp
>> groups?
>
> Using the xref header, I believe, which as far as I can tell, is
> generated on the fly for nnimap, and only lists the current group.

Ah.

> I wonder, is there a way to go through all the groups and ask the
> server if a certain set of message-ids are present in each?

Yes, iterate over the Gnus groups and use `imap-search' for each
group.  E.g., (imap-search "HEADER Message-Id foo@bar"
nnimap-server-buffer).

> It would probably be painfully slow if there are a lot of groups
> (I've got over 200 groups in my nnml hierarchy), and would require
> either staying connected until all the groups are scanned or storing
> the data in some magic place so the scan can be resumed if
> interrupted.

Yes, it doesn't sound like a realistic plan for a default behaviour,
but might be implemented as an add-on workaround.

> Worst case, perhaps Gnus could create a variant on the Xref header,
> which says which folders the article was going to be copied to, and
> leave out the message numbers, but that presumably requires Gnus to
> download, edit, and upload each message, and it wouldn't work with
> something like Sieve doing the refiling as messages come in, which is
> where I'd prefer to get eventually.

Hm.  What about: when marking an article, use the current
nnimap-split-rule to determine which groups would receive the current
message, and then use the imap-search logic above on only those
groups, to find any crossposted copies?  Will probably be rather slow
anyway, and assumes nnimap-split-rule knows about all split rules, but
it is an improvement.

>>    I suspect it might be possible to implement something like it
>> for nnimap by looking in the nnmail message-id split log when applying
>> marks to a group (i.e., for each article with new marks, look up the
>> message id in the split log and find out what other groups it was
>> crossposted to and apply the same mark to it).
>
> That means this split log would have to be permanent, not per-session,
> so I could read (and mark expirable across groups) in this session a
> message I got during the last session.  That per-server data will get
> very big, and slow to scan.

I think the nnmail split-log already is permanent, it is used for,
e.g., `nnmail-split-fancy-with-parent' which have similar needs (e.g.,
find destination groups for earlier messages based on message-id).

> And it would need to be shared between different client systems one
> might run Gnus on.  On the one hand, that might not be much worse than
> the .newsrc.eld file, but I'd kinda like to be able to browse my mail
> on my home server from my Gnus session at work on occasion, and vice
> versa, and they really shouldn't have to share all the .newsrc.eld
> data.  And isn't it part of the point of IMAP that such data lives on
> the server?

Yes.  It would only work if you only use one instance of Gnus, and it
does all the splitting, or if you somehow synchronize the information
between all Gnus instances.

So here is a more IMAPish, but still Gnus-specific, idea:

When nnimap crosspost split an article into several groups, it add
client-specific message flags on each message, indicating the mailbox
and article number of the other crossposted copies.  For example, if
article 4711 in mailbox INBOX.foo have a mark
gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
mark be applied to INBOX.bar:42 as well.  There is one immediate
problem: article numbers may change over time.  Therefor, perhaps just
indicating the destination mailbox name might be enough, and after
knowing the mailbox name, the imap-search logic above could be used.
Alternatively, the mark can contain UIDVALIDITY too, and nnimap would
only try to set the mark on the other articles if the destination
group have the same UIDVALIDITY and the UID still exist.  The latter
alternative would fast, the only delay is opening all other groups to
set flags in.  A minor problem would be that these crosspost marks are
never garbage collected, so they will stick around forever.  Perhaps
that's OK.

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
> Hm.  What about: when marking an article, use the current
> nnimap-split-rule to determine which groups would receive the current
> message, and then use the imap-search logic above on only those
> groups, to find any crossposted copies?  Will probably be rather slow
> anyway, and assumes nnimap-split-rule knows about all split rules, but
> it is an improvement.

Good idea ... except that one of my group names is constructed when
the init file is loaded, using (format-time-string "all.%Y-%m").  And
if I get to use Sieve eventually, the spam rating calculation, at
least, would be likely to differ between Gnus and Sieve.

I want those format-time-string groups to be "all mail I got during
month X and haven't deleted", so I can search for messages I got, but
am not sure from what list, as long as I know approximately when,
without having to pull down a summary of *all* my mail.  If I had a
way to do that with Gnus, I could probably make do with it being one
group instead of one per month.

If I could specify a date range with an nnvirtual group, and have that
range propagated into a search on any component nnimap groups, I could
probably even make do with a huge nnvirtual group to gather together
all the list-oriented and other groups.  That wouldn't eliminate
cross-posting, but it would cut out a lot of it.

> I think the nnmail split-log already is permanent, it is used for,
> e.g., `nnmail-split-fancy-with-parent' which have similar needs (e.g.,
> find destination groups for earlier messages based on message-id).

Ah, you mean the nnmail message-id cache file?  Good point.  But I'd
have to set nnmail-message-id-cache-length to something absurdly
large, or the data will get thrown away.  It defaults to 1000, which
won't have nearly the same performance problems as a complete list.
(And see the aforementioned problems with Sieve.)

> When nnimap crosspost split an article into several groups, it add
> client-specific message flags on each message, indicating the mailbox
> and article number of the other crossposted copies.  For example, if
> article 4711 in mailbox INBOX.foo have a mark
> gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
> mark be applied to INBOX.bar:42 as well.  There is one immediate

I'm told some IMAP servers like Cyrus will limit the number of flags
to something like 32 per mailbox.  Even listing the group names and
not the message numbers would quickly exceed that limit.  A single
flag, gnus-(not-)crossposted, might be enough to tell Gnus whether or
not the other groups need to be scanned, but it would still suck to
have to rescan all of them.

Too bad IMAP doesn't allow arbitrary client-defined string attributes
for messages...

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Good idea ... except that one of my group names is constructed when
> the init file is loaded, using (format-time-string "all.%Y-%m").  And
> if I get to use Sieve eventually, the spam rating calculation, at
> least, would be likely to differ between Gnus and Sieve.

I use the same, although most of my split rules are already moved into
Sieve.  So the idea doesn't work well in these cases.

>> When nnimap crosspost split an article into several groups, it add
>> client-specific message flags on each message, indicating the mailbox
>> and article number of the other crossposted copies.  For example, if
>> article 4711 in mailbox INBOX.foo have a mark
>> gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
>> mark be applied to INBOX.bar:42 as well.  There is one immediate
>
> I'm told some IMAP servers like Cyrus will limit the number of flags
> to something like 32 per mailbox.  Even listing the group names and
> not the message numbers would quickly exceed that limit.  A single
> flag, gnus-(not-)crossposted, might be enough to tell Gnus whether or
> not the other groups need to be scanned, but it would still suck to
> have to rescan all of them.
>
> Too bad IMAP doesn't allow arbitrary client-defined string attributes
> for messages...

Well, IMAP does; the client-specific message flags are just that.  If
servers cannot handle many client-specific flags, then we can't use
this idea, though.  OTOH, it looks difficult to find any other
portable solution, so perhaps we can fix the arbitrary limit in Cyrus
and use this idea anyway.  Otherwise I think we're left with
client-instance-specific solutions like the nnmail message-id cache
backlog, which I agree are no good.

Even increasing the arbitrary limit from 32 to, say, 512, would
suffice for most people, I think.  For each specific mailbox, you
won't crosspost to that many other mailboxes, will you?  Hm.  Except
if you crosspost between a personal mailbox and list mailboxes, then
the personal mailbox will likely have crosspost flags to almost all
other mailboxes, eventually.

Perhaps bring this up with the Cyrus folks?

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Furthermore, if you move all split rules into Sieve, there is no way
to implement even the client-specific message flag idea.  Perhaps a
new sieve extension is warranted for, that add a Xref header (or
something like it), to help locate the mailboxes with crossposted
copies.  That will likely take some time to implement, though.

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Kai Grossjohann]

Ken Raeburn <raeburn@raeburn.org> writes:

> I want those format-time-string groups to be "all mail I got during
> month X and haven't deleted", so I can search for messages I got, but
> am not sure from what list, as long as I know approximately when,
> without having to pull down a summary of *all* my mail.  If I had a
> way to do that with Gnus, I could probably make do with it being one
> group instead of one per month.

Does the IMAP "SEARCH" command allow you to do this?  If so, then it
might be possible to tweak nnir.el so that it can do this.

But the IMAP "SEARCH" command might be slow.

Kai

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
> Well, IMAP does; the client-specific message flags are just that.  If
> servers cannot handle many client-specific flags, then we can't use
> this idea, though.  OTOH, it looks difficult to find any other
> portable solution, so perhaps we can fix the arbitrary limit in Cyrus
> and use this idea anyway.  Otherwise I think we're left with
> client-instance-specific solutions like the nnmail message-id cache
> backlog, which I agree are no good.

I wonder if there's some way to use a message in a magic mailbox to
carry the message-id cache info, somehow.  Updates from multiple
clients connected at the same time would make that an, um,
interesting problem.

> Even increasing the arbitrary limit from 32 to, say, 512, would
> suffice for most people, I think.  For each specific mailbox, you
> won't crosspost to that many other mailboxes, will you?  Hm.  Except
> if you crosspost between a personal mailbox and list mailboxes, then
> the personal mailbox will likely have crosspost flags to almost all
> other mailboxes, eventually.

Even so, for me, 512 group-name flags would still be plenty,
especially if I punt the per-month mailboxes.  That still means
there'd be search time required to scan the listed groups for the
desired message, though.

> Perhaps bring this up with the Cyrus folks?

Maybe.  I should look into it more.  If it means a lot more storage
space per message just on the off chance someone wants to use a 33rd
flag, I probably need to put some positive spin on it. :-)


> Furthermore, if you move all split rules into Sieve, there is no way
> to implement even the client-specific message flag idea.  Perhaps a
> new sieve extension is warranted for, that add a Xref header (or
> something like it), to help locate the mailboxes with crossposted
> copies.  That will likely take some time to implement, though.

Actually, I just found two interesting Internet-Drafts:
draft-melnikov-sieve-imapflags-05.txt, which proposes a technique for
specifying flags on a message to be stored, and
draft-degener-sieve-editheader-01.txt, which proposes a mechanism for
adding and deleting headers on a message (but, despite the name, not
actually for *editing* arbitrary headers).  I'm not in touch with the
Sieve world, though, so I've got no idea how well received these ideas
have been.

The flags approach has the benefit(?) that the set of flags can be
changed after the message has been initially stored -- say, if I use
Gnus to copy/move the message.

Okay, I need to look into the Sieve side more closely, I guess.  And
things look bright enough that I might consider switching soon
anyways, and hope I can get the other bits implemented someday....

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Kai Grossjohann <kai@emptydomain.de> writes:

>> I want those format-time-string groups to be "all mail I got during
>> month X and haven't deleted", so I can search for messages I got, but
>> am not sure from what list, as long as I know approximately when,
>> without having to pull down a summary of *all* my mail.  If I had a
>> way to do that with Gnus, I could probably make do with it being one
>> group instead of one per month.
>
> Does the IMAP "SEARCH" command allow you to do this?  If so, then it
> might be possible to tweak nnir.el so that it can do this.

Yep, according to the RFC, the search criteria include BEFORE <date>
and SINCE <date>.

> But the IMAP "SEARCH" command might be slow.

Only until enough people use it that there's enough pressure on the
implementors to make it faster. :-)

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Even so, for me, 512 group-name flags would still be plenty,
> especially if I punt the per-month mailboxes.  That still means
> there'd be search time required to scan the listed groups for the
> desired message, though.

Searching for flags is typically fast, at least with Cyrus, so this
might not be a problem.  (I don't know if there is a difference
between client-specific flags and otherwise, but I assume not.)