Moving from nnml to nnimap...

Moving from nnml to nnimap...

[Jinhyok Heo]

Now I'm moving from nnml to nnimap and I want to hear some advice from
you.

First of all, is there an easy way to move an existing nnml group to
nnimap one? I did try to use "B c" to move one to a new nnimap group,
but it was too slow for an large or huge group.

Secondly, what size of one nnimap group is appropriate? I think nnimap
will be slower than nnml and it is good to maintain appropriate size
of nnimap groups. Can you give me a rough size?

Thirdly, is there a way to share gnus marks among xemacses on other
machines? I remember I saw an thread regarding this issue. I want to
know the current status of it.

Lastly, I use UW imap server, which doesn't seem to be recommended for
gnus. Does it have any defect unlike Courier or Cyrus imap?

Many questions. :)

Thanks in advance.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" -- a Ladakhi monk

Re: Moving from nnml to nnimap...

[Daniel Pittman]

On Mon, 24 Nov 2003, Jinhyok Heo wrote:
> First of all, is there an easy way to move an existing nnml group to
> nnimap one? I did try to use "B c" to move one to a new nnimap group,
> but it was too slow for an large or huge group.

I never found one, although you could possibly write a routine if you
knew lisp well enough...

> Secondly, what size of one nnimap group is appropriate? I think nnimap
> will be slower than nnml and it is good to maintain appropriate size
> of nnimap groups. Can you give me a rough size?

With a real IMAP server, performance should be as good as, or better
than, Gnus and nnml, in my experience.

> Thirdly, is there a way to share gnus marks among xemacses on other
> machines? I remember I saw an thread regarding this issue. I want to
> know the current status of it.

If you use IMAP, the marks are stored on the server...

> Lastly, I use UW imap server, which doesn't seem to be recommended for
> gnus. 

That would be because it's ... not very good, to put it mildly. The
upstream policy on security issues ("use something else") was also not
inspiring...

> Does it have any defect unlike Courier or Cyrus imap?

Well, the default use of 'mbox' folders will make your performance suck
completely. :)

I strongly recommend the Cyrus IMAP server. While it is much harder to
get up and running than the competition, the performance is great and
it's extremely powerful and useful.

Courier is fine for small groups, but isn't a well featured IMAP server
for the "shared" or "group" stuff that you can do (public and shared
folders, for example).

         Daniel

-- 
Anyone who doubts the value of [probabilistic] statements sould try to
determine the total assets of insurance companies in America. This astonishing
sum has been accumulated through skillful and successful prediction of the
behaviour of large groups of people.
        -- Roy Weatherford, _Philosophical Foundations of Probability Theory_

Re: Moving from nnml to nnimap...

[Ted Zlatanov]

On Mon, 24 Nov 2003, daniel@rimspace.net wrote:

On Mon, 24 Nov 2003, Jinhyok Heo wrote:
>> First of all, is there an easy way to move an existing nnml group
>> to nnimap one? I did try to use "B c" to move one to a new nnimap
>> group, but it was too slow for an large or huge group.
> 
> I never found one, although you could possibly write a routine if
> you knew lisp well enough...

I think it's easiest to condense the nnml group into a single mbox
file and import that into the IMAP server if you have shell access to
it, and it allows mbox storage.  I've done that with UW IMAP, and it
works nicely.  Unfortunately, marks will be lost - if you need those,
and you have some choice in the IMAP server, use one that allows
Maildir storage.  It will be much faster to move articles into the
IMAP server then.

> I strongly recommend the Cyrus IMAP server. While it is much harder
> to get up and running than the competition, the performance is great
> and it's extremely powerful and useful.

I'm also evaluating dovecot for Gnus, I'll let the group know how
performance compares between dovecot and Cyrus, when I have a few
months of experience.  UW IMAP performance with Gnus is pretty bad.
The only reason I'm using UW IMAP is because that's what my ISP
provides.  It is occasionally convenient to access folders as mbox
files. that's the only benefit to UW IMAP.

Ted

Re: Moving from nnml to nnimap...

[Ken Raeburn]

I'm also interested in possibly making the move to nnimap.

I use crossposting a lot in my incoming mail splitting, though.
I want the messages marked for expiration in all groups at the same
time it's marked for expiration in one of the other groups.  Do marks
get propagated for cross-posted articles in nnimap?

Ken

nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

The silence is deafening....  So, is no one actually using
cross-posting when splitting articles in an nnimap server?

My experimentation so far (playing with imtest, don't have the guest
account and nnimap config set up yet) seems to indicate that the marks
won't be propagated.  If this is the case, I think it would be a good
idea for the documentation to indicate this.

(Actually, seen and expire are the only marks I really care about, if
they're done any differently.)

I wrote, in November:
> I use crossposting a lot in my incoming mail splitting, though.
> I want the messages marked for expiration in all groups at the same
> time it's marked for expiration in one of the other groups.  Do marks
> get propagated for cross-posted articles in nnimap?

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> The silence is deafening....  So, is no one actually using
> cross-posting when splitting articles in an nnimap server?
>
> My experimentation so far (playing with imtest, don't have the guest
> account and nnimap config set up yet) seems to indicate that the marks
> won't be propagated.  If this is the case, I think it would be a good
> idea for the documentation to indicate this.

IMAP doesn't understand the concept of crossposting, I think, so this
sounds normal.  How does crosspost mark propagation work for nntp
groups?  I suspect it might be possible to implement something like it
for nnimap by looking in the nnmail message-id split log when applying
marks to a group (i.e., for each article with new marks, look up the
message id in the split log and find out what other groups it was
crossposted to and apply the same mark to it).

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
>> My experimentation so far (playing with imtest, don't have the guest
>> account and nnimap config set up yet) seems to indicate that the marks
>> won't be propagated.  If this is the case, I think it would be a good
>> idea for the documentation to indicate this.
>
> IMAP doesn't understand the concept of crossposting, I think, so this
> sounds normal.  How does crosspost mark propagation work for nntp
> groups?

Using the xref header, I believe, which as far as I can tell, is
generated on the fly for nnimap, and only lists the current group.

I wonder, is there a way to go through all the groups and ask the
server if a certain set of message-ids are present in each?  It would
probably be painfully slow if there are a lot of groups (I've got over
200 groups in my nnml hierarchy), and would require either staying
connected until all the groups are scanned or storing the data in some
magic place so the scan can be resumed if interrupted.

Worst case, perhaps Gnus could create a variant on the Xref header,
which says which folders the article was going to be copied to, and
leave out the message numbers, but that presumably requires Gnus to
download, edit, and upload each message, and it wouldn't work with
something like Sieve doing the refiling as messages come in, which is
where I'd prefer to get eventually.

>    I suspect it might be possible to implement something like it
> for nnimap by looking in the nnmail message-id split log when applying
> marks to a group (i.e., for each article with new marks, look up the
> message id in the split log and find out what other groups it was
> crossposted to and apply the same mark to it).

That means this split log would have to be permanent, not per-session,
so I could read (and mark expirable across groups) in this session a
message I got during the last session.  That per-server data will get
very big, and slow to scan.

And it would need to be shared between different client systems one
might run Gnus on.  On the one hand, that might not be much worse than
the .newsrc.eld file, but I'd kinda like to be able to browse my mail
on my home server from my Gnus session at work on occasion, and vice
versa, and they really shouldn't have to share all the .newsrc.eld
data.  And isn't it part of the point of IMAP that such data lives on
the server?

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>>> My experimentation so far (playing with imtest, don't have the guest
>>> account and nnimap config set up yet) seems to indicate that the marks
>>> won't be propagated.  If this is the case, I think it would be a good
>>> idea for the documentation to indicate this.
>>
>> IMAP doesn't understand the concept of crossposting, I think, so this
>> sounds normal.  How does crosspost mark propagation work for nntp
>> groups?
>
> Using the xref header, I believe, which as far as I can tell, is
> generated on the fly for nnimap, and only lists the current group.

Ah.

> I wonder, is there a way to go through all the groups and ask the
> server if a certain set of message-ids are present in each?

Yes, iterate over the Gnus groups and use `imap-search' for each
group.  E.g., (imap-search "HEADER Message-Id foo@bar"
nnimap-server-buffer).

> It would probably be painfully slow if there are a lot of groups
> (I've got over 200 groups in my nnml hierarchy), and would require
> either staying connected until all the groups are scanned or storing
> the data in some magic place so the scan can be resumed if
> interrupted.

Yes, it doesn't sound like a realistic plan for a default behaviour,
but might be implemented as an add-on workaround.

> Worst case, perhaps Gnus could create a variant on the Xref header,
> which says which folders the article was going to be copied to, and
> leave out the message numbers, but that presumably requires Gnus to
> download, edit, and upload each message, and it wouldn't work with
> something like Sieve doing the refiling as messages come in, which is
> where I'd prefer to get eventually.

Hm.  What about: when marking an article, use the current
nnimap-split-rule to determine which groups would receive the current
message, and then use the imap-search logic above on only those
groups, to find any crossposted copies?  Will probably be rather slow
anyway, and assumes nnimap-split-rule knows about all split rules, but
it is an improvement.

>>    I suspect it might be possible to implement something like it
>> for nnimap by looking in the nnmail message-id split log when applying
>> marks to a group (i.e., for each article with new marks, look up the
>> message id in the split log and find out what other groups it was
>> crossposted to and apply the same mark to it).
>
> That means this split log would have to be permanent, not per-session,
> so I could read (and mark expirable across groups) in this session a
> message I got during the last session.  That per-server data will get
> very big, and slow to scan.

I think the nnmail split-log already is permanent, it is used for,
e.g., `nnmail-split-fancy-with-parent' which have similar needs (e.g.,
find destination groups for earlier messages based on message-id).

> And it would need to be shared between different client systems one
> might run Gnus on.  On the one hand, that might not be much worse than
> the .newsrc.eld file, but I'd kinda like to be able to browse my mail
> on my home server from my Gnus session at work on occasion, and vice
> versa, and they really shouldn't have to share all the .newsrc.eld
> data.  And isn't it part of the point of IMAP that such data lives on
> the server?

Yes.  It would only work if you only use one instance of Gnus, and it
does all the splitting, or if you somehow synchronize the information
between all Gnus instances.

So here is a more IMAPish, but still Gnus-specific, idea:

When nnimap crosspost split an article into several groups, it add
client-specific message flags on each message, indicating the mailbox
and article number of the other crossposted copies.  For example, if
article 4711 in mailbox INBOX.foo have a mark
gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
mark be applied to INBOX.bar:42 as well.  There is one immediate
problem: article numbers may change over time.  Therefor, perhaps just
indicating the destination mailbox name might be enough, and after
knowing the mailbox name, the imap-search logic above could be used.
Alternatively, the mark can contain UIDVALIDITY too, and nnimap would
only try to set the mark on the other articles if the destination
group have the same UIDVALIDITY and the UID still exist.  The latter
alternative would fast, the only delay is opening all other groups to
set flags in.  A minor problem would be that these crosspost marks are
never garbage collected, so they will stick around forever.  Perhaps
that's OK.

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
> Hm.  What about: when marking an article, use the current
> nnimap-split-rule to determine which groups would receive the current
> message, and then use the imap-search logic above on only those
> groups, to find any crossposted copies?  Will probably be rather slow
> anyway, and assumes nnimap-split-rule knows about all split rules, but
> it is an improvement.

Good idea ... except that one of my group names is constructed when
the init file is loaded, using (format-time-string "all.%Y-%m").  And
if I get to use Sieve eventually, the spam rating calculation, at
least, would be likely to differ between Gnus and Sieve.

I want those format-time-string groups to be "all mail I got during
month X and haven't deleted", so I can search for messages I got, but
am not sure from what list, as long as I know approximately when,
without having to pull down a summary of *all* my mail.  If I had a
way to do that with Gnus, I could probably make do with it being one
group instead of one per month.

If I could specify a date range with an nnvirtual group, and have that
range propagated into a search on any component nnimap groups, I could
probably even make do with a huge nnvirtual group to gather together
all the list-oriented and other groups.  That wouldn't eliminate
cross-posting, but it would cut out a lot of it.

> I think the nnmail split-log already is permanent, it is used for,
> e.g., `nnmail-split-fancy-with-parent' which have similar needs (e.g.,
> find destination groups for earlier messages based on message-id).

Ah, you mean the nnmail message-id cache file?  Good point.  But I'd
have to set nnmail-message-id-cache-length to something absurdly
large, or the data will get thrown away.  It defaults to 1000, which
won't have nearly the same performance problems as a complete list.
(And see the aforementioned problems with Sieve.)

> When nnimap crosspost split an article into several groups, it add
> client-specific message flags on each message, indicating the mailbox
> and article number of the other crossposted copies.  For example, if
> article 4711 in mailbox INBOX.foo have a mark
> gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
> mark be applied to INBOX.bar:42 as well.  There is one immediate

I'm told some IMAP servers like Cyrus will limit the number of flags
to something like 32 per mailbox.  Even listing the group names and
not the message numbers would quickly exceed that limit.  A single
flag, gnus-(not-)crossposted, might be enough to tell Gnus whether or
not the other groups need to be scanned, but it would still suck to
have to rescan all of them.

Too bad IMAP doesn't allow arbitrary client-defined string attributes
for messages...

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Good idea ... except that one of my group names is constructed when
> the init file is loaded, using (format-time-string "all.%Y-%m").  And
> if I get to use Sieve eventually, the spam rating calculation, at
> least, would be likely to differ between Gnus and Sieve.

I use the same, although most of my split rules are already moved into
Sieve.  So the idea doesn't work well in these cases.

>> When nnimap crosspost split an article into several groups, it add
>> client-specific message flags on each message, indicating the mailbox
>> and article number of the other crossposted copies.  For example, if
>> article 4711 in mailbox INBOX.foo have a mark
>> gnus-crosspost-INBOX.bar-42, then when marking INBOX.foo:4711 the same
>> mark be applied to INBOX.bar:42 as well.  There is one immediate
>
> I'm told some IMAP servers like Cyrus will limit the number of flags
> to something like 32 per mailbox.  Even listing the group names and
> not the message numbers would quickly exceed that limit.  A single
> flag, gnus-(not-)crossposted, might be enough to tell Gnus whether or
> not the other groups need to be scanned, but it would still suck to
> have to rescan all of them.
>
> Too bad IMAP doesn't allow arbitrary client-defined string attributes
> for messages...

Well, IMAP does; the client-specific message flags are just that.  If
servers cannot handle many client-specific flags, then we can't use
this idea, though.  OTOH, it looks difficult to find any other
portable solution, so perhaps we can fix the arbitrary limit in Cyrus
and use this idea anyway.  Otherwise I think we're left with
client-instance-specific solutions like the nnmail message-id cache
backlog, which I agree are no good.

Even increasing the arbitrary limit from 32 to, say, 512, would
suffice for most people, I think.  For each specific mailbox, you
won't crosspost to that many other mailboxes, will you?  Hm.  Except
if you crosspost between a personal mailbox and list mailboxes, then
the personal mailbox will likely have crosspost flags to almost all
other mailboxes, eventually.

Perhaps bring this up with the Cyrus folks?

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Furthermore, if you move all split rules into Sieve, there is no way
to implement even the client-specific message flag idea.  Perhaps a
new sieve extension is warranted for, that add a Xref header (or
something like it), to help locate the mailboxes with crossposted
copies.  That will likely take some time to implement, though.

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Simon Josefsson <jas@extundo.com> writes:
> Well, IMAP does; the client-specific message flags are just that.  If
> servers cannot handle many client-specific flags, then we can't use
> this idea, though.  OTOH, it looks difficult to find any other
> portable solution, so perhaps we can fix the arbitrary limit in Cyrus
> and use this idea anyway.  Otherwise I think we're left with
> client-instance-specific solutions like the nnmail message-id cache
> backlog, which I agree are no good.

I wonder if there's some way to use a message in a magic mailbox to
carry the message-id cache info, somehow.  Updates from multiple
clients connected at the same time would make that an, um,
interesting problem.

> Even increasing the arbitrary limit from 32 to, say, 512, would
> suffice for most people, I think.  For each specific mailbox, you
> won't crosspost to that many other mailboxes, will you?  Hm.  Except
> if you crosspost between a personal mailbox and list mailboxes, then
> the personal mailbox will likely have crosspost flags to almost all
> other mailboxes, eventually.

Even so, for me, 512 group-name flags would still be plenty,
especially if I punt the per-month mailboxes.  That still means
there'd be search time required to scan the listed groups for the
desired message, though.

> Perhaps bring this up with the Cyrus folks?

Maybe.  I should look into it more.  If it means a lot more storage
space per message just on the off chance someone wants to use a 33rd
flag, I probably need to put some positive spin on it. :-)


> Furthermore, if you move all split rules into Sieve, there is no way
> to implement even the client-specific message flag idea.  Perhaps a
> new sieve extension is warranted for, that add a Xref header (or
> something like it), to help locate the mailboxes with crossposted
> copies.  That will likely take some time to implement, though.

Actually, I just found two interesting Internet-Drafts:
draft-melnikov-sieve-imapflags-05.txt, which proposes a technique for
specifying flags on a message to be stored, and
draft-degener-sieve-editheader-01.txt, which proposes a mechanism for
adding and deleting headers on a message (but, despite the name, not
actually for *editing* arbitrary headers).  I'm not in touch with the
Sieve world, though, so I've got no idea how well received these ideas
have been.

The flags approach has the benefit(?) that the set of flags can be
changed after the message has been initially stored -- say, if I use
Gnus to copy/move the message.

Okay, I need to look into the Sieve side more closely, I guess.  And
things look bright enough that I might consider switching soon
anyways, and hope I can get the other bits implemented someday....

Ken

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Simon Josefsson]

Ken Raeburn <raeburn@raeburn.org> writes:

> Even so, for me, 512 group-name flags would still be plenty,
> especially if I punt the per-month mailboxes.  That still means
> there'd be search time required to scan the listed groups for the
> desired message, though.

Searching for flags is typically fast, at least with Cyrus, so this
might not be a problem.  (I don't know if there is a difference
between client-specific flags and otherwise, but I assume not.)

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Kai Grossjohann]

Ken Raeburn <raeburn@raeburn.org> writes:

> I want those format-time-string groups to be "all mail I got during
> month X and haven't deleted", so I can search for messages I got, but
> am not sure from what list, as long as I know approximately when,
> without having to pull down a summary of *all* my mail.  If I had a
> way to do that with Gnus, I could probably make do with it being one
> group instead of one per month.

Does the IMAP "SEARCH" command allow you to do this?  If so, then it
might be possible to tweak nnir.el so that it can do this.

But the IMAP "SEARCH" command might be slow.

Kai

Re: nnimap and crossposting (Re: Moving from nnml to nnimap...)

[Ken Raeburn]

Kai Grossjohann <kai@emptydomain.de> writes:

>> I want those format-time-string groups to be "all mail I got during
>> month X and haven't deleted", so I can search for messages I got, but
>> am not sure from what list, as long as I know approximately when,
>> without having to pull down a summary of *all* my mail.  If I had a
>> way to do that with Gnus, I could probably make do with it being one
>> group instead of one per month.
>
> Does the IMAP "SEARCH" command allow you to do this?  If so, then it
> might be possible to tweak nnir.el so that it can do this.

Yep, according to the RFC, the search criteria include BEFORE <date>
and SINCE <date>.

> But the IMAP "SEARCH" command might be slow.

Only until enough people use it that there's enough pressure on the
implementors to make it faster. :-)

Ken

moving from nnml to nnimap

[Randal L. Schwartz]

I have about 300 nnml groups, some dating back a decade or so.

I'd like to move them to IMAP, so I can get to them offbox and also
with Mail.app on my iphone.

I already have my incoming mail split to both
.incoming/[nnml-group-name] and IMAP folders, and I can see
nnimap+localhost:INBOX just fine.

First, is there info in the nnml format (like marks) that I'll lose when
I go to nnimap?

Second, is there an easy way to take everything from nnml:MUMBLE and
move it to nnimap:MUMBLE while retaining as much meta-info as possible?

Thanks.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Mon, 14 Mar 2011 12:55:13 -0700 merlyn@stonehenge.com (Randal L. Schwartz) wrote: 

RLS> I have about 300 nnml groups, some dating back a decade or so.

RLS> I'd like to move them to IMAP, so I can get to them offbox and also
RLS> with Mail.app on my iphone.

RLS> I already have my incoming mail split to both
RLS> .incoming/[nnml-group-name] and IMAP folders, and I can see
RLS> nnimap+localhost:INBOX just fine.

RLS> First, is there info in the nnml format (like marks) that I'll lose when
RLS> I go to nnimap?

No, but the marks will now live on the IMAP server instead of locally.

RLS> Second, is there an easy way to take everything from nnml:MUMBLE and
RLS> move it to nnimap:MUMBLE while retaining as much meta-info as possible?

I don't know of one.  You basically need to do (open group with C-u to
see all articles, mark all articles, copy to 
(gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
automated I guess, but no one has needed the automation badly enough.
If you want I can write a function to do it.

Ted

Re: moving from nnml to nnimap

[Dave Goldberg]

RLS> Second, is there an easy way to take everything from nnml:MUMBLE and
RLS> move it to nnimap:MUMBLE while retaining as much meta-info as possible?

> I don't know of one.  You basically need to do (open group with C-u to
> see all articles, mark all articles, copy to 
> (gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
> automated I guess, but no one has needed the automation badly enough.
> If you want I can write a function to do it.

I did this a number of years ago.  Going through my .xemacs/init.el version history I find this:

(defun nnml2nnimap-group ()
  (interactive)
  (gnus-group-make-group
   (concat "INBOX."
	   (buffer-substring
	    (region-beginning)
	    (region-end)))
   "nnimap:cyrus"))

which I used to create a destination IMAP group for each nnml group.

I can't find what I wrote to do the actual moves.  My memory says I did it with a keyboard macro, editing the destination group in the macro before each run, but apparently I never saved that out.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't know of one.  You basically need to do (open group with C-u to
> see all articles, mark all articles, copy to 
> (gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
> automated I guess, but no one has needed the automation badly enough.
> If you want I can write a function to do it.

It sounds like a good idea.  The ability to, say, slurp down all
groups/messages from IMAP server A to move them to server B sounds
useful, too, so this should be a quite general command.  Perhaps in the
server buffer?  "Copy (or move?) all messages from server Foo to server
Bar?"

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Tue, 15 Mar 2011 16:28:00 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I don't know of one.  You basically need to do (open group with C-u to
>> see all articles, mark all articles, copy to 
>> (gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
>> automated I guess, but no one has needed the automation badly enough.
>> If you want I can write a function to do it.

LMI> It sounds like a good idea.  The ability to, say, slurp down all
LMI> groups/messages from IMAP server A to move them to server B sounds
LMI> useful, too, so this should be a quite general command.  Perhaps in the
LMI> server buffer?  "Copy (or move?) all messages from server Foo to server
LMI> Bar?"

It's extremely likely to get interrupted, though.  So really what you
want is something like imapsync but for Gnus.  Perhaps that's better
left as a Cloudy Gnus TODO item.

Ted

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's extremely likely to get interrupted, though.  So really what you
> want is something like imapsync but for Gnus.

Yes.  But it should be possible to keep track of what's been copied
over, so that the copying process can be restarted when it (inevitably)
fails.  So it'll have to be implemented carefully, but it would be
useful.

> Perhaps that's better left as a Cloudy Gnus TODO item.

It's kinda orthogonal to the Cloudy functionality...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Tue, 15 Mar 2011 16:58:32 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> It's extremely likely to get interrupted, though.  So really what you
>> want is something like imapsync but for Gnus.

LMI> Yes.  But it should be possible to keep track of what's been copied
LMI> over, so that the copying process can be restarted when it (inevitably)
LMI> fails.  So it'll have to be implemented carefully, but it would be
LMI> useful.

>> Perhaps that's better left as a Cloudy Gnus TODO item.

LMI> It's kinda orthogonal to the Cloudy functionality...

I don't mean it will be "cloudy" functionality, only that it's major
enough that I'd like to do it after No Gnus is released.

Ted

P.S. Ha ha!  I've got you saying "Cloudy Gnus" too!

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't mean it will be "cloudy" functionality, only that it's major
> enough that I'd like to do it after No Gnus is released.

True, but you don't have to wait if you just want to start implementing
it right now, either.  :-)

> P.S. Ha ha!  I've got you saying "Cloudy Gnus" too!

Eek!

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Reiner Steib]

On Tue, Mar 15 2011, Lars Magne Ingebrigtsen wrote:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> I don't know of one.  You basically need to do (open group with C-u to
>> see all articles, mark all articles, copy to 
>> (gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
>> automated I guess, but no one has needed the automation badly enough.
>> If you want I can write a function to do it.
>
> It sounds like a good idea.  The ability to, say, slurp down all
> groups/messages from IMAP server A to move them to server B sounds
> useful, too, so this should be a quite general command.  Perhaps in the
> server buffer?  "Copy (or move?) all messages from server Foo to server
> Bar?"

I have been using this code[1]:

--8<---------------cut here---------------start------------->8---
(defvar rs-gnus-migrate-old-group-regexp "^nnmaildir.personal:"
  "Regexp matching old groups.")
(defvar rs-gnus-migrate-target-group-prefix "nnml+personal:"
  "Prefix for new groups.")
;; (defvar rs-gnus-migrate-transform-function nil
;;   ;;  'rs-gnus-migrate-transform
;;   "Transform old name to new name")
;; (defun rs-gnus-migrate-transform (oldname)
;;   "Transform old name, return new name"
;;   (gnus-replace-in-string oldname "^.*:\\(News\\|Lists\\|Root\\)\\." ""))
 
;;;###autoload
(defun rs-gnus-migrate-loop (&optional copy)
  (interactive)
  (let (gnus-large-newsgroup
	gnus-parameters
	case-fold-search
	g-list full oldmethod g-name fulltarget
	level)
    (dolist (group gnus-newsrc-alist) ;; loop over all groups
      (setq full (car group))
      (setq oldmethod
	    (gnus-replace-in-string
	     full (regexp-quote (gnus-replace-in-string full "^.*:" "")) ""))
      (if (fboundp rs-gnus-migrate-transform-function)
	  (setq g-name (funcall rs-gnus-migrate-transform-function full))
	(setq g-name (gnus-replace-in-string full "^.*:" "")))
      ;;
      (setq fulltarget (concat rs-gnus-migrate-target-group-prefix g-name))
      (when (string-match rs-gnus-migrate-old-group-regexp full)
	(when (string= oldmethod "")
	  (setq oldmethod nil))
	(gnus-message 5
		      "full=`%s',\n oldmethod=`%s',\n g-name=`%s',\n fulltarget=`%s'"
		      full oldmethod g-name fulltarget)
	;;))));; when DUMMY
	;; Create target group next to the old one:
	(gnus-group-jump-to-group full) ;; j
	(setq level (gnus-group-group-level))
	;; create target group unless it already exists:
	(save-excursion
	  (if (gnus-gethash fulltarget gnus-newsrc-hashtb)
	      (gnus-group-jump-to-group fulltarget)
	    (gnus-group-make-group
	     g-name
	     ;; FIXME: Use Gnus functions instead of regexps.  Which one?
	     ;; FIXME: Use gnus-group-real-name?
	     (gnus-replace-in-string rs-gnus-migrate-target-group-prefix
				     "^\\([^+:]*\\).*$" "\\1")
	     (gnus-replace-in-string rs-gnus-migrate-target-group-prefix
				     "^[^+]+\\+\\([^:]+\\).*$" "\\1")))
	  (gnus-group-set-current-level nil level))
	(if (and (gnus-activate-group full)
		 (gnus-group-quick-select-group t)) ;; 0 M-RET
	    (progn
	      (gnus-summary-toggle-threads -1)
	      (gnus-summary-sort-by-date) ;; optional
	      (gnus-uu-mark-buffer) ;; M P b
	      ;; Maybe use `-move-' here:
	      (if copy
		  (gnus-summary-copy-article nil fulltarget)
		(gnus-summary-move-article nil fulltarget))
	      (sit-for 1)
	      (gnus-summary-exit))
	  (gnus-message 1 "Skipped inactivate group %s" full))
	;; Maybe unsubscribe/delete old group here:
	;; ...
	(gnus-group-change-level full gnus-level-killed)
	(gnus-message 7 "Processed `%s'" g-name)
	(sit-for 1)))))
--8<---------------cut here---------------end--------------->8---

[1] See also ...

,----[ http://article.gmane.org/gmane.emacs.gnus.general/53300 ]
| From: Reiner Steib
| Subject: Re: POP to IMAP
| Newsgroups: gmane.emacs.gnus.general
| Date: 2003-07-01 13:33:22 GMT
`----

Bye, Reiner.


-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Thu, 17 Mar 2011 09:13:50 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Tue, Mar 15 2011, Lars Magne Ingebrigtsen wrote:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> I don't know of one.  You basically need to do (open group with C-u to
>>> see all articles, mark all articles, copy to 
>>> (gnus-newsgroup-name =~ s/nnml/nnimap/) ) 300 times.  It can be
>>> automated I guess, but no one has needed the automation badly enough.
>>> If you want I can write a function to do it.
>> 
>> It sounds like a good idea.  The ability to, say, slurp down all
>> groups/messages from IMAP server A to move them to server B sounds
>> useful, too, so this should be a quite general command.  Perhaps in the
>> server buffer?  "Copy (or move?) all messages from server Foo to server
>> Bar?"

RS> (defun rs-gnus-migrate-loop (&optional copy)

Any chance this can be added to Gnus?

Ted

Re: moving from nnml to nnimap

[Randal L. Schwartz]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

Reiner> I have been using this code[1]:

Speaking of which, since I'm in the right place now to ask this, the
quoted code was in a completely unreadable black-type-on-dark-blue.
Sure, I can blindly cut-n-paste it, but it'd be nice to read it "in
place".

(a) why is this ugly combo the default?
(2) how do I customize that?  I'm not even sure how to search.
    M-x customize-ugly-dark-blue-on-black-when-in-quoted-mail  :)


-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> (a) why is this ugly combo the default?
> (2) how do I customize that?  I'm not even sure how to search.
>     M-x customize-ugly-dark-blue-on-black-when-in-quoted-mail  :)

(face-at-point) will tell you the name of the face under point.  In this
instance, it's `mm-uu-extract'.

I think it must be this part of the spec:

			 (((type tty)
			   (class color)
			   (background dark))
			  (:background "dark blue"))


Are you running your Emacs on a tty with a dark background?  Perhaps
`set-background-color' would allow Emacs to become unconfused...
                          
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Randal L. Schwartz]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Lars> 			 (((type tty)
Lars> 			   (class color)
Lars> 			   (background dark))
Lars> 			  (:background "dark blue"))

No, my tty is a light background, so the letters are black.  And thus, I
get black on dark blue.  Ugh.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> No, my tty is a light background, so the letters are black.  And thus, I
> get black on dark blue.  Ugh.

Does

(set-background-color "white")

in your .emacs fix the problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Randal L. Schwartz]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Lars> (set-background-color "white")

Lars> in your .emacs fix the problem?

It does!  So why doesn't emacs know that already?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion

Re: moving from nnml to nnimap

[Lars Magne Ingebrigtsen]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> It does!  So why doesn't emacs know that already?

When running Emacs on a tty, it has no way to query the tty for what
background colour it's using.  It defaults to assuming that the
background colour is black, which is probably as likely as anything
else.

So you have to tell Emacs what the background colour it's running under.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: moving from nnml to nnimap

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> No, but the marks will now live on the IMAP server instead of locally.

Also, gnus uses some custom marks that other MUAs may not recognize.

What IMAP server, btw?  If the choice hasn't already been made, I
recommend dovecot.  Easy to set up, easy to maintain, and it has been
completely stable for me, for the last 2 years 25 weeks and 5 days
(that's when I did a switch from cyrus, and... when all of my IMAP pains
went away).

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Wed, 16 Mar 2011 09:38:47 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> No, but the marks will now live on the IMAP server instead of locally.

SB> Also, gnus uses some custom marks that other MUAs may not recognize.

This is OK, IMAP supports it (well, it's an extension, but most IMAP
servers have it these days).

SB> What IMAP server, btw?  If the choice hasn't already been made, I
SB> recommend dovecot.  Easy to set up, easy to maintain, and it has been
SB> completely stable for me, for the last 2 years 25 weeks and 5 days
SB> (that's when I did a switch from cyrus, and... when all of my IMAP pains
SB> went away).

Yeah, my experience has been that Dovecot > Courier >= Cyrus > Zimbra

Ted

Re: moving from nnml to nnimap

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

SB> Also, gnus uses some custom marks that other MUAs may not recognize.

> This is OK, IMAP supports it (well, it's an extension, but most IMAP
> servers have it these days).

Yup.  From a gnus POV, no information is lost, when moving from nnml to
nnimap.  But other MUAs (like eg. the Mail.App of iPhone the OP
mentioned), may not be able to utilize them.

Re: moving from nnml to nnimap

[Ted Zlatanov]

On Thu, 17 Mar 2011 09:12:07 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
SB> Also, gnus uses some custom marks that other MUAs may not recognize.

>> This is OK, IMAP supports it (well, it's an extension, but most IMAP
>> servers have it these days).

SB> Yup.  From a gnus POV, no information is lost, when moving from nnml to
SB> nnimap.  But other MUAs (like eg. the Mail.App of iPhone the OP
SB> mentioned), may not be able to utilize them.

They couldn't use nnml at all, so it's better anyway in the end :)

Ted