Migrate from Maildir/IMAP/Exchange to NNML

Migrate from Maildir/IMAP/Exchange to NNML

[brettrandall]

Hey everyone

I used to use Gnus as my mail client, years ago... Used it for about 5
years. It was great. Then I needed to have more integration with our
internal Exchange server so had to move away to Outlook. Now I'm back
to needing Gnus. Outlook, kmail, Evolution, Entourage - they all suck.
I need a mail/news client that lets me send an e-mail in a handful of
keystrokes rather than needing to click all over the place and wait
for God-only-knows-what to finish doing what its doing in the
background. Only problem is I need to move all my mail (3gb) of it
back from the Exchange server.

I've sync'd all 3gb of mail to a local Maildir using offlineimap. I've
tried using nnimap to read my mail (by using Courier IMAP as a local
server and pointing it to the sync'd Maildir), however its not as
stable as I'd like. So, I'd like to turn all my IMAP folders (~/
Maildir/.TopLevelFolder, ~/Maildir/.TopLevelFolder.2ndLevelFolder,
etc) into NNML folders. Has anyone got a script or other that can
already do this? I don't care if it uses the locally sync'd Maildir,
whether it talks to an IMAP server (local or remote), or if it reads
from a PST - I just need to figure a way of converting all my folders
to NNML and then getting Gnus reading them.

TIA

Brett.

Re: Migrate from Maildir/IMAP/Exchange to NNML

[Nicolas KOWALSKI]

brettrandall <brett.randall@gmail.com> writes:

> Hey everyone

Hello,

> Has anyone got a script or other that can already do this? I don't
> care if it uses the locally sync'd Maildir, whether it talks to an
> IMAP server (local or remote), or if it reads from a PST - I just
> need to figure a way of converting all my folders to NNML and then
> getting Gnus reading them.

I would use 'mailutil', coming from UW-imapd distribution. The general
idea is to transfer mails from an IMAP server to local 'mx' format.
The 'mx' format works like 'mh', so the data files are fully readable
in the Gnus nnml backend after rebuilding its databases using 
`M-x nnml-generate-nov-databases'.

- the command I use is:

mailutil transfer -v -m append '{localhost/notls/norsh/user=niko}' '#driver.mx//home/niko/Mail/'


- then I remove the mx index files:

find /home/niko/Mail -type f -name .mxindex -exec rm -f {} \;


- then I rebuild the nnml databases in Gnus.


Hope this helps,
-- 
Nicolas

Re: Migrate from Maildir/IMAP/Exchange to NNML

[Ted Zlatanov]

On Sun, 20 Apr 2008 20:00:37 -0700 (PDT) brettrandall <brett.randall@gmail.com> wrote: 

b> Hey everyone
b> I used to use Gnus as my mail client, years ago... Used it for about 5
b> years. It was great. Then I needed to have more integration with our
b> internal Exchange server so had to move away to Outlook. Now I'm back
b> to needing Gnus. Outlook, kmail, Evolution, Entourage - they all suck.
b> I need a mail/news client that lets me send an e-mail in a handful of
b> keystrokes rather than needing to click all over the place and wait
b> for God-only-knows-what to finish doing what its doing in the
b> background. Only problem is I need to move all my mail (3gb) of it
b> back from the Exchange server.

b> I've sync'd all 3gb of mail to a local Maildir using offlineimap. I've
b> tried using nnimap to read my mail (by using Courier IMAP as a local
b> server and pointing it to the sync'd Maildir), however its not as
b> stable as I'd like. So, I'd like to turn all my IMAP folders (~/
b> Maildir/.TopLevelFolder, ~/Maildir/.TopLevelFolder.2ndLevelFolder,
b> etc) into NNML folders. Has anyone got a script or other that can
b> already do this? I don't care if it uses the locally sync'd Maildir,
b> whether it talks to an IMAP server (local or remote), or if it reads
b> from a PST - I just need to figure a way of converting all my folders
b> to NNML and then getting Gnus reading them.

Hi,

Gnus has a native Maildir backend (nnmaildir) which is very good.  Would
that work for you?  It will work fine with your existing setup without
migration work.

Ted

Re: Migrate from Maildir/IMAP/Exchange to NNML

[brettrandall]

On Apr 22, 1:35 am, Ted Zlatanov <t...@lifelogs.com> wrote:

> Gnus has a native Maildir backend (nnmaildir) which is very good.  Would
> that work for you?  It will work fine with your existing setup without
> migration work.

Do you know what it will do with the "subfolders" in the format they
are in at the moment? (Maildir/.TopLevelFolder,
Maildir/.TopLevelFolder.2ndLevelFolder), etc? I have a look in the
Gnus manual and it doesn't seem to mention how it would deal with that
- whether it would even show them, and if so, how they would show up.
I guess I could just give it a go and see what happens :)

Thanks

Brett.

Re: Migrate from Maildir/IMAP/Exchange to NNML

[brettrandall]

On Apr 21, 5:47 pm, Nicolas KOWALSKI <n...@petole.dyndns.org> wrote:
> brettrandall <brett.rand...@gmail.com> writes:
> > Has anyone got a script or other that can already do this? I don't
> > care if it uses the locally sync'd Maildir, whether it talks to an
> > IMAP server (local or remote), or if it reads from a PST - I just
> > need to figure a way of converting all my folders to NNML and then
> > getting Gnus reading them.
>
> I would use 'mailutil', coming from UW-imapd distribution. The general
> idea is to transfer mails from an IMAP server to local 'mx' format.
> The 'mx' format works like 'mh', so the data files are fully readable
> in the Gnus nnml backend after rebuilding its databases using
> `M-x nnml-generate-nov-databases'.

That sounds like a great idea. I'll give that a go today and report
back. Thanks for your help!

Brett.

Re: Migrate from Maildir/IMAP/Exchange to NNML

[brettrandall]

On Apr 22, 7:24 am, brettrandall <brett.rand...@gmail.com> wrote:
> On Apr 21, 5:47 pm, Nicolas KOWALSKI <n...@petole.dyndns.org> wrote:
>
> > brettrandall <brett.rand...@gmail.com> writes:
> > > Has anyone got a script or other that can already do this? I don't
> > > care if it uses the locally sync'd Maildir, whether it talks to an
> > > IMAP server (local or remote), or if it reads from a PST - I just
> > > need to figure a way of converting all my folders to NNML and then
> > > getting Gnus reading them.
>
> > I would use 'mailutil', coming from UW-imapd distribution. The general
> > idea is to transfer mails from an IMAP server to local 'mx' format.
> > The 'mx' format works like 'mh', so the data files are fully readable
> > in the Gnus nnml backend after rebuilding its databases using
> > `M-x nnml-generate-nov-databases'.
>
> That sounds like a great idea. I'll give that a go today and report
> back. Thanks for your help!

Ok I've given that tool a go but it comes up with an error. This is
probably not the best place to post this question but just in case
you've had this before I thought I'd start here:

brett@secludedvalley:~$ mailutil transfer -v '{imapserver/notls/norsh/
user=brett}INBOX' '#driver.mx//home/brett/Mail/'
[Trying IP address [10.0.0.34]]
[Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1
(imapserver) ready.]
password:
Listing mailboxes...
Copying {imapserver:143/imap/notls/user="brett"}INBOX/Personal
  => #driver.mx//home/brett/Mail/INBOX/Personal
[Reusing connection to imapserver/user="brett"]
[Mailbox is empty]
Can't create mailbox /home/brett/Mail/INBOX/Personal: invalid MX-
format name
[Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1
signing off.]
brett@secludedvalley:~$

Any idea what that "invalid MX-format name" is? I tried it without
"INBOX" and it died on the "Calendar" folder.

If no idea I'll post this off to a UW list somewhere. Thanks!

Brett.

Re: Migrate from Maildir/IMAP/Exchange to NNML

[brettrandall]

On Apr 22, 9:33 am, brettrandall <brett.rand...@gmail.com> wrote:
> On Apr 22, 7:24 am, brettrandall <brett.rand...@gmail.com> wrote:
>
>
>
> > On Apr 21, 5:47 pm, Nicolas KOWALSKI <n...@petole.dyndns.org> wrote:
>
> > > brettrandall <brett.rand...@gmail.com> writes:
> > > > Has anyone got a script or other that can already do this? I don't
> > > > care if it uses the locally sync'd Maildir, whether it talks to an
> > > > IMAP server (local or remote), or if it reads from a PST - I just
> > > > need to figure a way of converting all my folders to NNML and then
> > > > getting Gnus reading them.
>
> > > I would use 'mailutil', coming from UW-imapd distribution. The general
> > > idea is to transfer mails from an IMAP server to local 'mx' format.
> > > The 'mx' format works like 'mh', so the data files are fully readable
> > > in the Gnus nnml backend after rebuilding its databases using
> > > `M-x nnml-generate-nov-databases'.
>
> > That sounds like a great idea. I'll give that a go today and report
> > back. Thanks for your help!
>
> Ok I've given that tool a go but it comes up with an error. This is
> probably not the best place to post this question but just in case
> you've had this before I thought I'd start here:
>
> brett@secludedvalley:~$ mailutil transfer -v '{imapserver/notls/norsh/
> user=brett}INBOX' '#driver.mx//home/brett/Mail/'
> [Trying IP address [10.0.0.34]]
> [Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1
> (imapserver) ready.]
> password:
> Listing mailboxes...
> Copying {imapserver:143/imap/notls/user="brett"}INBOX/Personal
>   => #driver.mx//home/brett/Mail/INBOX/Personal
> [Reusing connection to imapserver/user="brett"]
> [Mailbox is empty]
> Can't create mailbox /home/brett/Mail/INBOX/Personal: invalid MX-
> format name
> [Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1
> signing off.]
> brett@secludedvalley:~$
>
> Any idea what that "invalid MX-format name" is? I tried it without
> "INBOX" and it died on the "Calendar" folder.

I've figured this one out. I was running the version that came with
Debian Etch's uw-mailutils, which apparently 6 years old. I downloaded
the latest UW-IMAP from the official server and mailutil worked great,
first time.

Thanks for your help.

Brett.

Re: Migrate from Maildir/IMAP/Exchange to NNML

[Ted Zlatanov]

On Mon, 21 Apr 2008 14:23:18 -0700 (PDT) brettrandall <brett.randall@gmail.com> wrote: 

b> On Apr 22, 1:35 am, Ted Zlatanov <t...@lifelogs.com> wrote:
>> Gnus has a native Maildir backend (nnmaildir) which is very good.  Would
>> that work for you?  It will work fine with your existing setup without
>> migration work.

b> Do you know what it will do with the "subfolders" in the format they
b> are in at the moment? (Maildir/.TopLevelFolder,
b> Maildir/.TopLevelFolder.2ndLevelFolder), etc? I have a look in the
b> Gnus manual and it doesn't seem to mention how it would deal with that
b> - whether it would even show them, and if so, how they would show up.
b> I guess I could just give it a go and see what happens :)

The Maildir root is your server's root level (like INBOX on most IMAP
servers, for example).

Every directory under it is just another group you can subscribe to.
The '.' in the name doesn't matter to Gnus, it doesn't care and will
just show you the directory name as the group name.  You can hit `^'
from the group buffer to see the server list, then ENTER on the maildir
server to see the groups it contains.

Ted

Re: Migrate from Maildir/IMAP/Exchange to NNML

[Nicolas KOWALSKI]

brettrandall <brett.randall@gmail.com> writes:

> I've figured this one out. I was running the version that came with
> Debian Etch's uw-mailutils, which apparently 6 years old. I downloaded
> the latest UW-IMAP from the official server and mailutil worked great,
> first time.

Glad this helped you.

I also read the thread on the imap-uw mailing-list. Do not forget (as
I did) what Mark wrote you about the CR-LF line termination of the
data files of the 'mx' format.

I converted (and kept dates of) the 'mx' files stored in my
~/Mail/mail/ directory with the following shell script. It takes a
long time, but it works well.

find . -type f -name '[0-9]*' |\
while read f ; do
      tr -d '\r' < $f > $f.new
      touch -r $f $f.new
      mv -f $f.new $f
done

-- 
Nicolas