Deleting duplicates from nnml:mail.misc

[Alex Schroeder <alex@gnu.org>]

I've started using Gnus again after many years. I had a ton of 
mbox files I had created using fetchmail from my Gmail account, 
but didn't really trust it, ran it a lot, thought that maybe it 
didn't get my sent messages and therefore moved a lot of messages 
from the Gmail IMAP to my nnml:mail.misc using B m ... and now I'm 
suddenly having second thoughts. Is my nnml:mail.misc of about 
60000 messages full of duplicates?

I've been wondering how to find them, if any. I've tried looking at the
.overview file...

#!/bin/env perl
my %count = ();
my %file = ();
my $overview = "/Volumes/Extern/Archives/Mail/mail/misc/.overview";
open(F, $overview) or die "Cannot open $overview: $!";
while(my $line = <F>) {
  my @field = split(/\t/, $line);
  $count{$field[4]}++;
  push(@{$file{$field[4]}}, $field[0]);
}
close(F);
my @keys = sort { $count{$b} cmp $count{$a} } keys %count;
print join("\n", map {
  $_ . "\t" . $count{$_}  . "\t" . join(", ", @{$file{$_}})
} @keys[0 .. 3]) . "\n";

How would I best use this script to delete the duplicate messages? Can I
"regenerate" the overview file without loosing anything, perhaps by
regenrating something in the Server buffer? Or is there an elisp version
of the above that does the right thing?

Cheers
Alex