code query

code query

[Eric Abrahamsen]

I've been trying for a while to write a function for marking all
duplicate messages in an nnmail group. So far I've got the below, which
works okay up to a point. It gets all the messages in a group, then
takes gnus-newsgroup-data, sorts it according to Message-ID, and sets
the process mark on any message that has the same Message-ID as the one
before. I was going to have it add Gnus-Warning headers, but have so far
failed to edit the messages programmatically.

The problem with the below is that it seems to create one "pseudo article"
for each block of consecutive identical messages -- the pseudo article
is a new copy of that message, but can't be opened or displayed
properly, nor deleted. They're also persistent, so the more I run my
function the more "duplicate" messages I get!

From what I can tell, pseudo articles are created in the course of
decoding raw messages. I assume I need to do some cleanup on the article
or on the summary buffer, either before `gnus-summary-set-process-mark'
is called, or afterwards (or both). Or perhaps after the whole process
is done?

Can anyone shed any light on this?

Thanks!
Eric


--8<---------------cut here---------------start------------->8---

(defun my-gnus-warn-duplicates (group)
  "Go through a group and mark all messages that are duplicates
  of other messages, according to the value of Message-ID."
  (interactive (list (gnus-group-group-name)))
  (let ((gnus-large-newsgroup nil)
	(gnus-registry-enabled nil))	;doesn't work
    (gnus-message 5 "Marking duplicate messages, this may take a while...")
    (gnus-summary-read-group group t t nil)
    (my-gnus-mark-duplicates gnus-newsgroup-data)
    (gnus-message 5 "Marked %d duplicate messages")))

(defun my-gnus-mark-duplicates (data)
  (let ((data (sort data (lambda (one two)
			   (string< (mail-header-id (gnus-data-header one))
				    (mail-header-id (gnus-data-header two))))))
	last-id)
    (mapcar (lambda (d)
	      (let ((id (mail-header-id (gnus-data-header d))))
		(if (string= id last-id)
		    (gnus-summary-set-process-mark (gnus-data-number d))
		  ;; there may be spurious marks
		  (gnus-summary-remove-process-mark (gnus-data-number d)))
		(setq last-id id)))
	    data)))
--8<---------------cut here---------------end--------------->8---

Re: code query

[Andreas Schwab]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I've been trying for a while to write a function for marking all
> duplicate messages in an nnmail group.

Have you looked at gnus-suppress-duplicates?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: code query

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I've been trying for a while to write a function for marking all
>> duplicate messages in an nnmail group.
>
> Have you looked at gnus-suppress-duplicates?

I have not -- thanks for that tip. But this appears to apply to newly
received mail, right? I've actually got nnmail-treat-duplicates set to
'delete, so that takes care of that. But I've got huge amounts of
duplicate messages in existing boxes, after rebuilding lost archives
from several sources. So I need something that applies to existing
messages...

Thanks,
Eric

Re: code query

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The problem with the below is that it seems to create one "pseudo article"
> for each block of consecutive identical messages -- the pseudo article
> is a new copy of that message, but can't be opened or displayed
> properly, nor deleted. They're also persistent, so the more I run my
> function the more "duplicate" messages I get!

I don't quite understand that problem, but I'm surprised the code works
at all.  :-)

>     (my-gnus-mark-duplicates gnus-newsgroup-data)

[...]

>   (let ((data (sort data (lambda (one two)

This `sort' is destructive, so you're altering the `gnus-newsgroup-data'
list (most likely truncating it).  Put a `copy-sequence' in there
somewhere.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: code query

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The problem with the below is that it seems to create one "pseudo article"
>> for each block of consecutive identical messages -- the pseudo article
>> is a new copy of that message, but can't be opened or displayed
>> properly, nor deleted. They're also persistent, so the more I run my
>> function the more "duplicate" messages I get!
>
> I don't quite understand that problem, but I'm surprised the code works
> at all.  :-)

Me too!

>>     (my-gnus-mark-duplicates gnus-newsgroup-data)
>
> [...]
>
>>   (let ((data (sort data (lambda (one two)
>
> This `sort' is destructive, so you're altering the `gnus-newsgroup-data'
> list (most likely truncating it).  Put a `copy-sequence' in there
> somewhere.

Thanks, I'll give that a shot.