Re: Why am I getting duplicate messages on RSS groups?

[James Thomas <jimjoe@gmx.net>]

Tim Landscheidt wrote:

> Nasser Alkmim <nasser.alkmim@gmail.com> wrote:
>
>> Not sure how to debug this situation, but some RSS feeds that I have
>> in groups end up with duplicate messages.
>
>> I use this "five filters full-text RSS" to extract the full text
>> from some RSS feeds, and it has a limit of 3 items per feed and
>> 12-hours refresh rate.
>> Maybe after this 12-hours, the messages are obtained again.
>
>> The duplicate messages have different "Message-ID", but same subject/date and everything else.
>
>> Any ideas?
>
> I'm not sure the /internal/ dates are actually the same: If
> I write the data for such duplicate entries to disk (*1):
>
> | (dolist (i '(58302 58461 58609 58757 58905 59053))
> |   (with-temp-file (format "/tmp/%d.el" i)
> |     (pp (cddr (assoc i nnrss-group-data)) (current-buffer))))
>
> and diff them, some entries change from file to file
> (pubDate, author, URL, etc.).  For example, pubDate is:
>
> | $ grep -i date /tmp/*.el
> | /tmp/58302.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 GMT")
> | /tmp/58461.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 -0400")
> | /tmp/58609.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 -0400")
> | /tmp/58757.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 -0400")
> | /tmp/58905.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 +0000")
> | /tmp/59053.el:       (pubDate nil "Thu, 10 Dec 2020 22:01:00 +0100")
> | $
>
> But for all six messages, Gnus says:
>
> | Date: Thu, 10 Dec 2020 22:01:00 +0000 (3 years, 18 weeks ago)
>
> Now if I understand nnrss.el correctly, it considers two en-
> tries the same if they only differ in fields listed in
> nnrss-ignore-article-fields (which is 'slash:comments by de-
> fault), so any changes to an RSS feed entry will create a
> new Gnus nnrss message.  What appears to be missing is
> treating guid as an indicator that an entry has not changed.

Nasser,

Maybe you haven't tried adding 'pubDate (or better: everything other
than 'guid) to nnrss-ignore-article-fields.

--