[gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.

[gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.

[Michaël Cadilhac]

[-- Attachment #1.1: Type: text/plain, Size: 51 bytes --]


Seems that this message never reached the list.



[-- Attachment #1.2: Type: message/rfc822, Size: 18349 bytes --]

From: Micha <micha@lrde.epita.fr>
Subject: Fwd: nnrss and some (partially redundant) RSS feeds.

=2D-=3D=3D-=3D-=3D
Content-Type: multipart/mixed; boundary=3D"=3D-=3D-=3D"

=2D-=3D-=3D-=3D
Content-Type: text/plain; charset=3Diso-8859-1
Content-Transfer-Encoding: quoted-printable


Some people told me to follow up this message in the ding mailing
list, =3DAB bugs =3DBB being overflowed by spam. Thanks in advance.


Michael Cadilhac <michael.cadilhac@lrde.org> writes:

>   Hi !
>
>   I've some issues with an RSS feed that comes from a Trac (project
>   managing tool).
>
> 1) The feed [1] has the following entries :
>
> //////////////////////////////////////////////////////////////////////
>
>    <item>
>     <title>Ticket #13 (defect) created by pouchet@lrde.epita.fr</title>
>     <author>pouchet@lrde.epita.fr</author>
>     <pubDate>Mon, 27 Mar 2006 10:00:52 GMT</pubDate>
>     <link>http://vaucanson.lrde.org/trac.cgi/ticket/13</link>
>     <description>Correction on homepage</description>
>    </item>
>
>    <item>
>     <title>Ticket #13 (defect) closed by cadilh_m</title>
>     <author>michael.cadilhac@lrde.org</author>
>     <pubDate>Mon, 27 Mar 2006 11:21:43 GMT</pubDate>
>     <link>http://vaucanson.lrde.org/trac.cgi/ticket/13</link>
>     <description>Fixed.</description>
>    </item>
>
> //////////////////////////////////////////////////////////////////////
>
>   As you can see, a ticket has been opened then closed.
>
> 2) The nnrss.el (I'm using CVS Gnus) code looks like:
>
>   Check if an item is already stored:
>
> 		 (if (setq url (nnrss-decode-entities-string
> 				(nnrss-node-text rss-ns 'link (cddr item))))
> 		     (not (gethash url nnrss-group-hashtb))
> 		   (setq extra (or (nnrss-node-text content-ns 'encoded item)
> 				   (nnrss-node-text rss-ns 'description item)))
> 		   (not (gethash extra nnrss-group-hashtb))))
>
>   Here,  the  hash  table  is  indexed  by, first,  the  URL,  and  as
>   a fallback, by the =3DAB encoded =3DBB or =3DAB description =3DBB field.
>
>
> 1 with 2)
>
>   Gosh  ! Both messages  have the  same =3DAB  link =3DBB  (URL) !  So th=
ey're
>   hashed by the same index and  the first message will be in the group
>   while the other one will never appear !
>
> 3) Then why not hash by URL _AND_ Description ?
>
>   In the RSS  field [1], we also have entries with  the same URL *and*
>   the same  description (only the  title and the date  differ). Beside
>   that, the description could be a large message.
>
> 4) So what would be a good hash index ?
>
>   What about the  concatenation between =3DAB date =3DBB (or  =3DAB pubda=
te =3DBB=3D
) and
>   =3DAB link =3DBB (or its fallback) ? I find that meaningful because a t=
icket
>   (here, in my case) couldn't be edited twice at the same time.
>
>   Alternatively, an even better hash  index would be the md5sum of the
>   whole  entry  from  XML   ;  the  drawback  being,  obviously,  the
>   computation time of this thing.
>
>   If needed, patches attached ; comments welcome :-)
>
>
> Thanks in advance.
>
> Footnotes:=3D20
> [1]  http://vaucanson.lrde.org/trac.cgi/timeline?milestone=3D3Don&ticket=
=3D3D=3D
on&changeset=3D3Don&wiki=3D3Don&max=3D3D50&daysback=3D3D90&format=3D3Drss
>
>
> ----
> Note on the patches :
>
>   For the first patch :
>
>     I  haven't kept  a back-compatibility  for el-rss  files :  in the
>     current code,  if the =3DAB date  =3DBB field is empty  (well, it's r=
arely
>     the case,  but it  could be), it  is set  to the current  time and
>     that's OK.
>
>     But we  compute the hash  index from the  original =3DAB date  =3DBB =
field
>     (i.e. the one from the RSS feed) ;  so I add to store it in the el
>     file and  additionally in the  `nnrss-group-data' list as  the 4th
>     element of each elements in order to recompute it rightly on group
>     loading.
>
>   For the second one :
>
>     Same thing,  but I preferred to  store the md5sum as  the 9th (and
>     last) element  of each elements of  `nnrss-group-data' directly in
>     order to avoid (an hard) re-computation.


=2D-=3D-=3D-=3D
Content-Type: text/x-patch
Content-Disposition: attachment; filename=3Dnnrss.patch
Content-Transfer-Encoding: quoted-printable
Content-Description: Patch 1: with the date field.

Index: ChangeLog
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1099
diff -c -r7.1099 ChangeLog
*** ChangeLog	27 Mar 2006 09:42:59 -0000	7.1099
=3D2D-- ChangeLog	27 Mar 2006 16:06:35 -0000
***************
*** 1,3 ****
=3D2D-- 1,12 ----
+ 2006-03-27  Michael Cadilhac  <michael.cadilhac@lrde.org>  (tiny change)
+=3D20=3D20
+ 	* nnrss.el (nnrss-check-group): Hash messages with the `date'
+ 	field together with the previous criteria. Store the original
+ 	`date' field in `nnrss-group-data'.
+ 	(nnrss-read-group-data): Update accordingly.
+ 	(nnrss-retrieve-headers): Update access to `nnrss-group-data' elements.
+ 	(nnrss-request-article): Likewise.
+=3D20
  2006-03-26  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>  (tiny change)
=3D20=3D20
  	* message.el (message-resend): Bind message-generate-hashcash to
Index: nnrss.el
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D
RCS file: /usr/local/cvsroot/gnus/lisp/nnrss.el,v
retrieving revision 7.43
diff -c -r7.43 nnrss.el
*** nnrss.el	16 Jan 2006 22:57:40 -0000	7.43
=3D2D-- nnrss.el	27 Mar 2006 16:06:36 -0000
***************
*** 125,131 ****
  		    (or (nth 4 e) "(nobody)")
  		    "\t"
  		    ;; date
! 		    (or (nth 5 e) "")
  		    "\t"
  		    ;; id
  		    (format "<%d@%s.nnrss>" (car e) group)
=3D2D-- 125,131 ----
  		    (or (nth 4 e) "(nobody)")
  		    "\t"
  		    ;; date
! 		    (or (nth 6 e) "")
  		    "\t"
  		    ;; id
  		    (format "<%d@%s.nnrss>" (car e) group)
***************
*** 138,149 ****
  		    "-1" "\t"
  		    ;; Xref
  		    "" "\t"
! 		    (if (and (nth 6 e)
  			     (memq nnrss-description-field
  				   nnmail-extra-headers))
  			(concat (symbol-name nnrss-description-field)
  				": "
! 				(nnrss-format-string (nth 6 e))
  				"\t")
  		      "")
  		    (if (and (nth 2 e)
=3D2D-- 138,149 ----
  		    "-1" "\t"
  		    ;; Xref
  		    "" "\t"
! 		    (if (and (nth 7 e)
  			     (memq nnrss-description-field
  				   nnmail-extra-headers))
  			(concat (symbol-name nnrss-description-field)
  				": "
! 				(nnrss-format-string (nth 7 e))
  				"\t")
  		      "")
  		    (if (and (nth 2 e)
***************
*** 198,210 ****
  	    (insert "Subject: " (nth 3 e) "\n"))
  	(if (nth 4 e)
  	    (insert "From: " (nth 4 e) "\n"))
! 	(if (nth 5 e)
! 	    (insert "Date: " (nnrss-format-string (nth 5 e)) "\n"))
  	(let ((header (buffer-string))
! 	      (text (nth 6 e))
  	      (link (nth 2 e))
! 	      (enclosure (nth 7 e))
! 	      (comments (nth 8 e))
  	      ;; Enable encoding of Newsgroups header in XEmacs.
  	      (default-enable-multibyte-characters t)
  	      (rfc2047-header-encoding-alist
=3D2D-- 198,210 ----
  	    (insert "Subject: " (nth 3 e) "\n"))
  	(if (nth 4 e)
  	    (insert "From: " (nth 4 e) "\n"))
! 	(if (nth 6 e)
! 	    (insert "Date: " (nnrss-format-string (nth 6 e)) "\n"))
  	(let ((header (buffer-string))
! 	      (text (nth 7 e))
  	      (link (nth 2 e))
! 	      (enclosure (nth 8 e))
! 	      (comments (nth 9 e))
  	      ;; Enable encoding of Newsgroups header in XEmacs.
  	      (default-enable-multibyte-characters t)
  	      (rfc2047-header-encoding-alist
***************
*** 576,582 ****
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (or (nth 2 e) (nth 6 e)) t nnrss-group-hashtb)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
=3D2D-- 576,582 ----
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (concat (nth 5 e) (or (nth 2 e) (nth 7 e))) t nnrss-group-hasht=
=3D
b)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
***************
*** 657,663 ****
  ;;; Snarf functions
=3D20=3D20
  (defun nnrss-check-group (group server)
!   (let (file xml subject url extra changed author date feed-subject
  	     enclosure comments rss-ns rdf-ns content-ns dc-ns)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
=3D2D-- 657,663 ----
  ;;; Snarf functions
=3D20=3D20
  (defun nnrss-check-group (group server)
!   (let (file xml subject url extra changed author date-field date feed-su=
=3D
bject
  	     enclosure comments rss-ns rdf-ns content-ns dc-ns)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
***************
*** 690,701 ****
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")=
=3D
) xml)))
        (when (and (listp item)
  		 (string=3D3D (concat rss-ns "item") (car item))
! 		 (if (setq url (nnrss-decode-entities-string
! 				(nnrss-node-text rss-ns 'link (cddr item))))
! 		     (not (gethash url nnrss-group-hashtb))
! 		   (setq extra (or (nnrss-node-text content-ns 'encoded item)
! 				   (nnrss-node-text rss-ns 'description item)))
! 		   (not (gethash extra nnrss-group-hashtb))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
  	(setq extra (or extra
  			(nnrss-node-text content-ns 'encoded item)
=3D2D-- 690,705 ----
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")=
=3D
) xml)))
        (when (and (listp item)
  		 (string=3D3D (concat rss-ns "item") (car item))
! 		 (progn
! 		   (setq date-field (or (nnrss-node-text dc-ns 'date item)
! 					(nnrss-node-text rss-ns 'pubDate item)
! 					""))
! 		   (if (setq url (nnrss-decode-entities-string
! 				  (nnrss-node-text rss-ns 'link (cddr item))))
! 		       (not (gethash (concat date-field url) nnrss-group-hashtb))
! 		     (setq extra (or (nnrss-node-text content-ns 'encoded item)
! 				     (nnrss-node-text rss-ns 'description item)))
! 		     (not (gethash (concat date-field extra) nnrss-group-hashtb)))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
  	(setq extra (or extra
  			(nnrss-node-text content-ns 'encoded item)
***************
*** 705,713 ****
  	(setq author (or (nnrss-node-text rss-ns 'author item)
  			 (nnrss-node-text dc-ns 'creator item)
  			 (nnrss-node-text dc-ns 'contributor item)))
! 	(setq date (nnrss-normalize-date
! 		    (or (nnrss-node-text dc-ns 'date item)
! 			(nnrss-node-text rss-ns 'pubDate item))))
  	(setq comments (nnrss-node-text rss-ns 'comments item))
  	(when (setq enclosure (cadr (assq (intern (concat rss-ns "enclosure")) i=
=3D
tem)))
  	  (let ((url (cdr (assq 'url enclosure)))
=3D2D-- 709,715 ----
  	(setq author (or (nnrss-node-text rss-ns 'author item)
  			 (nnrss-node-text dc-ns 'creator item)
  			 (nnrss-node-text dc-ns 'contributor item)))
! 	(setq date (nnrss-normalize-date date-field))
  	(setq comments (nnrss-node-text rss-ns 'comments item))
  	(when (setq enclosure (cadr (assq (intern (concat rss-ns "enclosure")) i=
=3D
tem)))
  	  (let ((url (cdr (assq 'url enclosure)))
***************
*** 737,748 ****
  	  url
  	  (and subject (nnrss-mime-encode-string subject))
  	  (and author (nnrss-mime-encode-string author))
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
  	  comments)
  	 nnrss-group-data)
! 	(puthash (or url extra) t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed
=3D2D-- 739,751 ----
  	  url
  	  (and subject (nnrss-mime-encode-string subject))
  	  (and author (nnrss-mime-encode-string author))
+ 	  date-field
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
  	  comments)
  	 nnrss-group-data)
! 	(puthash (concat date-field (or url extra)) t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed

=2D-=3D-=3D-=3D
Content-Type: text/x-patch
Content-Disposition: attachment; filename=3Dnnrss-md5.patch
Content-Transfer-Encoding: quoted-printable
Content-Description: Patch 2: with md5sum.

Index: ChangeLog
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1099
diff -c -r7.1099 ChangeLog
*** ChangeLog	27 Mar 2006 09:42:59 -0000	7.1099
=3D2D-- ChangeLog	27 Mar 2006 16:54:36 -0000
***************
*** 1,3 ****
=3D2D-- 1,9 ----
+ 2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
+=3D20
+ 	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
+ 	item as its hash index. Store this hash in `nnrss-group-data'.
+ 	(nnrss-read-group-data): Update accordingly.
+=3D20
  2006-03-26  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>  (tiny change)
=3D20=3D20
  	* message.el (message-resend): Bind message-generate-hashcash to
Index: nnrss.el
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D
RCS file: /usr/local/cvsroot/gnus/lisp/nnrss.el,v
retrieving revision 7.43
diff -c -r7.43 nnrss.el
*** nnrss.el	16 Jan 2006 22:57:40 -0000	7.43
=3D2D-- nnrss.el	27 Mar 2006 16:54:36 -0000
***************
*** 576,582 ****
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (or (nth 2 e) (nth 6 e)) t nnrss-group-hashtb)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
=3D2D-- 576,582 ----
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (nth 9 e) t nnrss-group-hashtb)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
***************
*** 658,664 ****
=3D20=3D20
  (defun nnrss-check-group (group server)
    (let (file xml subject url extra changed author date feed-subject
! 	     enclosure comments rss-ns rdf-ns content-ns dc-ns)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
  					(nnrss-translate-file-chars
=3D2D-- 658,664 ----
=3D20=3D20
  (defun nnrss-check-group (group server)
    (let (file xml subject url extra changed author date feed-subject
! 	     enclosure comments rss-ns rdf-ns content-ns dc-ns hash-index)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
  					(nnrss-translate-file-chars
***************
*** 690,704 ****
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")=
=3D
) xml)))
        (when (and (listp item)
  		 (string=3D3D (concat rss-ns "item") (car item))
! 		 (if (setq url (nnrss-decode-entities-string
! 				(nnrss-node-text rss-ns 'link (cddr item))))
! 		     (not (gethash url nnrss-group-hashtb))
! 		   (setq extra (or (nnrss-node-text content-ns 'encoded item)
! 				   (nnrss-node-text rss-ns 'description item)))
! 		   (not (gethash extra nnrss-group-hashtb))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
! 	(setq extra (or extra
! 			(nnrss-node-text content-ns 'encoded item)
  			(nnrss-node-text rss-ns 'description item)))
  	(if (setq feed-subject (nnrss-node-text dc-ns 'subject item))
  	    (setq extra (concat feed-subject "<br /><br />" extra)))
=3D2D-- 690,701 ----
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")=
=3D
) xml)))
        (when (and (listp item)
  		 (string=3D3D (concat rss-ns "item") (car item))
! 		 (progn (setq hash-index (md5 (prin1-to-string item)))
! 			(not (gethash hash-index nnrss-group-hashtb))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
! 	(setq url (nnrss-decode-entities-string
! 		   (nnrss-node-text rss-ns 'link (cddr item))))
! 	(setq extra (or (nnrss-node-text content-ns 'encoded item)
  			(nnrss-node-text rss-ns 'description item)))
  	(if (setq feed-subject (nnrss-node-text dc-ns 'subject item))
  	    (setq extra (concat feed-subject "<br /><br />" extra)))
***************
*** 740,748 ****
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
! 	  comments)
  	 nnrss-group-data)
! 	(puthash (or url extra) t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed
=3D2D-- 737,746 ----
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
! 	  comments
! 	  hash-index)
  	 nnrss-group-data)
! 	(puthash hash-index t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed

=2D-=3D-=3D-=3D
Content-Type: text/plain; charset=3Diso-8859-1
Content-Transfer-Encoding: quoted-printable


=3D2D-=3D20
|  Mieux vaut se taire                        Micha=3DEBl 'Micha' Cadilhac |
|   Que de parler trop fort.               cadilh_m - Epita 2007 - CSI |
|           -- As de tr=3DE8fle                     JID: micha@amessage.be |
`--  -  -                                                        - - --'

=2D-=3D-=3D-=3D--

=2D-=3D=3D-=3D-=3D
Content-Type: application/pgp-signature

=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFELWa2ZYNf4qobZ7kRAkvsAKCUlD4mdI05rYdHU7gNuRWxWWa65gCg7A7X
9C+Q8WhS5m7XlLtYlAS4sVM=3D
=3DPp2O
=2D----END PGP SIGNATURE-----
=2D-=3D=3D-=3D-=3D--

[-- Attachment #1.3: Type: text/plain, Size: 323 bytes --]


-- 
 |      Michaël `Micha' Cadilhac   |   Un certain Blaise Pascal             |
 |         Epita/LRDE Promo 2007   |     etc... etc...                      |
 | http://www.lrde.org/~cadilh_m   |   -- Prévert (Les paris stupides)      |
 `--  -   JID: micha@amessage.be --'                                   -  --'

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: [gnus.ding] Fwd: nnrss and some (partially redundant) RSS

[Jason L Tibbitts III]

MC> Seems that this message never reached the list.

I can verify that it did indeed reach the list.

 - J<

Re: [gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.

[Michaël Cadilhac]

Jason L Tibbitts III <tibbs@math.uh.edu> writes:

> MC> Seems that this message never reached the list.
>
> I can verify that it did indeed reach the list.
>
>  - J<

   Oh, my bad ; I don't see it in the news://news.gnus.org/gnus.ding
   newsgroup, but the news.gmane.org archive indeed contains it.


   Thanks :-)

-- 
 |      Michaël `Micha' Cadilhac   |   Un certain Blaise Pascal             |
 |         Epita/LRDE Promo 2007   |     etc... etc...                      |
 | http://www.lrde.org/~cadilh_m   |   -- Prévert (Les paris stupides)      |
 `--  -   JID: micha@amessage.be --'                                   -  --'

Re: [gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.

[Reiner Steib]

On Mon, Apr 10 2006, Michaël Cadilhac wrote:

> Seems that this message never reached the list.

Your message reached the list, see
<http://mid.gmane.org/87lkuq8qx9.fsf@mahaena.lrde>¹.

But since we indent to have a new release in the next days, we don't
want to add new code new.  We could add it after the release if
someone can review it (I don't use RSS).

+2006-03-27  Michael Cadilhac  <michael.cadilhac@lrde.org>  (tiny change)
+ 
+	* nnrss.el (nnrss-check-group): Hash messages with the `date'
+	field together with the previous criteria. Store the original
+	`date' field in `nnrss-group-data'.
+	(nnrss-read-group-data): Update accordingly.
+	(nnrss-retrieve-headers): Update access to `nnrss-group-data' elements.
+	(nnrss-request-article): Likewise.

+2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
+
+	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
+	item as its hash index. Store this hash in `nnrss-group-data'.
+	(nnrss-read-group-data): Update accordingly.


In total your contribution is larger than 10-15 lines (i.e. it's not a
"tiny change") so we need legal papers to install it (there are also
several tiny changes listed in emacs/lisp/ChangeLog).  Gnus is a part
of Emacs and since Emacs is a core GNU project, the Free Software
Foundation (FSF) requires an assignment of copyright for
contributions, so that it is able to defend Emacs' legal status
without doubts, should the need arise.

You don't seem to have an assignment on file yet.  If you agree, I can
send you the required request form.  After the assignment process
completes (you get sent forms which you have to read and sign and send
back), we could install your patch in Gnus.

You can either sign papers only for Gnus or for Emacs (which covers
Gnus, too).  Given your changes in emacs/lisp/*.el, I'd recommend you
to sign papers for Emacs.

Bye, Reiner.

¹ BTW, your FQDN is bogus.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: [gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.

[Lars Magne Ingebrigtsen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> But since we indent to have a new release in the next days, we don't
> want to add new code new. 

Er, I forgot about the freeze.  And I was just in the mood to hack
Gnus a bit...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Gnus hacking mood (was: [gnus.ding] Fwd: nnrss and some (partially redundant) RSS feeds.)

[Reiner Steib]

On Tue, Apr 11 2006, Lars Magne Ingebrigtsen wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> But since we indent to have a new release in the next days, we don't
>> want to add new code new. 
>
> Er, I forgot about the freeze.  And I was just in the mood to hack
> Gnus a bit...  :-)

Let's hope this mood will stay...  :-)

BTW, when committing bug fixes, it would be nice if you could add them
either to the v5-10 branch or to v5-10 and to the trunk.  See the
section "Syncing" in `texi/gnus-coding.texi'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus hacking mood

[Lars Magne Ingebrigtsen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> Let's hope this mood will stay...  :-)

Tomorrow I should probably pay a courtesy call at the office.  We'll
see...  :-)

> BTW, when committing bug fixes, it would be nice if you could add them
> either to the v5-10 branch or to v5-10 and to the trunk.  See the
> section "Syncing" in `texi/gnus-coding.texi'.

My deity!  How organised!

So you generally have two checked-out trees, and then apply patches to
both, and then check in both?

Hm...  how do you check out a branch?  I've never actually touched a
cvs branch in my life.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus hacking mood

[Reiner Steib]

On Tue, Apr 11 2006, Lars Magne Ingebrigtsen wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>> BTW, when committing bug fixes, it would be nice if you could add them
>> either to the v5-10 branch or to v5-10 and to the trunk.  See the
>> section "Syncing" in `texi/gnus-coding.texi'.
>
> My deity!  How organised!

;-)

> So you generally have two checked-out trees, and then apply patches to
> both, and then check in both?

Checking it in *either* in v5-10 (when it's a bug fix, doc fix, ...)
*or* in the trunk (new feature, fix to stuff not existing in v5-10,
...) is sufficient.  After a couple of days, Miles will sync the stuff
from v5-10 to trunk (and also to Emacs CVS) semi-automatically.

(Maybe someone want to clarify the text in `texi/gnus-coding.texi' if
it's not clear.)

> Hm...  how do you check out a branch?  I've never actually touched a
> cvs branch in my life.

Just add "-r v5-10" to the "checkout" command:

$ cvs -d ... checkout -r v5-10 gnus

$ cvs co -?
co: invalid option -- ?
Usage:
  cvs checkout [-ANPRcflnps] [-r rev] [-D date] [-d dir]
    [-j rev1] [-j rev2] [-k kopt] modules...
[...]
        -r rev  Check out revision or tag. (implies -P) (is sticky)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus hacking mood

[Lars Magne Ingebrigtsen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> Just add "-r v5-10" to the "checkout" command:
>
> $ cvs -d ... checkout -r v5-10 gnus

Great.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnrss and some (partially redundant) RSS feeds.

[Michaël Cadilhac]

[-- Attachment #1.1: Type: text/plain, Size: 245 bytes --]


  Hi!

  I come back to you as my assignment paper is, at last, signed,
  returned and validated.

  This message is a follow-up of my message of March:
  http://article.gmane.org/gmane.emacs.gnus.general/62464

  The patch is again attached:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: nnrss-md5.patch --]
[-- Type: text/x-patch, Size: 4297 bytes --]

Index: ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1099
diff -c -r7.1099 ChangeLog
*** ChangeLog	27 Mar 2006 09:42:59 -0000	7.1099
--- ChangeLog	27 Mar 2006 16:54:36 -0000
***************
*** 0,0 ****
--- 1,6 ----
+ 2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
+ 
+ 	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
+ 	item as its hash index. Store this hash in `nnrss-group-data'.
+ 	(nnrss-read-group-data): Update accordingly.
+ 
Index: nnrss.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnrss.el,v
retrieving revision 7.43
diff -c -r7.43 nnrss.el
*** nnrss.el	16 Jan 2006 22:57:40 -0000	7.43
--- nnrss.el	27 Mar 2006 16:54:36 -0000
***************
*** 576,582 ****
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (or (nth 2 e) (nth 6 e)) t nnrss-group-hashtb)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
--- 576,582 ----
  	  (insert-file-contents file)
  	  (eval-region (point-min) (point-max))))
        (dolist (e nnrss-group-data)
! 	(puthash (nth 9 e) t nnrss-group-hashtb)
  	(when (and (car e) (> nnrss-group-min (car e)))
  	  (setq nnrss-group-min (car e)))
  	(when (and (car e) (< nnrss-group-max (car e)))
***************
*** 658,664 ****
  
  (defun nnrss-check-group (group server)
    (let (file xml subject url extra changed author date feed-subject
! 	     enclosure comments rss-ns rdf-ns content-ns dc-ns)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
  					(nnrss-translate-file-chars
--- 658,664 ----
  
  (defun nnrss-check-group (group server)
    (let (file xml subject url extra changed author date feed-subject
! 	     enclosure comments rss-ns rdf-ns content-ns dc-ns hash-index)
      (if (and nnrss-use-local
  	     (file-exists-p (setq file (expand-file-name
  					(nnrss-translate-file-chars
***************
*** 690,704 ****
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")) xml)))
        (when (and (listp item)
  		 (string= (concat rss-ns "item") (car item))
! 		 (if (setq url (nnrss-decode-entities-string
! 				(nnrss-node-text rss-ns 'link (cddr item))))
! 		     (not (gethash url nnrss-group-hashtb))
! 		   (setq extra (or (nnrss-node-text content-ns 'encoded item)
! 				   (nnrss-node-text rss-ns 'description item)))
! 		   (not (gethash extra nnrss-group-hashtb))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
! 	(setq extra (or extra
! 			(nnrss-node-text content-ns 'encoded item)
  			(nnrss-node-text rss-ns 'description item)))
  	(if (setq feed-subject (nnrss-node-text dc-ns 'subject item))
  	    (setq extra (concat feed-subject "<br /><br />" extra)))
--- 690,701 ----
      (dolist (item (nreverse (nnrss-find-el (intern (concat rss-ns "item")) xml)))
        (when (and (listp item)
  		 (string= (concat rss-ns "item") (car item))
! 		 (progn (setq hash-index (md5 (gnus-prin1-to-string item)))
! 			(not (gethash hash-index nnrss-group-hashtb))))
  	(setq subject (nnrss-node-text rss-ns 'title item))
! 	(setq url (nnrss-decode-entities-string
! 		   (nnrss-node-text rss-ns 'link (cddr item))))
! 	(setq extra (or (nnrss-node-text content-ns 'encoded item)
  			(nnrss-node-text rss-ns 'description item)))
  	(if (setq feed-subject (nnrss-node-text dc-ns 'subject item))
  	    (setq extra (concat feed-subject "<br /><br />" extra)))
***************
*** 740,748 ****
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
! 	  comments)
  	 nnrss-group-data)
! 	(puthash (or url extra) t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed
--- 737,746 ----
  	  date
  	  (and extra (nnrss-decode-entities-string extra))
  	  enclosure
! 	  comments
! 	  hash-index)
  	 nnrss-group-data)
! 	(puthash hash-index t nnrss-group-hashtb)
  	(setq changed t))
        (setq extra nil))
      (when changed

[-- Attachment #1.3: Type: text/plain, Size: 401 bytes --]


  I use it since March and don't have any problem.

  Thanks in advance.

-- 
 |      Michaël `Micha' Cadilhac   |     Le copillage-collage               |
 |         Epita/LRDE Promo 2007   |        tue le programmeur.             |
 | http://www.lrde.org/~cadilh_m   |            -- Dictons LRDE             |
 `--  -   JID: micha@amessage.be --'                                   -  --'

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: nnrss and some (partially redundant) RSS feeds.

[Reiner Steib]

On Tue, May 23 2006, Michaël Cadilhac wrote:

> +2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
> +
> +	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
> +	item as its hash index. Store this hash in `nnrss-group-data'.
> +	(nnrss-read-group-data): Update accordingly.

Installed, thanks.

The byte-complier complained about an unbound variable `hash-index'.
I think this should fix it, but please verify that the fix is correct
(I don't use nnrss).

--- nnrss.el	23 May 2006 14:45:56 -0000	7.44
+++ nnrss.el	23 May 2006 14:47:29 -0000	7.45
@@ -658,7 +658,8 @@
 
 (defun nnrss-check-group (group server)
   (let (file xml subject url extra changed author date feed-subject
-	     enclosure comments rss-ns rdf-ns content-ns dc-ns)
+	     enclosure comments rss-ns rdf-ns content-ns dc-ns
+	     hash-index)
     (if (and nnrss-use-local
 	     (file-exists-p (setq file (expand-file-name
 					(nnrss-translate-file-chars

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnrss and some (partially redundant) RSS feeds.

[Andreas Seltenreich]

Reiner Steib schrob:

> On Tue, May 23 2006, Michaël Cadilhac wrote:
>
>> +2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
>> +
>> +	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
>> +	item as its hash index. Store this hash in `nnrss-group-data'.
>> +	(nnrss-read-group-data): Update accordingly.
>
> Installed, thanks.

The approach of hashing whole items doesn't work well with some of the
feeds I'm reading. E.g., <http://rss.slashdot.org/Slashdot/slashdot>
has a readily mutating content:

--8<---------------cut here---------------start------------->8---
*** 144,155 ****
  <dc:creator>CmdrTaco</dc:creator>
  <dc:date>2006-05-28T14:37:00+00:00</dc:date>
  <dc:subject>nasa</dc:subject>
  <slash:department>everyone-likes-upgrades</slash:department>
  <slash:section>science</slash:section>
! <slash:comments>105</slash:comments>
! <slash:hit_parade>105,100,88,54,19,14,14</slash:hit_parade>
  <feedburner:origLink>http://science.slashdot.org/article.pl?sid=06/05/28/148245&amp;from=rss</feedburner:origLink></item>

  <item rdf:about="http://yro.slashdot.org/article.pl?sid=06/05/28/136247&amp;from=rss">
  <title>Amnesty International vs. Internet Censorship</title>
  <link>http://rss.slashdot.org/Slashdot/slashdot?m=5712</link>
--- 144,155 ----
  <dc:creator>CmdrTaco</dc:creator>
  <dc:date>2006-05-28T14:37:00+00:00</dc:date>
  <dc:subject>nasa</dc:subject>
  <slash:department>everyone-likes-upgrades</slash:department>
  <slash:section>science</slash:section>
! <slash:comments>106</slash:comments>
! <slash:hit_parade>106,101,89,55,19,14,14</slash:hit_parade>
  <feedburner:origLink>http://science.slashdot.org/article.pl?sid=06/05/28/148245&amp;from=rss</feedburner:origLink></item>

  <item rdf:about="http://yro.slashdot.org/article.pl?sid=06/05/28/136247&amp;from=rss">
  <title>Amnesty International vs. Internet Censorship</title>
  <link>http://rss.slashdot.org/Slashdot/slashdot?m=5712</link>
***************
--8<---------------cut here---------------end--------------->8---

Michaël's alternative solution of just adding the date to the hash
would work for all the feeds I'm reading.

regards,
andreas

Re: nnrss and some (partially redundant) RSS feeds.

[Michaël Cadilhac]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

> Reiner Steib schrob:
>
>> On Tue, May 23 2006, Michaël Cadilhac wrote:
>>
>>> +2006-03-27  Michael Cadilhac  <micha@mahaena.lrde> (tiny change)
>>> +
>>> +	* nnrss.el (nnrss-check-group): Use the md5sum of the whole RSS
>>> +	item as its hash index. Store this hash in `nnrss-group-data'.
>>> +	(nnrss-read-group-data): Update accordingly.
>>
>> Installed, thanks.
>
> The approach of hashing whole items doesn't work well with some of the
> feeds I'm reading. E.g., <http://rss.slashdot.org/Slashdot/slashdot>
> has a readily mutating content:
>
> ! <slash:comments>105</slash:comments>
> ! <slash:comments>106</slash:comments>

Oh, I wasn't aware of such a thing. However, Gnus will not update its
messages, isn't it?

> Michaël's alternative solution of just adding the date to the hash
> would work for all the feeds I'm reading.

As it will for mines, AFAICT.

-- 
|  Un certain Blaise Pascal                   Michaël `Micha' Cadilhac |
|    etc... etc...                         cadilh_m - Epita 2007 - CSI |
|  -- Prévert (Les paris stupides)              JID: micha@amessage.be |
`--  -  -                                                        - - --'

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: nnrss and some (partially redundant) RSS feeds.

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

Michaël Cadilhac schrob:

> Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
>
>> The approach of hashing whole items doesn't work well with some of the
>> feeds I'm reading. E.g., <http://rss.slashdot.org/Slashdot/slashdot>
>> has a readily mutating content:
>>
>> ! <slash:comments>105</slash:comments>
>> ! <slash:comments>106</slash:comments>
>
> Oh, I wasn't aware of such a thing. However, Gnus will not update its
> messages, isn't it?

Currently, Gnus presents a new article differing from the previous one
only by the generated Message-Id everytime someone writes a comment
for it. So the feed is pretty much unusable at the moment.

regards,
andreas

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]