nnweb fix

nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 107 bytes --]

Hi,

attached is an attempt to get web groups working again with the new
google groups.

regards,
Andreas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5062 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.10
diff -c -r7.10 nnweb.el
*** lisp/nnweb.el	7 Sep 2005 13:50:08 -0000	7.10
--- lisp/nnweb.el	31 Dec 2005 09:32:34 -0000
***************
*** 27,37 ****
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Google Groups and Gmane, stuff
! ;; related to web groups (gnus-group-make-web-group) doesn't work anymore.
! 
! ;; Fetching an article by MID (cf. gnus-refer-article-method) over Google
! ;; Groups should work.
  
  ;;; Code:
  
--- 27,34 ----
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to
! ;; web groups doesn't work anymore.
  
  ;;; Code:
  
***************
*** 61,66 ****
--- 58,64 ----
  (defvar nnweb-type-definition
    '((google
       (id . "http://www.google.com/groups?as_umsgid=%s&hl=en&dmode=source")
+      (result . "http://groups.google.com/group/%s/msg/%s?dmode=source")
       (article . nnweb-google-wash-article)
       (reference . identity)
       (map . nnweb-google-create-mapping)
***************
*** 337,342 ****
--- 335,341 ----
        (mm-url-decode-entities))))
  
  (defun nnweb-google-parse-1 (&optional Message-ID)
+   "Parse search result in current buffer."
    (let ((i 0)
  	(case-fold-search t)
  	(active (cadr (assoc nnweb-group nnweb-group-alist)))
***************
*** 349,358 ****
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a href=/groups\\(\\?[^ \">]*selm=\\([^ &\">]+\\)\\)" nil t)
!       (setq mid (match-string 2)
  	    url (format
! 		 (nnweb-definition 'id) mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
--- 348,360 ----
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a +href=\"/group/\\([^>\"]+\\)/browse_thread/[^>]+#\\([0-9a-f]+\\)" nil t)
!       (setq Newsgroups (match-string-no-properties 1)
! 	    ;; Note: with groups-ng, mid is no longer a common
! 	    ;; Message-ID, but some internal id.
! 	    mid (match-string-no-properties 2)
  	    url (format
! 		 (nnweb-definition 'result) Newsgroups mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
***************
*** 360,384 ****
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (forward-line 2)
!       (when (looking-at "<br><font[^>]+>")
! 	(goto-char (match-end 0)))
!       (if (not (looking-at "<a[^>]+>"))
! 	  (skip-chars-forward " \t")
! 	(narrow-to-region (point)
! 			  (search-forward "</a>" nil t))
! 	(mm-url-remove-markup)
! 	(mm-url-decode-entities)
! 	(setq Newsgroups (buffer-string))
! 	(goto-char (point-max))
! 	(widen)
! 	(skip-chars-forward "- \t"))
        (when (looking-at
! 	     "\\([0-9]+\\)[/ ]\\([A-Za-z]+\\)[/ ]\\([0-9]+\\)[ \t]*by[ \t]*\\([^<]*\\) - <a")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 2) (match-string 1)
! 			   (match-string 3))))
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
--- 362,384 ----
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (narrow-to-region (point)
! 			(search-forward "</td" nil t))
! 
!       (search-backward ">- ")
!       (setq From nil)
        (when (looking-at
! 	     ">- \\([a-zA-Z]+\\) \\([0-9]+\\)\\(?: \\([0-9]\\{4\\}\\)\\)?, [^-<]+by \\([^<\n]+\\)\n")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 1)
! 			   (match-string 2)
! 			   (or (match-string 3)
! 			       (substring (current-time-string) -4)))))
! 
!       (unless From (debug))
! 
!       (widen)
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
***************
*** 419,425 ****
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td nowrap><a href=\\([^>]+\\).*<span class=b>Next</span>" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
--- 419,425 ----
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td><a href=\"\n\\([^>\"]+\\)\"><img src=\"/img/nav_next" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
***************
*** 443,449 ****
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")))))
    t)
  
  (defun nnweb-google-identity (url)
--- 443,450 ----
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")
!        ("filter" . "0")))))
    t)
  
  (defun nnweb-google-identity (url)

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

Andreas Seltenreich writes:

> attached is an attempt to get web groups working again with the new
> google groups.

Oops, I missed some html entity decoding and a debug statement.
Attached is an updated patch.

regards,
Andreas

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5069 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.10
diff -c -r7.10 nnweb.el
*** lisp/nnweb.el	7 Sep 2005 13:50:08 -0000	7.10
--- lisp/nnweb.el	31 Dec 2005 10:47:14 -0000
***************
*** 27,37 ****
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Google Groups and Gmane, stuff
! ;; related to web groups (gnus-group-make-web-group) doesn't work anymore.
! 
! ;; Fetching an article by MID (cf. gnus-refer-article-method) over Google
! ;; Groups should work.
  
  ;;; Code:
  
--- 27,34 ----
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to
! ;; web groups doesn't work anymore.
  
  ;;; Code:
  
***************
*** 61,66 ****
--- 58,64 ----
  (defvar nnweb-type-definition
    '((google
       (id . "http://www.google.com/groups?as_umsgid=%s&hl=en&dmode=source")
+      (result . "http://groups.google.com/group/%s/msg/%s?dmode=source")
       (article . nnweb-google-wash-article)
       (reference . identity)
       (map . nnweb-google-create-mapping)
***************
*** 337,342 ****
--- 335,341 ----
        (mm-url-decode-entities))))
  
  (defun nnweb-google-parse-1 (&optional Message-ID)
+   "Parse search result in current buffer."
    (let ((i 0)
  	(case-fold-search t)
  	(active (cadr (assoc nnweb-group nnweb-group-alist)))
***************
*** 349,358 ****
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a href=/groups\\(\\?[^ \">]*selm=\\([^ &\">]+\\)\\)" nil t)
!       (setq mid (match-string 2)
  	    url (format
! 		 (nnweb-definition 'id) mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
--- 348,360 ----
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a +href=\"/group/\\([^>\"]+\\)/browse_thread/[^>]+#\\([0-9a-f]+\\)" nil t)
!       (setq Newsgroups (match-string-no-properties 1)
! 	    ;; Note: with groups-ng, mid is no longer a common
! 	    ;; Message-ID, but some internal id.
! 	    mid (match-string-no-properties 2)
  	    url (format
! 		 (nnweb-definition 'result) Newsgroups mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
***************
*** 360,384 ****
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (forward-line 2)
!       (when (looking-at "<br><font[^>]+>")
! 	(goto-char (match-end 0)))
!       (if (not (looking-at "<a[^>]+>"))
! 	  (skip-chars-forward " \t")
! 	(narrow-to-region (point)
! 			  (search-forward "</a>" nil t))
! 	(mm-url-remove-markup)
! 	(mm-url-decode-entities)
! 	(setq Newsgroups (buffer-string))
! 	(goto-char (point-max))
! 	(widen)
! 	(skip-chars-forward "- \t"))
        (when (looking-at
! 	     "\\([0-9]+\\)[/ ]\\([A-Za-z]+\\)[/ ]\\([0-9]+\\)[ \t]*by[ \t]*\\([^<]*\\) - <a")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 2) (match-string 1)
! 			   (match-string 3))))
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
--- 362,383 ----
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (narrow-to-region (point)
! 			(search-forward "</td" nil t))
! 
!       (mm-url-remove-markup)
!       (mm-url-decode-entities)
!       (search-backward " - ")
        (when (looking-at
! 	     " - \\([a-zA-Z]+\\) \\([0-9]+\\)\\(?: \\([0-9]\\{4\\}\\)\\)?, [^\n]+by \\([^<\n]+\\)\n")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 1)
! 			   (match-string 2)
! 			   (or (match-string 3)
! 			       (substring (current-time-string) -4)))))
! 
!       (widen)
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
***************
*** 419,425 ****
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td nowrap><a href=\\([^>]+\\).*<span class=b>Next</span>" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
--- 418,424 ----
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td><a href=\"\n\\([^>\"]+\\)\"><img src=\"/img/nav_next" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
***************
*** 443,449 ****
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")))))
    t)
  
  (defun nnweb-google-identity (url)
--- 442,449 ----
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")
!        ("filter" . "0")))))
    t)
  
  (defun nnweb-google-identity (url)

Re: nnweb fix

[Reiner Steib]

On Sat, Dec 31 2005, Andreas Seltenreich wrote:

> Andreas Seltenreich writes:
>
>> attached is an attempt to get web groups working again with the new
>> google groups.
>
> Oops, I missed some html entity decoding and a debug statement.
> Attached is an updated patch.

Your patch works fine for me.  Thanks.

However, your contribution larger than 10-15 lines (i.e. it's not a
"tiny change") so we need legal papers to install it.  Gnus is a part
of Emacs and since Emacs is a core GNU project, the Free Software
Foundation (FSF) requires an assignment of copyright for
contributions, so that it is able to defend Emacs' legal status
without doubts, should the need arise.  You can either sign papers
only for Gnus or for Emacs (which covers Gnus, too).

Have you filed a copyright assignment for work on Gnus or Emacs to the
Free Software Foundation?  If not, I can ask someone to send you the
required request form.  After the assignment process completes (you
get sent forms which you have to read and sign and send back), we can
install your patch.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

Reiner Steib writes:

> Have you filed a copyright assignment for work on Gnus or Emacs to the
> Free Software Foundation?

I'm afraid not.

> If not, I can ask someone to send you the required request form.
> After the assignment process completes (you get sent forms which you
> have to read and sign and send back), we can install your patch.

Agreed.

regards,
Andreas

Re: nnweb fix

[Andreas Seltenreich]

Reiner Steib writes:

> On Sat, Dec 31 2005, Andreas Seltenreich wrote:
>
>> Andreas Seltenreich writes:
>>
>>> attached is an attempt to get web groups working again with the new
>>> google groups.
>>
>> Oops, I missed some html entity decoding and a debug statement.
>> Attached is an updated patch.
>
> Your patch works fine for me.  Thanks.

While it works for ephemeral groups, solid ones seem still broken.

ISTM this broke solid groups back in 2003:
<http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>

I guess the solution is to adapt nnweb to use the server variables
instead of those custom args to nnweb-request-create-group for solid
groups?

> required request form.  After the assignment process completes (you
> get sent forms which you have to read and sign and send back), we can
> install your patch.

According to a mail from the FSF, the process is now complete.

regards,
Andreas

Re: nnweb fix

[Reiner Steib]

On Fri, Jan 27 2006, Andreas Seltenreich wrote:

> Reiner Steib writes:
>> On Sat, Dec 31 2005, Andreas Seltenreich wrote:
>>> Andreas Seltenreich writes:
>>>> attached is an attempt to get web groups working again with the new
>>>> google groups.
[...]
>> Your patch works fine for me.  Thanks.
>
> While it works for ephemeral groups, solid ones seem still broken.
>
> ISTM this broke solid groups back in 2003:
> <http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>
>
> I guess the solution is to adapt nnweb to use the server variables
> instead of those custom args to nnweb-request-create-group for solid
> groups?

Could you suggest a patch, please?

>> required request form.  After the assignment process completes (you
>> get sent forms which you have to read and sign and send back), we can
>> install your patch.
>
> According to a mail from the FSF, the process is now complete.

Thanks, I have installed your patch in the trunk and in the stable
branch (v5-10).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]

Reiner Steib writes:

> On Fri, Jan 27 2006, Andreas Seltenreich wrote:
>> While it works for ephemeral groups, solid ones seem still broken.
>>
>> ISTM this broke solid groups back in 2003:
>> <http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>
>>
>> I guess the solution is to adapt nnweb to use the server variables
>> instead of those custom args to nnweb-request-create-group for solid
>> groups?
>
> Could you suggest a patch, please?

How about the one attached?

2006-01-31  Andreas Seltenreich <uwi7@stud.uni-karlsruhe.de>

	* nnweb.el (nnweb-group-alist): Use defvar instead of defvoo, there's
	only one active file for all servers.
	(nnweb-request-scan): Make sure nnweb-articles is initialized on solid
	groups -- Gnus might have used a FAST request to select the group.
	(nnweb-request-group, nnweb-google-parse-1): nnweb-type and
	nnweb-search are no longer redundantly kept in the active file.
	(nnweb-request-list): Don't list bogus groups.  There can only be one.
	(nnweb-request-create-group): ARGS is no longer used.
	(nnweb-possibly-change-server, nnweb-request-group): Some
	initialisation removed -- let nnoo do the work.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5947 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.12
diff -c -r7.12 nnweb.el
*** lisp/nnweb.el	30 Jan 2006 14:55:15 -0000	7.12
--- lisp/nnweb.el	31 Jan 2006 05:43:57 -0000
***************
*** 1,7 ****
  ;;; nnweb.el --- retrieving articles via web search engines
  
  ;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
! ;;   2004, 2005 Free Software Foundation, Inc.
  
  ;; Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
  ;; Keywords: news
--- 1,7 ----
  ;;; nnweb.el --- retrieving articles via web search engines
  
  ;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
! ;;   2004, 2005, 2006 Free Software Foundation, Inc.
  
  ;; Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
  ;; Keywords: news
***************
*** 30,39 ****
  ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to Gmane
  ;; web groups (`gnus-group-make-web-group') doesn't work anymore.
  
- ;; FIXME: Solid web groups are currently broken because ARGS are no longer
- ;; passed from `gnus-group-make-web-group' to `nnweb-request-create-group'.
- ;; See revision 6.96 of `gnus-group.el' (2003-01-06).
- 
  ;;; Code:
  
  (eval-when-compile (require 'cl))
--- 30,35 ----
***************
*** 103,109 ****
  
  (defvoo nnweb-articles nil)
  (defvoo nnweb-buffer nil)
! (defvoo nnweb-group-alist nil)
  (defvoo nnweb-group nil)
  (defvoo nnweb-hashtb nil)
  
--- 99,105 ----
  
  (defvoo nnweb-articles nil)
  (defvoo nnweb-buffer nil)
! (defvar nnweb-group-alist nil)
  (defvoo nnweb-group nil)
  (defvoo nnweb-hashtb nil)
  
***************
*** 126,150 ****
  (deffoo nnweb-request-scan (&optional group server)
    (nnweb-possibly-change-server group server)
    (if nnweb-ephemeral-p
!       (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (funcall (nnweb-definition 'map))
    (unless nnweb-ephemeral-p
      (nnweb-write-active)
      (nnweb-write-overview group)))
  
  (deffoo nnweb-request-group (group &optional server dont-check)
!   (nnweb-possibly-change-server nil server)
!   (when (and group
! 	     (not (equal group nnweb-group))
! 	     (not nnweb-ephemeral-p))
!     (setq nnweb-group group
! 	  nnweb-articles nil)
!     (let ((info (assoc group nnweb-group-alist)))
!       (when info
! 	(setq nnweb-type (nth 2 info))
! 	(setq nnweb-search (nth 3 info))
! 	(unless dont-check
! 	  (nnweb-read-overview group)))))
    (cond
     ((not nnweb-articles)
      (nnheader-report 'nnweb "No matching articles"))
--- 122,140 ----
  (deffoo nnweb-request-scan (&optional group server)
    (nnweb-possibly-change-server group server)
    (if nnweb-ephemeral-p
!       (setq nnweb-hashtb (gnus-make-hashtable 4095))
!     (unless nnweb-articles
!       (nnweb-read-overview group)))
    (funcall (nnweb-definition 'map))
    (unless nnweb-ephemeral-p
      (nnweb-write-active)
      (nnweb-write-overview group)))
  
  (deffoo nnweb-request-group (group &optional server dont-check)
!   (nnweb-possibly-change-server group server)
!   (unless (or nnweb-ephemeral-p
! 	      dont-check)
!     (nnweb-read-overview group))
    (cond
     ((not nnweb-articles)
      (nnheader-report 'nnweb "No matching articles"))
***************
*** 208,214 ****
    (nnweb-possibly-change-server nil server)
    (save-excursion
      (set-buffer nntp-server-buffer)
!     (nnmail-generate-active nnweb-group-alist)
      t))
  
  (deffoo nnweb-request-update-info (group info &optional server)
--- 198,204 ----
    (nnweb-possibly-change-server nil server)
    (save-excursion
      (set-buffer nntp-server-buffer)
!     (nnmail-generate-active (list (assoc server nnweb-group-alist)))
      t))
  
  (deffoo nnweb-request-update-info (group info &optional server)
***************
*** 220,226 ****
  (deffoo nnweb-request-create-group (group &optional server args)
    (nnweb-possibly-change-server nil server)
    (nnweb-request-delete-group group)
!   (push `(,group ,(cons 1 0) ,@args) nnweb-group-alist)
    (nnweb-write-active)
    t)
  
--- 210,216 ----
  (deffoo nnweb-request-create-group (group &optional server args)
    (nnweb-possibly-change-server nil server)
    (nnweb-request-delete-group group)
!   (push `(,group ,(cons 1 0)) nnweb-group-alist)
    (nnweb-write-active)
    t)
  
***************
*** 290,307 ****
      def))
  
  (defun nnweb-possibly-change-server (&optional group server)
-   (nnweb-init server)
    (when server
      (unless (nnweb-server-opened server)
!       (nnweb-open-server server)))
    (unless nnweb-group-alist
      (nnweb-read-active))
    (unless nnweb-hashtb
      (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (when group
!     (when (and (not nnweb-ephemeral-p)
! 	       (equal group nnweb-group))
!       (nnweb-request-group group nil t))))
  
  (defun nnweb-init (server)
    "Initialize buffers and such."
--- 280,295 ----
      def))
  
  (defun nnweb-possibly-change-server (&optional group server)
    (when server
      (unless (nnweb-server-opened server)
!       (nnweb-open-server server))
!     (nnweb-init server))
    (unless nnweb-group-alist
      (nnweb-read-active))
    (unless nnweb-hashtb
      (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (when group
!     (setq nnweb-group group)))
  
  (defun nnweb-init (server)
    "Initialize buffers and such."
***************
*** 347,354 ****
  	Subject Score Date Newsgroups From
  	map url mid)
      (unless active
!       (push (list nnweb-group (setq active (cons 1 0))
! 		  nnweb-type nnweb-search)
  	    nnweb-group-alist))
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
--- 335,341 ----
  	Subject Score Date Newsgroups From
  	map url mid)
      (unless active
!       (push (list nnweb-group (setq active (cons 1 0)))
  	    nnweb-group-alist))
      ;; Go through all the article hits on this page.
      (goto-char (point-min))

Re: nnweb fix

[Reiner Steib]

On Tue, Jan 31 2006, Andreas Seltenreich wrote:

> Reiner Steib writes:
>> On Fri, Jan 27 2006, Andreas Seltenreich wrote:
>>> I guess the solution is to adapt nnweb to use the server variables
>>> instead of those custom args to nnweb-request-create-group for solid
>>> groups?
>>
>> Could you suggest a patch, please?
>
> How about the one attached?

Fine, thanks.  Committed to trunk and v5-10.

Would you like to work on Gmane search <http://search.gmane.org/>,
too?  ;-) We could ask Lars and Olly Betts (he did much work on the
Xapian-powered Gmane search) on gmane.discuss to generate more
nnweb-friendly HTML output ("&output=nnweb"; maybe no frames?) if that
would make the task easier.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Reiner Steib writes:

> Fine, thanks.  Committed to trunk and v5-10.
>
> Would you like to work on Gmane search <http://search.gmane.org/>,
> too?  ;-)

Heh, I guess hunting bugs I just introduced has a higher priority.
E.g., those exponentially growing overview files :-/


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 580 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.13
diff -u -r7.13 nnweb.el
--- lisp/nnweb.el	31 Jan 2006 15:51:27 -0000	7.13
+++ lisp/nnweb.el	31 Jan 2006 16:37:03 -0000
@@ -133,7 +133,8 @@
 (deffoo nnweb-request-group (group &optional server dont-check)
   (nnweb-possibly-change-server group server)
   (unless (or nnweb-ephemeral-p
-	      dont-check)
+	      dont-check
+	      nnweb-articles)
     (nnweb-read-overview group))
   (cond
    ((not nnweb-articles)

[-- Attachment #3: Type: text/plain, Size: 38 bytes --]


Sorry for the inconvenience,
Andreas

Re: nnweb fix

[Reiner Steib]

On Tue, Jan 31 2006, Andreas Seltenreich wrote:

> Heh, I guess hunting bugs I just introduced has a higher priority.
> E.g., those exponentially growing overview files :-/
[...]
>    (unless (or nnweb-ephemeral-p
> -	      dont-check)
> +	      dont-check
> +	      nnweb-articles)

Thanks.  Committed.

(BTW, your entry has been added to the copyright list in the
meantime.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Hi,

it looks like the regexps in nnweb-google-wash-article broke in the
meantime. Attached is an update.

I also reverted the change to declare nnweb-group-alist with defvar
instead of defvoo. I didn't realise that it is held on disk
/multiplexed/ with the defvoo'd nnweb-directory, which would result in
bogus active file entries if one would use more than one directory for
nnweb.

Also, nnlistserv seems to inherit from nnweb via nnoo, and I'm not
sure what the implications are, if it wasn't declared with defvoo
(anyone)?

regards,
Andreas

2006-02-12  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>

        * nnweb.el (nnweb-google-wash-article): Update regexps.
        (nnweb-group-alist): Use defvoo instead of defvar,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 1593 bytes --]

Index: nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.14
diff -u -r7.14 nnweb.el
--- nnweb.el	3 Feb 2006 13:15:13 -0000	7.14
+++ nnweb.el	12 Feb 2006 05:55:27 -0000
@@ -99,7 +99,7 @@
 
 (defvoo nnweb-articles nil)
 (defvoo nnweb-buffer nil)
-(defvar nnweb-group-alist nil)
+(defvoo nnweb-group-alist nil)
 (defvoo nnweb-group nil)
 (defvoo nnweb-hashtb nil)
 
@@ -309,22 +309,26 @@
 
 (defun nnweb-google-wash-article ()
   ;; We have Google's masked e-mail addresses here.  :-/
-  (let ((case-fold-search t))
+  (let ((case-fold-search t)
+	(start-re "<pre>\n *")
+	(end-re "\n *</pre>"))
     (goto-char (point-min))
     (if (save-excursion
 	  (or (re-search-forward "The requested message.*could not be found."
 				 nil t)
-	      (not (and (re-search-forward "^<pre>" nil t)
-			(re-search-forward "^</pre>" nil t)))))
+	      (not (and (re-search-forward start-re nil t)
+			(re-search-forward end-re nil t)))))
 	;; FIXME: Don't know how to indicate "not found".
 	;; Should this function throw an error?  --rsteib
 	(progn
 	  (gnus-message 3 "Requested article not found")
 	  (erase-buffer))
       (delete-region (point-min)
-		     (1+ (re-search-forward "^<pre>" nil t)))
+		     (re-search-forward start-re))
       (goto-char (point-min))
-      (delete-region (- (re-search-forward "^</pre>" nil t) (length "</pre>"))
+      (delete-region (progn
+		       (re-search-forward end-re)
+		       (match-beginning 0))
 		     (point-max))
       (mm-url-decode-entities))))
 

Re: nnweb fix

[Reiner Steib]

On Sun, Feb 12 2006, Andreas Seltenreich wrote:

> I also reverted the change to declare nnweb-group-alist with defvar
> instead of defvoo. I didn't realise that it is held on disk
> /multiplexed/ with the defvoo'd nnweb-directory, which would result in
> bogus active file entries if one would use more than one directory for
> nnweb.
>
> Also, nnlistserv seems to inherit from nnweb via nnoo, and I'm not
> sure what the implications are, if it wasn't declared with defvoo
> (anyone)?

I don't know.

> 2006-02-12  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>
>
>         * nnweb.el (nnweb-google-wash-article): Update regexps.
>         (nnweb-group-alist): Use defvoo instead of defvar,

Thanks.  Applied to the trunk.  I'll update v5-10 once the Gmane
search is updated, too.  ;-)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

nnweb + Gmane search (was: nnweb fix)

[Andreas Seltenreich]

Reiner Steib writes:

> Would you like to work on Gmane search <http://search.gmane.org/>,
> too?  ;-) We could ask Lars and Olly Betts (he did much work on the
> Xapian-powered Gmane search) on gmane.discuss to generate more
> nnweb-friendly HTML output ("&output=nnweb"; maybe no frames?) if that
> would make the task easier.

It seems to work quite well after adjusting URLs and regexps. The only
drop of bitterness is that there are only 10 hits per page. So
nnweb-max-hits' default value of 999 would mean ten requests for
Google searches, but 100 for Gmane ones.

If an option to customise the number of results per page could be
added to the Xapian-powered search, it'd be as usable via nnweb as the
google one.

Of course, if there was an option to output the result set as S-exps
or nov-lines it'd be even better :-).

Btw, is there a way to inline group or author search in the main
search string? With google you could search for, e.g.,

   "group:*nethack insubject:pudding-farming"

...which works nicely with nnweb's single-search-string approach.

regards,
Andreas

Re: nnweb + Gmane search (was: nnweb fix)

[Olly Betts]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
> It seems to work quite well after adjusting URLs and regexps. The only
> drop of bitterness is that there are only 10 hits per page. So
> nnweb-max-hits' default value of 999 would mean ten requests for
> Google searches, but 100 for Gmane ones.

Please don't do that!

> If an option to customise the number of results per page could be
> added to the Xapian-powered search, it'd be as usable via nnweb as the
> google one.

It already exists, just pass a CGI parameter called HITSPERPAGE.  The value
is clipped to the range 10-1000.

> Of course, if there was an option to output the result set as S-exps
> or nov-lines it'd be even better .

I don't have enough information available to produce nov-lines unless some
fields aren't required (no message-id for example).  Other output formats
aren't hard to add, but require me to update two places if anything changes
so are a bit more work to support.  So I'd prefer something more generic than
S-exps.

> Btw, is there a way to inline group or author search in the main
> search string? With google you could search for, e.g.,
> 
>    "group:*nethack insubject:pudding-farming"
> 
> ...which works nicely with nnweb's single-search-string approach.

There's partial support for group: but it needs more work.  You need to
strip the "gmane." prefix from the groupname:

    group:emacs.gnus.general xapian

Only trailing wildcards are supported and must be after a ".":

    group:emacs.gnus.* xapian

Cheers,
    Olly

Re: nnweb + Gmane search

[Andreas Seltenreich]

Olly Betts writes:
> Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
>> If an option to customise the number of results per page could be
>> added to the Xapian-powered search, it'd be as usable via nnweb as the
>> google one.
>
> It already exists, just pass a CGI parameter called HITSPERPAGE.  The value
> is clipped to the range 10-1000.

Great, would it be okay to just set HITSPERPAGE to nnweb-max-hits and
doing a single request per search? One could also decrease the default
upper bound of 999 articles, but if this really is an issue, it would
be better to hardcode a limit for the Gmane part of nnweb, since
nnweb-max-hits is a user-customizable variable.

>> Of course, if there was an option to output the result set as S-exps
>> or nov-lines it'd be even better .
>
> I don't have enough information available to produce nov-lines unless some
> fields aren't required (no message-id for example).

It wouldn't have to be complete, just enough to present the hits to
the user plus something in the xref header one can construct links to
the raw articles from. Currently, nnweb parses the html output and
generates nov-lines like the following:

19	(gmane.emacs.gnus.general) Re: nnweb fix (96%)	Andreas Seltenreich <andreas+ding <at> gate450 <dot> dyndns <dot> org>	2006-01-31 05:48:02 GMT	<87vevtxjjv.fsf@totally-fudged-out-message-id>		0	0	Xref: http://article.gmane.org/gmane.emacs.gnus.general/61805/raw	

> Other output formats aren't hard to add, but require me to update
> two places if anything changes so are a bit more work to support. So
> I'd prefer something more generic than S-exps.

Now, does "more generic than S-exps" apply to nov-lines :-)? As
stated, it already works by parsing the html output format, but it
kind of pains me to waste bandwidth on useless markup, and those
regexps are likely to break on markup changes.

Thanks!
Andreas

Re: nnweb + Gmane search

[Olly Betts]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
> Great, would it be okay to just set HITSPERPAGE to nnweb-max-hits and
> doing a single request per search? One could also decrease the default
> upper bound of 999 articles, but if this really is an issue, it would
> be better to hardcode a limit for the Gmane part of nnweb, since
> nnweb-max-hits is a user-customizable variable.

I've just tried fetching 1000 matches, and it typically takes several
seconds (note that the "search time" is printed before the records
are read so it under-reports in this case).  

If this proves to be a problem I can easily impose a lower limit on the
server though.

> It wouldn't have to be complete, just enough to present the hits to
> the user plus something in the xref header one can construct links to
> the raw articles from. Currently, nnweb parses the html output and
> generates nov-lines like the following:

Anything you can produce from the HTML output I can produce on the server
of course.

> > Other output formats aren't hard to add, but require me to update
> > two places if anything changes so are a bit more work to support. So
> > I'd prefer something more generic than S-exps.
> 
> Now, does "more generic than S-exps" apply to nov-lines ?

I think so.  NOV is really just tab-separated fields.
dd  Just don't be tempted to work around by running more
searches to fetch 999 if I do...

How does this first attempt look:

http://search.gmane.org/cgi-bin/omega.cgi?FMT=nov&query=gnus+nnweb

The first line is either an error, or:

OK <total (estimated) matches> <number of matches which follow> <match time>

Beware the extra blank line at the end (which is hard to avoid right
now).  And note that encrypted addresses will be missing the
"@public.gmane.org" (because that's how they're stored internally to
save space).

Beware the URL and format may not be stable, so don't build it into
anything just yet.  

Cheers,
    Olly

Re: nnweb + Gmane search

[Olly Betts]

Olly Betts <olly@survex.com> writes:
> If this proves to be a problem I can easily impose a lower limit on the
> server though.
[...]
> I think so.  NOV is really just tab-separated fields.
> dd  Just don't be tempted to work around by running more
> searches to fetch 999 if I do...

Erm, those last two lines should be with the first.  I posted using Loom
but I'm more used to vim key bindings...

> How does this first attempt look:
> 
> http://search.gmane.org/cgi-bin/omega.cgi?FMT=nov&query=gnus+nnweb
> 
> The first line is either an error, or:
> 
> OK <total (estimated) matches> <number of matches which follow> <match time>

Forgot to say - if any of the current output format is inconvenient, I
can probably easily fix it.  In particular, I suspect the current date
format will probably want adjusting to be a valid date in news format.

Cheers,
    Olly

Re: nnweb + Gmane search

[Andreas Seltenreich]

Olly Betts writes:

> Olly Betts <olly@survex.com> writes:
>> How does this first attempt look:
>> 
>> http://search.gmane.org/cgi-bin/omega.cgi?FMT=nov&query=gnus+nnweb
>> 
>> The first line is either an error, or:
>> 
>> OK <total (estimated) matches> <number of matches which follow> <match time>
>
> Forgot to say - if any of the current output format is inconvenient, I
> can probably easily fix it.  In particular, I suspect the current date
> format will probably want adjusting to be a valid date in news format.

If I correctly interpreted <news:nov-faq-1-817902143@agate.Berkeley.EDU>,
one would also have to put numbers in the first column, and the
optional xref header comes one field too early and would need to
actually be prefixed with "Xref: " to yield a proper nov-line.

Don't get me wrong, the current format is /perfectly/ fine with nnweb,
since nnweb generates article numbers using the Xref content anyway,
but if more software is supposed to use this interface, one might want
to get it right the first time.

| Beware the URL and format may not be stable, so don't build it into
| anything just yet.  

I'll hold my breath :-)

Thanks!
Andreas

Re: nnweb + Gmane search

[Olly Betts]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
> If I correctly interpreted <news:nov-faq-1-817902143 <at> agate.Berkeley.EDU>,
> one would also have to put numbers in the first column

OK, I seem to have skipped over that when entering the fields.  I've just used
the hit number (starting from 1), which seems about as good as anything.

> and the
> optional xref header comes one field too early and would need to
> actually be prefixed with "Xref: " to yield a proper nov-line.

The missing tab is due to a typo.  I've fixed that and added "Xref: ", plus
and "X-Score: " prefix on the percentage score.

How's it looking now?

Cheers,
    Olly

Re: nnweb + Gmane search

[Andreas Seltenreich]

Olly Betts writes:

> The missing tab is due to a typo.  I've fixed that and added "Xref: ", plus
> and "X-Score: " prefix on the percentage score.
>
> How's it looking now?

Very well, Gnus' NOV parsing code swallows it nicely. So I guess it is
save now to make a patch using the current URL and format?

Btw, one very minor issue I stumbled upon while testing i18n is that
the charset is declared incorrectly:

| Content-Type: text/plain; charset=iso-8859-1

regards,
Andreas

Re: nnweb + Gmane search

[Reiner Steib]

On Tue, Feb 14 2006, Andreas Seltenreich wrote:

> Olly Betts writes:

[ http://search.gmane.org/cgi-bin/omega.cgi?FMT=nov&query=gnus+nnweb ]

>> The missing tab is due to a typo.  I've fixed that and added "Xref: ", plus
>> and "X-Score: " prefix on the percentage score.
>>
>> How's it looking now?
>
> Very well, Gnus' NOV parsing code swallows it nicely. So I guess it is
> save now to make a patch using the current URL and format?

Please do, unless Olly isn't happy with the current output.  We can
easily adjust the URL in Gnus later if Olly doesn't want to include
FMT=nov in the default CGI script yet.  But it would be nice to have a
permanent URL.

> Btw, one very minor issue I stumbled upon while testing i18n is that
> the charset is declared incorrectly:
>
> | Content-Type: text/plain; charset=iso-8859-1

Maybe just an omega.cgi issue.
http://search.gmane.org/?query=Seltenreich+gmane sends charset=utf-8.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb + Gmane search

[Olly Betts]

On 2006-02-23, Reiner Steib <reinersteib+gmane@imap.cc> wrote:
> Please do, unless Olly isn't happy with the current output.  We can
> easily adjust the URL in Gnus later if Olly doesn't want to include
> FMT=nov in the default CGI script yet.  But it would be nice to have a
> permanent URL.

I'm happy with the output (apart from the extra newline at the end but
that's a very minor issue).

I'd rather not be running omega.cgi directly like that, because it's
really an implementation detail.  I've been meaning to set up an
"official" wrapper URL for it but I've been a bit busy recently.

>> Btw, one very minor issue I stumbled upon while testing i18n is that
>> the charset is declared incorrectly:
>>
>> | Content-Type: text/plain; charset=iso-8859-1
>
> Maybe just an omega.cgi issue.
> http://search.gmane.org/?query=Seltenreich+gmane sends charset=utf-8.

Yeah, that'll just be a default charset coming from somewhere or other.
The output will be in utf-8.

Cheers,
    Olly

Re: nnweb + Gmane search

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Olly Betts writes:

> On 2006-02-23, Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>> Please do, unless Olly isn't happy with the current output.  We can
>> easily adjust the URL in Gnus later if Olly doesn't want to include
>> FMT=nov in the default CGI script yet.  But it would be nice to have a
>> permanent URL.
>
> I'm happy with the output (apart from the extra newline at the end but
> that's a very minor issue).

Ok, I've attached a patch. Future changes in the output to anything
besides the Xref header should be transparent to the code. E.g., if
users wanted a "Newsgroups:" extra header, it should "just work" as
long as it is valid nov.

regards,
andreas

2006-02-24  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>

	* nnweb.el (nnweb-type-definition, nnweb-gmane-create-mapping,
        nnweb-gmane-wash-article, nnweb-gmane-search): Fix Gmane web
        groups.  Kudos to Olly Betts for providing NOV output on the
        server side.
        (nnweb-google-create-mapping): Update regexps and add some
        progress indication.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 7042 bytes --]

Index: nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.15
diff -c -r7.15 nnweb.el
*** nnweb.el	13 Feb 2006 13:32:28 -0000	7.15
--- nnweb.el	23 Feb 2006 19:39:02 -0000
***************
*** 27,35 ****
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
- ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to Gmane
- ;; web groups (`gnus-group-make-web-group') doesn't work anymore.
- 
  ;;; Code:
  
  (eval-when-compile (require 'cl))
--- 27,32 ----
***************
*** 82,88 ****
       (reference . identity)
       (map . nnweb-gmane-create-mapping)
       (search . nnweb-gmane-search)
!      (address . "http://gmane.org/")
       (identifier . nnweb-gmane-identity)))
    "Type-definition alist.")
  
--- 79,85 ----
       (reference . identity)
       (map . nnweb-gmane-create-mapping)
       (search . nnweb-gmane-search)
!      (address . "http://search.gmane.org/cgi-bin/omega.cgi")
       (identifier . nnweb-gmane-identity)))
    "Type-definition alist.")
  
***************
*** 407,412 ****
--- 404,410 ----
    (save-excursion
      (set-buffer nnweb-buffer)
      (erase-buffer)
+     (nnheader-message 7 "Searching google...")
      (when (funcall (nnweb-definition 'search) nnweb-search)
  	(let ((more t)
  	      (i 0))
***************
*** 417,431 ****
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td><a href=\"\n\\([^>\"]+\\)\"><img src=\"/img/nav_next" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
  	      (setq more (concat (nnweb-definition 'base) (match-string 1)))
  	    (when more
  	      (erase-buffer)
  	      (mm-url-insert more))))
  	  ;; Return the articles in the right order.
  	  (setq nnweb-articles
  		(sort nnweb-articles 'car-less-than-car))))))
  
--- 415,432 ----
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<a href=\"\n\\([^>\"]+\\)\"><img src=\"[^\"]+next"
! 			  nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
  	      (setq more (concat (nnweb-definition 'base) (match-string 1)))
  	    (when more
  	      (erase-buffer)
+ 	      (nnheader-message 7 "Searching google...(%d)" i)
  	      (mm-url-insert more))))
  	  ;; Return the articles in the right order.
+ 	  (nnheader-message 7 "Searching google...done")
  	  (setq nnweb-articles
  		(sort nnweb-articles 'car-less-than-car))))))
  
***************
*** 458,503 ****
    "Perform the search and create a number-to-url alist."
    (save-excursion
      (set-buffer nnweb-buffer)
!     (erase-buffer)
!     (when (funcall (nnweb-definition 'search) nnweb-search)
!       (let ((more t)
! 	    (case-fold-search t)
! 	    (active (or (cadr (assoc nnweb-group nnweb-group-alist))
! 			(cons 1 0)))
! 	    subject group url
! 	    map)
! 	  ;; Remove stuff from the beginning of results
! 	(goto-char (point-min))
! 	(search-forward "Search Results</h1><ul>" nil t)
! 	(delete-region (point-min) (point))
  	(goto-char (point-min))
! 	;; Iterate over the actual hits
! 	(while (re-search-forward ".*href=\"\\([^\"]+\\)\">\\(.*\\)" nil t)
! 	    (setq url (concat "http://gmane.org/" (match-string 1)))
! 	    (setq subject (match-string 2))
! 	  (unless (nnweb-get-hashtb url)
! 	    (push
! 	     (list
! 	      (incf (cdr active))
! 	      (make-full-mail-header
! 	       (cdr active) (concat  "(" group ") " subject) nil nil
! 	       nil nil 0 0 url))
! 	     map)
! 	    (nnweb-set-hashtb (cadar map) (car map))))
! 	;; Return the articles in the right order.
! 	(setq nnweb-articles
! 	      (sort (nconc nnweb-articles map) 'car-less-than-car))))))
  
  (defun nnweb-gmane-wash-article ()
    (let ((case-fold-search t))
      (goto-char (point-min))
!     (search-forward "<!--X-Head-of-Message-->" nil t)
!     (delete-region (point-min) (point))
!     (goto-char (point-min))
!     (while (looking-at "^<li><em>\\([^ ]+\\)</em>.*</li>")
!       (replace-match "\\1\\2" t)
!       (forward-line 1))
!     (mm-url-remove-markup)))
  
  (defun nnweb-gmane-search (search)
    (mm-url-insert
--- 459,519 ----
    "Perform the search and create a number-to-url alist."
    (save-excursion
      (set-buffer nnweb-buffer)
!     (let ((case-fold-search t)
! 	  (active (or (cadr (assoc nnweb-group nnweb-group-alist))
! 		      (cons 1 0)))
! 	  map)
!       (erase-buffer)
!       (nnheader-message 7 "Searching Gmane..." )
!       (when (funcall (nnweb-definition 'search) nnweb-search)
  	(goto-char (point-min))
! 	;; Skip the status line
! 	(forward-line 1)
! 	;; Thanks to Olly Betts we now have NOV lines in our buffer!
! 	(while (not (eobp))
! 	  (unless (eolp)
! 	    (let ((header (nnheader-parse-nov)))
! 	      (let ((xref (mail-header-xref header))
! 		    (from (mail-header-from header))
! 		    (subject (mail-header-subject header))
! 		    (rfc2047-encoding-type 'mime))
! 		(when (string-match " \\([^:]+\\):\\([0-9]+\\)" xref)
! 		  (mail-header-set-xref
! 		   header
! 		   (format "http://article.gmane.org/%s/%s/raw"
! 			   (match-string 1 xref)
! 			   (match-string 2 xref))))
! 
! 		;; Add host part to gmane-encrypted addresses
! 		(when (string-match "@$" from)
! 		  (mail-header-set-from header
! 					(concat from "public.gmane.org")))
! 
! 		(mail-header-set-subject header
! 					 (rfc2047-encode-string subject))
! 
! 		(unless (nnweb-get-hashtb (mail-header-xref header))
! 		  (push
! 		   (list
! 		    (incf (cdr active))
! 		    header)
! 		   map)
! 		  (nnweb-set-hashtb (cadar map) (car map))))))
! 	  (forward-line 1)))
!       (nnheader-message 7 "Searching Gmane...done")
!       (setq nnweb-articles
! 	    (sort (nconc nnweb-articles map) 'car-less-than-car)))))
  
  (defun nnweb-gmane-wash-article ()
    (let ((case-fold-search t))
      (goto-char (point-min))
!     (when (search-forward "<!--X-Head-of-Message-->" nil t)
!       (delete-region (point-min) (point))
!       (goto-char (point-min))
!       (while (looking-at "^<li><em>\\([^ ]+\\)</em>.*</li>")
! 	(replace-match "\\1\\2" t)
! 	(forward-line 1))
!       (mm-url-remove-markup))))
  
  (defun nnweb-gmane-search (search)
    (mm-url-insert
***************
*** 505,514 ****
      (nnweb-definition 'address)
      "?"
      (mm-url-encode-www-form-urlencoded
!      `(("query" . ,search)))))
    (setq buffer-file-name nil)
    t)
- 
  
  (defun nnweb-gmane-identity (url)
    "Return a unique identifier based on URL."
--- 521,533 ----
      (nnweb-definition 'address)
      "?"
      (mm-url-encode-www-form-urlencoded
!      `(("query" . ,search)
!        ("FMT" . "nov")
!        ("HITSPERPAGE" . ,(number-to-string nnweb-max-hits))))))
    (setq buffer-file-name nil)
+   (set-buffer-multibyte t)
+   (mm-decode-coding-region (point-min) (point-max) 'utf-8)
    t)
  
  (defun nnweb-gmane-identity (url)
    "Return a unique identifier based on URL."

Re: nnweb + Gmane search

[Olly Betts]

On 2006-02-23, Olly Betts <olly@survex.com> wrote:
> I'd rather not be running omega.cgi directly like that, because it's
> really an implementation detail.  I've been meaning to set up an
> "official" wrapper URL for it but I've been a bit busy recently.

OK, done (and this sets the charset correctly too):

http://search.gmane.org/nov.php?query=gnus+nnweb

Cheers,
    Olly

Re: nnweb + Gmane search

[Reiner Steib]

On Fri, Feb 24 2006, Andreas Seltenreich wrote:

> 2006-02-24  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>
>
> 	* nnweb.el (nnweb-type-definition, nnweb-gmane-create-mapping,
>         nnweb-gmane-wash-article, nnweb-gmane-search): Fix Gmane web
>         groups.  Kudos to Olly Betts for providing NOV output on the
>         server side.
>         (nnweb-google-create-mapping): Update regexps and add some
>         progress indication.

Thanks, installed on the trunk.  I will merge it to v5-10 later.

Andreas, would you like to provide an update for the manual
(texi/gnus.texi)?  I'd like to have this example...

,----[ (info "(gnus)Foreign Groups") ]
|      If you use the `google' search engine, you can limit the search to
|      a particular group by using a match string like `shaving
|      group:alt.sysadmin.recovery'.
`----

... moved to (info "(gnus)Web Searches") and have some additional
examples such as restricting a search to a certain author, newsgroup
or subject (if possible) for both, Gmane and Google in (info
"(gnus)Web Searches").  It's not a problem if you're not familiar with
texinfo: You write it; we can fix the markup before committing it.
(While at it, "dejanews" could be removed in the manual.)


On Fri, Feb 24 2006, Olly Betts wrote:

> On 2006-02-23, Olly Betts <olly@survex.com> wrote:
>> I'd rather not be running omega.cgi directly like that, because it's
>> really an implementation detail.  I've been meaning to set up an
>> "official" wrapper URL for it but I've been a bit busy recently.
>
> OK, done (and this sets the charset correctly too):

Andreas, do we simply need to change...

-  (mm-decode-coding-region (point-min) (point-max) 'utf-8)
+  (mm-decode-coding-region (point-min) (point-max))

... or anything else?

> http://search.gmane.org/nov.php?query=gnus+nnweb

Installed.  Thanks.

Is it also possible to submit the "Sort by" flag (Relevance, Date,
Reverse Date) to nov.php?

This could also be included in the examples section of the manual.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb + Gmane search

[Olly Betts]

On 2006-02-24, Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>> http://search.gmane.org/nov.php?query=gnus+nnweb
>
> Is it also possible to submit the "Sort by" flag (Relevance, Date,
> Reverse Date) to nov.php?

Yes.  It supports all the CGI parameters which the HTML output search
does.

Cheers,
    Olly

Re: nnweb + Gmane search

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 3816 bytes --]

(removed gmane-discuss)

Reiner Steib writes:

> On Fri, Feb 24 2006, Andreas Seltenreich wrote:
>
> Andreas, would you like to provide an update for the manual
> (texi/gnus.texi)?  I'd like to have this example...

Oh, there's even a gnus-coding.texi in there. I guess I should have
read that before starting to work on nnweb :-)

> ,----[ (info "(gnus)Foreign Groups") ]
> |      If you use the `google' search engine, you can limit the search to
> |      a particular group by using a match string like `shaving
> |      group:alt.sysadmin.recovery'.
> `----
>
> ... moved to (info "(gnus)Web Searches") and have some additional
> examples such as restricting a search to a certain author, newsgroup
> or subject (if possible) for both, Gmane and Google in (info
> "(gnus)Web Searches").

Hmm, that paragraph about solid groups not being useful seems out of
date, too. Was it written at a time when solid groups weren't backed
up by hash tables, or am I missing something?

--8<---------------cut here---------------start------------->8---
   `nnweb' groups don't really lend themselves to being solid
groups--they have a very fleeting idea of article numbers.  In fact,
each time you enter an `nnweb' group (not even changing the search
pattern), you are likely to get the articles ordered in a different
manner.  Not even using duplicate suppression (*note Duplicate
Suppression::) will help, since `nnweb' doesn't even know the
`Message-ID' of the articles before reading them using some search
engines (Google, for instance).  The only possible way to keep track of
which articles you've read is by scoring on the `Date' header--mark all
articles posted before the last date you read the group as read.
--8<---------------cut here---------------end--------------->8---

> It's not a problem if you're not familiar with
> texinfo: You write it; we can fix the markup before committing it.
> (While at it, "dejanews" could be removed in the manual.)

I'm more concerned about my English skills than texinfo.

> Andreas, do we simply need to change...
>
> -  (mm-decode-coding-region (point-min) (point-max) 'utf-8)
> +  (mm-decode-coding-region (point-min) (point-max))
>
> ... or anything else?

It still doesn't work for me without explicit decoding using
mm-url-use-external + wget. However, it seems to decode fine
implicitly when using internal fetching. I dunno if there is a more
elegant way to solve this than testing for mm-url-use-external. And
wouldn't using external programs nevertheless break or unbreak
depending on the system locale?

Btw, internal url fetching yields a \0d at the end of the buffer,
which makes nnweb-gmane-create-mapping throw an error. I've attached a
patch.

>> http://search.gmane.org/nov.php?query=gnus+nnweb
>
> Installed.  Thanks.
>
> Is it also possible to submit the "Sort by" flag (Relevance, Date,
> Reverse Date) to nov.php?

At the moment one could achieve that by customising
nnweb-type-definition:

--8<---------------cut here---------------start------------->8---
(defun my-nnweb-gmane-search-by-date (search)
  (mm-url-insert
   (concat
    (nnweb-definition 'address)
    "?"
    (mm-url-encode-www-form-urlencoded
     `(("query" . ,search)
       ("sort" . "date")
       ("HITSPERPAGE" . ,(number-to-string nnweb-max-hits))))))
  (setq buffer-file-name nil)
  (set-buffer-multibyte t)
  (mm-decode-coding-region (point-min) (point-max) 'utf-8)
  t)

(push '(gmane-by-date
	(article . nnweb-gmane-wash-article)
	(id . "http://gmane.org/view.php?group=%s")
	(reference . identity)
	(map . nnweb-gmane-create-mapping)
	(search . my-nnweb-gmane-search-by-date)
	(address . "http://search.gmane.org/nov.php")
	(identifier . nnweb-gmane-identity))
      nnweb-type-definition)
--8<---------------cut here---------------end--------------->8---

regards,
andreas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 1155 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.17
diff -u -r7.17 nnweb.el
--- lisp/nnweb.el	24 Feb 2006 14:01:44 -0000	7.17
+++ lisp/nnweb.el	24 Feb 2006 21:55:56 -0000
@@ -471,7 +471,7 @@
 	(forward-line 1)
 	;; Thanks to Olly Betts we now have NOV lines in our buffer!
 	(while (not (eobp))
-	  (unless (eolp)
+	  (unless (or (eolp) (looking-at "\x0d"))
 	    (let ((header (nnheader-parse-nov)))
 	      (let ((xref (mail-header-xref header))
 		    (from (mail-header-from header))
Index: lisp/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1034
diff -u -r7.1034 ChangeLog
--- lisp/ChangeLog	24 Feb 2006 20:49:02 -0000	7.1034
+++ lisp/ChangeLog	24 Feb 2006 21:56:00 -0000
@@ -1,3 +1,7 @@
+2006-02-24  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>
+
+	* nnweb.el (nnweb-gmane-create-mapping): Don't choke on ^M.
+
 2006-02-20  Lars Magne Ingebrigtsen  <larsi@gnus.org>
 
 	* gnus-int.el (gnus-open-server): Respect gnus-batch-mode.

Re: nnweb + Gmane search

[Reiner Steib]

On Fri, Feb 24 2006, Andreas Seltenreich wrote:

> Reiner Steib writes:
>> Andreas, would you like to provide an update for the manual
>> (texi/gnus.texi)?  I'd like to have this example...
>
> Oh, there's even a gnus-coding.texi in there. I guess I should have
> read that before starting to work on nnweb :-)

Maybe gnus-coding.texi needs more improvement than nnweb.  I would
have complained if something obvious is wrong with your patches WRT
coding style.

> Hmm, that paragraph about solid groups not being useful seems out of
> date, too. Was it written at a time when solid groups weren't backed
> up by hash tables, or am I missing something?

I don't know.  `vc-annotate' says that the stuff in gnus.texi was last
change on 11-Nov-99 whereas some places in nnweb.el with the string
"hash" are unchanged since to 04-Mar-97.

>> It's not a problem if you're not familiar with
>> texinfo: You write it; we can fix the markup before committing it.
>> (While at it, "dejanews" could be removed in the manual.)
>
> I'm more concerned about my English skills than texinfo.

Don't worry.  We can also polish this.  Though I'm not sure if my
English is better than yours.  :-)

>> -  (mm-decode-coding-region (point-min) (point-max) 'utf-8)
>> +  (mm-decode-coding-region (point-min) (point-max))
>>
>> ... or anything else?
>
> It still doesn't work for me without explicit decoding using
> mm-url-use-external + wget. However, it seems to decode fine
> implicitly when using internal fetching. I dunno if there is a more
> elegant way to solve this than testing for mm-url-use-external. 

I don't think the Gmane's output encoding will change, so maybe we can
leave it as it is.

> And wouldn't using external programs nevertheless break or unbreak
> depending on the system locale?

Katsumi Yamaoka might know about this.  I'd guess that emacs-w3m also
has to deal with this.

>> Is it also possible to submit the "Sort by" flag (Relevance, Date,
>> Reverse Date) to nov.php?
>
> At the moment one could achieve that by customising
> nnweb-type-definition: [...]

I had in mind to modify the interactive form of
`gnus-group-make-web-group' to query (with completing) for additional
criteria after the "Search engine" prompt.  (We could bind this
variant to `G W'.)

> +2006-02-24  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>
> +
> +	* nnweb.el (nnweb-gmane-create-mapping): Don't choke on ^M.

Thanks.  Installed.

(Most of the time it's better to include the ChangeLog entry as plain
text in the article because a patch usually fails.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/