nnweb fix

nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

Hi,

the google part of nnweb recently broke for me again (both, web groups
and referring articles). The attached patch seems to fix it.

I don't know if is related to the problems recently reported on
<info-gnus-english@gnu.org> though. While the symptoms are the same
(blank articles), I wasn't able to reproduce it back then, and web
groups weren't affected.

Thanks,
andreas

2006-04-22  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>

	* nnweb.el (nnweb-google-wash-article): Update regexp.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 943 bytes --]

Index: nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.19
diff -c -r7.19 nnweb.el
*** nnweb.el	4 Apr 2006 21:35:07 -0000	7.19
--- nnweb.el	22 Apr 2006 09:57:25 -0000
***************
*** 307,314 ****
  (defun nnweb-google-wash-article ()
    ;; We have Google's masked e-mail addresses here.  :-/
    (let ((case-fold-search t)
! 	(start-re "<pre>\n *")
! 	(end-re "\n *</pre>"))
      (goto-char (point-min))
      (if (save-excursion
  	  (or (re-search-forward "The requested message.*could not be found."
--- 307,314 ----
  (defun nnweb-google-wash-article ()
    ;; We have Google's masked e-mail addresses here.  :-/
    (let ((case-fold-search t)
! 	(start-re "<pre>[\r\n ]*")
! 	(end-re "[\r\n ]*</pre>"))
      (goto-char (point-min))
      (if (save-excursion
  	  (or (re-search-forward "The requested message.*could not be found."

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

> the google part of nnweb recently broke for me again (both, web groups
> and referring articles). The attached patch seems to fix it.

Thanks for the patch; I've applied it to No Gnus v0.5 (i. e., CVS).

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Tassilo Horn]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

Hi Andreas,

> the google part of nnweb recently broke for me again (both, web groups
> and referring articles). The attached patch seems to fix it.

I updated gnus, as Lars said he had applied the patch, but still
referring articles doesn't work for me. I always get empty articles.

I tried referring (`^' in summary buffer) an article and got a blank
one. Then I tried fetching the message by copying the message id in the
References header. I switched to *Group* buffer and did `G w RET google
RET <m-id> RET', which worked as expected.

Bye,
Tassilo
-- 
A morning without coffee is like something without something else.

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

> I tried referring (`^' in summary buffer) an article and got a blank
> one. Then I tried fetching the message by copying the message id in the
> References header. I switched to *Group* buffer and did `G w RET google
> RET <m-id> RET', which worked as expected.

Odd.  I just tried this myself, and it worked fine.

I'm using

(setq gnus-refer-article-method
      '((nnweb "refer" (nnweb-type google))))

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> I tried referring (`^' in summary buffer) an article and got a blank
>> one. Then I tried fetching the message by copying the message id in the
>> References header. I switched to *Group* buffer and did `G w RET google
>> RET <m-id> RET', which worked as expected.
>
> Odd.  I just tried this myself, and it worked fine.
>
> I'm using
>
> (setq gnus-refer-article-method
>       '((nnweb "refer" (nnweb-type google))))

Here it's:

(setq gnus-refer-article-method
      '(current
        (nntp "newshost.uni-koblenz.de")
        (nntp "news.gmane.org")
        (nnweb "google"
               (nnweb-type google))))

I did a test again and got these messages in *Messages* after
`gnus-summary-refer-parent-article' (^) on an article:

,----[ *Messages* ]
| Opening nntp server on newshost.uni-koblenz.de...
| Opening TLS connection to `newshost.uni-koblenz.de'...
| Opening TLS connection with `gnutls-cli -p %p %h'...done
| Opening TLS connection to `newshost.uni-koblenz.de'...done
| Opening nntp server on newshost.uni-koblenz.de...done
| Opening nntp server on news.gmane.org...done
| Opening nnweb server on google...done
| Loading url-auth...done
| Contacting host: www.google.com:80
| Loading url-cache...done
| Contacting host: groups.google.com:80 [2 times]
| Requested article not found
| Generating summary...done
| Opening nntp server on newshost.uni-koblenz.de...done
| Opening nntp server on news.gmane.org...done
| Fetched article <4449e072$0$11072$9b4e6d93@newsread4.arcor-online.net>
| No such article (may have expired or been canceled)
`----

The result was again an empty article in summary buffer. If I manually
search for this message ID at google, the article is found.

Bye,
Tassilo
-- 
A child of five could understand this! Fetch me a child of five!

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

> The result was again an empty article in summary buffer. If I manually
> search for this message ID at google, the article is found.

What group is this in, and what's the Message-ID of the article you
hit `^' on?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> The result was again an empty article in summary buffer. If I
>> manually search for this message ID at google, the article is found.
>
> What group is this in, and what's the Message-ID of the article you
> hit `^' on?

,----[ Message I hit ^ on ]
| Newsgroups: de.comp.os.unix.apps.kde
| Subject: Re: aktuelle KDE-Updates direkt einspielen (SUSE)?
| Date: Sat, 8 Apr 2006 00:27:44 +0200
| Message-ID: <e16p0r$a3r$01$1@news.t-online.com>
| References: <44291c19@news.ish.de>
| 	<e0b918$4oc$03$4@news.t-online.com>
| 	<44294aab@news.ish.de>
`----

I can get the first and the last message of the references at google.

Regards,
Tassilo
-- 
A morning without coffee is like something without something else.

Re: nnweb fix

[Tassilo Horn]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

> I can get the first and the last message of the references at google.

Short addition: I can get them via Gnus' web groups (G w RET <m-id>
RET), too.

Bye,
Tassilo
-- 
My opinions may have changed, but not the fact that I am right.

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

Tassilo Horn schrob:

> Tassilo Horn <heimdall@uni-koblenz.de> writes:
>
>> I can get the first and the last message of the references at google.
>
> Short addition: I can get them via Gnus' web groups (G w RET <m-id>
> RET), too.

Does that mean, fetching the articles <44291c19@news.ish.de> and
<44294aab@news.ish.de> by hitting ^ works, but fetching
<e0b918$4oc$03$4@news.t-online.com> that way does not?

While I am not able to reproduce it, I noticed that
nnweb-request-article doesn't do any URL-encoding on the
message-id. The attached patch fixes it.

Google doesn't seem to care if ?$ and ?@ are sent unencoded though,
maybe you have a proxy inbetween that cares? I suggest applying the
patch anyway, even if it turns out that it doesn't fix the problem.

regards,
andreas

2006-04-30  Andreas Seltenreich  <uwi7@rz.uni-karlsruhe.de>

	* nnweb.el (nnweb-request-article): Do proper xwfu encoding when
	fetching articles by message-id.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.el.patch --]
[-- Type: text/x-patch, Size: 906 bytes --]

Index: nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.20
diff -c -r7.20 nnweb.el
*** nnweb.el	23 Apr 2006 14:12:44 -0000	7.20
--- nnweb.el	30 Apr 2006 15:32:07 -0000
***************
*** 171,177 ****
  		       (when (string-match "^<\\(.*\\)>$" article)
  			 (setq art (match-string 1 article)))
  		       (when (and fetch art)
! 			 (setq url (format fetch art))
  			 (mm-with-unibyte-current-buffer
  			   (mm-url-insert url))
  			 (if (nnweb-definition 'reference t)
--- 171,178 ----
  		       (when (string-match "^<\\(.*\\)>$" article)
  			 (setq art (match-string 1 article)))
  		       (when (and fetch art)
! 			 (setq url (format fetch
! 					   (mm-url-form-encode-xwfu art)))
  			 (mm-with-unibyte-current-buffer
  			   (mm-url-insert url))
  			 (if (nnweb-definition 'reference t)

Re: nnweb fix

[Tassilo Horn]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

Hi Andreas,

>> Short addition: I can get them via Gnus' web groups (G w RET <m-id>
>> RET), too.
>
> Does that mean, fetching the articles <44291c19@news.ish.de> and
> <44294aab@news.ish.de> by hitting ^ works, but fetching
> <e0b918$4oc$03$4@news.t-online.com> that way does not?

No, fetching any of those with ^ does *not* work, but the two
<.*@news.ish.de> articles can be fetched with `G w' in *Group*.

<e0b918$4oc$03$4@news.t-online.com> is not available at all, even not
directly at googles web interface.

Bye,
Tassilo
-- 
A child of five could understand this! Fetch me a child of five!

Re: nnweb fix

[Andreas Seltenreich]

Tassilo Horn schrob:

> Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:
>
>>> Short addition: I can get them via Gnus' web groups (G w RET <m-id>
>>> RET), too.
>>
>> Does that mean, fetching the articles <44291c19@news.ish.de> and
>> <44294aab@news.ish.de> by hitting ^ works, but fetching
>> <e0b918$4oc$03$4@news.t-online.com> that way does not?
>
> No, fetching any of those with ^ does *not* work, but the two
> <.*@news.ish.de> articles can be fetched with `G w' in *Group*.
>
> <e0b918$4oc$03$4@news.t-online.com> is not available at all, even not
> directly at googles web interface.

While it doesn't work via the standard search, the message is found
when I put the message id sans angle brackets into the message id
field of the "Advanced Groups Search" form:
<http://groups.google.com/advanced_search>. The latter is used by
nnweb.el when referring articles by message id. So I guess only
lexically simple message ids can be searched using G w. However, Using
j or ^ in the Summary Buffer should work for all ids.

regards,
andreas

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

> 	* nnweb.el (nnweb-request-article): Do proper xwfu encoding when
> 	fetching articles by message-id.

Ok; applied.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

> | Message-ID: <e16p0r$a3r$01$1@news.t-online.com>
> | References: <44291c19@news.ish.de>
> | 	<e0b918$4oc$03$4@news.t-online.com>
> | 	<44294aab@news.ish.de>

I went to the message in the Message-ID header, and pushed `^'.  It
fetched the parent, and I could `^' up again to the root.

So I seem to be unable to reproduce this bug.

Is it possible that Google is directing you to a German version of
news.google.com that formats stuff differently?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Is it possible that Google is directing you to a German version of
> news.google.com that formats stuff differently?

Maybe. How do I find out?

Bye,
Tassilo
-- 
98% of us Americans are hard-working people. It's the other 2% that give
us a bad reputation. Then again, we did elect them.

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

>> Is it possible that Google is directing you to a German version of
>> news.google.com that formats stuff differently?
>
> Maybe. How do I find out?

Put a

(debug (current-buffer))

into `nnweb-google-wash-article' in nnweb.el, and see what's in the
buffer in question.

Alternatively, use

http://www.google.com/groups?as_umsgid=<message-id-goes-here>&hl=en&dmode=source

and see what you get.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> Maybe. How do I find out?
>
> Put a
>
> (debug (current-buffer))
>
> into `nnweb-google-wash-article' in nnweb.el, and see what's in the
> buffer in question.

Ok, it's the first command in it. Instead of ^ I pressed RET on a
message id, which workes not for me, too. That resulted in this
*Backtrace* buffer:

Debugger entered: (#<buffer  *nntpd*>)
  nnweb-google-wash-article()
  nnweb-request-article("<44291c19@news.ish.de>" "gmane.emacs.gnus.general" "google" nil)
  gnus-request-article("<44291c19@news.ish.de>" "nntp+news.gmane.org:gmane.emacs.gnus.general")
  gnus-request-head("<44291c19@news.ish.de>" "nntp+news.gmane.org:gmane.emacs.gnus.general")
  gnus-read-header("<44291c19@news.ish.de>")
  gnus-summary-insert-subject("<44291c19@news.ish.de>")
  byte-code("Ä Å\x18‰\x19ƒ-

Re: nnweb fix

[Lars Magne Ingebrigtsen]

Tassilo Horn <heimdall@uni-koblenz.de> writes:

> Ok, it's the first command in it. Instead of ^ I pressed RET on a
> message id, which workes not for me, too. That resulted in this
> *Backtrace* buffer:
>
> Debugger entered: (#<buffer  *nntpd*>)

Then look in the " *nntpd*" buffer (note leading space) and see what's
there.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnweb fix

[Reiner Steib]

On Sun, Feb 12 2006, Andreas Seltenreich wrote:

> I also reverted the change to declare nnweb-group-alist with defvar
> instead of defvoo. I didn't realise that it is held on disk
> /multiplexed/ with the defvoo'd nnweb-directory, which would result in
> bogus active file entries if one would use more than one directory for
> nnweb.
>
> Also, nnlistserv seems to inherit from nnweb via nnoo, and I'm not
> sure what the implications are, if it wasn't declared with defvoo
> (anyone)?

I don't know.

> 2006-02-12  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>
>
>         * nnweb.el (nnweb-google-wash-article): Update regexps.
>         (nnweb-group-alist): Use defvoo instead of defvar,

Thanks.  Applied to the trunk.  I'll update v5-10 once the Gmane
search is updated, too.  ;-)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Hi,

it looks like the regexps in nnweb-google-wash-article broke in the
meantime. Attached is an update.

I also reverted the change to declare nnweb-group-alist with defvar
instead of defvoo. I didn't realise that it is held on disk
/multiplexed/ with the defvoo'd nnweb-directory, which would result in
bogus active file entries if one would use more than one directory for
nnweb.

Also, nnlistserv seems to inherit from nnweb via nnoo, and I'm not
sure what the implications are, if it wasn't declared with defvoo
(anyone)?

regards,
Andreas

2006-02-12  Andreas Seltenreich  <uwi7@stud.uni-karlsruhe.de>

        * nnweb.el (nnweb-google-wash-article): Update regexps.
        (nnweb-group-alist): Use defvoo instead of defvar,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 1593 bytes --]

Index: nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.14
diff -u -r7.14 nnweb.el
--- nnweb.el	3 Feb 2006 13:15:13 -0000	7.14
+++ nnweb.el	12 Feb 2006 05:55:27 -0000
@@ -99,7 +99,7 @@
 
 (defvoo nnweb-articles nil)
 (defvoo nnweb-buffer nil)
-(defvar nnweb-group-alist nil)
+(defvoo nnweb-group-alist nil)
 (defvoo nnweb-group nil)
 (defvoo nnweb-hashtb nil)
 
@@ -309,22 +309,26 @@
 
 (defun nnweb-google-wash-article ()
   ;; We have Google's masked e-mail addresses here.  :-/
-  (let ((case-fold-search t))
+  (let ((case-fold-search t)
+	(start-re "<pre>\n *")
+	(end-re "\n *</pre>"))
     (goto-char (point-min))
     (if (save-excursion
 	  (or (re-search-forward "The requested message.*could not be found."
 				 nil t)
-	      (not (and (re-search-forward "^<pre>" nil t)
-			(re-search-forward "^</pre>" nil t)))))
+	      (not (and (re-search-forward start-re nil t)
+			(re-search-forward end-re nil t)))))
 	;; FIXME: Don't know how to indicate "not found".
 	;; Should this function throw an error?  --rsteib
 	(progn
 	  (gnus-message 3 "Requested article not found")
 	  (erase-buffer))
       (delete-region (point-min)
-		     (1+ (re-search-forward "^<pre>" nil t)))
+		     (re-search-forward start-re))
       (goto-char (point-min))
-      (delete-region (- (re-search-forward "^</pre>" nil t) (length "</pre>"))
+      (delete-region (progn
+		       (re-search-forward end-re)
+		       (match-beginning 0))
 		     (point-max))
       (mm-url-decode-entities))))
 

Re: nnweb fix

[Reiner Steib]

On Tue, Jan 31 2006, Andreas Seltenreich wrote:

> Heh, I guess hunting bugs I just introduced has a higher priority.
> E.g., those exponentially growing overview files :-/
[...]
>    (unless (or nnweb-ephemeral-p
> -	      dont-check)
> +	      dont-check
> +	      nnweb-articles)

Thanks.  Committed.

(BTW, your entry has been added to the copyright list in the
meantime.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Reiner Steib writes:

> Fine, thanks.  Committed to trunk and v5-10.
>
> Would you like to work on Gmane search <http://search.gmane.org/>,
> too?  ;-)

Heh, I guess hunting bugs I just introduced has a higher priority.
E.g., those exponentially growing overview files :-/


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 580 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.13
diff -u -r7.13 nnweb.el
--- lisp/nnweb.el	31 Jan 2006 15:51:27 -0000	7.13
+++ lisp/nnweb.el	31 Jan 2006 16:37:03 -0000
@@ -133,7 +133,8 @@
 (deffoo nnweb-request-group (group &optional server dont-check)
   (nnweb-possibly-change-server group server)
   (unless (or nnweb-ephemeral-p
-	      dont-check)
+	      dont-check
+	      nnweb-articles)
     (nnweb-read-overview group))
   (cond
    ((not nnweb-articles)

[-- Attachment #3: Type: text/plain, Size: 38 bytes --]


Sorry for the inconvenience,
Andreas

Re: nnweb fix

[Reiner Steib]

On Tue, Jan 31 2006, Andreas Seltenreich wrote:

> Reiner Steib writes:
>> On Fri, Jan 27 2006, Andreas Seltenreich wrote:
>>> I guess the solution is to adapt nnweb to use the server variables
>>> instead of those custom args to nnweb-request-create-group for solid
>>> groups?
>>
>> Could you suggest a patch, please?
>
> How about the one attached?

Fine, thanks.  Committed to trunk and v5-10.

Would you like to work on Gmane search <http://search.gmane.org/>,
too?  ;-) We could ask Lars and Olly Betts (he did much work on the
Xapian-powered Gmane search) on gmane.discuss to generate more
nnweb-friendly HTML output ("&output=nnweb"; maybe no frames?) if that
would make the task easier.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]

Reiner Steib writes:

> On Fri, Jan 27 2006, Andreas Seltenreich wrote:
>> While it works for ephemeral groups, solid ones seem still broken.
>>
>> ISTM this broke solid groups back in 2003:
>> <http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>
>>
>> I guess the solution is to adapt nnweb to use the server variables
>> instead of those custom args to nnweb-request-create-group for solid
>> groups?
>
> Could you suggest a patch, please?

How about the one attached?

2006-01-31  Andreas Seltenreich <uwi7@stud.uni-karlsruhe.de>

	* nnweb.el (nnweb-group-alist): Use defvar instead of defvoo, there's
	only one active file for all servers.
	(nnweb-request-scan): Make sure nnweb-articles is initialized on solid
	groups -- Gnus might have used a FAST request to select the group.
	(nnweb-request-group, nnweb-google-parse-1): nnweb-type and
	nnweb-search are no longer redundantly kept in the active file.
	(nnweb-request-list): Don't list bogus groups.  There can only be one.
	(nnweb-request-create-group): ARGS is no longer used.
	(nnweb-possibly-change-server, nnweb-request-group): Some
	initialisation removed -- let nnoo do the work.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5947 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.12
diff -c -r7.12 nnweb.el
*** lisp/nnweb.el	30 Jan 2006 14:55:15 -0000	7.12
--- lisp/nnweb.el	31 Jan 2006 05:43:57 -0000
***************
*** 1,7 ****
  ;;; nnweb.el --- retrieving articles via web search engines
  
  ;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
! ;;   2004, 2005 Free Software Foundation, Inc.
  
  ;; Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
  ;; Keywords: news
--- 1,7 ----
  ;;; nnweb.el --- retrieving articles via web search engines
  
  ;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
! ;;   2004, 2005, 2006 Free Software Foundation, Inc.
  
  ;; Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
  ;; Keywords: news
***************
*** 30,39 ****
  ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to Gmane
  ;; web groups (`gnus-group-make-web-group') doesn't work anymore.
  
- ;; FIXME: Solid web groups are currently broken because ARGS are no longer
- ;; passed from `gnus-group-make-web-group' to `nnweb-request-create-group'.
- ;; See revision 6.96 of `gnus-group.el' (2003-01-06).
- 
  ;;; Code:
  
  (eval-when-compile (require 'cl))
--- 30,35 ----
***************
*** 103,109 ****
  
  (defvoo nnweb-articles nil)
  (defvoo nnweb-buffer nil)
! (defvoo nnweb-group-alist nil)
  (defvoo nnweb-group nil)
  (defvoo nnweb-hashtb nil)
  
--- 99,105 ----
  
  (defvoo nnweb-articles nil)
  (defvoo nnweb-buffer nil)
! (defvar nnweb-group-alist nil)
  (defvoo nnweb-group nil)
  (defvoo nnweb-hashtb nil)
  
***************
*** 126,150 ****
  (deffoo nnweb-request-scan (&optional group server)
    (nnweb-possibly-change-server group server)
    (if nnweb-ephemeral-p
!       (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (funcall (nnweb-definition 'map))
    (unless nnweb-ephemeral-p
      (nnweb-write-active)
      (nnweb-write-overview group)))
  
  (deffoo nnweb-request-group (group &optional server dont-check)
!   (nnweb-possibly-change-server nil server)
!   (when (and group
! 	     (not (equal group nnweb-group))
! 	     (not nnweb-ephemeral-p))
!     (setq nnweb-group group
! 	  nnweb-articles nil)
!     (let ((info (assoc group nnweb-group-alist)))
!       (when info
! 	(setq nnweb-type (nth 2 info))
! 	(setq nnweb-search (nth 3 info))
! 	(unless dont-check
! 	  (nnweb-read-overview group)))))
    (cond
     ((not nnweb-articles)
      (nnheader-report 'nnweb "No matching articles"))
--- 122,140 ----
  (deffoo nnweb-request-scan (&optional group server)
    (nnweb-possibly-change-server group server)
    (if nnweb-ephemeral-p
!       (setq nnweb-hashtb (gnus-make-hashtable 4095))
!     (unless nnweb-articles
!       (nnweb-read-overview group)))
    (funcall (nnweb-definition 'map))
    (unless nnweb-ephemeral-p
      (nnweb-write-active)
      (nnweb-write-overview group)))
  
  (deffoo nnweb-request-group (group &optional server dont-check)
!   (nnweb-possibly-change-server group server)
!   (unless (or nnweb-ephemeral-p
! 	      dont-check)
!     (nnweb-read-overview group))
    (cond
     ((not nnweb-articles)
      (nnheader-report 'nnweb "No matching articles"))
***************
*** 208,214 ****
    (nnweb-possibly-change-server nil server)
    (save-excursion
      (set-buffer nntp-server-buffer)
!     (nnmail-generate-active nnweb-group-alist)
      t))
  
  (deffoo nnweb-request-update-info (group info &optional server)
--- 198,204 ----
    (nnweb-possibly-change-server nil server)
    (save-excursion
      (set-buffer nntp-server-buffer)
!     (nnmail-generate-active (list (assoc server nnweb-group-alist)))
      t))
  
  (deffoo nnweb-request-update-info (group info &optional server)
***************
*** 220,226 ****
  (deffoo nnweb-request-create-group (group &optional server args)
    (nnweb-possibly-change-server nil server)
    (nnweb-request-delete-group group)
!   (push `(,group ,(cons 1 0) ,@args) nnweb-group-alist)
    (nnweb-write-active)
    t)
  
--- 210,216 ----
  (deffoo nnweb-request-create-group (group &optional server args)
    (nnweb-possibly-change-server nil server)
    (nnweb-request-delete-group group)
!   (push `(,group ,(cons 1 0)) nnweb-group-alist)
    (nnweb-write-active)
    t)
  
***************
*** 290,307 ****
      def))
  
  (defun nnweb-possibly-change-server (&optional group server)
-   (nnweb-init server)
    (when server
      (unless (nnweb-server-opened server)
!       (nnweb-open-server server)))
    (unless nnweb-group-alist
      (nnweb-read-active))
    (unless nnweb-hashtb
      (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (when group
!     (when (and (not nnweb-ephemeral-p)
! 	       (equal group nnweb-group))
!       (nnweb-request-group group nil t))))
  
  (defun nnweb-init (server)
    "Initialize buffers and such."
--- 280,295 ----
      def))
  
  (defun nnweb-possibly-change-server (&optional group server)
    (when server
      (unless (nnweb-server-opened server)
!       (nnweb-open-server server))
!     (nnweb-init server))
    (unless nnweb-group-alist
      (nnweb-read-active))
    (unless nnweb-hashtb
      (setq nnweb-hashtb (gnus-make-hashtable 4095)))
    (when group
!     (setq nnweb-group group)))
  
  (defun nnweb-init (server)
    "Initialize buffers and such."
***************
*** 347,354 ****
  	Subject Score Date Newsgroups From
  	map url mid)
      (unless active
!       (push (list nnweb-group (setq active (cons 1 0))
! 		  nnweb-type nnweb-search)
  	    nnweb-group-alist))
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
--- 335,341 ----
  	Subject Score Date Newsgroups From
  	map url mid)
      (unless active
!       (push (list nnweb-group (setq active (cons 1 0)))
  	    nnweb-group-alist))
      ;; Go through all the article hits on this page.
      (goto-char (point-min))

Re: nnweb fix

[Reiner Steib]

On Fri, Jan 27 2006, Andreas Seltenreich wrote:

> Reiner Steib writes:
>> On Sat, Dec 31 2005, Andreas Seltenreich wrote:
>>> Andreas Seltenreich writes:
>>>> attached is an attempt to get web groups working again with the new
>>>> google groups.
[...]
>> Your patch works fine for me.  Thanks.
>
> While it works for ephemeral groups, solid ones seem still broken.
>
> ISTM this broke solid groups back in 2003:
> <http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>
>
> I guess the solution is to adapt nnweb to use the server variables
> instead of those custom args to nnweb-request-create-group for solid
> groups?

Could you suggest a patch, please?

>> required request form.  After the assignment process completes (you
>> get sent forms which you have to read and sign and send back), we can
>> install your patch.
>
> According to a mail from the FSF, the process is now complete.

Thanks, I have installed your patch in the trunk and in the stable
branch (v5-10).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

Reiner Steib writes:

> On Sat, Dec 31 2005, Andreas Seltenreich wrote:
>
>> Andreas Seltenreich writes:
>>
>>> attached is an attempt to get web groups working again with the new
>>> google groups.
>>
>> Oops, I missed some html entity decoding and a debug statement.
>> Attached is an updated patch.
>
> Your patch works fine for me.  Thanks.

While it works for ephemeral groups, solid ones seem still broken.

ISTM this broke solid groups back in 2003:
<http://quimby.gnus.org/cgi-bin/cvsweb.cgi/gnus/lisp/gnus-group.el.diff?r1=6.95&r2=6.96>

I guess the solution is to adapt nnweb to use the server variables
instead of those custom args to nnweb-request-create-group for solid
groups?

> required request form.  After the assignment process completes (you
> get sent forms which you have to read and sign and send back), we can
> install your patch.

According to a mail from the FSF, the process is now complete.

regards,
Andreas

Re: nnweb fix

[Andreas Seltenreich]

Reiner Steib writes:

> Have you filed a copyright assignment for work on Gnus or Emacs to the
> Free Software Foundation?

I'm afraid not.

> If not, I can ask someone to send you the required request form.
> After the assignment process completes (you get sent forms which you
> have to read and sign and send back), we can install your patch.

Agreed.

regards,
Andreas

Re: nnweb fix

[Reiner Steib]

On Sat, Dec 31 2005, Andreas Seltenreich wrote:

> Andreas Seltenreich writes:
>
>> attached is an attempt to get web groups working again with the new
>> google groups.
>
> Oops, I missed some html entity decoding and a debug statement.
> Attached is an updated patch.

Your patch works fine for me.  Thanks.

However, your contribution larger than 10-15 lines (i.e. it's not a
"tiny change") so we need legal papers to install it.  Gnus is a part
of Emacs and since Emacs is a core GNU project, the Free Software
Foundation (FSF) requires an assignment of copyright for
contributions, so that it is able to defend Emacs' legal status
without doubts, should the need arise.  You can either sign papers
only for Gnus or for Emacs (which covers Gnus, too).

Have you filed a copyright assignment for work on Gnus or Emacs to the
Free Software Foundation?  If not, I can ask someone to send you the
required request form.  After the assignment process completes (you
get sent forms which you have to read and sign and send back), we can
install your patch.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

Andreas Seltenreich writes:

> attached is an attempt to get web groups working again with the new
> google groups.

Oops, I missed some html entity decoding and a debug statement.
Attached is an updated patch.

regards,
Andreas

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5069 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.10
diff -c -r7.10 nnweb.el
*** lisp/nnweb.el	7 Sep 2005 13:50:08 -0000	7.10
--- lisp/nnweb.el	31 Dec 2005 10:47:14 -0000
***************
*** 27,37 ****
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Google Groups and Gmane, stuff
! ;; related to web groups (gnus-group-make-web-group) doesn't work anymore.
! 
! ;; Fetching an article by MID (cf. gnus-refer-article-method) over Google
! ;; Groups should work.
  
  ;;; Code:
  
--- 27,34 ----
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to
! ;; web groups doesn't work anymore.
  
  ;;; Code:
  
***************
*** 61,66 ****
--- 58,64 ----
  (defvar nnweb-type-definition
    '((google
       (id . "http://www.google.com/groups?as_umsgid=%s&hl=en&dmode=source")
+      (result . "http://groups.google.com/group/%s/msg/%s?dmode=source")
       (article . nnweb-google-wash-article)
       (reference . identity)
       (map . nnweb-google-create-mapping)
***************
*** 337,342 ****
--- 335,341 ----
        (mm-url-decode-entities))))
  
  (defun nnweb-google-parse-1 (&optional Message-ID)
+   "Parse search result in current buffer."
    (let ((i 0)
  	(case-fold-search t)
  	(active (cadr (assoc nnweb-group nnweb-group-alist)))
***************
*** 349,358 ****
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a href=/groups\\(\\?[^ \">]*selm=\\([^ &\">]+\\)\\)" nil t)
!       (setq mid (match-string 2)
  	    url (format
! 		 (nnweb-definition 'id) mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
--- 348,360 ----
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a +href=\"/group/\\([^>\"]+\\)/browse_thread/[^>]+#\\([0-9a-f]+\\)" nil t)
!       (setq Newsgroups (match-string-no-properties 1)
! 	    ;; Note: with groups-ng, mid is no longer a common
! 	    ;; Message-ID, but some internal id.
! 	    mid (match-string-no-properties 2)
  	    url (format
! 		 (nnweb-definition 'result) Newsgroups mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
***************
*** 360,384 ****
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (forward-line 2)
!       (when (looking-at "<br><font[^>]+>")
! 	(goto-char (match-end 0)))
!       (if (not (looking-at "<a[^>]+>"))
! 	  (skip-chars-forward " \t")
! 	(narrow-to-region (point)
! 			  (search-forward "</a>" nil t))
! 	(mm-url-remove-markup)
! 	(mm-url-decode-entities)
! 	(setq Newsgroups (buffer-string))
! 	(goto-char (point-max))
! 	(widen)
! 	(skip-chars-forward "- \t"))
        (when (looking-at
! 	     "\\([0-9]+\\)[/ ]\\([A-Za-z]+\\)[/ ]\\([0-9]+\\)[ \t]*by[ \t]*\\([^<]*\\) - <a")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 2) (match-string 1)
! 			   (match-string 3))))
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
--- 362,383 ----
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (narrow-to-region (point)
! 			(search-forward "</td" nil t))
! 
!       (mm-url-remove-markup)
!       (mm-url-decode-entities)
!       (search-backward " - ")
        (when (looking-at
! 	     " - \\([a-zA-Z]+\\) \\([0-9]+\\)\\(?: \\([0-9]\\{4\\}\\)\\)?, [^\n]+by \\([^<\n]+\\)\n")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 1)
! 			   (match-string 2)
! 			   (or (match-string 3)
! 			       (substring (current-time-string) -4)))))
! 
!       (widen)
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
***************
*** 419,425 ****
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td nowrap><a href=\\([^>]+\\).*<span class=b>Next</span>" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
--- 418,424 ----
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td><a href=\"\n\\([^>\"]+\\)\"><img src=\"/img/nav_next" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
***************
*** 443,449 ****
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")))))
    t)
  
  (defun nnweb-google-identity (url)
--- 442,449 ----
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")
!        ("filter" . "0")))))
    t)
  
  (defun nnweb-google-identity (url)

nnweb fix

[Andreas Seltenreich]

[-- Attachment #1: Type: text/plain, Size: 107 bytes --]

Hi,

attached is an attempt to get web groups working again with the new
google groups.

regards,
Andreas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnweb.patch --]
[-- Type: text/x-patch, Size: 5062 bytes --]

Index: lisp/nnweb.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnweb.el,v
retrieving revision 7.10
diff -c -r7.10 nnweb.el
*** lisp/nnweb.el	7 Sep 2005 13:50:08 -0000	7.10
--- lisp/nnweb.el	31 Dec 2005 09:32:34 -0000
***************
*** 27,37 ****
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Google Groups and Gmane, stuff
! ;; related to web groups (gnus-group-make-web-group) doesn't work anymore.
! 
! ;; Fetching an article by MID (cf. gnus-refer-article-method) over Google
! ;; Groups should work.
  
  ;;; Code:
  
--- 27,34 ----
  
  ;; Note: You need to have `w3' installed for some functions to work.
  
! ;; FIXME: Due to changes in the HTML output of Gmane, stuff related to
! ;; web groups doesn't work anymore.
  
  ;;; Code:
  
***************
*** 61,66 ****
--- 58,64 ----
  (defvar nnweb-type-definition
    '((google
       (id . "http://www.google.com/groups?as_umsgid=%s&hl=en&dmode=source")
+      (result . "http://groups.google.com/group/%s/msg/%s?dmode=source")
       (article . nnweb-google-wash-article)
       (reference . identity)
       (map . nnweb-google-create-mapping)
***************
*** 337,342 ****
--- 335,341 ----
        (mm-url-decode-entities))))
  
  (defun nnweb-google-parse-1 (&optional Message-ID)
+   "Parse search result in current buffer."
    (let ((i 0)
  	(case-fold-search t)
  	(active (cadr (assoc nnweb-group nnweb-group-alist)))
***************
*** 349,358 ****
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a href=/groups\\(\\?[^ \">]*selm=\\([^ &\">]+\\)\\)" nil t)
!       (setq mid (match-string 2)
  	    url (format
! 		 (nnweb-definition 'id) mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
--- 348,360 ----
      ;; Go through all the article hits on this page.
      (goto-char (point-min))
      (while (re-search-forward
! 	    "a +href=\"/group/\\([^>\"]+\\)/browse_thread/[^>]+#\\([0-9a-f]+\\)" nil t)
!       (setq Newsgroups (match-string-no-properties 1)
! 	    ;; Note: with groups-ng, mid is no longer a common
! 	    ;; Message-ID, but some internal id.
! 	    mid (match-string-no-properties 2)
  	    url (format
! 		 (nnweb-definition 'result) Newsgroups mid))
        (narrow-to-region (search-forward ">" nil t)
  			(search-forward "</a>" nil t))
        (mm-url-remove-markup)
***************
*** 360,384 ****
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (forward-line 2)
!       (when (looking-at "<br><font[^>]+>")
! 	(goto-char (match-end 0)))
!       (if (not (looking-at "<a[^>]+>"))
! 	  (skip-chars-forward " \t")
! 	(narrow-to-region (point)
! 			  (search-forward "</a>" nil t))
! 	(mm-url-remove-markup)
! 	(mm-url-decode-entities)
! 	(setq Newsgroups (buffer-string))
! 	(goto-char (point-max))
! 	(widen)
! 	(skip-chars-forward "- \t"))
        (when (looking-at
! 	     "\\([0-9]+\\)[/ ]\\([A-Za-z]+\\)[/ ]\\([0-9]+\\)[ \t]*by[ \t]*\\([^<]*\\) - <a")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 2) (match-string 1)
! 			   (match-string 3))))
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
--- 362,384 ----
        (setq Subject (buffer-string))
        (goto-char (point-max))
        (widen)
!       (narrow-to-region (point)
! 			(search-forward "</td" nil t))
! 
!       (search-backward ">- ")
!       (setq From nil)
        (when (looking-at
! 	     ">- \\([a-zA-Z]+\\) \\([0-9]+\\)\\(?: \\([0-9]\\{4\\}\\)\\)?, [^-<]+by \\([^<\n]+\\)\n")
  	(setq From (match-string 4)
  	      Date (format "%s %s 00:00:00 %s"
! 			   (match-string 1)
! 			   (match-string 2)
! 			   (or (match-string 3)
! 			       (substring (current-time-string) -4)))))
! 
!       (unless From (debug))
! 
!       (widen)
        (forward-line 1)
        (incf i)
        (unless (nnweb-get-hashtb url)
***************
*** 419,425 ****
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td nowrap><a href=\\([^>]+\\).*<span class=b>Next</span>" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
--- 419,425 ----
  	    (goto-char (point-min))
  	    (incf i 100)
  	    (if (or (not (re-search-forward
! 			  "<td><a href=\"\n\\([^>\"]+\\)\"><img src=\"/img/nav_next" nil t))
  		    (>= i nnweb-max-hits))
  		(setq more nil)
  	      ;; Yup, there are more articles
***************
*** 443,449 ****
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")))))
    t)
  
  (defun nnweb-google-identity (url)
--- 443,450 ----
         ("hl" . "en")
         ("lr" . "")
         ("safe" . "off")
!        ("sites" . "groups")
!        ("filter" . "0")))))
    t)
  
  (defun nnweb-google-identity (url)