Fetching by MID: wget vs. curl (was: Support fuer das neue groups.google.com)

Fetching by MID: wget vs. curl (was: Support fuer das neue groups.google.com)

[Reiner Steib]

[ Redirecting this from de.comm.software.gnus to ding.
  See http://www.google.de/groups?as_umsgid=m2hdc64h9e.fsf@seki.fr&hl=en
  for the rest of the thread. ]

On Tue, Sep 27 2005, Sébastien Kirche wrote:

[ Summary: Fetching an article by MID doesn't work for Sébastien when
  using curl or the internal url.el ]

> It does not work here, possibly  due to a 302 http redirection (document
> has moved) from the Google server.
>
> I tried to fetch a  post from fr.comp.applications.emacs dated from Wed,
> 21 Apr 2004 01:29:01 +0200
>
> (MID: <plop87isfu1b5u.fsf@gnu-rox.org>) that one can see it is available
> From the google advanced search form.

With wget, it works for me:

(setq
 ;; for testing:
 gnus-refer-article-method
 '((nnweb "google" (nnweb-type google)))
 ;; Use wget instead of url.el (for nnweb-type google), seems to work more
 ;; reliable.
 mm-url-use-external t
 mm-url-program 'wget)

,----
| From: Xavier Maillard <z...@gnu-rox.org>
| Subject: Re: un mode pour maintenir un dictionnaire/index/repertoire ?
| Newsgroups: fr.comp.applications.emacs
| Date: Wed, 21 Apr 2004 01:29:01 +0200
| Message-ID: <plop87isfu1b5u.fsf@gnu-rox.org>
`----

> Here are some of my settings :
> ,----[ C-h v gnus-refer-article-method RET ]
> | gnus-refer-article-method is a variable defined in `gnus'.
> | Its value is 
> | (current
> |  (nntp "news.gmane.org")
> |  (nnweb "google"
> | 		(nnweb-type google)))
> `----
[...]
> ,----[ C-h v mm-url-program RET ]
> | mm-url-program is a variable defined in `mm-url'.
> | Its value is "curl"
> `----
>
> ,----[ C-h v mm-url-arguments RET ]
> | mm-url-arguments is a variable defined in `mm-url'.
> | Its value is ("--silent")
> `----
[...]
> I wonder if it could be a trick from google to prevent external queries ?

On the command line, I also get the redirection when using curl, even
when faking the User-Agent like mm-url does with wget:

$ mid='plop87isfu1b5u.fsf@gnu-rox.org'
$ curl --silent --user-agent mm-url \
  "http://www.google.com/groups?as_umsgid=$mid&hl=en&dmode=source"

With wget, everything is okay:

$ wget --user-agent=mm-url -q -O - \
  "http://www.google.com/groups?as_umsgid=$mid&hl=en&dmode=source"

I don't know what makes the difference between wget and curl.


[ order of quoting changed ]
> Erf.   Gmane    seems   to   be    broken   to   let   me    post   into
> gmane.emacs.gnus.general. I can read but not post. Although it is ok for
> several  other mailing  lists.  And  i have  already  subscribed to  the
> list...

You need a valid email address for Gmane and confirm the query.  I
have successfully posted some messages to ding via news.gmane.org
today (and I'm not even subscribed to ding).  See also the Gmane FAQ.
If all else fails, ask on gmane.discuss.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Fetching by MID: wget vs. curl

[Reiner Steib]

On Fri, Sep 30 2005, Sébastien Kirche wrote:

> I have discovered how to use curl for querying Google.
>
> I  have  spent  some  time  with ethereal  (network  monitor)  to  study
> carefully the dialog between curl/wget and the Google server.
>
> In both cases,  Google answers the request by  a redirection (actually 2
> successive redirections). 
>
> Simply  wget can follow  the redirections  "out of  the box"  by sending
> automatically some further requests.
>
> Curl can too, but you need to use the -L / --location setting.
>
> I  have  also noticed  that  the  user-agent  matters. Sometimes  Google
> returns an unauthorized response. Using "mm-url" is fine.
[...]
> curl --silent --user-agent mm-url --location \
> "http://www.google.com/groups?as_umsgid=$mid&hl=en&dmode=source"
>
> ...works now perfectly. Maybe it could  be worth to tell to others users
> of Gnus ?

I will add the options to `mm-url-predefined-programs'.  I don't have
CVS access in the next few days; maybe someone else can install it
earlier (in v5-10 and trunk):

(defvar mm-url-predefined-programs
  '((wget "wget" "--user-agent=mm-url" "-q" "-O" "-")
    (w3m  "w3m" "-dump_source")
    (lynx "lynx" "-source")
    (curl "curl" "--user-agent mm-url" "--location" "--silent")))

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/