From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/72100 Path: news.gmane.org!not-for-mail From: Steinar Bang Newsgroups: gmane.emacs.gnus.general Subject: Re: Gnus and googlegroups search Date: Tue, 28 Sep 2010 14:01:18 +0200 Organization: Probably a good idea Message-ID: References: <7y39sul8yu.fsf@news.eternal-september.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1285675307 24154 80.91.229.12 (28 Sep 2010 12:01:47 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Tue, 28 Sep 2010 12:01:47 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M20473@lists.math.uh.edu Tue Sep 28 14:01:44 2010 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1P0YsR-0003WX-KD for ding-account@gmane.org; Tue, 28 Sep 2010 14:01:43 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1P0YsJ-0002D7-Gb; Tue, 28 Sep 2010 07:01:35 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1P0YsH-0002Cu-RM for ding@lists.math.uh.edu; Tue, 28 Sep 2010 07:01:33 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.72) (envelope-from ) id 1P0YsE-0002m9-32 for ding@lists.math.uh.edu; Tue, 28 Sep 2010 07:01:33 -0500 Original-Received: from lo.gmane.org ([80.91.229.12]) by quimby.gnus.org with esmtp (Exim 3.36 #1 (Debian)) id 1P0YsD-0005ci-00 for ; Tue, 28 Sep 2010 14:01:29 +0200 Original-Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1P0YsC-0003QO-25 for ding@gnus.org; Tue, 28 Sep 2010 14:01:28 +0200 Original-Received: from 62.113.137.5 ([62.113.137.5]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 28 Sep 2010 14:01:28 +0200 Original-Received: from sb by 62.113.137.5 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 28 Sep 2010 14:01:28 +0200 X-Injected-Via-Gmane: http://gmane.org/ Mail-Followup-To: ding@gnus.org Original-Lines: 76 Original-X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 62.113.137.5 Mail-Copies-To: never User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/22.3 (gnu/linux) Cancel-Lock: sha1:AIpIKxUUhUyS071gO4ZA38q1ccU= X-Spam-Score: -0.7 (/) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:72100 Archived-At: >>>>> Steinar Bang : >>>>> Lars Magne Ingebrigtsen : >> It works by web-scraping the results, I think, which is notoriously >> unreliable. Doesn't Google have a search API? Rewriting nnweb to use >> that instead would probably be a good idea. > There is an AJAX REST API, but I only found documentation showing it > from the JS viewpoint, not documenting the actual HTTP calls made, and > their responses: > http://code.google.com/intl/no-NO/apis/ajaxsearch/documentation/ > Maybe the mappings from the API to what's sent over the wire are obvious > to JS-heads...? Something here: http://blog.programmableweb.com/2008/04/21/googles-rest-api-for-search/ Basically you do a GET HTTP request, and get a response with a JSON body. A request URL looks like this: http://ajax.googleapis.com/ajax/services/search/web?v=3D1.0&q=3DParis%20H= ilton You also need to set the referer header. To what it should be set from a client application, I don't know. I also don't know if it needs to DNS match the machine the calling application is done on. Perhaps it would be neccessary to proxy the queries through eg. http://www.gnus.org or something? Hm... let me try... $ curl -e http://www.gnus.org "http://ajax.googleapis.com/ajax/services/se= arch/web?v=3D1.0&q=3DParis%20Hilton" {"responseData": {"results":[{"GsearchResultClass":"GwebSearch","unescapedU= rl":"http://en.wikipedia.org/wiki/Paris_Hilton","url":"http://en.wikipedia.= org/wiki/Paris_Hilton","visibleUrl":"en.wikipedia.org","cacheUrl":"http://w= ww.google.com/search?q\u003dcache:TwrPfhd22hYJ:en.wikipedia.org","title":"\= u003cb\u003eParis Hilton\u003c/b\u003e - Wikipedia, the free encyclopedia",= "titleNoFormatting":"Paris Hilton - Wikipedia, the free encyclopedia","cont= ent":"\u003cb\u003eParis\u003c/b\u003e Whitney \u003cb\u003eHilton\u003c/b\= u003e (born February 17, 1981) is an American socialite, heiress, media p= ersonality, model, singer, author, fashion designer and actress . \u003cb\u= 003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"= http://www.tmz.com/person/paris-hilton/","url":"http://www.tmz.com/person/p= aris-hilton/","visibleUrl":"www.tmz.com","cacheUrl":"http://www.google.com/= search?q\u003dcache:t-eZlX9vZUYJ:www.tmz.com","title":"\u003cb\u003eParis H= ilton\u003c/b\u003e | TMZ.com","titleNoFormatting":"Paris Hilton| TMZ.com",= "content":"After Japan turned away \u003cb\u003eParis Hilton\u003c/b\u003e = -- because of her guilty plea to cocaine possession -- the heiress highta= iled it back to the U.S. of A ... arriving \u003cb\u003e...\u003c/b\u003e"}= ,{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.parishiltonz= one.com/","url":"http://www.parishiltonzone.com/","visibleUrl":"www.parishi= ltonzone.com","cacheUrl":"http://www.google.com/search?q\u003dcache:EXq0MGR= NiNoJ:www.parishiltonzone.com","title":"\u003cb\u003eParis Hilton\u003c/b\u= 003e Zone | \u003cb\u003eParis Hilton\u003c/b\u003e Pictures, Pics, Photos"= ,"titleNoFormatting":"Paris Hilton Zone | Paris Hilton Pictures, Pics, Phot= os","content":"8000+ new \u003cb\u003eParis Hilton\u003c/b\u003e pictures, = Paris wallpaper, lyrics, audio, video,daily Paris pics \u0026amp; news."}= ,{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.imdb.com/nam= e/nm0385296/","url":"http://www.imdb.com/name/nm0385296/","visibleUrl":"www= .imdb.com","cacheUrl":"http://www.google.com/search?q\u003dcache:1i34Kkqnso= oJ:www.imdb.com","title":"\u003cb\u003eParis Hilton\u003c/b\u003e","titleNo= Formatting":"Paris Hilton","content":"Socialite \u003cb\u003eParis Hilton\u= 003c/b\u003e was born on February 17, 1981 in New York City... Visit IMDb= for Photos, Filmography, Discussions, Bio, News, Awards, Agent, \u003cb\u0= 03e...\u003c/b\u003e"}],"cursor":{"pages":[{"start":"0","label":1},{"start"= :"4","label":2},{"start":"8","label":3},{"start":"12","label":4},{"start":"= 16","label":5},{"start":"20","label":6},{"start":"24","label":7},{"start":"= 28","label":8}],"estimatedResultCount":"19500000","currentPageIndex":0,"mor= eResultsUrl":"http://www.google.com/search?oe\u003dutf8\u0026ie\u003dutf8\u= 0026source\u003duds\u0026start\u003d0\u0026hl\u003den\u0026q\u003dParis+Hil= ton"}}, "responseDetails": null, "responseStatus": 200} $ Nah... the referer can be faked.