Gnus and googlegroups search

Gnus and googlegroups search

[Richard Riley]

Is there any easy way or plugin for gnus that would enable one to create
a temporary search result group from google-groups? It would be nice to
create a bunch of posts to usenet from a google groups search and be
able to process them etc in Gnus.

Re: Gnus and googlegroups search

[CHENG Gao]

*On Tue, 28 Sep 2010 07:42:49 +0200
* Also sprach Richard Riley <rileyrg@googlemail.com>:

> Is there any easy way or plugin for gnus that would enable one to create
> a temporary search result group from google-groups? It would be nice to
> create a bunch of posts to usenet from a google groups search and be
> able to process them etc in Gnus.

Seems nnweb is what you need - search messages from google or gmane groups.

Re: Gnus and googlegroups search

[Lars Magne Ingebrigtsen]

CHENG Gao <chenggao@cyberhut.org> writes:

> Seems nnweb is what you need - search messages from google or gmane groups.

I tried it a few weeks back -- the Google search no longer worked.

It works by web-scraping the results, I think, which is notoriously
unreliable.  Doesn't Google have a search API?  Rewriting nnweb to use
that instead would probably be a good idea.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> It works by web-scraping the results, I think, which is notoriously
> unreliable.  Doesn't Google have a search API?  Rewriting nnweb to use
> that instead would probably be a good idea.

There is an AJAX REST API, but I only found documentation showing it
from the JS viewpoint, not documenting the actual HTTP calls made, and
their responses:
	http://code.google.com/intl/no-NO/apis/ajaxsearch/documentation/

Maybe the mappings from the API to what's sent over the wire are obvious
to JS-heads...?

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

>> It works by web-scraping the results, I think, which is notoriously
>> unreliable.  Doesn't Google have a search API?  Rewriting nnweb to use
>> that instead would probably be a good idea.

> There is an AJAX REST API, but I only found documentation showing it
> from the JS viewpoint, not documenting the actual HTTP calls made, and
> their responses:
> 	http://code.google.com/intl/no-NO/apis/ajaxsearch/documentation/

> Maybe the mappings from the API to what's sent over the wire are obvious
> to JS-heads...?

Something here:
  http://blog.programmableweb.com/2008/04/21/googles-rest-api-for-search/

Basically you do a GET HTTP request, and get a response with a JSON
body.

A request URL looks like this:
  http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton

You also need to set the referer header.  To what it should be set from
a client application, I don't know.  I also don't know if it needs to
DNS match the machine the calling application is done on.

Perhaps it would be neccessary to proxy the queries through
eg. http://www.gnus.org or something?

Hm... let me try...
 $ curl -e http://www.gnus.org "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton"
{"responseData": {"results":[{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://en.wikipedia.org/wiki/Paris_Hilton","url":"http://en.wikipedia.org/wiki/Paris_Hilton","visibleUrl":"en.wikipedia.org","cacheUrl":"http://www.google.com/search?q\u003dcache:TwrPfhd22hYJ:en.wikipedia.org","title":"\u003cb\u003eParis Hilton\u003c/b\u003e - Wikipedia, the free encyclopedia","titleNoFormatting":"Paris Hilton - Wikipedia, the free encyclopedia","content":"\u003cb\u003eParis\u003c/b\u003e Whitney \u003cb\u003eHilton\u003c/b\u003e (born February 17, 1981) is an American socialite, heiress,   media personality, model, singer, author, fashion designer and actress . \u003cb\u003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.tmz.com/person/paris-hilton/","url":"http://www.tmz.com/person/paris-hilton/","visibleUrl":"www.tmz.com","cacheUrl":"http://www.google.com/search?q\u003dcache:t-eZlX9vZUYJ:www.tmz.com","title":"\u003cb\u003eParis Hilton\u003c/b\u003e | TMZ.com","titleNoFormatting":"Paris Hilton| TMZ.com","content":"After Japan turned away \u003cb\u003eParis Hilton\u003c/b\u003e -- because of her guilty plea to cocaine   possession -- the heiress hightailed it back to the U.S. of A ... arriving \u003cb\u003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.parishiltonzone.com/","url":"http://www.parishiltonzone.com/","visibleUrl":"www.parishiltonzone.com","cacheUrl":"http://www.google.com/search?q\u003dcache:EXq0MGRNiNoJ:www.parishiltonzone.com","title":"\u003cb\u003eParis Hilton\u003c/b\u003e Zone | \u003cb\u003eParis Hilton\u003c/b\u003e Pictures, Pics, Photos","titleNoFormatting":"Paris Hilton Zone | Paris Hilton Pictures, Pics, Photos","content":"8000+ new \u003cb\u003eParis Hilton\u003c/b\u003e pictures, Paris wallpaper, lyrics, audio, video,daily   Paris pics \u0026amp; news."},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.imdb.com/name/nm0385296/","url":"http://www.imdb.com/name/nm0385296/","visibleUrl":"www.imdb.com","cacheUrl":"http://www.google.com/search?q\u003dcache:1i34KkqnsooJ:www.imdb.com","title":"\u003cb\u003eParis Hilton\u003c/b\u003e","titleNoFormatting":"Paris Hilton","content":"Socialite \u003cb\u003eParis Hilton\u003c/b\u003e was born on February 17, 1981 in New York City... Visit   IMDb for Photos, Filmography, Discussions, Bio, News, Awards, Agent, \u003cb\u003e...\u003c/b\u003e"}],"cursor":{"pages":[{"start":"0","label":1},{"start":"4","label":2},{"start":"8","label":3},{"start":"12","label":4},{"start":"16","label":5},{"start":"20","label":6},{"start":"24","label":7},{"start":"28","label":8}],"estimatedResultCount":"19500000","currentPageIndex":0,"moreResultsUrl":"http://www.google.com/search?oe\u003dutf8\u0026ie\u003dutf8\u0026source\u003duds\u0026start\u003d0\u0026hl\u003den\u0026q\u003dParis+Hilton"}}, "responseDetails": null, "responseStatus": 200} $

Nah... the referer can be faked.

Re: Gnus and googlegroups search

[Steinar Bang]

Emacs lisp JSON parser:
 http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs

(hm... is it actually _in_ GNU Emacs I wonder?  And from what version?)

Linked to from here (parsers for different languages at the bottom):
 http://www.json.org/

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Emacs lisp JSON parser:
>  http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs

> (hm... is it actually _in_ GNU Emacs I wonder?  And from what version?)

Not in emacs 22.3, at least, according to locate-library.

Re: Gnus and googlegroups search

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

On Tue, Sep 28 2010, Steinar Bang wrote:

> (hm... is it actually _in_ GNU Emacs I wonder?  And from what
> version?)

It's in Emacs 23.2 at least. I already use it for the Google Maps
extension.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>> Emacs lisp JSON parser:
>> http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs

>> (hm... is it actually _in_ GNU Emacs I wonder?  And from what version?)

> Not in emacs 22.3, at least, according to locate-library.

Says here (search for "json.el"), "Included with Emacs as of 2008-02-21":
 http://edward.oconnor.cx/elisp/

The above URL also links on to "Introducing json.el":
 http://edward.oconnor.cx/2006/03/json.el

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> On Tue, Sep 28 2010, Steinar Bang wrote:
>> (hm... is it actually _in_ GNU Emacs I wonder?  And from what
>> version?)

> It's in Emacs 23.2 at least. I already use it for the Google Maps
> extension.

And Emacs 22 users can get version 1.2 of json.el easily from ELPA:
 http://tromey.com/elpa/

(git gnus won't run on emacs 21 anymore, so there is no point in
supporting that.  Personally I will be stuck with emacs 22 on at least
my UNE (Ubuntu Netbook Edition) netbook, until the issues with opening
emacs 23 are resolved...)

Re: Gnus and googlegroups search

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Basically you do a GET HTTP request, and get a response with a JSON
> body.

Looks good.  Now somebody just has to implement the nnweb thing, since
json.el is in Emacs already...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus and googlegroups search

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Basically you do a GET HTTP request, and get a response with a JSON
>> body.

> Looks good.  Now somebody just has to implement the nnweb thing, since
> json.el is in Emacs already...

Please make sure it still builds on emacs22 (which doesn't have it
included).  It is easily retrieved using ELPA if one _does_ need it, but
it should build without, I think.

(The netbook I'm writing this on needs to use emacs22 (Ubuntu Netbook
Edition), because when I try to open emacs23 it looks very weird.  It's
the wrong size, has an unreadable font, and can't get keyboard focus)