[Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Jun Furuse]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

Hi list,

I have launched a new OCaml API search, OCaml◎Scope at
http://ocamloscope.herokuapp.com .

OCaml◎Scope is a type directed library search, derived work from OCaml API
Search by Mizuno and its ancestor OCamlBrowser by Garrigue. It is also
inspired from Hoogle, the same API search engine for Haskell by MItchell,
which I regularly use in my Haskell :-) job.

* Fast and Portable. It loads everything in memory, unlike OCaml API Search
and OCamlBrowser which load compiled interface files (*.cmi) dynamically.
The data file is extracted from compiled files but self-contained, so the
search engine does not require to compile the libraries locally.
* No use of unification but edit distance of types like Hoogle. Unification
does not provide good results in type directed search, and is costy.
* OCamlFind and OPAM friendly. OCaml◎Scope knows which items are from which
OCamlFind and OPAM packages.
* OCamlDoc: it also extracts OCamlDoc comments, if possible.
* Small: it can even run as a heroku app. Currently it carries 245k entries
from 76 OCamlFind packages including Core and Batteries, but the data file
(as a marshalled OCaml value) is still 20Mb.

There are lots of todos but I think the search results look well sane so
far. If you find something strange please drop by
https://bitbucket.org/camlspotter/ocamloscope-server/issues?status=new&status=open
and
leave some comments. Thanks!

Jun Furuse

[-- Attachment #2: Type: text/html, Size: 1790 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]

Neat!

I played with it for a bit, and the main issue it saw with it is the
trickiness around type names for things like string.  For example, to find
core or batteries' string split function, you need to search for

t -> char -> t list

Rather than

string -> char -> string list

Which finds ocplib's equivalent, and the Core.Std.String.Escaping version
as well.

Fill on unification doesn't seem ideal, but I wonder if some ability to
recognize equivalent type names can be done without full-on search time
unification.
On Jul 5, 2013 5:12 AM, "Jun Furuse" <jun.furuse@gmail.com> wrote:

> Hi list,
>
> I have launched a new OCaml API search, OCaml◎Scope at
> http://ocamloscope.herokuapp.com .
>
> OCaml◎Scope is a type directed library search, derived work from OCaml API
> Search by Mizuno and its ancestor OCamlBrowser by Garrigue. It is also
> inspired from Hoogle, the same API search engine for Haskell by MItchell,
> which I regularly use in my Haskell :-) job.
>
> * Fast and Portable. It loads everything in memory, unlike OCaml API
> Search and OCamlBrowser which load compiled interface files (*.cmi)
> dynamically. The data file is extracted from compiled files but
> self-contained, so the search engine does not require to compile the
> libraries locally.
> * No use of unification but edit distance of types like Hoogle.
> Unification does not provide good results in type directed search, and is
> costy.
> * OCamlFind and OPAM friendly. OCaml◎Scope knows which items are from
> which OCamlFind and OPAM packages.
> * OCamlDoc: it also extracts OCamlDoc comments, if possible.
> * Small: it can even run as a heroku app. Currently it carries 245k
> entries from 76 OCamlFind packages including Core and Batteries, but the
> data file (as a marshalled OCaml value) is still 20Mb.
>
> There are lots of todos but I think the search results look well sane so
> far. If you find something strange please drop by
> https://bitbucket.org/camlspotter/ocamloscope-server/issues?status=new&status=open and
> leave some comments. Thanks!
>
> Jun Furuse
>
>

[-- Attachment #2: Type: text/html, Size: 2801 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

Does this supersede search.ocaml.jp? Shall we change the "OCaml API Search"
link on the front page of ocaml.org?

On Thu, Jul 4, 2013 at 10:11 PM, Jun Furuse <jun.furuse@gmail.com> wrote:

> Hi list,
>
> I have launched a new OCaml API search, OCaml◎Scope at
> http://ocamloscope.herokuapp.com .
>
> OCaml◎Scope is a type directed library search, derived work from OCaml API
> Search by Mizuno and its ancestor OCamlBrowser by Garrigue. It is also
> inspired from Hoogle, the same API search engine for Haskell by MItchell,
> which I regularly use in my Haskell :-) job.
>
> * Fast and Portable. It loads everything in memory, unlike OCaml API
> Search and OCamlBrowser which load compiled interface files (*.cmi)
> dynamically. The data file is extracted from compiled files but
> self-contained, so the search engine does not require to compile the
> libraries locally.
> * No use of unification but edit distance of types like Hoogle.
> Unification does not provide good results in type directed search, and is
> costy.
> * OCamlFind and OPAM friendly. OCaml◎Scope knows which items are from
> which OCamlFind and OPAM packages.
> * OCamlDoc: it also extracts OCamlDoc comments, if possible.
> * Small: it can even run as a heroku app. Currently it carries 245k
> entries from 76 OCamlFind packages including Core and Batteries, but the
> data file (as a marshalled OCaml value) is still 20Mb.
>
> There are lots of todos but I think the search results look well sane so
> far. If you find something strange please drop by
> https://bitbucket.org/camlspotter/ocamloscope-server/issues?status=new&status=open and
> leave some comments. Thanks!
>
> Jun Furuse
>
>

[-- Attachment #2: Type: text/html, Size: 2427 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Francois Berenger]

On 07/05/2013 11:11 AM, Jun Furuse wrote:
> Hi list,
>
> I have launched a new OCaml API search, OCaml◎Scope at
> http://ocamloscope.herokuapp.com .

Thanks god! At last!

Just a quick question:
are all the packages available in OPAM automatically added to ocamlscope
on a regular basis?

Thanks a lot for this extremely useful service,
F.

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Jun Furuse]

[-- Attachment #1: Type: text/plain, Size: 2622 bytes --]

Hi Yaron,

I have just updated it so that it expands manifest types. It is a dirty
hack but "split : string -> char -> string list" lists up
Core.Std.String.Escaping.split.

There are still lots of things to do before it becomes as powerful as
Hoogle is, but so far, I am working on its speed and size rather than
search quality.

Jun


On Fri, Jul 5, 2013 at 11:25 AM, Yaron Minsky <yminsky@janestreet.com>wrote:

> Neat!
>
> I played with it for a bit, and the main issue it saw with it is the
> trickiness around type names for things like string.  For example, to find
> core or batteries' string split function, you need to search for
>
> t -> char -> t list
>
> Rather than
>
> string -> char -> string list
>
> Which finds ocplib's equivalent, and the Core.Std.String.Escaping version
> as well.
>
> Fill on unification doesn't seem ideal, but I wonder if some ability to
> recognize equivalent type names can be done without full-on search time
> unification.
> On Jul 5, 2013 5:12 AM, "Jun Furuse" <jun.furuse@gmail.com> wrote:
>
>> Hi list,
>>
>> I have launched a new OCaml API search, OCaml◎Scope at
>> http://ocamloscope.herokuapp.com .
>>
>> OCaml◎Scope is a type directed library search, derived work from OCaml
>> API Search by Mizuno and its ancestor OCamlBrowser by Garrigue. It is also
>> inspired from Hoogle, the same API search engine for Haskell by MItchell,
>> which I regularly use in my Haskell :-) job.
>>
>> * Fast and Portable. It loads everything in memory, unlike OCaml API
>> Search and OCamlBrowser which load compiled interface files (*.cmi)
>> dynamically. The data file is extracted from compiled files but
>> self-contained, so the search engine does not require to compile the
>> libraries locally.
>> * No use of unification but edit distance of types like Hoogle.
>> Unification does not provide good results in type directed search, and is
>> costy.
>> * OCamlFind and OPAM friendly. OCaml◎Scope knows which items are from
>> which OCamlFind and OPAM packages.
>> * OCamlDoc: it also extracts OCamlDoc comments, if possible.
>> * Small: it can even run as a heroku app. Currently it carries 245k
>> entries from 76 OCamlFind packages including Core and Batteries, but the
>> data file (as a marshalled OCaml value) is still 20Mb.
>>
>> There are lots of todos but I think the search results look well sane so
>> far. If you find something strange please drop by
>> https://bitbucket.org/camlspotter/ocamloscope-server/issues?status=new&status=open and
>> leave some comments. Thanks!
>>
>> Jun Furuse
>>
>>

[-- Attachment #2: Type: text/html, Size: 3691 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Jun Furuse]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]

Hi Ashish

It is bit too early since OCamlOScope currently lacks hyperlinks from
search results to the corresponding module/package overviews. But apart
from that it covers lots more modules and packages than OCaml API Search :-)

Jun


On Fri, Jul 5, 2013 at 10:51 PM, Ashish Agarwal <agarwal1975@gmail.com>wrote:

> Does this supersede search.ocaml.jp? Shall we change the "OCaml API
> Search" link on the front page of ocaml.org?
>
>
> On Thu, Jul 4, 2013 at 10:11 PM, Jun Furuse <jun.furuse@gmail.com> wrote:
>
>> Hi list,
>>
>> I have launched a new OCaml API search, OCaml◎Scope at
>> http://ocamloscope.herokuapp.com .
>>
>> OCaml◎Scope is a type directed library search, derived work from OCaml
>> API Search by Mizuno and its ancestor OCamlBrowser by Garrigue. It is also
>> inspired from Hoogle, the same API search engine for Haskell by MItchell,
>> which I regularly use in my Haskell :-) job.
>>
>> * Fast and Portable. It loads everything in memory, unlike OCaml API
>> Search and OCamlBrowser which load compiled interface files (*.cmi)
>> dynamically. The data file is extracted from compiled files but
>> self-contained, so the search engine does not require to compile the
>> libraries locally.
>> * No use of unification but edit distance of types like Hoogle.
>> Unification does not provide good results in type directed search, and is
>> costy.
>> * OCamlFind and OPAM friendly. OCaml◎Scope knows which items are from
>> which OCamlFind and OPAM packages.
>> * OCamlDoc: it also extracts OCamlDoc comments, if possible.
>> * Small: it can even run as a heroku app. Currently it carries 245k
>> entries from 76 OCamlFind packages including Core and Batteries, but the
>> data file (as a marshalled OCaml value) is still 20Mb.
>>
>> There are lots of todos but I think the search results look well sane so
>> far. If you find something strange please drop by
>> https://bitbucket.org/camlspotter/ocamloscope-server/issues?status=new&status=open and
>> leave some comments. Thanks!
>>
>> Jun Furuse
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3176 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Jun Furuse]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

Hi Francois,

The ultimiate goal is to put all the available OPAM packages into the
search DB, and update the DB regularly. But now it is far from it: 35 OPAM
packages.

DB construction requires source code and bin-annot files (*.cmt and *.cmti)
therefore the libraries must be get compiled. Since it is very hard to
compile all the OPAM packages by one machine due to variety of architecture
and dependent libraries, we will have to build a public repository of
extracted portable DB files.

Regards,
Jun



On Mon, Jul 8, 2013 at 8:51 AM, Francois Berenger <berenger@riken.jp> wrote:

> On 07/05/2013 11:11 AM, Jun Furuse wrote:
>
>> Hi list,
>>
>> I have launched a new OCaml API search, OCaml◎Scope at
>> http://ocamloscope.herokuapp.**com <http://ocamloscope.herokuapp.com> .
>>
>
> Thanks god! At last!
>
> Just a quick question:
> are all the packages available in OPAM automatically added to ocamlscope
> on a regular basis?
>
> Thanks a lot for this extremely useful service,
> F.
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>

[-- Attachment #2: Type: text/html, Size: 2177 bytes --]

Re: [Caml-list] [ANN] OCamlOScope: a new OCaml API search

[Gerd Stolpmann]

Am Dienstag, den 09.07.2013, 00:06 +0800 schrieb Jun Furuse:
> Hi Francois,
> 
> 
> The ultimiate goal is to put all the available OPAM packages into the
> search DB, and update the DB regularly. But now it is far from it: 35
> OPAM packages.
> 
> 
> DB construction requires source code and bin-annot files (*.cmt and
> *.cmti) therefore the libraries must be get compiled. Since it is very
> hard to compile all the OPAM packages by one machine due to variety of
> architecture and dependent libraries, we will have to build a public
> repository of extracted portable DB files.

If you wanted to be 100% complete.

Look at docs.camlcity.org: This full-text search is for more than 200
GODI packages. While a full-text search has the advantage that it also
works for uncompiled sources, it nevertheless profits from build
information, and because of this, it is actually based on the output of
the GODI autobuilder. This is Linux-only (so far), but it was possible
to cover that many packages. Only very few packages do not run on Linux
or assume a special environment or libraries you cannot have on an
autobuilder box.

I'm wondering whether your type search could be made running on top of a
full-text search engine (basically, you can only search for words and
n-grams, and combine search results with set operators). That way you'd
get a very reasonable ranking.

Gerd


> 
> Regards,
> Jun
> 
> 
> 
> 
> On Mon, Jul 8, 2013 at 8:51 AM, Francois Berenger <berenger@riken.jp>
> wrote:
>         On 07/05/2013 11:11 AM, Jun Furuse wrote:
>         
>                 Hi list,
>                 
>                 I have launched a new OCaml API search, OCaml◎Scope at
>                 http://ocamloscope.herokuapp.com .
>         
>         
>         Thanks god! At last!
>         
>         Just a quick question:
>         are all the packages available in OPAM automatically added to
>         ocamlscope
>         on a regular basis?
>         
>         Thanks a lot for this extremely useful service,
>         F.
>         
>         
> 
> 
-- 
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------