[Caml-list] odb questions

[Caml-list] odb questions

[Daniel Bünzli]

Hello,  

A few questions about odb.

1) Is it possible to specify in ~/.odb/packages a remote packages file (instead of just individual remote packages) ?

2) How does odb find out and manage dependencies ? How can I list them before installing a package ?

3) Are multiple versions of the same package supported (à la homebrew on osx) ?  

4) Is something like odb install -file package.tbz possible ?  

The use case for 4) is both for testing and so that I can instruct my users once they have downloaded a tbz and went through the README that they can just install it with that command (provided they have installed other deps).  

Daniel

P.S. I think odb should use a odb <cmd> scheme. Instead of just installing on odb <pkg>. The former scheme has better scaling opportunities.   

Re: [Caml-list] odb questions

[Edgar Friendly]

On 03/07/2012 07:32 AM, Daniel Bünzli wrote:
> Hello,
>
> A few questions about odb.
>
> 1) Is it possible to specify in ~/.odb/packages a remote packages file (instead of just individual remote packages) ?

Not at the moment, but this is an interesting idea.  The only support 
for remote packages is the querying of oasis-db.  The protocol is nearly 
trivial, so you're welcome to make your own server (can be just a plain 
web server serving static text files) and set the odb repository URL to it.
>
> 2) How does odb find out and manage dependencies ? How can I list them before installing a package ?
>
Odb uses findlib to determine what libraries are installed.  For 
executables, the unix command `which` is used.  In the works is 
extracting the version of executables from running the command with 
`-version`.

The other end of this is handled by odb's metadata that specifies things 
like where to get the source code, dependencies, installation 
requirements, and other useful information for installing.  There are 
two sources for this metadata: 1) oasis-db by http 2) packages files 
locally.  The packages files are parsed on startup, if odb needs info 
about a package not in this cache, it queries oasis-db for the metadata.

I like the idea of printing deps without installing, and it should be 
pretty easy to implement.  It'd probably suffice to just print the whole 
metadata; it's quite readable.  Maybe metadata + dependency check will 
satisfy your needs?

> 3) Are multiple versions of the same package supported (à la homebrew on osx) ?
>
This is supported in two ways.  First through environment variables that 
basically switch between different ocaml installations / ocamlfind root 
paths.  Odb relies on findlib to manage libraries, and it supports this 
through tweaking install paths.

The other way is to have packages batteries1 and batteres2 that both 
install different versions of batteries, and as long as they register in 
findlib with the names batteries1 and batteries2, odb will treat them as 
different.

> 4) Is something like odb install -file package.tbz possible ?
>
It's planned to be able to give a full package info on the command line, 
so this would be more like:

odb tarball=package.tbz deps=odn,batteries

> The use case for 4) is both for testing and so that I can instruct my users once they have downloaded a tbz and went through the README that they can just install it with that command (provided they have installed other deps).
>
The current way to do testing is to put a line in your local packages 
file (two locations that are scanned for one: ~/.odb and the odb script 
dir), and use the info there for testing.  Then once you've got it 
working, make a simple _oasis file for it, and upload it to oasis-db - 
then everyone will gain access to it automatically.

> Daniel
>
> P.S. I think odb should use a odb<cmd>  scheme. Instead of just installing on odb<pkg>. The former scheme has better scaling opportunities.

Except I don't intend odb to scale.  It's hovering right around 500 
lines of code, and if I can keep it around that size, I'll be happy.  I 
don't intend odb to ever have a `odb remove foo` command, nor an `odb 
update`.  Maybe I'll have an `odb upgrade`, but I'm happy leaving this 
as `odb --upgrade`.  Odb's goal is to be an 80% solution, and it was 
that a while ago, I think.  It's approaching 90%, and I'm going to try 
to avoid it feature creeping to 100%.  Oddities should be handled by the 
package installers themselves or the metadata repository as much as 
possible, leaving pretty little work for odb.

E.

Re: [Caml-list] odb questions

[Daniel Bünzli]

Le mercredi, 7 mars 2012 à 15:37, Edgar Friendly a écrit :
> Not at the moment, but this is an interesting idea. The only support
> for remote packages is the querying of oasis-db. The protocol is nearly  
> trivial, so you're welcome to make your own server (can be just a plain  
> web server serving static text files) and set the odb repository URL to it.

  
Okay. Would be nice to have the support in the tool and documentation on how to make your own package source (basically GET on a ~/.odb/packages file ?). Package distribution should be distributed.

> The other end of this is handled by odb's metadata that specifies things  
> like where to get the source code, dependencies, installation  
> requirements, and other useful information for installing. There are  
> two sources for this metadata: 1) oasis-db by http 2) packages files  
> locally. The packages files are parsed on startup, if odb needs info  
> about a package not in this cache, it queries oasis-db for the metadata.


By 2) you mean what is described in "Local packages" here : https://github.com/thelema/odb  ? Because the syntax for specifiying deps is not documented.  
  
> I like the idea of printing deps without installing, and it should be
> pretty easy to implement. It'd probably suffice to just print the whole  
> metadata; it's quite readable. Maybe metadata + dependency check will  
> satisfy your needs?

  
Why not ?  
  
> This is supported in two ways. First through environment variables that
> basically switch between different ocaml installations / ocamlfind root  
> paths. Odb relies on findlib to manage libraries, and it supports this  
> through tweaking install paths.
>  
> The other way is to have packages batteries1 and batteres2 that both  
> install different versions of batteries, and as long as they register in  
> findlib with the names batteries1 and batteries2, odb will treat them as  
> different.


IMHO a package should be identified by a name and version.

> It's planned to be able to give a full package info on the command line,
> so this would be more like:
>  
> odb tarball=package.tbz deps=odn,batteries

That would be nice.  

> The current way to do testing is to put a line in your local packages
> file (two locations that are scanned for one: ~/.odb and the odb script  
> dir), and use the info there for testing.  


Okay but how do I specify deps ? I'm sure I can quickly find out but I want you to update the docs...

> Except I don't intend odb to scale.
  
Pity.  

> It's hovering right around 500  
> lines of code, and if I can keep it around that size, I'll be happy.

  
That's indeed a good thing, aswell as no deep deps. But hey, setup.ml is around 10x more (which makes it unusable for dev. btw.) you still have some way to go.  
  
> I  
> don't intend odb to ever have a `odb remove foo` command, nor an `odb  
> update`. Maybe I'll have an `odb upgrade`, but I'm happy leaving this  
> as `odb --upgrade`. Odb's goal is to be an 80% solution, and it was  
> that a while ago, I think. It's approaching 90%, and I'm going to try  
> to avoid it feature creeping to 100%. Oddities should be handled by the  
> package installers themselves or the metadata repository as much as  
> possible, leaving pretty little work for odb.

  
Pity. I like odb's down to earth approach.  

Best,

Daniel

Re: [Caml-list] odb questions

[Edgar Friendly]

On 03/07/2012 10:47 AM, Daniel Bünzli wrote:
> Okay. Would be nice to have the support in the tool and documentation
> on how to make your own package source (basically GET on a
> ~/.odb/packages file ?). Package distribution should be distributed.
>
one file per package, filename is package name, contents are same format 
as rest of packages line but with \n between key=value pairs instead of 
space.

> By 2) you mean what is described in "Local packages" here :
> https://github.com/thelema/odb  ? Because the syntax for specifiying
> deps is not documented.
>
Deps are just a comma-separated list of package names, with optional 
version numbers in parentheses.  I'll document this in the packages file 
now.

> IMHO a package should be identified by a name and version.
>
I've been thinking about this for a long time, and the full consequences 
of this involve not only deep changes to odb internals, but also expose 
the code to a ton more edge cases that need to be handled, as well as 
possibly some NP-hard problems of resolving version dependencies.  Odb 
may go this way if needed, but the current practice of having a single 
string as the package identifier is sufficient for my use, so...

> Pity. I like odb's down to earth approach.

You're more than welcome to fork and build a better odb.  I may even 
steal any good ideas you have and put them back in my odb.  :)

E.

Re: [Caml-list] odb questions

[Daniel Bünzli]

Le mercredi, 7 mars 2012 à 16:58, Edgar Friendly a écrit :
> > IMHO a package should be identified by a name and version.
>  
> I've been thinking about this for a long time, and the full consequences  
> of this involve not only deep changes to odb internals, but also expose  
> the code to a ton more edge cases that need to be handled, as well as  
> possibly some NP-hard problems of resolving version dependencies. Odb  
> may go this way if needed, but the current practice of having a single  
> string as the package identifier is sufficient for my use, so...


Yes, I understand that. I fear there's dependency hell waiting around the corner, but maybe this can be mitigated by implementing that idea of being able to easily specify your own package source in ~/.odb/packages. Well, in fact you can already do that locally by crafting your own ~/.odb/packages file. So maybe your approach is the right balancing act between usability and complexity.  

> You're more than welcome to fork and build a better odb. I may even  
> steal any good ideas you have and put them back in my odb. :)


If I get too annoyed with too much time on my hands (i.e. very unlikely) I may eventually do that.  

Thanks for the answers and the tool,  

Daniel

Re: [Caml-list] odb questions

[Daniel Bünzli]

Le mercredi, 7 mars 2012 à 16:58, Edgar Friendly a écrit :

> > IMHO a package should be identified by a name and version.
>  
> I've been thinking about this for a long time, and the full consequences  
> of this involve not only deep changes to odb internals, but also expose  
> the code to a ton more edge cases that need to be handled, as well as  
> possibly some NP-hard problems of resolving version dependencies.  

Without going into the dependency resolving thing, I still think it's an issue. Basically with odb you don't really know what you are downloading.  

For example I wanted to reinstall oasis, didn't have *any* clue which version I would get, went to oasis.ocamlcore.org, saw that v0.3.0rc2 was uploaded there so I expected to get that, ended with 0.2.1~alpha1. Had I knew before I would just have added a direct link to the new release in my `packages` files.  

Didn't really think hard about it, but instead of (or complementary to) defining it's own key-value language odb could maybe piggyback on oasis files (pretty easy to parse). There's a lot of info in there (deps etc.). So having something like :  

<pkg> tarball=http://link/to/<pkg>.tbz oasis=http://link/to/<pkg>.oasis

so that you can explore basic information about the package before even trying to download it (alternatively download the tarball and look for an oasis file to spit out, but then if that's not what you wanted you already wasted the bandwidth)

So an odb -info <pkg> that spits out an oasis file if any is available would be nice aswell as an odb -deps <pkg>.  

Best,

Daniel

Re: [Caml-list] odb questions

[Edgar Friendly]

On 03/09/2012 09:26 AM, Daniel Bünzli wrote:
> Without going into the dependency resolving thing, I still think it's
> an issue. Basically with odb you don't really know what you are
> downloading.
>
http://oasis.ocamlcore.org/dev/odb/

This page shows the stable, testing and unstable versions of each
package available in odb.  use --unstable or --testing to get a
non-stable version.  Unstable is the newest uploaded version, testing is
the newest version I've been able to make work.  Stable has all been
compiled together (sometimes there's version issues between packages).

> Didn't really think hard about it, but instead of (or complementary
> to) defining it's own key-value language odb could maybe piggyback on
> oasis files (pretty easy to parse). There's a lot of info in there
> (deps etc.).

While I agree it'd be nice to just parse oasis files, they're not nearly 
as trivial to parse as the k=v format.  If you can write a small enough 
parser with no external deps, I'll add it to odb and use its info to 
supplement the packages file.

E.

Re: [Caml-list] odb questions

[Daniel Bünzli]

Le vendredi, 9 mars 2012 à 17:26, Edgar Friendly a écrit :

> http://oasis.ocamlcore.org/dev/odb/

Ah missed that. Maybe you should mention the link in the README.md (searching for "odb ocaml" on the web brings me directly to github).  
  
> While I agree it'd be nice to just parse oasis files, they're not nearly  
> as trivial to parse as the k=v format. If you can write a small enough  
> parser with no external deps, I'll add it to odb and use its info to  
> supplement the packages file.


May happen, or not.

Thanks,

Daniel

Re: [Caml-list] odb questions

[Francois Berenger]

On 03/07/2012 11:37 PM, Edgar Friendly wrote:
> [...]
 >
> Except I don't intend odb to scale. It's hovering right around 500 lines
> of code, and if I can keep it around that size, I'll be happy. I don't
> intend odb to ever have a `odb remove foo` command

Hi,

Wouldn't it be possible to have 'odb remove foo' just call
'ocamlfind remove foo'?

Regards,
F.

 > nor an `odb update`.
> Maybe I'll have an `odb upgrade`, but I'm happy leaving this as `odb
> --upgrade`. Odb's goal is to be an 80% solution, and it was that a while
> ago, I think. It's approaching 90%, and I'm going to try to avoid it
> feature creeping to 100%. Oddities should be handled by the package
> installers themselves or the metadata repository as much as possible,
> leaving pretty little work for odb.
>
> E.

Re: [Caml-list] odb questions

[Edgar Friendly]

On 03/07/2012 08:02 PM, Francois Berenger wrote:
> Wouldn't it be possible to have 'odb remove foo' just call
> 'ocamlfind remove foo'?
>

Yes, this is possible.  Most details of this are already implemented; 
the code to do `ocamlfind remove foo` is already implemented as part of 
--force for reinstalling/upgrading packages that don't ocamlfind remove 
themselves as part of their `make install`, but...

This doesn't remove the package.  It only stops ocamlfind from 
complaining about double installation of the same package.  If users 
want to remove an ocamlfind package, they can type "ocamlfind remove 
foo" themselves; it's saving them only a few keystrokes to have an odb 
alias for the same thing, and even if it were added, it wouldn't work 
properly in many cases, as it wouldn't deal with executables and any 
other files installed by a program (documentation, etc.).

For myself, when I want to remove something installed by odb, I usually 
just nuke my whole ~/.odb directory and reinstall the packages I want to 
keep.

Thus my resistance to 'odb --remove foo'.

E.

Re: [Caml-list] odb questions

[Daniel Bünzli]

Le jeudi, 8 mars 2012 à 14:29, Edgar Friendly a écrit :
> For myself, when I want to remove something installed by odb, I usually
> just nuke my whole ~/.odb directory and reinstall the packages I want to  
> keep.

I like your down to earth approach !

Daniel