[Caml-list] Opam improvement request

[Caml-list] Opam improvement request

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3354 bytes --]

I'm not sure if opam already does this, and I'm not unhappy with opam at
all about it, but I'd like to be able to have installation or upgrade
commands have fast rollback/forward capability. Let me explain what I mean.

So far as I've seen or know about, if an install fails then opam will give
a file that describes the state it had and which you can execute using a
command it also gives. I'm too lazy to look that up, but that's a different
functionality than what I'm describing - and more importantly, it
represents part of the problem because it requires building all of the
packages from source and because it itself suffers from the fragility that
I'm trying to get around.

What I'd like is, if opam could quickly snap back and forth between
combinations of package versions. I don't want to manage specification
files, I'd kind of like a docker like interface where I could just put
notes by dated, named package selection options and have a command manage
it for me. I don't want to track files manually.

I've had some scenarios where opam tells me how to restore my state, after
attempting to do an install or upgrade but in the process had an error.
What has actually happened then was, I had had a working version of
packages, which then broke or went away on the possibility of some version
change. opam packages can change rather quickly too, so I remember I once
just had to quit and come back, and doing an opam update got the changes to
take successfully because someone pushed a change that fixed it. It would
nice if, if there are any errors, I could keep my old state without even
having to re-install that. And that's because I don't care if I have even
20 times as much space consumed by opam because opam is keeping old states.
I just want to always have some working system.

I hope it doesn't sound like I'm complaining or being negative. All I'm
saying is I don't want to have some dependency version mismatch cause the
current system to become inoperable, and I know that opam can maintain
multiple compilers and all of that, and that there are other levels where I
could apply this, even using docker I can entirely get rid of it. But if
opam were to use snapshotting as the default behavior and exchanging
snapshots merely meant swapping out folder names within the opam directory,
that would be really fantastic.

Often, doing an update or upgrade means that all the packages that depend
on it have to get rebuilt. OCaml is fantastic for having such a fast build
time, but this still takes hours if you have large dependencies. Again,
opam packages change all the time, and if you don't regularly stay up to
date, it's really easy to get downwind of drastic changes; leaping many
versions for a given package can trigger the kinds of unexpected breaks
that I'm talking about.

It makes sense to me to think of opam package installs as being in a kind
of git staging area, where I can consider them tentatively and test it
against a particular package, but easily and quickly revert back to other
built packages (without needing to rebuild something again unnecessarily)
that I know are working.

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list https://inbox.ocaml.org/caml-list
Forum: https://discuss.ocaml.org/
Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 3353 bytes --]

Re: [Caml-list] Opam improvement request

[SF Markus Elfring]

> I don't want to manage specification files, …

I propose to take another look around consequences for such a wish.


> And that's because I don't care if I have even 20 times as much space consumed
> by opam because opam is keeping old states.

Are there any circumstances under which you would start to care
(or become concerned about corresponding storage allocation challenges)?


> All I'm saying is I don't want to have some dependency version mismatch
> cause the current system to become inoperable, …

I can generally follow such a desire to some degree.
But safe dependency management can also require to express expectations
for system properties in precise ways, can't it?

Would you like to fiddle with application binary interfaces?


> Again, opam packages change all the time, and if you don't regularly stay
> up to date, it's really easy to get downwind of drastic changes;
> leaping many versions for a given package can trigger the kinds of
> unexpected breaks that I'm talking about.

There are more challenges to consider from usual software evolution.


> It makes sense to me to think of opam package installs as being in a kind of
> git staging area, where I can consider them tentatively and test it against
> a particular package, but easily and quickly revert back to other built packages

You might stumble on target conflicts which will make it harder to find
and agree on a solution.


> (without needing to rebuild something again unnecessarily) that I know are working.
It can take a while until it is noticed that something is finally not working
as expected, can't it?

Regards,
Markus

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list https://inbox.ocaml.org/caml-list
Forum: https://discuss.ocaml.org/
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Opam improvement request

[katherine]

i'm pretty new to opam here, but do know that many OS package managers,
like pacman or freebsd's pkg, will generally keep a package cache, so
that, if something goes wrong, it's possible to check the package
manager log for which packages have changed and then reinstall the
(already on disk) previous versions. is that something like you mean?

obviously that could be automated in a couple of ways while still
keeping things simple, like having the package manager write a file for
the most recent action, with before and after state (statefulness oh no
D=), and having something like a revert command that tries to reinstate
that most previous version.

add also some configuration for how many versions of a package should be
kept at a time (2 or 3 or something), with the older versions deleted
automatically, and the cache size could also be kept fairly low.

On Mon, Dec 24, 2018 at 04:10:44PM -0500, Kenneth Adam Miller wrote:
> I'm not sure if opam already does this, and I'm not unhappy with opam at
> all about it, but I'd like to be able to have installation or upgrade
> commands have fast rollback/forward capability. Let me explain what I mean.
> 
> So far as I've seen or know about, if an install fails then opam will give
> a file that describes the state it had and which you can execute using a
> command it also gives. I'm too lazy to look that up, but that's a different
> functionality than what I'm describing - and more importantly, it
> represents part of the problem because it requires building all of the
> packages from source and because it itself suffers from the fragility that
> I'm trying to get around.
> 
> What I'd like is, if opam could quickly snap back and forth between
> combinations of package versions. I don't want to manage specification
> files, I'd kind of like a docker like interface where I could just put
> notes by dated, named package selection options and have a command manage
> it for me. I don't want to track files manually.
> 
> I've had some scenarios where opam tells me how to restore my state, after
> attempting to do an install or upgrade but in the process had an error.
> What has actually happened then was, I had had a working version of
> packages, which then broke or went away on the possibility of some version
> change. opam packages can change rather quickly too, so I remember I once
> just had to quit and come back, and doing an opam update got the changes to
> take successfully because someone pushed a change that fixed it. It would
> nice if, if there are any errors, I could keep my old state without even
> having to re-install that. And that's because I don't care if I have even
> 20 times as much space consumed by opam because opam is keeping old states.
> I just want to always have some working system.
> 
> I hope it doesn't sound like I'm complaining or being negative. All I'm
> saying is I don't want to have some dependency version mismatch cause the
> current system to become inoperable, and I know that opam can maintain
> multiple compilers and all of that, and that there are other levels where I
> could apply this, even using docker I can entirely get rid of it. But if
> opam were to use snapshotting as the default behavior and exchanging
> snapshots merely meant swapping out folder names within the opam directory,
> that would be really fantastic.
> 
> Often, doing an update or upgrade means that all the packages that depend
> on it have to get rebuilt. OCaml is fantastic for having such a fast build
> time, but this still takes hours if you have large dependencies. Again,
> opam packages change all the time, and if you don't regularly stay up to
> date, it's really easy to get downwind of drastic changes; leaping many
> versions for a given package can trigger the kinds of unexpected breaks
> that I'm talking about.
> 
> It makes sense to me to think of opam package installs as being in a kind
> of git staging area, where I can consider them tentatively and test it
> against a particular package, but easily and quickly revert back to other
> built packages (without needing to rebuild something again unnecessarily)
> that I know are working.
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list https://inbox.ocaml.org/caml-list
> Forum: https://discuss.ocaml.org/
> Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list https://inbox.ocaml.org/caml-list
Forum: https://discuss.ocaml.org/
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Opam improvement request

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 5343 bytes --]

Yeah automating that is what I am talking about. I don't think I want to do
much in terms of meta data as much as I want the system of changes for any
one opam command to be atomic and revertible without having to rebuild.
Basically, opam shouldn't immediately delete things, but instead move them
while building new ones. Upon doing everything necessary, do a folder swap
to make the actions take.

You have the right idea, I think I was just much too wordy about it.

On Tue, Dec 25, 2018, 4:50 AM katherine <shmibs@airen-no-jikken.icu wrote:

> i'm pretty new to opam here, but do know that many OS package managers,
> like pacman or freebsd's pkg, will generally keep a package cache, so
> that, if something goes wrong, it's possible to check the package
> manager log for which packages have changed and then reinstall the
> (already on disk) previous versions. is that something like you mean?
>
> obviously that could be automated in a couple of ways while still
> keeping things simple, like having the package manager write a file for
> the most recent action, with before and after state (statefulness oh no
> D=), and having something like a revert command that tries to reinstate
> that most previous version.
>
> add also some configuration for how many versions of a package should be
> kept at a time (2 or 3 or something), with the older versions deleted
> automatically, and the cache size could also be kept fairly low.
>
> On Mon, Dec 24, 2018 at 04:10:44PM -0500, Kenneth Adam Miller wrote:
> > I'm not sure if opam already does this, and I'm not unhappy with opam at
> > all about it, but I'd like to be able to have installation or upgrade
> > commands have fast rollback/forward capability. Let me explain what I
> mean.
> >
> > So far as I've seen or know about, if an install fails then opam will
> give
> > a file that describes the state it had and which you can execute using a
> > command it also gives. I'm too lazy to look that up, but that's a
> different
> > functionality than what I'm describing - and more importantly, it
> > represents part of the problem because it requires building all of the
> > packages from source and because it itself suffers from the fragility
> that
> > I'm trying to get around.
> >
> > What I'd like is, if opam could quickly snap back and forth between
> > combinations of package versions. I don't want to manage specification
> > files, I'd kind of like a docker like interface where I could just put
> > notes by dated, named package selection options and have a command manage
> > it for me. I don't want to track files manually.
> >
> > I've had some scenarios where opam tells me how to restore my state,
> after
> > attempting to do an install or upgrade but in the process had an error.
> > What has actually happened then was, I had had a working version of
> > packages, which then broke or went away on the possibility of some
> version
> > change. opam packages can change rather quickly too, so I remember I once
> > just had to quit and come back, and doing an opam update got the changes
> to
> > take successfully because someone pushed a change that fixed it. It would
> > nice if, if there are any errors, I could keep my old state without even
> > having to re-install that. And that's because I don't care if I have even
> > 20 times as much space consumed by opam because opam is keeping old
> states.
> > I just want to always have some working system.
> >
> > I hope it doesn't sound like I'm complaining or being negative. All I'm
> > saying is I don't want to have some dependency version mismatch cause the
> > current system to become inoperable, and I know that opam can maintain
> > multiple compilers and all of that, and that there are other levels
> where I
> > could apply this, even using docker I can entirely get rid of it. But if
> > opam were to use snapshotting as the default behavior and exchanging
> > snapshots merely meant swapping out folder names within the opam
> directory,
> > that would be really fantastic.
> >
> > Often, doing an update or upgrade means that all the packages that depend
> > on it have to get rebuilt. OCaml is fantastic for having such a fast
> build
> > time, but this still takes hours if you have large dependencies. Again,
> > opam packages change all the time, and if you don't regularly stay up to
> > date, it's really easy to get downwind of drastic changes; leaping many
> > versions for a given package can trigger the kinds of unexpected breaks
> > that I'm talking about.
> >
> > It makes sense to me to think of opam package installs as being in a kind
> > of git staging area, where I can consider them tentatively and test it
> > against a particular package, but easily and quickly revert back to other
> > built packages (without needing to rebuild something again unnecessarily)
> > that I know are working.
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> https://inbox.ocaml.org/caml-list
> > Forum: https://discuss.ocaml.org/
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list https://inbox.ocaml.org/caml-list
Forum: https://discuss.ocaml.org/
Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 6225 bytes --]

Re: [Caml-list] Opam improvement request

[Louis Gesbert]

[-- Attachment #1: Type: text/plain, Size: 4449 bytes --]

This kind of feature wishes are best directed to the opam tracker (https://github.com/ocaml/opam/issues); but let me give a quick answer:

- we are working on allowing a binary cache, and there are ongoing experiments for that. The main issue is relocatability of packages themselves (which is not relevant in your case, but still). The point is that you would still do a `opam switch import backup.export`, but the recompilations wouldn't need to be done again since the packages are at the same versions, and with the same installed dependencies.

- using the hook mechanism in opam 2.0 [1], you could pretty easily add snapshotting to your opam switch. For example, create a git repository in your prefix dir, and use `git commit -a" as a `post-session-commands` hook. You can then freely checkout to previous revisions of the switch (all opam metadata is included in the `.opam-switch` subdir, so that should be safe, as long as opam is not running while you do it of course).

Hope this helps,
Louis Gesbert - OCamlPro

[1] http://opam.ocaml.org/doc/Manual.html#configfield-pre-session-commands

> - Kenneth Adam Miller, 24/12/2018 16:10 -
> I'm not sure if opam already does this, and I'm not unhappy with opam at
> all about it, but I'd like to be able to have installation or upgrade
> commands have fast rollback/forward capability. Let me explain what I mean.
> 
> So far as I've seen or know about, if an install fails then opam will give
> a file that describes the state it had and which you can execute using a
> command it also gives. I'm too lazy to look that up, but that's a different
> functionality than what I'm describing - and more importantly, it
> represents part of the problem because it requires building all of the
> packages from source and because it itself suffers from the fragility that
> I'm trying to get around.
> 
> What I'd like is, if opam could quickly snap back and forth between
> combinations of package versions. I don't want to manage specification
> files, I'd kind of like a docker like interface where I could just put
> notes by dated, named package selection options and have a command manage
> it for me. I don't want to track files manually.
> 
> I've had some scenarios where opam tells me how to restore my state, after
> attempting to do an install or upgrade but in the process had an error.
> What has actually happened then was, I had had a working version of
> packages, which then broke or went away on the possibility of some version
> change. opam packages can change rather quickly too, so I remember I once
> just had to quit and come back, and doing an opam update got the changes to
> take successfully because someone pushed a change that fixed it. It would
> nice if, if there are any errors, I could keep my old state without even
> having to re-install that. And that's because I don't care if I have even
> 20 times as much space consumed by opam because opam is keeping old states.
> I just want to always have some working system.
> 
> I hope it doesn't sound like I'm complaining or being negative. All I'm
> saying is I don't want to have some dependency version mismatch cause the
> current system to become inoperable, and I know that opam can maintain
> multiple compilers and all of that, and that there are other levels where I
> could apply this, even using docker I can entirely get rid of it. But if
> opam were to use snapshotting as the default behavior and exchanging
> snapshots merely meant swapping out folder names within the opam directory,
> that would be really fantastic.
> 
> Often, doing an update or upgrade means that all the packages that depend
> on it have to get rebuilt. OCaml is fantastic for having such a fast build
> time, but this still takes hours if you have large dependencies. Again,
> opam packages change all the time, and if you don't regularly stay up to
> date, it's really easy to get downwind of drastic changes; leaping many
> versions for a given package can trigger the kinds of unexpected breaks
> that I'm talking about.
> 
> It makes sense to me to think of opam package installs as being in a kind
> of git staging area, where I can consider them tentatively and test it
> against a particular package, but easily and quickly revert back to other
> built packages (without needing to rebuild something again unnecessarily)
> that I know are working.
> 
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]