[Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

Hi,

In preparation for the 113.24 release of Jane Street packages, which are
camlp4-free, we moved some Camlp4 syntax extensions into their own packages
[1]. As a result some findlib packages have been renamed. For instance the
old sexplib.syntax is now called pa_sexp_conv.

Details and upgrading instructions can be found in the opam repository
changelog [2].

Since the upcoming 113.24 release of sexplib, bin_prot and a few other
packages breaks compatibility due to these renamings, version constraints
were automatically added to the appropriate opam packages.

This is the list of affected packages:

- async-zmq
- bookaml
- camlhighlight
- charrua-core
- cohttp
- colors_js
- conduit
- cstruct
- extprot
- ezjsonm
- frenetic
- git-unix
- ibx
- ipaddr
- irmin
- lambdoc
- lazy-trie
- merge-queues
- merge-ropes
- mirage
- mirage-git
- mirage-http
- mirage-tc
- mirage-types
- mirage-types-lwt
- mustache
- nocrypto
- obeanstalk
- ocaml-markdown
- oloop
- openflow
- operf-macro
- opium
- otr
- packet
- perf
- planck
- protocol-9p
- qcow-format
- shared-block-ring
- sibylfs
- sill
- tls
- tlstunnel
- topology
- uri
- vchan
- x509
- xapi-backtrace
- xapi-stdext
- xapi-xenopsd
- xentropyd

If you maintain any of these packages, you are kindly invited to follow the
upgrading instructions [2].

Moving forward, we encourage people using any of our Camlp4 syntax
extensions to switch to their ppx replacements.

 [1] https://github.com/ocaml/opam-repository/pull/5414
 [2] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md

-- 
Jeremie
​ Dimino

[-- Attachment #2: Type: text/html, Size: 4745 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 2191 bytes --]

Hi,

I hope that the change will not modify library names retrospectively? I.e.,
the old versions will be still available under the `.syntax` names?

Also, presumably some tools rely on the `.syntax` extension, maybe it is
better to preserve the extension, i.e., to use `pa_sexp_conv.syntax`?

On Tue, Jan 26, 2016 at 5:44 AM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> Hi,
>
> In preparation for the 113.24 release of Jane Street packages, which are
> camlp4-free, we moved some Camlp4 syntax extensions into their own packages
> [1]. As a result some findlib packages have been renamed. For instance the
> old sexplib.syntax is now called pa_sexp_conv.
>
> Details and upgrading instructions can be found in the opam repository
> changelog [2].
>
> Since the upcoming 113.24 release of sexplib, bin_prot and a few other
> packages breaks compatibility due to these renamings, version constraints
> were automatically added to the appropriate opam packages.
>
> This is the list of affected packages:
>
> - async-zmq
> - bookaml
> - camlhighlight
> - charrua-core
> - cohttp
> - colors_js
> - conduit
> - cstruct
> - extprot
> - ezjsonm
> - frenetic
> - git-unix
> - ibx
> - ipaddr
> - irmin
> - lambdoc
> - lazy-trie
> - merge-queues
> - merge-ropes
> - mirage
> - mirage-git
> - mirage-http
> - mirage-tc
> - mirage-types
> - mirage-types-lwt
> - mustache
> - nocrypto
> - obeanstalk
> - ocaml-markdown
> - oloop
> - openflow
> - operf-macro
> - opium
> - otr
> - packet
> - perf
> - planck
> - protocol-9p
> - qcow-format
> - shared-block-ring
> - sibylfs
> - sill
> - tls
> - tlstunnel
> - topology
> - uri
> - vchan
> - x509
> - xapi-backtrace
> - xapi-stdext
> - xapi-xenopsd
> - xentropyd
>
> If you maintain any of these packages, you are kindly invited to follow
> the upgrading instructions [2].
>
> Moving forward, we encourage people using any of our Camlp4 syntax
> extensions to switch to their ppx replacements.
>
>  [1] https://github.com/ocaml/opam-repository/pull/5414
>  [2] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md
>
> --
> Jeremie
> ​ Dimino
>

[-- Attachment #2: Type: text/html, Size: 5530 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On Tue, Jan 26, 2016 at 1:57 PM, Ivan Gotovchits <ivg@ieee.org> wrote:

> I hope that the change will not modify library names retrospectively?
> I.e., the old versions will be still available under the `.syntax` names?
>

​The '.syntax' names are gone. I thought about keeping them as aliases, but
eventually they'll go away and this split already breaks opam dependencies,
so it's simpler to just do the full upgrade right now.

Also, presumably some tools rely on the `.syntax` extension, maybe it is
> better to preserve the extension, i.e., to use `pa_sexp_conv.syntax`?
>

So far we only did this for syntax extensions with a runtime part. For
instance type_conv doesn't have a .syntax package. Do you know of any tool
that rely on this?

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 1744 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

> Do you know of any tool that rely on this?

OASIS it the most notable [1].

> ​The '.syntax' names are gone. I thought about keeping them as aliases,
but eventually they'll go away and this split already breaks opam
dependencies, so it's simpler to just do the full upgrade right now.

Usually, when you release something, you shouldn't change it. The releasing
is a process of fixing things in time. Changing a name of library is the
same as changing the library.

And consider the following example, suppose a released project.0.3 uses one
of the removed syntax packages. Ok, you will fix the opam file, and a
correct set of dependencies will be installed after an update,
but the package build system will still contain hardcoded old names (in
_oasis, Makefile, etc). So what should a package maintainer do? I see two
options.

1. Delete `project.0.3` from the repository and add new `project.0.3-???`
with a fixed build system
2. Retrospectively modify `project.0.3` build system and upload a new
tarball without changing a library.

The second option will break an expected assumption, that library
`project.0.3` should always be the same in time. The first option is not an
option also.

Also, some packages, may encode library names in the code itself. For
example, bap uses this to track dependencies of dynamically loaded plugins.
Merlin has a heuristics, that
guesses requested syntax extensions based on package names.

[1]:
https://github.com/ocaml/oasis/blob/master/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml#L161

On Tue, Jan 26, 2016 at 9:50 AM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> On Tue, Jan 26, 2016 at 1:57 PM, Ivan Gotovchits <ivg@ieee.org> wrote:
>
>> I hope that the change will not modify library names retrospectively?
>> I.e., the old versions will be still available under the `.syntax` names?
>>
>
> ​The '.syntax' names are gone. I thought about keeping them as aliases,
> but eventually they'll go away and this split already breaks opam
> dependencies, so it's simpler to just do the full upgrade right now.
>
> Also, presumably some tools rely on the `.syntax` extension, maybe it is
>> better to preserve the extension, i.e., to use `pa_sexp_conv.syntax`?
>>
>
> So far we only did this for syntax extensions with a runtime part. For
> instance type_conv doesn't have a .syntax package. Do you know of any tool
> that rely on this?
>
> --
> Jeremie
>

[-- Attachment #2: Type: text/html, Size: 4127 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On Tue, Jan 26, 2016 at 3:15 PM, Ivan Gotovchits <ivg@ieee.org> wrote:

> > Do you know of any tool that rely on this?
>
> OASIS it the most notable [1].
>

​Ah, I didn't know about this hack​, and apparently it's the same hack for
merlin. Well, we can certainly add ".syntax" packages to pa_sexp_conv and
others then


> 1. Delete `project.0.3` from the repository and add new `project.0.3-???`
> with a fixed build system
> 2. Retrospectively modify `project.0.3` build system and upload a new
> tarball without changing a library.
>

​I was thinking of:

3. release project.0.3.1, keeping project.0.3 unchanged

That the same as what one would have to do if the API changed in an
incompatible way.


> Also, some packages, may encode library names in the code itself. For
> example, bap uses this to track dependencies of dynamically loaded plugins.
> Merlin has a heuristics, that
> guesses requested syntax extensions based on package names.
>

​For merlin and oasis we just need to add pa_sexp_conv.syntax, we'll do
that.
I don't know bap very well, does it have "sexplib.syntax" hardcoded ​in its
source code or is the same situation as merlin and oasis?

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 2912 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 2543 bytes --]

> 3. release project.0.3.1, keeping project.0.3 unchanged

The problem is that `project.0.3` will be broken for ever, as well as all
dependencies that depend on this particular version.

Maybe there is actually some misunderstanding, and we're talking about
different things. Just to clarify, are you going to go
through opam-repository and update all archives of old libraries (e.g.
sexplib.7.0.5, sexplib.108.00, ...) and remove `<library>.syntax`
findlib library from it? This is what I'm afraid of. And this is what I
name "retroactively".
As according to the PR there is nothing like this. If all old archives will
be still available, then it is perfectly fine.

> ​For merlin and oasis we just need to add pa_sexp_conv.syntax, we'll do
that.

This is what I'm proposing.

> I don't know bap very well, does it have "sexplib.syntax" hardcoded ​in
its source code or is the same situation as merlin and oasis?

Yep, but it is not a problem to me to fix it. I'm just wondering are there
any more packages, that provide build systems, that can have such problems.
Maybe just grepping the opam universe will help in answering this question.




On Tue, Jan 26, 2016 at 11:15 AM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> On Tue, Jan 26, 2016 at 3:15 PM, Ivan Gotovchits <ivg@ieee.org> wrote:
>
>> > Do you know of any tool that rely on this?
>>
>> OASIS it the most notable [1].
>>
>
> ​Ah, I didn't know about this hack​, and apparently it's the same hack for
> merlin. Well, we can certainly add ".syntax" packages to pa_sexp_conv and
> others then
>
>
>> 1. Delete `project.0.3` from the repository and add new `project.0.3-???`
>> with a fixed build system
>> 2. Retrospectively modify `project.0.3` build system and upload a new
>> tarball without changing a library.
>>
>
> ​I was thinking of:
>
> 3. release project.0.3.1, keeping project.0.3 unchanged
>
> That the same as what one would have to do if the API changed in an
> incompatible way.
>
>
>> Also, some packages, may encode library names in the code itself. For
>> example, bap uses this to track dependencies of dynamically loaded plugins.
>> Merlin has a heuristics, that
>> guesses requested syntax extensions based on package names.
>>
>
> ​For merlin and oasis we just need to add pa_sexp_conv.syntax, we'll do
> that.
> I don't know bap very well, does it have "sexplib.syntax" hardcoded ​in
> its source code or is the same situation as merlin and oasis?
>
> --
> Jeremie
>

[-- Attachment #2: Type: text/html, Size: 4933 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On Tue, Jan 26, 2016 at 4:56 PM, Ivan Gotovchits <ivg@ieee.org> wrote:

> > 3. release project.0.3.1, keeping project.0.3 unchanged
>
> The problem is that `project.0.3` will be broken for ever, as well as all
> dependencies that depend on this particular version.
>
> Maybe there is actually some misunderstanding, and we're talking about
> different things. Just to clarify, are you going to go
> through opam-repository and update all archives of old libraries (e.g.
> sexplib.7.0.5, sexplib.108.00, ...) and remove `<library>.syntax`
> findlib library from it? This is what I'm afraid of. And this is what I
> name "retroactively".
> As according to the PR there is nothing like this. If all old archives
> will be still available, then it is perfectly fine.
>

​Indeed, I misunderstood your original ​question. Sorry about that.
We definitely won't touch the archive of any released package. I agree that
would be disastrous...

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 1819 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

Just to follow up, the addition of `.syntax` packages was released this
week: https://github.com/ocaml/opam-repository/pull/5523

On Tue, Jan 26, 2016 at 5:17 PM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> On Tue, Jan 26, 2016 at 4:56 PM, Ivan Gotovchits <ivg@ieee.org> wrote:
>
>> > 3. release project.0.3.1, keeping project.0.3 unchanged
>>
>> The problem is that `project.0.3` will be broken for ever, as well as all
>> dependencies that depend on this particular version.
>>
>> Maybe there is actually some misunderstanding, and we're talking about
>> different things. Just to clarify, are you going to go
>> through opam-repository and update all archives of old libraries (e.g.
>> sexplib.7.0.5, sexplib.108.00, ...) and remove `<library>.syntax`
>> findlib library from it? This is what I'm afraid of. And this is what I
>> name "retroactively".
>> As according to the PR there is nothing like this. If all old archives
>> will be still available, then it is perfectly fine.
>>
>
> ​Indeed, I misunderstood your original ​question. Sorry about that.
> We definitely won't touch the archive of any released package. I agree
> that would be disastrous...
>
> --
> Jeremie
>



-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 2680 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[ygrek]

On Tue, 26 Jan 2016 10:44:22 +0000
Jeremie Dimino <jdimino@janestreet.com> wrote:

> In preparation for the 113.24 release of Jane Street packages, which are
> camlp4-free, we moved some Camlp4 syntax extensions into their own packages
> [1]. As a result some findlib packages have been renamed. For instance the
> old sexplib.syntax is now called pa_sexp_conv.

I wonder if keeping empty sexplib.syntax subpackage with only requires="pa_sexp_conv" line
will prevent build breakage (one will only need to fix opam dependencies)?

-- 

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On Wed, Feb 3, 2016 at 7:20 PM, ygrek <ygrek@autistici.org> wrote:

> I wonder if keeping empty sexplib.syntax subpackage with only
> requires="pa_sexp_conv" line
> will prevent build breakage (one will only need to fix opam dependencies)?
>

​That would work for now. However, sooner or latter some change in​

​sexplib will break the code generated by pa_sexp_conv and we'll have the
same problem again.
The current forced upgrade is simple (one sed command [1]) ​and it's a hint
that it's a good time to switch to ppx. We published a tool to help with
this BTW [2]

 [1] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md
 ​[2] ​
https://blogs.janestreet.com/converting-a-code-base-from-camlp4-to-ppx
​      ​
https://github.com/janestreet/camlp4-to-ppx

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 2324 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Junsong Li]

Hi Jeremie,

I have 4.02.1 OCaml installed on my Mac. opam update; opam upgrade
didn't upgrade the sexplib to 113.24. opam list still shows 113.00.
Any ideas?

My fresh installed 4.02.3 OCaml on a Linux machine has 113.24 sexplib though.

On Fri, Feb 5, 2016 at 1:36 AM, Jeremie Dimino <jdimino@janestreet.com> wrote:
> On Wed, Feb 3, 2016 at 7:20 PM, ygrek <ygrek@autistici.org> wrote:
>>
>> I wonder if keeping empty sexplib.syntax subpackage with only
>> requires="pa_sexp_conv" line
>> will prevent build breakage (one will only need to fix opam dependencies)?
>
>
> That would work for now. However, sooner or latter some change in
>
> sexplib will break the code generated by pa_sexp_conv and we'll have the
> same problem again.
> The current forced upgrade is simple (one sed command [1]) and it's a hint
> that it's a good time to switch to ppx. We published a tool to help with
> this BTW [2]
>
>  [1] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md
>  [2] https://blogs.janestreet.com/converting-a-code-base-from-camlp4-to-ppx
>
> https://github.com/janestreet/camlp4-to-ppx
>
> --
> Jeremie

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

Hi Junsong,

It's possible that you have a package installed that has no version
compatible with sexplib 113.24. You can try to explicitly upgrade sexplib
to see what solution opam suggests

On Wed, Mar 9, 2016 at 3:56 PM, Junsong Li <ljs.darkfish@gmail.com> wrote:

> Hi Jeremie,
>
> I have 4.02.1 OCaml installed on my Mac. opam update; opam upgrade
> didn't upgrade the sexplib to 113.24. opam list still shows 113.00.
> Any ideas?
>
> My fresh installed 4.02.3 OCaml on a Linux machine has 113.24 sexplib
> though.
>
> On Fri, Feb 5, 2016 at 1:36 AM, Jeremie Dimino <jdimino@janestreet.com>
> wrote:
> > On Wed, Feb 3, 2016 at 7:20 PM, ygrek <ygrek@autistici.org> wrote:
> >>
> >> I wonder if keeping empty sexplib.syntax subpackage with only
> >> requires="pa_sexp_conv" line
> >> will prevent build breakage (one will only need to fix opam
> dependencies)?
> >
> >
> > That would work for now. However, sooner or latter some change in
> >
> > sexplib will break the code generated by pa_sexp_conv and we'll have the
> > same problem again.
> > The current forced upgrade is simple (one sed command [1]) and it's a
> hint
> > that it's a good time to switch to ppx. We published a tool to help with
> > this BTW [2]
> >
> >  [1] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md
> >  [2]
> https://blogs.janestreet.com/converting-a-code-base-from-camlp4-to-ppx
> >
> > https://github.com/janestreet/camlp4-to-ppx
> >
> > --
> > Jeremie
>



-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 2737 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Junsong Li]

Ah, I see

$ opam upgrade --verbose sexplib
Everything as up-to-date as possible.
The following newer versions couldn't be installed:
  - sexplib.113.24.00 is not available because your system doesn't
comply with ocaml-version >= "4.02.3".

I'll install OCaml 4.02.3 then.

Thank you.

On Wed, Mar 9, 2016 at 8:06 AM, Jeremie Dimino <jdimino@janestreet.com> wrote:
> Hi Junsong,
>
> It's possible that you have a package installed that has no version
> compatible with sexplib 113.24. You can try to explicitly upgrade sexplib to
> see what solution opam suggests
>
> On Wed, Mar 9, 2016 at 3:56 PM, Junsong Li <ljs.darkfish@gmail.com> wrote:
>>
>> Hi Jeremie,
>>
>> I have 4.02.1 OCaml installed on my Mac. opam update; opam upgrade
>> didn't upgrade the sexplib to 113.24. opam list still shows 113.00.
>> Any ideas?
>>
>> My fresh installed 4.02.3 OCaml on a Linux machine has 113.24 sexplib
>> though.
>>
>> On Fri, Feb 5, 2016 at 1:36 AM, Jeremie Dimino <jdimino@janestreet.com>
>> wrote:
>> > On Wed, Feb 3, 2016 at 7:20 PM, ygrek <ygrek@autistici.org> wrote:
>> >>
>> >> I wonder if keeping empty sexplib.syntax subpackage with only
>> >> requires="pa_sexp_conv" line
>> >> will prevent build breakage (one will only need to fix opam
>> >> dependencies)?
>> >
>> >
>> > That would work for now. However, sooner or latter some change in
>> >
>> > sexplib will break the code generated by pa_sexp_conv and we'll have the
>> > same problem again.
>> > The current forced upgrade is simple (one sed command [1]) and it's a
>> > hint
>> > that it's a good time to switch to ppx. We published a tool to help with
>> > this BTW [2]
>> >
>> >  [1] https://github.com/ocaml/opam-repository/blob/master/CHANGES.md
>> >  [2]
>> > https://blogs.janestreet.com/converting-a-code-base-from-camlp4-to-ppx
>> >
>> > https://github.com/janestreet/camlp4-to-ppx
>> >
>> > --
>> > Jeremie
>
>
>
>
> --
> Jeremie

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

On Wed, Mar 9, 2016 at 10:56 AM, Junsong Li <ljs.darkfish@gmail.com> wrote:
>
>
> I have 4.02.1 OCaml installed on my Mac. opam update; opam upgrade
>

 Sexplib 113.24.0, is only available for ocaml >= 4.02.3
The same is true for all other core packages of this version and above.

[-- Attachment #2: Type: text/html, Size: 646 bytes --]

Re: [Caml-list] Package renamings for sexplib, bin_prot and a few other camlp4 syntax extensions

[Junsong Li]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

Thanks.

On Wednesday, March 9, 2016, Ivan Gotovchits <ivg@ieee.org> wrote:

>
>
> On Wed, Mar 9, 2016 at 10:56 AM, Junsong Li <ljs.darkfish@gmail.com
> <javascript:_e(%7B%7D,'cvml','ljs.darkfish@gmail.com');>> wrote:
>>
>>
>> I have 4.02.1 OCaml installed on my Mac. opam update; opam upgrade
>>
>
>  Sexplib 113.24.0, is only available for ocaml >= 4.02.3
> The same is true for all other core packages of this version and above.
>
>
>

[-- Attachment #2: Type: text/html, Size: 944 bytes --]