ABI compatibility between versions

ABI compatibility between versions

[Alexander Revin]

Hi all,

I know this have been briefly discussed here before, but still: does
musl guarantee in some way that executable/library compiled against
one musl version will work with  another (for example, 1.18 and 1.21)
?

I remember there were concerns against embedding versioning
information in musl like glibc does, but is there a way to somehow
ensure the stability between releases?

Thanks!
Alex

Re: ABI compatibility between versions

[Rich Felker]

On Tue, Feb 26, 2019 at 12:18:06AM +0100, Alexander Revin wrote:
> Hi all,
> 
> I know this have been briefly discussed here before, but still: does
> musl guarantee in some way that executable/library compiled against
> one musl version will work with  another (for example, 1.18 and 1.21)
> ?
> 
> I remember there were concerns against embedding versioning
> information in musl like glibc does, but is there a way to somehow
> ensure the stability between releases?

It guarantees that there is no ABI mismatch. That's not entirely the
same as guaranteeing that it will work. If the application was relying
on a bug in an old version to function, or was poking at some
accidentally-exposed libc internals not defined as a public interface,
it's possible that updating libc.so will expose this bug in the
application.

This is different from the glibc approach, which is to use symbol
versioning to attempt to retain "bug-compatibility" with the version
of glibc the application was linked with. Such a system forces new
application binaries that want to be able to run on systems with old
glibc to link against the old glibc, and thereby get the buggy
behaviors even if they're running on a system without the bugs. Myself
and most of the musl community I'm aware of consider this entirely
unreasonable, and that's why musl doesn't do it.

Rich

Re: ABI compatibility between versions

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2019-02-25 19:33:53 -0500]:
> On Tue, Feb 26, 2019 at 12:18:06AM +0100, Alexander Revin wrote:
> > Hi all,
> > 
> > I know this have been briefly discussed here before, but still: does
> > musl guarantee in some way that executable/library compiled against
> > one musl version will work with  another (for example, 1.18 and 1.21)
> > ?
> > 
> > I remember there were concerns against embedding versioning
> > information in musl like glibc does, but is there a way to somehow
> > ensure the stability between releases?
> 
> It guarantees that there is no ABI mismatch. That's not entirely the
> same as guaranteeing that it will work. If the application was relying
> on a bug in an old version to function, or was poking at some
> accidentally-exposed libc internals not defined as a public interface,
> it's possible that updating libc.so will expose this bug in the
> application.
> 
> This is different from the glibc approach, which is to use symbol
> versioning to attempt to retain "bug-compatibility" with the version
> of glibc the application was linked with. Such a system forces new
> application binaries that want to be able to run on systems with old
> glibc to link against the old glibc, and thereby get the buggy
> behaviors even if they're running on a system without the bugs. Myself
> and most of the musl community I'm aware of consider this entirely
> unreasonable, and that's why musl doesn't do it.

i just want to add that glibc makes a distinction as well
between public api contract and implementation internals
and it does not aim to be compatible with anything that
depends on internals (unless there is a strong reason
to do so) so a binary may not work across glibc versions.

other than the bug compatibility, a difference between the
two approaches is that glibc may do certain abi breaking
changes while keeping old binaries work, that musl cant do.
but for this reason a binary compiled against a new version
of glibc is unlikely to work with an older version (which
is why anybody who wants to distribute a binary that works
across different linux distros, compiles against a very old
version of glibc, which of course means lots of old bugs)
while for musl such breakage is much more rare (happens
when a new symbol is introduced and the binary uses that).

Re: ABI compatibility between versions

[Alexander Revin]

Thanks for your answers.

> but for this reason a binary compiled against a new version
> of glibc is unlikely to work with an older version (which
> is why anybody who wants to distribute a binary that works
> across different linux distros, compiles against a very old
> version of glibc, which of course means lots of old bugs)
> while for musl such breakage is much more rare (happens
> when a new symbol is introduced and the binary uses that).

So it generally similar to glibc approach – link against old musl,
which doesn't expose new symbols?

I'm asking this because I'm investigating efforts required to bring
Python native modules support to musl (at the present moment it's
impossible to install any Python native module on musl system without
recompiling) – discussion is here:
https://mail.python.org/archives/list/distutils-sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/

On Tue, Feb 26, 2019 at 10:58 AM Szabolcs Nagy <nsz@port70.net> wrote:
>
> * Rich Felker <dalias@libc.org> [2019-02-25 19:33:53 -0500]:
> > On Tue, Feb 26, 2019 at 12:18:06AM +0100, Alexander Revin wrote:
> > > Hi all,
> > >
> > > I know this have been briefly discussed here before, but still: does
> > > musl guarantee in some way that executable/library compiled against
> > > one musl version will work with  another (for example, 1.18 and 1.21)
> > > ?
> > >
> > > I remember there were concerns against embedding versioning
> > > information in musl like glibc does, but is there a way to somehow
> > > ensure the stability between releases?
> >
> > It guarantees that there is no ABI mismatch. That's not entirely the
> > same as guaranteeing that it will work. If the application was relying
> > on a bug in an old version to function, or was poking at some
> > accidentally-exposed libc internals not defined as a public interface,
> > it's possible that updating libc.so will expose this bug in the
> > application.
> >
> > This is different from the glibc approach, which is to use symbol
> > versioning to attempt to retain "bug-compatibility" with the version
> > of glibc the application was linked with. Such a system forces new
> > application binaries that want to be able to run on systems with old
> > glibc to link against the old glibc, and thereby get the buggy
> > behaviors even if they're running on a system without the bugs. Myself
> > and most of the musl community I'm aware of consider this entirely
> > unreasonable, and that's why musl doesn't do it.
>
> i just want to add that glibc makes a distinction as well
> between public api contract and implementation internals
> and it does not aim to be compatible with anything that
> depends on internals (unless there is a strong reason
> to do so) so a binary may not work across glibc versions.
>
> other than the bug compatibility, a difference between the
> two approaches is that glibc may do certain abi breaking
> changes while keeping old binaries work, that musl cant do.
> but for this reason a binary compiled against a new version
> of glibc is unlikely to work with an older version (which
> is why anybody who wants to distribute a binary that works
> across different linux distros, compiles against a very old
> version of glibc, which of course means lots of old bugs)
> while for musl such breakage is much more rare (happens
> when a new symbol is introduced and the binary uses that).

Re: ABI compatibility between versions

[Rich Felker]

On Tue, Feb 26, 2019 at 12:28:31PM +0100, Alexander Revin wrote:
> Thanks for your answers.
> 
> > but for this reason a binary compiled against a new version
> > of glibc is unlikely to work with an older version (which
> > is why anybody who wants to distribute a binary that works
> > across different linux distros, compiles against a very old
> > version of glibc, which of course means lots of old bugs)
> > while for musl such breakage is much more rare (happens
> > when a new symbol is introduced and the binary uses that).
> 
> So it generally similar to glibc approach – link against old musl,
> which doesn't expose new symbols?

This works but isn't necessarily needed. As long as your application
does not use any symbols that were introduced in a newer musl, it will
run with an older one, subject to any bugs the older one might have.
If configure is detecting and causing the program's build process to
link to new symbols in the newer musl, and you don't want to depend on
that, you can usually override the detections with configure variables
on the configure command line or in an explicit config.cache file, or
equivalent for other non-autoconf-based build systems.

> I'm asking this because I'm investigating efforts required to bring
> Python native modules support to musl (at the present moment it's
> impossible to install any Python native module on musl system without
> recompiling) – discussion is here:
> https://mail.python.org/archives/list/distutils-sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/
> 
> On Tue, Feb 26, 2019 at 10:58 AM Szabolcs Nagy <nsz@port70.net> wrote:
> >
> > * Rich Felker <dalias@libc.org> [2019-02-25 19:33:53 -0500]:
> > > On Tue, Feb 26, 2019 at 12:18:06AM +0100, Alexander Revin wrote:
> > > > Hi all,
> > > >
> > > > I know this have been briefly discussed here before, but still: does
> > > > musl guarantee in some way that executable/library compiled against
> > > > one musl version will work with  another (for example, 1.18 and 1.21)
> > > > ?
> > > >
> > > > I remember there were concerns against embedding versioning
> > > > information in musl like glibc does, but is there a way to somehow
> > > > ensure the stability between releases?
> > >
> > > It guarantees that there is no ABI mismatch. That's not entirely the
> > > same as guaranteeing that it will work. If the application was relying
> > > on a bug in an old version to function, or was poking at some
> > > accidentally-exposed libc internals not defined as a public interface,
> > > it's possible that updating libc.so will expose this bug in the
> > > application.
> > >
> > > This is different from the glibc approach, which is to use symbol
> > > versioning to attempt to retain "bug-compatibility" with the version
> > > of glibc the application was linked with. Such a system forces new
> > > application binaries that want to be able to run on systems with old
> > > glibc to link against the old glibc, and thereby get the buggy
> > > behaviors even if they're running on a system without the bugs. Myself
> > > and most of the musl community I'm aware of consider this entirely
> > > unreasonable, and that's why musl doesn't do it.
> >
> > i just want to add that glibc makes a distinction as well
> > between public api contract and implementation internals
> > and it does not aim to be compatible with anything that
> > depends on internals (unless there is a strong reason
> > to do so) so a binary may not work across glibc versions.
> >
> > other than the bug compatibility, a difference between the
> > two approaches is that glibc may do certain abi breaking
> > changes while keeping old binaries work, that musl cant do.
> > but for this reason a binary compiled against a new version
> > of glibc is unlikely to work with an older version (which
> > is why anybody who wants to distribute a binary that works
> > across different linux distros, compiles against a very old
> > version of glibc, which of course means lots of old bugs)
> > while for musl such breakage is much more rare (happens
> > when a new symbol is introduced and the binary uses that).

Re: ABI compatibility between versions

[Alexander Revin]

Thanks! I think that should be enough for Python problem

On Tue, Feb 26, 2019 at 4:11 PM Rich Felker <dalias@libc.org> wrote:
>
> On Tue, Feb 26, 2019 at 12:28:31PM +0100, Alexander Revin wrote:
> > Thanks for your answers.
> >
> > > but for this reason a binary compiled against a new version
> > > of glibc is unlikely to work with an older version (which
> > > is why anybody who wants to distribute a binary that works
> > > across different linux distros, compiles against a very old
> > > version of glibc, which of course means lots of old bugs)
> > > while for musl such breakage is much more rare (happens
> > > when a new symbol is introduced and the binary uses that).
> >
> > So it generally similar to glibc approach – link against old musl,
> > which doesn't expose new symbols?
>
> This works but isn't necessarily needed. As long as your application
> does not use any symbols that were introduced in a newer musl, it will
> run with an older one, subject to any bugs the older one might have.
> If configure is detecting and causing the program's build process to
> link to new symbols in the newer musl, and you don't want to depend on
> that, you can usually override the detections with configure variables
> on the configure command line or in an explicit config.cache file, or
> equivalent for other non-autoconf-based build systems.
>
> > I'm asking this because I'm investigating efforts required to bring
> > Python native modules support to musl (at the present moment it's
> > impossible to install any Python native module on musl system without
> > recompiling) – discussion is here:
> > https://mail.python.org/archives/list/distutils-sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/
> >
> > On Tue, Feb 26, 2019 at 10:58 AM Szabolcs Nagy <nsz@port70.net> wrote:
> > >
> > > * Rich Felker <dalias@libc.org> [2019-02-25 19:33:53 -0500]:
> > > > On Tue, Feb 26, 2019 at 12:18:06AM +0100, Alexander Revin wrote:
> > > > > Hi all,
> > > > >
> > > > > I know this have been briefly discussed here before, but still: does
> > > > > musl guarantee in some way that executable/library compiled against
> > > > > one musl version will work with  another (for example, 1.18 and 1.21)
> > > > > ?
> > > > >
> > > > > I remember there were concerns against embedding versioning
> > > > > information in musl like glibc does, but is there a way to somehow
> > > > > ensure the stability between releases?
> > > >
> > > > It guarantees that there is no ABI mismatch. That's not entirely the
> > > > same as guaranteeing that it will work. If the application was relying
> > > > on a bug in an old version to function, or was poking at some
> > > > accidentally-exposed libc internals not defined as a public interface,
> > > > it's possible that updating libc.so will expose this bug in the
> > > > application.
> > > >
> > > > This is different from the glibc approach, which is to use symbol
> > > > versioning to attempt to retain "bug-compatibility" with the version
> > > > of glibc the application was linked with. Such a system forces new
> > > > application binaries that want to be able to run on systems with old
> > > > glibc to link against the old glibc, and thereby get the buggy
> > > > behaviors even if they're running on a system without the bugs. Myself
> > > > and most of the musl community I'm aware of consider this entirely
> > > > unreasonable, and that's why musl doesn't do it.
> > >
> > > i just want to add that glibc makes a distinction as well
> > > between public api contract and implementation internals
> > > and it does not aim to be compatible with anything that
> > > depends on internals (unless there is a strong reason
> > > to do so) so a binary may not work across glibc versions.
> > >
> > > other than the bug compatibility, a difference between the
> > > two approaches is that glibc may do certain abi breaking
> > > changes while keeping old binaries work, that musl cant do.
> > > but for this reason a binary compiled against a new version
> > > of glibc is unlikely to work with an older version (which
> > > is why anybody who wants to distribute a binary that works
> > > across different linux distros, compiles against a very old
> > > version of glibc, which of course means lots of old bugs)
> > > while for musl such breakage is much more rare (happens
> > > when a new symbol is introduced and the binary uses that).

Re: ABI compatibility between versions

[u-uy74]

On Tue, Feb 26, 2019 at 10:58:38AM +0100, Szabolcs Nagy wrote:
> other than the bug compatibility, a difference between the
> two approaches is that glibc may do certain abi breaking
> changes while keeping old binaries work, that musl cant do.

I feel this statement is unfair to musl.

AFAICS symbol versioning does not allow "breaking" the ABI, but
*extending* it with new entry points (made to look like the old ones in
the API, which makes the old ones inaccessible via the API).

This is also exactly what musl can do, extend the ABI.
The difference is how to reflect those changes in the API.

Keeping the old API on top of a subset of the occasionally extended ABI
is what glibc does and musl does not.

Regarding old binaries API stability is irrelevant.

Rune