[discuss] Binary compatibility between Illumos distributions

[discuss] Binary compatibility between Illumos distributions

[Sad Clouds]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

Hi, if somebody is building binaries for Illumos x86_64 platform, is there a good binary compatibility between various Illumos distributions? I know they may use different packaging systems for additional software, but if we assume the binaries are only linked with the base OS libraries and built with the system compiler, then can the same set of binaries be safely distributed for different distributions?

Also, Solaris used to have very good ABI backward compatibility, however Illumos distributions don't seem to use major/minor version numbers, but instead use a date string, or some unique release number. So how does one track incompatible ABI changes, etc? OK it may be stated in the release notes, but ideally it would need to be scripted, i.e. make file or config script would parse the OS version from the uname command and adjust the build variables. Is there some common versioning scheme across all Illumos distributions, maybe via a different command or version file in /etc?

Thanks.
------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M77e70244c2b042a81f2c1990
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

[-- Attachment #2: Type: text/html, Size: 1815 bytes --]

Re: [discuss] Binary compatibility between Illumos distributions

[Peter Tribble]

[-- Attachment #1: Type: text/plain, Size: 4610 bytes --]

On Sun, Sep 29, 2024 at 8:26 AM Sad Clouds <cryintothebluesky@gmail.com>
wrote:

> Hi, if somebody is building binaries for Illumos x86_64 platform, is there
> a good binary compatibility between various Illumos distributions?
>

It's fairly good, with severe limitations.


> I know they may use different packaging systems for additional software,
> but if we assume the binaries are only linked with the base OS libraries
>

That's the first catch. Many applications end up being linked with
libraries that have a
relatively poor history of compatibility. And different distributions may
be shipping all sorts
of different versions. For example openssl might be 1.1.1 or 3.0 or 3.1 or
3.2. Interestingly,
while getting worse over time this is starting to become less of an issue
with runtimes
like go and rust where you often end with applications that are more
self-contained.


> and built with the system compiler,
>

That's the second catch. Different distributions have different compilers,
and may link
against libgcc_s and libcstd++. The basic advice there is that the binary
is only guaranteed
to run on a system with an equal or newer version of the gcc runtime (and
clang may
introduce other interesting constraints). There's an additional issue with
OpenIndiana
which ships multiple versions of the gcc runtime, hidden away.


> then can the same set of binaries be safely distributed for different
> distributions?
>

Again, within limitations.

And there's a third catch; there's a theoretical possibility that binaries
built on an LX
distribution might detect the presence of extra functionailty and implement
behaviour
based on that. In Tribblix I simply avoid this being an issue by building
all applications
on vanilla Tribblix rather than OmniTribblix, but the fact that the pkgsrc
binaries are built
on SmartOS and work everywhere indicates that this isn't much of a
practical issue,
and is a demonstration that compatibility does work (note that pkgsrc ships
its own
compilers and 3rd-party libraries, so avoiding the first two catches above).


> Also, Solaris used to have very good ABI backward compatibility, however
> Illumos distributions don't seem to use major/minor version numbers, but
> instead use a date string, or some unique release number. So how does one
> track incompatible ABI changes, etc?
>

What incompatible ABI changes?

The versioning of libc (which is the primary interfaces of interest - there
are relatively rare
changes to libsocket and libnsl, but the same rules apply) can be seen in
the source at

usr/src/lib/libc/port/mapfile-vers

so you can work out from the history there which interfaces were
introduced  in which
commits.

You can see the versioning history using 'pvs -vs'. Run it on libc and
you'll basically see
the same output as the mapfile-vers file in the source. Run it on any other
object
and you'll be able to see exactly which symbols are required, and which
library version
they correspond to. For example, on my system running 'pvs -vs' on bash and
the highest
version of libc that it requires is:

        libc.so.1 (ILLUMOS_0.29):
                getrandom;

So that my copy of bash will run on any illumos distribution that has
version 0.29 of
libc (which according to the git history came in with issue 9971 in
December 2018).
So, in this way, you can identify what the oldest compatible distribution
version is.

OK it may be stated in the release notes, but ideally it would need to be
> scripted, i.e. make file or config script would parse the OS version from
> the uname command and adjust the build variables. Is there some common
> versioning scheme across all Illumos distributions, maybe via a different
> command or version file in /etc?
>

So basically, not from anything other than the versioning of libc.so.

Ultimately, the solution is to find something adequately old and build on
that. I'm using
an old box with Tribblix m29 on it to generate my java builds, for example.
But java is
extremely precise and conservative in which library calls it uses. Other
software
(*cough* autoconf *cough*) may try and enable any functionality it finds on
the build system.

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M7094879799212154dff5c6b4
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

[-- Attachment #2: Type: text/html, Size: 7185 bytes --]

Re: [discuss] Binary compatibility between Illumos distributions

[Sad Clouds]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

OK thanks for the info. Maybe ABI was the wrong word to use, I guess I was referring to a mixture of ABI and API changes that may occur during OS major version upgrade. To give you a specific example, some of the software I work on is using thread affinity APIs to bind threads to specific CPUs. Solaris < 11.2 and Illumos only support processor_bind()which has a limitation where a thread can be bound to only one CPU, where Solaris >= 11.2 also supports processor_affinity() where a thread can be bound to multiple CPUs. I quite dislike GNU autoconf scripts and tend to extract OS name and major/minor version via uname commands in a makefile and then expose various C macros to work around API differences. So for OS name, Illumos reports SUNOS which is good, but then major/minor versions could be anything that a particular distro decided to use. Since they all fork their OS code from Illumos, I was looking for some common version number to identify which Illumos release their build was based on. For example, /etc/illumos-release file or something similar. In the future, if Illumos happens to implement processor_affinity() or some other new feature, it can cut down on the number of OS version probes one has to make.
------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M97e4498022e542b5ef540560
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

[-- Attachment #2: Type: text/html, Size: 2430 bytes --]

Re: [discuss] Binary compatibility between Illumos distributions

[Joshua M. Clulow via illumos-discuss]

On Sun, 29 Sept 2024 at 12:16, Sad Clouds <cryintothebluesky@gmail.com> wrote:
> OK thanks for the info. Maybe ABI was the wrong word to use, I guess I was referring to a mixture of ABI and API changes that may occur during OS major version upgrade. To give you a specific example, some of the software I work on is using thread affinity APIs to bind threads to specific CPUs. Solaris < 11.2 and Illumos only support processor_bind()which has a limitation where a thread can be bound to only one CPU, where Solaris >= 11.2 also supports processor_affinity() where a thread can be bound to multiple CPUs. I quite dislike GNU autoconf scripts and tend to extract OS name and major/minor version via uname commands in a makefile and then expose various C macros to work around API differences.

I would strongly advise feature detection instead of using versions.
I don't think anybody particularly enjoys autoconf, but in this area
they are often doing the right thing: if you care about the presence
of a specific interface, you should use some feature detection
mechanism to check for it.


Cheers.

-- 
Joshua M. Clulow
http://blog.sysmgr.org

------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M5373e902a4e28bd922dbb21a
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

Re: [discuss] Binary compatibility between Illumos distributions

[Sad Clouds]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]

Over the years I got frustrated with autoconf.

It is quite slow, especially on old machines like Sun Ultra 10. It could take several minutes to run autoconf scripts and then several seconds to compile some small open source package. If you are building large number of packages from something like pkgsrc, the issue is magnified significantly. Even on fast machines with many CPUs, autoconf inhibits parallelization, as it runs its tests sequentially and then many packages repeat the same tests over and over again, it is really bonkers. The issues are quite noticeable when bulk building many packages. 

It has issues correctly detecting features with cross compilers, which is why many packages cannot be cross compiled. For example, with NetBSD you can cross compile base OS on Linux, Solaris, BSD, etc. You could use the same cross compiler to cross compile pkgsrc for many other architectures, but unfortunately many packages just won't build this way.

So I try and keep it simple. I stick to POSIX and use OS name and version for specific cases where I need to resolve portability issues. I mostly link against base OS libraries, so don't need to worry too much about testing for 3rd party software.


------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-Mbad13a6d17f4050691a54180
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

[-- Attachment #2: Type: text/html, Size: 2104 bytes --]

Re: [discuss] Binary compatibility between Illumos distributions

[Bob Friesenhahn]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

On Sun, 29 Sep 2024, Sad Clouds wrote:

> Over the years I got frustrated with autoconf.
>
> It is quite slow, especially on old machines like Sun Ultra 10. It 
> could take several minutes to run autoconf scripts and then several 
> seconds to compile some small open source package. If you are 
> building large number of packages from something like pkgsrc, the 
> issue is magnified significantly. Even on fast machines with many 
> CPUs, autoconf inhibits parallelization, as it runs its tests 
> sequentially and then many packages repeat the same tests over and 
> over again, it is really bonkers. The issues are quite noticeable 
> when bulk building many packages. 

The Sun Ultra 10 is indeed very old.  But my early experiences with 
Autoconf started in 1993 on a SPARCclassic "server" (running Solaris 
2.1), which is much slower than that. :-)

In order to speed autoconf scripts for tests which always produce the 
same answer, one can create a 'config.site' script with cached 
results, and refer to it with the CONFIG_SITE environment variable. 
If you are bulk-building many packages using similar configuration 
options, this will result in a huge speed-up.

A good thing about autoconf is that it almost always produces the 
correct answer, and packages depending on autoconf+automake+libtool 
are extremely consistent and predictable.  I have found that projects 
based on CMake are not nearly as consistent and predictable.

Bob
-- 
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt
------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M9e9aa7d627e7d2c4235e91b0
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

Re: [discuss] Binary compatibility between Illumos distributions

[Sad Clouds]

On Mon, 30 Sep 2024 08:01:04 -0500 (CDT)
Bob Friesenhahn <bfriesen@simple.dallas.tx.us> wrote:

> A good thing about autoconf is that it almost always produces the 
> correct answer, and packages depending on autoconf+automake+libtool 
> are extremely consistent and predictable.  I have found that projects 
> based on CMake are not nearly as consistent and predictable.

That may be true, however over the years I mostly used NetBSD pkgsrc to
build packages on Solaris and the experience has not always been
positive, even with software which used GNU autotools. Not only because
of the tools, but due to how the software was designed and implemented.
Autotools do not automagically make the software portable and the
usual approach of "if it works on Linux then it is good enough for us"
did not help.

These days there are a lot of different build tools and they all have
issues in various areas and none of them are perfect. I think GNU
autotools were designed at a time when Unix standards were still
evolving and there was a lot of fragmentation. However now it is much
simpler, POSIX is the dominant standard and there is a finite
combination of OS versions, compilers and linkers.

I avoid autotools and similar by following these rules:

1. Do not test for every single feature. Instead assume POSIX
compatibility and work around specific portability issues. OS name and
major/minor versions should be enough for this.

2. Use plain GNU make and create your own make files. There is no need
for portable make files, because GNU make can be built on any POSIX OS
with minimum effort.

3. Use make variables to pass specific compiler and linker options
during build. Most platforms use either GCC or Clang, but some obsolete
compilers can be easily added when needed.

4. The most important rule - do not release anything for any untested
platforms, there is a good chance the software may not work properly.
I don't have Solaris 8 or 9 to test, so they are not supported until
somebody really needs it. It is our responsibility to make sure software
works correctly. The end users should never suffer the indignity of
broken builds, runtime bugs or crashes.

------------------------------------------
illumos: illumos-discuss
Permalink: https://illumos.topicbox.com/groups/discuss/Te614dba5c9d949ba-M4afdc1b290b77c6fff49424a
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription