sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Jonathan Rajotte-Julien]

Hi all,

We are currently in the process of making sure that lttng [1] (linux tracer) run
smoothly on system using musl (Yocto, Alpine etc.). Most things work
fine. Still, we currently have tests that are failing due to an issue regarding
the reported number of configured processors on the system (__SC_NPROCESSORS_CONF).
Note that users of LTTng are also affected by this if they chose to modify the
sched affinity of their instrumented apps. This is relatively a big deal for us.

Long story short, we start an app with "taskset -c 0" and we need to allocate
data structure internally but using the number of configured processors not the
number of online processors. To do so we call sysconf(__SC_NPROCESSORS_CONF).
Slight problem: the value returned is the _SC_NPROCESSORS_ONLN value instead of
__SC_NPROCESSORS_CONF.

From src/conf/sysconf.c:196

 case JT_NPROCESSORS_CONF & 255:
 case JT_NPROCESSORS_ONLN & 255: ;
   unsigned char set[128] = {1};
   int i, cnt;
   __syscall(SYS_sched_getaffinity, 0, sizeof set, set);
   for (i=cnt=0; i<sizeof set; i++)
     for (; set[i]; set[i]&=set[i]-1, cnt++);
   return cnt;

A simple command line to show this:

  taskset -c 0 nproc --all

This is equivalent to asking sysconf(__SC_NPROCESSORS_CONF).

On 4 cpu system using glibc:

  $ taskset -c 0 nproc --all
  4

On the same system using musl:

  $ taskset -c 0 nproc --all
  1

And as for _SC_NPROCESSORS_ONLN, we can use:

  taskset -c 0 nproc

Using glibc:
  $ taskset -c 0 nproc
  1

Using musl:
  $ taskset -c 0 nproc
  1

No problem there.

Not sure where to go from there. We will probably end-up having a fallback for
musl that will probably consist of parsing /sys/ like glibc does.

Still we wanted to get your feedback on the matter.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Szabolcs Nagy]

* Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> [2019-03-15 17:02:02 -0400]:
> We are currently in the process of making sure that lttng [1] (linux tracer) run
> smoothly on system using musl (Yocto, Alpine etc.). Most things work
> fine. Still, we currently have tests that are failing due to an issue regarding
> the reported number of configured processors on the system (__SC_NPROCESSORS_CONF).
> Note that users of LTTng are also affected by this if they chose to modify the
> sched affinity of their instrumented apps. This is relatively a big deal for us.
> 
> Long story short, we start an app with "taskset -c 0" and we need to allocate
> data structure internally but using the number of configured processors not the
> number of online processors. To do so we call sysconf(__SC_NPROCESSORS_CONF).
> Slight problem: the value returned is the _SC_NPROCESSORS_ONLN value instead of
> __SC_NPROCESSORS_CONF.
...
> A simple command line to show this:
> 
>   taskset -c 0 nproc --all
> 
> This is equivalent to asking sysconf(__SC_NPROCESSORS_CONF).

the right way to check the sysconf from a shell is getconf

on glibc system
$ taskset -c 0 getconf -a |grep NPROC
_NPROCESSORS_CONF                  8
_NPROCESSORS_ONLN                  8

on musl
$ taskset -c 0 getconf -a |grep NPROC
_NPROCESSORS_CONF                  1
_NPROCESSORS_ONLN                  1

so both values differ (plain nproc returns the affinity number,
*_ONLN is all the cpus that the kernel schedules to, *_CONF
includes offline cpus that may be hotplugged)

these are documented linux extensions so i think musl should follow
the linux sysconf man page. (but the semantics is not entirely clear
e.g. there is /sys/devices/system/cpu/possible which can have larger
number than echo /sys/devices/system/cpu/cpu[0-9]* |wc -w which is
what glibc seems to be doing for *_CONF)

i think we need to know why does a process care if musl returns
the wrong number? or what are the valid uses of such a number?
(there are heterogeous systems like arm big-little, numa systems
with many sockets, containers, virtualization,.. how deep may a
user process need to go down in this rabbit hole?)

note that most of /sys/devices/system/cpu/* is documented under
Documentation/ABI/testing in linux, not in Documentation/ABI/stable
and the format is not detailed, and some apis (e.g. /proc/cpuinfo)
are known to be different on android (and grsec?) kernels it may
be unmounted during early boot or in chroots, so sysfs parsing is
only done when really necessary.

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Mathieu Desnoyers]

----- On Mar 15, 2019, at 10:25 PM, Szabolcs Nagy nsz@port70.net wrote:

> * Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> [2019-03-15
> 17:02:02 -0400]:
>> We are currently in the process of making sure that lttng [1] (linux tracer) run
>> smoothly on system using musl (Yocto, Alpine etc.). Most things work
>> fine. Still, we currently have tests that are failing due to an issue regarding
>> the reported number of configured processors on the system
>> (__SC_NPROCESSORS_CONF).
>> Note that users of LTTng are also affected by this if they chose to modify the
>> sched affinity of their instrumented apps. This is relatively a big deal for us.

[...]

> i think we need to know why does a process care if musl returns
> the wrong number? or what are the valid uses of such a number?
> (there are heterogeous systems like arm big-little, numa systems
> with many sockets, containers, virtualization,.. how deep may a
> user process need to go down in this rabbit hole?)

The use-case for LTTng-UST is as follows: it implements a ring buffer
which is allocated (taking NUMA affinity into account) and owned by a
centralized daemon (lttng-consumerd), passed to traced applications as
file descriptors over unix sockets, and then memory-mapped into each
application address space.

This ring buffer is a per-cpu buffer for performance reasons.

The way LTTng-UST handles cpu hotplug is to pre-allocate buffers for
all possible CPUs, hand them over to applications, so those applications
can always find an allocated buffer to write into for the current cpu
reported by sched_getcpu(), even after a cpu hotplug.

In comparison, the LTTng-modules Linux kernel tracer can do a much more
clever use of memory resources, because it has access to cpu hotplug
kernel modules APIs, which allows it to allocate buffers only for online
CPUs.

Unfortunately, there is no way to do this for user-space buffers that
I'm aware of other than lazily allocating the resources on first use,
and we cannot do this because allocation is performed by a centralized
daemon (not by the application being traced and doing the actual "first
use"). Moreover, as an overall design guide-line, we keep lazy allocation
to a minimum in LTTng-UST, so tracing does not cause unexpected
application latency.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Jonathan Rajotte-Julien]

> > A simple command line to show this:
> > 
> >   taskset -c 0 nproc --all
> > 
> > This is equivalent to asking sysconf(__SC_NPROCESSORS_CONF).
> 
> the right way to check the sysconf from a shell is getconf

It was only to provide an easy reproducer. But you are right in that nproc
does not expose the complete picture. Thanks for taking the time to reproduce
the base problem.

I mixed up the  _NPROCESSORS_ONLN result for glibc, it should have been 4 in the
previous email since there is 4 online cpu even if we have sched_affinity only
set for cpu0. (nproc was not proving my point for the _NPROCESSORS_ONLN value)

As you know, you can take a cpu offline easily, but we still need to account for
it in userspace tracing since it can be put back online (see Mathieu Desnoyers
answer in this thread).

  echo 0 > /sys/devices/system/cpu/cpu3/online

Now on a glibc system:

  $ taskset -c 0 getconf -a |grep NPROC
  _NPROCESSORS_CONF                  4
  _NPROCESSORS_ONLN                  3

This is why we use _NPROCESSORS_CONF and expect it to represent the complete
picture. We do not care much for _NPROCESSORS_ONLN or affinity. This is why the
use of "nproc --all" was sufficient for me (I was wrong).

> 
> on glibc system
> $ taskset -c 0 getconf -a |grep NPROC
> _NPROCESSORS_CONF                  8
> _NPROCESSORS_ONLN                  8
> 
> on musl
> $ taskset -c 0 getconf -a |grep NPROC
> _NPROCESSORS_CONF                  1
> _NPROCESSORS_ONLN                  1
> 
> so both values differ (plain nproc returns the affinity number,
> *_ONLN is all the cpus that the kernel schedules to, *_CONF
> includes offline cpus that may be hotplugged)
> 
> these are documented linux extensions so i think musl should follow
> the linux sysconf man page. (but the semantics is not entirely clear
> e.g. there is /sys/devices/system/cpu/possible which can have larger
> number than echo /sys/devices/system/cpu/cpu[0-9]* |wc -w which is
> what glibc seems to be doing for *_CONF)
> 
> i think we need to know why does a process care if musl returns
> the wrong number? or what are the valid uses of such a number?
> (there are heterogeous systems like arm big-little, numa systems
> with many sockets, containers, virtualization,.. how deep may a
> user process need to go down in this rabbit hole?)

I'll refer you to Mathieu Desnoyers answer regarding that. (same thread).
It should be approved shortly by a moderator.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Florian Weimer]

* Szabolcs Nagy:

> i think we need to know why does a process care if musl returns
> the wrong number? or what are the valid uses of such a number?

If the value is 1, the process might use different concurrency
algorithms.

This is why glibc returns 2 if the appropriate value is not available.

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Jonathan Rajotte-Julien]

Hi Szabolcs,

> i think we need to know why does a process care if musl returns
> the wrong number? or what are the valid uses of such a number?
> (there are heterogeous systems like arm big-little, numa systems
> with many sockets, containers, virtualization,.. how deep may a
> user process need to go down in this rabbit hole?)

Does the answers from Mathieu Desnoyers [1] and Florian Weimer [2] fit the bill?

[1] https://www.openwall.com/lists/musl/2019/03/16/3
[2] https://www.openwall.com/lists/musl/2019/03/19/1
> 
> note that most of /sys/devices/system/cpu/* is documented under
> Documentation/ABI/testing in linux, not in Documentation/ABI/stable
> and the format is not detailed, and some apis (e.g. /proc/cpuinfo)
> are known to be different on android (and grsec?) kernels it may
> be unmounted during early boot or in chroots, so sysfs parsing is
> only done when really necessary.

For what it's worth, uclibc and uclibc-ng seem to iterate over
/sys/devices/system/cpu/* and fallback on online calculation if necessary.

https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/unistd/sysconf.c#n102

In the mean time, we implemented a fallback similar to this when we do not "know"
the libc used (since musl does not come with __musl__, I read the reasons why,
no need to discuss this).

Not sure of the direction musl should take but I strongly believe that the
behaviour regarding _SC_NPROCESSORS_CONF is not the appropriate one.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Szabolcs Nagy]

* Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> [2019-03-26 12:23:34 -0400]:
> > i think we need to know why does a process care if musl returns
> > the wrong number? or what are the valid uses of such a number?
> > (there are heterogeous systems like arm big-little, numa systems
> > with many sockets, containers, virtualization,.. how deep may a
> > user process need to go down in this rabbit hole?)
> 
> Does the answers from Mathieu Desnoyers [1] and Florian Weimer [2] fit the bill?

yes

> 
> [1] https://www.openwall.com/lists/musl/2019/03/16/3
> [2] https://www.openwall.com/lists/musl/2019/03/19/1
> > 
> > note that most of /sys/devices/system/cpu/* is documented under
> > Documentation/ABI/testing in linux, not in Documentation/ABI/stable
> > and the format is not detailed, and some apis (e.g. /proc/cpuinfo)
> > are known to be different on android (and grsec?) kernels it may
> > be unmounted during early boot or in chroots, so sysfs parsing is
> > only done when really necessary.
> 
> For what it's worth, uclibc and uclibc-ng seem to iterate over
> /sys/devices/system/cpu/* and fallback on online calculation if necessary.
> 
> https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/unistd/sysconf.c#n102
> 
> In the mean time, we implemented a fallback similar to this when we do not "know"
> the libc used (since musl does not come with __musl__, I read the reasons why,
> no need to discuss this).
> 
> Not sure of the direction musl should take but I strongly believe that the
> behaviour regarding _SC_NPROCESSORS_CONF is not the appropriate one.

i agree that the current behaviour is not ideal, but
iterating over /sys/devices/system/cpu/cpu* may not
be correct either.. based on current linux api docs.

i don't understand why is that number different from the
cpu set in /sys/devices/system/cpu/possible

it seems any upper bound on the number of cpus would be
valid but it's not clear how to provide that guarantee.

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Mathieu Desnoyers]

----- On Mar 26, 2019, at 1:45 PM, Szabolcs Nagy nsz@port70.net wrote:

> * Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> [2019-03-26
> 12:23:34 -0400]:
>> > i think we need to know why does a process care if musl returns
>> > the wrong number? or what are the valid uses of such a number?
>> > (there are heterogeous systems like arm big-little, numa systems
>> > with many sockets, containers, virtualization,.. how deep may a
>> > user process need to go down in this rabbit hole?)
>> 
>> Does the answers from Mathieu Desnoyers [1] and Florian Weimer [2] fit the bill?
> 
> yes
> 
>> 
>> [1] https://www.openwall.com/lists/musl/2019/03/16/3
>> [2] https://www.openwall.com/lists/musl/2019/03/19/1
>> > 
>> > note that most of /sys/devices/system/cpu/* is documented under
>> > Documentation/ABI/testing in linux, not in Documentation/ABI/stable
>> > and the format is not detailed, and some apis (e.g. /proc/cpuinfo)
>> > are known to be different on android (and grsec?) kernels it may
>> > be unmounted during early boot or in chroots, so sysfs parsing is
>> > only done when really necessary.
>> 
>> For what it's worth, uclibc and uclibc-ng seem to iterate over
>> /sys/devices/system/cpu/* and fallback on online calculation if necessary.
>> 
>> https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/unistd/sysconf.c#n102
>> 
>> In the mean time, we implemented a fallback similar to this when we do not
>> "know"
>> the libc used (since musl does not come with __musl__, I read the reasons why,
>> no need to discuss this).
>> 
>> Not sure of the direction musl should take but I strongly believe that the
>> behaviour regarding _SC_NPROCESSORS_CONF is not the appropriate one.
> 
> i agree that the current behaviour is not ideal, but
> iterating over /sys/devices/system/cpu/cpu* may not
> be correct either.. based on current linux api docs.
> 
> i don't understand why is that number different from the
> cpu set in /sys/devices/system/cpu/possible

I suspect both iteration over /sys/devices/system/cpu/cpu* and
content of /sys/devices/system/cpu/possible should provide the
same result. However, looking at Linux
Documentation/ABI/testing/sysfs-devices-system-cpu ,
it appears that /sys/devices/system/cpu/possible was introduced
in December 2008, whereas /sys/devices/system/cpu/cpu#/ was there
pre-git history.

This could explain why glibc uses the iteration method.

Thoughts ?

Thanks,

Mathieu

> 
> it seems any upper bound on the number of cpus would be
> valid but it's not clear how to provide that guarantee.





-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Szabolcs Nagy]

* Mathieu Desnoyers <mathieu.desnoyers@efficios.com> [2019-03-26 14:01:08 -0400]:
> ----- On Mar 26, 2019, at 1:45 PM, Szabolcs Nagy nsz@port70.net wrote:
> > i agree that the current behaviour is not ideal, but
> > iterating over /sys/devices/system/cpu/cpu* may not
> > be correct either.. based on current linux api docs.
> > 
> > i don't understand why is that number different from the
> > cpu set in /sys/devices/system/cpu/possible
> 
> I suspect both iteration over /sys/devices/system/cpu/cpu* and
> content of /sys/devices/system/cpu/possible should provide the
> same result. However, looking at Linux
> Documentation/ABI/testing/sysfs-devices-system-cpu ,
> it appears that /sys/devices/system/cpu/possible was introduced
> in December 2008, whereas /sys/devices/system/cpu/cpu#/ was there
> pre-git history.
> 
> This could explain why glibc uses the iteration method.
> 
> Thoughts ?

as far as i can tell the cpu iteration method is valid,
and that directory list cannot change after boot (is this
guaranteed by the linux abi in the future?), so as long
as /sys is mounted we can get the number, but it's fairly
ugly.. does lttng have fallback code if sysconf returns -1?
if it does maybe musl should just do that (or somebody has
to write cancellation safe directory traversal code)

Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Mathieu Desnoyers]

----- On Apr 2, 2019, at 3:02 PM, Szabolcs Nagy nsz@port70.net wrote:

> * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> [2019-03-26 14:01:08
> -0400]:
>> ----- On Mar 26, 2019, at 1:45 PM, Szabolcs Nagy nsz@port70.net wrote:
>> > i agree that the current behaviour is not ideal, but
>> > iterating over /sys/devices/system/cpu/cpu* may not
>> > be correct either.. based on current linux api docs.
>> > 
>> > i don't understand why is that number different from the
>> > cpu set in /sys/devices/system/cpu/possible
>> 
>> I suspect both iteration over /sys/devices/system/cpu/cpu* and
>> content of /sys/devices/system/cpu/possible should provide the
>> same result. However, looking at Linux
>> Documentation/ABI/testing/sysfs-devices-system-cpu ,
>> it appears that /sys/devices/system/cpu/possible was introduced
>> in December 2008, whereas /sys/devices/system/cpu/cpu#/ was there
>> pre-git history.
>> 
>> This could explain why glibc uses the iteration method.
>> 
>> Thoughts ?
> 
> as far as i can tell the cpu iteration method is valid,
> and that directory list cannot change after boot (is this
> guaranteed by the linux abi in the future?), so as long
> as /sys is mounted we can get the number, but it's fairly
> ugly.. does lttng have fallback code if sysconf returns -1?
> if it does maybe musl should just do that (or somebody has
> to write cancellation safe directory traversal code)

Nope, the lttng-ust ring buffer won't be usable anyway. I
*think* we need to improve the error handling for that scenario
within our consumer daemon which is responsible for buffer
allocation. I don't think we properly error out if the internal
variable tracking the number of CPUs in the system stays at its
unitialized value of 0.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com