Re: sysconf(_SC_NPROCESSORS_CONF) returns the wrong value

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

----- On Mar 15, 2019, at 10:25 PM, Szabolcs Nagy nsz@port70.net wrote:

> * Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com> [2019-03-15
> 17:02:02 -0400]:
>> We are currently in the process of making sure that lttng [1] (linux tracer) run
>> smoothly on system using musl (Yocto, Alpine etc.). Most things work
>> fine. Still, we currently have tests that are failing due to an issue regarding
>> the reported number of configured processors on the system
>> (__SC_NPROCESSORS_CONF).
>> Note that users of LTTng are also affected by this if they chose to modify the
>> sched affinity of their instrumented apps. This is relatively a big deal for us.

[...]

> i think we need to know why does a process care if musl returns
> the wrong number? or what are the valid uses of such a number?
> (there are heterogeous systems like arm big-little, numa systems
> with many sockets, containers, virtualization,.. how deep may a
> user process need to go down in this rabbit hole?)

The use-case for LTTng-UST is as follows: it implements a ring buffer
which is allocated (taking NUMA affinity into account) and owned by a
centralized daemon (lttng-consumerd), passed to traced applications as
file descriptors over unix sockets, and then memory-mapped into each
application address space.

This ring buffer is a per-cpu buffer for performance reasons.

The way LTTng-UST handles cpu hotplug is to pre-allocate buffers for
all possible CPUs, hand them over to applications, so those applications
can always find an allocated buffer to write into for the current cpu
reported by sched_getcpu(), even after a cpu hotplug.

In comparison, the LTTng-modules Linux kernel tracer can do a much more
clever use of memory resources, because it has access to cpu hotplug
kernel modules APIs, which allows it to allocate buffers only for online
CPUs.

Unfortunately, there is no way to do this for user-space buffers that
I'm aware of other than lazily allocating the resources on first use,
and we cannot do this because allocation is performed by a centralized
daemon (not by the application being traced and doing the actual "first
use"). Moreover, as an overall design guide-line, we keep lazy allocation
to a minimum in LTTng-UST, so tracing does not cause unexpected
application latency.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com