Re: broadcast fd

[Rich Felker <dalias@libc.org>]

On Tue, Oct 15, 2019 at 10:04:47AM -0700, Leonid Shamis wrote:
> Hi Folks,
> 
> TLDR:
> 
> How do I broadcast a notification across processes, without a broker?
> 
> Long story:
> 
> I'm building an ipc-over-shm library with some unusual restrictions:
> * There are multiple channels of communication (each in it's own shm).
> * Each channel contains non-consumable resources (not a message queue).
> * Multiple processes can produce resources on the same channel.
> * Multiple consumers can access/use the resources.
> * Resources are versioned and (some) old versions are kept available.
> * Spurious wakeups are ok, but missing notifications (starvation) is not.
> * The library must be robust (in the mutex sense), meaning a process may
> crash without taking down the rest of the system.
> 
> I'm trying to figure out how to notify a process that a new resource is
> available on a channel.
> 
> Right now, my library puts (the equivalent of) a robust conditional
> variable in shared memory, for each channel.
> Each process must spawn a thread per-channel, just to await the cv
> broadcast.
> 
> I'd ideally like to use a single thread with something like an epoll.
> For this, I'd need to get an fd (maybe eventfd) per channel into the hands
> of every process that wants to connect to a channel.
> fds, as I understand, cannot be shared via shared memory, only pipes, but
> I'm required to be brokerless.
> futex_fd are very deprecated.
> 
> Am I missing another approach?

If you want the events associated with the shared memory objects, and
you have file descriptors for these objects (they're POSIX shm or
hand-rolled tmpfs shm, as opposed to sysvipc), perhaps there's a way
to convey the event availability through the fd. For example is there
a way you could use inotify to do it?

Otherwise I think you already have the best way. Keep in mind the
cv-waiting thread can be made tiny if you use a minimal stack size,
run with signals blocked, and just have it convert cv events to
self-pipe/eventfd/etc.

Rich