From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 15814 invoked from network); 8 Jan 2024 17:17:05 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 8 Jan 2024 17:17:05 -0000 Received: (qmail 10192 invoked by uid 550); 8 Jan 2024 17:15:34 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 10157 invoked from network); 8 Jan 2024 17:15:34 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=s31663417; t=1704734211; x=1705339011; i=nullplan@gmx.net; bh=10w0rOkg1KNawLUERnpt+F92DJDLQkpkre1Pera9EB4=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:References: In-Reply-To; b=VXphPf8OGiIaxEz4HjD5idZS2q/HgHtbJprfI1cNFPFC5yP8NE6ERhZmJh0ktQg1 3Cu+nsCLWTO9/GjFLzQ42JTTcPV4Yu3qw4puVRmVAeFXwLpV8LV5GlvMy9xPkKLEp AAYmIxk8gIvyc4ZzbGjJbhTsx3AiU96pGc615bo4a79kLvWWUqlST9cAlHvcNnf3e F9FQBqYuou0C2sCWjsYuwkLu/Xm2xXrZXhAV6d/1orqBNMgjukvydn0wa2Gfdv6bo nIHro9MVHSmjOWEeV7iSuhfvWjAzGt7bMKBZqZKhkE4NXORHMRQVDQxhDv++W3fSD OYwQSOGMORkZgCvB8w== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Date: Mon, 8 Jan 2024 18:16:49 +0100 From: Markus Wichmann To: musl@lists.openwall.com Cc: Patrick Rauscher Message-ID: References: <2dcdd7436bd07e0b020644fac1b3a06256db8673.camel@prauscher.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <2dcdd7436bd07e0b020644fac1b3a06256db8673.camel@prauscher.de> X-Provags-ID: V03:K1:XnkgqGD5SqrE4lwWi4lecXg/6+Z4A4jyzM9eK/DaHT1dL5N+A19 p62v8jzOREr/HJqBuC26mQ5IfI1sBXf/dUEWaixtxySlp0zpSpUxBhR7zOxLgBlSWujvAGN 8f7IkU/iQiFsOpPEIDJLAs2TaUgPuDZ5CWZ1Hd9+ErMA0MHC2beQiXADKBhSO4/45iASJCz LGM4NfYv4SfvjTSppSVDg== UI-OutboundReport: notjunk:1;M01:P0:RjbTokPMDm0=;lSUpvhLz3ApjQxqXAK+9DLEz0qs Gtoi86q8S80myTnNbeIAzei2jDVYi08tINeYVAdfl5AcaIPQz4VGxTGTpkTh2hEiXw/4ia/4E fOkWgUIXGzpsUHDDSzmQe7oFcpahIa4de3n9KHKKjcrVENjiW25HQGbkdxPd/JmtS8Usq3Yn1 Gjhg+NKaKQIa3ijZu//bgTZiavN7+g53/vLssh0C3lIbYUzF6Iser476LxyHxSickix+S4f+y w0Mg9f2hgnohU2Sh4yWWUJCBxPszlVpjR8yRzCw/tDpvpwaXljzSh079jyS5VjikgQale9bPa 5QcSTbsxLBYR5etqg/6NIioayRBWNiC5jN83Fd250+Ri7lN7szPEu87n3Yg/cQy7P21X80Q6S uqnmoT1AiDd/+WqKKRe4f8/x2A5dMgOyU5CBtlc6HeQTamIsTXF/Xu8XOYr2IvjvrMMaDTkqn do0py/3UiGoBQ+MwzEtIH7ZrpfRL2xGaQqaq2AYc355xpLmx6KVzmUh4aMb9ni0ftFpSvWIWi 3mpCMswzR9AXsjyPGTGbwYP/RdgBQX9crcXw5rBdNHok72+91pomaSON9sSgM4qczwNHbnc/T +Pr3okuYlDuLROcc6LmvxamUhXcbycmV1JHEhI/JXhsnqbCFFILHBRU/uvvKlSDf9fQGXwe4N GrK++zD8+pV6c2i1cQS/27R6eG248XKctwXAjkM5gn2PF+F44yD+P8pXtrAsYzV3NH6W8m6Nc dNFh+zvjkFjGG9mM0MbwteL297ncJm9IN29MoeijDTE3Kq6MyD5tZQ+eleq68VetlCsdGrf98 AaX7VEK6V30Ek8G8uF+lHVSkcPuuBlriAncjl+ModywISaMCZXKYhrN1j1t+yzgoQLh6pm3fO QrjlAt+BxgeU8lON05w2PvYhtyCneyurMryCd/A/hnEKqbJheFgHpHHuYWlr9B+NJymcI+Un1 A91MWgBW5hxXzDXF+yDKIvgMN8InuvZgE3bkNr7tgz7O6NjV Subject: Re: [musl] ENOFILE after reaching SEM_NSEMS_MAX Am Mon, Jan 08, 2024 at 03:01:46PM +0100 schrieb Patrick Rauscher: > Hello everyone, > > in POSIX, the constant SEM_NSEMS_MAX is defined as 256 from some time. W= hile > glibc nowadays ignores the limit and will yield -1 when asked for it [1]= , musl > currently will return ENOFILE when asking for the 256th semaphore. > > This hit me (and obviously another person) when using python multiproces= sing > [2]: Jan H detected, that while allocating a large number of semaphores = (or > objects using at least one semaphore) works on Debian, it fails on alpin= e. > > Thanks to psykose on the IRC the issue could be identified to the differ= ent > libc. To make this finding less ephemeral than on the IRC log, I leave a= note > here. As sem_open works in the documented way, this is certainly no bug > report, rather a feature request if you will so. > > Due to my lack of C-knowledge I may only standby and marvel to further > discussion, but maybe someone can come up with ideas =F0=9F=99=82 > > Thanks for your time, > Patrick > > 1: https://sourceware.org/legacy-ml/libc-help/2012-02/msg00003.html > 2: https://stackoverflow.com/questions/77679957/python-multiprocessing-n= o-file-descriptors-available-error-inside-docker-alpine > I should probably explain what the limit is used for at the moment. POSIX requires that all successful calls to sem_open() with the same name argument have to return the same pointer and increase a refcount. Practically, this is only possible by having a list of all file-backed semaphores. The code goes through the file mapping code normally, then at the end checks if the semaphore was mapped already, and if so unmaps the new copy and returns the old one. With the limit in place, the memory for the semaphore map can be allocated once and never returned. Iterations on the map have a well-defined end, and there's a defined maximum run-time to all such loops. If the limit were no longer imposed, the semaphore map would have to become a data structure capable of growth, e.g. a list. But then you'd get all the negative effects of having a list vs. the array we currently have. Whatever data structure you choose, you basically always get worse performance than with a simple array. For the python example, I would ask whether this pattern of creating named semaphores with random names is really all that wise. What is the point? You have to transmit the name to the destination process somehow. This is not really what named semaphores are for. Mayhaps a shared memory would suit you better? Or foregoing semaphores in favor of another primitive altogether? Ciao, Markus