From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/6481 Path: news.gmane.org!not-for-mail From: Timo Teras Newsgroups: gmane.linux.lib.musl.general Subject: Re: Nonstandard functions with callbacks Date: Tue, 11 Nov 2014 09:22:53 +0200 Message-ID: <20141111092253.15b592df@vostro> References: <20141111045046.GA25461@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1415690641 28636 80.91.229.3 (11 Nov 2014 07:24:01 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 11 Nov 2014 07:24:01 +0000 (UTC) Cc: musl@lists.openwall.com To: Rich Felker Original-X-From: musl-return-6494-gllmg-musl=m.gmane.org@lists.openwall.com Tue Nov 11 08:23:54 2014 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1Xo5nq-0002Ra-F8 for gllmg-musl@m.gmane.org; Tue, 11 Nov 2014 08:23:50 +0100 Original-Received: (qmail 23947 invoked by uid 550); 11 Nov 2014 07:23:48 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 23939 invoked from network); 11 Nov 2014 07:23:48 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=/yo7sXL8JV+zybrvYWZQVcr8FulwJ1GJl8ppFjmo2QE=; b=JsK7EwCOK5yRylfxR5Pjgm8b+9JifcFC2E3v3DykWjeVU8WoDawg8i+wCNHtusxILK BAWVoJPExLctAlmYDCQdVUMUNxwBaIp1oywfa/36PnqVB0tDxyCf1kzSUWWB4nBoToaa eA6sTbrt+RpYcAJCPfaWh/h8S9XnTAwglaF/eLLJ6Xokp6iIEy8cObHStFiDMKFon5yN jJCQH+VIFcLECYA88mFfbdFGXXLyF0mkCkAh7dwp5ZgekQcTe2WwGumcpxrjLMtKGTCe 6x7gTbVrN4Nc4Svam5vGQcVrbKkgZVUiryJGYCPTlQ7bt+PqcqF0ir3SwTf3WJN9Rt5q +RyA== X-Received: by 10.152.43.229 with SMTP id z5mr7522984lal.86.1415690616970; Mon, 10 Nov 2014 23:23:36 -0800 (PST) Original-Sender: =?UTF-8?Q?Timo_Ter=C3=A4s?= In-Reply-To: <20141111045046.GA25461@brightrain.aerifal.cx> X-Mailer: Claws Mail 3.10.1 (GTK+ 2.24.23; x86_64-alpine-linux-musl) Xref: news.gmane.org gmane.linux.lib.musl.general:6481 Archived-At: On Mon, 10 Nov 2014 23:50:46 -0500 Rich Felker wrote: > The topic of fopencookie has just come up on #musl again in regards to > Asterisk's dependence on it (or the similar funopen function), and in > response I'd like to offer a few ideas on this kind of function in > general. > > Supporting nonstandard functions is always a potential pitfall. Unlike > functions with a rigorous or semi-rigorous specification in one or > more standards documents, they inevitably have all sorts of > underspecified or unspecified corner cases that some software ends up > depending on. And when they come from a single origin (e.g. glibc) > rather than various historical systems that all had their own quirks, > it's arguably reasonable for applications to expect the exact behavior > of the original (e.g. glibc) implementation. Understand. Especially for musl which aims to be mostly ABI compatible. Apparently both fopencookie() and funopen() had ABI issues. Most horribly fopencookie() changed the callback pointer struct layout/contents. funopen() seems to have some versions with fpos_t for seek; and others use off_t. NetBSD current seems to have funopen2() with alternative signatures for the callbacks using ssize_t/size_t for read/write instead of int. So yeah, it's a royal mess. > In this regard, functions that take a caller-provided callback > function are just about the worst possible. The documentation (e.g. > man pages or glibc manual) for the original version rarely specifies > constraints on what the callback can do, such as: > > - Does it have to return? > - Can it call longjmp? > - What happens if it causes pthread cancellation to be acted upon? > - Can it change the floating point environment? > - Does it run in the same thread as the caller? > - Does it need to take special precautions to avoid deadlock? > - What reentrancy requirements does it have? > - Are there specific standard library functions it can't call? > - Can it unwind or backtrace out of the callback context? > - Etc. >[snip] > The fopencookie situation is fairly similar. The callbacks to provide > a custom FILE type would run in a context with the relevant FILE > locked. It's likely that the underlying reads or writes performed on > the 'cookie' would be somewhat different under musl than under glibc > due to musl's readv/writev type model for stdio buffering, and these > differences could potentially break applications' assumptions about > how the underlying operations would be performed. Such assumptions in > turn may be valid if glibc is considered 'authoritative' for the > fopencookie function. There are a number of other considerations too > with regard to the FILE locking. In a naive fopencookie implementation > for musl, the callbacks would run with the FILE lock potentially held > in musl's light-locking mode rather than with a true recursive lock, > meaning that strange things could happen if the callback tries to > perform any operation on its own FILE. Avoiding that seems like it > would require some complex dance to 'upgrade' the lock type, which in > turn might impose long-term costs on maintaining the FILE locking > system. Generally applications seem to write things to work with funopen() or fopencookie(); so they do not do too many assumptions. But yes, having identical emulation seems hard and unfeasible to accomplish. Given, the vagueness of the situation and the above mentioned hassle I perfectly understand why it's not suitable for musl upstreaming. At least without having additional specification. Though the readv/writev architecture is no excuse the not implement them. It can always be split to individual read()/write() callbacks. But yes, the sizes (and alignment of sizes) may vary. And yes, it might mean the performance will be non-optimal. > My feeling is that "involves callbacks" should be an indication for > exclusion of nonstandard functions. In terms of what I've written > above, I think this follows from the existing principles of exclusion > based on cost of implementation complexity and high risk of > compatibility issues with applications. As distribution, we want things to work. And we can limit support to certain applications that use the specific API. And as we compile everything against musl, we can also make additional ABI considerations. So I think we'll still consider doing fopencookie() or funopen() as distro-mainted patch. These are additional constraints we can live with - compared to musl upstream which is committed to maintain ABI between releases. Though, we'll probably also file portability bugs against those apps that rely on this. /Timo