From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/15078 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: [ Guidance ] Potential New Routines; Requesting Help Date: Mon, 30 Dec 2019 12:31:06 -0500 Message-ID: <20191230173106.GI30412@brightrain.aerifal.cx> References: Reply-To: musl@lists.openwall.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="228651"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Mutt/1.5.21 (2010-09-15) Cc: musl@lists.openwall.com To: JeanHeyd Meneide Original-X-From: musl-return-15094-gllmg-musl=m.gmane.org@lists.openwall.com Mon Dec 30 18:31:22 2019 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.89) (envelope-from ) id 1ilyt0-000xKx-1L for gllmg-musl@m.gmane.org; Mon, 30 Dec 2019 18:31:22 +0100 Original-Received: (qmail 20227 invoked by uid 550); 30 Dec 2019 17:31:20 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 20209 invoked from network); 30 Dec 2019 17:31:19 -0000 Content-Disposition: inline In-Reply-To: Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:15078 Archived-At: On Tue, Dec 24, 2019 at 06:06:50PM -0500, JeanHeyd Meneide wrote: > Dear musl Maintainers and Contributors, > > I hope this e-mail finds you doing well this Holiday Season! I am > interested in developing a few fast routines for text encoding for > musl after the positive reception of a paper for the C Standard > related to fast conversion routines: > > https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html This is interesting, but I'm trying to understand the motivation. If __STDC_ISO_10646__ is defined, wchar_t is UTF-32/UCS-4, and the proposed functions are just the identity (for the c32 ones) and UTF-16/32 conversion. If it's not defined, you have the same problem as the current mb/cNN functions: there's no reason to believe arbitrary Unicode characters can round-trip through wchar_t any better than they can through multibyte characters. In fact on such implementations it's likely that wchar_t meanings are locale-dependent and just a remapping of the byte/multibyte characters. What situation do you envision where the proposed functions let you reliably do something that's not already possible? > While I have a basic implementation, I would like to use some > processor and compiler intrinsics to make it faster and make sure my > first contribution meets both quality and speed standards for a C > library. > > Is there a place in the codebase I can look to for guidance on > how to handle intrinsics properly within musl libc? If there is > already infrastructure and common idioms in place, I would rather use > that then starting to spin up my own. I'm not sure what you mean by intrinsics or why you're looking for them but I guess you're thinking of something as a performance optimization? musl favors having code in straight simple C except when there's a strong reason (known bottleneck in existing real-world software -- things like memcpy, strlen, etc.) to do otherwise. The existing mb/wc code is slightly "vectorized" (see mbsrtowcs) but doing so was probably a mistake. The motivation came along with one of the early motivations for musl: not making UTF-8 a major performance regression like it was in glibc. But it turned out the bigger issue was the performance of character-at-a-time and byte-at-a-time conversions, not bulk conversion. If we do adopt these functions, the right way to do it would be using them to refactor the existing c16/c32 functions. Basically, for example, the bulk of c16rtomb would become c16rtowc, and c16rtomb would be replaced with a call to c16rtowc followed by wctomb. And the string ones can all be simple loop wrappers. Rich