Re: [musl] Selecting locale source format

[Pablo Correa Gomez <pabloyoyoista@postmarketos.org>]

Thanks a lot Rich for the follow-up.

I have now called the attention of the translators, and asked them some
further questions, most importantly if there is something they think
won't be accommodated by their language.

Personally, I do not see anything that might break Spanish, as we have
in the current format. 

I also like option (1) best, mostly out of it being more compact, and
similar to what other people are doing. IIRC glibc locale translations
look very similar.

Best,
Pablo

El mar, 16-09-2025 a las 21:14 -0400, Rich Felker escribió:
> I have a proposed binary format for new locale files that I'm in the
> process of writing up, but Pablo brought it to my attention that,
> while binary format (ABI) is what's important to have down and stable
> at the time we integrate into musl, pinning down the source format is
> what's important/blocking for collaboration with localization folks.
> 
> I have two candidate formats in the works right now for this:
> 
> 
> 
> Option 1: subset+extension of POSIX localedef format.
> 
> The basis for this format is described in
> https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.h
> tml
> 
> If we go this way, it would be a "subset" because (1) some parts are
> not relevant, like LC_CTYPE, which does not vary by locale, (2) some
> parts will necessarily be represented in different ways, like
> collation where we're using UCA rather than the POSIX form, and (3)
> the format just has a lot of gratuitous cruft like symbolic character
> names. It will also necessarily be extended because POSIX localedef
> has no way to represent translated error strings etc. - keys for them
> have to be added.
> 
> Going this route would have the source data in a fairly compact and
> "well-known" (to certain audiences) form, but requires that the
> tooling to produce binary locale files be aware of how these fields
> translate to the data model for the binary form.
> 
> A sample (should be roughly correct C/POSIX locale) is attached for
> reference.
> 
> 
> 
> 
> Option 2: human-readable/text representation of the binary form
> 
> Describing this requires a basic intro to the binary form, which is a
> multi-level hierarchical table mapping a path of integer key values
> to
> a data blob. In text we can represent keys with symbolic constants,
> but they're just a way of writing the underlying numbers. For example
> the path strerror/0 leads to the "No error information" text,
> strerror/EACCES leads to the "Permission denied" text, etc. Here
> "strerror" just represents a number for the first-level path
> component
> where strerror strings are stored, subindexed by (the arch/generic
> versions of) the errno codes.
> 
> Going this route mostly avoids the need for smarts in the tooling,
> and
> "has more flexibility" to encode things. But this also potentially
> makes the encoding seem more arbitrary to localization folks.
> 
> Like in option 1, a sample (some hybrid between C/POSIX and a
> hypothetical US-English locale, whipped up quick by hand as an
> example) of one way this format could look is attached for reference.
> An obvious variant that might be friendlier/more-familiar to folks
> working with the data would be representing the same in json (which
> is
> easy).
> 
> 
> 
> 
> My leaning is towards option 1.