[musl] Selecting locale source format

[Rich Felker <dalias@libc.org>]

[-- Attachment #1: Type: text/plain, Size: 2627 bytes --]

I have a proposed binary format for new locale files that I'm in the
process of writing up, but Pablo brought it to my attention that,
while binary format (ABI) is what's important to have down and stable
at the time we integrate into musl, pinning down the source format is
what's important/blocking for collaboration with localization folks.

I have two candidate formats in the works right now for this:



Option 1: subset+extension of POSIX localedef format.

The basis for this format is described in
https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.html

If we go this way, it would be a "subset" because (1) some parts are
not relevant, like LC_CTYPE, which does not vary by locale, (2) some
parts will necessarily be represented in different ways, like
collation where we're using UCA rather than the POSIX form, and (3)
the format just has a lot of gratuitous cruft like symbolic character
names. It will also necessarily be extended because POSIX localedef
has no way to represent translated error strings etc. - keys for them
have to be added.

Going this route would have the source data in a fairly compact and
"well-known" (to certain audiences) form, but requires that the
tooling to produce binary locale files be aware of how these fields
translate to the data model for the binary form.

A sample (should be roughly correct C/POSIX locale) is attached for
reference.




Option 2: human-readable/text representation of the binary form

Describing this requires a basic intro to the binary form, which is a
multi-level hierarchical table mapping a path of integer key values to
a data blob. In text we can represent keys with symbolic constants,
but they're just a way of writing the underlying numbers. For example
the path strerror/0 leads to the "No error information" text,
strerror/EACCES leads to the "Permission denied" text, etc. Here
"strerror" just represents a number for the first-level path component
where strerror strings are stored, subindexed by (the arch/generic
versions of) the errno codes.

Going this route mostly avoids the need for smarts in the tooling, and
"has more flexibility" to encode things. But this also potentially
makes the encoding seem more arbitrary to localization folks.

Like in option 1, a sample (some hybrid between C/POSIX and a
hypothetical US-English locale, whipped up quick by hand as an
example) of one way this format could look is attached for reference.
An obvious variant that might be friendlier/more-familiar to folks
working with the data would be representing the same in json (which is
easy).




My leaning is towards option 1.


[-- Attachment #2: sample_posix_localedef.txt --]
[-- Type: text/plain, Size: 1459 bytes --]

LC_TIME
abday "Sun";"Mon";"Tue";"Wed";"Thu";"Fri";"Sat"
day "Sunday";"Monday";"Tuesday";"Wednesday";"Thursday";"Friday";"Saturday"
abmon "Jan";"Feb";"Mar";"Apr";"May";"Jun";"Jul";"Aug";"Sep";"Oct";"Nov";"Dec"
mon "Januar";"February";"March";"April";"May";"June";"July";"August";"September";"October";"November";"December"
d_t_fmt "%a %b %e %H:%M:%S %Y"
d_fmt "%m/%d/%y"
t_fmt "%H:%M:%S"
t_fmt_ampm "%I:%M:%S %p"
am_pm "AM";"PM"
END LC_TIME

LC_MESSAGES
yesexpr "^[yY]"
noexpr "^[nN]"
EILSEQ "Illegal byte sequence"
EDOM "Domain error"
ERANGE "Result not representable"
ENOTTY "Not a tty"
EACCES "Permission denied"
EPERM "Operation not permitted"
...
EAI_0 "Unknown error"
EAI_BADFLAGS "Invalid flags"
EAI_NONAME "Name does not resolve"
EAI_AGAIN "Try again"
...
REG_OK "No error"
REG_NOMATCH "No match"
REG_BADPAT "Invalid regexp"
...
END LC_MESSAGES

LC_NUMERIC
decimal_point "."
thousands_sep ""
grouping -1
END LC_NUMERIC

LC_MONETARY
int_curr_symbol      ""
currency_symbol      ""
mon_decimal_point    ""
mon_thousands_sep    ""
mon_grouping         -1
positive_sign        ""
negative_sign        ""
int_frac_digits      -1
frac_digits          -1
p_cs_precedes        -1
p_sep_by_space       -1
n_cs_precedes        -1
n_sep_by_space       -1
p_sign_posn          -1
n_sign_posn          -1
int_p_cs_precedes    -1
int_p_sep_by_space   -1
int_n_cs_precedes    -1
int_n_sep_by_space   -1
int_p_sign_posn      -1
int_n_sign_posn      -1
END LC_MONETARY

[-- Attachment #3: sample_binary_as_text.txt --]
[-- Type: text/plain, Size: 4349 bytes --]

[langinfo/LC_TIME]
ABDAY1="Sun"
ABDAY2="Mon"
ABDAY3="Tue"
ABDAY4="Wed"
ABDAY5="Thu"
ABDAY6="Fri"
ABDAY7="Sat"
DAY1="Sunday"
DAY2="Monday"
DAY3="Tuesday"
DAY4="Wednesday"
DAY5="Thursday"
DAY6="Friday"
DAY7="Saturday"
...

[langinfo/LC_NUMERIC]
decimal_point="."
thousands_sep=""
grouping_sc="\177"
grouping_uc="\377"

[langinfo/LC_MONETARY]
mon_grouping_sc="\3\177"
mon_grouping_uc="\3\377"
mon_thousands_sep=","
mon_decimal_point="."
int_curr_symbol="USD "
currency_symbol="$"

[strerror]
0="No error information"
EILSEQ="Illegal byte sequence"
EDOM="Domain error"
ERANGE="Result not representable"
ENOTTY="Not a tty"
EACCES="Permission denied"
EPERM="Operation not permitted"
ENOENT="No such file or directory"
ESRCH="No such process"
EEXIST="File exists"
EOVERFLOW="Value too large for data type"
ENOSPC="No space left on device"
ENOMEM="Out of memory"
EBUSY="Resource busy"
EINTR="Interrupted system call"
EAGAIN="Resource temporarily unavailable"
ESPIPE="Invalid seek"
EXDEV="Cross-device link"
EROFS="Read-only file system"
ENOTEMPTY="Directory not empty"
ECONNRESET="Connection reset by peer"
ETIMEDOUT="Operation timed out"
ECONNREFUSED="Connection refused"
EHOSTDOWN="Host is down"
EHOSTUNREACH="Host is unreachable"
EADDRINUSE="Address in use"
EPIPE="Broken pipe"
EIO="I/O error"
ENXIO="No such device or address"
ENOTBLK="Block device required"
ENODEV="No such device"
ENOTDIR="Not a directory"
EISDIR="Is a directory"
ETXTBSY="Text file busy"
ENOEXEC="Exec format error"
EINVAL="Invalid argument"
E2BIG="Argument list too long"
ELOOP="Symbolic link loop"
ENAMETOOLONG="Filename too long"
ENFILE="Too many open files in system"
EMFILE="No file descriptors available"
EBADF="Bad file descriptor"
ECHILD="No child process"
EFAULT="Bad address"
EFBIG="File too large"
EMLINK="Too many links"
ENOLCK="No locks available"
EDEADLK="Resource deadlock would occur"
ENOTRECOVERABLE="State not recoverable"
EOWNERDEAD="Previous owner died"
ECANCELED="Operation canceled"
ENOSYS="Function not implemented"
ENOMSG="No message of desired type"
EIDRM="Identifier removed"
ENOSTR="Device not a stream"
ENODATA="No data available"
ETIME="Device timeout"
ENOSR="Out of streams resources"
ENOLINK="Link has been severed"
EPROTO="Protocol error"
EBADMSG="Bad message"
EBADFD="File descriptor in bad state"
ENOTSOCK="Not a socket"
EDESTADDRREQ="Destination address required"
EMSGSIZE="Message too large"
EPROTOTYPE="Protocol wrong type for socket"
ENOPROTOOPT="Protocol not available"
EPROTONOSUPPORT,"Protocol not supported"
ESOCKTNOSUPPORT,"Socket type not supported"
ENOTSUP="Not supported"
EPFNOSUPPORT="Protocol family not supported"
EAFNOSUPPORT="Address family not supported by protocol"
EADDRNOTAVAIL,"Address not available"
ENETDOWN="Network is down"
ENETUNREACH="Network unreachable"
ENETRESET="Connection reset by network"
ECONNABORTED="Connection aborted"
ENOBUFS="No buffer space available"
EISCONN="Socket is connected"
ENOTCONN="Socket not connected"
ESHUTDOWN="Cannot send after socket shutdown"
EALREADY="Operation already in progress"
EINPROGRESS="Operation in progress"
ESTALE="Stale file handle"
EUCLEAN="Data consistency error"
ENAVAIL="Resource not available"
EREMOTEIO="Remote I/O error"
EDQUOT="Quota exceeded"
ENOMEDIUM="No medium found"
EMEDIUMTYPE="Wrong medium type"
EMULTIHOP="Multihop attempted"
ENOKEY="Required key not available"
EKEYEXPIRED="Key has expired"
EKEYREVOKED="Key has been revoked"
EKEYREJECTED="Key was rejected by service"

[gai_strerror]
0="Unknown error"
EAI_BADFLAGS="Invalid flags"
EAI_NONAME="Name does not resolve"
EAI_AGAIN="Try again"
EAI_FAIL="Non-recoverable error"
EAI_NODATA="Name has no usable address"
EAI_FAMILY="Unrecognized address family or invalid length"
EAI_SOCKTYPE="Unrecognized socket type"
EAI_SERVICE="Unrecognized service"
EAI_MEMORY="Out of memory"
EAI_SYSTEM="System error"
EAI_OVERFLOW="Overflow"

[regerror]
REG_OK="No error"
REG_NOMATCH="No match"
REG_BADPAT="Invalid regexp"
REG_ECOLLATE="Unknown collating element"
REG_ECTYPE="Unknown character class name"
REG_EESCAPE="Trailing backslash"
REG_ESUBREG="Invalid back reference"
REG_EBRACK="Missing ']'"
REG_EPAREN="Missing ')'"
REG_EBRACE="Missing '}'"
REG_BADBR="Invalid contents of {}"
REG_ERANGE="Invalid character range"
REG_ESPACE="Out of memory"
REG_BADRPT="Repetition not preceded by valid expression"
???="Unknown error"