Big5 "mostly" complete

Big5 "mostly" complete

[Rich Felker]

Hi all,

I just committed a revision to iconv Big5 support, adding current
standardized extensions up to nearly the maximum possible level. The
whatwg definition of Big5 is still a bit bigger, so maybe we should
research whether its additions make sense to support. There would be
no size or performance cost to supporting them, since they just fill
in holes in the table without extending the table size.

I'm still unclear on what, if any, action needs to be taken for
extended Taiwan variants of Big5. In theory UAO could be added (minus
the PUA mappings) but it's quite large and I still don't feel like I
have a good understanding of whether there are users who need it.
Whatwg has no mention of it with encoding standards
(http://encoding.spec.whatwg.org/) and only Mozilla seems to support
it. If it's needed in Taiwant for representing native-language text in
a Big5-based encoding, that's a strong argument for supporting it
anyway, but if it's just for representing mixed-language text (as a
poor substitute for Unicode) I feel like it would be hard to justify
including it, and maybe we could think about later supporting it as a
local nonstandard encoding via charmap files...

Comments?

Rich

Re: Big5 "mostly" complete

[Roy]

On Sun, 18 Aug 2013 04:57:57 +0800, Rich Felker <dalias@aerifal.cx> wrote:

> Hi all,
>
> I just committed a revision to iconv Big5 support, adding current
> standardized extensions up to nearly the maximum possible level. The
> whatwg definition of Big5 is still a bit bigger, so maybe we should
> research whether its additions make sense to support. There would be
> no size or performance cost to supporting them, since they just fill
> in holes in the table without extending the table size.
>
> I'm still unclear on what, if any, action needs to be taken for
> extended Taiwan variants of Big5. In theory UAO could be added (minus
> the PUA mappings) but it's quite large and I still don't feel like I
> have a good understanding of whether there are users who need it.
> Whatwg has no mention of it with encoding standards
> (http://encoding.spec.whatwg.org/) and only Mozilla seems to support
> it. If it's needed in Taiwant for representing native-language text in
> a Big5-based encoding, that's a strong argument for supporting it
> anyway, but if it's just for representing mixed-language text (as a
> poor substitute for Unicode) I feel like it would be hard to justify
> including it, and maybe we could think about later supporting it as a
> local nonstandard encoding via charmap files...
>
> Comments?

Both Big5-UAO and Big5-HKSCS are needed for those Taiwan people and Hong  
Kong people.
For Big5-UAO, there is some commonly used dingbats(for example "♡" mark)  
and numeric representations(for example "①") are in Big5-UAO but not in  
CP950.
and Big5-UAO is still being used not only in ptt.cc telnet BBS, but also  
in  text data files(file lists/cue sheets) because of not-supporting UTF-8  
in applications(for example, Perl File-system I/O in windows, CD-Rippers).
for Big5-HKSCS, it use used for storing commonly used Cantonese ideographs  
(for example, "𨋢" means "lift" in Cantonese) in Hong Kong.

>
> Rich

Re: Re: Big5 "mostly" complete

[Rich Felker]

On Sun, Aug 18, 2013 at 12:20:47PM +0800, Roy wrote:
> Both Big5-UAO and Big5-HKSCS are needed for those Taiwan people and
> Hong Kong people.
> For Big5-UAO, there is some commonly used dingbats(for example "♡"
> mark) and numeric representations(for example "①") are in Big5-UAO
> but not in CP950.
> and Big5-UAO is still being used not only in ptt.cc telnet BBS, but
> also in  text data files(file lists/cue sheets) because of
> not-supporting UTF-8 in applications(for example, Perl File-system
> I/O in windows, CD-Rippers).
> for Big5-HKSCS, it use used for storing commonly used Cantonese
> ideographs (for example, "𨋢" means "lift" in Cantonese) in Hong
> Kong.

HKSCS is supported as of yesterday's commit. I'm aware that it's
needed for representing Cantonese language in Big5, and that it's
widely used on the web.

What I'm not clear on is the necessity of UAO. Keep in mind that iconv
is an API for information interchange: things like interpreting web
content, email, old text files, etc. The fact that UAO exists is not
alone reason to support it; it has to actually have usefulness in
situations where the iconv interface should be used. If you want to
see it included, this is what you need to convince us of:

- That it's in widespread use in large volumes of existing data (on
  the web, text files, etc.) or data that is being newly generated
  (e.g. as a default encoding of popular mail software).

- That it's necessary to represent linguistic content in languages
  used in Taiwan, not just as a substitute for Unicode to represent
  foreign languages.

- That failure to support it would put musl's iconv in a worse
  position of compatibility than other iconv implementations or
  software-specific (e.g. in-browser) character set conversions.

The fact that UAO does not even have a MIME type, and that, per web
standards (http://encoding.spec.whatwg.org/), a MIME type of "Big5" is
to be interpreted as essentially Big5-2003+HKSCS, leaves me doubting
that UAO is in widespread use for information interchange. Mozilla
seems to be the only browser that support it. I previously cited a
thread on the whatwg mailing list in which research was being
conducted to determine what should constitute Big5, i.e. roughly what
characters outside the old CP950 were needed for real websites, and
UAO did not come up.

If I'm wrong and just missing its importance, please fill me in. I'm
not trying to be difficult, but if UAO is important, I want to see the
reasons why.

Rich

Re: Re: Big5 "mostly" complete

[Roy]

On Sun, 18 Aug 2013 15:32:29 +0800, Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Aug 18, 2013 at 12:20:47PM +0800, Roy wrote:
>> Both Big5-UAO and Big5-HKSCS are needed for those Taiwan people and
>> Hong Kong people.
>> For Big5-UAO, there is some commonly used dingbats(for example "♡"
>> mark) and numeric representations(for example "①") are in Big5-UAO
>> but not in CP950.
>> and Big5-UAO is still being used not only in ptt.cc telnet BBS, but
>> also in  text data files(file lists/cue sheets) because of
>> not-supporting UTF-8 in applications(for example, Perl File-system
>> I/O in windows, CD-Rippers).
>> for Big5-HKSCS, it use used for storing commonly used Cantonese
>> ideographs (for example, "𨋢" means "lift" in Cantonese) in Hong
>> Kong.
>
> HKSCS is supported as of yesterday's commit. I'm aware that it's
> needed for representing Cantonese language in Big5, and that it's
> widely used on the web.
>
> What I'm not clear on is the necessity of UAO. Keep in mind that iconv
> is an API for information interchange: things like interpreting web
> content, email, old text files, etc. The fact that UAO exists is not
> alone reason to support it; it has to actually have usefulness in
> situations where the iconv interface should be used. If you want to
> see it included, this is what you need to convince us of:
>
> - That it's in widespread use in large volumes of existing data (on
>   the web, text files, etc.) or data that is being newly generated
>   (e.g. as a default encoding of popular mail software).
>

People are told *NOT* to publish file with Big5-UAO to the web(or say,  
people, even the creator of UAO, appeal to people that not to publish file  
with Big5-UAO to the web), but still there are some that's in archive  
format.(Like I said before, for example cue-sheet file of CD-ROM image,  
etc.)
But for local data processing, UAO does facilitate file managing to  
windows users.

> - That it's necessary to represent linguistic content in languages
>   used in Taiwan, not just as a substitute for Unicode to represent
>   foreign languages.

It does, some Chinese ideographs are used as part of name, but not in  
CP950 mapping like "喆" and "堃".

>
> - That failure to support it would put musl's iconv in a worse
>   position of compatibility than other iconv implementations or
>   software-specific (e.g. in-browser) character set conversions.

Since people made Big5-UAO patch for libiconv and glibc(gconv)  
unofficially to meet their uses, if musl libc have an optional Big5-UAO  
mapping will be an advantage to Taiwan people.

> The fact that UAO does not even have a MIME type, and that, per web
> standards (http://encoding.spec.whatwg.org/), a MIME type of "Big5" is
> to be interpreted as essentially Big5-2003+HKSCS, leaves me doubting
> that UAO is in widespread use for information interchange.

Since UAO is used to be local encoding, MIME type does not apply here.
And it is not recommended to use UAO as information exchange *over  
internet*.

> Mozilla seems to be the only browser that support it. I previously cited  
> a
> thread on the whatwg mailing list in which research was being
> conducted to determine what should constitute Big5, i.e. roughly what
> characters outside the old CP950 were needed for real websites, and
> UAO did not come up.
>

Since both Hong Kong and Taiwan government are moving forward to Unicode  
(for example HKSCS no longer provides new Big5 codepoints for new  
ideographs), the importance of using Big5 on *web* is decreasing.
But for local data processing, Big5-HKSCS and Big5-UAO have their own  
position and userbase which keeping them exist.

> If I'm wrong and just missing its importance, please fill me in. I'm
> not trying to be difficult, but if UAO is important, I want to see the
> reasons why.
>
> Rich

Re: Re: Re: Big5 "mostly" complete

[Rich Felker]

On Sun, Aug 18, 2013 at 07:19:57PM +0800, Roy wrote:
> On Sun, 18 Aug 2013 15:32:29 +0800, Rich Felker <dalias@aerifal.cx> wrote:
> 
> >On Sun, Aug 18, 2013 at 12:20:47PM +0800, Roy wrote:
> >>Both Big5-UAO and Big5-HKSCS are needed for those Taiwan people and
> >>Hong Kong people.
> >>For Big5-UAO, there is some commonly used dingbats(for example "♡"
> >>mark) and numeric representations(for example "①") are in Big5-UAO
> >>but not in CP950.
> >>and Big5-UAO is still being used not only in ptt.cc telnet BBS, but
> >>also in  text data files(file lists/cue sheets) because of
> >>not-supporting UTF-8 in applications(for example, Perl File-system
> >>I/O in windows, CD-Rippers).
> >>for Big5-HKSCS, it use used for storing commonly used Cantonese
> >>ideographs (for example, "𨋢" means "lift" in Cantonese) in Hong
> >>Kong.
> >
> >HKSCS is supported as of yesterday's commit. I'm aware that it's
> >needed for representing Cantonese language in Big5, and that it's
> >widely used on the web.
> >
> >What I'm not clear on is the necessity of UAO. Keep in mind that iconv
> >is an API for information interchange: things like interpreting web
> >content, email, old text files, etc. The fact that UAO exists is not
> >alone reason to support it; it has to actually have usefulness in
> >situations where the iconv interface should be used. If you want to
> >see it included, this is what you need to convince us of:
> >
> >- That it's in widespread use in large volumes of existing data (on
> >  the web, text files, etc.) or data that is being newly generated
> >  (e.g. as a default encoding of popular mail software).
> 
> People are told *NOT* to publish file with Big5-UAO to the web(or
> say, people, even the creator of UAO, appeal to people that not to
> publish file with Big5-UAO to the web), but still there are some
> that's in archive format.(Like I said before, for example cue-sheet
> file of CD-ROM image, etc.)
> But for local data processing, UAO does facilitate file managing to
> windows users.

Based on this, I think:

(1) It's reasonable to omit UAO for now, and
(2) Support for iconv to load user-defined characer mappings would be
a worthwhile feature to work on post-1.0.

My reasoning is that the goal of iconv in musl, at least for the
built-in character set conversions, is to facilitate information
interchange, particularly reading of data that may be received in
email, as documents published on the web, via IRC or IM protocols,
etc. An encoding whose creators specifically request that it NOT be
used for publishing/interchange is well outside this scope.

I agree with your examples (CD-ROM cue sheets, archived text files,
that telnet BBS, etc.) that there is a need by some users to
process/import data encoded in UAO, but most of these usages do not
seem to require general applications, treating charsets in an
abstract, MIME-style manner, to be able to handle it. For many of the
examples, a command-line conversion utility (BTW, there are ones much
more powerful than iconv out there) would be the logical choice. For
the BBS, my understanding is that most of its users are using special
telnet/terminal apps with the conversion built-in.

> >- That it's necessary to represent linguistic content in languages
> >  used in Taiwan, not just as a substitute for Unicode to represent
> >  foreign languages.
> 
> It does, some Chinese ideographs are used as part of name, but not
> in CP950 mapping like "喆" and "堃".

How do these users send email or enter their names in web-based apps?
My guess would be that the email clients switch to UTF-8 when
encountering a character they can't encode in Big5, and that,
nowadays, most web apps are built on CMS that are Unicode-based. Is
this correct?

> >- That failure to support it would put musl's iconv in a worse
> >  position of compatibility than other iconv implementations or
> >  software-specific (e.g. in-browser) character set conversions.
> 
> Since people made Big5-UAO patch for libiconv and glibc(gconv)
> unofficially to meet their uses, if musl libc have an optional
> Big5-UAO mapping will be an advantage to Taiwan people.

*nod*

For what it's worth, how do those patches handle it? Do they add a new
"Big5-UAO" charset name to iconv, or do they modify the existing Big5
to treat it as UAO?

My feeling for now is to increase the priority of adding custom local
charmap files to iconv after musl 1.0 is released. My main reason is
that "intended for information interchange" vs "intended only for
local use" seems to be the best guideline for whether an encoding is
appropriate to include built-in.

Rich

Re: Re: Re: Big5 "mostly" complete

[Roy]

On Tue, 27 Aug 2013 09:53:49 +0800, Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Aug 18, 2013 at 07:19:57PM +0800, Roy wrote:
>> On Sun, 18 Aug 2013 15:32:29 +0800, Rich Felker <dalias@aerifal.cx>  
>> wrote:
>>
>> >On Sun, Aug 18, 2013 at 12:20:47PM +0800, Roy wrote:
>> >>Both Big5-UAO and Big5-HKSCS are needed for those Taiwan people and
>> >>Hong Kong people.
>> >>For Big5-UAO, there is some commonly used dingbats(for example "♡"
>> >>mark) and numeric representations(for example "①") are in Big5-UAO
>> >>but not in CP950.
>> >>and Big5-UAO is still being used not only in ptt.cc telnet BBS, but
>> >>also in  text data files(file lists/cue sheets) because of
>> >>not-supporting UTF-8 in applications(for example, Perl File-system
>> >>I/O in windows, CD-Rippers).
>> >>for Big5-HKSCS, it use used for storing commonly used Cantonese
>> >>ideographs (for example, "𨋢" means "lift" in Cantonese) in Hong
>> >>Kong.
>> >
>> >HKSCS is supported as of yesterday's commit. I'm aware that it's
>> >needed for representing Cantonese language in Big5, and that it's
>> >widely used on the web.
>> >
>> >What I'm not clear on is the necessity of UAO. Keep in mind that iconv
>> >is an API for information interchange: things like interpreting web
>> >content, email, old text files, etc. The fact that UAO exists is not
>> >alone reason to support it; it has to actually have usefulness in
>> >situations where the iconv interface should be used. If you want to
>> >see it included, this is what you need to convince us of:
>> >
>> >- That it's in widespread use in large volumes of existing data (on
>> >  the web, text files, etc.) or data that is being newly generated
>> >  (e.g. as a default encoding of popular mail software).
>>
>> People are told *NOT* to publish file with Big5-UAO to the web(or
>> say, people, even the creator of UAO, appeal to people that not to
>> publish file with Big5-UAO to the web), but still there are some
>> that's in archive format.(Like I said before, for example cue-sheet
>> file of CD-ROM image, etc.)
>> But for local data processing, UAO does facilitate file managing to
>> windows users.
>
> Based on this, I think:
>
> (1) It's reasonable to omit UAO for now, and
> (2) Support for iconv to load user-defined character mappings would be
> a worthwhile feature to work on post-1.0.
>

That is good. But I have few feature request about this:
- user-defined mapping can be overlayed to another coding, just like HKSCS  
does.
- user-defined mapping can be embedded to static-linked binary.

And for Unicode to CJK legacy encodings is a must (hope it is available  
before musl-1.0)

> My reasoning is that the goal of iconv in musl, at least for the
> built-in character set conversions, is to facilitate information
> interchange, particularly reading of data that may be received in
> email, as documents published on the web, via IRC or IM protocols,
> etc. An encoding whose creators specifically request that it NOT be
> used for publishing/interchange is well outside this scope.

Yeah it is not encouraged for publishing since it is not a standard and  
people are not encouraged to install UAO blindly, but people do use it for  
private interchange(like sending files via ftp/instant messaging)

>
> I agree with your examples (CD-ROM cue sheets, archived text files,
> that telnet BBS, etc.) that there is a need by some users to
> process/import data encoded in UAO, but most of these usages do not
> seem to require general applications, treating charsets in an
> abstract, MIME-style manner, to be able to handle it. For many of the
> examples, a command-line conversion utility (BTW, there are ones much
> more powerful than iconv out there) would be the logical choice. For
> the BBS, my understanding is that most of its users are using special
> telnet/terminal apps with the conversion built-in.
>
>> >- That it's necessary to represent linguistic content in languages
>> >  used in Taiwan, not just as a substitute for Unicode to represent
>> >  foreign languages.
>>
>> It does, some Chinese ideographs are used as part of name, but not
>> in CP950 mapping like "喆" and "堃".
>
> How do these users send email or enter their names in web-based apps?
> My guess would be that the email clients switch to UTF-8 when
> encountering a character they can't encode in Big5, and that,
> nowadays, most web apps are built on CMS that are Unicode-based. Is
> this correct?
>

Yes, most popular web apps are using UTF-8 nowadays.
In the past, people enter (方方土) as 堃 and (吉吉) as 喆, and they may  
install ChinaSea/UAO/etc. charset extensions for 堃 and 喆 as well.

>> >- That failure to support it would put musl's iconv in a worse
>> >  position of compatibility than other iconv implementations or
>> >  software-specific (e.g. in-browser) character set conversions.
>>
>> Since people made Big5-UAO patch for libiconv and glibc(gconv)
>> unofficially to meet their uses, if musl libc have an optional
>> Big5-UAO mapping will be an advantage to Taiwan people.
>
> *nod*
>
> For what it's worth, how do those patches handle it? Do they add a new
> "Big5-UAO" charset name to iconv, or do they modify the existing Big5
> to treat it as UAO?

The original patches by Tiberius Teng modify Big5 with Big5-UAO mappings.
I'm trying to reach Tiberius and get the patch if available.

And there is another libiconv patch that adds big5-uao encoding instead.
http://ku.myftp.org/goods/libiconv-1.11-uao.patch.bz2

>
> My feeling for now is to increase the priority of adding custom local
> charmap files to iconv after musl 1.0 is released. My main reason is
> that "intended for information interchange" vs "intended only for
> local use" seems to be the best guideline for whether an encoding is
> appropriate to include built-in.
>
> Rich