ruby-core@ruby-lang.org archive (unofficial mirror)
 help / color / mirror / Atom feed
* [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
@ 2023-01-26 20:52 stringsn88keys (Thomas Powell) via ruby-core
  2023-01-26 23:03 ` [ruby-core:112063] " austin (Austin Ziegler) via ruby-core
                   ` (9 more replies)
  0 siblings, 10 replies; 11+ messages in thread
From: stringsn88keys (Thomas Powell) via ruby-core @ 2023-01-26 20:52 UTC (permalink / raw)
  To: ruby-core; +Cc: stringsn88keys (Thomas Powell)

Issue #19383 has been reported by stringsn88keys (Thomas Powell).

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383

* Author: stringsn88keys (Thomas Powell)
* Status: Open
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112063] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
@ 2023-01-26 23:03 ` austin (Austin Ziegler) via ruby-core
  2023-01-26 23:59 ` [ruby-core:112064] " stringsn88keys (Thomas Powell) via ruby-core
                   ` (8 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: austin (Austin Ziegler) via ruby-core @ 2023-01-26 23:03 UTC (permalink / raw)
  To: ruby-core; +Cc: austin (Austin Ziegler)

Issue #19383 has been updated by austin (Austin Ziegler).


It’s been a long time since I’ve used Windows, but the Windows console is notoriously stuck in 1980s encodings and using codepage 65001 *should* fix this in general. Otherwise, you’re going to get Windows 1252 encoding as your default input/output encoding even if Ruby is otherwise using UTF-8.

I believe that since Ruby 3.0, Ruby by default uses UTF-8 but the boundaries caused by your console codepage may be a confounding factor.

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101497

* Author: stringsn88keys (Thomas Powell)
* Status: Open
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112064] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
  2023-01-26 23:03 ` [ruby-core:112063] " austin (Austin Ziegler) via ruby-core
@ 2023-01-26 23:59 ` stringsn88keys (Thomas Powell) via ruby-core
  2023-01-27  2:49 ` [ruby-core:112070] " austin (Austin Ziegler) via ruby-core
                   ` (7 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: stringsn88keys (Thomas Powell) via ruby-core @ 2023-01-26 23:59 UTC (permalink / raw)
  To: ruby-core; +Cc: stringsn88keys (Thomas Powell)

Issue #19383 has been updated by stringsn88keys (Thomas Powell).


By "console" do you mean irb are you referencing PowerShell or cmd.exe/Command Prompt? Windows Terminal produces the same results as well.

Also, the source for this is from one process to another without user interactivity.

Looking at the Code Page 437 vs. Windows-1252, 0xE4 would be ∑ in Code Page 437 and ä in Windows-1252

The byte sequence of "Mitteleuropäische Zeit" as encoded from "Time.now.zone" (which reports itself as "IBM437" is (hex values):
=> ["4d", "69", "74", "74", "65", "6c", "65", "75", "72", "6f", "70", "e4", "69", "73", "63", "68", "65", "20", "5a", "65", "69", "74"]

70 e4 69 would be "päi" in Windows-1252, but "p∑i" in IBM437 as reported. If UTF-8 is assumed, then e4 is a leading byte for a CJK script byte, but packing them doesn't associate the e4 with the following byte, which is confirmed by occasional invalid byte sequence errors depending on how the string is picked up.

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101498

* Author: stringsn88keys (Thomas Powell)
* Status: Open
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112070] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
  2023-01-26 23:03 ` [ruby-core:112063] " austin (Austin Ziegler) via ruby-core
  2023-01-26 23:59 ` [ruby-core:112064] " stringsn88keys (Thomas Powell) via ruby-core
@ 2023-01-27  2:49 ` austin (Austin Ziegler) via ruby-core
  2023-01-27  7:08 ` [ruby-core:112073] " nobu (Nobuyoshi Nakada) via ruby-core
                   ` (6 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: austin (Austin Ziegler) via ruby-core @ 2023-01-27  2:49 UTC (permalink / raw)
  To: ruby-core; +Cc: austin (Austin Ziegler)

Issue #19383 has been updated by austin (Austin Ziegler).


Yes, I mean cmd.exe or any other windows command-line. I repeat, that it has been *years* since I have used Windows in any serious manner, but this was the absolute *bane* of my existence, and process *boundaries* on Windows were nightmarish when I last dealt with Windows, primarily because of the emphasis on backwards compatibility at any cost. There does appear to be a bug if `Time.now.zone` is not returning the data in the code page expected for where it’s used (e.g., UTF-8).

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101505

* Author: stringsn88keys (Thomas Powell)
* Status: Open
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112073] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (2 preceding siblings ...)
  2023-01-27  2:49 ` [ruby-core:112070] " austin (Austin Ziegler) via ruby-core
@ 2023-01-27  7:08 ` nobu (Nobuyoshi Nakada) via ruby-core
  2023-01-27 13:22 ` [ruby-core:112083] " nobu (Nobuyoshi Nakada) via ruby-core
                   ` (5 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: nobu (Nobuyoshi Nakada) via ruby-core @ 2023-01-27  7:08 UTC (permalink / raw)
  To: ruby-core; +Cc: nobu (Nobuyoshi Nakada)

Issue #19383 has been updated by nobu (Nobuyoshi Nakada).

Status changed from Open to Feedback

What are:
* the output from `chcp.com` command
* `Encoding.locale_charmap`
?

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101507

* Author: stringsn88keys (Thomas Powell)
* Status: Feedback
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112083] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (3 preceding siblings ...)
  2023-01-27  7:08 ` [ruby-core:112073] " nobu (Nobuyoshi Nakada) via ruby-core
@ 2023-01-27 13:22 ` nobu (Nobuyoshi Nakada) via ruby-core
  2023-01-27 13:48 ` [ruby-core:112084] " stringsn88keys (Thomas Powell) via ruby-core
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: nobu (Nobuyoshi Nakada) via ruby-core @ 2023-01-27 13:22 UTC (permalink / raw)
  To: ruby-core; +Cc: nobu (Nobuyoshi Nakada)

Issue #19383 has been updated by nobu (Nobuyoshi Nakada).


Maybe msvcrt converts timezone names to ACP, not ConsoleCP.
If so, this patch may work, but I have no idea how to test this in a CI.

```diff
diff --git i/time.c w/time.c
index 9c4c93939e0..2e1a2dca29b 100644
--- i/time.c
+++ w/time.c
@@ -929,7 +929,7 @@ timegmw_noleapsecond(struct vtm *vtm)
 }
 
 static VALUE
-zone_str(const char *zone)
+zone_str_enc(const char *zone, rb_encoding *enc)
 {
     const char *p;
     int ascii_only = 1;
@@ -950,11 +950,18 @@ zone_str(const char *zone)
         str = rb_usascii_str_new(zone, len);
     }
     else {
-        str = rb_enc_str_new(zone, len, rb_locale_encoding());
+        if (!enc) enc = rb_locale_encoding();
+        str = rb_enc_str_new(zone, len, enc);
     }
     return rb_fstring(str);
 }
 
+static VALUE
+zone_str(const char *zone)
+{
+    return zone_str_enc(zone, NULL);
+}
+
 static void
 gmtimew_noleapsecond(wideval_t timew, struct vtm *vtm)
 {
@@ -1653,12 +1660,18 @@ localtime_with_gmtoff_zone(const time_t *t, struct tm *result, long *gmtoff, VAL
 #if defined(HAVE_TM_ZONE)
             *zone = zone_str(tm.tm_zone);
 #elif defined(HAVE_TZNAME) && defined(HAVE_DAYLIGHT)
+            rb_encoding *enc = NULL;
+# if defined(_WIN32)
+            char cp[DECIMAL_SIZE_OF_BITS(sizeof(UINT) * CHAR_BIT) + sizeof("CP")];
+            snprintf(cp, sizeof(cp), "CP%u", GetACP());
+            enc = rb_enc_find(cp);
+# endif
 # if defined(RUBY_MSVCRT_VERSION) && RUBY_MSVCRT_VERSION >= 140
 #  define tzname _tzname
 #  define daylight _daylight
 # endif
             /* this needs tzset or localtime, instead of localtime_r */
-            *zone = zone_str(tzname[daylight && tm.tm_isdst]);
+            *zone = zone_str_enc(tzname[daylight && tm.tm_isdst], enc);
 #else
             {
                 char buf[64];
```

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101517

* Author: stringsn88keys (Thomas Powell)
* Status: Feedback
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112084] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (4 preceding siblings ...)
  2023-01-27 13:22 ` [ruby-core:112083] " nobu (Nobuyoshi Nakada) via ruby-core
@ 2023-01-27 13:48 ` stringsn88keys (Thomas Powell) via ruby-core
  2023-01-27 21:05 ` [ruby-core:112095] " stringsn88keys (Thomas Powell) via ruby-core
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: stringsn88keys (Thomas Powell) via ruby-core @ 2023-01-27 13:48 UTC (permalink / raw)
  To: ruby-core; +Cc: stringsn88keys (Thomas Powell)

Issue #19383 has been updated by stringsn88keys (Thomas Powell).


nobu (Nobuyoshi Nakada) wrote in #note-4:
> What are:
> * the output from `chcp.com` command
> * `Encoding.locale_charmap`
> 
> ?

chcp.com output:
"Aktive Codepage: 437." ("Active code page: 437" on English display language.

Encoding.locale_charmap # => "CP437" (German and English)

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101518

* Author: stringsn88keys (Thomas Powell)
* Status: Feedback
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112095] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (5 preceding siblings ...)
  2023-01-27 13:48 ` [ruby-core:112084] " stringsn88keys (Thomas Powell) via ruby-core
@ 2023-01-27 21:05 ` stringsn88keys (Thomas Powell) via ruby-core
  2023-01-30  2:23 ` [ruby-core:112107] " nobu (Nobuyoshi Nakada) via ruby-core
                   ` (2 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: stringsn88keys (Thomas Powell) via ruby-core @ 2023-01-27 21:05 UTC (permalink / raw)
  To: ruby-core; +Cc: stringsn88keys (Thomas Powell)

Issue #19383 has been updated by stringsn88keys (Thomas Powell).


The top level status of this bug says "Closed" but last updated status says "Feedback". Can anyone clarify?

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101528

* Author: stringsn88keys (Thomas Powell)
* Status: Feedback
* Priority: Normal
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112107] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (6 preceding siblings ...)
  2023-01-27 21:05 ` [ruby-core:112095] " stringsn88keys (Thomas Powell) via ruby-core
@ 2023-01-30  2:23 ` nobu (Nobuyoshi Nakada) via ruby-core
  2023-03-03  2:15 ` [ruby-core:112672] " nobu (Nobuyoshi Nakada) via ruby-core
  2024-10-21  8:50 ` [ruby-core:119558] " YO4 (Yoshinao Muramatsu) via ruby-core
  9 siblings, 0 replies; 11+ messages in thread
From: nobu (Nobuyoshi Nakada) via ruby-core @ 2023-01-30  2:23 UTC (permalink / raw)
  To: ruby-core; +Cc: nobu (Nobuyoshi Nakada)

Issue #19383 has been updated by nobu (Nobuyoshi Nakada).

Status changed from Feedback to Assigned
Assignee set to windows

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-101544

* Author: stringsn88keys (Thomas Powell)
* Status: Assigned
* Priority: Normal
* Assignee: windows
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:112672] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (7 preceding siblings ...)
  2023-01-30  2:23 ` [ruby-core:112107] " nobu (Nobuyoshi Nakada) via ruby-core
@ 2023-03-03  2:15 ` nobu (Nobuyoshi Nakada) via ruby-core
  2024-10-21  8:50 ` [ruby-core:119558] " YO4 (Yoshinao Muramatsu) via ruby-core
  9 siblings, 0 replies; 11+ messages in thread
From: nobu (Nobuyoshi Nakada) via ruby-core @ 2023-03-03  2:15 UTC (permalink / raw)
  To: ruby-core; +Cc: nobu (Nobuyoshi Nakada)

Issue #19383 has been updated by nobu (Nobuyoshi Nakada).


How about the patch at #note-5?


----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-102122

* Author: stringsn88keys (Thomas Powell)
* Status: Assigned
* Priority: Normal
* Assignee: windows
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [ruby-core:119558] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
  2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
                   ` (8 preceding siblings ...)
  2023-03-03  2:15 ` [ruby-core:112672] " nobu (Nobuyoshi Nakada) via ruby-core
@ 2024-10-21  8:50 ` YO4 (Yoshinao Muramatsu) via ruby-core
  9 siblings, 0 replies; 11+ messages in thread
From: YO4 (Yoshinao Muramatsu) via ruby-core @ 2024-10-21  8:50 UTC (permalink / raw)
  To: ruby-core; +Cc: YO4 (Yoshinao Muramatsu)

Issue #19383 has been updated by YO4 (Yoshinao Muramatsu).


There is another remaining locale related issue.
ref: https://bugs.ruby-lang.org/issues/20774

As an alternative solution, if setlocale(LC_CTYPE, ".65001") could be used, it would solve the problem, but it does not seem to work in all environments.
Also, could there be an impact on the extension library?

Another idea is to use SetProcessPreferredUILanguages or SetThreadPreferredUILanguages. It is possible to lock in a fallback language.
It's results may or may not be desirable. Similarly, the result will affect another program running in the same process.

Saying to #note-5 patch,
It seems to work fine, except for corner cases like setting Unicode-specific characters in TZ.
I also think that in today's environment there are many situations where we want UTF-8 results.

----------------------------------------
Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect
https://bugs.ruby-lang.org/issues/19383#change-110174

* Author: stringsn88keys (Thomas Powell)
* Status: Assigned
* Assignee: windows
* ruby -v: 3.1.3
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
OS:
Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3

Display language:
Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z].

Time zone:
CET (UTC +01:00) Amsterdam, Berlin, ...

Time.now.zone # => "Mitteleuro\xE3ische Zeit"
Time.now.zone.encoding # => #<Encoding:IBM437>
puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit")
Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit"

Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string:
Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit"
... but ...
Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit"

Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252.
https://github.com/chef/ohai/pull/1781 




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2024-10-21  9:12 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-26 20:52 [ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect stringsn88keys (Thomas Powell) via ruby-core
2023-01-26 23:03 ` [ruby-core:112063] " austin (Austin Ziegler) via ruby-core
2023-01-26 23:59 ` [ruby-core:112064] " stringsn88keys (Thomas Powell) via ruby-core
2023-01-27  2:49 ` [ruby-core:112070] " austin (Austin Ziegler) via ruby-core
2023-01-27  7:08 ` [ruby-core:112073] " nobu (Nobuyoshi Nakada) via ruby-core
2023-01-27 13:22 ` [ruby-core:112083] " nobu (Nobuyoshi Nakada) via ruby-core
2023-01-27 13:48 ` [ruby-core:112084] " stringsn88keys (Thomas Powell) via ruby-core
2023-01-27 21:05 ` [ruby-core:112095] " stringsn88keys (Thomas Powell) via ruby-core
2023-01-30  2:23 ` [ruby-core:112107] " nobu (Nobuyoshi Nakada) via ruby-core
2023-03-03  2:15 ` [ruby-core:112672] " nobu (Nobuyoshi Nakada) via ruby-core
2024-10-21  8:50 ` [ruby-core:119558] " YO4 (Yoshinao Muramatsu) via ruby-core

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).