Is there any desire to support languages other than English in zsh?

Is there any desire to support languages other than English in zsh?

[Agron Selimaj]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

Hello,

It seems that zsh prints messages in English only. I would like to add
support for Albanian and German by using environment variables like this?

    case ${profile} in

      SQ|SQ_AL|sq_AL)
          LC_ALL="sq_AL.UTF-8"
          LANG="sq_AL.UTF-8"
          LANGUAGE="sq_AL:sq:en_US:en"
          ;;

      DE|DE_DE|de_DE)

          LC_ALL="de_DE.UTF-8"
          LANG="de_DE.UTF-8"
          LANGUAGE="de_DE:de:en_US:en"
          ;;

EN|EN_US|en|en_US) LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8"
LANGUAGE="en_US:en" ;; *)


However, I am not sure the zsh project supports gettext (and POEdit) throughout.

So, in order to support localization in general, we first must use
gettext everywhere.

Is this something this team is willing to do?


Thanks,

//Agron

[-- Attachment #2: Type: text/html, Size: 9505 bytes --]

Re: Is there any desire to support languages other than English in zsh?

[Peter Stephenson]

> On 06/06/2023 16:20 Agron Selimaj <as9902613@gmail.com> wrote:
> However, I am not sure the zsh project supports gettext (and POEdit) throughout.
> So, in order to support localization in general, we first must use gettext everywhere. 
> Is this something this team is willing to do?

This is certainly a useful thing to do, however previous discussions suggest there is
no one around to do the considerable amount of work that supporting databases on this
scale would require.  We would certainly be able and willing to support anyone who did
have the time to do the work, though.

I could see an alternative of allowing zsh's own message system to use zsh associative
variables to retrieve messages --- the advantage being that the internal support
required within the shell is much more limited as everything else can be done at
the shell language level.   But I have detected no enthusiasm for this proposal in
the past.

pws

Re: Is there any desire to support languages other than English in zsh?

[Bart Schaefer]

On Tue, Jun 6, 2023 at 8:54 AM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> > On 06/06/2023 16:20 Agron Selimaj <as9902613@gmail.com> wrote:
> > However, I am not sure the zsh project supports gettext (and POEdit) throughout.
> > So, in order to support localization in general, we first must use gettext everywhere.
> > Is this something this team is willing to do?
>
> This is certainly a useful thing to do, however previous discussions suggest there is
> no one around to do the considerable amount of work that supporting databases on this
> scale would require.

Having previously overseen the gettext-ification of a piece of
software of similar original vintage and complexity, I would say that
supporting the databases is often the least of the issues.  Sentence
structure varies, which requires converting catenated substrings into
snprintf formats or sets of alternatives; pluralization may differ,
requiring 0 / 1 / >1 conditionals even with gettext intervention; etc.
Also, because of the pervasive changes required, we found it necessary
to merge all outstanding code changes and call a halt to other
development until the "l10n" pass was complete, or we couldn't track
deltas accurately (though we were stuck with CVS for version control
at the time).

This is definitely not something we would want to dive into until
after whatever the next zsh release is, is done.

Re: Is there any desire to support languages other than English in zsh?

[Agron Selimaj]

[-- Attachment #1: Type: text/plain, Size: 3313 bytes --]

What if we take the long path by starting small?

Let me add gettext support, and we don't have to use it everywhere from day
one.

Let me just make the "--help" output support Albanian and German. This
shouldn't affect anything in the operation of the program since it is in
the exit path. It's small and safe. Plus it gives us a chance to discuss
implementation further.

❯ zsh --help
> Usage: zsh [<options>] [<argument> ...]
>
> Special options:
>  --help     show this message, then exit
>  --version  show zsh version number, then exit
>  -b         end option processing, like --
>  -c         take first argument as a command to execute
>  -o OPTION  set an option by name (see below)
>
> Normal options are named.  An option may be turned on by
> `-o OPTION', `--OPTION', `+o no_OPTION' or `+-no-OPTION'.  An
> option may be turned off by `-o no_OPTION', `--no-OPTION',
> `+o OPTION' or `+-OPTION'.  Options are listed below only in
> `--OPTION' or `--no-OPTION' form.
>
>
This means, I'll make this text be extractable to generate *.POT files and
then we can use all kinds of tools and websites to do the actual
translation.

I'll also use a certain syntax for parameters in the strings, and that can
also be used as an example for new strings to be added to the zsh.

I don't know what is the anticipated release date for 6.0, but maybe we can
put this much in for 6.0?

And then, since zsh 5.xx lasted 11 years, and before that 4.xx lasted 12
years, I assume 6.xx will also last about 10+ years, can we make a long
term plan to fully implement gettext support during the lifetime of 6.xx
for the next 10 years?

What do you guys think? Shall we try it?

Thanks,
//Agron




On Wed, Jun 7, 2023 at 10:56 PM Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Tue, Jun 6, 2023 at 8:54 AM Peter Stephenson
> <p.w.stephenson@ntlworld.com> wrote:
> >
> > > On 06/06/2023 16:20 Agron Selimaj <as9902613@gmail.com> wrote:
> > > However, I am not sure the zsh project supports gettext (and POEdit)
> throughout.
> > > So, in order to support localization in general, we first must use
> gettext everywhere.
> > > Is this something this team is willing to do?
> >
> > This is certainly a useful thing to do, however previous discussions
> suggest there is
> > no one around to do the considerable amount of work that supporting
> databases on this
> > scale would require.
>
> Having previously overseen the gettext-ification of a piece of
> software of similar original vintage and complexity, I would say that
> supporting the databases is often the least of the issues.  Sentence
> structure varies, which requires converting catenated substrings into
> snprintf formats or sets of alternatives; pluralization may differ,
> requiring 0 / 1 / >1 conditionals even with gettext intervention; etc.
> Also, because of the pervasive changes required, we found it necessary
> to merge all outstanding code changes and call a halt to other
> development until the "l10n" pass was complete, or we couldn't track
> deltas accurately (though we were stuck with CVS for version control
> at the time).
>
> This is definitely not something we would want to dive into until
> after whatever the next zsh release is, is done.
>

[-- Attachment #2: Type: text/html, Size: 4461 bytes --]

RE: Re: Is there any desire to support languages other than English in zsh?

[zeurkous]

Haai,

Me thinks it's about time for me to speak up.

On Wed, 7 Jun 2023 23:52:59 -0400, Agron Selimaj <as9902613@gmail.com> wrote:
> What if we take the long path by starting small?
>
> Let me add gettext support, and we don't have to use it everywhere from day
> one.

An incomplete, underdesigned "feature" upsets people more than a lack of
such, IME. Besides, sprinkling gettext calls around just leads to more
bloat, and more importantly, is inconsistent with the existing design
of zsh.

But me has deeper objections to this whole idea.

Whether we like it or not... the main language of computing as we know
it, is English. Me's not a fan of hiding that fact. zsh is aimed at a
moderately technical audience. It's not unreasonable to expect zsh users
to learn English (either up front, as many do in school; or for the rest
of us, as we go). 

The real issue is to process text correctly, no matter what natlang
it's written in. As long as we're dandy there, me doesn't see the need
for all this messing about translating shell messages.

Indeed, it's actively harmful: while it's somewhat rare, shell messages
are occasionally parsed by invoking programs (such as test scripts), and
those programs would have to be updated to deal with localized versions
of them as well.

Besides, in the specific case of the shell language: the names of many
commands and concepts like "if", "exec", etc. will remain untranslated
by necessity. This will create a jarring discontinuity that people may
trip over (and IIRC there have been environments where the next step
was to localize the names, making code incompatible between natlangs...
ugh.).

To summarize: the posts use French; me's been informed that for world
trade, the most prominent language is Spanish; and in computing, we have
English as our primary language. 

Is that really a bug? Or just a fact of life? Or perhaps even an asset,
as many globalists would argue?

Either way, steps towards splitting the shell along natlang lines
should not be taken lightly.

Hope this gives some insight,

        --zeurkous.

-- 
Friggin' Machines!

FU: RE: Re: Is there any desire to support languages other than English in zsh?

[zeurkous]

Me also has another criticism, that me neglected to write in me previous
message. Allow me to repair--

On Thu, 08 Jun 2023 06:53:12 +0000 (UTC), zeurkous@blaatscaahp.org wrote:
> Haai,
>
> Me thinks it's about time for me to speak up.
>
> On Wed, 7 Jun 2023 23:52:59 -0400, Agron Selimaj <as9902613@gmail.com> wrote:
>> What if we take the long path by starting small?
>>
>> Let me add gettext support, and we don't have to use it everywhere from day
>> one.

Adding a feature like this, which will by all probability never be
complete, will create a major rabbit hole for us to go down in.

At the very best, we'll be swarmed by user patches that we cannot verify
(does anyone on the workers list speak Cree or Armenian?) -- for all we
know, we'll put be approving all sorts of "funny" insults, and be
completely unaware of it --, for a somewhat tangential "feature"; at
worst, it'll be limited to the couple of easily-translated phrases, in
the natlangs the developers know, leaving mere mortals to bang their
heads in frustration when a message pops up that has not been
translated (or sometimes: /cannot/ be translated, due to sufficiently
subtle connotations that do not exist in the target language).

Either way, like mesaid: it's a major rabbit hole. One perhaps not worth
going into.

        --zeurkous.

-- 
Friggin' Machines!

Re: Is there any desire to support languages other than English in zsh?

[Oliver Kiddle]

Agron Selimaj wrote:
> What if we take the long path by starting small?
>
> Let me add gettext support, and we don't have to use it everywhere from day
> one.

This seems like a sensible approach to me. Adding gettext(), is mostly
just about identifying information messages so doesn't need extensive
knowledge of the existing internals. A --disable-nls configure option
can be added to satisfy anyone who is concerned about the implied bloat.

There may be border-line cases such as the prompts for history searching
("bck-i-search:") where we may want configurability at a zsh associative
array level like Peter was suggesting. But that can perhaps come later.

> This means, I'll make this text be extractable to generate *.POT files and then
> we can use all kinds of tools and websites to do the actual translation.

I'm vaguely aware that websites exist to allow non-technical people to
contribute translations. Do you know of any good ones? It would be best
if we have an individual or small team that keep that in step with the
git sources going forward so that people interested in the C internals
need not be too concerned with the translations.

> On Wed, Jun 7, 2023 at 10:56 PM Bart Schaefer <[1]schaefer@brasslantern.com>
> wrote:
>
>     Having previously overseen the gettext-ification of a piece of
>     software of similar original vintage and complexity, I would say that
>     supporting the databases is often the least of the issues.  Sentence
>     structure varies, which requires converting catenated substrings into
>     snprintf formats or sets of alternatives; pluralization may differ,
>     requiring 0 / 1 / >1 conditionals even with gettext intervention; etc.

Isn't that what ngettext() is for?
The domains - dgettext() - looks like something we might need to be
concerned with. Should modules use a different domain?

>     Also, because of the pervasive changes required, we found it necessary
>     to merge all outstanding code changes and call a halt to other
>     development until the "l10n" pass was complete, or we couldn't track
>     deltas accurately (though we were stuck with CVS for version control
>     at the time).

I can understand that having been necessary. It may matter less in zsh
if the pass is incomplete. And as software goes we probably have
relatively few user-readable strings (unless you count completion which
I assume is firmly out of scope for this).

>     This is definitely not something we would want to dive into until
>     after whatever the next zsh release is, is done.

I don't think its especially helpful to put off any particular efforts
on that basis. If the release manager feels that some recent work is
half-baked, git makes it very easy to create a release branch off from
some earlier point and backout or cherry-pick changes. Or something new
can be diverted into a temporary feature branch. Currently, I'm not
aware of any vague plans around timing for whatever the next release
will be but that would be a separate discussion.

zeurkous@blaatscaahp.org wrote:
| > Let me add gettext support, and we don't have to use it everywhere from day
| > one.

| An incomplete, underdesigned "feature" upsets people more than a lack of
| such, IME. Besides, sprinkling gettext calls around just leads to more
| bloat, and more importantly, is inconsistent with the existing design

I don't see that as leaving the feature "underdesigned". It'd be fully
designed, it is just the coverage of strings that may be incomplete
("from day one"). Our coverage of the world's many languages can never be
complete. It'd probably be best to have decent coverage of strings and
at least a few common languages covered before it makes it to a release.
gettext() calls can be substituted by a macro that does essentially
nothing so no bloat is forced on binary builds.

Oliver

Re: Is there any desire to support languages other than English in zsh?

[Peter Stephenson]

Thanks, Oliver, this reduces the responses I need to write to "me too"...

> On 08/06/2023 11:13 Oliver Kiddle <opk@zsh.org> wrote:
> Agron Selimaj wrote:
> > What if we take the long path by starting small?
> >
> > Let me add gettext support, and we don't have to use it everywhere from day
> > one.
> 
> This seems like a sensible approach to me.

Yes, I agree.  Adding this and checking the basics compile is already useful
work and doesn't need to be visible to the user immediately.

Your efforts in this direction would certainly be hugely appreciated by
the developers, thank you.  (I take it you're already on zsh-workers?)

> >     This is definitely not something we would want to dive into until
> >     after whatever the next zsh release is, is done.
> 
> I don't think its especially helpful to put off any particular efforts
> on that basis.

I would tend to agree that as long as Agron is providing framework support
it can continue.  Branches are cheap if we decide we need them.  A roll
out of the feature is another matter entirely, but that's a very long
way off at this point.

It might even be useful to be able to check that our configure-time checks
are working with all the distributions --- if they're not this time then there's
no impact on users, so no panic^H^H^H^H^Hcarefully thought out update.

Thanks again
pws

RE: Re: Is there any desire to support languages other than English in zsh?

[zeurkous]

On Thu, 8 Jun 2023 11:26:56 +0100 (BST), Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> Your efforts in this direction would certainly be hugely appreciated by
> the developers, thank you.  (I take it you're already on zsh-workers?)

That would seem to imply you speak for everyone. Is this true?

(Me's contributed only 1 patch and thus me'll decline to be considered
 as one of "the developers" <g>.) 

>> >     This is definitely not something we would want to dive into until
>> >     after whatever the next zsh release is, is done.
>>=20
>> I don't think its especially helpful to put off any particular efforts
>> on that basis.

Depends on how invasive it'll be in practice, me'd say. Certainly a
sample patch would help...

> I would tend to agree that as long as Agron is providing framework support
> it can continue.  Branches are cheap if we decide we need them.  A roll
> out of the feature is another matter entirely, but that's a very long
> way off at this point.

The question is still if we can support these efforts long-term, even if 
Agron becomes unable to, or plainly loses interest.

> It might even be useful to be able to check that our configure-time checks
> are working with all the distributions --- if they're not this time then th=
> ere's
> no impact on users, so no panic^H^H^H^H^Hcarefully thought out update.

Yup.

But overall: me'd strongly suggest that me earlier objections &
observations be considered more carefully before comitting to the
proposed venture.

We certainly should not dive for this too eagerly.

        --zeurkous.

-- 
Friggin' Machines!

RE: Re: Is there any desire to support languages other than English in zsh?

[zeurkous]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3994 bytes --]

On Thu, 08 Jun 2023 12:13:41 +0200, Oliver Kiddle <opk@zsh.org> wrote:
> Agron Selimaj wrote:
>> What if we take the long path by starting small?
>>
>> Let me add gettext support, and we don't have to use it everywhere from day
>> one.
>
> This seems like a sensible approach to me. Adding gettext(), is mostly
> just about identifying information messages so doesn't need extensive
> knowledge of the existing internals. A --disable-nls configure option
> can be added to satisfy anyone who is concerned about the implied bloat.

The bloat is also on the source level. This can make maintaining code
{,significantly }harder.

> There may be border-line cases such as the prompts for history searching
> ("bck-i-search:") where we may want configurability at a zsh associative
> array level like Peter was suggesting. But that can perhaps come later.

Given the already relatively self-contained nature of zsh, me'd argue
for the associative array approach, zeroth.

>> This means, I'll make this text be extractable to generate *.POT files and then
>> we can use all kinds of tools and websites to do the actual translation.
>
> I'm vaguely aware that websites exist to allow non-technical people to
> contribute translations. Do you know of any good ones? It would be best
> if we have an individual or small team that keep that in step with the
> git sources going forward so that people interested in the C internals
> need not be too concerned with the translations.

Dear $DEITY, please let's not let people contribute translations
unchecked at all, and on top of that, "non-technical" people tend to get
"technical" things wrong.

Very wrong. Misleading and incorrect translations will be a fact of
life.

>>     This is definitely not something we would want to dive into until
>>     after whatever the next zsh release is, is done.
>
> I don't think its especially helpful to put off any particular efforts
> on that basis. If the release manager feels that some recent work is
> half-baked, git makes it very easy to create a release branch off from
> some earlier point and backout or cherry-pick changes. Or something new
> can be diverted into a temporary feature branch. Currently, I'm not
> aware of any vague plans around timing for whatever the next release
> will be but that would be a separate discussion.

While seeming simple initially, these kind of changes have the potential
to become quite invasive. Perhaps deferring any such work is best, but
a feature branch could work.

> | An incomplete, underdesigned "feature" upsets people more than a lack of
> | such, IME. Besides, sprinkling gettext calls around just leads to more
> | bloat, and more importantly, is inconsistent with the existing design
>
> I don't see that as leaving the feature "underdesigned". It'd be fully
> designed,

That sounds like a pipe dream for now. Working w/ natlangs is
notoriously difficult, even for native speakers (most of them haven't
quite mastered their mother tongue{,s}, either). Giving the amount of
messages in zsh, we'll be finding grammar problems forever. And me'll
hazard the prediction that some of them just cannot be fixed without
compromising the exact meaning (mehopes medoesn't have to elaborate on
how very important exact meanings are in computing).

> it is just the coverage of strings that may be incomplete
> ("from day one"). Our coverage of the world's many languages can never be
> complete.

Yes. And that leads to no end of work, while we're already struggling to
manage the many features we have (and especially their interactions). 

We'd be inducing expectations that we cannot hope to ever fulfill.

> It'd probably be best to have decent coverage of strings and
> at least a few common languages covered before it makes it to a release.
> gettext() calls can be substituted by a macro that does essentially
> nothing so no bloat is forced on binary builds.

A macro hides bloat; it does not remove it.

        --zeurkous.

-- 
Friggin' Machines!