Subversion completion don't work with UTF8 (and other) file names

Subversion completion don't work with UTF8 (and other) file names

[Paul Romanchenko]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

When trying to autocomplete svn operations in path containing UTF8 file
names, the following error occurs:

$ svn svn: E000022: Can't convert string from native encoding to 'UTF-8':
svn: E000022: /path/to/working/copy/2.?\208?\160...

I suppose this is because of $(LC_ALL=C _call_program....) in _subversion
completion code.

-- 
rmrfchik.

Re: Subversion completion don't work with UTF8 (and other) file names

[Vincent Lefevre]

On 2013-04-26 15:51:30 +0400, Paul Romanchenko wrote:
> When trying to autocomplete svn operations in path containing UTF8 file
> names, the following error occurs:
> 
> $ svn svn: E000022: Can't convert string from native encoding to 'UTF-8':
> svn: E000022: /path/to/working/copy/2.?\208?\160...
> 
> I suppose this is because of $(LC_ALL=C _call_program....) in _subversion
> completion code.

Since the error message is about native encoding (US-ASCII here,
due to the LC_ALL=C) to UTF-8, it concerns a file that is in the
repository with a non-ASCII filename (internal encoding is UTF-8).

A solution might be to retrieve the computed LC_CTYPE with the
"locale" command, then before executing svn, do the following:

_ Unset LC_ALL
_ Set LC_CTYPE to the locale determined above.
_ Set every other LC_* and LANG environment variables to "C".

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Subversion completion don't work with UTF8 (and other) file names

[Peter Stephenson]

On Fri, 26 Apr 2013 14:39:21 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> Since the error message is about native encoding (US-ASCII here,
> due to the LC_ALL=C) to UTF-8, it concerns a file that is in the
> repository with a non-ASCII filename (internal encoding is UTF-8).
> 
> A solution might be to retrieve the computed LC_CTYPE with the
> "locale" command, then before executing svn, do the following:
> 
> _ Unset LC_ALL
> _ Set LC_CTYPE to the locale determined above.
> _ Set every other LC_* and LANG environment variables to "C".

Unsetting all the LC_* variables (including LC_ALL) and setting only
LC_CTYPE and LANG should be good enough, shouldn't it?  Something like:

_comp_locale() {
   # This exports new locale settings, so should only
   # be run in a subshell.  A typical use is in a $(...).
   local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
   unset -m LC_\*
   [[ -n $ctype ]] && eval export $ctype
   export LANG=C
}

seems to do the trick here:

LANG=C
LC_CTYPE=en_GB.UTF-8
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

I've actually lost track of which of the above we're trying to fix up
when we set the locale in this sort of case.  LC_COLLATE, certainly,
and I think LC_MESSAGES since otherwise we can't parse it (although
the disadvantage here is the output may be in a language the user
doesn't understand).

pws

Re: Subversion completion don't work with UTF8 (and other) file names

[Vincent Lefevre]

On 2013-04-26 15:44:27 +0100, Peter Stephenson wrote:
> Unsetting all the LC_* variables (including LC_ALL) and setting only
> LC_CTYPE and LANG should be good enough, shouldn't it?  Something like:
> 
> _comp_locale() {
>    # This exports new locale settings, so should only
>    # be run in a subshell.  A typical use is in a $(...).
>    local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
>    unset -m LC_\*
>    [[ -n $ctype ]] && eval export $ctype
>    export LANG=C
> }

I think it should work, but these settings should affect only the
svn command; I suppose that you need to run that in a subshell.

Note that since LC_ALL is unset and LANG is set to C, the LANGUAGE
environment variable will be ignored with the GNU libc (so, there
is no need to do anything about it).

> I've actually lost track of which of the above we're trying to fix up
> when we set the locale in this sort of case.  LC_COLLATE, certainly,
> and I think LC_MESSAGES since otherwise we can't parse it (although
> the disadvantage here is the output may be in a language the user
> doesn't understand).

If this is done only for parsing, the user wouldn't see the message.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Subversion completion don't work with UTF8 (and other) file names

[Peter Stephenson]

On Fri, 26 Apr 2013 17:35:46 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> On 2013-04-26 15:44:27 +0100, Peter Stephenson wrote:
> > Unsetting all the LC_* variables (including LC_ALL) and setting only
> > LC_CTYPE and LANG should be good enough, shouldn't it?  Something like:
> > 
> > _comp_locale() {
> >    # This exports new locale settings, so should only
> >    # be run in a subshell.  A typical use is in a $(...).
> >    local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
> >    unset -m LC_\*
> >    [[ -n $ctype ]] && eval export $ctype
> >    export LANG=C
> > }
> 
> I think it should work, but these settings should affect only the
> svn command; I suppose that you need to run that in a subshell.

Here's a change for _subversion only, if this works we can expand the
use.

diff --git a/Completion/Base/Utility/.distfiles b/Completion/Base/Utility/.distfiles
index 97d86bd..91ddf88 100644
--- a/Completion/Base/Utility/.distfiles
+++ b/Completion/Base/Utility/.distfiles
@@ -1,9 +1,25 @@
 DISTFILES_SRC='
 .distfiles
-_alternative      _call_program     _nothing          _sub_commands
-_arg_compile      _combination      _regex_arguments  _values
-_arguments        _set_command      _retrieve_cache   _guard
-_cache_invalid    _describe         _sep_parts        _pick_variant
-_call_function    _multi_parts      _store_cache      _regex_words
+_alternative
+_arg_compile
+_arguments
+_cache_invalid
+_call_function
+_comp_locale
 _complete_help_generic
+_call_program
+_combination
+_set_command
+_describe
+_multi_parts
+_nothing
+_regex_arguments
+_retrieve_cache
+_sep_parts
+_store_cache
+_sub_commands
+_values
+_guard
+_pick_variant
+_regex_words
 '
diff --git a/Completion/Base/Utility/_comp_locale b/Completion/Base/Utility/_comp_locale
new file mode 100644
index 0000000..9273ad0
--- /dev/null
+++ b/Completion/Base/Utility/_comp_locale
@@ -0,0 +1,9 @@
+#autoload
+
+# This exports new locale settings, so should only
+# be run in a subshell.  A typical use is in a $(...).
+
+local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
+unset -m LC_\*
+[[ -n $ctype ]] && eval export $ctype
+export LANG=C
diff --git a/Completion/Unix/Command/_subversion b/Completion/Unix/Command/_subversion
index 11d0b69..0fa3e8c 100644
--- a/Completion/Unix/Command/_subversion
+++ b/Completion/Unix/Command/_subversion
@@ -20,7 +20,7 @@ _svn () {
     typeset -gHA _svn_cmds
     if _cache_invalid svn-cmds || ! _retrieve_cache svn-cmds; then
       _svn_cmds=(
-	${=${(f)${${"$(LC_ALL=C _call_program commands svn help)"#l#*Available subcommands:}%%Subversion is a tool*}}/(#s)[[:space:]]#(#b)([a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
+	${=${(f)${${"$(_comp_locale; _call_program commands svn help)"#l#*Available subcommands:}%%Subversion is a tool*}}/(#s)[[:space:]]#(#b)([a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
       )
       _store_cache svn-cmds _svn_cmds
     fi
@@ -41,14 +41,14 @@ _svn () {
 	if _cache_invalid svn-${cmd}-usage || \
 	    ! _retrieve_cache svn-${cmd}-usage;
 	then
-	  usage=${${(M)${(f)"$(LC_ALL=C _call_program options svn help $cmd)"}:#usage:*}#usage:*$cmd] }
+	  usage=${${(M)${(f)"$(_comp_locale; _call_program options svn help $cmd)"}:#usage:*}#usage:*$cmd] }
 	  _store_cache svn-${cmd}-usage usage
 	fi
 	if _cache_invalid svn-${cmd}-usage || \
 	    ! _retrieve_cache svn-${cmd}-args;
 	then
 	  args=(
-	    ${=${${${(M)${(f)"$(LC_ALL=C _call_program options svn help $cmd)"#(*Valid options:|(#e))}:#* :*}%% #:*}/ (arg|ARG)/:arg:}/(#b)(-##)([[:alpha:]]##) \[--([a-z-]##)\](:arg:)#/(--$match[3])$match[1]$match[2]$match[4] ($match[1]$match[2])--$match[3]$match[4]}
+	    ${=${${${(M)${(f)"$(_comple_local; _call_program options svn help $cmd)"#(*Valid options:|(#e))}:#* :*}%% #:*}/ (arg|ARG)/:arg:}/(#b)(-##)([[:alpha:]]##) \[--([a-z-]##)\](:arg:)#/(--$match[3])$match[1]$match[2]$match[4] ($match[1]$match[2])--$match[3]$match[4]}
 	  )
           while (( idx=$args[(I)*--c(l|hangelist):arg:] )); do
             args[(I)*--c(l|hangelist):arg:]=( \*{--cl,--changelist}':change list:_svn_changelists' )
@@ -157,7 +157,7 @@ _svnadmin () {
   if [[ -n $state ]] && (( ! $+_svnadmin_cmds )); then
     typeset -gHA _svnadmin_cmds
     _svnadmin_cmds=(
-      ${=${(f)${${"$(LC_ALL=C _call_program commands svnadmin help)"#l#*Available subcommands:}}}/(#s)[[:space:]]#(#b)([-a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
+      ${=${(f)${${"$(_comp_locale; _call_program commands svnadmin help)"#l#*Available subcommands:}}}/(#s)[[:space:]]#(#b)([-a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
     )
   fi
 
@@ -172,9 +172,9 @@ _svnadmin () {
       if (( $#cmd )); then
         curcontext="${curcontext%:*:*}:svnadmin-${cmd}:"
 
-        usage=${${(M)${(f)"$(LC_ALL=C _call_program options svnadmin help $cmd)"}:#$cmd: usage:*}#$cmd: usage: svnadmin $cmd }
+        usage=${${(M)${(f)"$(_comp_locale; _call_program options svnadmin help $cmd)"}:#$cmd: usage:*}#$cmd: usage: svnadmin $cmd }
         args=(
-          ${=${${${(M)${(f)"$(LC_ALL=C _call_program options svnadmin help $cmd)"#(*Valid options:|(#e))}:#* :*}%% #:*}/ (arg|ARG)/:arg:}/(#b)-([[:alpha:]]) \[--([a-z-]##)\](:arg:)#/(--$match[2])-$match[1]$match[3] (-$match[1])--$match[2]$match[3]}
+          ${=${${${(M)${(f)"$(_comp_locale; _call_program options svnadmin help $cmd)"#(*Valid options:|(#e))}:#* :*}%% #:*}/ (arg|ARG)/:arg:}/(#b)-([[:alpha:]]) \[--([a-z-]##)\](:arg:)#/(--$match[2])-$match[1]$match[3] (-$match[1])--$match[2]$match[3]}
         )
         if [[ $usage == *REPOS_PATH* ]]; then
           args+=( ":path:_files -/" )
@@ -319,7 +319,7 @@ _svn_props() {
 _svn_changelists() {
   local cls
 
-  cls=( ${${${(M)${(f)"$(LC_ALL=C _call_program changelists svn status 2>/dev/null)"}:#--- Changelist*}%??}##*\'} )
+  cls=( ${${${(M)${(f)"$(_comp_locale; _call_program changelists svn status 2>/dev/null)"}:#--- Changelist*}%??}##*\'} )
   compadd "$@" -a cls && return 0
 }
 

pws

Re: Subversion completion don't work with UTF8 (and other) file names

[Bart Schaefer]

On Apr 29,  9:57am, Peter Stephenson wrote:
} 
} Here's a change for _subversion only, if this works we can expand the
} use.

You've got one typo in there.   s/_comple_local/_comp_locale/

} +	    ${=${${${(M)${(f)"$(_comple_local; _call_program options svn help $cmd)"#(*Valid options:|(#e))}:#* :*}%% #:*}/ (arg|ARG)/:arg:}/(#b)(-##)([[:alpha:]]##) \[--([a-z-]##)\](:arg:)#/(--$match[3])$match[1]$match[2]$match[4] ($match[1]$match[2])--$match[3]$match[4]}

Re: Subversion completion don't work with UTF8 (and other) file names

[Oliver Kiddle]

On 29 Apr, Peter Stephenson wrote:
> > > Unsetting all the LC_* variables (including LC_ALL) and setting only
> > > LC_CTYPE and LANG should be good enough, shouldn't it?  Something like:
> > > 
> > > _comp_locale() {
> > >    # This exports new locale settings, so should only
> > >    # be run in a subshell.  A typical use is in a $(...).
> > >    local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
> > >    unset -m LC_\*
> > >    [[ -n $ctype ]] && eval export $ctype
> > >    export LANG=C
> > > }
> > 
> Here's a change for _subversion only, if this works we can expand the
> use.

Is the locale command that portable?.
I'm not sure this is even necessary in the cases where it is just calling
svnadmin help or svn help rather than svn status.

Originally, _subversion set LC_MESSAGES for these commands but LC_ALL
overrides LC_MESSAGES. Would it be better to do it as follows:
  LC_CTYPE=${LC_ALL:-${LC_CTYPE:-$LANG}} LC_ALL= LC_MESSAGES=C _call_program ...

Or should we change the patterns so they'll work even with output in
other languages. For svn status, the changelists are introduced with
lines starting '---' and for svn help, sub commands all start with
three spaces so that may be the best option.

Oliver

Re: Subversion completion don't work with UTF8 (and other) file names

[Peter Stephenson]

On Sat, 04 May 2013 04:15:01 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Is the locale command that portable?.

That's easy to work around.

> I'm not sure this is even necessary in the cases where it is just calling
> svnadmin help or svn help rather than svn status.
> 
> Originally, _subversion set LC_MESSAGES for these commands but LC_ALL
> overrides LC_MESSAGES. Would it be better to do it as follows:
>   LC_CTYPE=${LC_ALL:-${LC_CTYPE:-$LANG}} LC_ALL= LC_MESSAGES=C _call_program ...

So that goes back to my previous question, what do we actually need to
change to make it work regardless of locale?

diff --git a/Completion/Base/Utility/_comp_locale b/Completion/Base/Utility/_comp_locale
index 1987043..7cf8511 100644
--- a/Completion/Base/Utility/_comp_locale
+++ b/Completion/Base/Utility/_comp_locale
@@ -7,7 +7,14 @@
 # This exports new locale settings, so should only
 # be run in a subshell.  A typical use is in a $(...).
 
-local ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}
-unset -m LC_\*
-[[ -n $ctype ]] && eval export $ctype
+local ctype
+
+if ctype=${${(f)"$(locale 2>/dev/null)"}:#^LC_CTYPE=*}; then
+    unset -m LC_\*
+    [[ -n $ctype ]] && eval export $ctype
+else
+    ctype=${LC_ALL:-${LC_CTYPE:-$LANG}}
+    unset -m LC_\*
+    export LC_CTYPE=$ctype
+fi
 export LANG=C

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Subversion completion don't work with UTF8 (and other) file names

[Oliver Kiddle]

On 4 May, Peter Stephenson wrote:
> So that goes back to my previous question, what do we actually need to
> change to make it work regardless of locale?

For subcommands, lines starting with three spaces works. For
changelists, we can use svn status --xml. That breaks if the changelist
name needs XML entities (which it only uses for things like &<>") but
you can already confuse it with suitably contrived characters, anyway.

For everything else, it isn't so easy, mainly because we also match
strings like ARG, TARGET, URL etc to find out what to complete after the
option or subcommand. And those get translated. If anyone, is tempted to
try to get further on this, note that svn 1.7 translates slightly more
than 1.6 such as the word "usage".

Oliver

diff --git a/Completion/Unix/Command/_subversion b/Completion/Unix/Command/_subversion
index 597f232..014b0ac 100644
--- a/Completion/Unix/Command/_subversion
+++ b/Completion/Unix/Command/_subversion
@@ -20,7 +20,7 @@ _svn () {
     typeset -gHA _svn_cmds
     if _cache_invalid svn-cmds || ! _retrieve_cache svn-cmds; then
       _svn_cmds=(
-	${=${(f)${${"$(_comp_locale; _call_program commands svn help)"#l#*Available subcommands:}%%Subversion is a tool*}}/(#s)[[:space:]]#(#b)([a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
+        ${=${${${(M)${(f)"$(_call_program commands svn help)"}:#   [a-z]*}//[ )]/}//[(,]/:}/(#m)*/${MATCH%%:*} :$MATCH:}
       )
       _store_cache svn-cmds _svn_cmds
     fi
@@ -157,7 +157,7 @@ _svnadmin () {
   if [[ -n $state ]] && (( ! $+_svnadmin_cmds )); then
     typeset -gHA _svnadmin_cmds
     _svnadmin_cmds=(
-      ${=${(f)${${"$(_comp_locale; _call_program commands svnadmin help)"#l#*Available subcommands:}}}/(#s)[[:space:]]#(#b)([-a-z]##)[[:space:]]#(\([a-z, ?]##\))#/$match[1] :$match[1]${match[2]:+:${${match[2]//[(),]}// /:}}:}
+      ${=${${${(M)${(f)"$(_call_program commands svnadmin help)"}:#   [a-z]*}//[ )]/}//[(,]/:}/(#m)*/${MATCH%%:*} :$MATCH:}
     )
   fi
 
@@ -319,7 +319,7 @@ _svn_props() {
 _svn_changelists() {
   local cls
 
-  cls=( ${${${(M)${(f)"$(_comp_locale; _call_program changelists svn status 2>/dev/null)"}:#--- Changelist*}%??}##*\'} )
+  cls=( ${${${(M)${(s.<.)"$(_call_program changelists svn status --xml)"}:#changelist*}#*name=?}%%?>*} )
   compadd "$@" -a cls && return 0
 }
 

Re: Subversion completion don't work with UTF8 (and other) file names

[Peter Stephenson]

On Tue, 07 May 2013 09:38:35 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> On 4 May, Peter Stephenson wrote:
> > So that goes back to my previous question, what do we actually need to
> > change to make it work regardless of locale?
> 
> For subcommands, lines starting with three spaces works. For
> changelists, we can use svn status --xml.

Thanks for the patch --- I suppose I was asking the more general
question: given a utility we need output from, is it OK to keep LC_CTYPE
and set everything else to C or is there something else going on?

The bug report with subversion was the first I've seen in this area for
a while, so presumably anything else is pretty subtle.

I should try to get _comp_locale into other places where we currently
set the locale to C.

pws

Re: Subversion completion don't work with UTF8 (and other) file names

[Vincent Lefevre]

On 2013-05-07 10:21:15 +0100, Peter Stephenson wrote:
> I suppose I was asking the more general question: given a utility we
> need output from, is it OK to keep LC_CTYPE and set everything else
> to C or is there something else going on?

It depends on the utility. With some utilities, you may want to
set LC_CTYPE to C in order to avoid non-ASCII characters and/or
get charset-independent output.

For instance, with the US-ASCII charmap,

$ LANG=C LC_CTYPE=C cp foo
cp: missing destination file operand after 'foo'
Try 'cp --help' for more information.

but with the UTF-8 charmap,

$ LANG=C LC_CTYPE=C.UTF-8 cp foo
cp: missing destination file operand after ‘foo’
Try 'cp --help' for more information.

The quotes of foo are non-ASCII ones in the latter case.

But if the filename can have non-ASCII characters...

$ LANG=C LC_CTYPE=C cp aéb
cp: missing destination file operand after 'a\303\251b'
Try 'cp --help' for more information.

while one gets the accented character in UTF-8:

$ LANG=C LC_CTYPE=C.UTF-8 cp aéb
cp: missing destination file operand after ‘aéb’
Try 'cp --help' for more information.

but things get complex if the filename has special characters
(even ASCII ones)...

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)