Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Sven Wischnowsky]

Tanaka Akira wrote:

> Thanks.
> 
> > +    tmp1=( ${tmp1//(#b)([][()|*?^#~<>])/\\${match[1]}} )
> 
> I tried test some special characters other than # and found that > and
> \ has a problem.

Rats. I had tried some others, too, but missed (inter alia) these two.

> Z(2):akr@flux% Src/zsh -f
> flux% bindkey -e; autoload -U compinit; compinit -D; compdef _tst tst
> flux% mkdir -p '>aa/bb' '\cc/dd'
> flux% ls -ld '>aa' '\cc'
> drwxr-xr-x  3 akr  akr  512 Aug  3 20:46 >aa
> drwxr-xr-x  3 akr  akr  512 Aug  3 20:46 \cc
> flux% ls \>a<TAB>
> flux% ls \\c<TAB>
> 
> They completes nothing.
> 
> Although I tried to change the character range to [][()|*?^#~<\\], the
> problem is not fixed.

It's *much* worse:

  % ls
  <aa  >bb
  % a='\<*' b='\>*'
  % echo $~a
  <aa
  % echo $~b
  zsh: no match
  % setopt shglob
  % echo $~a
  zsh: no match
  % echo $~b
  zsh: no match
  %

All this is caused by tokenize() treating backslashes differently.

I wanted to not commit this patch right now, but to compile the shell
I had to fix a pathmax bug in parameter.c and that has to go in. So,
let's just try it. Most of the patch is for unsetopting shglob in
completion functions and for _expand and _path_files to get quoting
right. At least I've tried several things needing quoting and they
were all handled correctly.


There is another problem with the pathmax stuff: with the files
module, doing `mkdir foo' fails because it tries to get the pathname
length of a non-existing directory. Haven't checked if there are other 
builtins with this problem.


Bye
 Sven

Index: Completion/Commands/_bash_completions
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_bash_completions,v
retrieving revision 1.4
diff -u -r1.4 _bash_completions
--- Completion/Commands/_bash_completions	2000/06/22 08:42:36	1.4
+++ Completion/Commands/_bash_completions	2000/08/03 13:29:13
@@ -25,7 +25,7 @@
 # that will not have been overridden, so you should add '~' to the
 # list of keys at the top of the for-loop.
 
-setopt localoptions nullglob rcexpandparam extendedglob
+setopt localoptions nullglob rcexpandparam extendedglob noshglob
 unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
 local key=$KEYS[-1] expl
Index: Completion/Commands/_complete_debug
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_complete_debug,v
retrieving revision 1.7
diff -u -r1.7 _complete_debug
--- Completion/Commands/_complete_debug	2000/05/31 09:56:12	1.7
+++ Completion/Commands/_complete_debug	2000/08/03 13:29:13
@@ -1,6 +1,6 @@
 #compdef -k complete-word \C-x?
 
-setopt localoptions nullglob rcexpandparam extendedglob
+setopt localoptions nullglob rcexpandparam extendedglob noshglob
 unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
 setopt localtraps noerrexit ; trap - ZERR
Index: Completion/Commands/_complete_help
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_complete_help,v
retrieving revision 1.8
diff -u -r1.8 _complete_help
--- Completion/Commands/_complete_help	2000/05/23 08:54:30	1.8
+++ Completion/Commands/_complete_help	2000/08/03 13:29:13
@@ -1,7 +1,7 @@
 #compdef -k complete-word \C-xh
 
 _complete_help() {
-  setopt localoptions nullglob rcexpandparam extendedglob
+  setopt localoptions nullglob rcexpandparam extendedglob noshglob
   unsetopt markdirs globsubst shwordsplit nounset ksharrays
   exec </dev/null	# ZLE closes stdin, which can cause errors
 
Index: Completion/Commands/_correct_word
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_correct_word,v
retrieving revision 1.2
diff -u -r1.2 _correct_word
--- Completion/Commands/_correct_word	2000/06/07 06:47:40	1.2
+++ Completion/Commands/_correct_word	2000/08/03 13:29:13
@@ -7,7 +7,7 @@
 # If configurations keys with the prefix `correctword_' are
 # given they override those starting with `correct_'.
 
-setopt localoptions nullglob rcexpandparam extendedglob
+setopt localoptions nullglob rcexpandparam extendedglob noshglob
 unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
 local curcontext="$curcontext"
Index: Completion/Commands/_expand_word
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_expand_word,v
retrieving revision 1.3
diff -u -r1.3 _expand_word
--- Completion/Commands/_expand_word	2000/06/07 06:47:40	1.3
+++ Completion/Commands/_expand_word	2000/08/03 13:29:13
@@ -2,7 +2,7 @@
 
 # Simple completion front-end implementing expansion.
 
-setopt localoptions nullglob rcexpandparam extendedglob
+setopt localoptions nullglob rcexpandparam extendedglob noshglob
 unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
 local curcontext="$curcontext"
Index: Completion/Commands/_history_complete_word
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_history_complete_word,v
retrieving revision 1.6
diff -u -r1.6 _history_complete_word
--- Completion/Commands/_history_complete_word	2000/06/22 08:42:36	1.6
+++ Completion/Commands/_history_complete_word	2000/08/03 13:29:13
@@ -16,7 +16,7 @@
 #   range -- range of history words to complete
 
 _history_complete_word () {
-  setopt localoptions nullglob rcexpandparam extendedglob
+  setopt localoptions nullglob rcexpandparam extendedglob noshglob
   unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
   local expl direction stop curcontext="$curcontext"
Index: Completion/Commands/_next_tags
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Commands/_next_tags,v
retrieving revision 1.8
diff -u -r1.8 _next_tags
--- Completion/Commands/_next_tags	2000/08/01 08:27:18	1.8
+++ Completion/Commands/_next_tags	2000/08/03 13:29:13
@@ -3,7 +3,7 @@
 # Main widget.
 
 _next_tags() {
-  setopt localoptions nullglob rcexpandparam extendedglob
+  setopt localoptions nullglob rcexpandparam extendedglob noshglob
   unsetopt markdirs globsubst shwordsplit nounset ksharrays
 
   local ins ops="$PREFIX$SUFFIX"
Index: Completion/Core/_expand
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Core/_expand,v
retrieving revision 1.26
diff -u -r1.26 _expand
--- Completion/Core/_expand	2000/07/26 09:16:13	1.26
+++ Completion/Core/_expand	2000/08/03 13:29:13
@@ -63,7 +63,7 @@
       eval exp\=\( ${${(q)exp}:gs/\\{/\{/:gs/\\}/\}/} \)
   eval 'exp=( ${${(e)exp//\\[ 	
 ]/ }//(#b)([ 	
-])/\\$match[1]} )' 2>/dev/null
+\\])/\\$match[1]} )' 2>/dev/null
 else
   exp=( ${exp:s/\\\$/\$} )
 fi
@@ -77,7 +77,7 @@
 # Now try globbing.
 
 [[ "$force" = *g* ]] || zstyle -T ":completion:${curcontext}:" glob &&
-    eval 'exp=( ${~exp} )' 2>/dev/null
+    eval 'exp=( ${~exp} ); exp=( ${exp//(#b)([][()|*?^#~<>\\])/\\${match[1]}} )' 2>/dev/null
 
 # If we don't have any expansions or only one and that is the same
 # as the original string, we let other completers run.
Index: Completion/Core/_main_complete
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Core/_main_complete,v
retrieving revision 1.39
diff -u -r1.39 _main_complete
--- Completion/Core/_main_complete	2000/07/27 10:59:44	1.39
+++ Completion/Core/_main_complete	2000/08/03 13:29:13
@@ -16,7 +16,7 @@
 # which makes the output of setopt and unsetopt reflect a different
 # state than the global one for which you are completing.
 
-setopt localoptions nullglob rcexpandparam extendedglob
+setopt localoptions nullglob rcexpandparam extendedglob noshglob
 unsetopt markdirs globsubst shwordsplit nounset ksharrays
 exec </dev/null	# ZLE closes stdin, which can cause errors
 
Index: Completion/Core/_path_files
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Core/_path_files,v
retrieving revision 1.28
diff -u -r1.28 _path_files
--- Completion/Core/_path_files	2000/08/03 11:12:31	1.28
+++ Completion/Core/_path_files	2000/08/03 13:29:13
@@ -421,7 +421,7 @@
     # There are more components, so skip over the next components and make a
     # slash be added.
 
-    tmp1=( ${tmp1//(#b)([][()|*?^#~<>])/\\${match[1]}} )
+    tmp1=( ${tmp1//(#b)([][()|*?^#~<>\\])/\\${match[1]}} )
     tmp2="${(M)tpre##((.|..|)/)##}"
     if [[ -n "$tmp2" ]]; then
       skipped="/$tmp2"
Index: Completion/Core/compdump
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Core/compdump,v
retrieving revision 1.7
diff -u -r1.7 compdump
--- Completion/Core/compdump	2000/07/04 16:17:00	1.7
+++ Completion/Core/compdump	2000/08/03 13:29:13
@@ -14,7 +14,7 @@
 # to see if auto-dump should re-dump the dump-file.
 
 emulate -L zsh
-setopt extendedglob
+setopt extendedglob noshglob
 
 typeset _d_file _d_f _d_bks _d_line _d_als _d_files
 
Index: Src/glob.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/glob.c,v
retrieving revision 1.9
diff -u -r1.9 glob.c
--- Src/glob.c	2000/06/02 10:09:13	1.9
+++ Src/glob.c	2000/08/03 13:29:14
@@ -2388,6 +2388,7 @@
 	case ')':
 	    if (isset(SHGLOB))
 		break;
+	case '>':
 	case '^':
 	case '#':
 	case '~':
Index: Src/Modules/parameter.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/parameter.c,v
retrieving revision 1.13
diff -u -r1.13 parameter.c
--- Src/Modules/parameter.c	2000/08/02 14:20:42	1.13
+++ Src/Modules/parameter.c	2000/08/03 13:29:15
@@ -1465,9 +1465,9 @@
 #endif
 	    if (!(val = getstrvalue(&v)) || *val != '/' ||
 #ifdef HAVE_PATHCONF
-		strlen(val) >= PATH_MAX)
-#else
                 ((strlen(val) >= pathmax)) && pathmax != -1)
+#else
+		strlen(val) >= PATH_MAX)
 #endif
 		zwarn("invalid value: %s", val, 0);
 	    else

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Clint Adams]

> There is another problem with the pathmax stuff: with the files
> module, doing `mkdir foo' fails because it tries to get the pathname
> length of a non-existing directory. Haven't checked if there are other 
> builtins with this problem.

I tested this, and it worked fine.  Perhaps this means that glibc 2.1.3 is
broken.  I just now tried it under FreeBSD 4.0, and OpenBSD 2.7, and ran
into the error you mentioned.

I think that statting to find the mount point would be too wasteful.
Dropping the tail could work except for things like mkdir -p.

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Clint Adams]

> I tested this, and it worked fine.  Perhaps this means that glibc 2.1.3 is

Yup, this is the relevant code in pathconf().

    case _PC_PATH_MAX:
#ifdef  PATH_MAX
      return PATH_MAX;
#else
      return -1;
#endif

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Tanaka Akira]

In article <200008031330.PAA22832@beta.informatik.hu-berlin.de>,
  Sven Wischnowsky <wischnow@informatik.hu-berlin.de> writes:

> Rats. I had tried some others, too, but missed (inter alia) these two.

\ has the problem yet.

Z(2):akr@flux% Src/zsh -f  
flux% bindkey -e; autoload -U compinit; compinit -D
flux% mkdir -p '\ab/cd'
flux% ls \\<TAB>

This completes nothing.

I made a test to check this problem and found other problems.

flux% mkdir -p '=ab/cd'
flux% ls \=<TAB>
_path_files:327: * not found

Note that the completion with = behaves curiously as follows.

flux% ls ./=<TAB>
flux% ls ./\=ab/<TAB>
flux% ls ./=ab/cd/

Also note that metafied characters (0x83 to 0x9c) is printed in
metafied form.

The test is follows but I'll not commit it because I feel it's too
slow.  It spends over two minutes on my machine until first test
(0x00 - 0x7f) is failed.

Index: 53completion.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/53completion.ztst,v
retrieving revision 1.7
diff -u -r1.7 53completion.ztst
--- 53completion.ztst	2000/07/17 10:36:00	1.7
+++ 53completion.ztst	2000/08/04 21:33:47
@@ -13,8 +13,61 @@
   touch file1
   touch file2
 
+  cd dir1
+  setopt braceccl
+  dirs=()
+  for x in {0-9a-f}{0-9a-f}
+  do
+    if [[ $x == 00 || $x == 2f ]]; then continue; fi
+    eval "ch=\$'\\x$x'"
+    dirs=("$dirs[@]" "./${ch}ab")
+  done
+  mkdir $dirs ${^dirs}/cd      
+  cd ..
+
+  special_character_test() {
+    sp_input=''
+    sp_output=''
+    for x in "$@" # {0-9a-f}{0-9a-f}
+    do
+      eval "ch=\$'\\x$x'"
+      case $x in
+      00|2f) continue;; # filename cannot contain `\0' or `/'.
+      09) ch_input=$'\\\C-v'"$ch" ch_output="\\$ch";;
+      0a) continue;; # cannot input `\n'.
+      0[1-8b-f]|1[0-9a-f]|7f) ch_input=$'\C-v'"$ch" ch_output="$ch";;
+      2[012346789a]|3[bcdef]|5[bcde]|60|7[bcde]) ch_input="\\$ch" ch_output="\\$ch";;
+      *) ch_input="$ch" ch_output="$ch";;
+      esac
+      sp_input="$sp_input: $ch_input"$'\t\t\C-u'
+      sp_output="${sp_output}line: {: ${ch_output}ab/}{}"$'\n'"line: {: ${ch_output}ab/cd/}{}"$'\n'
+    done
+  }
+ 
+  special_character_test {0-7}{0-9a-f}
+  sp_input1="$sp_input"
+  sp_output1="$sp_output"
+
+  special_character_test {89a-f}{0-9a-f}
+  sp_input2="$sp_input"
+  sp_output2="$sp_output"
+
 %test
 
+  # stty is required to input C-c, C-q and C-s.
+  comptesteval $'stty intr "" stop "" start ""'
+  comptesteval $'cd dir1'
+  comptest "$sp_input1"
+  echo
+0q:filename with special characters (0x00 - 0x7f)
+>$sp_output1
+
+  comptest "$sp_input2"
+  echo
+0q:filename with special characters (0x80 - 0xff)
+>$sp_output2
+
+  comptesteval 'cd ..'
   comptest $': \t\t\t\t\t\t\t'
 0:directories and files
 >line: {: }{}
-- 
Tanaka Akira

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Clint Adams]

>   For pathconf(), the path parameter points to the pathname of a file or
>   directory.  Read, write, or execute permission of the	named file is not
>   required, but	all directories	in the path leading to the file	must be
>   searchable.

This is valid; it needs to determine the mount point.  glibc is broken
for having a hardcoded return value.

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Clint Adams]

> Or fall back to _POSIX_PATH_MAX (if system supports pathconf(), it rorbably
> suports it as well):

Unless it supports unlimited path lengths, which is actually the point
of the mess that Bart cleaned up.

RE: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Andrej Borsenkow]

>
> There are a couple of things we could do.
>
> One, we could pitch out the path length test and simply let domkdir()
> fail -- and perhaps test the value of errno to decide whether to break
> or continue as a result.
>
> Two, we could cause zpathmax() to lop off the tail of the path and call
> itself until it actually does find a maximum path length, then add the
> tail back on and repeat the strlen() test.  That might be a good thing
> to have for the general case anyway.
>
> Or we could do both of the above.
>

Or fall back to _POSIX_PATH_MAX (if system supports pathconf(), it rorbably
suports it as well):

PATH_MAX
Maximum number of bytes in a pathname, including the terminating null
character. Minimum Acceptable Value: _POSIX_PATH_MAX

-andrej

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Sven Wischnowsky]

Bart Schaefer wrote:

> On Aug 4,  7:16am, Bart Schaefer wrote:
> } Subject: Re: PATCH: Re: completion problem with filename including # (and 
> }
> } One, we could pitch out the path length test and simply let domkdir()
> } fail -- and perhaps test the value of errno to decide whether to break
> } or continue as a result.
> 
> Sorry, that doesn't work if the goal is that no directories in the path
> get created if the whole path can't be created.  So that leaves us to
> dig around for a max path length.
> 
> I suspect the True64 pathconf() is actually broken in this respect, Sven.

It probably depends on how one defines `broken'. It *is*
documented. From the manual:


  For pathconf(), the path parameter points to the pathname of a file or
  directory.  Read, write, or execute permission of the	named file is not
  required, but	all directories	in the path leading to the file	must be
  searchable.

  ...

  If name is an	invalid	value, both pathconf() and fpathconf() return -1 and
  errno	is set to indicate the error.


And if mkdir calls pathconf on the name it should create...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Bart Schaefer]

On Aug 4,  7:16am, Bart Schaefer wrote:
} Subject: Re: PATCH: Re: completion problem with filename including # (and 
}
} One, we could pitch out the path length test and simply let domkdir()
} fail -- and perhaps test the value of errno to decide whether to break
} or continue as a result.

Sorry, that doesn't work if the goal is that no directories in the path
get created if the whole path can't be created.  So that leaves us to
dig around for a max path length.

I suspect the True64 pathconf() is actually broken in this respect, Sven.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Bart Schaefer]

On Aug 4,  9:01am, Sven Wischnowsky wrote:
} Subject: Re: PATCH: Re: completion problem with filename including # (and 
}
} > > There is another problem with the pathmax stuff: with the files
} > > module, doing `mkdir foo' fails because it tries to get the pathname
} > > length of a non-existing directory.
} > 
} > I think that statting to find the mount point would be too wasteful.
} > Dropping the tail could work except for things like mkdir -p.
} 
} But we need some kind of solution. I'm getting this with a normal
} True64 Unix installation (so it has nothing to do with any glibc
} whatsoever).

There are a couple of things we could do.

One, we could pitch out the path length test and simply let domkdir()
fail -- and perhaps test the value of errno to decide whether to break
or continue as a result.

Two, we could cause zpathmax() to lop off the tail of the path and call
itself until it actually does find a maximum path length, then add the
tail back on and repeat the strlen() test.  That might be a good thing
to have for the general case anyway.

Or we could do both of the above.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: PATCH: Re: completion problem with filename including # (and pathmax stuff)

[Sven Wischnowsky]

Clint Adams wrote:

> > There is another problem with the pathmax stuff: with the files
> > module, doing `mkdir foo' fails because it tries to get the pathname
> > length of a non-existing directory. Haven't checked if there are other 
> > builtins with this problem.
> 
> I tested this, and it worked fine.  Perhaps this means that glibc 2.1.3 is
> broken.  I just now tried it under FreeBSD 4.0, and OpenBSD 2.7, and ran
> into the error you mentioned.
> 
> I think that statting to find the mount point would be too wasteful.
> Dropping the tail could work except for things like mkdir -p.

But we need some kind of solution. I'm getting this with a normal
True64 Unix installation (so it has nothing to do with any glibc
whatsoever).

And I want my file-module builtins back ;-)


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de