* OpenSSH scp completion reminder @ 2022-04-14 15:20 Phil Pennock 2022-04-14 16:18 ` Axel Beckert 0 siblings, 1 reply; 7+ messages in thread From: Phil Pennock @ 2022-04-14 15:20 UTC (permalink / raw) To: zsh-workers I kept meaning to take a look at this, but haven't, so before I forget again ... OpenSSH 9.0 switches the default underlying application-layer transport for the scp command from the RCP protocol to the SFTP protocol. This is available as an option from OpenSSH 8.9, I believe. This change impacts quoting of things like whitespace. Which I have assumed will impact zsh completion. If someone has a _quick_ fix for this, it's probably worth getting it into the upcoming zsh release, so that there's not an extended period of time where there's an incompatibility. -Phil ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 15:20 OpenSSH scp completion reminder Phil Pennock @ 2022-04-14 16:18 ` Axel Beckert 2022-04-14 17:04 ` Bart Schaefer 2022-04-14 20:44 ` Phil Pennock 0 siblings, 2 replies; 7+ messages in thread From: Axel Beckert @ 2022-04-14 16:18 UTC (permalink / raw) To: zsh-workers Hi, On Thu, Apr 14, 2022 at 11:20:34AM -0400, Phil Pennock wrote: > OpenSSH 9.0 switches the default underlying application-layer transport > for the scp command from the RCP protocol to the SFTP protocol. This is > available as an option from OpenSSH 8.9, I believe. IIRC even since 8.8. > This change impacts quoting of things like whitespace. > Which I have assumed will impact zsh completion. I thought so as well, but I so far couldn't find anything which didn't work with zsh remote file completion with OpenSSH 9.0 on the client side. Otherwise I would have reported it. I at least tried tab-completing a file name with spaces, added a "." and pressed Enter and to my surprise it worked. > If someone has a _quick_ fix for this, it's probably worth getting it > into the upcoming zsh release, so that there's not an extended period of > time where there's an incompatibility. IMHO this would be very preferable, yes, at least if there's actually something to fix. Kind regards, Axel -- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe@deuxchevaux.org \ / Say No to HTML in E-Mail and Usenet Mail+Jabber: abe@noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 16:18 ` Axel Beckert @ 2022-04-14 17:04 ` Bart Schaefer 2022-04-14 17:27 ` Axel Beckert 2022-04-14 20:44 ` Phil Pennock 1 sibling, 1 reply; 7+ messages in thread From: Bart Schaefer @ 2022-04-14 17:04 UTC (permalink / raw) To: Zsh hackers list On Thu, Apr 14, 2022 at 9:43 AM Axel Beckert <abe@deuxchevaux.org> wrote: > > On Thu, Apr 14, 2022 at 11:20:34AM -0400, Phil Pennock wrote: > > This change impacts quoting of things like whitespace. > > Which I have assumed will impact zsh completion. > > I thought so as well, but I so far couldn't find anything which didn't > work with zsh remote file completion with OpenSSH 9.0 on the client > side. Completion itself doesn't use scp to do the remote file name generation, it uses plain ssh. Of course once the file names are returned, getting the quoting right for the actual scp invocation is down to what appears on the command line, so the only way completion could be implicated in messing this up is if it somehow automatically inserted the wrong quoting. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 17:04 ` Bart Schaefer @ 2022-04-14 17:27 ` Axel Beckert 2022-04-14 17:34 ` Mikael Magnusson 0 siblings, 1 reply; 7+ messages in thread From: Axel Beckert @ 2022-04-14 17:27 UTC (permalink / raw) To: zsh-workers Hi, On Thu, Apr 14, 2022 at 10:04:42AM -0700, Bart Schaefer wrote: > On Thu, Apr 14, 2022 at 9:43 AM Axel Beckert <abe@deuxchevaux.org> wrote: > > On Thu, Apr 14, 2022 at 11:20:34AM -0400, Phil Pennock wrote: > > > This change impacts quoting of things like whitespace. > > > Which I have assumed will impact zsh completion. > > > > I thought so as well, but I so far couldn't find anything which didn't > > work with zsh remote file completion with OpenSSH 9.0 on the client > > side. > > Completion itself doesn't use scp to do the remote file name > generation, it uses plain ssh. Of course once the file names are > returned, getting the quoting right for the actual scp invocation is > down to what appears on the command line, so the only way completion > could be implicated in messing this up is if it somehow automatically > inserted the wrong quoting. Yep, and that's the point: In theory, the amount of quoting (e.g. three backslashes vs one backslash) is expected to be different with the different scp backends. But for some reason the variant with three backslashes still worked for me with OpenSSH 9.0 on the client side which defaults to use the SFTP backend which does no shell globbing or escaping on the server side. To copy a file named "tmp/foo bar" from remotehost to my local directory "tmp/", I did: % scp -p remotehost:tmp/foo<Tab><Tab> tmp/<Enter> It expanded to: % scp -p remotehost:tmp/foo\\\ bar tmp/ Accoding to the docs, this does no more work with OpenSSH 9.0's default settings, but it did. Running "scp" with "-vvvv" confirms that the new default (SFTP backend) is used: scp -pvvvv remotehost:tmp/foo\\\ bar tmp/ |& fgrep tmp/foo scp: debug3: sink_sftp: copying remote tmp/foo\\ bar to local tmp/ scp: debug1: Fetching tmp/foo bar to tmp/foo bar scp: debug2: do_download: download remote "tmp/foo bar" to local "tmp/foo bar" scp: debug2: Sending SSH2_FXP_STAT "tmp/foo bar" scp: debug2: Sending SSH2_FXP_OPEN "tmp/foo bar" scp: debug3: Sent remote message SSH2_FXP_OPEN I:4 P:tmp/foo bar M:0x0001 It seems as if scp has some capability to recognize of there's too much shell quoting in a file name and reducres it, since there are still two backslashes in the first line with "debug3:" which vanish after that line. And this even seems to go wrong if I want to copy a file created with "touch 'foo\ bar'", i.e. with an backslash in there. I would have had expected that OpenSSH 9.0 would fetch that file with "scp -p remotehost:tmp/foo\\\ bar tmp/" but it actually stripped the file name down and I needed five backslashes: scp -pvvvv remotehost:tmp/foo\\\\\ bar tmp/ |& fgrep tmp/foo scp: debug3: sink_sftp: copying remote tmp/foo\\\\ bar to local tmp/ scp: debug1: Fetching tmp/foo\\ bar to tmp/foo\\ bar scp: debug2: do_download: download remote "tmp/foo\\ bar" to local "tmp/foo\\ bar" scp: debug2: Sending SSH2_FXP_STAT "tmp/foo\\ bar" scp: debug2: Sending SSH2_FXP_OPEN "tmp/foo\\ bar" scp: debug3: Sent remote message SSH2_FXP_OPEN I:4 P:tmp/foo\\ bar M:0x0001 Looks like either a bug in OpenSSH 9.0 or an undocumented compatibility feature to me. Kind regards, Axel -- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe@deuxchevaux.org \ / Say No to HTML in E-Mail and Usenet Mail+Jabber: abe@noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 17:27 ` Axel Beckert @ 2022-04-14 17:34 ` Mikael Magnusson 0 siblings, 0 replies; 7+ messages in thread From: Mikael Magnusson @ 2022-04-14 17:34 UTC (permalink / raw) To: zsh-workers On 4/14/22, Axel Beckert <abe@deuxchevaux.org> wrote: > Hi, > > On Thu, Apr 14, 2022 at 10:04:42AM -0700, Bart Schaefer wrote: >> On Thu, Apr 14, 2022 at 9:43 AM Axel Beckert <abe@deuxchevaux.org> wrote: >> > On Thu, Apr 14, 2022 at 11:20:34AM -0400, Phil Pennock wrote: >> > > This change impacts quoting of things like whitespace. >> > > Which I have assumed will impact zsh completion. >> > >> > I thought so as well, but I so far couldn't find anything which didn't >> > work with zsh remote file completion with OpenSSH 9.0 on the client >> > side. >> >> Completion itself doesn't use scp to do the remote file name >> generation, it uses plain ssh. Of course once the file names are >> returned, getting the quoting right for the actual scp invocation is >> down to what appears on the command line, so the only way completion >> could be implicated in messing this up is if it somehow automatically >> inserted the wrong quoting. > > Yep, and that's the point: In theory, the amount of quoting (e.g. > three backslashes vs one backslash) is expected to be different with > the different scp backends. But for some reason the variant with three > backslashes still worked for me with OpenSSH 9.0 on the client side > which defaults to use the SFTP backend which does no shell globbing or > escaping on the server side. > > To copy a file named "tmp/foo bar" from remotehost to my local > directory "tmp/", I did: > > % scp -p remotehost:tmp/foo<Tab><Tab> tmp/<Enter> > > It expanded to: > > % scp -p remotehost:tmp/foo\\\ bar tmp/ > > Accoding to the docs, this does no more work with OpenSSH 9.0's > default settings, but it did. Running "scp" with "-vvvv" confirms that > the new default (SFTP backend) is used: > > scp -pvvvv remotehost:tmp/foo\\\ bar tmp/ |& fgrep tmp/foo > scp: debug3: sink_sftp: copying remote tmp/foo\\ bar to local tmp/ > scp: debug1: Fetching tmp/foo bar to tmp/foo bar > scp: debug2: do_download: download remote "tmp/foo bar" to local "tmp/foo > bar" > scp: debug2: Sending SSH2_FXP_STAT "tmp/foo bar" > scp: debug2: Sending SSH2_FXP_OPEN "tmp/foo bar" > scp: debug3: Sent remote message SSH2_FXP_OPEN I:4 P:tmp/foo bar > M:0x0001 > > It seems as if scp has some capability to recognize of there's too > much shell quoting in a file name and reducres it, since there are > still two backslashes in the first line with "debug3:" which vanish > after that line. > > And this even seems to go wrong if I want to copy a file created with > "touch 'foo\ bar'", i.e. with an backslash in there. I would have had > expected that OpenSSH 9.0 would fetch that file with "scp -p > remotehost:tmp/foo\\\ bar tmp/" but it actually stripped the file name > down and I needed five backslashes: > > scp -pvvvv remotehost:tmp/foo\\\\\ bar tmp/ |& fgrep tmp/foo > scp: debug3: sink_sftp: copying remote tmp/foo\\\\ bar to local tmp/ > scp: debug1: Fetching tmp/foo\\ bar to tmp/foo\\ bar > scp: debug2: do_download: download remote "tmp/foo\\ bar" to local > "tmp/foo\\ bar" > scp: debug2: Sending SSH2_FXP_STAT "tmp/foo\\ bar" > scp: debug2: Sending SSH2_FXP_OPEN "tmp/foo\\ bar" > scp: debug3: Sent remote message SSH2_FXP_OPEN I:4 P:tmp/foo\\ bar > M:0x0001 > > Looks like either a bug in OpenSSH 9.0 or an undocumented > compatibility feature to me. It's definitely a bug (I'm using 8.9 though): % touch '\;\[' % scp -s localhost:\\\;\\\[ . scp: \\;\\[: No such file or directory % scp -s localhost:\;\[ . scp: ;[: No such file or directory It doesn't seem possible to copy files with backslashes in the name at all using -s. -- Mikael Magnusson ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 16:18 ` Axel Beckert 2022-04-14 17:04 ` Bart Schaefer @ 2022-04-14 20:44 ` Phil Pennock 2022-04-15 12:46 ` Axel Beckert 1 sibling, 1 reply; 7+ messages in thread From: Phil Pennock @ 2022-04-14 20:44 UTC (permalink / raw) To: zsh-workers On 2022-04-14 at 18:18 +0200, Axel Beckert wrote: > I thought so as well, but I so far couldn't find anything which didn't > work with zsh remote file completion with OpenSSH 9.0 on the client > side. Otherwise I would have reported it. What version was the server running? With localhost being 9.0 for client and server and zsh 5.8, I see this reproduction case; setup, then the results of tab-completion, and then at the end a manually-constructed filename which definitely would not have worked with older scp. % pwd /home/pdp/t3 % ls -lR .: total 4 drwxr-xr-x 2 pdp pdp 4096 Apr 14 16:31 'foo bar'$'\t''baz' './foo bar'$'\t''baz': total 8 -rw-r--r-- 1 pdp pdp 4785 Apr 14 16:31 'An Artist - Foo Bar Baz.mp3' % cd ~/Downloads/t % scp localhost:t3/<TAB><TAB><ENTER> % scp localhost:t3/foo\\\ bar\$\'\\t\'baz/An\\\ Artist\\\ -\\\ Foo\\\ Bar\\\ Baz.mp3 . /opt/openssh/bin/scp: t3/foo\\ bar$'\\t'baz/An\\ Artist\\ -\\ Foo\\ Bar\\ Baz.mp3: No such file or directory % scp $'localhost:t3/foo bar\tbaz/An Artist - Foo Bar Baz.mp3' . An Artist - Foo Bar Baz.mp3 100% 4785 2.1MB/s 00:00 % ssh -V OpenSSH_9.0p1, OpenSSL 1.1.1f 31 Mar 2020 So the good news is that scp in the new mode is "pass the filename exactly as it really is, no escaping for the remote side, whatever reaches us in argv[] is what will be used, so you only need to worry about local shell escaping". Note that scp(1) does not have a version flag, whereas ssh(1) does. It might be reasonable to expect them to be paired. Invoking `scp -O` exits 1 in both versions below, but you can see the output text differences. -O to use original SCP protocol, -s to use sftp protocol. -D is for local debugging. So I think full support means knowing the version, knowing which version switched the default behaviour (looks like 9.0 man-page might be wrong), and then knowing which options have been set on the command-line already for this version. OpenSSH 9.0: usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-J destination] [-l limit] [-o ssh_option] [-P port] [-S program] source ... target OpenSSH 8.6: /usr/bin/scp: unrecognized option: O usage: scp [-346ABCpqrTv] [-c cipher] [-F ssh_config] [-i identity_file] [-J destination] [-l limit] [-o ssh_option] [-P port] [-S program] source ... target -Phil ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: OpenSSH scp completion reminder 2022-04-14 20:44 ` Phil Pennock @ 2022-04-15 12:46 ` Axel Beckert 0 siblings, 0 replies; 7+ messages in thread From: Axel Beckert @ 2022-04-15 12:46 UTC (permalink / raw) To: zsh-workers Hi, On Thu, Apr 14, 2022 at 04:44:33PM -0400, Phil Pennock wrote: > On 2022-04-14 at 18:18 +0200, Axel Beckert wrote: > > I thought so as well, but I so far couldn't find anything which didn't > > work with zsh remote file completion with OpenSSH 9.0 on the client > > side. Otherwise I would have reported it. > > What version was the server running? Good point. I tried it with 8.4p1 (Debian 11 Bullseye) as well as with 9.0p1 (Debian Unstable) on the server side. And both times OpenSSH 9.0p1 and Zsh 5.8.1.2-test on the client side. > With localhost being 9.0 for client and server and zsh 5.8, I see this > reproduction case; setup, then the results of tab-completion, and then > at the end a manually-constructed filename which definitely would not > have worked with older scp. Oh, ok. Now I am confused, though. > So the good news is that scp in the new mode is "pass the filename > exactly as it really is, no escaping for the remote side, whatever > reaches us in argv[] is what will be used, so you only need to worry > about local shell escaping". Correct. That's probably one of the things they wanted to get rid of. > -O to use original SCP protocol, -s to use sftp protocol. Thanks for that summary! I searched through the scp(1) man page several times with different search terms for something like -s, but didn't find it. *lookingagain* Ah, they removed it again with the 9.0 release. No traces of it in the 9.0p1 scp(1) man page. But it's in the one from 8.4p1. (I assumed that there is a way to revert a previously set -O — maybe via config or so — on the commandline. But it seems as if already the latter is not intended to be possible.) > -D is for local debugging. Huh? That's not in the scp(1) man page as of 8.4p1 and in the scp(1) man page of 9.0p1, -D is documented as follows: -D sftp_server_path When using the SFTP protocol support via -M, connect directly to a local SFTP server pro‐ gram rather than a remote one via ssh(1). This option may be useful in debugging the client and server. Ah, I think you forgot to mention that "local debugging" meant "local debugging on the server side" as sshd(8) mentions: -D When this option is specified, sshd will not detach and does not become a daemon. This allows easy monitoring of sshd. > So I think full support means knowing the version, knowing which version > switched the default behaviour (looks like 9.0 man-page might be > wrong), Yes, noticed that, too. It says, the SFTP backend is default since 8.8. But if you look at the upstream changelog, they made SFTP the default multiple times and then reverted it and applied it again: commit a4537e79ab4ac6db4493c5158744b9ebde5efcb0 Date: Wed Feb 23 21:21:16 2022 +0000 upstream: put back the scp manpage changes for SFTP mode too OpenBSD-Commit-ID: 05dc53921f927e1b5e5694e1f3aa314549f2e768 commit 449bcb8403adfb9724805d02a51aea76046de185 Date: Wed Feb 23 19:01:00 2022 +0000 upstream: and we go back to testing sftp-scp after the 8.9 release... OpenBSD-Commit-ID: a80440168258adca543a4607b871327a279c569c commit 9699151b039ecc5fad9ac6c6c02e9afdbd26f15f Date: Thu Feb 10 04:12:38 2022 +0000 upstream: revert for imminent OpenSSH release, which wil ship with scp in RCP mode. > revision 1.106 > date: 2021/10/15 14:46:46; author: deraadt; state: Exp; lines: +13 -9; commitid: w5n9B2RE38tFfggl; > openbsd 7.0 release shipped with the (hopefully last) scp that uses RCP > protocol for copying. Let's get back to testing the SFTP protocol. This will be put back once the OpenSSH release is done. OpenBSD-Commit-ID: 0c725481a78210aceecff1537322c0b2df03e768 I assume that they didn't fix up the "Since OpenSSH 8.8, scp has use the SFTP protocol for transfers by default" (sic!) phrase in the scp(1) man page after reapplying the patches from after 8.7 again for 9.0. > and then knowing which options have been set on the command-line already > for this version. I also wondered if there's a possibility to configure the settings relevant for rcp vs sftp as scp backend in ~/.ssh/config or /etc/ssh/ssh_config. But I haven't found anything in that direction so far. Kind regards, Axel -- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe@deuxchevaux.org \ / Say No to HTML in E-Mail and Usenet Mail+Jabber: abe@noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/ ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-04-15 12:46 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-14 15:20 OpenSSH scp completion reminder Phil Pennock 2022-04-14 16:18 ` Axel Beckert 2022-04-14 17:04 ` Bart Schaefer 2022-04-14 17:27 ` Axel Beckert 2022-04-14 17:34 ` Mikael Magnusson 2022-04-14 20:44 ` Phil Pennock 2022-04-15 12:46 ` Axel Beckert
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).