[TUHS] Be there a "remote diff" utility?

[TUHS] Be there a "remote diff" utility?

[Dave Horsfall]

Every so often I want to compare files on remote machines, but all I can 
do is to fetch them first (usually into /tmp); I'd like to do something 
like:

    rdiff host1:file1 host2:file2

Breathes there such a beast?  I see that Penguin/OS has already taken 
"rdiff" which doesn't seem to do what I want.

Think of it as an extension to the Unix philosophy of "Everything looks 
like a file"...

-- Dave

[TUHS] Re: Be there a "remote diff" utility?

[arnold]

Maybe

	diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)

?

That could be put into a shell script that does the approriate
text manipulations on the original $1 and $2.

HTH,

Arnold

Dave Horsfall <dave@horsfall.org> wrote:

> Every so often I want to compare files on remote machines, but all I can 
> do is to fetch them first (usually into /tmp); I'd like to do something 
> like:
>
>     rdiff host1:file1 host2:file2
>
> Breathes there such a beast?  I see that Penguin/OS has already taken 
> "rdiff" which doesn't seem to do what I want.
>
> Think of it as an extension to the Unix philosophy of "Everything looks 
> like a file"...
>
> -- Dave

[TUHS] Re: Be there a "remote diff" utility?

[Ralph Corderoy]

Hi,

I've set ‘mail-followup-to: coff@tuhs.org’.

> > Every so often I want to compare files on remote machines, but all
> > I can do is to fetch them first (usually into /tmp); I'd like to do
> > something like:
> >
> >     rdiff host1:file1 host2:file2
> >
> > Breathes there such a beast?

No, nor should there.  It would be slain less it beget rcmp, rcomm,
rpaste, ...

> > Think of it as an extension to the Unix philosophy of "Everything
> > looks like a file"...

Then make remote files look local as far as their access is concerned.
Ideally at the system-call level.  Less ideal, at libc.a.

> Maybe
>
>     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)

This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
‘Banner’ set which spews the contents of a file before authentication,
e.g. the pointless

    This computer system is the property of ...

    Disconnect NOW if you have not been expressly authorised to use this
    system.  Unauthorised use is a criminal offence under the Computer
    Misuse Act 1990.

    Communications on or through ...uk's computer systems may be
    monitored or recorded to secure effective system operation and for
    other lawful purposes.

It appears on stderr so doesn't upset the diff but does clutter.
And discarding stderr is too sloppy.

-- 
Cheers, Ralph.

[TUHS] Re: Be there a "remote diff" utility?

[George Michaelson]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

Sshfs

G

On Thu, 16 May 2024, 5:34 pm Ralph Corderoy, <ralph@inputplus.co.uk> wrote:

> Hi,
>
> I've set ‘mail-followup-to: coff@tuhs.org’.
>
> > > Every so often I want to compare files on remote machines, but all
> > > I can do is to fetch them first (usually into /tmp); I'd like to do
> > > something like:
> > >
> > >     rdiff host1:file1 host2:file2
> > >
> > > Breathes there such a beast?
>
> No, nor should there.  It would be slain less it beget rcmp, rcomm,
> rpaste, ...
>
> > > Think of it as an extension to the Unix philosophy of "Everything
> > > looks like a file"...
>
> Then make remote files look local as far as their access is concerned.
> Ideally at the system-call level.  Less ideal, at libc.a.
>
> > Maybe
> >
> >     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)
>
> This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
> ‘Banner’ set which spews the contents of a file before authentication,
> e.g. the pointless
>
>     This computer system is the property of ...
>
>     Disconnect NOW if you have not been expressly authorised to use this
>     system.  Unauthorised use is a criminal offence under the Computer
>     Misuse Act 1990.
>
>     Communications on or through ...uk's computer systems may be
>     monitored or recorded to secure effective system operation and for
>     other lawful purposes.
>
> It appears on stderr so doesn't upset the diff but does clutter.
> And discarding stderr is too sloppy.
>
> --
> Cheers, Ralph.
>

[-- Attachment #2: Type: text/html, Size: 2151 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[arnold]

Ralph Corderoy <ralph@inputplus.co.uk> wrote:

> > Maybe
> >
> >     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)
>
> This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
> ‘Banner’ set which spews the contents of a file before authentication,
> e.g. the pointless
>
> [....]
>
> It appears on stderr so doesn't upset the diff but does clutter.

All true, I didn't think about that.

> And discarding stderr is too sloppy.

But the author of a personal script knows his/her remote machines
and can decide if

     diff -u <(ssh host1 cat file1 2>/dev/null) <(ssh host2 cat file2 2>/dev/null)

is appropriate or not.

My main point was that the problem is easily solved with a
few lines of shell, so no need for a utility, especially one
written in C or some other compiled language.

Thanks,

Arnold

[TUHS] Re: Be there a "remote diff" utility?

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 1689 bytes --]

" The 9import tool allows an arbitrary file on a remote system, with the
capability of running the Plan 9 exportfs(4) service, to be imported into
the local name space. Usually file is a directory, so the complete file
tree under the directory is made available."
https://9fans.github.io/plan9port/man/man4/9import.html

9import host1 / /tmp/host1
9import host2 /tmp/host2
diff /tmp/host1/a/b/c /tmp/host2/a/b/c
(or whatever command you want that works with files. No need for stuff like
'rdiff' etc.)

stuff you take for granted on some systems ...

I have the plan 9 cpu command working (written in Go) and I think it's time
I get import working more widely, it's just too useful.

On Thu, May 16, 2024 at 2:01 AM <arnold@skeeve.com> wrote:

> Ralph Corderoy <ralph@inputplus.co.uk> wrote:
>
> > > Maybe
> > >
> > >     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)
> >
> > This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
> > ‘Banner’ set which spews the contents of a file before authentication,
> > e.g. the pointless
> >
> > [....]
> >
> > It appears on stderr so doesn't upset the diff but does clutter.
>
> All true, I didn't think about that.
>
> > And discarding stderr is too sloppy.
>
> But the author of a personal script knows his/her remote machines
> and can decide if
>
>      diff -u <(ssh host1 cat file1 2>/dev/null) <(ssh host2 cat file2
> 2>/dev/null)
>
> is appropriate or not.
>
> My main point was that the problem is easily solved with a
> few lines of shell, so no need for a utility, especially one
> written in C or some other compiled language.
>
> Thanks,
>
> Arnold
>

[-- Attachment #2: Type: text/html, Size: 2376 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

To add to Ron's post, Plan 9's cpu exports the origination's namespace to
the destination; by convention it is mounted on /mnt/term at destination.

host1% cpu -h host2
host2% diff file2 /mnt/term/usr/me/file1


On Thu, May 16, 2024 at 10:09 AM ron minnich <rminnich@gmail.com> wrote:

> " The 9import tool allows an arbitrary file on a remote system, with the
> capability of running the Plan 9 exportfs(4) service, to be imported into
> the local name space. Usually file is a directory, so the complete file
> tree under the directory is made available."
> https://9fans.github.io/plan9port/man/man4/9import.html
>
> 9import host1 / /tmp/host1
> 9import host2 /tmp/host2
> diff /tmp/host1/a/b/c /tmp/host2/a/b/c
> (or whatever command you want that works with files. No need for stuff
> like 'rdiff' etc.)
>
> stuff you take for granted on some systems ...
>
> I have the plan 9 cpu command working (written in Go) and I think it's
> time I get import working more widely, it's just too useful.
>
> On Thu, May 16, 2024 at 2:01 AM <arnold@skeeve.com> wrote:
>
>> Ralph Corderoy <ralph@inputplus.co.uk> wrote:
>>
>> > > Maybe
>> > >
>> > >     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)
>> >
>> > This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
>> > ‘Banner’ set which spews the contents of a file before authentication,
>> > e.g. the pointless
>> >
>> > [....]
>> >
>> > It appears on stderr so doesn't upset the diff but does clutter.
>>
>> All true, I didn't think about that.
>>
>> > And discarding stderr is too sloppy.
>>
>> But the author of a personal script knows his/her remote machines
>> and can decide if
>>
>>      diff -u <(ssh host1 cat file1 2>/dev/null) <(ssh host2 cat file2
>> 2>/dev/null)
>>
>> is appropriate or not.
>>
>> My main point was that the problem is easily solved with a
>> few lines of shell, so no need for a utility, especially one
>> written in C or some other compiled language.
>>
>> Thanks,
>>
>> Arnold
>>
>

[-- Attachment #2: Type: text/html, Size: 3050 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Marc Donner]

[-- Attachment #1: Type: text/plain, Size: 2684 bytes --]

If I recall correctly, there is a combination of flags to rsync that will
generate a report on a file, a set of files, or a set of directories to
tell if they are different.

I seem to recall DPK or RCT doing something clever with rsync and cksum to
get this sort of result without having to stream a lot of data across the
long-haul network back in the day.

Best,

Marc
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>


On Thu, May 16, 2024 at 2:51 PM Skip Tavakkolian <fariborz.t@gmail.com>
wrote:

> To add to Ron's post, Plan 9's cpu exports the origination's namespace to
> the destination; by convention it is mounted on /mnt/term at destination.
>
> host1% cpu -h host2
> host2% diff file2 /mnt/term/usr/me/file1
>
>
> On Thu, May 16, 2024 at 10:09 AM ron minnich <rminnich@gmail.com> wrote:
>
>> " The 9import tool allows an arbitrary file on a remote system, with the
>> capability of running the Plan 9 exportfs(4) service, to be imported into
>> the local name space. Usually file is a directory, so the complete file
>> tree under the directory is made available."
>> https://9fans.github.io/plan9port/man/man4/9import.html
>>
>> 9import host1 / /tmp/host1
>> 9import host2 /tmp/host2
>> diff /tmp/host1/a/b/c /tmp/host2/a/b/c
>> (or whatever command you want that works with files. No need for stuff
>> like 'rdiff' etc.)
>>
>> stuff you take for granted on some systems ...
>>
>> I have the plan 9 cpu command working (written in Go) and I think it's
>> time I get import working more widely, it's just too useful.
>>
>> On Thu, May 16, 2024 at 2:01 AM <arnold@skeeve.com> wrote:
>>
>>> Ralph Corderoy <ralph@inputplus.co.uk> wrote:
>>>
>>> > > Maybe
>>> > >
>>> > >     diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)
>>> >
>>> > This is annoyingly noisy if the remote SSH server has sshd_config(5)'s
>>> > ‘Banner’ set which spews the contents of a file before authentication,
>>> > e.g. the pointless
>>> >
>>> > [....]
>>> >
>>> > It appears on stderr so doesn't upset the diff but does clutter.
>>>
>>> All true, I didn't think about that.
>>>
>>> > And discarding stderr is too sloppy.
>>>
>>> But the author of a personal script knows his/her remote machines
>>> and can decide if
>>>
>>>      diff -u <(ssh host1 cat file1 2>/dev/null) <(ssh host2 cat file2
>>> 2>/dev/null)
>>>
>>> is appropriate or not.
>>>
>>> My main point was that the problem is easily solved with a
>>> few lines of shell, so no need for a utility, especially one
>>> written in C or some other compiled language.
>>>
>>> Thanks,
>>>
>>> Arnold
>>>
>>

[-- Attachment #2: Type: text/html, Size: 4390 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Theodore Ts'o]

On Thu, May 16, 2024 at 04:34:54PM +1000, Dave Horsfall wrote:
> Every so often I want to compare files on remote machines, but all I can 
> do is to fetch them first (usually into /tmp); I'd like to do something 
> like:
> 
>     rdiff host1:file1 host2:file2
> 
> Breathes there such a beast?  I see that Penguin/OS has already taken 
> "rdiff" which doesn't seem to do what I want.

rdiff is something which someone on the internet had created, as part
of the librsync package[1].  Thia isn't considered part of the core
package (for example, Debian consideres it as an "optional" package)
but rather something which various distributions have packaged for the
convenience for their users.

[1] https://librsync.github.io/

So if this is considered part of Penguin/OS, would we also consider
"nethack" or X11 part of BSD 4.3, since it was available and often
would be commonly installed on BSD 4.3 systems?  Or are all packages
which are in FreeBSD's ports "part of FreeBSD"?  Or all packages in
MacPorts part of MacOS?

In any case, the way I'd suggest that you do this that works as an
extention to the Unix philosohy of "Everything looks like a file" is
to use FUSE:

sshfs host1:/ ~/mnt/host1
sshfs host2:/ ~/mnt/host2
diff ~/mnt/host1/file1 ~/mnt/host2/file2

Cheers,

					- Ted

[TUHS] Re: Be there a "remote diff" utility?

[Alan D. Salewski]

On Thu, May 16, 2024, at 15:53, Theodore Ts'o wrote:
> On Thu, May 16, 2024 at 04:34:54PM +1000, Dave Horsfall wrote:
>> Every so often I want to compare files on remote machines, but all I can 
>> do is to fetch them first (usually into /tmp); I'd like to do something 
>> like:
>> 
>>     rdiff host1:file1 host2:file2
>> 
>> Breathes there such a beast?  I see that Penguin/OS has already taken 
>> "rdiff" which doesn't seem to do what I want.
[...]

> In any case, the way I'd suggest that you do this that works as an
> extention to the Unix philosohy of "Everything looks like a file" is
> to use FUSE:
>
> sshfs host1:/ ~/mnt/host1
> sshfs host2:/ ~/mnt/host2
> diff ~/mnt/host1/file1 ~/mnt/host2/file2
>
> Cheers,
>
> 					- Ted

Mentioning this since I just came across it today: Eric S. Raymond
(ESR) has a 'netdiff' tool available here:

     http://www.catb.org/~esr/netdiff/
     https://gitlab.com/esr/netdiff

<quote>
    Perform diff across network links.

    Takes two arguments which may be host:path pairs in the style of
    ssh/scp, make temporary local copies as needed, applies
    diff(1). All options are passed to diff(1).
</quote>

Usage:
    netdiff [diff-options] [host1:]path1 [host2:]path2

It's 56 lines of POSIX shell script that invokes 'scp' behind the
scenes and creates temp files that are fed to 'diff', with some file
name post-processing.

Compared to the one line of shell script mentioned by Arnold:

    diff -u <(ssh host1 cat file1) <(ssh host2 cat file2)

the main benefits I see to 'netdiff' in its current form are that it
encapsulates the mechanism behind a descriptive name that can be
found in the file system, and it has a man page. So it can be found,
used and/or studied by those with larval shell fu.

However, the comments in the 'netdiff' source suggest openness to
supporting different transport mechanisms, which (if added) might
allow using the same command line interface regardless of the
underlying transport being used (scp, ssh, rsync, curl, wget,
whatever), possibly with different transports for each of the
requested files.

Of course, that would not offer a superior user experience to the
'9import', 'sshfs', and NFS approaches also mentioned.

But if there were an 'rf' ("remote file") tool that did just the
transport abstraction portion, then tools such as 'netdiff' could
use it internally, or folks could use it directly:

    diff -u <(rf user@host1:/path/to/file1) <(rf user@host2:/path/to/file2)

The presumption here is that the user has control over some local
configuration that maps user@host pairs to transports (maybe
defaulting to 'ssh' invoking 'cat' on the remote host).

Maybe somebody who needs more than 'ssh' would find value in such a
thing...

-Al

-- 
a l a n   d.   s a l e w s k i
ads@salewski.email
salewski@att.net
https://github.com/salewski

[TUHS] Re: Be there a "remote diff" utility?

[Douglas McIlroy]

[-- Attachment #1: Type: text/plain, Size: 118 bytes --]

With the disclaimer that I have never used it, I note that FUSE/sshfs
allows one to  mount remote file systems.

Doug

[-- Attachment #2: Type: text/html, Size: 165 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Bakul Shah via TUHS]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

An interesting question is whether there exists a diff algorithm
which *minimizes* data movement across the network. Assuming similar
lengths, you can halve it by running the diff at one of the hosts
but can one do better if the two files are fairly similar? Is this
even a theoretical possibility? I don't see links to any such
algorithm on wikipedia's diff page but I figured there might be
someone on TUHS who may have speculated or know about this!

Bakul

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Rich Salz]

[-- Attachment #1: Type: text/plain, Size: 142 bytes --]

The rsync protocol might be appropriate.  See
https://www.samba.org/~tridge/phd_thesis.pdf and
https://rsync.samba.org/tech_report/node2.html

[-- Attachment #2: Type: text/html, Size: 311 bytes --]

[TUHS] Re: Be there a "remote diff" utility?

[Ben Greenfield via TUHS]

[-- Attachment #1: Type: text/plain, Size: 277 bytes --]

I use rsync for that with the -n dry run flag

> On May 16, 2024, at 2:12 PM, Rich Salz <rich.salz@gmail.com> wrote:
> 
> The rsync protocol might be appropriate.  See https://www.samba.org/~tridge/phd_thesis.pdf and https://rsync.samba.org/tech_report/node2.html
> 


[-- Attachment #2: Type: text/html, Size: 788 bytes --]