[TUHS] x/y/zmodem on Unix

[TUHS] x/y/zmodem on Unix

[Paul Ruizendaal]

Last december Matt brought up xmodem and recently I needed it for an almost identical use case. Studied it a bit and in the context of Unix, it is an interesting piece of software history.

Although xmodem originated on CP/M in 1977, it seems to have arrived on Unix soon thereafter. The first Unix implementation seems to have gone by the name of “rbsb” and must have originated when V7 was prevalent: it uses alarm() to simulate non-blocking I/O. Over the course of the 80’s ymodem and zmodem were added and the package became lrzsz; the source continued to have a very V7-ish feel to it at least to the early 90’s. In the mid-90’s it was converted to ansi-C and had some other modernization, but it looks like it would still have run on SysIII (apart from the ansi).The last update to the upstream source package appears to be from 1998, i.e. 25 years ago.

It is still packaged today for FreeBSD and major Linux distros. On Unix it must be one the oldest code bases still in regular use, with the 1980 source still recognizable in its current incarnation. Any other contenders come to mind?

I had always associated x/y/zmodem with CP/M and MSDOS, not so much with Unix. Last December Clem already pointed out that it was popular for file exchange in the Unix scene as well, along with several other similar tools. Also, the ymodem approach to file metadata is very unix oriented, suggesting it originated on Unix or at least that Unix users were an important user demographic. Yet, I could find little trace of x/y/zmodem in the TUHS Unix Tree. The search tool finds it in 2.11BSD, in Minix 1.5 and 2.0 and in V10. Kermit is in those as well, and in 4.3BSD and 4.4BSD on top.

Maybe these programs were commonly pulled from a bulletin board and hence not on distribution media. Not sure how that would be bootstrapped, although the 2.11BSD files for x/y/zmodem include “minirb.c”, a 175 line implementation to receive files using the ymodem protocol. It is possible that this was transferred as plain text as a first step, or even retyped.

Any recollections?

[TUHS] Re: x/y/zmodem on Unix

[Dan Cross]

On Mon, Sep 18, 2023 at 6:31 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> Last december Matt brought up xmodem and recently I needed it for an almost identical use case. Studied it a bit and in the context of Unix, it is an interesting piece of software history.
>
> Although xmodem originated on CP/M in 1977, it seems to have arrived on Unix soon thereafter. The first Unix implementation seems to have gone by the name of “rbsb” and must have originated when V7 was prevalent: it uses alarm() to simulate non-blocking I/O. Over the course of the 80’s ymodem and zmodem were added and the package became lrzsz; the source continued to have a very V7-ish feel to it at least to the early 90’s. In the mid-90’s it was converted to ansi-C and had some other modernization, but it looks like it would still have run on SysIII (apart from the ansi).The last update to the upstream source package appears to be from 1998, i.e. 25 years ago.
>
> It is still packaged today for FreeBSD and major Linux distros. On Unix it must be one the oldest code bases still in regular use, with the 1980 source still recognizable in its current incarnation. Any other contenders come to mind?
>
> I had always associated x/y/zmodem with CP/M and MSDOS, not so much with Unix. Last December Clem already pointed out that it was popular for file exchange in the Unix scene as well, along with several other similar tools. Also, the ymodem approach to file metadata is very unix oriented, suggesting it originated on Unix or at least that Unix users were an important user demographic. Yet, I could find little trace of x/y/zmodem in the TUHS Unix Tree. The search tool finds it in 2.11BSD, in Minix 1.5 and 2.0 and in V10. Kermit is in those as well, and in 4.3BSD and 4.4BSD on top.
>
> Maybe these programs were commonly pulled from a bulletin board and hence not on distribution media. Not sure how that would be bootstrapped, although the 2.11BSD files for x/y/zmodem include “minirb.c”, a 175 line implementation to receive files using the ymodem protocol. It is possible that this was transferred as plain text as a first step, or even retyped.
>
> Any recollections?

xmodem was an outgrowth of Christensen's MODEM.ASM for CP/M, and is
almost criminally simple: a small block of data coupled with a little
bit of header information, wait for an acknowledgement, and repeat.
Metadata was small (a start byte, a block number, inverse block
number, and a single checksum byte), but there was no total byte
count, so it was assumed that the final block would be padded with a
throwaway character.

Chuck Forsberg did YMODEM and ZMODEM; YMODEM is sort of a
super-XMODEM: it adds a 16-bit CRC, increases the block size, and adds
a "0 block" with some metadata (total file size and file name); thus,
it no longer needed a padding byte removed from the final packet.
Otherwise, it retains most of the overall structure of XMODEM.

ZMODEM was, as I understood it, designed for transfers across telenet,
which was pretty reliable; instead of the highly synchronous
send/wait-for-ack cycle of xmodem and ymodem, zmodem relies on error
detection and correction and is basically a streaming protocol: a
packet in a sliding window could be NAK'ed, thus rewinding the
transfer, but otherwise it basically just sends data until done.

lrzsz came kind of later; I remember Forsberg's rzsz package on Unix
back in the day, and then there was a GNU reimplementation. For a
while, the Omen Technologies BBS was listed in the BSD phone numbers
files.

xmodem lives on in a lot of embedded applications because of its
overall simplicity. We used it to bootstrap kernels onto Oxide
computers, for example, while we were doing active development.

        - Dan C.

[TUHS] Re: x/y/zmodem on Unix

[Theodore Ts'o]

On Mon, Sep 18, 2023 at 07:07:25PM -0400, Dan Cross wrote:
> 
> xmodem lives on in a lot of embedded applications because of its
> overall simplicity. We used it to bootstrap kernels onto Oxide
> computers, for example, while we were doing active development.

Today, KDE's Konsole terminal window has a "ZModem upload" function,
which I've used for sending a file up to some system where the
administrator on the remote machine has disabled scp and sftp for
random security $REASONS.

I could compress the file and use uuencode/uudecode, but for larger
files where cut and paste isn't terribly convenient, using zmodem can
be one of the simpler and more effective.

						- Ted

[TUHS] Re: x/y/zmodem on Unix

[Jonathan Gray]

On Tue, Sep 19, 2023 at 12:31:30AM +0200, Paul Ruizendaal wrote:
> I had always associated x/y/zmodem with CP/M and MSDOS, not so much with Unix. Last December Clem already pointed out that it was popular for file exchange in the Unix scene as well, along with several other similar tools. Also, the ymodem approach to file metadata is very unix oriented, suggesting it originated on Unix or at least that Unix users were an important user demographic. Yet, I could find little trace of x/y/zmodem in the TUHS Unix Tree. The search tool finds it in 2.11BSD, in Minix 1.5 and 2.0 and in V10. Kermit is in those as well, and in 4.3BSD and 4.4BSD on top.

have a look at
tuhs/Applications/Shoppa_Tapes/usenix878889.tar.gz
usenix87/Comm/
usenix89/Comm/

also appears in
386bsd/othersrc/public/zmodem-3.03/
where the license was changed to prohibit commercial use after
the RLE changes in April 1989

https://www.ohse.de/uwe/software/lrzsz.html
is derived from an earlier version with the license changed to GPLv2

[TUHS] Re: x/y/zmodem on Unix

[Serge Burjak]

Chuck Forsberg wrote Zmodem as a sliding Window transfer protocol
alternative to Kermit and superkermit. It was very high performance
over slow links. He provided source code support for the protocol
driver on the server and client end, in various versions. His client
programs  were Yam (yet another modem) and ProYam. You could shell out
of the client, run other programs. Other client platforms supported
parts of the protocol. Available as binaries and source when licenced.
It had a scripting language and out of the box had the zmodem protocol
which has the ability to send commands, resume transfers, traverse non
8 bit or unreliable links, remove or rename source file, deal with
conflicting file names, synchronise folders and lots of other things.
I have written many scripts to help dial up users into a unix box and
download financial research data. Scripts still work across the
internet. He made the code very portable, clean and I believe it is
open source now. Chuck is no longer with us.

SecureCRT ssh client from VanDyke.com still has native support. To
send a file from the unix host, typically sz filename and it just
arrives. Sz- u and it will remove the source file after successful
transfer, plus lots of other options. The binary was typically linked
so if you type sx it would try to do a transfer with xmodem, even
though it's the same binary.

Hope this helps.

Serge


On Tue, 19 Sept 2023 at 12:47, Jonathan Gray <jsg@jsg.id.au> wrote:
>
> On Tue, Sep 19, 2023 at 12:31:30AM +0200, Paul Ruizendaal wrote:
> > I had always associated x/y/zmodem with CP/M and MSDOS, not so much with Unix. Last December Clem already pointed out that it was popular for file exchange in the Unix scene as well, along with several other similar tools. Also, the ymodem approach to file metadata is very unix oriented, suggesting it originated on Unix or at least that Unix users were an important user demographic. Yet, I could find little trace of x/y/zmodem in the TUHS Unix Tree. The search tool finds it in 2.11BSD, in Minix 1.5 and 2.0 and in V10. Kermit is in those as well, and in 4.3BSD and 4.4BSD on top.
>
> have a look at
> tuhs/Applications/Shoppa_Tapes/usenix878889.tar.gz
> usenix87/Comm/
> usenix89/Comm/
>
> also appears in
> 386bsd/othersrc/public/zmodem-3.03/
> where the license was changed to prohibit commercial use after
> the RLE changes in April 1989
>
> https://www.ohse.de/uwe/software/lrzsz.html
> is derived from an earlier version with the license changed to GPLv2

[TUHS] Re: x/y/zmodem on Unix

[John Cowan]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

On Mon, Sep 18, 2023 at 7:08 PM Dan Cross <crossd@gmail.com> wrote:

but there was no total byte
> count, so it was assumed that the final block would be padded with a
> throwaway character.
>

CP/M, like RT-11 and OS/8 before it, did not track the sizes of files in
bytes or words, only in blocks.  Text files ended in ^Z and were then
padded with either NULs or more ^Zs if necessary; binary files were usually
padded with NULs.

ZMODEM was, as I understood it, designed for transfers across telenet,
> which was pretty reliable; instead of the highly synchronous
> send/wait-for-ack cycle of xmodem and ymodem, zmodem relies on error
> detection and correction and is basically a streaming protocol: a
> packet in a sliding window could be NAK'ed, thus rewinding the
> transfer, but otherwise it basically just sends data until done.
>

It's an analogue of TCP/IP, in fact.

[-- Attachment #2: Type: text/html, Size: 1832 bytes --]

[TUHS] Re: x/y/zmodem on Unix

[Serge Burjak]

If anyone wants more zmodem/Chuck Forsberg history, a prolific usenet
poster, here are news archives going back to 1994. It gives a good
indication of his personality. I had been dealing with him for about 6
years by then... Still have the licence keys.

https://groups.google.com/g/comp.protocols.misc/c/kjQ3HWqR2Ck/m/QVrdXdbsz1EJ

Kermit discussion and comparisons,
https://groups.google.com/g/comp.dcom.modems/c/GuEzWARpVQY/m/nZX5QdmY7xwJ

On Tue, 19 Sept 2023 at 13:42, Serge Burjak <sburjak@systech.com.au> wrote:
>
> Chuck Forsberg wrote Zmodem as a sliding Window transfer protocol
> alternative to Kermit and superkermit. It was very high performance
> over slow links. He provided source code support for the protocol
> driver on the server and client end, in various versions. His client
> programs  were Yam (yet another modem) and ProYam. You could shell out
> of the client, run other programs. Other client platforms supported
> parts of the protocol. Available as binaries and source when licenced.
> It had a scripting language and out of the box had the zmodem protocol
> which has the ability to send commands, resume transfers, traverse non
> 8 bit or unreliable links, remove or rename source file, deal with
> conflicting file names, synchronise folders and lots of other things.
> I have written many scripts to help dial up users into a unix box and
> download financial research data. Scripts still work across the
> internet. He made the code very portable, clean and I believe it is
> open source now. Chuck is no longer with us.
>
> SecureCRT ssh client from VanDyke.com still has native support. To
> send a file from the unix host, typically sz filename and it just
> arrives. Sz- u and it will remove the source file after successful
> transfer, plus lots of other options. The binary was typically linked
> so if you type sx it would try to do a transfer with xmodem, even
> though it's the same binary.
>
> Hope this helps.
>
> Serge
>
>
> On Tue, 19 Sept 2023 at 12:47, Jonathan Gray <jsg@jsg.id.au> wrote:
> >
> > On Tue, Sep 19, 2023 at 12:31:30AM +0200, Paul Ruizendaal wrote:
> > > I had always associated x/y/zmodem with CP/M and MSDOS, not so much with Unix. Last December Clem already pointed out that it was popular for file exchange in the Unix scene as well, along with several other similar tools. Also, the ymodem approach to file metadata is very unix oriented, suggesting it originated on Unix or at least that Unix users were an important user demographic. Yet, I could find little trace of x/y/zmodem in the TUHS Unix Tree. The search tool finds it in 2.11BSD, in Minix 1.5 and 2.0 and in V10. Kermit is in those as well, and in 4.3BSD and 4.4BSD on top.
> >
> > have a look at
> > tuhs/Applications/Shoppa_Tapes/usenix878889.tar.gz
> > usenix87/Comm/
> > usenix89/Comm/
> >
> > also appears in
> > 386bsd/othersrc/public/zmodem-3.03/
> > where the license was changed to prohibit commercial use after
> > the RLE changes in April 1989
> >
> > https://www.ohse.de/uwe/software/lrzsz.html
> > is derived from an earlier version with the license changed to GPLv2

[TUHS] Re: x/y/zmodem on Unix

[Dave Horsfall]

On Tue, 19 Sep 2023, Serge Burjak wrote:

> If anyone wants more zmodem/Chuck Forsberg history, a prolific usenet 
> poster, here are news archives going back to 1994. It gives a good 
> indication of his personality. I had been dealing with him for about 6 
> years by then... Still have the licence keys.
> 
> https://groups.google.com/g/comp.protocols.misc/c/kjQ3HWqR2Ck/m/QVrdXdbsz1EJ

I remember him well; thanks!

> Kermit discussion and comparisons,
> https://groups.google.com/g/comp.dcom.modems/c/GuEzWARpVQY/m/nZX5QdmY7xwJ

I was wondering when someone was going to mention Kermit...

-- Dave, with a 1200/75 software modem at the time

[TUHS] Re: x/y/zmodem on Unix

[Michael Stiller via TUHS]

I just wanted to mention, that you can also get zmodem support for iTerm2 if you are on mac.

In fact i use it a lot.

See here:

https://github.com/robberphex/iTerm2-zmodem

Best regards,

Michael


> On 19. Sep 2023, at 04:04, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> On Mon, Sep 18, 2023 at 07:07:25PM -0400, Dan Cross wrote:
>> 
>> xmodem lives on in a lot of embedded applications because of its
>> overall simplicity. We used it to bootstrap kernels onto Oxide
>> computers, for example, while we were doing active development.
> 
> Today, KDE's Konsole terminal window has a "ZModem upload" function,
> which I've used for sending a file up to some system where the
> administrator on the remote machine has disabled scp and sftp for
> random security $REASONS.
> 
> I could compress the file and use uuencode/uudecode, but for larger
> files where cut and paste isn't terribly convenient, using zmodem can
> be one of the simpler and more effective.
> 
> 						- Ted