[COFF] Shell script advice: using 'dd' to write multiple media

[COFF] Shell script advice: using 'dd' to write multiple media

[steve jenkin]

Apropos the ESIX SVR4 distro on floppies or streaming tape mentioned by Bill Corcoran

	<https://www.tuhs.org/mailman3/hyperkitty/list/coff@tuhs.org/message/WEJQQCJHH7BVVRGR4QNLS4AQ2OCAJRU7/>

In the mid 1980’s I worked for a small Australian outfit that did “Unix”.

One of the things we did was distributing software, which required writing to many media.

There was a very clever script that broke the distribution into many parts, if needed,
to suit the size of the distribution media. [ tape, 3.5” floppy, 2.5” floppy, etc ]

Over the years I’ve tried to recreate a version and not succeeded :(

There was a ‘create the distro’ step of the pipeline which gathered the input,
followed by a loop that used ‘dd’ to block the stream into media-sized parts.

I’ve never figured out how to use ‘dd’ so it returns after a single block is written
doesn’t close the input, killing the pipeline, or cause the rest of the data
to be discarded.

The script let our admin staff reliably create distros on whatever media was requested.

Any suggestions or hints?
I’m thinking this is obvious, but in the man pages i’ve read, not found an answer.

It could be modern versions of ‘dd’ don’t have this behaviour.

cheers
steve

--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Dave Horsfall]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

On Tue, 27 Jun 2023, steve jenkin wrote:

> In the mid 1980’s I worked for a small Australian outfit that did 
> “Unix”.

Neology?  Softway?

> One of the things we did was distributing software, which required 
> writing to many media.

[...]

> There was a ‘create the distro’ step of the pipeline which gathered the 
> input, followed by a loop that used ‘dd’ to block the stream into 
> media-sized parts.

Sounds like you want to put "dd" into a loop, driven by the block size and 
the media size, then using the "iseek" option on the input file to burrow 
through it.

I would use "expr" for the arithmetic because I've never bothered to
learn the Bash syntax :-)

Or have I missed something?

-- Dave

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Mon, Jun 26, 2023 at 7:09 PM steve jenkin <sjenkin@canb.auug.org.au>
wrote:

>
>
> I’ve never figured out how to use ‘dd’ so it returns after a single block
> is written
> doesn’t close the input, killing the pipeline, or cause the rest of the
> data
> to be discarded.
>

The trick here is understanding how ibs & obs, EOF is handled and probably
osync, then how to use iseek (iskip in modern versions).
It's not that hard. Its been done many times.

One of the more interesting issues with dd is most versions still are
single threaded which sucks for most media - particularly f you want to
stream things.
There was a wonderful hack to dd done in the. early 1980s by Tapani
Lindgren ddd - double dd - which used two processes and a pipe to control
them, so one process was always reading while the other was writing.   You
can one it Volume 14, issue 85 in the comp.sources.unix archives.

Years ago, I hacked up a version using threads to do the same thing with a
mutex to protect everything (It ever ran on Windows at some point).
ᐧ

[-- Attachment #2: Type: text/html, Size: 2338 bytes --]

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[steve jenkin]

Dave,

I’ve tried doing the ‘dd’ in a for or while loop many ways, many times.

Perhaps you could write & yesy a little demo script for me.

Say, “ls -1 | for (); do dd bs=10b; done”

cheers
steve

> On 27 Jun 2023, at 09:21, Dave Horsfall <dave@horsfall.org> wrote:
> 
> Sounds like you want to put "dd" into a loop, driven by the block size and 
> the media size, then using the "iseek" option on the input file to burrow 
> through it.
> 
> I would use "expr" for the arithmetic because I've never bothered to
> learn the Bash syntax :-)
> 
> Or have I missed something?
> 
> -- Dave

--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Bakul Shah]

On Jun 26, 2023, at 4:09 PM, steve jenkin <sjenkin@canb.auug.org.au> wrote:
> 
> There was a ‘create the distro’ step of the pipeline which gathered the input,
> followed by a loop that used ‘dd’ to block the stream into media-sized parts.

If space is not an issue, you can use split(1) to divide
the input in N pieces and then use a separate loop to copy
them to the media. If you want to stream the distribution
on stdin but still copy to N disks or whatever, you can
write a simple C program that will prompt the user to switch
media, print out checksum etc. If you want to *not* split
files across media (and no file is greater than media size),
you can use makekit from Rich Salz's cshar (comp.sources.unix
Volume 15).

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[segaloco via COFF]

> There was a ‘create the distro’ step of the pipeline which gathered the input,
> followed by a loop that used ‘dd’ to block the stream into media-sized parts.

If you were taking a stream and breaking it up arbitrarily with dd, does that imply that filesystem data defining the contents were isolated to the first volume (or however many it took to describe the contents)?  Or is there something you could do with dd in that circumstance to amount to more than just bytes in bytes out to prop up a filesystem superblock on each individual volume?  Granted, I don't have a lot of experience in this area, so I could be comparing apples to your oranges for all I know.

- Matt G.

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Steve Jenkin]

> On 27 Jun 2023, at 09:44, Bakul Shah <bakul@iitbombay.org> wrote:
> 
> f space is not an issue, you can use split(1) to divide
> the input in N pieces and then use a separate loop to copy
> them to the media. If you want to stream the distribution
> on stdin but still copy to N disks or whatever, you can
> write a simple C program that will prompt the user to switch
> media, print out checksum etc. If you want to *not* split
> files across media (and no file is greater than media size),
> you can use makekit from Rich Salz's cshar (comp.sources.unix
> Volume 15).

Bakul,

Thanks for the note.
I’d not heard of “makeit” before, will go hunt it down.

1. this was the 1980’s, space was always an issue :)

2. There are many cases where you’ve only got a 
    stream. This distributions were the first
    time I hit this problem type.

3. The distributions were a compressed tar or cpio
     of a directory structure, to be exploded at destination.
     There wasn’t a need to fit files onto media.

cheers
steve j

--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Steve Jenkin]

> On 27 Jun 2023, at 09:32, Clem Cole <clemc@ccc.com> wrote:
> 
> The trick here is understanding how ibs & obs, EOF is handled and probably osync, then how to use iseek (iskip in modern versions).
> It's not that hard. Its been done many times. 
> 
> One of the more interesting issues with dd is most versions still are single threaded which sucks for most media - particularly f you want to stream things.
> There was a wonderful hack to dd done in the. early 1980s by Tapani Lindgren ddd - double dd - which used two processes and a pipe to control them, so one process was always reading while the other was writing.   You can one it Volume 14, issue 85 in the comp.sources.unix archives.
> 
> Years ago, I hacked up a version using threads to do the same thing with a mutex to protect everything (It ever ran on Windows at some point). 
> ᐧ


Clem,

Thanks very much for the note & comments.

Of course, amongst the many fine things you’ve done,
you’d written a proper, high-quality solution to this problem.
I’d expect no less :)

I take your point:

	‘dd’ is a beast, very subtle with lots of options.
	Very good idea to use ‘osync’ to pad the last block written out to same size.
	[ I’ve never though to do that ]

What I remember about the script is that ‘dd’ didn’t have to be told the number of media to be written.
It detected ‘End of File’ on the input (pipe) and terminated the loop.
It’s this action I’ve never solved.

There were a few other things done in the body of the loop:

	writing to the operator the number of the media, to label it
	reading a response from /dev/tty, to allow the operator to change media

		eg:  read -p "OK? " resp </dev/tty 

I’ve constructed two examples below of splitting input into fixed blocks.
using the easiest repeatable & limited stream I thought of:
	ls -1

When I know the number of blocks, a for loop does exactly what I hope & expect.
Including the trailing short block.

But writing a ‘while’ loop with ‘dd’ as the condition - it’s an infinite loop.

Even though STDIN is exhausted and ‘dd’ knows it, read returns zero bytes,
and ‘dd' doesn’t return an error (because there’s no ‘read error’, just ’No Data').

This Mac example uses ‘bash’.

I think in 1985 it was Bourne shell, maybe Korn shell, definitely not ‘csh’.

cheers
steve

==========

iMac1:~/ steve$ ls -1 | wc -c
 1071404

iMac1:~/ steve$ ls -1 | for i in {1..108}; do dd of=/dev/null bs=10000 count=1;  done
1+0 records in
1+0 records out
10000 bytes transferred in 0.249052 secs (40152 bytes/sec)

… another 105 times

10000 bytes transferred in 0.000011 secs (911805217 bytes/sec)

0+1 records in
0+1 records out
1404 bytes transferred in 0.000007 secs (203062166 bytes/sec)

==========

iMac1:~/ steve$ ls -1 | while dd of=/dev/null bs=10000 count=1; do : ;  done	# ‘:’ is a null command within loop

1+0 records in
1+0 records out
10000 bytes transferred in 0.251171 secs (39813 bytes/sec)

… another 105 times

0+1 records in
0+1 records out
1404 bytes transferred in 0.000009 secs (154968495 bytes/sec)

0+0 records in
0+0 records out
0 bytes transferred in 0.000006 secs (0 bytes/sec)

0+0 records in
0+0 records out
0 bytes transferred in 0.000007 secs (0 bytes/sec)

… and it's an infinite loop, repeating the last group of lines :(

==========
--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Ralph Corderoy]

Hi Steve,

> I’ve never figured out how to use ‘dd’ so it returns after a single
> block is written doesn’t close the input, killing the pipeline, or
> cause the rest of the data to be discarded.

I think this meets your description and complies with POSIX's dd(1p)
here.

    $ seq 33 126 | sed 's/$/P/' | dc |
    > while :; do
    >     LC_ALL=C dd bs=10 count=1 2>dd.err | sed -n l
    >     grep -q '^[^0].* records in$' dd.err || break
    > done
    !"#$%&'()*$
    +,-./01234$
    56789:;<=>$
    ?@ABCDEFGH$
    IJKLMNOPQR$
    STUVWXYZ[\\$
    ]^_`abcdef$
    ghijklmnop$
    qrstuvwxyz$
    {|}~$
    $
    $ rm dd.err

I set the locale so the format of dd's stderr report is known.

-- 
Cheers, Ralph.

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Adam Thornton]

[-- Attachment #1: Type: text/plain, Size: 1120 bytes --]

My approach would have been to use "split" on the original file and then dd
the resulting files.  But now I find myself wondering how old "split" is.
It was certainly already a well-established thing by the early 90s.

On Mon, Jun 26, 2023 at 11:23 PM Ralph Corderoy <ralph@inputplus.co.uk>
wrote:

> Hi Steve,
>
> > I’ve never figured out how to use ‘dd’ so it returns after a single
> > block is written doesn’t close the input, killing the pipeline, or
> > cause the rest of the data to be discarded.
>
> I think this meets your description and complies with POSIX's dd(1p)
> here.
>
>     $ seq 33 126 | sed 's/$/P/' | dc |
>     > while :; do
>     >     LC_ALL=C dd bs=10 count=1 2>dd.err | sed -n l
>     >     grep -q '^[^0].* records in$' dd.err || break
>     > done
>     !"#$%&'()*$
>     +,-./01234$
>     56789:;<=>$
>     ?@ABCDEFGH$
>     IJKLMNOPQR$
>     STUVWXYZ[\\$
>     ]^_`abcdef$
>     ghijklmnop$
>     qrstuvwxyz$
>     {|}~$
>     $
>     $ rm dd.err
>
> I set the locale so the format of dd's stderr report is known.
>
> --
> Cheers, Ralph.
>

[-- Attachment #2: Type: text/html, Size: 1593 bytes --]

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Ralph Corderoy]

Hi Adam,

> My approach would have been to use "split" on the original file

I think it's already been said that the input is a stream and the chunks
should be written to the output media on the fly.

-- 
Cheers, Ralph.

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[steve jenkin]

Ralph,

Nice solution:
	examine dd’s STDERR then ‘break’.

Thanks for the test script - I like your generation of characters using ‘dc’.

cheers!
steve

> On 27 Jun 2023, at 16:23, Ralph Corderoy <ralph@inputplus.co.uk> wrote:
> 
> Hi Steve,
> 
>> I’ve never figured out how to use ‘dd’ so it returns after a single
>> block is written doesn’t close the input, killing the pipeline, or
>> cause the rest of the data to be discarded.
> 
> I think this meets your description and complies with POSIX's dd(1p)
> here.
> 
>    $ seq 33 126 | sed 's/$/P/' | dc |
>> while :; do
>>    LC_ALL=C dd bs=10 count=1 2>dd.err | sed -n l
>>    grep -q '^[^0].* records in$' dd.err || break
>> done
>    !"#$%&'()*$
>    +,-./01234$
>    56789:;<=>$
>    ?@ABCDEFGH$
>    IJKLMNOPQR$
>    STUVWXYZ[\\$
>    ]^_`abcdef$
>    ghijklmnop$
>    qrstuvwxyz$
>    {|}~$
>    $
>    $ rm dd.err
> 
> I set the locale so the format of dd's stderr report is known.
> 
> -- 
> Cheers, Ralph.

--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Steffen Nurpmeso]

Steve Jenkin wrote in
 <5C26BE4B-2E6C-4E02-9536-3EA774FB6B47@canb.auug.org.au>:
 |> On 27 Jun 2023, at 09:44, Bakul Shah <bakul@iitbombay.org> wrote:
 |> f space is not an issue, you can use split(1) to divide
 |> the input in N pieces and then use a separate loop to copy
 |> them to the media. If you want to stream the distribution

I had

   act mkdir -p "$target"
   act btrfs send $parent "$this" '|' \
      zstd -zc -T0 $ZSTD_LEVEL '|' \
      '('cd "$target" '&&' \
        echo "$this" '>' .stamp '&&' \
        split -a 4 -b 2000000000 -d -')'
   ) || exit $?

for splitting BTRFS snapshots to VFAT filesystems.
You then did

   act cat "$ball"/"$mydir"/* '|' zstd -dc '|' btrfs receive .

to receive them.  (Where "act" is

  act() {
          if [ -n "$DEBUG" ]; then
                  echo eval "$@"
          else
                  eval "$@"
                  if [ $? -ne 0 ]; then
                          echo 'PANIC: '$*
                          exit 1
                  fi
          fi
  }

).  I dropped that "ball" support, all my backup media now uses
EXT4 or simply BTRFS directly.  (Thing was that Linux cannot drive
some external Seagate USB disks in a way that allows EXT4 or BTRFS
on them, the "final sync" or fails, will all kernels tried, and
some USB hints, too.  MacOS X could create HFS?? just like that.)
(That ".stamp" was

   if [ -f "$ball"/"$mydir"/.stamp ]; then
      snap=$(cat "$ball"/"$mydir"/.stamp)
  ...
   cd snapshots/"$mydir" || exit 11
   if [ -d "$snap" ]; then
      echo '=== '$mydir': snapshot '$snap' already exists'
      exit 0
   fi

.).

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Steffen Nurpmeso]

And i'd wish the shell had "set -o pipefail" much much earlier.
So you (sometimes) do things like

  #(set -o pipefail) >/dev/null 2>&1 && set -o pipefail
  es=$(exec 3>&1 1>&2; { the_worker "$cmd"; echo $? >&3; } | mytee | mymail)

that are super neat (ie that this is possible syntax-wise!), but
terrible and cryptical and surely error-prone to do.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Dave Horsfall]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On Tue, 27 Jun 2023, steve jenkin wrote:

> I’ve tried doing the ‘dd’ in a for or while loop many ways, many times.
> 
> Perhaps you could write & yesy a little demo script for me.
> 
> Say, “ls -1 | for (); do dd bs=10b; done”

The problem is the pipe, of course; no Unix I know will allow a seek on a 
pipe (?), so it will have to be buffered somehow.

I'm a bit tied up right now, so I'll take a look later.  In the meantime 
you seem to have lots of responses...

-- Dave