[COFF] Re: Shell script advice: using 'dd' to write multiple media

[Steve Jenkin <sjenkin@canb.auug.org.au>]

> On 27 Jun 2023, at 09:32, Clem Cole <clemc@ccc.com> wrote:
> 
> The trick here is understanding how ibs & obs, EOF is handled and probably osync, then how to use iseek (iskip in modern versions).
> It's not that hard. Its been done many times. 
> 
> One of the more interesting issues with dd is most versions still are single threaded which sucks for most media - particularly f you want to stream things.
> There was a wonderful hack to dd done in the. early 1980s by Tapani Lindgren ddd - double dd - which used two processes and a pipe to control them, so one process was always reading while the other was writing.   You can one it Volume 14, issue 85 in the comp.sources.unix archives.
> 
> Years ago, I hacked up a version using threads to do the same thing with a mutex to protect everything (It ever ran on Windows at some point). 
> ᐧ


Clem,

Thanks very much for the note & comments.

Of course, amongst the many fine things you’ve done,
you’d written a proper, high-quality solution to this problem.
I’d expect no less :)

I take your point:

	‘dd’ is a beast, very subtle with lots of options.
	Very good idea to use ‘osync’ to pad the last block written out to same size.
	[ I’ve never though to do that ]

What I remember about the script is that ‘dd’ didn’t have to be told the number of media to be written.
It detected ‘End of File’ on the input (pipe) and terminated the loop.
It’s this action I’ve never solved.

There were a few other things done in the body of the loop:

	writing to the operator the number of the media, to label it
	reading a response from /dev/tty, to allow the operator to change media

		eg:  read -p "OK? " resp </dev/tty 

I’ve constructed two examples below of splitting input into fixed blocks.
using the easiest repeatable & limited stream I thought of:
	ls -1

When I know the number of blocks, a for loop does exactly what I hope & expect.
Including the trailing short block.

But writing a ‘while’ loop with ‘dd’ as the condition - it’s an infinite loop.

Even though STDIN is exhausted and ‘dd’ knows it, read returns zero bytes,
and ‘dd' doesn’t return an error (because there’s no ‘read error’, just ’No Data').

This Mac example uses ‘bash’.

I think in 1985 it was Bourne shell, maybe Korn shell, definitely not ‘csh’.

cheers
steve

==========

iMac1:~/ steve$ ls -1 | wc -c
 1071404

iMac1:~/ steve$ ls -1 | for i in {1..108}; do dd of=/dev/null bs=10000 count=1;  done
1+0 records in
1+0 records out
10000 bytes transferred in 0.249052 secs (40152 bytes/sec)

… another 105 times

10000 bytes transferred in 0.000011 secs (911805217 bytes/sec)

0+1 records in
0+1 records out
1404 bytes transferred in 0.000007 secs (203062166 bytes/sec)

==========

iMac1:~/ steve$ ls -1 | while dd of=/dev/null bs=10000 count=1; do : ;  done	# ‘:’ is a null command within loop

1+0 records in
1+0 records out
10000 bytes transferred in 0.251171 secs (39813 bytes/sec)

… another 105 times

0+1 records in
0+1 records out
1404 bytes transferred in 0.000009 secs (154968495 bytes/sec)

0+0 records in
0+0 records out
0 bytes transferred in 0.000006 secs (0 bytes/sec)

0+0 records in
0+0 records out
0 bytes transferred in 0.000007 secs (0 bytes/sec)

… and it's an infinite loop, repeating the last group of lines :(

==========
--
Steve Jenkin, IT Systems and Design 
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA

mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin