'>;' redirection operator [was: [1003.1(2008)/Issue 7 0000530]: Support in-place editing in sed (-iEXTENSION)]

'>;' redirection operator [was: [1003.1(2008)/Issue 7 0000530]: Support in-place editing in sed (-iEXTENSION)]

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2100 bytes --]

[cc'ing bash, dash, mksh, and zsh developers; feel free to avoid
cross-posted replies on content not relevant to all the groups]

On 12/22/2011 08:39 AM, David Korn wrote:
> Subject: Re: Re: [1003.1(2008)/Issue 7 0000530]: Support in-place editing in sed  (-iEXTENSION)
> --------
> 
> There are many commands other than sed that want the output to replace
> an input file.  That is why I added the >; redirection operator to ksh93.
> 
> With >; you can do
> 	sed -e s/foo/bar/ file >; file
> to do in place sed.  The >; operator generates the output in a temporary file
> and moves the file to the original file only if the command terminates
> with 0 exit status.

I agree that engineering a single fix into the shell that can apply to
multiple situations, rather than chasing down a set of applications to
add an in-place editing option to each, is a much more flexible and
powerful approach.  Can we get buy-in from other shell developers to
support '>;' as an atomic temp-file replacement-on-success idiom, if
POSIX were to standardize the existing practice of ksh93 as the basis?

I assume on the ksh implementation that the temp file is discarded if
the command (simple or compound) feeding the redirection failed?  If the
redirection is used on a simple command, is there any shorthand for
specifying that the destination name on success also be fed as an
argument to the command, to avoid the redundancy of having to type
'file' both before and after the '>;' operator?  I assume that this is
like any other redirection operator, where an optional fd number can be
prepended, as in '2>; file' to collect stderr and overwrite file on
success?  What happens if there is more than one '>;' redirection in the
same command, and both target the same end file (whether or not by the
same file name)?  What happens if the command succeeds, but the rename
of the temp file to the destination fails?  Are there clobber ('>|') or
append ('>>') variants?

-- 
Eric Blake   eblake@redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: '>;' redirection operator [was: [1003.1(2008)/Issue 7 0000530]: Support in-place editing in sed (-iEXTENSION)]

[Bruce Korb]

On 12/22/11 13:03, Eric Blake wrote:
> I assume on the ksh implementation that the temp file is discarded if
> the command (simple or compound) feeding the redirection failed?

One would hope!

>  If the
> redirection is used on a simple command, is there any shorthand for
> specifying that the destination name on success also be fed as an
> argument to the command, to avoid the redundancy of having to type
> 'file' both before and after the'>;' operator?

Doesn't the shell already have enough hieroglyphs?  It is what
intimidates many folks from figuring it out.

>  I assume that this is
> like any other redirection operator, where an optional fd number can be
> prepended, as in '2>; file' to collect stderr and overwrite file on
> success?

When the exact opposite is the useful variation?  I.e. keep-on-failure.
"-i" for sed is simple, understandable and implemented a lot.
Please don't add another glyph to the standardized shell.  Let us not
slide on slippery slopes.  Shells can always add some useful builtins:

    sh_move_if_changed
    sh_save_on_success
    sh_save_on_failure

to cope with this stuff.  Or you can write your own such library for
your own use.  ">;" is not an answer for sed-as-a-batch-editor anyway,
which is what "-i" really is.

Re: '>;' redirection operator

[Thorsten Glaser]

Eric Blake dixit:

>powerful approach.  Can we get buy-in from other shell developers to
>support '>;' as an atomic temp-file replacement-on-success idiom, if

Urgh, PLEASE NOT!

People complain about the readability of code enough already, and as
practice shows, things like [[ have been around and nobody uses them
anyway (often using just POSIX, but not even knowing – myself included
– that POSIX sh has $((…))⁺; or even using less-than-POSIX, e.g. in
autoconf, which means that anything we were to introduce now would not
be used in the places where it counts anyway, for compatibility).

⁺) Reminds me to write to the list about that. Buried in dayjob work
   atm though. Expect something about that next year.


Bruce Korb dixit:

> slide on slippery slopes.  Shells can always add some useful builtins:
>
>   sh_move_if_changed
>   sh_save_on_success
>   sh_save_on_failure

In mksh, practice is to keep such things out of the core code and
optionally put it into ~/.mkshrc instead. The pushd/popd/dirs code
is a prime example of it. Also, this way, the shell is extended in
shell instead of in C. (I’ve seen the C201x draft this week. This
drives home _that_ point even better. That’s bloat, not C any more.)

Many languages have standard libraries written in that language
itself, for better portability and easier maintenance, so I’d say
do it like that. Heck, https://evolvis.org/projects/shellsnippets/
(disclaimer: a pet project of a coworker and me) is waiting for
more contributions. (Hosted at my current employer, that’s why I
untypically-for-me chose git so nobody needs to fear they could
take it down.)


Oh, and: sed. has. no. -i. option. either. Please. There’s a
perfectly fine ed, man! man ed! for that.

bye,
//mirabilos
-- 
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh

Re: '>;' redirection operator [was: [1003.1(2008)/Issue 7 0000530]: Support in-place editing in sed (-iEXTENSION)]

[Geir Hauge]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

2011/12/22 Bruce Korb <bkorb@gnu.org>
>
> When the exact opposite is the useful variation?  I.e. keep-on-failure.
> "-i" for sed is simple, understandable and implemented a lot.
>

As far as I know, -i is only implemented with GNU sed and BSD sed, and they
are incompatible, BSD sed's -i takes a mandatory argument, while GNU sed's
-i takes an optional string which must be provided in the same argument.
E.g.

gnused -i.bak sed-script file
bsdsed -i .bak sed-script file

So the only portable way of using sed to "edit" (read: overwrite) a file is
with

sed sed-script file > file.tmp && mv file.tmp file

I'd welcome this >; syntax.

-- 
Geir Hauge

<> redirection operator

[Clint Adams]

Is this doing what it's meant to do?

% echo test >/tmp/redirtest

% sed 's/e/Z/g' <>/tmp/redirtest
tZst

Re: <> redirection operator

[Bart Schaefer]

On Jun 3,  4:10pm, Clint Adams wrote:
}
} Is this doing what it's meant to do?

Yes.

<> WORD
     Open file WORD for reading and writing as standard input.
                                            ^^^^^^^^^^^^^^^^^

All this means is that fd 0 is O_RDWR rather than O_RDONLY.  It still
opens exactly one descriptor.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: <> redirection operator

[Clint Adams]

> All this means is that fd 0 is O_RDWR rather than O_RDONLY.  It still
> opens exactly one descriptor.

Ah.  What is the point of that?

Re: <> redirection operator

[Peter Stephenson]

(Sigh.  After half a dozen years of the mailing list sending replies to the
original sender, I'm still forgetting to change the address.)

Clint Adams wrote:
> > All this means is that fd 0 is O_RDWR rather than O_RDONLY.  It still
> > opens exactly one descriptor.
> 
> Ah.  What is the point of that?

To put it another way,

% echo test >/tmp/redirtest
% sed 's/e/Z/g' <>/tmp/redirtest 1>&0
% cat /tmp/redirtest
tZst

(This happens to work in place even on a mult-line file, but I think you
have to be pretty sure what you're doing to use it like that.  Finally I've
got an example of this for the user guide, anyway.)

It's just consistently applying the rule that one redirection redirects
exactly one file descriptor, and to do the other you need to do it
explcitly as above.

-- 
Peter Stephenson <pws@pwstephenson.fsnet.co.uk>
Work: pws@csr.com
Web: http://www.pwstephenson.fsnet.co.uk

Re: <> redirection operator

[Bart Schaefer]

On Jun 4, 12:14am, Peter Stephenson wrote:
} 
} % echo test >/tmp/redirtest
} % sed 's/e/Z/g' <>/tmp/redirtest 1>&0
} % cat /tmp/redirtest
} tZst
} 
} (This happens to work in place even on a mult-line file, but I think you
} have to be pretty sure what you're doing to use it like that.  Finally I've
} got an example of this for the user guide, anyway.)

I gave an example of using <> a few months ago ... let's see ... ah, here it
is, sent to zsh-users in December:

------------
Yes, <> means "open for both reading and writing," which is sometimes
useful as it's the mode in which stdin/out/err are typically opened on
a terminal.  The implicit descriptor to the left of <> is 0, so unless
you supply another number it opens stdin.

If you give a descriptor number, you can use <> to open a file for
overwrite-without-truncation:

    zsh% print "foo\nbar" >| foo
    zsh% <foo
    foo
    bar
    zsh% print xxx 1<>foo
    zsh% <foo
    xxx
    bar

So if you know that e.g. sed is either going to fail to produce any output
at all, or will succeed with exactly as much output as the length of the
original input, you can do

    sed 's/foo/bar/g' <foo 1<>foo

But this is pretty dangerous because if sed ever writes more bytes than it
has already read, it'll start re-reading what it wrote.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net