* [RFC] adding zmktemp command
@ 2019-03-27 21:16 ` Clinton Bunch
2019-03-28 9:38 ` Daniel Shahaf
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Clinton Bunch @ 2019-03-27 21:16 UTC (permalink / raw)
To: zsh-workers
I'm thinking of adding a zmktemp command either in a new module (e.g.
zsh/tempfile) or in the zsh/files module.
My rationale is that mktemp is really useful for creating temporary
files but is not available by default on all systems and on some on
which it is available its options and function can diverge wildly (HP-UX
is an especially egregious example). Building it into the shell would
give a consistent interface for zsh scripts to create temporary files.
The advantage I can see to being it's own module, is it could be
backported and separately packed for Long-Term Supported OS. (Mostly
thinking of Enterprise Linux distributions)
But it also seems like it would be a good fit into the zsh/files module
(mktemp and zf_mktemp instead of zmktemp)
Thoughts?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-27 21:16 ` [RFC] adding zmktemp command Clinton Bunch
@ 2019-03-28 9:38 ` Daniel Shahaf
2019-03-28 15:00 ` Clinton Bunch
2019-03-28 15:17 ` Peter Stephenson
2019-03-28 15:36 ` Oliver Kiddle
2 siblings, 1 reply; 10+ messages in thread
From: Daniel Shahaf @ 2019-03-28 9:38 UTC (permalink / raw)
To: zsh-workers
Clinton Bunch wrote on Wed, 27 Mar 2019 21:18 +00:00:
> I'm thinking of adding a zmktemp command either in a new module (e.g.
> zsh/tempfile) or in the zsh/files module.
...
> Thoughts?
A few.
- I wonder if implementing mktemp in the shell is easier than expecting
people to install a third-party mktemp(1) binary with whatever
functionality they desire. BSD systems often have both BSD make and
GNU make, so it's conceivable that HP-UX systems could have both the
native mktemp(1) and a third-party one.
(To be clear, I do not object to your RFC; I just wonder if there's a
better solution to the underlying problem.)
- There's already a gettempname() function in the shell's C implementation;
it relies on mktemp(3) being available. A module implementation might
be able to reuse that.
- O_EXCL is exposed by zsh/system's 'sysopen' builtin, so a pure zsh
implementation should be possible.
(I think you're aware of the following, but for the record:)
There is a *limited* workaround: '() { … } =(:)' creates a tempfile (in
${TMPPREFIX:h}, if that's set). The catch is that the file is deleted
when the anonymous function returns, so it's effectively a lexically
scoped tempfile. And, of course, it's not a substitute for «mktemp -d».
(So, no, that's not a replacement to a proper mktemp(1).)
Cheers,
Daniel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 9:38 ` Daniel Shahaf
@ 2019-03-28 15:00 ` Clinton Bunch
2019-03-28 21:13 ` Daniel Shahaf
0 siblings, 1 reply; 10+ messages in thread
From: Clinton Bunch @ 2019-03-28 15:00 UTC (permalink / raw)
To: Daniel Shahaf, zsh-workers
On 3/28/2019 4:38 AM, Daniel Shahaf wrote:
> Clinton Bunch wrote on Wed, 27 Mar 2019 21:18 +00:00:
>> I'm thinking of adding a zmktemp command either in a new module (e.g.
>> zsh/tempfile) or in the zsh/files module.
> ...
>> Thoughts?
> A few.
>
> - I wonder if implementing mktemp in the shell is easier than expecting
> people to install a third-party mktemp(1) binary with whatever
> functionality they desire. BSD systems often have both BSD make and
> GNU make, so it's conceivable that HP-UX systems could have both the
> native mktemp(1) and a third-party one.
>
> (To be clear, I do not object to your RFC; I just wonder if there's a
> better solution to the underlying problem.)
That situation is why I proposed this. On my HP-UX systems I use gnu
coreutils mktemp, but either I have to order my path so that it's before
/usr/bin, which can get me non-standard versions of standard commands
which might affect the script, or name it something else (which I did,
gmktemp). Either way this makes for less portable scripts. That also
requires that the script writer have access to install packages or the
wherewithal to build these packages and install them in their home
directory themselves.
>
> - There's already a gettempname() function in the shell's C implementation;
> it relies on mktemp(3) being available. A module implementation might
> be able to reuse that.
Actually I was thinking about modifying this as well to use either a
system mkstemps where available or a hand-rolled one. The names
generated on HP-UX by mktemp(3) are extremely predictable (basically it
zero-pads the pid and starts incrementing the leftmost digit on
subsequent calls. This is likely to be a bigger deal in a shell script
where the tempfile might be repeatedly opened and closed and the name
handed off as an argument to an external command, than internally where
all I/O to the file is done with an already opened file descriptor.
>
> - O_EXCL is exposed by zsh/system's 'sysopen' builtin, so a pure zsh
> implementation should be possible.
I didn't think about a pure zsh implementation, but modifying the
template character by character in zsh sounds like at least as much work
as it is in C, but slower.
>
> (I think you're aware of the following, but for the record:)
>
> There is a *limited* workaround: '() { … } =(:)' creates a tempfile (in
> ${TMPPREFIX:h}, if that's set). The catch is that the file is deleted
> when the anonymous function returns, so it's effectively a lexically
> scoped tempfile. And, of course, it's not a substitute for «mktemp -d».
> (So, no, that's not a replacement to a proper mktemp(1).)
>
> Cheers,
>
> Daniel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-27 21:16 ` [RFC] adding zmktemp command Clinton Bunch
2019-03-28 9:38 ` Daniel Shahaf
@ 2019-03-28 15:17 ` Peter Stephenson
2019-03-28 15:25 ` Clinton Bunch
2019-03-28 15:36 ` Oliver Kiddle
2 siblings, 1 reply; 10+ messages in thread
From: Peter Stephenson @ 2019-03-28 15:17 UTC (permalink / raw)
To: zsh-workers
On Wed, 2019-03-27 at 16:16 -0500, Clinton Bunch wrote:
> But it also seems like it would be a good fit into the zsh/files module
> (mktemp and zf_mktemp instead of zmktemp)
>
> Thoughts?
It occurs to me that one reason for doing this might be that you could
add a zf_mkstemp function (no reason for compatibility with any external
command) that both creates and opens a file using mkstemp(), returning
the file descriptor in REPLY as usual. This is then secure.
That's something that's hard to do without a builtin.
pws
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 15:17 ` Peter Stephenson
@ 2019-03-28 15:25 ` Clinton Bunch
0 siblings, 0 replies; 10+ messages in thread
From: Clinton Bunch @ 2019-03-28 15:25 UTC (permalink / raw)
To: Peter Stephenson, zsh-workers
On 3/28/2019 10:17 AM, Peter Stephenson wrote:
> On Wed, 2019-03-27 at 16:16 -0500, Clinton Bunch wrote:
>> But it also seems like it would be a good fit into the zsh/files module
>> (mktemp and zf_mktemp instead of zmktemp)
>>
>> Thoughts?
> It occurs to me that one reason for doing this might be that you could
> add a zf_mkstemp function (no reason for compatibility with any external
> command) that both creates and opens a file using mkstemp(), returning
> the file descriptor in REPLY as usual. This is then secure.
> That's something that's hard to do without a builtin.
>
> pws
I was actually planning on adding a -f option to return a fd as the
zsystem flock command does. This could then be used in print -u,
sysseek, and read -u statements. I would also probably use mkstemps
(hand-rolled where not available in libc) as suffixes can be a good
thing. This would also relieve the predictability issue with HP-UX's
mkstemp.
I didn't think about using REPLY by default. That could cause issues
with use of read -u if they rewind the file
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-27 21:16 ` [RFC] adding zmktemp command Clinton Bunch
2019-03-28 9:38 ` Daniel Shahaf
2019-03-28 15:17 ` Peter Stephenson
@ 2019-03-28 15:36 ` Oliver Kiddle
2019-03-28 15:40 ` Clinton Bunch
2 siblings, 1 reply; 10+ messages in thread
From: Oliver Kiddle @ 2019-03-28 15:36 UTC (permalink / raw)
To: Clinton Bunch; +Cc: zsh-workers
Clinton Bunch wrote:
> I'm thinking of adding a zmktemp command either in a new module (e.g.
> The advantage I can see to being it's own module, is it could be
> backported and separately packed for Long-Term Supported OS. (Mostly
> thinking of Enterprise Linux distributions)
I doubt you'd be able to persuade such distributions to include even a
completely new and separate module.
> But it also seems like it would be a good fit into the zsh/files module
> (mktemp and zf_mktemp instead of zmktemp)
zsh/files would mostly make sense for a mktemp builtin that closely
follows the interface of the normal mktemp Unix utility. If the focus is
to provide a consistent and convenient way to create temporary files,
then following mktemp is perhaps not the best choice. For security
reasons, an interface that returns a file handle is, I think, generally
preferred these days. Perhaps sysopen (in zsh/system) could accept a -t
option instead of a filename.
Oliver
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 15:36 ` Oliver Kiddle
@ 2019-03-28 15:40 ` Clinton Bunch
2019-03-28 20:26 ` Daniel Shahaf
0 siblings, 1 reply; 10+ messages in thread
From: Clinton Bunch @ 2019-03-28 15:40 UTC (permalink / raw)
To: Oliver Kiddle; +Cc: zsh-workers
On 3/28/2019 10:36 AM, Oliver Kiddle wrote:
> Clinton Bunch wrote:
>> I'm thinking of adding a zmktemp command either in a new module (e.g.
>> The advantage I can see to being it's own module, is it could be
>> backported and separately packed for Long-Term Supported OS. (Mostly
>> thinking of Enterprise Linux distributions)
> I doubt you'd be able to persuade such distributions to include even a
> completely new and separate module.
>
>> But it also seems like it would be a good fit into the zsh/files module
>> (mktemp and zf_mktemp instead of zmktemp)
> zsh/files would mostly make sense for a mktemp builtin that closely
> follows the interface of the normal mktemp Unix utility. If the focus is
> to provide a consistent and convenient way to create temporary files,
> then following mktemp is perhaps not the best choice. For security
> reasons, an interface that returns a file handle is, I think, generally
> preferred these days. Perhaps sysopen (in zsh/system) could accept a -t
> option instead of a filename.
>
> Oliver
I was planning on adding a -f option to return a fd in a parameter. The
sysopen solution though wouldn't give you the filename which would mean
you couldn't pass it to an external program. I realize that that has
some security issues which is why an unpredictable filename is needed.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 15:40 ` Clinton Bunch
@ 2019-03-28 20:26 ` Daniel Shahaf
0 siblings, 0 replies; 10+ messages in thread
From: Daniel Shahaf @ 2019-03-28 20:26 UTC (permalink / raw)
To: zsh-workers
Clinton Bunch wrote on Thu, 28 Mar 2019 15:42 +00:00:
> On 3/28/2019 10:36 AM, Oliver Kiddle wrote:
> > preferred these days. Perhaps sysopen (in zsh/system) could accept a -t
> > option instead of a filename.
> >
> > Oliver
>
> I was planning on adding a -f option to return a fd in a parameter. The
> sysopen solution though wouldn't give you the filename which would mean
> you couldn't pass it to an external program. I realize that that has
> some security issues which is why an unpredictable filename is needed.
The filename could be passed back from sysopen in a parameter, just like
that -f you describe here.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 15:00 ` Clinton Bunch
@ 2019-03-28 21:13 ` Daniel Shahaf
2019-03-29 15:20 ` Clinton Bunch
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Shahaf @ 2019-03-28 21:13 UTC (permalink / raw)
To: zsh-workers
Clinton Bunch wrote on Thu, Mar 28, 2019 at 10:00:24 -0500:
> On 3/28/2019 4:38 AM, Daniel Shahaf wrote:
> > Clinton Bunch wrote on Wed, 27 Mar 2019 21:18 +00:00:
> > > I'm thinking of adding a zmktemp command either in a new module (e.g.
> > > zsh/tempfile) or in the zsh/files module.
> > ...
> > > Thoughts?
> > A few.
> >
> > - I wonder if implementing mktemp in the shell is easier than expecting
> > people to install a third-party mktemp(1) binary with whatever
> > functionality they desire. BSD systems often have both BSD make and
> > GNU make, so it's conceivable that HP-UX systems could have both the
> > native mktemp(1) and a third-party one.
> > (To be clear, I do not object to your RFC; I just wonder if there's a
> > better solution to the underlying problem.)
>
> That situation is why I proposed this. On my HP-UX systems I use gnu
> coreutils mktemp, but either I have to order my path so that it's before
> /usr/bin, which can get me non-standard versions of standard commands which
> might affect the script,
You neglected to explain why none of the other possible solutions to
this subproblem is suitable for you. (For starters, there are 'add a
directory to the front of $PATH that contains just GNU mktemp and
nothing else' and 'use the "hash" builtin to specify a different mktemp
than the one in $PATH'.)
> or name it something else (which I did, gmktemp).
> Either way this makes for less portable scripts.
So your problem statement is "HP-UX and Linux use incompatible mktemp(1)
binaries". I don't understand why that should be fixed in zsh;
shouldn't that be fixed by getting HP-UX to improve their mktemp?
Compare how there is any number of instances in the FreeBSD man pages of
option flags that have been added for compatibility with coreutils (see
ls(1) and find(1) for example).
(By the way: I wonder if mktemp(1) will be added to POSIX?)
As to your proposal itself, I initially thought you were proposing to
implement a drop-in replacement of some mktemp(1) out there (probably
GNU's, though for license reasons it'd be easier to crib BSD's);
however, reading your responses to Peter and Oliver I see that you might
be thinking of adding an *idiomatic* make-temporary-files interface,
e.g., one that returns an fd and/or returns the filename in REPLY to
save a fork. Which is it? Could you sketch the API that will be
provided to script authors? Is it "see GNU coreutils' mktemp(1) man
page, plus the -f option to return an fd"?
> That also requires that
> the script writer have access to install packages or the wherewithal to
> build these packages and install them in their home directory themselves.
By this argument, we should ship an rsync implementation in zsh if HP-UX
doesn't happen to ship rsync in part of its (HP-UX's) default installation.
> > - O_EXCL is exposed by zsh/system's 'sysopen' builtin, so a pure zsh
> > implementation should be possible.
>
> I didn't think about a pure zsh implementation, but modifying the template
> character by character in zsh sounds like at least as much work as it is in
> C, but slower.
Given that there's going to be a syscall at the end anyway [open(O_EXCL)],
I'm not sure if the overhead of zsh over pure C would be noticeable.
I mentioned a pure zsh implementation because it could be implemented as
an autoloaded function and released as a plugin (rather than a module),
so it would be installable by users who don't or can't compile their own
zsh, and it would even be compatible with existing zsh binaries out
there.
Cheers,
Daniel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] adding zmktemp command
2019-03-28 21:13 ` Daniel Shahaf
@ 2019-03-29 15:20 ` Clinton Bunch
0 siblings, 0 replies; 10+ messages in thread
From: Clinton Bunch @ 2019-03-29 15:20 UTC (permalink / raw)
To: Daniel Shahaf, zsh-workers
On 3/28/2019 4:13 PM, Daniel Shahaf wrote:
> Clinton Bunch wrote on Thu, Mar 28, 2019 at 10:00:24 -0500:
>> On 3/28/2019 4:38 AM, Daniel Shahaf wrote:
>>> Clinton Bunch wrote on Wed, 27 Mar 2019 21:18 +00:00:
>>>> I'm thinking of adding a zmktemp command either in a new module (e.g.
>>>> zsh/tempfile) or in the zsh/files module.
>>> ...
>>>> Thoughts?
>>> A few.
>>>
>>> - I wonder if implementing mktemp in the shell is easier than expecting
>>> people to install a third-party mktemp(1) binary with whatever
>>> functionality they desire. BSD systems often have both BSD make and
>>> GNU make, so it's conceivable that HP-UX systems could have both the
>>> native mktemp(1) and a third-party one.
>>> (To be clear, I do not object to your RFC; I just wonder if there's a
>>> better solution to the underlying problem.)
>> That situation is why I proposed this. On my HP-UX systems I use gnu
>> coreutils mktemp, but either I have to order my path so that it's before
>> /usr/bin, which can get me non-standard versions of standard commands which
>> might affect the script,
> You neglected to explain why none of the other possible solutions to
> this subproblem is suitable for you. (For starters, there are 'add a
> directory to the front of $PATH that contains just GNU mktemp and
> nothing else' and 'use the "hash" builtin to specify a different mktemp
> than the one in $PATH'.)
Didn't think of the hash builtin.
>
>> or name it something else (which I did, gmktemp).
>> Either way this makes for less portable scripts.
> So your problem statement is "HP-UX and Linux use incompatible mktemp(1)
> binaries". I don't understand why that should be fixed in zsh;
> shouldn't that be fixed by getting HP-UX to improve their mktemp?
> Compare how there is any number of instances in the FreeBSD man pages of
> option flags that have been added for compatibility with coreutils (see
> ls(1) and find(1) for example).
Fixing the incompatibilities of HP-UX mktemp requires more than adding a
few ignored options. It doesn't make temporary directories and in fact
uses -d to specify the directory to create the file. It doesn't take a
template string. And most importantly doesn't create the temporary file
by default (you have to give a -c option to do so). It also uses the
highly predictable HP-UX mktemp(3) to generate the name.
>
> (By the way: I wonder if mktemp(1) will be added to POSIX?)
It should, but who knows if it will.
>
> As to your proposal itself, I initially thought you were proposing to
> implement a drop-in replacement of some mktemp(1) out there (probably
> GNU's, though for license reasons it'd be easier to crib BSD's);
> however, reading your responses to Peter and Oliver I see that you might
> be thinking of adding an *idiomatic* make-temporary-files interface,
> e.g., one that returns an fd and/or returns the filename in REPLY to
> save a fork. Which is it? Could you sketch the API that will be
> provided to script authors? Is it "see GNU coreutils' mktemp(1) man
> page, plus the -f option to return an fd"?
I'm thinking of something that *could* be used as a drop-in replacement
for mktemp (from the online man pages I can find, other than HP-UX, the
other *nix with mktemp accept the same short options as GNU), but I want
to make it better (that seems to be the zsh way :)
The -f var option mentioned elsewhere to return the opened file descriptor.
a -i (invisible) that will unlink the file as soon as it's opened. (Not
sure what the use case for this is, but there must be one as both
stdio's tempfile and perl's File::Temp (as an option) offer this
functionality)
a -E to erase the file on exit (if I can figure out how to use the exit
hook)
a -e to clear the FD_CLOEXEC on the file descriptor so external programs
could inherit the open file descriptor (perhaps useful on those systems
with /dev/fd or with other specially written programs)
>
>> That also requires that
>> the script writer have access to install packages or the wherewithal to
>> build these packages and install them in their home directory themselves.
> By this argument, we should ship an rsync implementation in zsh if HP-UX
> doesn't happen to ship rsync in part of its (HP-UX's) default installation.
I suppose that argument could be made, but opening temp files is a much
more common use case than rsync. To be honest I've used temp files in
scripts a lot more often than some of the other builtins in zsh/files
like chgrp or chown
>
>>> - O_EXCL is exposed by zsh/system's 'sysopen' builtin, so a pure zsh
>>> implementation should be possible.
>> I didn't think about a pure zsh implementation, but modifying the template
>> character by character in zsh sounds like at least as much work as it is in
>> C, but slower.
> Given that there's going to be a syscall at the end anyway [open(O_EXCL)],
> I'm not sure if the overhead of zsh over pure C would be noticeable.
>
> I mentioned a pure zsh implementation because it could be implemented as
> an autoloaded function and released as a plugin (rather than a module),
> so it would be installable by users who don't or can't compile their own
> zsh, and it would even be compatible with existing zsh binaries out
> there.
That wouldn't allow the -i option or the -e mentioned above.
> Cheers,
>
> Daniel
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-03-29 15:23 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CGME20190327211847epcas2p395ce077891a4535f19e1e9605217df7c@epcas2p3.samsung.com>
2019-03-27 21:16 ` [RFC] adding zmktemp command Clinton Bunch
2019-03-28 9:38 ` Daniel Shahaf
2019-03-28 15:00 ` Clinton Bunch
2019-03-28 21:13 ` Daniel Shahaf
2019-03-29 15:20 ` Clinton Bunch
2019-03-28 15:17 ` Peter Stephenson
2019-03-28 15:25 ` Clinton Bunch
2019-03-28 15:36 ` Oliver Kiddle
2019-03-28 15:40 ` Clinton Bunch
2019-03-28 20:26 ` Daniel Shahaf
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/zsh/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).