cat as a builtin command

cat as a builtin command

[Izumi Natsuka]

Hello, I'm going to write a shell function that provides a basic functionality(print the content of a file or stdin) of cat[0], in order to avoid forking too many process when I call it in a loop[1]. And I have put the following in my zshrc:

cat() {
	if [[ $# -le 1 ]] && [[ ${1:0:1} != '-' || ${1} == '-' ]];then
		local file
		exec {file}<${1:-0}
		read -Erd '' -u ${file}
		exec {file}>&-
	else
		command cat $@
	fi
}

It works perfectly except when I want to cat a binary file:

$ zstat +size archlinux-2012.09.07-dual.iso
411041792
$ cat archlinux-2012.09.07-dual.iso | wc -c
39

Seems that the file was cut by some special raw bytes. As I don't known how to avoid that I tried to use another method(via sysread):

cat() {
	if [[ $# -le 1 ]] && [[ ${1:0:1} != '-' || ${1} == '-' ]];then
		local file REPLY
		exec {file}<${1:-0}
		if [[ -t 1 ]] && [[ $# -eq 0 || ${1} == '-' ]];then
			read -Erd '' -u ${file}
		else
			sysread -i ${file} -o 1 -s $(zstat +size ${1})
		fi
		exec {file}>&-
	else
		command cat $@
	fi
}

It works with raw files. Unfortunately this method doesn't avoid the use of subshell[2], it doesn't fit my need[1] and also doesn't work when you want to cat a fifo.

Is there any way that can perform the basic functionality of cat without calling external command?

Izumi Natsuka

0: https://www.gnu.org/software/coreutils/manual/html_node/cat-invocation.html
1: https://www.ohnekontur.de/2013/02/17/making-cat-a-shell-builtin/
2: http://stackoverflow.com/questions/21976606

Re: cat as a builtin command

[Roman Neuhauser]

# izumi.natsuka@hotmail.com / 2014-08-29 09:40:29 +0800:
> Hello, I'm going to write a shell function that provides a basic
> functionality (print the content of a file or stdin) of cat[0], in
> order to avoid forking too many process when I call it in a loop[1].

> 		exec {file}<${1:-0}
> 		read -Erd '' -u ${file}
 
> It works perfectly except when I want to cat a binary file:
> 
> $ zstat +size archlinux-2012.09.07-dual.iso
> 411041792
> $ cat archlinux-2012.09.07-dual.iso | wc -c
> 39
> 
> Seems that the file was cut by some special raw bytes.

i thought i could get it through with empty $IFS, alas, it stops on
the first ^M anyway.  which is what the manpage says:

  Read one line and break it into fields using the characters in $IFS
  as separators, except as noted below.

there *is* a way, it's just slow as hell:

  setopt localoptions nomultibyte
  declare -A st
  zstat -H st +size ${1:-0}
  read -Erd '' -u ${file} -k ${st[size]}

> As I don't known how to avoid that I tried to use another method (via
> sysread):

> 			sysread -i ${file} -o 1 -s $(zstat +size ${1})

see `zstat -H` above.

none of that solves the FIFO problem.

> Is there any way that can perform the basic functionality of cat
> without calling external command?

i'd go for a zsh module (see zshmodules(1)).  the sources for
zsh/example and zsh/files should give you a kickstart (i found
them useful for my needs).

-- 
roman

Re: cat as a builtin command

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

Am 29.08.2014 um 03:40 schrieb Izumi Natsuka:

> Hello, I'm going to write a shell function that provides a basic
> functionality(print the content of a file or stdin) of cat[0], in
> order to avoid forking too many process when I call it in a loop[1].

I try to avoid cat alltogether, as many uses belong to the "Useless use
of cat" category, like this one:

>
> $ cat archlinux-2012.09.07-dual.iso | wc -c
> 39

Could also be "wc -c archlinux-2012.09.07-dual.iso".

Results in mostly cat-free shell scripts (I like dog(s) more, anyway ;-) )

HTH...

    Dirk
-- 

*Dirk Heinrichs*, Senior Systems Engineer, Engineering Solutions
*Recommind GmbH*, Von-Liebig-Straße 1, 53359 Rheinbach
*Tel*: +49 2226 1596666 (Ansage) 1149
*Email*: dhs@recommind.com <mailto:dhs@recommind.com>
*Skype*: dirk.heinrichs.recommind
www.recommind.com <http://www.recommind.com>

[-- Attachment #2.1: Type: text/html, Size: 1717 bytes --]

[-- Attachment #2.2: Logo.gif --]
[-- Type: image/gif, Size: 1537 bytes --]

Re: cat as a builtin command

[Peter Stephenson]

On Fri, 29 Aug 2014 09:40:29 +0800
Izumi Natsuka <izumi.natsuka@hotmail.com> wrote:
> Hello, I'm going to write a shell function that provides a basic
> functionality(print the content of a file or stdin) of cat[0], in
> order to avoid forking too many process when I call it in a
> loop[1]. And I have put the following in my zshrc:

If it's just printing the contents of a file, investigate zsh/mapfile.
This turns off multibyte mode to make it easier to do binary files ---
there should be no disadvantage here as you just want a byte stream.


mycat() {
  emulate -L zsh
  unsetopt multibyte
  zmodload zsh/mapfile
  print -r $mapfile[$1]
}


This only works for files in the file system, not file descriptors,
pipes etc.  For those you can often optimise out the cat entirely.

pws

Re: cat as a builtin command

[Roman Neuhauser]

# dhs@recommind.com / 2014-08-29 14:24:38 +0200:
> Am 29.08.2014 um 03:40 schrieb Izumi Natsuka:
> 
> > Hello, I'm going to write a shell function that provides a basic
> > functionality(print the content of a file or stdin) of cat[0], in
> > order to avoid forking too many process when I call it in a loop[1].
> 
> I try to avoid cat alltogether, as many uses belong to the "Useless use
> of cat" category, like this one:
> 
> >
> > $ cat archlinux-2012.09.07-dual.iso | wc -c
> > 39
> 
> Could also be "wc -c archlinux-2012.09.07-dual.iso".

definitely, but that doesn't fit the no-fork(2) restriction;
i understand the OP used wc(1) simply to demonstrate the defect
in his implementation.

-- 
roman

Re: cat as a builtin command

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

Am 29.08.2014 um 15:20 schrieb Roman Neuhauser:
> definitely, but that doesn't fit the no-fork(2) restriction;
> i understand the OP used wc(1) simply to demonstrate the defect
> in his implementation.

Sure. But OTOH he also didn't state how he uses cat in his loop, so I
wanted to mention it, just un case...

Bye...

    Dirk
-- 

*Dirk Heinrichs*, Senior Systems Engineer, Engineering Solutions
*Recommind GmbH*, Von-Liebig-Straße 1, 53359 Rheinbach
*Tel*: +49 2226 1596666 (Ansage) 1149
*Email*: dhs@recommind.com <mailto:dhs@recommind.com>
*Skype*: dirk.heinrichs.recommind
www.recommind.com <http://www.recommind.com>

[-- Attachment #2.1: Type: text/html, Size: 1238 bytes --]

[-- Attachment #2.2: Logo.gif --]
[-- Type: image/gif, Size: 1537 bytes --]

Re: cat as a builtin command

[Mikael Magnusson]

On 29 August 2014 15:03, Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Fri, 29 Aug 2014 09:40:29 +0800
> Izumi Natsuka <izumi.natsuka@hotmail.com> wrote:
>> Hello, I'm going to write a shell function that provides a basic
>> functionality(print the content of a file or stdin) of cat[0], in
>> order to avoid forking too many process when I call it in a
>> loop[1]. And I have put the following in my zshrc:
>
> If it's just printing the contents of a file, investigate zsh/mapfile.
> This turns off multibyte mode to make it easier to do binary files ---
> there should be no disadvantage here as you just want a byte stream.
>
>
> mycat() {
>   emulate -L zsh
>   unsetopt multibyte
>   zmodload zsh/mapfile
>   print -r $mapfile[$1]
> }
>
>
> This only works for files in the file system, not file descriptors,
> pipes etc.  For those you can often optimise out the cat entirely.
>
> pws

mycat() { for i { print -r - "$(< $i)" } }
(also only works for files)

-- 
Mikael Magnusson

Re: cat as a builtin command

[Han Pingtian]

On Fri, Aug 29, 2014 at 09:40:29AM +0800, Izumi Natsuka wrote:
> Hello, I'm going to write a shell function that provides a basic functionality(print the content of a file or stdin) of cat[0], in order to avoid forking too many process when I call it in a loop[1]. And I have put the following in my zshrc:
> 
> cat() {
> 	if [[ $# -le 1 ]] && [[ ${1:0:1} != '-' || ${1} == '-' ]];then
> 		local file
> 		exec {file}<${1:-0}

If there is no argument to cat, this line would be

    exec {file}<0

It should be 

    exec {file}<&0

right? But I get error mesage for it:

    localhost% (){
    function> local fd
    function> exec {fd}<&0
    function> read -E -u $fd
    function> exec {fd}<&-
    function> }
    (anon):2: 0: bad file descriptor
    (anon):read:3: argument expected: -u
    (anon):4: failed to close file descriptor 0: bad file descriptor

0 isn't a good descriptor here?

Re: cat as a builtin command

[Bart Schaefer]

On Sep 1,  3:53pm, Han Pingtian wrote:
}
} It should be 
} 
}     exec {file}<&0
} 
} right? But I get error mesage for it:
} 
}     localhost% (){
}     function> local fd
}     function> exec {fd}<&0
}     function> read -E -u $fd
}     function> exec {fd}<&-
}     function> }
}     (anon):2: 0: bad file descriptor
}     (anon):read:3: argument expected: -u
}     (anon):4: failed to close file descriptor 0: bad file descriptor

I'm not able to reproduce this.  Is this in a newly started shell?

    torch% (){           
    local fd  
    function> exec {fd}<&0
    function> read -E -u $fd
    function> exec {fd}<&-
    function> }
    hello
    hello
    torch% 

I *suspect* that what happened is that while you were experimenting, some
previous "exec <&-" has already closed descriptor 0.  Closing stdin is
not fatal to an interactive zsh, it maintains its own descriptor for ZLE
to access /dev/tty.

E.g., if I explicitly do:

torch% exec 0<&-   

Then up-history a couple of times and:

torch% (){         
local fd
exec {fd}<&0
read -E -u $fd
exec {fd}<&-
}
(anon):2: 0: bad file descriptor
(anon):read:3: argument expected: -u
(anon):4: failed to close file descriptor 0: bad file descriptor
torch% 

What's puzzling to me is why line 4 says "0: bad file descriptor" rather
than this:

torch% exec {fd}<&-
zsh: parameter fd does not contain a file descriptor

It appears that only an UNSET parameter name triggers the "does not
contain" error; a set-but-empty parameter is treated as 0 and closes
standard input, which is likely how you got into this situation.

Re: cat as a builtin command

[Han Pingtian]

On Mon, Sep 01, 2014 at 11:39:40AM -0700, Bart Schaefer wrote:
> On Sep 1,  3:53pm, Han Pingtian wrote:
> }
> } It should be 
> } 
> }     exec {file}<&0
> } 
> } right? But I get error mesage for it:
> } 
> }     localhost% (){
> }     function> local fd
> }     function> exec {fd}<&0
> }     function> read -E -u $fd
> }     function> exec {fd}<&-
> }     function> }
> }     (anon):2: 0: bad file descriptor
> }     (anon):read:3: argument expected: -u
> }     (anon):4: failed to close file descriptor 0: bad file descriptor
> 
> I'm not able to reproduce this.  Is this in a newly started shell?
> 
>     torch% (){           
>     local fd  
>     function> exec {fd}<&0
>     function> read -E -u $fd
>     function> exec {fd}<&-
>     function> }
>     hello
>     hello
>     torch% 
> 
> I *suspect* that what happened is that while you were experimenting, some
> previous "exec <&-" has already closed descriptor 0.  Closing stdin is
> not fatal to an interactive zsh, it maintains its own descriptor for ZLE
> to access /dev/tty.
> 
> E.g., if I explicitly do:
> 
> torch% exec 0<&-   
> 
> Then up-history a couple of times and:
> 
> torch% (){         
> local fd
> exec {fd}<&0
> read -E -u $fd
> exec {fd}<&-
> }
> (anon):2: 0: bad file descriptor
> (anon):read:3: argument expected: -u
> (anon):4: failed to close file descriptor 0: bad file descriptor
> torch% 
Thanks, I cannot reproduce it after a reboot now. I think you are right,
I must closed 0 before the test. Thanks so much.