backing up with zsh

backing up with zsh

[Sven Guckes]

I am trying to use the zsh to back my webpages.
I just want to backup text files, ie no pictures.

They webpages all reside in "~/.P".  The pictures are within ~/.P/pics/ -
excluding this one dir is not a problem, but the problem is that some of
the pictures are scattered across the subdir of ~/.P still.
How do I exclude these pictures?

Possible filename extensions for pictures are:  bmp gif jpg jpeg
But how do I specify these as case insensitive?
Do I have to list all possible combinations of upper- and lowercase spelling?

Should I use zsh's globbing at all?
Or should I exclude filename with gtar's "-X File" feature?

	gtar cvvzf /tmp/guckes.web.980417.tar.gz ~/.P -X files.exclude

I may also need a restriction for file size (for all files), say 100K at most.
Is that possible?

Any hints?

Sven

Re: backing up with zsh

[Rob Windsor]

Verily did Sven Guckes write:

> I am trying to use the zsh to back my webpages.
> I just want to backup text files, ie no pictures.

> They webpages all reside in "~/.P".  The pictures are within ~/.P/pics/ -
> excluding this one dir is not a problem, but the problem is that some of
> the pictures are scattered across the subdir of ~/.P still.
> How do I exclude these pictures?

> Possible filename extensions for pictures are:  bmp gif jpg jpeg
> But how do I specify these as case insensitive?
> Do I have to list all possible combinations of upper- and lowercase spelling?

possibly.

A good start would be:

cd ~/.P ; find . -type f \! \( -name '*.gif' -a -name '*.jpg' -a -name '*.jpeg' -a -name '*.bmp' \) -print | xargs cvvzf /tmp/guckes.web.980417.tar.gz

> Should I use zsh's globbing at all?
> Or should I exclude filename with gtar's "-X File" feature?

> 	gtar cvvzf /tmp/guckes.web.980417.tar.gz ~/.P -X files.exclude

> I may also need a restriction for file size (for all files), say 100K at
> most.

> Is that possible?

Sure, check the manpage for find(1), look for "-size".

-- Rob
----------------------------------------
Internet: windsor@warthog.com
Life: Rob@Carrollton.Texas.USA.Earth

The weather is here, wish you were beautiful.

Re: backing up with zsh

[TGAPE!]

Rob Windsor wrote:
> Verily did Sven Guckes write:
> 
>> I am trying to use the zsh to back my webpages.
>> I just want to backup text files, ie no pictures.
> 
>> They webpages all reside in "~/.P".  The pictures are within ~/.P/pics/ -
>> excluding this one dir is not a problem, but the problem is that some of
>> the pictures are scattered across the subdir of ~/.P still.
>> How do I exclude these pictures?
> 
>> Possible filename extensions for pictures are:  bmp gif jpg jpeg
>> But how do I specify these as case insensitive?
>> Do I have to list all possible combinations of upper- and lowercase spelling?
<snip&splice>
>> I may also need a restriction for file size (for all files), say 100K at
>> most.
> 
> possibly.
> 
> A good start would be:
> 
> cd ~/.P ; find . -type f \! \( -name '*.gif' -a -name '*.jpg' -a -name '*.jpeg' -a -name '*.bmp' \) -print | xargs cvvzf /tmp/guckes.web.980417.tar.gz

A better start would be:

gtar czf /tmp/guckes.web.980417.tgz `cd ~/.P; find *(D) \! -type d -size 101k \
 \! \( -iname '*.gif' -o -iname '*.jpg' -o -iname '*.jpeg' -o \
 -iname '*.bmp' \) -print`

Also, this is larger (though more efficient, I hate to admit) than the
zsh

gtar czf /tmp/guckes.web.19980418.tgz \
**/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]|[jJ][pP][eE][gG])(DL-102401)

which is already fairly insane.  (Of course, I use find jobs more
complicated than that every few months or so.  However, I'm trying to
get some stars for my blue pointy hat.)

Hmm...
cd ~/.P;for x in **/*(DL-102401^/)
do
	if [ $(file $x |grep -c picture) -eq 0 ]; then
		tar cf - $x
	fi
done | gzip -9 > /tmp/guckes.web.`date +%Y%m%d`.tgz

would be much more general; it's more keystrokes, though, and it'll run
slower than a siamese cat pulling a 1-ton stone block up a pyramid.
(Well, OK, maybe not *quite* that slow, but pretty damn slow.)

zsh is fairly resistant to the 'line too long' problem, it takes a
fairly large expansion to trigger it, and some cases appears to be
completely licked.

If you really need multiple v options to gtar, I'm not the guy you need
to talk to, I don't have a couch.

Uh,
~/.P;for x in **/*(D)
do
	if [ $x != ${x#pics} && $(file $x | grep -c picture) -gt 0 ]; then
		n=pics/${x##*/}
		ln $x $n
		for y in **/*(D.)
		do
			sed "s@$x@$n@g" $y > /tmp/nopics.$$
			mv /tmp/nopics.$$ $y
		done
		rm $x
	fi
done

should go a long way to moving all your pictures and fixing your links.
Other people's links, however, are another matter.  I would suggest
backing up your webpage (probably including pictures this time) before
hazarding it.  (If you're curious about the ln followed by rm, I work in
a 24x7x365 environment, where second outages can be costly.)

Ed

Re: backing up with zsh

[Bart Schaefer]

On Apr 18,  9:36am, TGAPE! wrote:
} Subject: Re: backing up with zsh
}
} ~/.P;for x in **/*(D)
} do
} 	if [ $x != ${x#pics} && $(file $x | grep -c picture) -gt 0 ]; then
} 		n=pics/${x##*/}
} 		ln $x $n
} 		for y in **/*(D.)
} 		do
} 			sed "s@$x@$n@g" $y > /tmp/nopics.$$
} 			mv /tmp/nopics.$$ $y
} 		done
} 		rm $x
} 	fi
} done
} 
} should go a long way to moving all your pictures and fixing your links.

Various remarks:

Using ~/.P; like that only works if autocd is set.

What's the reason for including directories in the glob for the outer loop?

It's probably a -lot- faster to run "file" on multiple filenames, rather
than on every name individually.

Most implementations of /etc/magic (the "file" database) use "image" rather
than "picture" for GIF, JPEG, etc. files.

${x:t} is equivalent to ${x##*/}.

The second parameter to "ln" can be the name of a directory, so it can be
run once on all the picture files.

The inner loop runs sed on both the html files and the picture files, which
is probably going to corrupt the pictures.

You're running sed and then mv once per picture file per every other file.
Once per html document should be enough.

/tmp is probably in another filesystem, which means the mv is no better
than a cp.  Why put the temp file there?

So that leaves us with:

    function shufflepics() {
	setopt localoptions extendedglob
	local pics docs sedit
	cd ~/.P
	file **/*~pics/*(D.) | while read name type
	do
	    name=${name%:*}
	    case ${(L)type} in
	    (*image*|*picture*)
		sedit=($sedit -e "s@${name}@pics/${name:t}@g")
		pics=($pics $name);;
	    (*html*)
		docs=($docs $name);;
	    esac
	done
	ln $pics pics
	for f in $docs
	do
	    sed $sedit $f > $$-${f:t} && mv $$-${f:t} $f
	done
	rm -f $pics $$-*
    }

How's that?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: backing up with zsh

[Sven Guckes]

Hi!

Thanks for feedback, Mark!  :-)

Quoting Mark Borges (mdb@cdc.noaa.gov):
>  $ echo **/*~*.[Gg][Ii][Ff]

ok, but - this just excludes the GIFs.
Now, how do I combine this with the BMPs and JPGs, too?

Can I use '|' for "or"?  Example:

   $ echo **/*~*.[Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]

I'd rather be using something like this:

   $ echo **/*~(*.[bmp|gif|jpeg|jpg]/i)

.. where "/i" means "case insensitive".
Is there a way with zsh globbing?

> If you need to do more, it might be worth
> piping the output of find(1) into tar instead.

I didn't get this to work, either.
I also seem to get into trouble with
different kinds of regular expressions.
*sigh*

Sven  [yes, I should have left three hours ago...]

Re: backing up with zsh

[Sven Guckes]

Quoting Mark Borges (mdb@cdc.noaa.gov):
> >    $ echo **/*~*.[Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]
> Close -- you forgot the parens:
>     $ echo **/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG])
> I think you'll also have to put `jpeg' and it's case variant as well.

You mean
      $ echo **/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]|[jJ][pP][eE][gG])

This is getting "out of hand".  :-(

> There may be an equivalent for `jpe?g', but I don't see it offhand.

Anyone?

> SG> I'd rather be using something like this:
> SG>    $ echo **/*~(*.[bmp|gif|jpeg|jpg]/i)
> SG> .. where "/i" means "case insensitive".
> Sure. perl -e ''. ;-).

So - perl has better regexpr than the zsh here?

Sven

Perl and ZSH (WAS: Re: backing up with zsh)

[tep]

Thus Spake Sven Guckes:
> Quoting Mark Borges (mdb@cdc.noaa.gov):
> > >    $ echo **/*~*.[Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]
> > Close -- you forgot the parens:
> >     $ echo **/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG])
> > I think you'll also have to put `jpeg' and it's case variant as well.
> 
> You mean
>       $ echo **/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]|[jJ][pP][eE][gG])
> 
> This is getting "out of hand".  :-(
> 
> > There may be an equivalent for `jpe?g', but I don't see it offhand.
> 
> Anyone?
> 
> > SG> I'd rather be using something like this:
> > SG>    $ echo **/*~(*.[bmp|gif|jpeg|jpg]/i)
> > SG> .. where "/i" means "case insensitive".
> > Sure. perl -e ''. ;-).
> 
> So - perl has better regexpr than the zsh here?
> 
> Sven
> 

I've actually been tinkering with the idea of embedding a Perl
interpreter inside ZSH but I got stumped when it came time to
think about sharing data (i.e. variables and such).

Anyone care to assist in this endeavor??  :-)

Tim.

-- 
 ____________________________________________   _                       _ 
                           Timothy E. Peoples  |_| C o l l e c t i v e |_|
                            Senior Consultant   |_    technologies     _| 
                             tep@colltech.com     []                 []   
                                                    a pencom company

Re: Perl and ZSH (WAS: Re: backing up with zsh)

[Bart Schaefer]

On Apr 17,  7:44pm, tep@colltech.com wrote:
} Subject: Perl and ZSH (WAS: Re: backing up with zsh)
}
} I've actually been tinkering with the idea of embedding a Perl
} interpreter inside ZSH

I can't convince myself that this is a good idea.  You'd have to change
zexecve() in order to get perl programs to run in the internal interpreter
(i.e., you couldn't let the kernel interpret #!/usr/bin/perl), which would
be a performance hit on every binary exec.  Also, there's a fundamental
difference between shell scripts and perl programs that would make it very
impractical to have the perl interpreter state hang around for the entire
time the shell is running; but if you shut down the interpeter after every
call, you haven't gained much by making part of the shell.

That said, however:

} but I got stumped when it came time to
} think about sharing data (i.e. variables and such).

Make it a loadable module using the new facilities in 3.1.x.  Forget about
sharing anyting other than environment variables, which will "just work";
it's easy enough to "export" variables from zsh and stuff things into the
%ENV hash in perl, if values really need to be passed back and forth.  (I
suppose you might need to do a little tweaking, in case perl ignores the
actual environ array after first setting up %ENV.)

Make the interface to perl look like a builtin shell function, so that the
existing zsh code handles argv and i/o redirection.

} Anyone care to assist in this endeavor??  :-)

Not more than I just did.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Perl and ZSH

[Jonas Bofjall]

On Fri, 17 Apr 1998 tep@colltech.com wrote:

> I've actually been tinkering with the idea of embedding a Perl
> interpreter inside ZSH but I got stumped when it came time to

zsh is good because it keeps the executables size within bounds even
while having all these features. But Perl is *complex*... Wouldn't it
be time to modularize zsh some? And I don't mean autoloading of shell
script functions but of compiled code.

  // Jonas <job@abc.se> [2:201/262.37]

Re: Perl and ZSH

[TGAPE!]

Jonas Bofjall wrote:
> On Fri, 17 Apr 1998 tep@colltech.com wrote:
> 
>> I've actually been tinkering with the idea of embedding a Perl
>> interpreter inside ZSH but I got stumped when it came time to
> 
> zsh is good because it keeps the executables size within bounds even
> while having all these features. But Perl is *complex*... Wouldn't it
> be time to modularize zsh some? And I don't mean autoloading of shell
> script functions but of compiled code.

First, I seem to recall hearing something about that already.  Second,
no.  Do not encourage them.  Perl is embeded *enough*, simply by it
being a unix shell.  If I need the power of perl, I'd rather write a
perl script.  I realize a powerful shell can be nice, but there is
something called sanity which must be kept track of.

I'm actually of the opinion that the zshexpn is possibly getting a
little too complete - benchmarking shows that find is better suited for
most complex tasks.  That having been said, I do find virtually every
feature of zshexpn useful, so long as you use them within reason.
Remember that find still exists, and is better for the major endevor.

Perl is not only complex, but it's a fairly efficient piece of code.  It
could be improved significantly, but I don't believe the people who are
talking about putting it in zsh will necessarily have the time or
patience to even match its efficiency.

All this having been said, I wouldn't complain too loudly over an
optional perl expression parser, though I'd like it to be a compile time
option.

>On 17 Apr 1998, Karsten Thygesen wrote:
>
>> another project. We have a lot of resources available to projects like
>> Infoseek searchengine, newsgroups (if that is prefered over
>> mailinglists) and even CVS servers for distributed development.
>
>btw is there a cvs repository for zsh?

Not that I know of[1]; what's wrong with RCS?  You have it on your
system if you have CVS.  Since CVS uses RCS to do its work, it should be
trivial to convert.  Unfortunately, I've just shared pretty much my sum
total knowledge of CVS...

Ed

[1] Though Bart Schaefer might know better.

Re: Perl and ZSH

[Bart Schaefer]

On Apr 18,  9:53am, TGAPE! wrote:
} Subject: Re: Perl and ZSH
}
} >On 17 Apr 1998, Karsten Thygesen wrote:
} >
} >btw is there a cvs repository for zsh?
} 
} Not that I know of[1]; what's wrong with RCS?  You have it on your
} system if you have CVS.  Since CVS uses RCS to do its work, it should be
} trivial to convert.
} 
} [1] Though Bart Schaefer might know better.

I don't know of any cvs repository for zsh either (well, except my own,
which is only 3.0.x and littered with local experiments and selectively-
applied patches from zsh-workers).

You can make RCS'd sources accessible through CVS just by dropping the ,v
files into a directory tree under a new or existing CVS repository root;
but manipulating any RCS branch revisions probably won't work becaus CVS
uses a very particular branch numbering scheme.  To do it "right" you'd
have to check out each RCS revision and then import it into CVS, to make
a CVS revision history that matches the original RCS one.

You'd be better off to simply select a stable starting point, import that,
and begin to use CVS exclusively from there.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Zsh and CVS

[Scott RoLanD]

On Sat, Apr 18, 1998 at 11:21:51AM -0700, Bart Schaefer wrote: 
> On Apr 18,  9:53am, TGAPE! wrote:
> } Subject: Re: Perl and ZSH
> }
> } >On 17 Apr 1998, Karsten Thygesen wrote:
> } >
> } >btw is there a cvs repository for zsh?
> } 
> } Not that I know of[1]; what's wrong with RCS?  You have it on your
> } system if you have CVS.  Since CVS uses RCS to do its work, it should be
> } trivial to convert.
> } 
> } [1] Though Bart Schaefer might know better.
> 
> I don't know of any cvs repository for zsh either (well, except my own,
> which is only 3.0.x and littered with local experiments and selectively-
> applied patches from zsh-workers).
> 
> You can make RCS'd sources accessible through CVS just by dropping the ,v
> files into a directory tree under a new or existing CVS repository root;
> but manipulating any RCS branch revisions probably won't work becaus CVS
> uses a very particular branch numbering scheme.  To do it "right" you'd
> have to check out each RCS revision and then import it into CVS, to make
> a CVS revision history that matches the original RCS one.
> 
> You'd be better off to simply select a stable starting point, import that,
> and begin to use CVS exclusively from there.
> 
As an aside I would encourage everyone to look into using Perforce
(www.perforce.com) for work on free projects (and commercial too; but
it is free for free projects). I find it much more usable the CVS.

RoLanD
--
Just a happy user.

Re: backing up with zsh

[Mirar]

Sven Guckes <guckes@math.fu-berlin.de> writes:

> Hi!
> 
> Thanks for feedback, Mark!  :-)
> 
> Quoting Mark Borges (mdb@cdc.noaa.gov):
> >  $ echo **/*~*.[Gg][Ii][Ff]

Well, ls -1 **/*.{[Gg][Ii][Ff],[Jj][Pp][gG]} lists all gif/jpg files
for me. My zsh seems to old to understand **/*~foo, so you'll have to
experiment for yourself. :)

Re: backing up with zsh

[Andrew Main]

Sven Guckes wrote:
>Can I use '|' for "or"?  Example:
>
>   $ echo **/*~*.[Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP][gG]

echo **/*~*.([Gg][Ii][Ff]|[bB][mM][pP]|[jJ][pP](|[Ee])[gG])

>I'd rather be using something like this:
>
>   $ echo **/*~(*.[bmp|gif|jpeg|jpg]/i)
>
>.. where "/i" means "case insensitive".
>Is there a way with zsh globbing?

No.  zsh has glob patterns, not regular expressions.

-zefram