Prompt expansion, multi-job for

Prompt expansion, multi-job for

[Andre Pang]

Hiya guys,

    I've got two completely separate questions about zsh.

    1) Let's say I'm in the /usr/local/src path at the moment.  Normally, if
you use the %~ prompt expansion, it will expand to /usr/local/src - no
surprises there.  What I'd like to do is trim each path element to one
character unless it's the last path element, in which case it should be
displayed to a maximum of, say, 15 characters.  eg, /usr/local/src should be
displayed as "/u/l/src", /usr/local/src/linux would be displayed as
"/u/l/s/linux", and /usr/local/src/linux-2.2.14+reiserfs+raid+ide might be
displayed as "/u/l/s/linux-2.2.14...".  Is this possible with the current
prompt expansion codes, or will I have to write up a function to do it, if
it's possible to do at all?

    2) Is there any way to emulate make's -jn option in the for command? 
This would be *really* useful for SMP systems.  Currently, doing something
like "for i in ***/***.wav; do l3enc $i; done" doesn't take advantage of
multiple CPUs (on a Linux system, at least); hacking the for command to
accept a 'make -j'-like parameter (eg: for -j2 i in *; do l3enc $i; done)
would allow us lucky ones who have SMP systems to do many things similar to
the above without going mad and writing silly Makefiles simply so we can
utilise make -j.  If someone's written a function to do this already (Bart?
;) it'd be *very* cool.

    Thanks,


-- 
: Andre Pang <andrep@vjolnir.org> - Purruna Pty Ltd - ph# 0411.882299 :
:               #ozone - http://www.vjolnir.org/ozone/                :

Re: Prompt expansion, multi-job for

[Oliver Kiddle]

Andre Pang wrote:

>     2) Is there any way to emulate make's -jn option in the for command?
> This would be *really* useful for SMP systems.  Currently, doing something
> like "for i in ***/***.wav; do l3enc $i; done" doesn't take advantage of
> multiple CPUs (on a Linux system, at least); hacking the for command to
> accept a 'make -j'-like parameter (eg: for -j2 i in *; do l3enc $i; done)
> would allow us lucky ones who have SMP systems to do many things similar to
> the above without going mad and writing silly Makefiles simply so we can
> utilise make -j.  If someone's written a function to do this already (Bart?
> ;) it'd be *very* cool.

If I'm not mistaken, you should be able to do this by running l3enc as a
background process, i.e. for i in *.wav; do l3enc $i& done. I haven't
got an SMP system available to me to test but I'd assume that it would
allocate different l3enc processes to different processors. Putting the
process in the background means that zsh will get control back
straight-away and can run the next l3enc.

The trouble with this is that it will attempt to run all the l3encs
together. If you have say 4 processors, it may be most efficient to run
a maximum of 4 l3encs at once. This would be a bit messy to implement in
Zsh (compared to something like Ada) but can be done.

Oliver Kiddle

Re: Prompt expansion, multi-job for

[Thomas Köhler]

On Thu, Jan 06, 2000 at 01:04:55PM +0100,
Oliver Kiddle <opk@u.genie.co.uk> wrote:
> 
> Andre Pang wrote:
> 
> >     2) Is there any way to emulate make's -jn option in the for command?
> > This would be *really* useful for SMP systems.  Currently, doing something
> > like "for i in ***/***.wav; do l3enc $i; done" doesn't take advantage of
> > multiple CPUs (on a Linux system, at least); hacking the for command to
> > accept a 'make -j'-like parameter (eg: for -j2 i in *; do l3enc $i; done)
> > would allow us lucky ones who have SMP systems to do many things similar to
> > the above without going mad and writing silly Makefiles simply so we can
> > utilise make -j.  If someone's written a function to do this already (Bart?
> > ;) it'd be *very* cool.
> 
> If I'm not mistaken, you should be able to do this by running l3enc as a
> background process, i.e. for i in *.wav; do l3enc $i& done. I haven't
> got an SMP system available to me to test but I'd assume that it would
> allocate different l3enc processes to different processors. Putting the
> process in the background means that zsh will get control back
> straight-away and can run the next l3enc.

Well, bad idea if you have a few hundred wavs :)

> The trouble with this is that it will attempt to run all the l3encs
> together. If you have say 4 processors, it may be most efficient to run
> a maximum of 4 l3encs at once. This would be a bit messy to implement in
> Zsh (compared to something like Ada) but can be done.

Well - let's try something like this:

for i in **/*.wav ; do jspool l3enc "'$i'" ; done
jrun 4

Now, we need jspool and jrun. jspool is simple:
function jspool() {
   print $@ >> ~/.zjspool/jobs
}

Now, jrun:
function jrun() {
   print rm -f ~/.zjspool/jobs >> ~/.zjspool/jobs
   print 1 > ~/.zjspool/current
   i=0
   while [ $i -lt $1 ] ; do
      zjrun $i &!
      i=$[i+1]
   done
}

zjrun is a shell script:

#!/usr/bin/zsh
while : ; do
   sleep 1
   while [ -f ~/.zjspool/lock ] ; do sleep 5 ; done
   print $$ >> ~/.zjspool/lock
   PID=$(head -1 ~/.zjspool/lock)
   if [ $PID -ne $$ ] ; then continue ; fi
   LINE=$(cat ~/.zjspool/current)
   if [ ! -f ~/.zjspool/jobs ] ; then
      rm -f ~/.zjspool/lock
      exit
   fi
   JOB=$(head -$LINE ~/.zjspool/jobs | tail -1)
   print $[LINE + 1] > ~/.zjspool/current
   rm -f ~/.zjspool/lock
   print -n "$1: "
   eval $JOB
done

You get the idea...

Ah - you have 4 jobs running, but you want another important job to run
in between?
echo "XXX" > ~/.zjspool/lock
wait for the 4 running jobs to finish
run your important other stuff 
rm ~/.zjspool/lock

> Oliver Kiddle

CU,
Thomas

-- 
 Thomas Köhler Email:   jean-luc@picard.franken.de   | LCARS - Linux for
     <><        WWW:  http://home.pages.de/~jeanluc/ | Computers on All
                IRC:             jeanluc             | Real Starships
   PGP public key: http://www.mayn.de/users/jean-luc/PGP-Public.asc

Re: Prompt expansion, multi-job for

[Oliver Kiddle]

Thomas Köhler wrote:

> Well, bad idea if you have a few hundred wavs :)
> 
> > The trouble with this is that it will attempt to run all the l3encs
> > together. If you have say 4 processors, it may be most efficient to run
> > a maximum of 4 l3encs at once. This would be a bit messy to implement in
> > Zsh (compared to something like Ada) but can be done.
> 
> Well - let's try something like this:

That's certainly one way. I seem to remember seeing someone else's
solution using a co-routine at some point (Bart maybe?). Here is my
method which I think is quite simple:

trap '(( jobs=jobs-1 ))' CHLD

for i in 10 9 2 3 1 7 1 2 4; do
  while (( jobs >= 4 )); do
    :
  done
   
  echo Running sleep $i.
  (( jobs=jobs+1 ))
  sleep $i &
done
 
echo Wait for last job to finish
wait

For the original problem, you would want to replace sleep with l3enc and
the list of numbers with your glob for .wav files. I just thought that
this demonstrates it better. You might also want to use a 'sleep 1'
instead of ':' so that the while loop loops more slowly.

In theory you could have problems if the increment and decrement for
$jobs didn't run atomicly and ran at the same time but this is hardly a
safety-critical system and with jobs as long as l3enc, I'd doubt it
would happen. At worst $jobs would end up a little out so you'd have
more or less concurrent jobs.

Other jobs finishing might also mess this up but you can always avoid
that by running it all in a sub-shell.

Anyone else got any other solutions (maybe using co-routines, fifos or
something)?

Oliver

Re: Prompt expansion, multi-job for

[Oliver Kiddle]

Kevin Sullivan wrote:
> I use 8hz-mp3 instead of l3enc and I use 'wait'. It is
> granted simplistic. But then again took about 1 min to think up and 2
> min to write. I added a bunch of worthless printing but this has
> worked well for some 20Gig of mp3(s) and growing...

The trouble with wait is that it will wait for all the jobs to finish - there is no way to get it to wait for the first job to finish so you are setting 4 jobs off, waiting for them all to finish and doing the next 4 instead of doing the next each time a job finishes. Your solution is certainly nice and simple but it won't utilise multiple processors as well. If all the jobs are going to take roughly similar time then this makes less of a difference.

Oliver

Re: Prompt expansion, multi-job for

[Bart Schaefer]

On Jan 6,  3:21pm, Oliver Kiddle wrote:
} Subject: Re: Prompt expansion, multi-job for
}
} That's certainly one way. I seem to remember seeing someone else's
} solution using a co-routine at some point (Bart maybe?).

You must be thinking of this.  It's from a thread with the subject
"Running N jobs from M all the time".


function run_parallel {
    # Runs N copies of command at once until M copies have run.

    emulate -L zsh
    if (( $# < 3 )); then
	print "usage: run_parallel N M command [args ...]" >&2
	return 1
    fi

    integer N=$1 M=$2
    shift 2

    if ((M <= N)); then
	repeat $M do $==* & done
	wait
    else
	# Set up a loop to read and write one byte at a time.  This
	# semaphores between the parallel children and the parent zsh.  
	# It's important to pass exactly one byte here and not rely on
	# reading complete lines, lest a race condition develop.

	# Unfortunately, a race still can occur, it's just less likely.
	# Maybe SMP systems are better about multiple writers on one FD?

	coproc while ((--M > 0)) && read -u0k; do print -n .; done

	# Start the first N children

	repeat $N do
	    ((--M))
	    { $==* ; print -np . } &
	done

	# Wait for the semaphore from each child and start another as
	# soon as we get it, up to M children.

	while ((--M > 0)) && read -pk; do
	    { $==* ; print -np . } &
	done

	# Wait for the last N semaphores.

	repeat $N do read -pk; done
    fi

    return $?
}

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: Prompt expansion, multi-job for

[Andre Pang]

On Thu, Jan 06, 2000 at 12:03:09PM +0000, Oliver Kiddle wrote:

> Andre Pang wrote:
> 
> >     2) Is there any way to emulate make's -jn option in the for command?
> > This would be *really* useful for SMP systems.  Currently, doing something
> > like "for i in ***/***.wav; do l3enc $i; done" doesn't take advantage of
> > multiple CPUs (on a Linux system, at least); hacking the for command to
> > accept a 'make -j'-like parameter (eg: for -j2 i in *; do l3enc $i; done)
> > would allow us lucky ones who have SMP systems to do many things similar to
> > the above without going mad and writing silly Makefiles simply so we can
> > utilise make -j.  If someone's written a function to do this already (Bart?
> > ;) it'd be *very* cool.
> 
> If I'm not mistaken, you should be able to do this by running l3enc as a
> background process, i.e. for i in *.wav; do l3enc $i& done. I haven't
> got an SMP system available to me to test but I'd assume that it would
> allocate different l3enc processes to different processors. Putting the

    Yes, you're right there.  I've done this a few times when I've had < 15
jobs to run, but today, I did a 'for i in ***/***.wav; do shorten $i; done',
and I have easily >200 .wavs which I wanted to compress.  Running 200
background processes isn't my idea of fun :).

> a maximum of 4 l3encs at once. This would be a bit messy to implement in
> Zsh (compared to something like Ada) but can be done.

    Which brings me back to my original question - if someone has a function
to do this already, I'll be most happy.  I'm just not sure whether it's
possible to make a function which can emulate the behaviour of the for
command perfectly, because it's treated as a complex command and not as a
function.  I'm thinking that the only way to implement this gracefully would
be to either (a) change the syntax of the new function, thus having a for
command with a different syntax - annoying[1], or (b) modify the zsh source
to deal with it, which I would love to do, but I lack the skill for.

    Suggestions, guys?

    [1] Perhaps the closest you'd get is ``jfor -2 i in * do 'l3enc -q $i;
echo done $i' done'' -- note the lack of semicolons and the single quotes
around the commands to run.  Even then, the syntax is different from normal
for.  Syntax change == bad.


-- 
: Andre Pang <andrep@vjolnir.org> - Purruna Pty Ltd - ph# 0411.882299 :
:               #ozone - http://www.vjolnir.org/ozone/                :

RE: Prompt expansion, multi-job for

[Andrej Borsenkow]

>
>     1) Let's say I'm in the /usr/local/src path at the moment.  Normally, if
> you use the %~ prompt expansion, it will expand to /usr/local/src - no
> surprises there.  What I'd like to do is trim each path element to one
> character unless it's the last path element, in which case it should be
> displayed to a maximum of, say, 15 characters.  eg, /usr/local/src should be
> displayed as "/u/l/src", /usr/local/src/linux would be displayed as
> "/u/l/s/linux", and /usr/local/src/linux-2.2.14+reiserfs+raid+ide might be
> displayed as "/u/l/s/linux-2.2.14...".  Is this possible with the current
> prompt expansion codes, or will I have to write up a function to do it, if
> it's possible to do at all?
>

In  the current zsh

bor@itsrm2% echo ${PWD//(#m)[^\\/]##\\//$MATCH[1]/}
/t/s/zsh-3.1.6-dev-14

But you'll have to clamp the last component at 15 chars yourself anyway ... so
your are better off writing your own fuction for it.

/andrej

Re: Prompt expansion, multi-job for

[Andre Pang]

On Thu, Jan 06, 2000 at 03:44:26PM +0300, Andrej Borsenkow wrote:

> > surprises there.  What I'd like to do is trim each path element to one
> > character unless it's the last path element, in which case it should be
> > displayed to a maximum of, say, 15 characters.  eg, /usr/local/src should be
> > displayed as "/u/l/src", /usr/local/src/linux would be displayed as
> > "/u/l/s/linux", and /usr/local/src/linux-2.2.14+reiserfs+raid+ide might be
> > displayed as "/u/l/s/linux-2.2.14...".  Is this possible with the current
> 
> In  the current zsh
> 
> bor@itsrm2% echo ${PWD//(#m)[^\\/]##\\//$MATCH[1]/}
> /t/s/zsh-3.1.6-dev-14

    Thanks; I might wait for 3.1.7 to try it out.  It appears that 3.1.6
doesn't support it (at least my 3.1.6 returns 'zsh: bad pattern:
(#m)[^/]##/').

    While on the subject of prompt stuff, is there any reason why the
'PROMPT EXPANSION' section of the zsh manual is under zshmisc.1?  Would it
be more sensible to put it in zshexpn.1?


-- 
: Andre Pang <andrep@vjolnir.org> - Purruna Pty Ltd - ph# 0411.882299 :
:               #ozone - http://www.vjolnir.org/ozone/                :

Re: Prompt expansion, multi-job for

[Peter Stephenson]

Andre Pang wrote:
> Hiya guys,
> 
>     I've got two completely separate questions about zsh.
> 
>     1) Let's say I'm in the /usr/local/src path at the moment.  Normally, if
> you use the %~ prompt expansion, it will expand to /usr/local/src - no
> surprises there.  What I'd like to do is trim each path element to one
> character unless it's the last path element, in which case it should be
> displayed to a maximum of, say, 15 characters.  eg, /usr/local/src should be
> displayed as "/u/l/src", /usr/local/src/linux would be displayed as
> "/u/l/s/linux", and /usr/local/src/linux-2.2.14+reiserfs+raid+ide might be
> displayed as "/u/l/s/linux-2.2.14...".  Is this possible with the current
> prompt expansion codes, or will I have to write up a function to do it, if
> it's possible to do at all?

Without ~ backsubstitution (i.e. full paths with no insertion of ~'s), you
can do it with the latest code as follows --- it won't work even with
3.1.6, because of the use of backreferences, as you've already discovered,
so you'll need to get zsh-3.1.6-dev-14 from the development directory on
the standard archives (if they're up to date).

setopt extendedglob promptsubst
PS1='${${PWD:h}//(#b)(\\/[^\\/])[^\\/]#/$match[1]}${${PWD:t}:+/}%15>...>${PWD:t}'

I'll explain it if you like, but that would take a good deal longer than
inventing it.  I can't think of a way of getting the ~'s back just using
parameter code, sorry (anyone else?).  If you can live with just $HOME put
back, you can replace PWD both times with ${PWD/$HOME/~}.

-- 
Peter Stephenson <pws@pwstephenson.fsnet.co.uk>

Re: Prompt expansion, multi-job for

[Sven Wischnowsky]

Andre Pang wrote:

> Hiya guys,
> 
>     I've got two completely separate questions about zsh.
> 
>     1) Let's say I'm in the /usr/local/src path at the moment.  Normally, if
> you use the %~ prompt expansion, it will expand to /usr/local/src - no
> surprises there.  What I'd like to do is trim each path element to one
> character unless it's the last path element, in which case it should be
> displayed to a maximum of, say, 15 characters.  eg, /usr/local/src should be
> displayed as "/u/l/src", /usr/local/src/linux would be displayed as
> "/u/l/s/linux", and /usr/local/src/linux-2.2.14+reiserfs+raid+ide might be
> displayed as "/u/l/s/linux-2.2.14...".  Is this possible with the current
> prompt expansion codes, or will I have to write up a function to do it, if
> it's possible to do at all?

Not built-in. But you can always do any calculation you need in the
precmd function, stuff the result in the psvar array and use %v (or
any parameter if you use PROMPT_SUBST).


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de