I wrote a bash script that executes a Java jar for a n number of times.
In practice, I defined e foo() function, containing the invocation of the jar, and then I run this script:
for RUN in $(seq 1 $RUNS)
do
foo &
done
Now, I would like to not execute runs times in parallel the jar.
There is a wait to limit the number of parallel execution (like a wait every 10 process for instance)?
bash version 4.4 introduced a useful new idiom called parameter transformation that can help you out in this case. In the code snippet below, note the use of ${num_jobs@P}. The @P is a type of parameter transformation that causes the variable to be expanded as if it were a bash prompt string. See man bash for other parameter transformation options.
#!/bin/bash
num_procs=$1
num_iters=$2
num_jobs="\j" # The prompt escape for number of jobs currently running
for ((i=0; i<num_iters; i++)); do
while (( ${num_jobs@P} >= num_procs )); do
wait -n
done
foo &
done
Credits to chepner at https://stackoverflow.com/a/38775799/6631810.
Following Kusalananda's comment, if needed, in order to make this group of processes independent of any other background jobs that would affect counting, you can wrap them with their own shell. For this a few changes are needed.
#!/bin/bash
# start a wrapper shell for the group of jobs
cat<<EOS | bash &
num_procs="$1"
num_iters="$2"
for ((i=0; i<num_iters; i++)); do
# escape what's not supposed to be expanded
# at the time of here-doc redirection
while (( \${num_jobs@P} >= num_procs )); do
wait -n
done
foo &
done
EOS
# now you can do other things
for RUN in $(seq 1 $RUNS); do
foo &
if (( (RUN % 10) == 0 )); then
wait
fi
done
or, with an alternative loop construct (IMHO nicer looking):
for (( r = 1; r <= RUNS; ++i )); do
foo &
if (( (r % 10) == 0 )); then
wait
fi
done
You may additionally want a lone wait after the loop if $RUNS is not a multiple of 10.
Instead of having RUNS as the total number of runs to do, one could also imagine having n batches of 10 jobs:
for (( i = 0; i < n; ++i )); do
printf 'starting batch %d...\n' "$i"
for (( j = 0; j < 10; ++j )); do
foo &
done
echo 'waiting...'
wait
done
Alternative solution using xargs and no explicit wait:
seq 1 "$RUNS" | xargs -n 1 -P 10 foo
This would however give the foo process a command line argument (one of the integers produced by seq) which may not be wanted. This gets rid of that issue:
seq 1 "$RUNS" | xargs -n 1 -P 10 sh -c 'foo'
-I parameter to it. Check seq 1 10 | xargs -P 10 -I# bash -c 'echo "$0"; echo "$@"'. With xargs -n 1 -P 10 sh -c 'foo' it's still passed as $0.
sh rather than to foo.
-I helps avoid starting the wrapper shell process, right?
-I still starts a new shell.
xargs -P10 -I# foo of course. Where's the shell here?
GNU Parallel is made for exactly this:
seq 1 $RUNS | parallel -j 10 -N0 foo
Default is to run one job per CPU core:
seq 1 $RUNS | parallel -N0 foo
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
