xlat/gen.sh: speedup xlat generation

This commit implements the following changes:
- increases the number of concurrent jobs to twice the CPU count;
- creates a circular buffer, so instead of running multiple jobs
  at once, the generator tries to keep about the same number of jobs
  being run concurrently;
- runs gen_git and gen_make concurrently in order to squeeze in
  one more bit of concurrency.

With my deeply scientific measurements, this approach achieves
up to 15% speedup with dash and about 30-40% with bash as /bin/sh
on a 4-core machine.

* xlat/gen.sh (main): Declare pids local variable, append pid of every
run job to it; increase the limit of jobs to ncpus * 2; when the limit
is reached, wait for the first pid in pids instead of resetting jobs
to zero and waiting for all the run jobs; put gen_git and gen_make
into background.
This commit is contained in:
Eugene Syromyatnikov 2017-09-16 02:30:40 +02:00 committed by Dmitry V. Levin
parent 4d8eedbd4f
commit 04a65a183b

View File

@ -290,6 +290,7 @@ main()
local name
local jobs=0
local ncpus="$(getconf _NPROCESSORS_ONLN)"
local pids=
[ "${ncpus}" -ge 1 ] ||
ncpus=1
@ -300,15 +301,20 @@ main()
name=${f##*/}
name=${name%.in}
gen_header "${f}" "${output}/${name}.h" "${name}" &
pids="$pids $!"
names="${names} ${name}"
: $(( jobs += 1 ))
if [ ${jobs} -ge ${ncpus} ]; then
jobs=0
wait
fi
if [ "${jobs}" -gt "$(( ncpus * 2 ))" ]; then
read wait_pid rest
pids="$rest"
wait -n 2>/dev/null || wait "$wait_pid"
: $(( jobs -= 1 ))
fi <<- EOF
$pids
EOF
done
gen_git "${output}/.gitignore" ${names}
gen_make "${output}/Makemodule.am" ${names}
gen_git "${output}/.gitignore" ${names} &
gen_make "${output}/Makemodule.am" ${names} &
wait
else
name=${input##*/}