rtems-bsp-builder Job Load Calculation

[Joel Sherrill]

On Fri, Apr 3, 2020 at 5:26 PM Chris Johns <chrisj at rtems.org> wrote:

> On 2020-04-04 07:57, Joel Sherrill wrote:
> > How does rtems-bsp-builder calculate the amount of parallelism to use?
>
> The option is documented here ...
>
>
> https://docs.rtems.org/branches/master/user/tools/bsp-builder.html#cmdoption-jobs
>
> and is ...
>
>   --jobs
>
>   The jobs options where the format is build-jobs/make-jobs. The default
>   is 1/num-cores where num-cores is the operating system reported number
>   of cores.
>

What's the definition of build job vs make job?

Is make jobs equivalent to N in make -jN?





>
> > I ask because I have realized that I selected jobs=6/12 on a very beefy
> > machine with 12 cores and 32 GB RAM. The main test machine I use is 8
> > cores/32 GB RAM. But the slowest ones are old with 4 cores/4 GB RAM.
> > They are hitting the swap space. All have SSDs.
> >
> > Should I drop the --jobs parameter? Will the script select a decent load
> > automatically? The machines are essentially dedicated to testing.
>
> You will need a --jobs option or the machine will not be fully loaded.
> You need to find a balance between the number of parallel BSPs that are
> built at once and the number of parallel build jobs each BSP makes runs.
>
> The critical factor is the amount of bootstrap and configure time. This
> is a single core task and if you use the default a lot of the build time
> the machine is basically idle.
>

But if you go too high, the number of compiles will swamp a machine
without a huge amount of memory. So the 4 core/4 GB machines
probably would be something like --jobs=2/4. if that's two bsp builds
in parallel with a -j4.

That generally gives you a 2x factor on processes which is close
to the old school rule of the -jN number should be between 1.5 and
2 times the number of cores as long as the RAM and disk subsystem
could handle it. That tends to overlap IO and computation.

Maybe the default should be (roundup(cores)/2) instead of 1.

Temporarily I may have to add a jobs option to my script and hand-tune
it to the machines I am testing on.


> There is no way I know of to automatically find this balance for all the
> different types of host machines.
>
> Chris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20200403/e4933a5a/attachment.html>