rtems-bsp-builder Job Load Calculation

[Joel Sherrill]

On Fri, Apr 3, 2020 at 6:23 PM Joel Sherrill <joel at rtems.org> wrote:

>
>
> On Fri, Apr 3, 2020 at 5:26 PM Chris Johns <chrisj at rtems.org> wrote:
>
>> On 2020-04-04 07:57, Joel Sherrill wrote:
>> > How does rtems-bsp-builder calculate the amount of parallelism to use?
>>
>> The option is documented here ...
>>
>>
>> https://docs.rtems.org/branches/master/user/tools/bsp-builder.html#cmdoption-jobs
>>
>> and is ...
>>
>>   --jobs
>>
>>   The jobs options where the format is build-jobs/make-jobs. The default
>>   is 1/num-cores where num-cores is the operating system reported number
>>   of cores.
>>
>
> What's the definition of build job vs make job?
>
> Is make jobs equivalent to N in make -jN?
>
>
>
>
>
>>
>> > I ask because I have realized that I selected jobs=6/12 on a very beefy
>> > machine with 12 cores and 32 GB RAM. The main test machine I use is 8
>> > cores/32 GB RAM. But the slowest ones are old with 4 cores/4 GB RAM.
>> > They are hitting the swap space. All have SSDs.
>> >
>> > Should I drop the --jobs parameter? Will the script select a decent
>> load
>> > automatically? The machines are essentially dedicated to testing.
>>
>> You will need a --jobs option or the machine will not be fully loaded.
>> You need to find a balance between the number of parallel BSPs that are
>> built at once and the number of parallel build jobs each BSP makes runs.
>>
>> The critical factor is the amount of bootstrap and configure time. This
>> is a single core task and if you use the default a lot of the build time
>> the machine is basically idle.
>>
>
> But if you go too high, the number of compiles will swamp a machine
> without a huge amount of memory. So the 4 core/4 GB machines
> probably would be something like --jobs=2/4. if that's two bsp builds
> in parallel with a -j4.
>
> That generally gives you a 2x factor on processes which is close
> to the old school rule of the -jN number should be between 1.5 and
> 2 times the number of cores as long as the RAM and disk subsystem
> could handle it. That tends to overlap IO and computation.
>
> Maybe the default should be (roundup(cores)/2) instead of 1.
>
> Temporarily I may have to add a jobs option to my script and hand-tune
> it to the machines I am testing on.
>
>
>> There is no way I know of to automatically find this balance for all the
>> different types of host machines.
>>
>
Thinking a bit more, I wonder if there could be some named
build/job ratios. 1/cores, (cores/2)/cores, etc.

>
>> Chris
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/devel/attachments/20200403/3a07fe6e/attachment.html>