waf default number of jobs

[Sebastian Huber]

On 03/10/2020 08:37, Chris Johns wrote:

> On 2/10/20 5:53 am, Joel Sherrill wrote:
>> On Thu, Oct 1, 2020 at 2:39 PM Sebastian Huber
>> <sebastian.huber at embedded-brains.de  <mailto:sebastian.huber at embedded-brains.de>>
>> wrote:
>>
>>      On 01/10/2020 20:09, Joel Sherrill wrote:
>>
>>      > What was the rationale behind the choice of the default for the number
>>      > of jobs for the waf build system?
>>      I don't know. It is the default behaviour of waf. I noticed also
>>      scalability problems but had no time to track them down. It could be
>>      also a limitation of the Python multiprocessing capabilities. I am also
>>      not a waf expert, I may have produced some bottlenecks.
>>
>>
>> I don't think you produced any bottlenecks. Here are a couple of fragments
>> from my build. I have attached the ini file so you can double check that it
>> does match. Please let me know if the ini matches the configure command
>> or not. If it doesn't, then I have a lot of bogus builds. :(
>>
>>   ../rtems/configure --target=arm-rtems6 --enable-rtemsbsp=gumstix
>> --prefix=/home/joel/rtems-cron-6/tools/6/bsp-install --disable-networking
>> --enable-posix --disable-smp --disable-multiprocessing --enable-rtems-debug
>> --enable-profiling --enable-tests --enable-cxx --enable-maintainer-mode
>> + make -j24
>>
>> real    1m53.064s
>> user    7m14.675s
>> sys     1m45.913s
>> .....
>>
>> + ./waf -j 24
>>
>> real    1m1.404s
>> user    0m9.601s
>> sys     0m1.547s
>>
>> Notice how the build time for waf is 50 seconds faster but that doesn't
>> match the user and system time. Perhaps the Python jobs model doesn't
>> keep them as children the same way so time doesn't get to count them.
>>
>> I honestly can't explain that.
>>
> Which host OS?
>
> I am running a waf build of 5 BSPs with tests on FreeBSD 12.1 and I ran `systat
> -vmstat 1` at the same time and it shows idle is around 0%, 25% is in sys and
> 75% is in user. The machine has with 8 cores. The powerpc/mvme5500 took 58
> seconds. Python is 3.7.6.
I noticed the scalability issues on systems with more than 64 processors 
of the GCC compile farm.