source-builder randomly hangs

[Chris Johns]

On 2/5/20 10:04 pm, Anders Montonen wrote:
> A couple of times I’ve had source-builder hang seemingly randomly, with the following Python backtrace:

Do you see this problem with the bootstrap script 'rtems-bootstrap' in 
the top of the rtems.git rep?

I think the RSB version could be retired now the rtems.git repo has a 
decent version, plud this is the last autotools release of RTEMS so the 
need to bootstrap will go and both will go.

However fixing execute.py would be nice ...

> Exception in thread _stderr[]:
> Traceback (most recent call last):
>    File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
>      self.run()
>    File "/usr/lib/python3.8/threading.py", line 870, in run
>      self._target(*self._args, **self._kwargs)
>    File "/home/anders/work/rtems/rsb/source-builder/sb/execute.py", line 204, in _readthread
>      data = data.decode(sys.stdout.encoding)
> UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data
> 
> My guess is that since the loop in execute._readthread() is reading in fixed 4096-byte chunks, occasionally the boundary will fall in the middle of an UTF-8 code point, causing the decode call to throw an error. Since source-builder stops reading the file handle at that point, it will then deadlock the child process. Unfortunately, I haven’t yet been able to reliably trigger the issue. Usually, re-running the same build command succeeds.

Interesting and your analysis sounds right. I am not sure how this could 
be handled. A test case with execute.py would be a good start, see the 
end of the file.

Chris