source-builder randomly hangs

[Chris Johns]

On 6/5/20 2:24 am, Anders Montonen wrote:
> Hi,
> 
>> On 5 May 2020, at 10:54, Chris Johns <chrisj at rtems.org> wrote:
>>
>> On 5/5/20 5:26 pm, Anders Montonen wrote:
>>>> On 4 May 2020, at 4:05, Chris Johns <chrisj at rtems.org> wrote:
>>>> Interesting and your analysis sounds right. I am not sure how this could be handled. A test case with execute.py would be a good start, see the end of the file.
>>> We could try passing “universal_newlines=True” to the subprocess.Popen() constructor. This will open stdin/stdout/stderr in text mode, using the current locale setting. Python 3.6+ allows specifying the encoding in the constructor, if that makes a difference. Another option would be to manually wrap the output streams in an io.TextIOWrapper, which is essentially what Popen() does when given any of these arguments.
>> We need to support python 2.7 but I like what you are suggesting. Any chance of a patch for execute.py? Anything that helps in this area is welcome. It is a hot spot for performance as well.
> 
> I can try.

Thanks.

> In the meanwhile, here’s a reproducer which fails with Python 3:

Nice. We can merge this once you have something.

Chris

> 
> diff --git a/source-builder/sb/execute.py b/source-builder/sb/execute.py
> index 06f9b7d..a719da8 100755
> --- a/source-builder/sb/execute.py
> +++ b/source-builder/sb/execute.py
> @@ -577,6 +577,7 @@ if __name__ == "__main__":
>   
>       cmd_shell_test = 'if "%OS%" == "Windows_NT" (echo It is WinNT) else echo Is is not WinNT'
>       sh_shell_test = 'x="me"; if [ $x = "me" ]; then echo "It was me"; else "It was him"; fi'
> +    long_output_test = 'echo ' + 4095*'a' + u'\U00010000'
>   
>       commands = {}
>       commands['windows'] = {}
> @@ -587,7 +588,7 @@ if __name__ == "__main__":
>       commands['windows']['csubsts'] = [('netstat %0', ['-a']),
>                                         ('netstat %0 %1', ['-a', '-n'])]
>       commands['windows']['pipe'] = ('ftp', None, 'help\nquit')
> -    commands['unix']['shell'] = ['pwd', 'ls -las', './xyz', sh_shell_test]
> +    commands['unix']['shell'] = ['pwd', 'ls -las', './xyz', sh_shell_test, long_output_test]
>       commands['unix']['spawn'] = ['ls', 'execute.pyc', ['ls', '-i']]
>       commands['unix']['cmd'] = [('date'), ('date', '-R'), ('date', ['-u', '+%d %D']),
>                                  ('date', '-u "+%d %D %S"’)]
> 
> Regards,
> Anders
>