source-builder randomly hangs

[Anders Montonen]

Hi,

> On 5 May 2020, at 10:54, Chris Johns <chrisj at rtems.org> wrote:
> 
> On 5/5/20 5:26 pm, Anders Montonen wrote:
>>> On 4 May 2020, at 4:05, Chris Johns <chrisj at rtems.org> wrote:
>>> Interesting and your analysis sounds right. I am not sure how this could be handled. A test case with execute.py would be a good start, see the end of the file.
>> We could try passing “universal_newlines=True” to the subprocess.Popen() constructor. This will open stdin/stdout/stderr in text mode, using the current locale setting. Python 3.6+ allows specifying the encoding in the constructor, if that makes a difference. Another option would be to manually wrap the output streams in an io.TextIOWrapper, which is essentially what Popen() does when given any of these arguments.
> We need to support python 2.7 but I like what you are suggesting. Any chance of a patch for execute.py? Anything that helps in this area is welcome. It is a hot spot for performance as well.

I can try. In the meanwhile, here’s a reproducer which fails with Python 3:

diff --git a/source-builder/sb/execute.py b/source-builder/sb/execute.py
index 06f9b7d..a719da8 100755
--- a/source-builder/sb/execute.py
+++ b/source-builder/sb/execute.py
@@ -577,6 +577,7 @@ if __name__ == "__main__":
 
     cmd_shell_test = 'if "%OS%" == "Windows_NT" (echo It is WinNT) else echo Is is not WinNT'
     sh_shell_test = 'x="me"; if [ $x = "me" ]; then echo "It was me"; else "It was him"; fi'
+    long_output_test = 'echo ' + 4095*'a' + u'\U00010000'
 
     commands = {}
     commands['windows'] = {}
@@ -587,7 +588,7 @@ if __name__ == "__main__":
     commands['windows']['csubsts'] = [('netstat %0', ['-a']),
                                       ('netstat %0 %1', ['-a', '-n'])]
     commands['windows']['pipe'] = ('ftp', None, 'help\nquit')
-    commands['unix']['shell'] = ['pwd', 'ls -las', './xyz', sh_shell_test]
+    commands['unix']['shell'] = ['pwd', 'ls -las', './xyz', sh_shell_test, long_output_test]
     commands['unix']['spawn'] = ['ls', 'execute.pyc', ['ls', '-i']]
     commands['unix']['cmd'] = [('date'), ('date', '-R'), ('date', ['-u', '+%d %D']),
                                ('date', '-u "+%d %D %S"’)]

Regards,
Anders