[RTEMS Project] #4726: RSB decode exception stops build

[RTEMS trac]

#4726: RSB decode exception stops build
-------------------------+--------------------------
 Reporter:  Chris Johns  |       Owner:  Chris Johns
     Type:  defect       |      Status:  assigned
 Priority:  normal       |   Milestone:  6.1
Component:  admin        |     Version:  6
 Severity:  normal       |  Resolution:
 Keywords:               |  Blocked By:
 Blocking:               |
-------------------------+--------------------------

Comment (by Frank Kuehndel):

 I have seen similar errors in special build environments. These two are
 from March 2022, RSB for RTEMS 6 (note the slight difference in the error
 message in the last line):

 {{{
   GIT BUILD HEAD: 4cdec141b1320a1e5a04b898e13e04c43ec233c3 ubuntu-nios2
   DevTools: Build #124 ubuntu-nios2 (Mar 9, 2022, 12:38:59
   building: nios2-rtems6-gcc-0f001dd-newlib-85f2dca-x86_64-linux-gnu-1
   21:08:10  Exception in thread _stderr[]:
   21:08:10  Traceback (most recent call last):
   21:08:10    File "/usr/lib/python3.8/threading.py", line 932, in
 _bootstrap_inner
   21:08:10      self.run()
   21:08:10    File "/usr/lib/python3.8/threading.py", line 870, in run
   21:08:10      self._target(*self._args, **self._kwargs)
   21:08:10    File "/home/minna/src/rtems-source-builder/source-
 builder/sb/execute.py", line 204, in _readthread
   21:08:10      data = data.decode(sys.stdout.encoding)
   21:08:10  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in
 position 4095: unexpected end of data
 }}}

 {{{
   GIT_HEAD="6fe98f91d94bbf965bc0e78015585ff8823d17bd
   BUILD_ACTUAL_RSB_OPTIONS="--with_ada --with_cxx --with_objc --jobs=12"
   building: v850-rtems6-gcc-0f001dd-newlib-d88cbd0-x86_64-linux-gnu-1
   18:55:36  Exception in thread _stderr[]:
   18:55:36  Traceback (most recent call last):
   18:55:36    File "/usr/lib/python3.8/threading.py", line 932, in
 _bootstrap_inner
   18:55:36      self.run()
   18:55:36    File "/usr/lib/python3.8/threading.py", line 870, in run
   18:55:36      self._target(*self._args, **self._kwargs)
   18:55:36    File "/home/minna/src/rtems-source-builder/source-
 builder/sb/execute.py", line 204, in _readthread
   18:55:36      data = data.decode(sys.stdout.encoding)
   18:55:36  UnicodeDecodeError: 'utf-8' codec can't decode bytes in
 position 4094-4095: unexpected end of data
 }}}

 Setting the ''locale'' to any "only-one-byte-unicode" linke
 `LANG=en_US.iso885915` when invoking the source-builder avoids the error.

 There is most likely no error in the input data but the loop decoding it
 has probably a bug. Unicode code points encoded in UTF-8 can be up to 4
 bytes long. The input data is read by `source-builder/sb/execute.py` in
 blocks of 4096 bytes. I guess it cuts through a multi-byte code point with
 the first byte(s) at the end of the current block and the rest of the
 bytes in the beginning of the next block. Note that the error always
 points to the end of the block `position 4095` or `position 4094-4095`.
 Moreover, `byte 0xe2` indicates that this is not the last byte of the code
 point (`0xe2` is the first byte of a 3 byte code point?).

 I know in Linux there exist some GNU Unicode decoding function which keeps
 a state between consecutive calls. This function is intended for use in
 cases where not the complete text can be decoded at once. I guess there is
 such a function for Python too. An alternative may be to read and decode
 the input in one large piece.

--
Ticket URL: <http://devel.rtems.org/ticket/4726#comment:1>
RTEMS Project <http://www.rtems.org/>
RTEMS Project