[RTEMS Project] #4966: Source Builder doesn't recognize Linux distributions w/ Python >= 3.8
RTEMS trac
trac at rtems.org
Sat Nov 11 00:22:38 UTC 2023
#4966: Source Builder doesn't recognize Linux distributions w/ Python >= 3.8
----------------------------+--------------------
Reporter: Michael South | Owner: (none)
Type: defect | Status: new
Priority: normal | Milestone:
Component: tool/rsb | Version: 7
Severity: normal | Keywords:
Blocked By: | Blocking:
----------------------------+--------------------
If a Linux build host's installed Python >= 3.8 then sb-set-builder fails
to recognize the Linux distribution, and defaults the paths for OS
utilities such as grep and sed. This fails on Gentoo, no doubt others.
The problem is that source-builder/sb/linux.py tries to use
"platform.dist()"; dist was removed from the module in Python 3.8. That
fails, so linux.py falls back to reading /etc/issue, which fails miserably
on Gentoo and I'd guess most other distros. This causes the paths for
utilities to fall back to defaults which may not work (e.g., default
/usr/bin/bzip2 vs. Gentoo's /bin/bzip2).
The recommended replacement is the "distro" package. In was introduced in
Python 3.6, and was back-ported to 2.6 and 2.7.
== Recreate:
{{{
$ sb-set-builder --list-host
RTEMS Source Builder - Set Builder, 6 (633023de6517)
warning: exe: absolute exe found in path: (__bzip2) /usr/bin/bzip2
warning: exe: absolute exe found in path: (__chgrp) /usr/bin/chgrp
warning: exe: absolute exe found in path: (__chown) /usr/sbin/chown
warning: exe: absolute exe found in path: (__grep) /usr/bin/grep
warning: exe: absolute exe found in path: (__sed) /usr/bin/sed
Host operating system information:
Operating system: linux
Number of processors: 4
Build architecture: x86_64
Host triplet: x86_64-linux-gnu
$ ls /usr/bin/bzip2
ls: cannot access '/usr/bin/bzip2': No such file or directory
$ which bzip2
/bin/bzip2
$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.14"
}}}
== Proposed fix:
Prefer the new "dist" module. If that isn't available, try
platform.dist(), and finally, for backwards compatibility, fall back to
/etc/issue. We could also try parsing /etc/os-release directly, but that's
just duplicating what "dist" does.
If its available, use the handy "dist.like()" property to fall back from
derived distributions to the root "debian", etc.
"dist" may not be installed by default, so include warning messages
describing the problem, and suggest "pip install dist".
(I'm including this as a verbatim block since some code is moved around
and the patch is a bit choppy to read.)
Old:
{{{
# platform.dist() was removed in Python 3.8
if hasattr(platform, 'dist'):
# Works for LSB distros
try:
distro = platform.dist()[0]
distro_ver = float(platform.dist()[1])
except ValueError:
# Non LSB distro found, use failover"
pass
else:
distro = ''
# Non LSB - fail over to issue
if distro == '':
try:
issue = open('/etc/issue').read()
distro = issue.split(' ')[0]
distro_ver = float(issue.split(' ')[2])
except:
pass
distro = distro.lower()
# Manage distro aliases
if distro in ['centos']:
distro = 'redhat'
elif distro in ['fedora']:
if distro_ver < 17:
distro = 'redhat'
elif distro in ['ubuntu', 'mx', 'linuxmint']:
distro = 'debian'
variations = {
'debian' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required', '/bin/sed')
},
'redhat' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__install_info': ('exe', 'required', '/sbin
/install-info'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required',
'/bin/sed'),
'__touch': ('exe', 'required',
'/bin/touch') },
'fedora' : { '__chown': ('exe', 'required',
'/usr/bin/chown'),
'__install_info': ('exe', 'required', '/usr/sbin
/install-info') },
'arch' : { '__gzip': ('exe', 'required',
'/usr/bin/gzip'),
'__chown': ('exe', 'required',
'/usr/bin/chown') },
'suse' : { '__chgrp': ('exe', 'required',
'/usr/bin/chgrp'),
'__chown': ('exe', 'required',
'/usr/sbin/chown') },
'gentoo' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__gzip': ('exe', 'required',
'/bin/gzip'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required', '/bin/sed')
},
}
if distro in variations:
for v in variations[distro]:
if path.exists(variations[distro][v][2]):
defines[v] = variations[distro][v]
}}}
New:
{{{
# platform.dist() was removed in Python 3.8
# The distro module (introduced in Python 3.6, back-ported to 2.6)
# is preferred.
distro = ''
distro_like = ''
distro_ver = 0
variations = {
'debian' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required', '/bin/sed')
},
'redhat' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__install_info': ('exe', 'required', '/sbin
/install-info'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required',
'/bin/sed'),
'__touch': ('exe', 'required',
'/bin/touch') },
'fedora' : { '__chown': ('exe', 'required',
'/usr/bin/chown'),
'__install_info': ('exe', 'required', '/usr/sbin
/install-info') },
'arch' : { '__gzip': ('exe', 'required',
'/usr/bin/gzip'),
'__chown': ('exe', 'required',
'/usr/bin/chown') },
'suse' : { '__chgrp': ('exe', 'required',
'/usr/bin/chgrp'),
'__chown': ('exe', 'required',
'/usr/sbin/chown') },
'gentoo' : { '__bzip2': ('exe', 'required',
'/bin/bzip2'),
'__chgrp': ('exe', 'required',
'/bin/chgrp'),
'__chown': ('exe', 'required',
'/bin/chown'),
'__gzip': ('exe', 'required',
'/bin/gzip'),
'__grep': ('exe', 'required',
'/bin/grep'),
'__sed': ('exe', 'required', '/bin/sed')
},
}
try:
import distro as distro_mod
distro = distro_mod.id()
distro_like = distro_mod.like()
try:
distro_ver = float(distro_mod.version())
except ValueError:
pass
except:
pass
if distro == '' and hasattr(platform, 'dist'):
distro = platform.dist()[0]
try:
distro_ver = float(platform.dist()[1])
except ValueError:
pass
# Non LSB - last resort, try issue
if distro == '':
try:
with open('/etc/issue') as f:
issue = f.read().split(' ')
distro = issue[0]
distro_ver = float(issue[2])
except:
pass
if distro:
distro = distro.lower()
if distro_like:
distro_like = distro_like.lower().split(' ')[0]
# Some additional distro aliases
if distro in ['centos']:
distro_like = 'redhat'
elif distro in ['fedora']:
if distro_ver < 17:
distro_like = 'redhat'
elif distro in ['ubuntu', 'mx', 'linuxmint']:
distro_like = 'debian'
if not (distro in variations) and (distro_like in variations):
distro = distro_like
# Versions don't carry over to likes; e.g. linuxmint 21.6 !=
debian 21.6.
distro_ver = 0
if distro in variations:
for v in variations[distro]:
if path.exists(variations[distro][v][2]):
defines[v] = variations[distro][v]
else:
log.warning('Unrecognized OS distro; assuming defaults for grep,
sed, etc.')
try:
distro_mod
except:
log.warning("The 'distro' package may fix this problem; try
'pip install distro'.")
}}}
== Tested on:
Gentoo, userland as-of July 2023.
Linuxmint 21.6
--
Ticket URL: <http://devel.rtems.org/ticket/4966>
RTEMS Project <http://www.rtems.org/>
RTEMS Project
More information about the bugs
mailing list