<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en" style="--code-editor-font: var(--default-mono-font, "GitLab Mono"), JetBrains Mono, Menlo, DejaVu Sans Mono, Liberation Mono, Consolas, Ubuntu Mono, Courier New, andale mono, lucida console, monospace;">
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<title>
GitLab
</title>
<style data-premailer="ignore" type="text/css">
a { color: #1068bf; }
</style>
<style>img {
max-width: 100%; height: auto;
}
body {
font-size: .875rem;
}
body {
-webkit-text-shadow: hsla(0,0%,100%,.01) 0 0 1px;
}
body {
font-family: "GitLab Sans",-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"; font-size: inherit;
}
</style>
</head>
<body style="font-size: inherit; -webkit-text-shadow: hsla(0,0%,100%,.01) 0 0 1px; font-family: "GitLab Sans",-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";">
<div class="content">
<p class="details" style="font-style: italic; color: #626168;">
Epic created by <a href="https://gitlab.rtems.org/gedare">Gedare Bloom</a>: <a href="https://gitlab.rtems.org/groups/rtems/-/work_items/39">#39</a>
</p>
<div class="md" style="position: relative; z-index: 1; color: #3a383f; word-wrap: break-word;">
<h2 id="user-content-summary" dir="auto" style="margin-top: 0px; margin-bottom: 10px;" align="initial">Summary<a href="#summary" aria-label="Link to heading 'Summary'" data-heading-content="Summary" class="anchor" style="margin-top: 0px;"></a>
</h2>
<p dir="auto" style="color: #3a383f; margin: 0px 0px 1rem;" align="initial">Our <code style="font-size: 90%; color: #18171d; word-wrap: break-word; background-color: #ececef; border-radius: .25rem; margin-top: 0px; font-weight: inherit; overflow-wrap: break-word; white-space: break-spaces; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: keep-all; padding: 0.125rem 0.25rem;">testsuites</code> has grown overly cumbersome and contributes to excessive build and test run times. The goal of this epic is to reduce and consolidate tests without losing test coverage so that RTEMS development work can be simpler and faster.</p>
<p dir="auto" style="color: #3a383f; margin: 0px 0px 1rem;" align="initial">As an example of the current state and problem, building a (rv64imafdc) BSP with <code style="font-size: 90%; color: #18171d; word-wrap: break-word; background-color: #ececef; border-radius: .25rem; margin-top: 0px; font-weight: inherit; overflow-wrap: break-word; white-space: break-spaces; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: keep-all; padding: 0.125rem 0.25rem;">BUILD_TESTS = true</code> compiles 4638 files and on a workstation took 1 minute 30 seconds, while building with <code style="font-size: 90%; color: #18171d; word-wrap: break-word; background-color: #ececef; border-radius: .25rem; font-weight: inherit; overflow-wrap: break-word; white-space: break-spaces; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: keep-all; padding: 0.125rem 0.25rem;">BUILD_TESTS = false</code> compiles 1483 files and took 27 seconds on the same workstation. Although there are several options to reduce the test suites at a coarse-grained level, this puts the burden on a developer to know and configure which tests are relevant for a specific change.</p>
<p dir="auto" style="color: #3a383f; margin: 0px 0px 1rem;" align="initial">The validation tests trade off building time for running. They run really long, but build many fewer tests. They also brute-force API limits, that's a big part of the time overhead. Since their purpose is to support qualification efforts, their suitability for general-purpose or even coverage testing of RTEMS is not clear.</p>
<h2 id="user-content-approach" dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">Approach<a href="#approach" aria-label="Link to heading 'Approach'" data-heading-content="Approach" class="anchor" style="margin-top: 0px;"></a>
</h2>
<h3 id="user-content-unit-testing" dir="auto" style="margin-top: 1.5rem; margin-bottom: 1rem; font-weight: 600; font-size: 1.125rem; letter-spacing: inherit; line-height: 1.25; color: #18171d;" align="initial">Unit Testing<a href="#unit-testing" aria-label="Link to heading 'Unit Testing'" data-heading-content="Unit Testing" class="anchor" style="margin-top: 0px;"></a>
</h3>
<p dir="auto" style="color: #3a383f; margin: 0px 0px 1rem;" align="initial">Improving the ability to write unit tests for RTEMS has been attempted in various ways in the past. There is an obsolete port of <code style="font-size: 90%; color: #18171d; word-wrap: break-word; background-color: #ececef; border-radius: .25rem; margin-top: 0px; font-weight: inherit; overflow-wrap: break-word; white-space: break-spaces; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: keep-all; padding: 0.125rem 0.25rem;">rtems-gtest</code> that has been archived on our FTP server. The current direction that has been taken is the <a href="https://docs.rtems.org/docs/main/eng/test-framework.html" rel="nofollow noreferrer noopener" target="_blank">RTEMS Test Framework</a> but adoption and use of this framework has been slow going. Converting existing tests into the Test Framework could be a good direction for making improvements. Improving the parsing ability of <code style="font-size: 90%; color: #18171d; word-wrap: break-word; background-color: #ececef; border-radius: .25rem; font-weight: inherit; overflow-wrap: break-word; white-space: break-spaces; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: keep-all; padding: 0.125rem 0.25rem;">rtems-test</code> to understand the RTEMS Testing Framework is integral to the long-term success of such an effort, along with improving the documentation "how to" write tests with it.</p>
<h3 id="user-content-requirements-and-documentation" dir="auto" style="margin-top: 1.5rem; margin-bottom: 1rem; font-weight: 600; font-size: 1.125rem; letter-spacing: inherit; line-height: 1.25; color: #18171d;" align="initial">Requirements and Documentation<a href="#requirements-and-documentation" aria-label="Link to heading 'Requirements and Documentation'" data-heading-content="Requirements and Documentation" class="anchor" style="margin-top: 0px;"></a>
</h3>
<p dir="auto" style="color: #3a383f; margin: 0px 0px 1rem;" align="initial">Part of this effort would need to establish the requirements/expectations around testsuites. That will give us something concrete for making any decisions. There are brief sections in the <a href="https://docs.rtems.org/docs/main/user/testing/index.html" rel="nofollow noreferrer noopener" target="_blank" style="margin-top: 0px;">User Manual</a> and the <a href="https://docs.rtems.org/docs/main/eng/test-plan.html" rel="nofollow noreferrer noopener" target="_blank">Software Engineering Manual</a> that might serve as a starting point for documenting the course of action. Some initial requirements that come to mind for any work done in this direction follows:</p>
<ul dir="auto" style="text-align: initial; list-style-type: disc; margin: 0px; padding: 0;">
<li style="margin-top: 0px; line-height: 1.6em; margin-left: 25px; padding-left: 3px;">Merge requests that refactor and remove existing tests needs to <a href="https://docs.rtems.org/docs/main/user/testing/coverage.html" rel="nofollow noreferrer noopener" target="_blank" style="margin-top: 0px;">run coverage</a> before and after major changes to ensure nothing is lost.</li>
<li style="line-height: 1.6em; margin-left: 25px; padding-left: 3px;">Test sizes need to accommodate the variety of targets supported by RTEMS.</li>
<li style="line-height: 1.6em; margin-left: 25px; padding-left: 3px;">Tests that exercise tight resource allocations need to preserve those constraints.</li>
</ul>
</div>
</div>
<div class="footer" style="margin-top: 10px;">
<p style="font-size: small; color: #626168;">
—
<br>
<a href="https://gitlab.rtems.org/groups/rtems/-/work_items/39">View it on GitLab</a>.
<br>
You're receiving this email because of your account on <a target="_blank" rel="noopener noreferrer" href="https://gitlab.rtems.org">gitlab.rtems.org</a>. <a href="https://gitlab.rtems.org/-/sent_notifications/4-a5hqwpjp16vrcpsyzw8pihqxs-3/unsubscribe" target="_blank" rel="noopener noreferrer">Unsubscribe</a> from this thread · <a href="https://gitlab.rtems.org/-/profile/notifications" target="_blank" rel="noopener noreferrer" class="mng-notif-link">Manage all notifications</a> · <a href="https://gitlab.rtems.org/help" target="_blank" rel="noopener noreferrer" class="help-link">Help</a>
<span style="color: transparent; font-size: 0; display: none; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0;">
Notification message regarding https://gitlab.rtems.org/groups/rtems/-/work_items/39 at 1783020316
</span>
<script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","action":{"@type":"ViewAction","name":"View Work item","url":"https://gitlab.rtems.org/groups/rtems/-/work_items/39"}}</script>
</p>
</div>
</body>
</html>