[RTEMS Project] #4784: Update servers to latest FreeBSD 13.1

[RTEMS trac]

#4784: Update servers to latest FreeBSD 13.1
------------------------------------------------+--------------------------
 Reporter:  Amar Takhar                         |       Owner:  Amar Takhar
     Type:  infra                               |      Status:  accepted
 Priority:  highest                             |   Milestone:  Indefinite
Component:  admin                               |     Version:
 Severity:  normal                              |  Resolution:
 Keywords:  funded project-1                    |  Blocked By:
 Blocking:  3606, 4785, 4786, 4787, 4789, 4790  |
------------------------------------------------+--------------------------

Comment (by Amar Takhar):

 A lot has been going on the past week with the general server update.
 There are dozens of services we run and we need a system that will be easy
 to maintain going forward.

 Each service really needs to be insular so we can change out one for
 another as updates happen or the requirements of the project change.  This
 has been a current difficulty.

 Overall as a general methodology the following rules have been followed:

   * Spend more time on setup for less longterm maintenance.
   * A longer restoration is OK for less longterm maintenance.
   * Heavyweight solutions that require more upkeep are dropped in lieu of
 simple crontabs and scripts located in one directory.
   * Setup is consistent across all servers and Jails.

 We have not had any major issues in the last 9 years.  If we had chosen
 more heavyweight options we would have garnered no benefit other than more
 complexity and ongoing maintenance.

 As the project expands horizontal scaling is OK but we don't want
 maintenance to grow exponentially.  By having a static cost associated
 with maintaining services it will make it far easier for the project to
 expand.

 If, in the future we need to go with more heavyweight options we can.
 Going backwards is a lot more work but switching from a say a crontabbed
 backup to a service is trivial.

 The following software will be deployed on each server:

   * Jails will be managed using https://bastillebsd.org/
   * Backups are crontabbed using https://restic.net
   * Logs rotated using https://github.com/logrotate/logrotate
   * https://certbot.eff.org/ for certificate creation and updating

 There are still a lot more decisions to be made but those will have to
 happen as setup progresses.  The major point in doing in depth planning
 was to avoid any major rewind of deployments if another, better solution
 is discovered late in the process.

--
Ticket URL: <http://devel.rtems.org/ticket/4784#comment:9>
RTEMS Project <http://www.rtems.org/>
RTEMS Project