RESF Board Proposal: Increase Rocky's Sig/AltArch SBC support via purchase of needed tools

Meta Proposals
1

Note to community. This is a proposal to RESF for funds. In the spirit of open-ness, we want full community transparency and feedback. This proposal has been submitted to the RESF Board but has not been voted on yet. There is time for feedback, comment, and suggestions from the Rocky community. At this time, we (the Rocky leadership) are still trying to figure out the best way of handling proposals so this process will get better. For the moment, I took a lot of inspiration from Fedora as I like how they handle proposals (thank you to everyone at Fedora!). Please feel free to leave relevant thoughts, comments, feedback, suggestions below.

RESF Board Proposal: Increase Rocky’s Sig/AltArch SBC support via purchase of needed tools

Summary

Single Board Computers (SBC’s) are a small part of Rocky Linux numbers but the community is growing. There also exists a chicken-v-egg scenario in that many in the SBC community have turned away from Rocky because of a perceived lack of support yet a small demand has left a few dedicated Rocky members doing the majority of the work in supporting the SBC’s. This proposal aims to address this issue by requesting funds for developers, testers, and potentially others in the community as a method to help grow this part of the Rocky community.

Estimated cost: $19,500

Owner

Stack

Current Status

This idea has been discussed multiple times in the community. Polls were run for a week mid-December across five social media platforms and results collected. Many have informally volunteered assistance dependent on having hardware to do the work on. This proposal is addressing the immediate needs.

Detailed Description

Overview and problem description

Single Board Computers (SBC’s) are complete, or nearly complete, computer systems built on a single circuit board. Generally, they have a CPU, memory, networking, and general I/O features. They are commonly found in embedded systems, portable computers, hobbyist labs, educational systems, and personal computers.

The Raspberry Pi is, by far, the most well known. The Raspberry Pi foundation has done an excellent job in developing a community around their products. Until recent years, it was trivial to purchase a Raspberry Pi and thus there are many in the community who already have a lot of Raspberry Pi hardware but wish for a different OS then the official Raspbian. However, there are still a lot of SBC challengers who bring additional features that the Raspberry Pi does not have, or they provide more or less features at a more or less expensive price. The community for these alternative SBC projects is not as strong.

Author, Software Developer, popular Youtuber, and Rocky supported Jeff Geerling has made multiple comments about how bad Linux support is on alternative SBC’s. See the first minute of this video for one recent example: [WAY faster than a Raspberry Pi—but is it enough? - YouTube]. Jeff is only one voice in the crowd, but he’s got a voice many listen to and he’s only reflecting what the SBC community feels. Far too many of the SBC’s are challenging to get started with for most people who simply want an OS to launch their project from.

There has been an amazing effort by a few dedicated Rocky members to bring Rocky to the Raspberry Pi and a few other SBC’s. However, not all of these members have the tools and resources they need. Further, testing these images is difficult for many on Rocky releases as only a few members have access to the tools and resources needed to sufficiently test. Lastly, when issues do arise in the community there are only a few who can help.

This proposal seeks to address the issues of lack of resources and tools with the hope that this will encourage a more vibrant Rocky SBC community.

Proposal specifics

Rocky’s SIG/AltArch should “officially” support several SBC’s. As these are not supported by the upstream vendor, this “support” will always be community based inside of SIG/AltArch and other related means of Rocky Linux communication (eg: forums, Reddit, ect). Further, some SBC’s may require newer kernels from Rocky’s SIG/Kernel and thus “break” upstream kernel compatibility adding to “support” existing in SIG/AltArch. To ensure the best chance of support it is recommended that :

  1. Developers: Rocky will have a few people who have these devices who can ensure packages are compiled and images are built.
  2. Testers: Rocky will have a few people who have these devices who can run the tests for the Testing Team to satisfy the release checklists (install the image, verify functionality, run a few known-issues tests).
  3. Community: Rocky will have a few people who have these devices in_such_a_capacity that they can be used to help troubleshoot and debug community issues. This would mean an agreement between the community member and Rocky for a specific duration (EG: one year) that they could use these devices for their own purposes so long as they are willing to be active participants in the Rocky SIG/AltArch community. Ideally, these community members would have higher visibility for community to ask them questions (EG: a pinned topic in forums or headers suggesting posts tag these members in related questions).

The key take away is that we should have people in the Rocky community with each of these boards in each of these three categories. While there can be overlap, ideally the more the better. In the proposals perfect-world-use-case, Rocky would have at minimal five to ten people for each SBC. Two or more willing to assist in the “Developer” role ensuring packages, images, and releases are properly built. Two or more willing to assist in the “Tester” role ensuring tests pass on future releases. And two or more willing to participate in SIG/AltArch community conversations.

Community Polling

In mid-December 2022, community polls were run on Rocky’s Mattermost, Rocky’s forum, Reddit, Mastodon, and Twitter for a week. There were eighty-five responses evaluating fourteen of the most popular SBC’s. This poll did not include the 64bit Raspberry Pi 3 nor 4’s as Rocky SIG/AltArch already produces an image for these SBC’s.

The clear winner was a request for (armhfp/armv7) 32bit Raspberry Pi. Second place was (RISC-V) VisionFive 2. Third place was (ARM) ROCKPro64.

The hardest issue for the first and second place option is that they both require an entirely new architecture to support. While there is community work evaluating these architectures for future Rocky Linux releases, these two will be considered more of a footnote and extra-bonus to this proposal as the work involved in new architectures is quite large.

The RockPro64, the Pine A64-LTS (fourth place), the ODROID-N2+ (fifth place), and the Radxa ROCK 5 Model B (sixth place) are all similar architectures to the Raspberry Pi though they have their own quirks.

An additional SBC has been requested from within the Rocky community, the Nvidia Jetson Nano. As it should run the aarch64 architecture there should be little additional work needed to support and it falls into the same price categories as the others.

Cost Analysis

The pricing of these SBC’s are between $40 and $150 US dollars per SBC. This proposal uses the $150 price point for calculations as that easily covers the cost of the board, power, and shipping for the vast majority of the boards (other peripherals we can leave to the individual to supply). And the cost difference between the cheaper ones and the expensive ones I think is great enough that the buffer will cover the shipping & power of the expensive ones.

At a minimal

Sherif and Skip currently do nearly all of the development, testing, and support for SIG/AltArch. To purchase a RockPro64, Pine A64-LTS, ODROID-N2+, Radxa ROCK 5 Model B, Nvidia Jetson Nano, and one Raspberry Pi model that they don’t currently have is an estimated $1,800.

At least two Testers are needed to assist on release day: $1,800.

The minimal proposal for these six SBC’s: $3,600

Preferred

Ideally, Rocky SIG/AltArch would have ten members assisting in one or more of all three roles (Developer, Tester, Community) for each of the six SBCs: Raspberry Pi 3/4, RockPro64, Pine A64-LTS, ODROID-N2+, Radxa ROCK 5 Model B, NVidia Jetson Nano. However, it would not be ideal if the same ten people were responsible for all of these devices. As there have been volunteers in the past who wanted to help but did not have access to SBC’s, this proposal believes this is the best option to bring in more community assistance to both SIG/AltArch and Testing. Thus, there would be a process of identifying Rocky Members and Rocky community to take a smaller number of the SBC’s for different roles and capacities.

Ten members with six devices at $150 dollars each would cost roughly $9,000.

Above and beyond

By far the two most popular votes were for two SBC’s that require different architectures: 32 bit armhfp/armv7 and RISC-V. The prices on each of these is far cheaper then many of the other SBC’s mentioned. Further, many in the community have access to one or the other. The SBC cost is estimated to be under $500 (for two of each) to ensure that more community support is available for both of these boards. However, the cost for developing the architectures is more considerable and out of scope for this proposal at this time. This addendum is added because of the popularity of these boards in the polls reflecting a desire to have Rocky Linux supported. It is not a significant detriment to provide these additional boards to those willing to assist in future porting of these architectures but the immediate ROI will be low on these SBCs.

Shooting for the stars

An additional ten in the Rocky community receiving one of each of the six SBC’s. These would fall under the Community aspect and be “seeds” in the community to both draw in more of the SBC community and assist the Rocky community.

$9,000 for the Preferred core group + $500 for the additional Arch + $9,000 for community seeds + $1000 for overhead for shipping to multiple countries = a rough total of $19,500.

Additional thoughts

Due to both supply chain issues and the need to find proper volunteers, these purchases do not need to be done all at the same time. SIG/AltArch members could prioritize certain SBC’s. The cost analysis breakdown is not included here as the order in which SBC’s could be prioritized would be a messy matrix. Further breakdowns of price could be done if necessary once a priority list has been made.

Additionally, purchasing SBC’s that currently have no Rocky Linux image for community members decreases the desired goal of sparking excitement for Rocky Linux on these devices. The problem this proposal is trying to solve is the “I have the SBC but I can’t use it” thus getting the Developers access is the highest priority followed by Testing then Community.

Feedback

At this time, all Rocky Member and Rocky community comments have been positive. Several in the Rocky community have already volunteered to assist in one or more capacities dependent on having a device to work on. While Raspberry Pi 3’s and 4’s are quite popular, there are few that have both in a capacity to assist Rocky though the current lack of supply of Raspberry Pis may prevent Rocky from purchasing the required amounts.

Benefit to Rocky

While the Raspberry Pi community is quite strong, other SBC’s do have such strong communities and many are quite difficult to find information about. Having a stable OS like Rocky Linux would bring many of these SBC owners to Rocky and increase the community engagement as they develop more projects. As Rocky SIG Alt/Arch support grows, it is possible for relationships to develop with the upstream vendors for community referrals and co-operation to resolve issues.

Further, this provides additional opportunity for Rocky mentorship programs. It would identify those in the community willing to assist Rocky in exchange for hardware. These Community members would need to be guided and instructed on how to best participate in the Rocky Linux community in these new roles. This is a perfect opportunity to increase the mentorship required for eventual Rocky Membership.

Scope

  • Proposal owners: Stack
  • Other developers: Sherif, Skip
  • Other testing team members: Lukas, Al, Alan
  • Release engineering: Neil
  • Policies and guidelines: Rocky Members should be given first priority as they already have investment in Rocky. However, broader community engagement would be ideal. We would need to develop a loose guideline for Rocky community outside of Rocky Membership. It should not be complicated legal process. Rather a publicly acknowledged “handshake deal” for participation in exchange for the SBC’s. Guidance on how to properly mentor the community into membership roles would need to be defined.
  • Trademark approval: N/A
  • Alignment with Objectives: Yes. Rocky Linux objective is stated as “Rocky Linux is an open-source enterprise operating system designed to be 100% bug-for-bug compatible with Red Hat Enterprise LinuxÂŽ. It is under intensive development by the community.” This proposal falls under the “intensive development by the community” as it seeks to better support the SBC community within Rocky and grow contributions by the community by providing needed resources.

Upgrade/Compatibility Impact

N/A

How to Test

As each SBC becomes “officially supported by the Rocky SIG/AltArch” a testing plan will be developed by the Testing Team for each. A general testing plan would cover: installation, primary feature testing, and regression testing against previously known issues.

User experience

User experience should be simple and straight forward though this will change slightly for each SBC.

Dependencies

For the current proposal, there are no immediate dependencies. For future considerations of the alternative architectures (32 bit armhfp/armv7 and RISC-V), a much bigger conversation needs to be had with RelEng.

Contingency Plan

If this proposal fails for any reason, limited support by SIG/AltArch will remain a limited best-effort by a few members with access to the SBC’s.

Documentation

Will be developed as needed for each SBC.

Release Notes

N/A at this time. Though as support for each SBC grows, identifying these SBC’s publicly would be of great benefit.

Further information

As there were many Raspberry Pi devices discussed, the following two links are provided. A complete official list of all Raspberry Pi processors and their corresponding models, and the official store for all currently sold versions of the Raspberry Pi. Not all would need to be collected, but rather a general sampling comprised of both 64bit and 32bit processors.

The VisionFive2 community is still new and as such there are several sources for information:
VisionFive2 information: VisionFive 2 - open source quad-core RISC-V dev board by StarFive Tech — Kickstarter
VisionFive2 Github: GitHub - starfive-tech/VisionFive2
VisionFive2 official vendor list: How to purchase VisionFive 2 - VisionFive 2 English Forum) - RVspace Forum

RockPro64: ROCKPro64 | PINE64
Pine A64-LTS: PINE A64-LTS V2 Single Board Computer - PINE STORE
ODROID-N2+: ODROID-N2+ — ameriDroid
Radxa ROCK 5 Model B: ROCK 5 Model B — ameriDroid
NVidia Jetson Nano: NVIDIA Jetson Nano Developer Kit

2 Likes
2

It would be useful to add in where you expect to obtain the funds from, possible options might include:

  • RESF General fund.
  • Grant
  • Sponsor
  • go fund me?
    …etc
3

I support this and would also like to donate to it. I can’t commit to being a developer, but would definitely help test on the SBCs I have!

4

If this is a request for comments from the greater community (ie us users) then I’m mostly supportive of this. Seeing Rocky on more devices can only be a good thing.

But I’m not sure RESF should be funding community testing; the “shooting for the stars” part. If there’s sufficient demand and desire for Rocky on these devices then the community should step up and provide their own test equipment. You state these other SBCs have strong communities and, after all, people must have these devices if they want to run Rocky on them! If no one is willing to step up then it’s not clear the demand is there.

Anyway, just my zero-pennies (all I paid for Rocky; cheap at twice the price!) worth of thought. Apologies if I’ve overstepped any bounds.

5

Thanks for the feedback. The new board governance has only recently been put into place. They are still working out the details of funding so this is going to the RESF board as it is an expense that is not yet allocated thus they must approve it for now. There is a pool of donation funds for improving Rocky that I expect this to come from. But I don’t know those details yet.

6

Thanks for the feedback. As we get further along with this proposal, we will post more information. But if you have SBC’s now and can join SIG/AltArch over at https://chat.rockylinux.org that would be the easiest way to stay in the loop. We expect the next releases to be May-ish and would love assistance in testing then.

7

Greetings, thanks for the feedback and I’m glad you chimed in. The proposal is open for community feedback but it ultimately will be approved of by the RESF board. At this time, the board is overseeing the finances which is why they would need to approve of these purchases.

As the proposal mentions, the problem is a bit of a chicken-and-egg one. We have a few in the community providing most of the work for the SBC’s but they don’t have all the resources they need to support community requests nor do we have enough testers. However, the SBC community is growing within Rocky but many of them do not have spares for testing Rocky as they are actively using the SBC for their projects. This proposal is to purchase additional SBC’s for the intended purpose of assisting Rocky in various aspects.

I hope that helps clarify. Please feel free to add more.

8

Just as another point. I think that right now the SBC landscape is relatively small due to the chip shortage making it difficult to come across boards, but once that issue is solved I think that the demand for support will skyrocket and so it is very much a good thing for us to stay ahead of it with this SIG. Therefore I very much approve of this proposal FWIW.

1 Like
9

…oh and one more thing. This proposal should be considered an initial investment, but there’s going to be new manufacturers, new board designs and chip designs and newer board versions from existing manufacturers going forward, so some sort of ongoing funding and investment for this would be ideal going forward.

10

Seeing the mention in OP, I’m flattered :slight_smile:

The default on almost every other non-RPi board seems to be ‘Debian, and maybe Ubuntu’. Ubuntu/Canonical themselves have dedicated some resources towards remaining a viable option on many different ARM SBCs, which is nice, but there’s traditionally been much less awareness of RHEL/derivative OSes.

So it would be nice to combat some of that hegemony in the space by having Rocky Linux available on more boards. Doubly so if Rocky Linux’s builds focused on server usage in particular (e.g. lower memory footprint, small image size, lower resource usage out of the box). At least, that’s my opinion!

2 Likes
11

What’s the possibility to get these companies (Rock, Pine, Rpi, Odroid, etc) to maybe donate a few of these boards - as in self-interest. Like a sponsor/semi-sponsor-deal. I mean, they litterally hand out all these products from left to right to all kind of youtubers, to test and review. A few xtra boards donated for some real testing wouldn’t be too much of a cost for them, one would think. :slight_smile:

12

Absolutely. Thanks for adding that into the discussion.

Thanks for chiming in! Absolutely. This seems to be a popular usage for those that are inquiring about Rocky on various SBC’s.

There was an attempt to a few of the companies early on that didn’t go far. This was briefly brought up again in our team meeting and the consensus was that it’s worth trying again now that we are a bit more established. We also have a few corporate sponsors that might be willing to assist that we are going to reach out to. I think it also helps that things like this proposal show that Rocky Linux is serious in our desire to support the SBC’s and that we have community asking for/wanting it. I don’t know what will happen, but it’s worth asking. If you happen to have contacts… or just want to @ them on your favorite social media it wouldn’t hurt. :grinning:
Thanks for chiming in with the suggestion!