Rocky Linux HCI setup with libvirt/KVM?

Hi all,

Has anyone here built an HCI-style setup using Rocky Linux as the base OS?

I’m mainly curious about running a small cluster with:

  • libvirt/KVM for VMs

  • Ceph or another shared/distributed storage solution

  • standard Rocky packages where possible: BaseOS, AppStream, Extras, maybe EPEL

  • no Kubernetes

What tools are people using for this kind of setup?

For example:

  • storage: Ceph, ZFS, NFS/iSCSI, something else?

  • HA/cluster management: Pacemaker/Corosync, custom scripts, Ansible, etc.?

  • VM migration and node maintenance

  • provisioning/reinstalling nodes

Mostly just looking for real-world experiences and what people recommend or avoid.

Thanks!

Well no idea if I even posted it the right place. Just wanted to know if anyone has done this as I think HCI are kinda interesting. And even though I prefer and like Proxmox. I just wanna have a alternative too it as well. And really enjoyed using Rocky for vms and so on.

I ran a gaggle of RL9 hosts offering libvirt as a precursor to jumping off ESXi, all of it while testing alternatives like oVirt (and ALVM/OLVM) and OpenNebula. Finally, the weird requirement to run ansible in a container to get the dependencies right as a dependency to running any oVirt was a bridge way, way too far.

I’ve used Puppet, Chef/Cinc, and 25 years ago we used RPM installs, triggers, make, cron and sed/awkas templaters for DevOps long before they called it that. I’ve used Ansible exensively for the last few years, at my day job. Puppet, or the make/cron/sed/awk/rpmsimple watch-and-converge setup, and especially chef, were all absolutely light-years ahead of what ansible does now. And orders of magnitude faster. I will switch cinc for mgmt(configmgmt.com) when I get up on it because it’s amazing and should be the orchestration under oVirt because it’s real-time. I competed for and won my last job while requiring a temporary wage adjustment while they continued to use ansible. It was a couple-thousand a year, and it’s now part of any interview with a company using ansible or outlook. But aside from a recommendation to learn and run anything else, that’s all I’ll say about Ansible.

I’m deploying proxmox - the only non-Enterprise Linux device at any site - and since it’s replacing ESXi, the amount of out-of-band patching we’ll need is about the same effort. Snowflake patch, bounce, hold nose, move on. Sleep at night.

oVirt looked great. But its install was crunchy and frail … usually because ansible would choke on a repo URL or something. OLVM’s install was better, actually, due to a little fresher maintenance. ALVM was almost as good, and Alma people are also great people. If you wanna run an oVirt and you’re okay with ansible, then go grab ALVM and try that.

But here’s what’s gonna happen. You’re gonna worry about how much you have to futz with things – I mean, it’s fun when you have time, but you need to have a stable daily driver so you can get work done, and leave experimentation to the weekend. You’re gonna come back to Proxmox, you’re going to explore Gluster and DRBD/Linstor, because you heard about ceph freaking out and losing reams of data. But, I mean, xfs did too. And Ceph got better. So you’re gonna pick ceph, using the webUI to set it up, and you’re gonna stop messing with your production/tools cluster at that point, and you’re gonna mess with virtual-in-virtual and try all the things again and it’s gonna be super fun.

But when you’re doing that, if you go VM on LIbvirt VM host on Proxmox, and you start working with pacemaker/keepalive (the latter’s easy as pie) and gluster and all those things, and you build some scaffolding scripts to manage disks and vMotion and all that, I think it’s going to be a great topic for discussion.

If hardware isn’t an issue, OpenShift Virtualization is HCI. Or you can do Kubernetes with Kubevirt and add something like Rook on Kubernetes to get Ceph.

oh wow, thats many! some I have heard of like oVirt, OLVM and Proxmox of course. And some I have not heard of like OpenNebula and ALVM (could not find much about it online).

Also been using ansible a lot on vms and bearly heard of puppet. Chef/Cinc is new for me and need to look up. Mgmt and the link you gave me pointed to nowhere.

But thanks for the information and experience you have had! Kinda looking at this for homelab setup now (do run proxmox now but always fun to try something else). And just wanna see how it goes. But have been looking at replacing stuff at work too. We run Azure local (its been a pain) and Morpheus from HPE (just got it). But honestly all of these solutions have been more painful, expensive and annoying to use then what Proxmox gives. But also just wanted to look into setting stuff on my own is a viable option too. Primarily run everything in vms (containers as well inside vms for islation and security) and my experience on Kubernetes is limited.

Again thanks for sharing!

I see! been also looking at “okd” as i do want something fully open sourced and available for me. Been kinda looking at Kubernetes as well? but so many considerations to be careful there. I guess looking into Kubevirt would be a thing? but not sure yet. Also been looking at Talos for Containers? but again no experience there.

Right now I keep it simple and run most container works isolated inside vms as then i know its the safest and easiest option to manage at my level of skills.