What servers/services do we need to bootstrap ourselves

  • FreeIPA (and associated items)
  • AWX (and associated items)
  • Build things? (What, how many, etc)

Comment and I’ll add them, we can discuss.

If listing services, can they share a (VM/Server) - keeping in mind we may be limited in terms of ability to perform HA at first.

You all know this more than me, I hope :slight_smile:

1 Like

git forge?

Something along the lines of self-hosted gitlab.com or pagure.io

1 Like

I think we should put the general requirement down with the current agreed upon recommendation so we know why the item was chosen instead of specifying the products that we have recommended. and build out how that architecture works.

  • Identity Provider - FreeIPA
  • SAML Provider - Keycloak ipsilon (nice to have)
  • Configuration Management (Ansible)
  • Configuration Management Orchestration (AWX)
  • Forge - Gittea/lab?
  • Build system - Koji
  • bug tracking - mantis?
  • wiki - wiki.js moinmo
  • backup - restic rsync
  • monitoring - Icinga+prometheus+influxdb? zabbix
  • secrets management - Vault/FreeIPA Encrypt

I am sure i am missing quite a bit here but i wanted to get the list down.

2 Likes

For monitoring, definitely look into Prometheus; it’s pretty simple to set up and quite powerful.
Sourcehut even has a neat public instance of their infrastructure monitoring:

I’m personally meh on zabbix. If people are into it… we can. But something more modern feels better…

grafana/prometheus/influxdb type stack.

While we’re all chatting - if we hypothetically had access to dedicated hosts, how would we want to use them? Hypervisors/clustered workloads (openstack/other)… or just baremetal.

Nothing is set in stone, and we don’t know what or how many of anything we might have.

3 Likes

I’d definitely be in favour of running virtualization on top of hardware hosts where possible. It just makes life simpler.

Having a cloud system to use would also be useful, but I don’t want to inflict OpenStack on anyone :stuck_out_tongue: I vote on putting it in the “maybe later” pile. I do have experience with it if needed, though…

2 Likes

Agreed for prometheus/influxdb stack instead of zabbix that list will make more sense in a minute with my next two posts. Definitely will need some sort of virtualisation but more importantly we will need some sor of IPMI regardless of the stack since the team will be so distributed

my vote there is netbox. and also we’ll need DNS infrastructure…

we could see if ns1 is open to donating stuff. they even have DDI…

1 Like

netbox and Grafana are great! also are we gonna use Gitlab / github in general ?

1 Like

Pagure would be my vote for a git forge, with gitea as a close second. It may be worth also investigating mirrored repos with github/gitlab in the future.

1 Like

Have a look also to Icinga for host/service health monitoring and alerting. It has good integration with Graphite and a C style DSL for host checks configurations and it’s fully compatible with existing Nagios checks. Its functionality can also be extended with other tools via modules and plugins (eg. director, reporting etc).

Grafana+Prometheus is another good option. Prometheus has also builtin alerting functionality via Alertmanager which has to be configured separately though. Grafana has also pretty interesting “satellite” projects like Loki for log exploration.

My two cents for netbox. Pretty neat and simple DCIM tool with all the features you probably need to organize your infra information.

We have been running all of these on prem (single node setups) quite some time now without any major issues. Something that has to be considered carefully is the storage needs of Prometheus (if you would like to have long data retention policies) and that you need a separate project in order to have a scalable highly available underline storage for it like Cortex or Thanos. Additionally for Icinga, it has builtin HA functionality but it needs some searching and testing to make it work without issues with Icinga’s Web2 GUI.

Is it fair to assume everything on this list would benefit from being logically networked together? I’m thinking in terms of zabbix really only being effective if it’s in pretty constant contact with these servers.

Though if that was the main concern, there are likely alternatives that could be reached out to like New Relic and Datadog.

I suspect we will want to run some sort of overlay to connect everything together… basically cluster what we can and vpn the rest together.

1 Like

For monitoring: I’m a developer of openITCOCKPIT which is basically Nagios on steroids. It cams with Graphite, Grafana, Web UI and API. I could assist on setup.

I don’t want to make any AD so please checkout the website / GitHub repo for more information.

For SAML maybe keycloak is worth looking at. Supports SAML2 and also others and can be used on top of FreeIPA.

2 Likes

Also not sure what the specs are for the wiki but I really like WikiJS. Stil heavily under development but looks really promising. Has also some features as syncing to a GIT repository (pull, push or 2-way)

why not use github for git, it offers issue tracking and a wiki. so it would solve many issues.

also it has excellent integration with more or less everything. and hosting these components on there own, in an reliable way can be pretty hard.

also it makes contributing way easier, since most people already have a github account.

also the python lib for interacting with it is pretty neat.

Agreed, I really prefer working with Keycloak to be honest but the ipsilon was just something that was mentioned previously.

I never used Keycloak, but isn’t that what RH uses for SSO? also won’t gitlab be a bit better and “open” than github?

running a group of mirrors distributed among all the continents will be necessary. Count on us.
Ernesto Pérez
https://mirror.cedia.org.ec

2 Likes