Yes, @sthornton has done a lot of work here so that when Apollo runs, errata is correctly processed. The next step was to make it easier and faster to run Apollo, and Sam just submitted a PR for exactly that. You can have a look at this here:
This PR introduces a comprehensive API key authentication system and admin interface for Apollo, enabling secure programmatic access to workflow management and database operations. The changes span 6 commits and add significant new functionality while maintaining backward compatibility […] and represents a significant enhancement to Apollo’s automation capabilities while maintaining production stability and security.
[…]
New Functionality Available After Deployment
API-driven workflow triggers for automation and CI/CD integration
Programmatic access to workflow status and management
Admin interface enhancements for better user and key management
Development database reset capabilities for testing environments
We’ll update this thread as soon as this is merged.
Currently, Apollo risks missing (or “skipping”) some Red Hat advisories when cloning them for Rocky Linux version 9.X. This occurs due to the interaction between Peridot-hosted repositories and the manual, error-prone nature of advisory matching:
Peridot Repositories: Rocky Linux Peridot RPM repositories only retain the most recently built artifact for each package. When a new build is published, older builds are removed from the repository.
Manual Workflow Execution: The RhMatcherWorkflow is typically run manually during repository staging or release. In practice, this step is often skipped or delayed, especially during busy update cycles.
Window for Matching: The workflow matches advisories only for packages currently present in the repository. If multiple new package versions (e.g., kernel v1, v2, v3) are published before the workflow runs, advisories for earlier versions (v1, v2) can be permanently missed, as those builds are no longer available for matching.
Staging vs. Live: Matching advisories against staging repositories is a manual process and presents risk, as these may contain unreleased content. Matching against live repositories is safe and suitable for automation.
Example – Rocky 9.6 Kernel Advisories:
On a development instance of Apollo, only the most recent Rocky 9.6 kernel advisory (RLSA-2025:14420) was cloned and published. Earlier advisories, such as RHSA-2025:13962, were skipped. This can be confirmed by searching the dev instance, which shows only one 9.6 kernel advisory. This demonstrates the real-world impact of the current workflow limitations.
Apollo’s current advisory cloning process:
Pulls Red Hat advisories and related metadata (CVEs, Bugzilla, affected products) into a PostgreSQL database.
Matches Red Hat packages to Rocky Linux packages using repository metadata.
Clones and adapts advisories for Rocky Linux, creating new records and mapping associated packages, CVEs, and fixes.
Currently relies on manual triggers for the matching process, which is the root of the issue described above.
Summary:
To close this gap, a fully automated matcher workflow is needed. This workflow should run on a regular schedule (cron) and target only the live public Rocky Linux repositories, serving as a reliable backup to ensure all applicable advisories are cloned, regardless of manual process reliability. This is a crucial stopgap until broader improvements are implemented.
Objective
Develop and deploy a new, automated matcher workflow that:
Runs on a configurable schedule (default: every hour via cron).
Only scans Rocky Linux live public repository URLs (not staging).
Ensures all relevant advisories are cloned as soon as possible after new builds are released.
Minimizes the risk of missing the manual step for an engineer running the matcher workflow.
Implementation Plan
Database Changes
Add a new table or extend existing product/repo tables to store Rocky Linux live public repository URLs (explicitly separate from staging).
New Matcher Workflow
Implement a new workflow (parallel to/complementing RHMatcherWorkflow) that:
Queries only the live public repo URLs from the database.
Matches Red Hat advisories to only those packages present in live repos.
Avoids advisory generation for packages not yet published to users (i.e., present only in staging).
Cron Integration
Deploy the new matcher workflow as a scheduled task (cronjob).
Default frequency: every hour (should be configurable via environment variable or config DB entry).
Documentation & Communication
Document the workflow, configuration options, and rationale.
Communicate to stakeholders that this closes the gap where advisories could be missed due to infrequent matcher runs.
Why This Is Needed
Completeness: Guarantees all relevant security and bugfix advisories are cloned and published for Rocky users.
Reliability: Reduces manual operations and risk of human error/omission.
Trust: Maintains alignment with Red Hat advisories, supporting compliance and user expectations.
Flexibility: Configurable schedule allows tuning based on repo activity and resource availability.
Timeline
I expect to have a pre-PR ready for review today which adds some needed functionality for managing Apollo repomd.xml URLs using the admin portal built into the Apollo UI.
I expect to prepare an initial pull request for community review within approximately 1 week.
Summary:
By automating the matcher workflow on a regular schedule and focusing exclusively on public live repositories, we ensure that no advisories are skipped—even if there are multiple package builds between workflow runs. This adaptation is critical for maintaining the integrity and security of the Rocky Linux advisory system.
I’ve opened a PR with some initial groundwork that will make the development of these proposed features much easier and which should provide some easier to use functionality for managing Apollo configuration updates in the future.
This PR was huge and as such I’ve decided to cancel it. I’m working on breaking it apart into smaller PRs that I will get reviewed by CIQ folks before pushing a comprehensive PR to the RESF with links to the PRs reviewed by CIQ. The goal here is to reduce the load on RESF engineers in the need to review a massive PR in extreme detail (though this is still welcomed).
Thank you for all the work that’s gone into the Apollo errata initiative — it’s really appreciated.
Could you please share if there have been any recent updates on the errata data publishing? Our monthly patching cycle depends on having accurate and current information, so even a manual update once a month would be very helpful in the meantime.
Thanks for reaching out. The RESF has merged in changes that have allowed for regular errata generation. Can you help me understand how you are consuming the RESF errata? There are number of different ways:
These are being updated daily though there are some days where we don’t have anything to generate. Please feel free to reach out with further questions either here on on Mattermost.
In our case, we use a patching and compliance tool that pulls information directly from the standard Rocky repositories — mainly the metadata files like repomd.xml and the *-UPDATEINFO.xml.gz under /repodata/ (for example, here).
The tool relies on those files to determine which packages have new errata or security advisories.
I unfortunately don’t have any ability to deploy the updated repomedata. This is part of series of workflows that are managed by the RESF Release Engineering team. I would highly recommend asking in the Mattermost Security channel as you may get a more timely response there. My current understanding is that whenever new content is deployed into the repos the underlying tooling should be calling the Apollo API to generate fresh updateinfo.xml so ideally the updated repometadata isn’t too far delayed behind the advisories showing up in the Apollo system. You’ve got me thinking now and I may see if I can get something pulled together that could track the statistics for delays between and advisory showing up in Apollo and that same advisory being available in updateinfo.xml.
I’m very happy to report that after the release of 9.7, the latest security errata are being ingested into my third party patch/vuln management systems in a timely manner. The systemd vuln was available instantly and we were tracking that down to completion.
This is a tremendous step forward and hoping the momentum for RLBAs and RLEAs to be reinstated as well. It’s a large win and congrats to the RL leaders and their development teams.