Apollo, Errata, & You: a CIQ OSPO request for comment

Teams Release Engineering
24

Problem Overview

Currently, Apollo risks missing (or “skipping”) some Red Hat advisories when cloning them for Rocky Linux version 9.X. This occurs due to the interaction between Peridot-hosted repositories and the manual, error-prone nature of advisory matching:

  • Peridot Repositories: Rocky Linux Peridot RPM repositories only retain the most recently built artifact for each package. When a new build is published, older builds are removed from the repository.

  • Manual Workflow Execution: The RhMatcherWorkflow is typically run manually during repository staging or release. In practice, this step is often skipped or delayed, especially during busy update cycles.

  • Window for Matching: The workflow matches advisories only for packages currently present in the repository. If multiple new package versions (e.g., kernel v1, v2, v3) are published before the workflow runs, advisories for earlier versions (v1, v2) can be permanently missed, as those builds are no longer available for matching.

  • Staging vs. Live: Matching advisories against staging repositories is a manual process and presents risk, as these may contain unreleased content. Matching against live repositories is safe and suitable for automation.

Example – Rocky 9.6 Kernel Advisories:
On a development instance of Apollo, only the most recent Rocky 9.6 kernel advisory (RLSA-2025:14420) was cloned and published. Earlier advisories, such as RHSA-2025:13962, were skipped. This can be confirmed by searching the dev instance, which shows only one 9.6 kernel advisory. This demonstrates the real-world impact of the current workflow limitations.

Apollo’s current advisory cloning process:

  • Pulls Red Hat advisories and related metadata (CVEs, Bugzilla, affected products) into a PostgreSQL database.

  • Matches Red Hat packages to Rocky Linux packages using repository metadata.

  • Clones and adapts advisories for Rocky Linux, creating new records and mapping associated packages, CVEs, and fixes.

  • Currently relies on manual triggers for the matching process, which is the root of the issue described above.

Summary:
To close this gap, a fully automated matcher workflow is needed. This workflow should run on a regular schedule (cron) and target only the live public Rocky Linux repositories, serving as a reliable backup to ensure all applicable advisories are cloned, regardless of manual process reliability. This is a crucial stopgap until broader improvements are implemented.

Objective

Develop and deploy a new, automated matcher workflow that:

  • Runs on a configurable schedule (default: every hour via cron).

  • Only scans Rocky Linux live public repository URLs (not staging).

  • Ensures all relevant advisories are cloned as soon as possible after new builds are released.

  • Minimizes the risk of missing the manual step for an engineer running the matcher workflow.

Implementation Plan

  1. Database Changes

    • Add a new table or extend existing product/repo tables to store Rocky Linux live public repository URLs (explicitly separate from staging).

  2. New Matcher Workflow

    • Implement a new workflow (parallel to/complementing RHMatcherWorkflow) that:

      • Queries only the live public repo URLs from the database.

      • Matches Red Hat advisories to only those packages present in live repos.

      • Avoids advisory generation for packages not yet published to users (i.e., present only in staging).

  3. Cron Integration

    • Deploy the new matcher workflow as a scheduled task (cronjob).

    • Default frequency: every hour (should be configurable via environment variable or config DB entry).

  4. Documentation & Communication

    • Document the workflow, configuration options, and rationale.

    • Communicate to stakeholders that this closes the gap where advisories could be missed due to infrequent matcher runs.

Why This Is Needed

  • Completeness: Guarantees all relevant security and bugfix advisories are cloned and published for Rocky users.

  • Reliability: Reduces manual operations and risk of human error/omission.

  • Trust: Maintains alignment with Red Hat advisories, supporting compliance and user expectations.

  • Flexibility: Configurable schedule allows tuning based on repo activity and resource availability.

Timeline

  • I expect to have a pre-PR ready for review today which adds some needed functionality for managing Apollo repomd.xml URLs using the admin portal built into the Apollo UI.
  • I expect to prepare an initial pull request for community review within approximately 1 week.

Summary:
By automating the matcher workflow on a regular schedule and focusing exclusively on public live repositories, we ensure that no advisories are skipped—even if there are multiple package builds between workflow runs. This adaptation is critical for maintaining the integrity and security of the Rocky Linux advisory system.