AppStream repo giving error "Error: No available modular metadata for modular package"

Getting below error while installing the packages from AppStream repo.

Downloading Packages:
[SKIPPED] php-xml-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-opcache-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-common-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-cli-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-mbstring-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-pdo-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-gd-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
[SKIPPED] php-fpm-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64.rpm: Already downloaded
Running transaction check
No available modular metadata for modular package 'php-xml-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-opcache-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-common-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-cli-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-mbstring-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-pdo-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-gd-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
No available modular metadata for modular package 'php-fpm-8.1.14-1.module+el9.2.0+14732+817c3ec2.x86_64', it cannot be installed on the system
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: No available modular metadata for modular package

this repos have been synced in Foreman. While using the local repo we are not seeing this error.

able to solve by doing below changes, not the perfect solution but for now it help me to move on.

1. enable debug for rhsm

/etc/rhsm/rhsm.conf

[logging]
default_log_level = DEBUG



2. check the log 

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 176, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 246, in do_transaction
    tid = super(BaseCli, self).do_transaction(display)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 977, in do_transaction
    self.transaction._populate_rpm_ts(self._ts)
  File "/usr/lib/python3.9/site-packages/dnf/db/group.py", line 361, in _populate_rpm_ts
    raise dnf.exceptions.Error(_("No available modular metadata for modular package"))
dnf.exceptions.Error: No available modular metadata for modular package
2023-08-07T16:38:10+0000 CRITICAL Error: No available modular metadata for modular package

3.  made change in below file 

vim /usr/lib/python3.9/site-packages/dnf/db/group.py

changed below :

    def _test_fail_safe(self, hdr, pkg):
        if pkg._from_cmdline:
            return 0
        if pkg.repo.module_hotfixes:
            return 0
        #try:
        #    if hdr['modularitylabel'] and not pkg._is_in_active_module():
        #        logger.critical(_("No available modular metadata for modular package '{}', "
        #                          "it cannot be installed on the system").format(pkg))
        #        return 1
        #except ValueError:
        #    return 0
        return 0


The solution is a hacky workaround and does not solve the problem (I would argue that it will make it worse). What you think is the problem isn’t actually the problem at all.

You need to look into how you have foreman configured to sync the repositories. There should be no reason that foreman (pulp) should be making any changes to the repository data as it is coming in. It should be a flat sync.

Thanks for suggestion. Considering this as workaround, will see how can we address it permanently. Meantime if anyone having same issue and if you found solution pls keep posted.