So i got this to work and can confirm the 2.9.1-4.el9_3.5 does seem to have issues with sssd and ssh. I can now connected all the time without intermitted issues such as authorisation not allowed. I used the oracle repository which seemed to have the sssd-2.9.1-4-el9_3.1.x86_64.
Only issue is I need to do a yum upgrade rather than downgrade and it gets installed.
Reason must be when the package is installed it shows up as 2.9.1-4.0.1.el9_3.1.x86_64
When you say its newer it has _3.1 and the other packages which has issues is _3.5 which I assumed is newer. In the actual repository it shows up as 2.9.1-4.el9_3.1 but when installed it has a -4.0.1.