Freeipa schema issue and replication

I inherited a freeipa cluster, hand cranked and all of that. for some reason, wrong schema was replicated from a bad server that I was trying to add to cluster using ipa-replica-install to working ones. 10% of ipa server are left and I am afraid i may loose them. the freeipa server upgrade was done with --skip-version-check. some server have 4.9.13-12.module+el8, other 4.9.13-18.module+el8. the following is a snippet of multipl error lines

ERR - NSACLPlugin - __aclp__init_targetattr - targetattr “ipauserdefaultsubordinateid” does not exist in schema. Please add attributeTypes “ipauserdefaultsubordinateid” to schema if necessary.
ERR - NSACLPlugin - acllist_insert_aci_needsLock_ext - ACL PARSE ERR(rv=-5): (targetattr = "cn
ERR - NSACLPlugin - __aclinit_handler - This ((targetattr = “cn || createtimestamp || entryusn || ipacertificatesubjectbase || ipaconfigstring || ipacustomfields || ipadefaultemaildomain || ipadefaultloginshell || ipadefaultprimarygroup || ipadomainresolutionorder || ipagroupobjectclasses || ipagroupsearchfields || ipahomesrootdir || ipakrbauthzdata || ipamaxhostnamelength || ipamaxusernamelength || ipamigrationenabled || ipapwdexpadvnotify || ipasearchrecordslimit || ipasearchtimelimit || ipaselinuxusermapdefault || ipaselinuxusermaporder || ipauserauthtype || ipauserdefaultsubordinateid || ipauserobjectclasses || ipausersearchfields || modifytimestamp || objectclass”)(targetfilter = “(objectclass=ipaguiconfig)”)(version 3.0;acl “permission:System: Read Global Configuration”;allow (compare,read,search) userdn = “ldap:///all”;)) ACL will not be considered for evaluation because of syntax errors.
ERR - NSACLPlugin - __aclp__init_targetattr - targetattr “ipaautoprivategroups” does not exist in schema. Please add attributeTypes “ipaautoprivategroups” to schema if necessary.
ERR - NSACLPlugin - acllist_insert_aci_needsLock_ext - ACL PARSE ERR(rv=-5): (targetattr = "cn
ERR - NSACLPlugin - __aclinit_handler - This ((targetattr = “cn || createtimestamp || entryusn || ipaautoprivategroups || ipabaseid || ipabaserid || ipaidrangesize || ipanttrusteddomainsid || iparangetype || ipasecondarybaserid || modifytimestamp || objectclass”)(targetfilter = “(objectclass=ipaidrange)”)(version 3.0;acl “permission:System: Read ID Ranges”;allow (compare,read,search) userdn = “ldap:///all”;)) ACL will not be considered for evaluation because of syntax errors.
WARN - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
INFO - slapi_vattrspi_regattr - Because krbPwdPolicyReference is a new registered virtual attribute , nsslapd-ignore-virtual-attrs was set to ‘off’
ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=example,dc=com–no CoS Templates found, which should be added before the CoS Definition.

i am not sure where to began, i am kind of lost. help is appreciated.

You knew there were bad domain controllers mixed with good ones, why didn’t you remove the bad ones from the domain first?

Why did you use this switch? There are very, very few edge cases you should be using this.

What versions of IPA are you actually running on your domain controllers? Meaning, all of them. How different are they? What are the actual steps you took to get to this point? Without much to go off of, this is looking like a situation that you need to start fresh or potentially migrate data to a new domain.

You may also want to consider asking this question on the freeipa-users mail list.

I did not know it. I assumed things were working fine.

it was not me who did the upgrades. It was mentioned to me by my colleague.

clusters are mixed 4.9.13-12 and 4.9.13-18, if this is what you are asking about.

one machine had expired certificates and certmonger renewal did not do anything. so I want to create new replica so that i pulls the data and renew certs, etc. but cert_request got reject, so I wanted to check if the master has updated certs. I ran ipa-certupdate on the master and so that the command got suspended. after checking dirsrv service, I so the schema output. then it spread through multiple replicas.