Validating Compliance AI Systems

Although we are probably currently at the Peak of Inflated Expectations with respect to the use of Generative AI in corporate and compliance settings and the Trough of Disillusionment is yet to come, in the not-TOO-distant future AI tools will likely become ubiquitous in corporate compliance settings, helping automate and scale compliance capabilities, assisting human decision makers on compliance matters, and overall reducing the risk of non-compliance.

However, with increasing reliance on AI systems and the critical stakes presented by many compliance issues, it will be absolutely crucial to validate the reliability and accuracy (among other things) of such AI systems to ensure that they are making decisions and/or advising in an accurate, reliable, explainable, and safe way.

So how should such AI systems be put through their paces? NIST has recently published the first version of its AI Risk Management Framework, which is necessary reading for anyone working on or looking to implement such systems. Although, like many NIST frameworks, it can be initially daunting to consume, within it is a very pithy and helpful articulation of the characteristics of a “Trustworthy AI System”:

Keeping these characteristics in mind and pulling on other ideas articulated in the NIST AI RMF, other NIST publications, as well as my own experience as an engineer, lawyer, compliance professional, and auditor, I posit that personnel tasked with validating and auditing AI systems (vendors, procurement personnel, IT, internal risk management, external auditors, investigators, etc.) will need to review, test, and generally keep the following in mind.

This is of course a first pass and I will likely return to this in the future to further refine and expand. Welcome your thoughts on what else might need to be included (or removed) from this list.

1. Governance

• Are there policies, procedures, work instructions, etc. regarding the scope, usage, maintenance, and governance of the AI system?
• Is there a human governance framework in place that has ultimate responsibility and authority with respect to the AI system in question?
• Is there a backup system/plan in place to continue operations if the AI system/capabilities have to be paused?
• Is the business risk and impact of pausing/pulling AI capabilities being deployed well understood?
• Is there a risk-aware, appropriately scoped, and vetted incident response plan in place in case something goes wrong?
• Has the company implemented a relevant risk management framework (e.g., NIST AI RMF) to govern its deployment and usage of AI technologies?

2. System and Data Acquisition

• Are the algorithms and models being used by the AI system known and subject to academic study? Are the strengths and shortcomings of such algorithms and models known and factored into system design and usage?
• Is the provenance of the training data well understood? Is all training data owned by the company or sourced from external sources?
• Has training data been vetted for:

i.    Ownership and other IP/legal risks?

ii.    Data privacy and authorized usage risks?

• Is liability and risk associated with the use of the AI system appropriate allocated between the using organization and the vendor?

3. Design & Initialization

• Has the scope of the AI system been documented and approved? What functions will the AI system support? What are the use cases? What are the boundaries?
• Has the usage of the AI system been evaluated by relevant compliance functions? E.g., Data privacy, international trade, HR. Have some compliance programs been updated to address relevant implications of AI system usage?
• Has the IT infrastructure hosting the AI system been vetted for data privacy, export control, and other issues that may restrict cross-border data transfers?
• How are critical emerging regulatory or business changes that may impact AI function handled from a design perspective?

4. AI Training

• Was the training data vetted for

i.    Accuracy and completeness?

ii.    Bias, when relevant?

• If the training data is from internal sources, is the ETL process for the data fully documented?
• What testing was initially done on the system to validate proper functioning?
• Is the accuracy / success rate of the system well understood? Has it been reviewed through appropriate governance channels?

5. Human Training

• Are individuals that may use or interact with the system appropriately trained on

i.    Appropriate usage?

ii.    How to recognize and respond to erroneous outcomes?

• Are relevant individuals trained on contingency procedures in case the AI system has to be brought offline?

6. Recordkeeping and Explainability

• Does the AI system keep human-readable records that give insight into / explain decision-making process / factors?
• Are there records related to model training?
• Are there records related to (and copies of) the data used to train the AI model/tool?
• Are there records related to cases where the AI system took erroneous actions / gave erroneous advice?
• Are these records stored centrally and access-controlled?
• Are these records kept in compliance with larger corporate-side recordkeeping policy?

7. System Maintenance & Update

• How often and how is the system retrained on new data? How is such data identified?
• Is responsibility for system maintenance and update clearly identified and allocated?
• Are there policies and procedures governing system maintenance and update?

8. Monitoring, Investigations, and Auditing

• Are there (automated) mechanisms in place to identify when the AI system may be providing erroneous responses or taking erroneous actions?
• Are there channels for system users and others to report issues related with the AI system?
• Are there policies and procedures in place regarding how such issues are:

i.    investigated and remediated?

ii.    Escalated up the governance chain?

iii.    Notified to internal compliance and legal, when appropriate?

• Are there regularly scheduled audits of the system? Have prior findings been fully addressed?

9. Validation and Continuous Improvement

AI validation should be an ongoing process rather than a one-time test. Organizations should regularly evaluate accuracy, consistency, false positives, false negatives, and the system’s ability to handle unusual or changing compliance scenarios.

Validation results should be documented and reviewed by appropriate compliance, legal, IT, or audit personnel. When significant errors are identified, organizations should investigate the cause, apply corrective measures, and determine whether additional human oversight is required.

As models, data, regulations, and business processes change, AI systems should be retested to ensure they continue to operate within their approved scope. Continuous validation helps organizations maintain confidence in AI while reducing the risk that automation introduces new compliance failures.