Assess your AI

Description

Provide an initial description of the system, project overview, and points of contact who will be responsible for the audit or AI system. Note: You may want to include unique ID, tags or other project identifiers to facilitate identification and record keeping downstream.

1. Working name of AI system and owner/organization:
2. Briefly describe the AI system under consideration:
3. List team members, their roles and responsibilities:
4. Projected (or past) date(s) of AI system release:
5. Who is the point of contact for the AI project? Who is the point of contact for the Impact Assessment?

Context

Section 1.1. Intended purpose(s) and limitations

It is important to provide context about the specific, envisioned use cases for which the AI system is designed to be deployed. In the absence of advanced knowledge about all potential settings in which a system will be deployed, examination of the bounds of acceptable deployment is instructive to identify unanticipated or untested downstream uses.

1. Describe the AI system's task, purpose, minimum functionality, and the benefits it offers:
2. Can any applicable non-AI systems solve the same problem without AI? Can any non-technical procedural changes or adaptations help solve the same problem without the need for deploying an AI system? Explain if and why the AI system is preferred:
3. List the prospective users of the AI system and/or data subjects and their expectations for the AI system's use and limits:
4. Delineate the expected and acceptable AI system context of use, including: operational environment, user/operator characteristics, and social environment:
5. List the AI system's known limitations:
6. Describe the roles and responsibilities for human oversight of the deployed system:

Section 1.2. Interdisciplinary Collaborations

A team of AI actors with a diversity of experience, expertise, abilities and backgrounds, and with the resources and independence to engage in critical inquiry is critical to build a robust and comprehensive mapping system.

1. What interdisciplinary expert teams are you engaging to identify and manage risks in all stages of the AI life cycle? Have you included individuals with expertise in:
   System Design
   Engineering
   Ethics
   Sociology
   Psychology
   Privacy compliance
   Law and regulations
   Public policy
   Subject matter expertise for expected uses (domain experts)
2. Consider whether the teams responsible for developing and maintaining the AI system reflect diverse opinions, backgrounds, experiences, and perspectives. Identify gaps:
3. Review the demographics of those involved in the design and development of the AI system. This will help identify potential gaps in understanding and biases embedded during the development process.
4. Which communities and potential end users were consulted in the development? At which stages in the design and development process were they engaged?
5. Have stakeholders expressed potential negative impacts of the AI system? If so, how have you incorporated these concerns to address negative impacts?
6. What interdisciplinary expert teams are you engaging to identify and manage risks in all stages of the AI life cycle?

Section 1.3. The business value or context of business use

There are instances in which the AI solution is not the appropriate choice. The most significant example are situations where the AI solution would cause more harm than good. Another situation where the decision to terminate development could be appropriate is when AI systems do not present a business benefit beyond the status quo. Inherent risks and implicit or explicit costs should be weighed in the evaluation of whether an AI solution should be developed or deployed. Defining and documenting the specific business purpose of an AI system in a broader context of societal values helps teams evaluate risks and increases the clarity of “go/no-go” deployment decisions.

1. Have you reviewed the documented system's purpose from a socio-technical perspective, i.e., considering characteristics such as explainability, interpretability, privacy, safety, and bias management, and in consideration of societal values?
   Yes
   No
2. What potential latent business incentives may contribute to the AI system inflicting negative impacts? Note: possible misalignment between societal values, stated organizational principles and code of ethics, as compared to business incentives could result in negative societal impacts.

Section 1.4. Organization's mission and goals

By establishing comprehensive and explicit enumeration of AI system purpose and expectations, organizations can identify and manage socio-technical risks and benefits that could be supported or jeopardized by the AI system.

1. What goals and objectives does the organization expect to achieve by designing, developing, and/or deploying the AI system?
2. Review organization's stated values, mission statements, social responsibility commitments, and AI principles. Are there misalignments between organization’s goals and commitments and system’s purpose/context of use? Have you established the organization’s AI principles?
3. To what extent are the model outputs inconsistent with the values of fostering public trust? Broader equity?

Section 1.5. Organizational risk tolerances

Risk tolerance reflects the level and type of risk the organization will accept while conducting its mission and functions. Deployment decisions should be the outcome of a clearly defined process that is reflective of an organization’s values, including its risk tolerances. Go/no-go decisions should be incorporated throughout the AI system’s lifecycle. For systems deemed “higher risk,” such decisions should include approval from sufficiently senior technical or otherwise specialized or empowered executives (for more information on risk tolerance, see NIST AI RMF document Section 3.2.2. Risk Tolerance).

1. Provide justifications for the assumptions, boundaries, and limitations of the AI system including wh the system should (or should not) be deployed based on its limitations, boundaries, or assumptions.
2. What are the maximum allowable risk thresholds above which the system will not be deployed or will be prematurely decommissioned?
3. Establish risk tolerance levels for the AI system and allocate the appropriate oversight resources and authorities/ supervisors to each level:

Classification

Section 2.1. Learning Task

AI actors should define the technical learning or decision-making task that an AI system is designed to accomplish.

1. Which category of learning tasks does the AI system support?
   Computer Vision
   Natural Language Processing (NLP)
   Recommender system
   Classification system
   Image or text generation
   Other:
2. Have you defined technical specifications and requirements for the AI system? Provide a link to, or attach, the technical specifications document.
3. Have you documented the AI system's development, testing methodology, metrics, and performance outcomes? Provide a link to, or attach, the documentation.
4. How do the technical specifications and requirements align with the AI system’s goals and objectives?
5. Did your organization implement accountability-based practices in data management and protection and, if so, which standard(s) (e.g. the PDPA and OECD Privacy Principles)?
6. What assessments has your organization conducted on data security and privacy impacts associated with the AI system? Consult NIST privacy and cybersecurity frameworks for a comprehensive assessment of these risk areas.
7. How are outputs marked to clearly show that they came from an AI system?

2.2. Operational Context

Once deployed and in use, there are times when AI systems can perform poorly, manifest unanticipated negative impacts, or violate legal or ethical norms. Human oversight and stakeholder engagement can provide important contextual awareness.

1. Risks can arise from deploying a system in an environment that differs from the original controlled or envisioned environment. Identify potential risks due to unanticipated or unintended deployment contexts or human-AI configurations:
2. What dependencies does the AI system have on upstream data and other AI systems? (See Section 2.3. and Section 5).
3. Does the AI system have connections to external networks (including the internet), financial markets, and critical infrastructure that have potential for negative externalities?
   Yes
   No

2.3. Data Collection and Selection

Many AI system risks and vulnerabilities can be traced to insufficient testing and evaluation processes as well as oversight in data collection and curation.

1. How was the data collected, cleaned, and curated? Who was involved in the data collection process?
2. What, if any, are known errors, sources of noise, or redundancies in the data?
3. Over what time-frame was the data collected?
4. [After extended use(s)] Is the training and testing data still representative of the current operational environment(s)?
   Yes
   No
5. What is the variable selection and evaluation process?
6. If the dataset relates to, or derives from individuals (e.g., their attributes) were they informed about the data collection?
   Yes
   No
7. Is the training and testing data representative of the demographic population with which the AI system will be used (e.g., age, gender, race, etc.)?
   Yes
   No
8. Do you know why the dataset was created? (e.g., what was the specific need or utility that it was created to fulfill?)
9. How do you ensure that the collected data are adequate, relevant, and limited to what is necessary in relation to the purposes for which they are processed (‘data minimization’)? See Article 5 of GDPR for more information.
10. Do the data collection processes adhere to organization policies related to bias, privacy and security for AI systems?
    Yes
    No
11. Do the data collection processes comply with relevant legal or regulatory requirements applicable to data or AI systems? (See Section 6 on Legal)
    Yes
    No

Bias

Trustworthy and Responsible AI has been evaluated to determine whether a given AI system is biased, fair and does what it claims to accomplish. Processes to ensure trustworthy and responsible AI often focus on computational factors such as representativeness of datasets and fairness of machine learning algorithms, which indeed are vital for mitigating bias. However, a robust process must include considerations of human, systemic, institutional and societal factors that can present significant sources of AI bias as well. An effective governance system requires expanding the perspective beyond the machine learning pipeline to recognize and investigate how this technology is both created within and impacts our society.

Understanding AI as a socio-technical system acknowledges that the processes used to develop technology are more than their mathematical and computational constructs. A socio-technical approach to AI takes into account the values and behavior embedded in the datasets, the humans who interact with the AI systems, and numerous other factors that go into the design, development, and deployment of that system.

The importance of engaging in an evaluation process considering the transparency, datasets, and test, evaluation, validation, and verification (TEVV) cannot be overstated. Participatory design techniques and multi-stakeholder approaches, as well as human-in-the-loop processes are also important steps to help mitigate risks related to AI bias. However none of these practices provide a panacea against the introduction or scaling of bias in AI systems, and each can introduce potential pitfalls. It is critical to recognize the reality that it is not possible to achieve zero risk of bias in an AI system. Rather, we create and adopt AI governance to enable better identification, understanding, management, and reduction of bias, as well as other potential harms.

This section incorporates elements of SP1270 NIST taxonomy of bias in AI to help organizations initiate a probe for common types of biases that are likely to occur in the AI systems outcomes. Fairness evaluation and remediation is a fast-evolving, interdisciplinary field of research, which requires a variety of perspectives from different fields. Engaging a social scientist, an AI fairness specialist or similar an individual with expertise in this area could help facilitate responses to the questions in this section.

The NIST Special Publication on Bias categorizes AI biases into three categories:

3.1. Pre deployment check

1. Are the users of the AI system properly trained to interpret AI model output and decisions? Are the staff developing the AI system trained to detect and manage bias in data?
   Yes
   No
2. Statistical and Computational biases stem from errors that result when a sample is not representative of the population. Consider whether your AI system should be tested for one or more of the following statistical/computational biases:
   Sampling bias: Proper randomization is not used during data collection.
   Measurement bias: arises when features and labels are proxies for desired quantities.
   Model selection bias: introduced while using the data to select a single seemingly “best” model from a large set of potential models.
   Activity bias: a type of selection bias that occurs when systems get their training data from their most active users.
   Temporal bias: arises from differences in populations and behaviors over time.
   Automation bias: tendency to favor results generated by automated systems over those generated by non-automated systems, regardless of accuracy.
   Other (refer to enumerations in Pg. 8 of NIST Special Publication or see some examples here):
3. Systemic biases affect how organizations and teams are structured and who controls the decision making processes, which can result in certain social groups being advantaged or favored. Which types of systemic biases should your system be tested for?
   historical
   societal
   institutional
4. Human Cognitive biases reflect systematic errors in human thought based on a limited number of heuristic principles and predicting values to simpler judgmental operations. These biases are often implicit and tend to relate to how an individual or group perceives information (such as automated AI output) to make a decision or fill in missing or unknown information. Consider which types of human bias should your system be tested for, including:
   Availability: overly weighting a resource or response that comes easily or quickly to mind, e.g., which datasets are readily available.
   Funding bias: biased results are reported to satisfy the needs or interests of the funding agency, individual, or organization.
   Groupthink: individuals in a group can slant decisions toward a desire to conform to the group, or fear of dissenting with the group.
   Other (refer to enumerations in Pg. 8 of NIST Special Publication):
5. Could the AI system unfairly advantage or disadvantage a particular social group?
6. Identify other sources of biases in your AI system that could lead to inequitable or discriminatory outcomes? (See NIST SP1270 Glossary, pg 58) Identify ways to reduce such biases:
7. List other potential impacts of bias from use of the AI system, including stereotyping or offensive outcomes for certain demographic groups:
8. What tools are available (and applicable to your system) for identifying and mitigating AI bias that you have identified above as a potential risk?
9. What procedures and mitigations have you put in place to reduce harmful outcomes of AI bias that you have identified above as a potential risk?

3.2. Post deployment check: Are additional or newly established procedures necessary to continue mitigating bias or inequity?

1. Which types of bias will you test for and on what cadence going forward?
2. Have rechecks continued within stated time frames?
   Yes
   No
3. How are you documenting checks (e.g, is there an owner who is responsible in your c-suite, is this process overseen by your office of general counsel, where are testing results stored and who is in charge of verifying results for safe continued use of the AI system?)

Cost/Benefits Evaluations

4.1. System Benefits

AI systems should be checked periodically to ensure benefits outweigh inherent risks as well as implicit and explicit related costs. To identify system benefits, organizations should define and document system purpose and utility, along with foreseeable costs, risks, and negative impacts.

1. What benefits does the AI system offer?
2. How are you quantifying and measuring AI costs and benefits:
3. Have the benefits of the AI system been communicated to users? How?
   Yes
   No
   Describe the risk management plan:
4. Who is responsible for oversight of the risk management system?
5. Describe the process for collecting and incorporating stakeholder feedback on perceived system benefits:
6. Have any employees or users raised concerns about the systems’ safety or inclusivity, or other potentially negative impacts arising from the AI system?
   Yes
   No
   If so, what concerns have been raised and by whom (user/stakeholder/role within company)? What steps have been taken to remedy the negative impact?

4.2. Potential Costs

Negative impacts can be due to many factors, such as poor system performance, and may range from minor annoyance to serious injury, financial losses, or regulatory enforcement actions.

1. What constitutes an internal or external failure for this AI system? Are there various levels of failure?
2. Describe procedures for regularly evaluating the qualitative and quantitative costs of internal and external AI system failures:
3. Can users or parties affected by the outputs of the AI system test the AI system and provide feedback?
   Yes
   No

4.3. Application Scope

Systems that function in a narrow scope tend to enable better mapping, measurement, and management of risks in the learning or decision-making tasks and the system context. Areas that help narrow contexts for system deployment include:

1. Are there areas where the narrowing of application scope could help reduce risks exhibited by the AI system?
   Yes
   No
2. Have you consulted with experts (e.g., legal and procurement experts) to identify whether the application scope needs to be further narrowed or refined?
   Yes
   No

Third-party Technologies

5.1. Third-party risks

Technologies, such as pre-trained models, and personnel from third-parties are another source of risk to consider during AI risk management activities.

1. Did you acquire datasets from a third party for this AI system?
   Yes
   No
2. Did you assess and manage the risks of using third-party datasets consistent with the process above?
   Yes
   No
3. To what extent were you unable to answer key questions about potential risks:
4. Did you acquire third-party material (open-source software, pre-trained models, open-source datasets, etc.) for developing the AI system?
   Yes
   No
5. Evaluate the risks associated with third-party material inventoried above. For example, pretrained models could carry historical biases as they are trained on internet-scraped data, or certain demographic groups could be under-represented in an open-source dataset.

5.2. Controls for third-party risks

AI actors often utilize open-source software, freely available datasets, or third-party technologies—some of which have been reported to have privacy, bias, and security risks.

1. Have you applied controls—such as procurement, security, and data privacy controls—to all acquired third-party technologies, for example, when procuring a third-party AI model?
   Yes
   No
2. Have you reviewed any audit reports, testing results, product roadmaps, warranties, terms of service, end-user license agreements, contracts, model/system cards, or other documentation available for third-party resources used in your AI system? For example, if you are using models such as DALL-E 2 have you reviewed its system card?
   Yes
   No

Legal and Compliance

In addition to upcoming legal developments, such as the EU AI Act, there are numerous Federal and state laws in the US and across the globe currently on the books that are or could be applicable to AI systems. Legal professionals should review and assess AI systems for legal risks and identify mitigations at the earliest stages and throughout the lifecycle and assessment processes.

1. Have you considered whether General Data Protection Regulation (GDPR) or related U.S. state privacy laws (e.g., California Privacy Rights Act) apply to your system?
   Yes
   No
2. Have you undertaken a privacy review?
   Yes
   No
3. Does your system process personal data?
   Yes
   No
4. Have you considered whether data used to train AI contains personal identifying information (PII) or other data that could create liability?
   Yes
   No
5. Have you considered whether your system accesses private data after deployment?
   Yes
   No
6. If applicable to your system, have you established appropriate mechanisms to notify the user (data subject) about their data being collected and to obtain required consent? (For an example see GDPR requirements of notice and consent)
   Yes
   No
7. Does your system use facial recognition technology that could implicate state regulations (e.g., Washington Facial Recognition legislation SB2856)?
   Yes
   No
8. Does your system use biometric information that could implicate regulations (e.g., Illinois Biometric Information Privacy Act)?
   Yes
   No
9. Have you checked whether your AI system has biases that could result in discriminatory outcomes in violation of civil rights laws (e.g., Americans with Disabilities Act (ADA), HIPAA, Title VII of Civil Rights Act, etc.)?
   Yes
   No
10. Have you checked whether your AI system exhibits unfair or deceptive practices in violation of Section 5 of the FTC Act or results in a discriminatory act that violates the Fair Credit Reporting Act or Equal Credit Opportunity Act?
    Yes
    No
11. Have you consulted legal counsel on compliance with other applicable laws (e.g., privacy, security, intellectual property, discrimination, contract, tort liability) and whether the AI system could be used in a regulated domain, such as healthcare or finance?
    Yes
    No
12. What other legal system (global, federal, state) could be implicated by the AI development or use:
    Yes
    No

Final Decision

7.1. Identifying Impacts

Risk assessment enables organizations to create a baseline for system monitoring and to increase opportunities for detecting emergent risks.

1. What stakeholder engagement processes have you established to identify potential impacts from the AI system on individuals, groups, communities, organizations, and society?
2. Identify misalignments between organizational or societal values and system implementation and impact:
3. Are there systems or mechanisms in place to ensure continuous monitoring for impacts and emergent risks?
   Yes
   No
4. If the AI system relates to people, could it expose people to harm or legal action (e.g., financial, social or otherwise)? What was done to mitigate or reduce the potential for harm?
5. If the AI system relates to subjects protected by international standards or bodies, have appropriate obligations been met (e.g., medical data might include information collected from animals)?
   Yes
   No

7.2. Likelihood and Magnitude of Impact

If an organization decides to proceed with deploying the system, the ‘likelihood estimate’ can be used to assign oversight resources appropriate for the risk level and triage the likelihood of the system’s impacts. A ‘likelihood estimate’ includes:

1. Describe how you measure AI system impact. For example, you can establish qualitative assessment scales, such as red-amber-green, as well as simulations or econometric approaches. Apply scales uniformly across the organization’s AI portfolio.
2. Identify the likelihood and magnitude of each identified impact based on expected use, past uses of AI systems in similar contexts, public incident reports, stakeholder feedback, or other data are identified and documented.

7.3. Final Decision

The final decision on go/no go should take into account the risks mapped from previous steps and the organizational capacity for their management.

1. Review and examine documentation, including system purpose and benefits, and mapped potential impacts with associated magnitude and likelihoods.
2. Document a summary of the system’s estimated risk.
3. Document why a go/no-go determination was made based on magnitude and likelihood of impact and estimated risk. If a decision is made to proceed, assign the system to an appropriate risk tolerance and align oversight resources with the assessed risk.