Risk Self Assessment
Every organization and AI implementation is unique. We developed this self-assessment for security practitioners, to help identify which AI risks may be most relevant to you.
Use this assessment to start conversations and guide further research.
Risk Self Assessment
This is a resource to help you understand which SAIF risks may be relevant to your organization.
It is for informational purposes only and does not constitute or replace professional advice. Your answers are not collected or shared. By continuing, you agree with the terms.
About this self-assessment
For informational purposes only
This self-assessment is intended to help you understand what AI security risks may be relevant to your organization. By using this self-assessment, you agree that this resource is for informational purposes only. Consult a security professional for advice. You're encouraged to talk to a security professional about what your results may mean for your organization. Google disclaims any liability, loss, or risk incurred as a consequence of using this resource.
Private and secure
Google does not collect or share answers or results from the self-assessment. We collect data only to analyze traffic, and we use that data only in ways that do not identify you and that follow our Privacy Policy.
Risk Self Assessment
Risk Assessment Questions
Which of the following best describes your organization’s use of Generative AI models? You may select more than one option.
Risk Self Assessment
Risk Assessment Questions
Do you have robust management of all training, tuning, or evaluation data used with your models to ensure that sensitive, unauthorized, or malicious data does not enter your models?
Risk Assessment Questions
Are you able to detect, remove, and remediate malicious or accidental changes in your training, tuning, or evaluation data?
Risk Assessment Questions
Is any sensitive user data used in training, tuning, or evaluating your AI models?
Risk Assessment Questions
Do you have robust management of all user data that results from your Generative AI applications to ensure that user data is stored, processed, and used in accordance with user consents and user policies?
Risk Assessment Questions
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Risk Assessment Questions
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Risk Assessment Questions
Are you able to ensure that all data, models, and code used to train, tune, or evaluate models cannot be tampered without detection during model development and during deployment?
Risk Assessment Questions
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Risk Assessment Questions
Do you protect your Generative AI applications and models against large-scale malicious queries from user accounts, devices, or via APIs?
Risk Assessment Questions
Are you using secure-by-default designs and coding frameworks in applications integrated with Generative AI applications?
Risk Assessment Questions
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Risk Assessment Questions
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
SAIF Risk Report
This report has been shared with you. Start over to create your own report.
Assessment & risks
Risks which are relevant to your organisation
Data Poisoning
Altering data sources used to train the model. In terms of impact, Data Poisoning is comparable to modifying the logic of an application to change its behavior.
The following controls may help mitigate this risk:
- Training Data Sanitization
- Secure-by-Default ML Tooling
- Model and Data Integrity Management
- Model and Data Access Control
Read more about the risk of Data Poisoning.
Are you able to detect, remove, and remediate malicious or accidental changes in your training, tuning, or evaluation data?
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are you able to ensure that all data, models, and code used to train, tune, or evaluate models cannot be tampered without detection during model development and during deployment?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Unauthorized Training Data
Using unauthorized data for model training. Using a model trained with Unauthorized Training Data might lead to legal or ethical challenges.
The following controls may help mitigate this risk:
Read more about the risk of Unauthorized Training Data.
Do you have robust management of all training, tuning, or evaluation data used with your models to ensure that sensitive, unauthorized, or malicious data does not enter your models?
Model Source Tampering
Tampering with the model's code or data. Model Source Tampering is similar to tampering with traditional software code, and can create vulnerabilities or unintended behavior.
The following controls may help mitigate this risk:
- Secure-by-Default ML Tooling
- Model and Data Integrity Management
- Model and Data Access Controls
- Model and Data Inventory Management
Read more about the risk of Model Source Tampering.
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are you able to ensure that all data, models, and code used to train, tune, or evaluate models cannot be tampered without detection during model development and during deployment?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Excessive Data Handling
Unauthorized collection, retention, processing, or sharing of user data. Excessive Data Handling may lead to policy and legal challenges.
The following controls may help mitigate this risk:
Read more about the risk of Excessive Data Handling.
Do you have robust management of all user data that results from your Generative AI applications to ensure that user data is stored, processed, and used in accordance with user consents and user policies?
Model Exfiltration
Theft of a model. Similar to stealing code, this threat has both intellectual property and security implications.
The following controls may help mitigate this risk:
- Model and Data Inventory Management
- Model and Data Access Controls
- Model and Data Integrity Management
- Secure-by-Default ML Tooling
Read more about the risk of Model Exfiltration.
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Model Deployment Tampering
Unauthorized changes to model deployment components. Model Deployment Tampering can result in changes to model behavior.
The following controls may help mitigate this risk:
Read more about the risk of Model Deployment Tampering.
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Denial of ML Service
Overloading ML systems with resource-intensive queries. Like traditional DoS attacks, Denial of ML Service can reduce availability of or entirely disrupt a service.
The following controls may help mitigate this risk:
Read more about the risk of Denial of ML Service.
Do you protect your Generative AI applications and models against large-scale malicious queries from user accounts, devices, or via APIs?
Model Reverse Engineering
Recreating a model by analyzing its inputs, outputs, and behaviors. A reverse engineer model can be used to create imitation products or adversarial attacks.
The following controls may help mitigate this risk:
Read more about the risk of Model Reverse Engineering.
Do you protect your Generative AI applications and models against large-scale malicious queries from user accounts, devices, or via APIs?
Insecure Integrated Component
Software vulnerabilities that can be leveraged to compromise AI models. Insecure Integrated Component can lead to privacy and security concerns, as well as potential ethical and legal challenges.
The following controls may help mitigate this risk:
Read more about the risk of Insecure Integrated Component.
Are you using secure-by-default designs and coding frameworks in applications integrated with Generative AI applications?
Prompt Injection
Tricking a model to run unintended commands. In terms of impact, Prompt Injection can change a model's behavior.
The following controls may help mitigate this risk:
Read more about the risk of Prompt Injection.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
Model Evasion
Changes to a prompt input to cause the model to produce incorrect inferences. Model Evasion can lead to reputational, legal, security, and privacy risks.
The following controls may help mitigate this risk:
Read more about the risk of Model Evasion.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Sensitive Data Disclosure
Disclosure of sensitive data by the model. Sensitive Data Disclosure poses a threat to user privacy, organizational reputation, and intellectual property.
The following controls may help mitigate this risk:
- Privacy Enhancing Technologies
- User Data Management
- Input and Output Validation and Sanitization
Read more about the risk of Sensitive Data Disclosure.
Is any sensitive user data used in training, tuning, or evaluating your AI models?
Do you have robust management of all user data that results from your Generative AI applications to ensure that user data is stored, processed, and used in accordance with user consents and user policies?
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
Inferred Sensitive Data
Model inferring personal information not contained in training data or inputs. Inferred Sensitive Data may be considered a data privacy incident.
The following controls may help mitigate this risk:
Read more about the risk of Inferred Sensitive Data.
Do you have robust management of all training, tuning, or evaluation data used with your models to ensure that sensitive, unauthorized, or malicious data does not enter your models?
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Insecure Model Output
Unvalidated model output passed to the end user. Insecure Model Output poses risks to organizational reputation, security, and user safety.
The following controls may help mitigate this risk:
Read more about the risk of Insecure Model Output.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Rogue Actions
Unintentional model-based actions executed via extensions. Rogue Actions can create a cascading, risk to organizational reputation, user trust, security, and safety.
The following controls may help mitigate this risk:
Read more about the risk of Rogue Actions.
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
Risks which you have mitigated
Data Poisoning
Altering data sources used to train the model. In terms of impact, Data Poisoning is comparable to modifying the logic of an application to change its behavior.
The following controls may help mitigate this risk:
- Training Data Sanitization
- Secure-by-Default ML Tooling
- Model and Data Integrity Management
- Model and Data Access Control
Read more about the risk of Data Poisoning.
Are you able to detect, remove, and remediate malicious or accidental changes in your training, tuning, or evaluation data?
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are you able to ensure that all data, models, and code used to train, tune, or evaluate models cannot be tampered without detection during model development and during deployment?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Unauthorized Training Data
Using unauthorized data for model training. Using a model trained with Unauthorized Training Data might lead to legal or ethical challenges.
The following controls may help mitigate this risk:
Read more about the risk of Unauthorized Training Data.
Do you have robust management of all training, tuning, or evaluation data used with your models to ensure that sensitive, unauthorized, or malicious data does not enter your models?
Model Source Tampering
Tampering with the model's code or data. Model Source Tampering is similar to tampering with traditional software code, and can create vulnerabilities or unintended behavior.
The following controls may help mitigate this risk:
- Secure-by-Default ML Tooling
- Model and Data Integrity Management
- Model and Data Access Controls
- Model and Data Inventory Management
Read more about the risk of Model Source Tampering.
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are you able to ensure that all data, models, and code used to train, tune, or evaluate models cannot be tampered without detection during model development and during deployment?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Excessive Data Handling
Unauthorized collection, retention, processing, or sharing of user data. Excessive Data Handling may lead to policy and legal challenges.
The following controls may help mitigate this risk:
Read more about the risk of Excessive Data Handling.
Do you have robust management of all user data that results from your Generative AI applications to ensure that user data is stored, processed, and used in accordance with user consents and user policies?
Model Exfiltration
Theft of a model. Similar to stealing code, this threat has both intellectual property and security implications.
The following controls may help mitigate this risk:
- Model and Data Inventory Management
- Model and Data Access Controls
- Model and Data Integrity Management
- Secure-by-Default ML Tooling
Read more about the risk of Model Exfiltration.
Do you have a complete inventory of all models, datasets (for training, tuning, or evaluation), and related ML artifacts (such as code)?
Do you have robust access controls on all models, datasets, and related ML artifacts to minimize, detect, and prevent unauthorized reading or copying?
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Model Deployment Tampering
Unauthorized changes to model deployment components. Model Deployment Tampering can result in changes to model behavior.
The following controls may help mitigate this risk:
Read more about the risk of Model Deployment Tampering.
Are the frameworks, libraries, software systems, and hardware components used in the development and deployment of your models analyzed for and protected against security vulnerabilities?
Denial of ML Service
Overloading ML systems with resource-intensive queries. Like traditional DoS attacks, Denial of ML Service can reduce availability of or entirely disrupt a service.
The following controls may help mitigate this risk:
Read more about the risk of Denial of ML Service.
Do you protect your Generative AI applications and models against large-scale malicious queries from user accounts, devices, or via APIs?
Model Reverse Engineering
Recreating a model by analyzing its inputs, outputs, and behaviors. A reverse engineer model can be used to create imitation products or adversarial attacks.
The following controls may help mitigate this risk:
Read more about the risk of Model Reverse Engineering.
Do you protect your Generative AI applications and models against large-scale malicious queries from user accounts, devices, or via APIs?
Insecure Integrated Component
Software vulnerabilities that can be leveraged to compromise AI models. Insecure Integrated Component can lead to privacy and security concerns, as well as potential ethical and legal challenges.
The following controls may help mitigate this risk:
Read more about the risk of Insecure Integrated Component.
Are you using secure-by-default designs and coding frameworks in applications integrated with Generative AI applications?
Prompt Injection
Tricking a model to run unintended commands. In terms of impact, Prompt Injection can change a model's behavior.
The following controls may help mitigate this risk:
Read more about the risk of Prompt Injection.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
Model Evasion
Changes to a prompt input to cause the model to produce incorrect inferences. Model Evasion can lead to reputational, legal, security, and privacy risks.
The following controls may help mitigate this risk:
Read more about the risk of Model Evasion.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Sensitive Data Disclosure
Disclosure of sensitive data by the model. Sensitive Data Disclosure poses a threat to user privacy, organizational reputation, and intellectual property.
The following controls may help mitigate this risk:
- Privacy Enhancing Technologies
- User Data Management
- Input and Output Validation and Sanitization
Read more about the risk of Sensitive Data Disclosure.
Is any sensitive user data used in training, tuning, or evaluating your AI models?
Do you have robust management of all user data that results from your Generative AI applications to ensure that user data is stored, processed, and used in accordance with user consents and user policies?
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
Inferred Sensitive Data
Model inferring personal information not contained in training data or inputs. Inferred Sensitive Data may be considered a data privacy incident.
The following controls may help mitigate this risk:
Read more about the risk of Inferred Sensitive Data.
Do you have robust management of all training, tuning, or evaluation data used with your models to ensure that sensitive, unauthorized, or malicious data does not enter your models?
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Insecure Model Output
Unvalidated model output passed to the end user. Insecure Model Output poses risks to organizational reputation, security, and user safety.
The following controls may help mitigate this risk:
Read more about the risk of Insecure Model Output.
Do you perform adversarial testing and training on models and Generative AI applications to improve resistance to adversarial inputs?
Rogue Actions
Unintentional model-based actions executed via extensions. Rogue Actions can create a cascading, risk to organizational reputation, user trust, security, and safety.
The following controls may help mitigate this risk:
Read more about the risk of Rogue Actions.
Do you build or deploy Generative AI powered agents or tools that can take actions on behalf of internal or external users?
