My current mission leads me to work on AI systems security and in particular generative AI.
With any new technology comes its share of opportunities and risks, Generative AI is not without risk.
I have already mentioned in previous posts about possible data leaks when companies don't make the effort to provide their employees with internal chatbots, thus they use public chatbots (see the notion of shadow AI in the RAG blog).
Of course, ANSSI is the French reference in cyber security and Generative AI could not escape its vigilance. The organization recently published a set of recommendations that covers the risk of data leakage and many more.
ANSSI also cosigned a document with NSCS if you are looking for English details. Otherwise I have translated the ANSSI recommendations for you.
The document provides an overview of technology, attack vectors and risk mitigation approaches in the form of detailed recommendations.
It's very interesting and easy to read but it's 38 pages long. I therefore offer you this summary, starting with a summary table of these ANSSI recommendations. This can be used as a checklist when reviewing the risks of your AI systems.
Summary table of Generative AI security recommendations
R1 | |
R2 | |
R3 | |
R4 | |
R5 | |
R6 | |
R7 | |
R8 | |
R9 | |
R10 | |
R11 | |
R12 | |
R13 | |
R14 | |
R15 | |
R16 | |
R17 | |
R18 | |
R19 | |
R20 | |
R21 | |
R22 | |
R23 | |
R24 | |
R25 | |
R26 | |
R27 | |
R28 | |
R29 | |
R30 | |
R31 | |
R32 | |
R33 | |
R34 | |
R35 |
And to better understand this table, I also share with you the excerpts of these recommendations that the ANSSI has established to help you set up a governance of your AI systems adapted to your risks.
Generative AI system Lifecycle
R1 Embedding security in all phases of an AI system's lifecycle
Security measures must be identified and applied in each of the 3 phases of an AI system's lifecycle: training, deployment, and production.
These measures depend heavily on the shared responsibility scenario chosen and the associated subcontracting. They must also take into account interactions with other applications or components internal or external to the IS.
It is possible to refer to the ANSSI hygiene guide to have a basic safety basis to apply.
Generative AI attack scenarios
R2
R2 Conduct a risk analysis on AI systems before the training phase
The risk analysis of an AI system must integrate the following items :
Map all the elements related to the AI model: third-party libraries, data sources, interconnected applications, etc. ;
identify the sub-parts of the AI system that will process the organization's data, especially those contained in user queries ;
take into account the scenario of sharing responsibilities and the issue of subcontracting for each of the phases ;
identify the direct and indirect impacts of erroneous or malicious responses from the AI model to users ;
consider the protection of the AI model's training data.
General recommendations
R3
R3 Assess the confidence level of the libraries and plug-ins used
It is recommended that you map all libraries and plug-ins used in the project and assess their confidence level.
R4
R4 Assess the level of confidence of external data sources used in the AI system
It is recommended to map all external data sources used in the project and assess their level of confidence.
R5
R5 Apply DevSecOps principles across all phases of the project
It is recommended that you apply secure development best practices across all phases of the project, for example :
Deploy and secure continuous integration and deployment (CI/CD) chains by enforcing the principle of least privilege access to tools in those CI/CD chains ;
Implement secure management of secrets used in all phases of the project ;
Plan for automated security tests on the source code (static code analysis) and during the execution of this source code (dynamic code analysis) ;
Protect the integrity of the source code and secure its access (multi-factor authentication, code signing, access rights, etc.) ;
Use secure development languages ( fine tuning scripts, model development, maintenance, deployment, etc.).
6
R6 Use secure AI model formats
It is recommended to use state-of-the-art formats from a security point of view, such as the safetensor format for example. Some insecure formats, such as pickle, should be avoided.
R7
R7 Take data privacy issues into consideration from the design of the AI system
The project study must map all the datasets used in each phase of the AI system: training (training datasets), deployment (test sets) and production (additional data, vector database, etc.).
This study must include the usage data of the AI system in production, i.e. the user queries as well as the answers provided by the AI model.
The analysis can also deal with the case of the confidentiality protection of the parameters of the model itself, for example for proprietary models.
R8
R8 Factor in the need-to-know basis from the design of the AI system
It is important to define the structuring options of the project model upstream to manage the need to know basis :
the choice of data used for training (without the possibility of managing access rights) and additional data in production (with the possibility of managing roles and access rights) ;
the model training strategy, i.e. when the model is retrained and on the basis of which data (additional business data, user queries, model responses, etc.).
R9
R9 Prohibit the automated use of AI systems for critical actions on the IS
An AI system must be configured in such a way that it is not able to automatically execute critical actions on the IS.
These actions can be business-critical actions (banking transactions, production of public content, direct impact on people, etc.) or critical actions on the IS infrastructure (reconfiguration of network components, creation of privileged users, deployment of virtual servers, etc.)..
R10
R10 Control and secure privileged access for AI system developers and administrators
All privileged operations on the AI system must adhere to secure administration best practices, including: :
Privileged operations must be defined and their triggering must be validated: re-training, modification of datasets, new interconnection with an application, change of hosting, etc. ;
Privileged operations must be carried out with dedicated accounts and from an administration station dedicated to this use ;
The principle of least privilege must be applied and the use of temporary authentication tokens must be preferred ;
The development environment must be controlled and administered with the same level of security as the production environment.
R11
R11 Host the AI system in trusted environments consistent with security needs
The hosting of the AI system during the 3 phases of the life cycle must be consistent with the security needs of the project, and in particular the needs of confidentiality and integrity. In particular, securing the model's training data (at rest, in transit, during processing) should not be neglected.
R12
R12 Compartmentalize each phase of the AI system in a dedicated environment
It is recommended to compartmentalize the 3 technical environments corresponding to each of the phases of the AI system's life cycle. This compartmentalization can relate to :
Network partitioning: each environment is integrated into a physically or logically dedicated network ;
System partitioning: Each environment has its own dedicated physical servers or hypervisors ;
Storage partitioning: Each environment has its own storage hardware or dedicated disks. At the very least, logical compartmentalization is applied ;
Silos of accounts and secrets: Each environment has its own user and administrator accounts and separate secrets.
R13
R13 Implement a secure internet gateway in the case of an AI system exposed on the internet
In the case of an AI system exposed on the Internet, it is recommended to follow the best practices for compartmentalization in the ANSSI guide on this subject, in particular :
dedicate a reverse-proxy function before accessing the AI system's web service ;
Implement two logical zones for network filtering using firewalls: external filtering on the Internet front-end and internal filtering before access to the AI system ;
not expose an internal directory of the entity for authentication on the AI system ;
avoid pooling security functions that are separate from the secure Internet gateway (firewalls, reverse-proxy, logging server, etc.) on the same hypervisor.
R14
R14 Favor SecNumCloud hosting when deploying an AI system in a public cloud
If the entity chooses to use hosting in a public cloud, it is recommended to favor a SecNumCloud trusted offer in the following cases :
the data processed by the AI system is considered sensitive ;
the impact of the AI system on the business is considered critical ;
users of the AI system are not considered trusted.
R15
R15 Prepare a degraded mode of business services that can run without the AI system
In order to prevent malfunctions or inconsistencies in the responses provided by the AI model, it is recommended to provide at least one procedure for bypassing the AI system for users, in order to meet business needs.
R16
R16 Have dedicated GPU components for the AI system
It is recommended to dedicate the physical GPU components to the processing performed by the AI system. In the case of virtualization, it is recommended that the hypervisors with access to the GPU cards be dedicated to the AI system, or at least that there be a hardware filtering function (e.g., IOMMU) to restrict the access of virtual machines to the memory of these GPU cards.
R17
R17 Take into account possible side-channel attacks on the AI system
It is recommended to ensure that the AI system is not vulnerable to side-channel attacks (temporal, consumption, etc.) that could, for example, allow an attacker to reconstruct a response provided by an AI model.
Recommendations for the training phase
R18
R18 Only train an AI model with data that was legitimately accessed by users
It is strongly recommended to train a model with data whose sensitivity is consistent with the need to know basis of its users.
R19
R19 Protect the AI model's training data integrity
It is recommended that the integrity of the model's training data be ensured throughout the training cycle. This protection can take the form of a systematic verification of the signature or hash of the files used or compressed archives of all this data.
R20
R20 Protect the AI system files integrity
It is recommended to protect the integrity of the files of the trained model, and to regularly check that they have not been altered. The recommendation also applies by extension to all files inherent to the operation of the AI system (scripts, binaries, etc.).
R21
R21 Prohibit the re-training of the AI model in production
It is strongly recommended not to re-train an AI model directly in production. This re-training action must start with the 3-phase cycle, in the appropriate environments for each of the phases.
R22
Recommendations for the deployment phase
R22 Secure the AI systems production deployment chain
It is recommended to deploy generative AI systems from an administrative IS, in accordance with the best practices of the ANSSI's secure administration guide.
R23
R23 Schedule security audits of AI systems before deployment in production
It is recommended to provide robustness and security tests for AI systems.
These tests can be :
Standard penetration tests on the usual technical components of an AI system: web servers, orchestrator, database, etc.
security tests on developments made in the AI system (via SAST or DAST tools for example) ;
automated testing specifically targeting vulnerabilities related to AI models (adversarial attacks, model extraction, etc.) ;
Manual auditors testing specifically the robustness of a generative AI model on more sophisticated attack scenarios.
R24
R24 Plan business functional tests of AI systems before deployment in production
It is recommended to provide performance and quality tests of the answers provided by a generative AI system.
Recommendations for the production phase
R25 Protect the AI system by filtering user inputs and outputs
It is recommended to set up functions to protect against a data leak or a model leak in the responses :
A function to filter malicious user requests before sending to the model ;
A function to filter queries that are deemed not legitimate from a business point of view ;
a filter function of the responses that could leak model internal information (parameters, training) ;
a filter function for information defined as sensitive in the responses (e.g., personal contact information, project references, etc.) ;
a limit on the size of the answers (maximum number of characters).
R26
R26 Control and secure the interactions of the AI system with other business applications
All interactions and network flows of the AI system must be documented and validated. Network flows between the AI system and other resources must adhere to state-of-the-art security :
they must be strictly filtered at the network level, encrypted and authenticated (e.g., by following the ANSSI TLS guide);
they must use secure protocols (e.g. OpenID Connect) when using an identity provider;
They must include a control of access permissions to the resource in addition to authentication ;
They should be logged at the appropriate level of granularity.
R27
R27 Limit automatic actions from an AI system processing uncontrolled inputs
It is strongly recommended to limit or even prohibit automatic actions on the IS triggered from an AI system and from uncontrolled inputs (e.g. data from the Internet or emails, etc.).
R28
R28 Compartmentalize the AI system in one or more dedicated technical environments
It is recommended that the AI system be partitioned into dedicated logical zones, in order to limit the risk of lateralization of an attacker who has compromised this system.
R29
R29 Log all processing performed within the AI system
It is recommended to log all the processing performed on the AI system at the right level of granularity, in particular :
user requests (taking care to protect them if those requests contain sensitive data) ;
the input processing performed on this query before sending it to the model ;
Calls to plugins ;
calls for additional data ;
the processing carried out by the output filters;
User responses.
Special case of AI-assisted source code generation
R30
R30 Systematically control AI-generated source code
AI-generated source code must be subject to security measures in order to verify its safety :
Prohibit the automatic execution of AI-generated source code in the environment development;
Prohibit automatic commit of AI-generated source code in repositories ;
integrate an AI-generated source code sanitization tool into the environment development;
verify the safety of referenced libraries in the AI-generated source code output ;
have a human regularly check the quality of the source code generated from sufficiently sophisticated queries.
R31
R31 Limit AI source code generation for critical application modules
It is strongly recommended that you do not use a generative AI tool to generate blocks of source code for critical application modules :
Cryptography modules (authentication, encryption, signature, etc.) ;
modules for managing user and administrator access rights ;
Sensitive data processing modules.
R32
R32 Educate developers about the risks of AI-generated source code
It is recommended to carry out awareness campaigns on the risks associated with the use of AI-generated source code. This awareness can be supported by public reports on this topic or research papers demonstrating the presence of vulnerabilities in AI-generated code.
In addition, developers can also be trained on AI tools for the optimization of their queries (prompt engineering) to improve the quality and security of the generated code.
R33
Special case of consumer AI services exposed on the Internet
R33 Tighten security measures for consumer AI services exposed on the Internet
It is recommended that special attention be paid to certain safety measures for services exposed to the general public, including:
train the AI model only from public data ;
ensure that users of the AI system have been pre-authenticated ;
systematically analyze user queries on the AI system ;
Check and validate responses before sending them to users ;
protect user data (request and response history, etc.) ;
implement measures against distributed denial of service (DDoS) ;
Securing the front-end web service for users.
R34
Special case of using third-party generative AI solutions
R34 Prohibit the use of generative AI tools on the internet for business use involving sensitive data
Since the client entity does not have control over the generative AI service, it is not possible to ensure that the confidentiality protection of the data submitted as input respects the security needs of the entity.
As a precautionary measure, it is therefore mandatory to never include sensitive data from the entity in user requests.
R35
R35 Conduct a regular review of generative AI tool rights configuration on business applications
It is recommended to review the access rights of generative AI tools as soon as the product is activated in the entity, to ensure that the rights positioned by default are not too lax or too open by design.
Finally, a regular review of access rights must be carried out (e.g., every month), in order to ensure that functional and security updates to the product do not impact users' need to know basis.
Contacts
Feel free to contact Hervé @ biZNov if you have any questions or to let me know if there are any other topics you would like to see covered on this blog.
And don't forget to spread the word if you liked this blog post, just click on the social media buttons below.
Sharing is caring :-)
Commentaires