AI-Self Security

Striving Toward an Intelligent World

With the accumulation of big data, dramatic improvements in computing power, and continuous innovation in Machine Learning (ML) methods, Artificial Intelligence (AI) technologies such as image recognition, voice recognition, and natural language processing have become ubiquitous. Meanwhile, AI poses a significant impact on computer security: on the one hand, AI can be used to build defensive systems such as malware and network attack detection; on the other hand, AI might be exploited to launch more effective attacks. In some scenarios, the security of AI systems is a matter of life and death. Thus, building robust AI systems that are immune to external interference is essential. AI can benefit security and vice versa.

The main purpose of this paper is to explore the security of AI, in terms of protecting the integrity and confidentiality of AI models and data, and thus preventing attackers from changing the inference results or stealing the data.

Unlike security vulnerabilities in traditional systems, the root cause of security weaknesses in ML systems lies in the lack of explainability in AI systems. This lack of explainability leaves openings that can be exploited by adversarial machine learning methods such as evasion, poisoning, and backdoor attacks. These attacks are very effective and have strong transferability among different ML models, and thus pose serious security threats to Deep Neural Network (DNN)-based AI applications. For instance, attackers can inject malicious data in the training stage to affect the inference of AI models or add a small perturbation to the input samples in the inference stage to alter the inference result. Attackers may also implant backdoors in models and launch targeted attacks or extract model parameters or training data from query results.

In order to tackle the new AI security challenges, this paper proposes three layers of defense for deploying AI systems:

• Attack mitigation: Design defense mechanisms for known attacks.
• Model security: Enhance model robustness by various mechanisms such as model verification.
• Architecture security: Build a secure architecture with multiple security mechanisms to ensure business security.

Five Challenges to AI Security

AI has great potential to build a better, smarter world, but at the same time faces severe security risks. Due to the lack of security consideration at the early development of AI algorithms, attackers are able to manipulate the inference results in ways that lead to misjudgment. In critical domains such as healthcare, transportation, and surveillance, security risks can be devastating. Successful attacks on AI systems can result in property loss or endanger personal safety.

AI security risks exist not only in theoretical analyses but also in AI deployments. For instance, attackers can craft files to bypass AI-based detection tools or add noise to smart home voice control command to invoke malicious applications. Attackers can also tamper with data returned by a terminal or deliberately engage in malicious dialogs with a chat robot to cause a prediction error in the backend AI system. It is even possible to apply small stickers on traffic signs or vehicles that cause false inferences by autonomous vehicles.

To mitigate these AI security risks, AI system design must overcome five security challenges:

• Software and hardware security: The code of applications, models, platforms, and chips may have vulnerabilities or backdoors that attackers can exploit. Further, attackers may implant backdoors in models to launch advanced attacks. Due to the inexplainability of AI models, the backdoors are difficult to discover.
• Data integrity: Attackers can inject malicious data in the training stage to affect the inference capability of AI models or add a small perturbation to input samples in the inference stage to change the inference result.
• Model confidentiality: Service providers generally want to provide only query services without exposing the training models. However, an attacker may create a clone model through a number of queries.
• Model robustness: Training samples typically do not cover all possible corner cases, resulting in the insufficiency of robustness. Therefore the model may fail to provide correct inference on adversarial examples.
• Data privacy: For scenarios in which users provide training data, attackers can repeatedly query a trained model to obtain users’ private information.

AI Security Layered Defense

As illustrated in Figure 3-1, three layers of defense are needed for deploying AI systems in service scenarios: Attack mitigation, Model security and Architecture security.

Figure 1-1 AI security defense architecture

Attack mitigation: Design defense mechanisms for known attacks. The typical AI security attacks include evasion attacks, poisoning attacks, backdoor and model extraction. For these attacks, many countermeasures have been put forward in literature, such as adversarial training, Network Distillation, adversarial detection, DNN model verification, data filtering, ensemble analysis, model pruning, PATE, etc.

Model security: adversarial ML exists extensively. Evasion attacks, poisoning attacks, and all kinds of methods that take advantage of vulnerabilities and backdoors are not only accurate, but also have strong transferability, leading to high risks of misjudgment by AI models. Thus, in addition to defense against known attacks, the security of an AI model itself must be enhanced to avoid the damage caused by other potential attacks. Potential techniques include model detectability, model verifiability and model explainability.

Architecture security: When developing AI systems, we must pay close attention to their potential security risks; strengthen prevention mechanisms and constraint conditions; minimize risks; and ensure AI’s secure, reliable, and controllable development. When applying AI models, we must analyze and determine the risks in using AI models based on the characteristics and architecture of specific services, and design a robust AI security architecture and deployment solution using security mechanisms involving isolation, detection, failsafe, and redundancy.