AI security - the new face of application security
How AI based application risks are a new blind spot in cybersecurity
While the AI market has been growing by leaps and bounds for many years, ChatGPT in late 2022 really made it mainstream.
Suddenly everyone from content creators to big tech CEOs are pushing the AI hype train even more than ever
If you work in a company then you can expect it to adopt AI in some form or the other in the coming years
At the same time, AI applications also introduce new risks which a lot of cybersecurity teams seem to not be aware of
Just like application security was a blind stop to traditional network firewalls a few decades ago, AI targeted attacks seem to be flying under the radar
Lets take a look a few of them
How these attacks work
AI unique attacks target how machine learning works. Machine learning can be thought of as the engine that drives AI and gives it the magical ability to make decisions .. just like humans learn from experience
Machine learning models learn from data they are provided and build up the ability to make decisions based on that data.
Initially the model may make mistakes which are checked against actual data and used to fine tune it
The more data that you provide a model , the more accurate it becomes over time
Taking a look at the above diagram .. how would you assess its security ?
Are you going to scan the underlying infrastructure ? the access controls ? or the security of the APIs it exposes ?
All are good actions to take from a risk perspective
But what about the unique attacks that take place against AI
Most attackers want to compromise machine learning and AI applications and tamper with their decision making process OR find out what data was used to train them
How would you find out if the model was vulnerable or not ?
Let us take a look at a few common attacks
Attack 1 — Evasion
Evasion attacks like the name says are attempts to bypass or “evade” machine learning models by attackers
These are usually done by making small changes to the data that is provided to the model.
Small things that imperceptible to humans can result in drastically different decision being made by ML models
What if an attacker could bypass the machine learning model used to detect malware in companies ?
Or trick a facial recognition system by making small changes to an image ?
Or make a self driving car think a tree is an open road ?
How to protect
The first step to protect against evasion attacks is to find out if your model is vulnerable to it or not
Just like cybersecurity teams conduct penetration testing and security reviews to find security flaws, AI based security testing needs to happen
Evasion attacks can be found out via using “adversarial samples” which are designed to test models against these attacks.
Make this a part of your security testing ASAP
The below is an excellent resource for finding out examples of such attacks
SecML: A library for Secure and Explainable Machine Learning¶
Attack 2 — Inference attacks
Similar to evasion, inference attacks “trick” models and make them act in a way they were never designed to do
Inference is a way of getting the model to disclose information about how the model works and what data was used to train it
Models typically expose APIs that attackers can keep on querying until they are able to build up a picture of how the model is working
The below paper goes into deep detail about how attackers were able to reconstruct the faces used in training a machine learning model
This attack can be highly dangerous especially against models used in sensitive industries like financial institutions, law enforcement etc.
How to protect
Just like one of the most basic principle of application security is to sanitize outputs .. ML models have to sanitize what they are responding with
Giving away too much information in responses can allow malicious attacks to reconstruct the machine learning logic AND the data that was used.
Try and provide summarized responses which are sanitized and anonymized making it difficult for attackers to find out the inner workings of AI systems.
Attack 3 — Poisoning attacks
Ransomware is a nightmare for companies in which attackers encrypt or “pollute”the data they need to run their business
Similarly data is the lifeblood of ML models and can be targeted by attackers and polluted
By poisoning the well, the attackers can make the machine learning model reach incorrect decisions
How about adding a malware to the data used to train an anti-malware solution so that it effectively does not consider it malicious ?
It is critical to ensure the data used in training is secure right from the start and throughout the ML lifecycle
How to protect
The controls around the data used to train a system must be strictly checked to make sure they cannot be tampered with.
This is easier said than done given that most companies outsource their data models and do not maintain them in-house
Just as quality tests are done on applications to make sure they are working correct, ML models have to be checked after every data refresh to make sure their functionality has not been tampered by faulty data.
How to create an AI security culture
Implementing controls to protect against these attacks does not happen in a vacuum
Below are a few tips on how to create one to protect against these attacks
Cybersecurity teams need to be trained and up-skilled with regards to these attacks. We dont have AI application firewalls (yet) so it is crucial to learn how these attacks happen
Threat Modeling sessions need to incorporate these new attacks for any AI or machine learning model that is being considered for the company
Can you SIEM solutions detect if an attacker is scanning a ML model’s API for the purposes ? Check this and start building up your monitoring capabilities
Get your pen-testing teams to study resources like MITRE ATLAS which is a great resource for understand these attacks and how to protect against them
Thanks for reading this !
If you found the whole topic of AI governance and cybersecurity interesting then check out my book on the same below