Over the years, hackers have been able to penetrate AI systems through inference attacks.
Such attacks are becoming a concern for many organisations globally. For instance, in 2009 similar attacks took place against the National Institutes of Health (NIH) in the United States, and NIH had to change their access policies to sensitive medical data.
The attack involves hackers repeatedly asking the AI service to generate information and analysing the data to detect patterns. Once they have determined the pattern, hackers can steal sensitive information or even reconstruct the original dataset that was used to train the AI engine.
The National University of Singapore (NUS) School of Computing has reported that it has developed a full-fledged open-source tool that can help companies determine if their AI services are vulnerable to such inference attacks.
Assistant Professor Reza Shokri and his team developing a tool to check for data leakage
To help tackle the problem, Assistant Professor Shokri from NUS Computing and his team created “Machine Learning Privacy Meter” (ML Privacy Meter). The tool analyses the Membership Inference Attacks to decipher if that particular data record was part of the model’s training data.
By replicating these attacks, the privacy analysis algorithm then quantifies the extent of which the model leaks individual data records in its training set — thus detailing the risk of different attacks that attempt to reconstruct the dataset either completely or partially. The tool would also help generate reports detailing the vulnerable areas in the training data that were used.
The ML Privacy Meter would then analyse the result of the privacy analysis to issue a scorecard that indicates how accurately attackers can identify the original datasets used for training. These scorecards will assist organisations by identifying the vulnerabilities in their datasets, and instructs them how to pre-emptively mitigate a possible Membership Inference Attack.
By developing a standardised general attack formula, the NUS Computing team provided a framework for their AI algorithm to properly test and quantify various types of membership interference attacks.
The ML Privacy Meter is based on the research led by the team over the last three years, upon identifying that there was no standardised method to sufficiently test and quantify the privacy risks of machine learning algorithms, making it difficult to provide a concrete analysis.
“When building AI systems using sensitive data, organisations should ensure that the data processed in such systems are adequately protected. Our tool can help organisations perform internal privacy risk analysis or audits before deploying an AI system.
“Also, data protection regulations such as the General Data Protection Regulation mandate the need to assess the privacy risks to data when using machine learning. Our tool can aid companies in achieving regulatory compliance by generating reports for Data Protection Impact Assessments,” says Assistant Professor Shokri.
