How you can protect large language models from data poisoning

There is an escalating demand for advanced security solutions today, marked by the robust growth of the security software market in Asia/Pacific, excluding Japan and China, which according to IDC rose 21.1 percent in 2023 and reached US$6.3 billion (S$8.5 billion).

Additionally, a recent IDC FutureScape report predicts that by the end of 2024, a quarter of Top 2000 Asia companies will leverage Generative AI (Gen AI) within their Security Operation Centers (SOCs). The rapid digital transformation sweeping across Asia has made the region a center for innovation and growth.

Asia is today at the epicentre of a burgeoning cyber security crisis, fueled by rapid digitalisation and the increasing interconnectedness of systems.

Within cyber security, AI and large language models (LLMs) have emerged as powerful tools that not only mimic human writing and tackle complex questions but also engage in meaningful conversations that enhance security operations. By integrating them, we bolster analysts’ skills in detecting and responding to cyber threats, improving efficiency in managing cybersecurity incidents.

On the flip side, the same advancements that make LLMs so promising is also making them vulnerable to a new and insidious threat: data poisoning.

As communication service providers (CSPs) and enterprises in Asia plan to incorporate LLMs into their operations, protecting these AI models from malicious data injection is critical. This means making sure AI systems are tamper-proof, preventing any attempts to jailbreak or manipulate them with contaminated data.

Understanding data poisoning

Data poisoning is a malicious technique where attackers introduce corrupted or biased data into the training process of an LLM. This can lead to a model that generates harmful, misleading, or discriminatory outputs. For example, a poisoned AI model can be weaponised in harmful ways, from delivering misleading advice to exposing sensitive customer information or even creating malicious code.

There are many ways LLMs can be hacked. Two primary methods attackers use to poison LLMs are:

• During the Training Phase: During this critical phase, attackers can exploit gaps in the training process if security measures are not robust. Attackers could jailbreak the LLM, leading to the creation of harmful content. Foundational models such as OpenAI GPT and Anthropic Claude undergo extensive red teaming and testing to mitigate these issues before launch.
• During the Model’s Inference Time: Approaches such as RAG (Retrieval-Augmented Generation) are powerful and legitimate ways to enhance the AI model's knowledge without re-training it. However, once an LLM is deployed, attackers can manipulate inputs or the model's environment to produce incorrect or harmful outputs.

A compromised LLM is a ticking time bomb for Asian businesses

Safe deployment of large language models in CSPs and enterprises is critical as the consequences of data poisoning can be severe for Asian businesses. A compromised LLM could lead to financial losses, reputational damage, and even national security breaches. It is imperative that organisations in Asia prioritise the protection of their LLMs.

While specific, publicly documented cases of data poisoning targeting LLMs in Asia may be limited due to the secretive nature of cyberattacks, it is essential to recognise the potential for such incidents.

-Rodrigo Brito, Global Head of Cybersecurity Portfolio, Nokia

Attackers can exploit LLMs to carry out phishing attacks, such as poisoning an AI-powered chatbot to deceive users into sharing sensitive information.

How can we prevent data poisoning in LLMs?

Communication service providers (CSPs) are at the heart of Asia's digital transformation, offering essential infrastructure and services to businesses and consumers alike. As such, they are prime targets for cyberattacks, including data poisoning. Protecting LLMs within CSP environments is crucial to maintaining the integrity and reliability of cloud services.

Frederic Giron, VP and senior research director at Forrester said, “Asia Pacific is bracing itself for a year of exploration and potential growth, with Gen AI at the center of it all…The promise and potential of Gen AI, combined with a new wave of technological innovations, will inspire more APAC tech and business leaders to follow in the footsteps of early trailblazers and fuse the power of AI with their transformation efforts to drive business outcomes.”

This emphasises some positive steps that are being taken. LLMs, which are a subset of GenAI, and a recent Omdia report cites that 64 percent of telecom businesses in APAC have incorporated, or are in the process of incorporating Gen AI into their cyber security portfolios.

This proves crucial for CSPs in Asia to implement security measures to leverage the full capabilities of implementing LLM-powered offerings in their cyber security operations. But how do we address this?

It requires a multi-faceted approach. Researchers and developers need to implement strong data validation techniques to detect and filter out poisoned data during training. Key strategies for preventing data poisoning include:

• Rely on curated, human-verified data for accuracy
• Strengthen LLM security with anomaly detection using fresh validation sets
• Perform thorough negative testing to uncover vulnerabilities in flawed data
• Leverage precise language models in benchmark tests to reduce risks and prevent negative impacts

Secure your LLMs or risk disaster

Don’t wait for a breach – start by sanitising your training data to prevent leaks, enforcing strong user authentication, and filtering outputs for airtight content safety. CSPs in Asia can strengthen their defenses by securing data storage, maintaining continuous monitoring with thorough risk assessments, and strictly adhering to ethical and compliance standards.

AI-specific defenses like adversarial training can help reinforce LLMs against emerging cyber threats. Combined, these practices ensure LLMs operate securely, protecting both the technology and its users from potential risks.

It is clear that leaders in Asia are increasingly aware of both the immense opportunities and the significant challenges heralded by the rapid adoption of LLMs. For example, LLMs can respond to intricate questions, engage in meaningful conversations with security analysts, and dramatically improve incident forensics and resolution times in security operations centers.

However, to move forward, leaders must understand the threat of data poisoning and implement security measures to protect their organisations, ensuring the responsible development and deployment of these powerful technologies.

Rodrigo Brito is Nokia’s Global Head of Cybersecurity Portfolio