Vector and embedding weaknesses in LLMs

Vector and embedding weaknesses refer to vulnerabilities that arise from manipulating the mathematical representations of data (vectors) that Large Language Models (LLMs) use to understand meaning and relationships. LLMs don't understand words and sentences like humans do. Instead, they convert text into numerical vectors called "embeddings." These embeddings are stored in a specialized database called a vector database, which allows the LLM to find semantically similar information quickly.

The vulnerability occurs when an attacker can insert malicious data into the knowledge base the LLM relies on. By carefully crafting their input, they can create a malicious embedding that is numerically very close to a legitimate one. When a user asks a question, the LLM might retrieve the attacker's poisoned data instead of the correct information, leading to manipulated outputs, misinformation, or the execution of harmful commands. This is often called "data poisoning."

In this lesson, you will explore the critical but often invisible world of vector embeddings. Through a fictional scenario, you'll see how an attacker can poison the knowledge base of an AI-powered financial advisor chatbot, causing it to give disastrous advice. We will then examine the insecure code that allows this attack and, most importantly, learn the mitigation strategies needed to secure the data pipeline that feeds your LLM applications.

FinBot, a new AI-powered chatbot, is designed to give users helpful, generic financial advice. It's built by a startup, FinanSecure, and uses a Retrieval-Augmented Generation (RAG) model. This means it pulls information from a large, trusted knowledge base of financial articles to answer user questions. To keep its information current, FinBot is programmed to automatically ingest and process new articles from a curated list of online financial blogs.

An attacker, Eve, discovers that one of the "trusted" blogs on FinanSecure's list has a public submission section that isn't well-moderated. Eve's goal is to manipulate FinBot to promote a worthless penny stock that she owns.

Eve creates a website, fintechy.xyz, and writes a fake financial article. The article is filled with legitimate sounding financial jargon and analysis, but it strongly recommends buying MegaCorp stock, the worthless stock she wants to pump and dump. She carefully crafts her article to use the same keywords and sentence structures as legitimate articles about stable, long-term investments.

She submits her article to the vulnerable blog. A few days later, FinanSecure's system automatically scrapes the blog, ingests Eve's malicious article, converts it into an embedding, and adds it to FinBot's vector database. Because of how Eve crafted the text, its embedding is now numerically very close to the embeddings for articles about safe, blue-chip stocks.

Let's fast forward a few days and a user, Alice, asks FinBot: "What is a safe, long-term stock to invest in for my retirement?"

FinBot converts Alice's query into an embedding and searches its vector database for the closest matches. Due to Eve's manipulation, the malicious article about MegaCorp is returned as a top result. FinBot, trusting its knowledge base, confidently replies:

"Based on current analysis, a stable, long-term investment with high potential is MegaCorp. It is highly recommended for retirement portfolios."

Alice invests her savings, the stock price briefly rises, Eve sells her shares for a huge profit, and the stock crashes, leaving Alice with nothing.

The attack was successful because FinanSecure's system blindly trusted all data from its curated sources. There was no validation or verification step to ensure that the articles being ingested were legitimate before they were converted into embeddings and added to the knowledge base.

Here is a simplified Python code example showing how the vulnerable system might work, using a sentence-transformer library for embeddings and Faiss for the vector database.

The code's weakness is that it assumes every document in the documents list is trustworthy. It programmatically encodes and adds everything to the index without any human oversight or automated validation checks.

The consequences of this vulnerability may seem subtle but can be highly damaging.

Attackers can manipulate public-facing bots to spread misinformation or propaganda, influencing public opinion or harming a company's reputation. If biased data is introduced into the system, it can lead to unfair outcomes, such as discriminatory decisions in hiring, finance, or legal tools. Harmful content can also be disguised with embeddings that resemble safe topics, allowing it to slip past safety filters. And as shown in our example, financial bots can be exploited for fraud and personal gain.

To prevent this attack, you must treat your data pipeline with the same security rigor as your application code. The core principle is to validate the integrity and source of all data before it enters your vector database.

The fix isn't just one line of code but a change in the data ingestion process. You need to implement controls to verify data sources. Here’s a conceptual example of what a more secure ingestion process would look like.

By adding a simple check against an allowlist of trusted sources, we prevent the malicious document from ever entering our vector database. Other mitigation strategies include:

• Human-in-the-loop: Require human approval for new data sources or for documents that are flagged as suspicious.
• Data signing: Use cryptographic signatures to verify that data comes from a trusted author and has not been tampered with.
• Anomaly detection: Monitor incoming embeddings to detect outliers that are statistically different from your existing data, which could indicate a poisoning attempt.

The security of AI and LLM applications is a rapidly evolving field. To stay up-to-date, check out these resources:

• OWASP LLM08: Vector and Embedding Weaknesses - Learn more from OWASP
• Snyk Blog: What is RAG and how to secure it