Summary

Data poisoning can manipulate training or retrieval data to alter AI model behavior, making data integrity, RAG security, and trusted recovery key to protecting AI systems.

image_pdfimage_print

Data poisoning is the deliberate manipulation of the data an AI model is trained on, fine-tuned on, or grounded in, so that the model behaves the way an attacker wants. Its effects can include degraded accuracy, hidden trigger behavior, or targeted false answers. It can also create data-leakage risk if a poisoned model or retrieval path causes sensitive information to be surfaced.

The impact can also be financial. Responding may require investigating affected data, rebuilding a clean training or retrieval pipeline, and validating outputs again. When retraining or fine-tuning is necessary, it can add compute and engineering costs; RAG recovery may instead require removing the poisoned source and re-indexing the knowledge base.

Consider a retrieval-augmented assistant that answers questions from an internal knowledge base. It starts recommending the wrong vendor. Nobody changed the model. Someone added one carefully written document to the knowledge base, and that document now ranks near the top for every related question.

The amount of bad data needed can be small. In a study with the UK AI Security Institute and the Alan Turing Institute, Anthropic researchers found that as few as 250 malicious documents could plant a backdoor in language models from 600M to 13B parameters, regardless of model size or training data volume.

This article explains what AI data poisoning is, how it works at each stage of the AI lifecycle, the main types of data poisoning attacks, why RAG and vector databases change the threat, and how to prevent and recover from it.

What is data poisoning in AI and machine learning?

Data poisoning is an integrity attack on data rather than on code. The model’s software can be unchanged and fully patched. What changes is the information it learns from or retrieves and that information shapes every answer it gives.

It applies to classic machine learning classifiers, such as spam filters and fraud models, and to large language models. In both cases, the attacker’s job is the same: get manipulated data into the pipeline without being noticed.

Data poisoning is often confused with prompt injection. Prompt injection manipulates the input a model receives at inference time. Data poisoning corrupts what the model has learned or what it retrieves. The two can combine, for example, when a poisoned document carries instructions that act as an injection once it’s retrieved.

The term has a second use outside enterprise security. Tools such as Nightshade, from the University of Chicago, let artists turn images into “poison” samples that disrupt models trained on them without consent. This article covers poisoning from the defender’s side.

How data poisoning works across the AI lifecycle

The OWASP Top 10 for LLM Applications 2025 lists data and model poisoning as LLM04:2025. It describes poisoning as manipulation of pre-training, fine-tuning, or embedding data to introduce vulnerabilities, backdoors, or biases. Each of the three stages has a different exposure.

Pre-training

Pre-training uses large volumes of broadly collected data, often scraped from the web or bought from third parties. The scale makes manual review impossible, and it gives attackers a way in: publish or alter content that a scraper will later collect. OWASP names split-view data poisoning and frontrunning poisoning as techniques that exploit these training dynamics.

Fine-tuning

Fine-tuning uses smaller, task-specific data sets. Because each example carries more weight, a few bad records can have a large effect. OWASP’s guidance includes tailoring models to each use case with specific fine-tuning data sets, which also narrows what any single poisoned set can reach.

Embedding

Embedding converts text into vectors so it can be searched and retrieved. Poisoned content at this stage doesn’t change the model at all. It changes what the model is shown when it answers. That makes embedding the stage most relevant to RAG systems, covered below.

Why LLM poisoning doesn’t need much data

The Anthropic study is the clearest recent evidence on LLM poisoning. The researchers found that 250 poisoned documents compromised models across every size they tested, even though the largest models trained on more than 20 times as much clean data. The practical lesson is that an attacker’s challenge is getting data into the pipeline, not producing a large volume of it.

The result has limits. The backdoor tested was a “denial-of-service” behavior that made the model produce gibberish when it saw a trigger phrase, and the authors note it’s still unclear whether the pattern holds for larger models or more harmful behaviors.

Types of data poisoning attacks

The NIST AI 100-2 E2025 taxonomy of adversarial machine learning organizes data poisoning attacks by what the attacker wants to achieve:

  • Availability poisoning: Degrades the model broadly. NIST describes these attacks as causing indiscriminate degradation of the model on all samples. Example: mislabeled records flood a fraud model’s training set until its accuracy drops across every transaction type.
  • Targeted poisoning: Changes the model’s output for a small number of specific inputs while leaving the rest alone. Example: a spam filter learns to let one sender’s messages through.
  • Backdoor poisoning: Teaches the model to behave normally until it sees a trigger pattern, then perform an attacker-chosen behavior. Example: a model answers correctly until a rare phrase appears in the prompt.
  • Clean-label poisoning: The attacker controls the training examples but not their labels, so every label looks correct and review doesn’t catch the manipulation. It can be used for availability or backdoor goals.
  • Model poisoning: Modifies the trained model directly to inject malicious functionality, rather than poisoning its data. A tampered model downloaded from a shared repository is the common supply chain route.

Security teams map these attacks to MITRE ATLAS, the adversarial AI knowledge base, which catalogs training-time poisoning as technique AML.T0020.

RAG poisoning and vector database security

Retrieval-augmented generation (RAG) changes the poisoning threat. A RAG system answers from a knowledge base that can be updated without retraining the model. That makes fixes fast, but it also means write access to the knowledge base and its ingestion pipeline is now a security boundary.

RAG poisoning works through vector search. A planted document written to sit close to common questions in embedding space gets retrieved for those questions and shapes the answer. MITRE tracks this as AML.T0070, RAG Poisoning: adversaries inject malicious content into data a RAG system indexes to contaminate later answers.

Research shows how little it takes. In PoisonedRAG, presented at USENIX Security 2025, researchers reported a 90% attack success rate when they injected five malicious texts per target question into a knowledge database of millions of texts. They also evaluated several defenses and found them insufficient.

Poisoning is one of several threats to vector databases. Embedding inversion, membership and attribute inference, and semantic deception are covered in this vector database security guide.

How to prevent and recover from data poisoning

No single control stops data poisoning. Defense works in three layers: keep poisoned data out, detect it when it gets in, and recover to a known-good state when it’s found.

Prevent

  • Track provenance and lineage: OWASP recommends tracking data origins and transformations with tools such as OWASP CycloneDX or an ML-BOM. Data lineage shows where each training or retrieval source came from and what touched it.
  • Vet data sources and vendors: Third-party and scraped data carry the most exposure, so they need the most scrutiny before use.
  • Control write access: Limit who and what can write to training sets, fine-tuning data, and vector stores. For RAG, the ingestion pipeline is part of the attack surface.
  • Know what’s in the corpus: Classifying data before ingestion shows what sensitive and unexpected content is entering training sets and knowledge bases.

Detect

  • Version data sets: OWASP recommends data version control (DVC) to track changes in data sets and detect manipulation.
  • Watch for anomalies: Unexpected shifts in training loss, evaluation results, or model behavior can signal poisoned data.
  • Red-team the pipeline: Test whether planted content can reach training data or the knowledge base, and whether it changes outputs.

Recover

  • Trace and restore: Once poisoning is confirmed, trace it to its source and restore a known-good version of the data set or vector store from before the attack, then re-index or retrain. Cleaning poisoned data in place is slow and uncertain because it requires knowing every record the attacker touched.
  • Protect the recovery point: Recovery depends on a clean copy that the attacker could not alter or delete. If the same credentials that wrote the poisoned data can also change or remove the backups, the recovery point cannot be trusted.

How Everpure helps

Everpure can help teams improve visibility into data environments that support AI initiatives and protect data with immutable snapshots that can support recovery.

  • Visibility before ingestion: Everpure Data Intelligence discovers and classifies data across DQL and NoSQL databases, SaaS, public cloud, and third-party storage, and identifies PII, PHI, and other sensitive data. Teams can see what is entering training sets and RAG knowledge bases.
  • A clean recovery point: SafeMode™ Snapshots are immutable and indelible by default. Deleting or weakening a protected snapshot requires multi-party authorization and an eradication timer, so a pre-attack copy of a training data set or vector store survives even if a single administrator account is compromised.

Explore Everpure cyber resilience and AI infrastructure solutions.

Conclusion

Data poisoning targets the data, not the model. A small, fixed number of documents can be enough to plant a backdoor, and RAG turns the knowledge base into a live attack surface that changes every day.

That shifts the defense toward the data layer: knowing what goes into training sets and vector stores, controlling who can write to them, and keeping a recovery point the attacker can’t reach. Detection will miss some attacks, so the ability to return to a known-good state is what limits the damage.

Data poisoning FAQs

What is an example of data poisoning?

A common example is a backdoor: An attacker adds a small number of training documents that pair a rare trigger phrase with a specific behavior. The model works normally until the trigger appears. In RAG systems, an attacker can plant a document in the knowledge base that is written to be retrieved for certain questions, so the assistant repeats false information without any change to the model itself.

How do you combat data poisoning?

Combating data poisoning takes prevention, detection, and recovery together. Prevention means tracking data provenance, vetting sources, and restricting write access to training data and vector stores. Detection means versioning data sets and watching for unexpected changes in model behavior. Recovery means restoring a known-good, unalterable copy of the data from before the attack, then re-indexing or retraining.

Is data poisoning the same as prompt injection?

No. Prompt injection manipulates the input a model receives at the moment it answers. Data poisoning corrupts the data a model learns from or retrieves, so the effect persists across many interactions. The two can overlap when a poisoned document in a RAG knowledge base contains instructions that act as an injection once retrieved.

Can data poisoning affect models that are already deployed?

Yes, especially RAG systems. A deployed model’s weights may be fixed, but its knowledge base keeps changing. Any content added to that knowledge base can influence answers right away, without retraining. Deployed models can also be retrained or fine-tuned on new data, which reopens the training-time risk.