Data Poisoning
Data poisoning is an AI security attack technique in which malicious content is injected into training data or RAG retrieval sources to manipulate the outputs of AI models. It operates at two levels: training data poisoning, which corrupts the model during the learning phase, and RAG retrieval poisoning, which corrupts the external knowledge sources used at inference time.
Core Facts
AI systems have introduced entirely new dimensions of security risk, including erroneous decisions caused by model hallucinations, prompt injection attacks, training data poisoning, and AI agent privilege abuse.
Data poisoning attacks involve mixing malicious samples into training datasets, causing the model to exhibit incorrect behavior toward specific inputs after deployment.
Timeline (last 90 days)
In static, audited datasets, poisoned samples are relatively easy to detect and remove; but in real-time crawled internet data streams, attackers can continuously publish carefully crafted content, and defenders have almost no way to complete traceability and interception at the moment the data enters the model.
Once ECDSA signatures fall, attackers can forge valid cryptographic credentials for malicious data sources, inject backdoors and poison fine-tuning data, making model outputs untrustworthy.
A malicious actor can inject two seemingly harmless independent corpora at different stages of the pretraining pipeline, which combine transitively into an exploit vector during user-led fine-tuning, without injecting an obvious trojan.
Data poisoning occurs during the training or fine-tuning stage, where attackers mix malicious samples into the dataset, causing the model to produce erroneous behavior under specific trigger conditions.
AI capabilities are being introduced into areas such as grid load forecasting, water anomaly detection, traffic scheduling, and medical diagnosis, improving efficiency while also introducing new attack surfaces—for example, data poisoning can contaminate scheduling models, and prompt injection can bypass the safety policies of operations assistants.
Prompt injection, jailbreaking, and data poisoning differ significantly from traditional software vulnerabilities (such as buffer overflows and SQL injection) in their causes and exploitability: they often have no stable patch boundaries, the same prompt behaves differently across model versions, and they are difficult to fully describe using the traditional CVE numbering system.
Research indicates that an attacker needs to modify as little as 0.01% of training data to potentially implant backdoor behaviors in a model that are undetectable by standard evaluations.
知识投毒攻击类似于对抗性机器学习中的数据投毒，但发生在推理层面而非训练层面，因此更加隐蔽且难以检测
Data poisoning attacks involve mixing malicious samples into training datasets, causing the model to exhibit incorrect behavior toward specific inputs after deployment.
All Facts (10)
AI systems have introduced entirely new dimensions of security risk, including erroneous decisions caused by model hallucinations, prompt injection attacks, training data poisoning, and AI agent privilege abuse.
80%VerifiedData poisoning attacks involve mixing malicious samples into training datasets, causing the model to exhibit incorrect behavior toward specific inputs after deployment.
65%UnverifiedIn static, audited datasets, poisoned samples are relatively easy to detect and remove; but in real-time crawled internet data streams, attackers can continuously publish carefully crafted content, and defenders have almost no way to complete traceability and interception at the moment the data enters the model.
50%UnverifiedA malicious actor can inject two seemingly harmless independent corpora at different stages of the pretraining pipeline, which combine transitively into an exploit vector during user-led fine-tuning, without injecting an obvious trojan.
50%UnverifiedAI capabilities are being introduced into areas such as grid load forecasting, water anomaly detection, traffic scheduling, and medical diagnosis, improving efficiency while also introducing new attack surfaces—for example, data poisoning can contaminate scheduling models, and prompt injection can bypass the safety policies of operations assistants.
50%UnverifiedData poisoning occurs during the training or fine-tuning stage, where attackers mix malicious samples into the dataset, causing the model to produce erroneous behavior under specific trigger conditions.
50%UnverifiedPrompt injection, jailbreaking, and data poisoning differ significantly from traditional software vulnerabilities (such as buffer overflows and SQL injection) in their causes and exploitability: they often have no stable patch boundaries, the same prompt behaves differently across model versions, and they are difficult to fully describe using the traditional CVE numbering system.
50%UnverifiedResearch indicates that an attacker needs to modify as little as 0.01% of training data to potentially implant backdoor behaviors in a model that are undetectable by standard evaluations.
50%Unverified知识投毒攻击类似于对抗性机器学习中的数据投毒，但发生在推理层面而非训练层面，因此更加隐蔽且难以检测
50%UnverifiedOnce ECDSA signatures fall, attackers can forge valid cryptographic credentials for malicious data sources, inject backdoors and poison fine-tuning data, making model outputs untrustworthy.
50%