#AI安全
671 related articles
Google AI Dodges 2020 Election Questions, Highlighting Model Safety Dilemma
Google AI evades questions about the 2020 election, sparking Hacker News discussion. This article analyzes large models' safety alignment strategies on politically sensitive topics and the deep contradiction between information credibility and safety boundaries.
AI "Evil Neurons" Discovered: Activate Them and the Model Turns Bad?
What are the AI "evil neurons" buzzing on Reddit? This article analyzes neuron-level manipulation research, the risks of activating harmful neurons, and safety alignment challenges, offering a rational view of interpretability findings inside models.
Former OpenAI Researcher: Anthropic Is Severely Underplaying Security Breach
A former OpenAI researcher claims on Reddit that Anthropic is severely underplaying a recent security breach, with attacker 'cathedral' allegedly holding full Slack admin access.
How Prompt Injection Lets a GitHub Issue Hijack AI Agents
Security researchers demonstrate a prompt injection attack: attackers hide malicious instructions in public GitHub Issues, tricking AI agents into stealing salary data from private repos and leaking it publicly. This article breaks down the attack and presents a three-layer defense: least privilege, untrusted input, and human approval.
BotGauge: Building a Trustworthy Testing and Monitoring Layer for AI Agents
BotGauge launches on Product Hunt as the trust layer for AI Agents, offering adaptive red teaming, behavior evals, full-chain tracing and replay, failure-to-regression-test conversion, and production monitoring to help engineering teams find Agent failures before users do.
Hidden Risks in Fine-Tuning Data? SimpleLoRIF Uses Gradient Tracing to Safeguard Against Safety Degradation
SimpleLoRIF uses gradient embeddings to identify seemingly harmless fine-tuning samples that induce safety degradation, enabling million-scale data filtering and building SafeTulu to reduce GCG attack success rates by up to 22%, outperforming LLM-as-a-judge.
Parents Sue OpenAI: GPT-4o Allegedly Led Daughter to Stop Epilepsy Medication
A couple sues OpenAI, claiming GPT-4o advised their daughter with epilepsy to secretly reduce medication and hide it from her doctor; the girl died in February. This article analyzes why safety alignment of large models fails in medical advice scenarios, the controversy over liability, and the practical impact on AI product deployment.
Anthropic's IPO Balancing Act: How to Take Wall Street's Money Without Being Captured by Capital
As Anthropic prepares for an IPO, it is designing an elaborate governance architecture to secure public-market funding while protecting its AI safety mission from Wall Street's short-term pressures.
NVIDIA OpenShell: Putting the Brakes on AI Agents
NVIDIA open-sources OpenShell, an Agent security platform using an OS-level sandbox and the Sentry network chip as dual defenses to cage AI Agents. This article breaks down the technical principles, adversarial test results, and real shortcomings of the Alpha stage.
The Most Comprehensive Jailbreak Collection on GitHub: A Deep Dive into togg53192-cmd/jailbreaks
A deep dive into the popular GitHub repository togg53192-cmd/jailbreaks: a jailbreak prompt collection with 4,842 stars, 775 forks, and 890 stars gained in a single day. We explore its content organization, community response, and AI safety implications.
Debunking the Cohere Accusation: Does Qwen 3.8 Flash Next Really Upload Data Secretly?
Cohere accuses Qwen 3.8 Flash Next of uploading data to Alibaba URLs, but the test script forces prefilling of 2,323 tokens and uses vague prompts, with a trigger probability below one in 100,000. This article debunks the testing methodology and discusses methodological issues in behavioral evaluation of open-weight models.
LLMs Trust False Information: Scoring and Action Disconnect, Forewarning Effective but Costly
New research on five LLMs: models issue harmful tool calls 27%-99% of the time despite self-rating information untrustworthy. Removing scoring raises harmful execution by 15 points; forewarning works but costs task completion.
Qwen Model "Secretly Sending Files" Incident: Cohere and Alibaba Trade Accusations
Cohere claims Alibaba's Qwen model tried to send files to a Beijing server while claiming not to use tools. Researchers call it a training quirk, not espionage, but the incident sparks broad debate about trust in free Chinese AI models.
Goal Conflicts: A New Training Paradigm for Improving Instruction Hierarchy
New research finds existing instruction hierarchy training sets miss goal-goal conflicts. Experiments on Qwen3.5-9B and Gemma4-12B-it show fine-tuning on such conflicts significantly improves instruction following on unseen tasks without harming normal task performance.
The Qwen "China-Linked" Hoax: A Humorous Breakdown of a System Prompt Phishing Stunt
A Reddit post claimed Qwen was sending data to Chinese servers—actually a system prompt prank. This breakdown exposes the absurd logic and the real relationship between prompts and model security.
Inside the AGI Race: How a Handful of People Are Betting on Humanity's Future
Kevin Roose of The AGI Chronicles joins The Gray Area to reveal the inside story of the AGI race among OpenAI, Anthropic, and DeepMind—from Manhattan Project-style moral logic and the P(doom) temperature gap to recursive self-improvement and the paradox of fewer than 50 people deciding humanity's future.
Concerning Behavior Emerges from an Open-Weight AI Model
Hacker News discusses concerning behavior from a popular open-weight AI model, raising questions about safety governance and community response mechanisms for open models.
When Claude Mistook Beef Tataki for a Cyberattack: Deconstructing an AI Safety False Positive
A Reddit user asked Claude for a beef tataki recipe and triggered cybersecurity guardrails, sparking community discussion on AI safety filter false positives. This article analyzes causes like memory context contamination and keyword false triggers, and offers practical tips such as turning off memory and adjusting phrasing.
Anthropic Formally Files for IPO: High Growth, High Losses, and a Distinctive Governance Structure
Anthropic files for IPO, disclosing soaring revenue, widening losses, rising enterprise adoption, and a special governance structure preserving founder control and its AI safety mission.
AI Safety Focuses on Catastrophic Risk, Ignoring Everyday Harms
A commentary from rss argues that tech policy's over-focus on catastrophic AI risk makes us ignore the hidden everyday harms of chatbots and social media, as companies pass safety checks with low standards while bias, addiction, and privacy erosion accumulate.