AI Safety ≠ Cybersecurity: Gray Swan Founders Detail a New Paradigm for Red-Teaming

AI Safety Demands an Entirely New Mental Framework

OpenAI board member Zico Kolter and Gray Swan CEO Matt Fredrikson recently shared an in-depth conversation that addressed a widely misunderstood point in the industry: AI safety is far more than just "applying AI to cybersecurity." Coming from two scholar-entrepreneurs who have spent years in adversarial AI research, this assertion deserves serious attention across the entire industry.

Adversarial research is a core branch of machine learning security, originating from the "adversarial examples" phenomenon discovered by Szegedy et al. in 2013 — where imperceptible perturbations to input data can cause deep learning models to produce completely incorrect outputs. Kolter himself is one of the foundational researchers in this field; his PGD (Projected Gradient Descent) attack method, proposed in 2017, remains a standard benchmark for adversarial robustness evaluation. Fredrikson is known for his work on Model Inversion Attacks, demonstrating that attackers can reverse-engineer sensitive information from training data through model outputs. The academic expertise of both directly shaped Gray Swan's technical approach.

rss source

The Evolution of Red-Teaming: From Manual Exploration to Systematic Engineering

Why Traditional Red-Teaming Falls Short

Red-teaming is the core method for evaluating AI system security and has developed rapidly over the past few years. The concept originated in Cold War-era military exercises, where the U.S. military used a "red team" to simulate Soviet adversaries and test their own defenses. In cybersecurity, red-teaming refers to professional security personnel simulating real attackers to conduct comprehensive penetration testing. In the AI era, red-teaming has been adopted for large language model security evaluation, but its meaning has undergone a fundamental shift: the target is no longer deterministic code logic but probabilistic neural network behavior. Major AI labs including OpenAI, Anthropic, and Google DeepMind all conduct large-scale red-teaming before model releases, but methodologies are still rapidly iterating and lack unified standards.

However, as LLM capabilities advance at breakneck speed, traditional red-teaming methods face fundamental challenges. Kolter and Fredrikson point out that many organizations still use a cybersecurity mental framework to understand AI safety problems — a cognitive bias that can create serious security blind spots.

Traditional cybersecurity focuses on well-defined attack surfaces — vulnerabilities, ports, and privilege escalation paths. It is built on a mature conceptual ecosystem: the CVE (Common Vulnerabilities and Exposures) numbering system, CVSS (Common Vulnerability Scoring System), and penetration testing methodologies like the OWASP Top 10. These frameworks assume that vulnerabilities are discrete, catalogable, and patchable. But the "attack surface" of AI systems is fuzzy, probabilistic, and highly context-dependent. The "vulnerabilities" in AI systems are fundamentally different — they are not logical errors in code but statistical biases in high-dimensional parameter spaces. A jailbreak prompt works not because of a code bug, but because the probability distributions learned during training contain exploitable patterns. This means the traditional "discover-report-fix-verify" security workflow needs fundamental restructuring in the AI domain. NIST (National Institute of Standards and Technology) released its AI Risk Management Framework (AI RMF) in 2024, attempting to establish baseline standards for this new paradigm. A model that behaves safely under one prompt may completely go off the rails with a slightly modified context.

Three Unique Challenges in AI Safety

Compared to cybersecurity, AI safety differs fundamentally across several dimensions:

Uncertainty boundaries: Traditional software either has a vulnerability or it doesn't, but AI model behavior is probabilistic. You cannot declare a system "safe" after a single test, because the same model may exhibit entirely different behavior patterns in different contexts.

Adversarial adaptation: Attackers can dynamically adjust their strategies by observing a model's response patterns. The dynamics of this "cat-and-mouse game" far exceed those in traditional cybersecurity scenarios. In recent years, automated jailbreak techniques have emerged — such as GCG (Greedy Coordinate Gradient) attacks, genetic algorithm-based prompt mutations, and "progressive jailbreaking" that gradually breaches safety boundaries through multi-turn conversations — all demonstrating how attackers can systematically exploit a model's statistical properties to bypass safety alignment.

Unpredictability of emergent capabilities: As models scale up, new capabilities may suddenly emerge, introducing previously unforeseen risk categories. A 2022 paper from Google Research systematically documented this phenomenon: when model parameter counts cross certain thresholds, models suddenly exhibit capabilities entirely absent at smaller scales, such as multi-step reasoning, code generation, and even basic mathematical proofs. This nonlinear leap means safety evaluation faces a fundamental difficulty — you cannot reliably predict the risk profile of the next-generation model based on the current version's behavior. Stanford University researchers have called this the "impossibility of capability prediction," which is precisely why Kolter and Fredrikson emphasize that safety evaluation must be continuous rather than one-time.

Gray Swan's Methodology: Academic Depth Driving Industry Practice

Upgrading Red-Teaming from "Craft" to "Engineering Discipline"

As a company focused on AI safety, Gray Swan's core philosophy is making red-teaming systematic and quantifiable. Gray Swan AI was co-founded by three professors from Carnegie Mellon University's School of Computer Science — in addition to Kolter and Fredrikson, the team includes a researcher specializing in formal verification. Carnegie Mellon University consistently ranks among the world's top computer science programs, with deep research foundations in AI safety, robotics safety, and software security. The name "Gray Swan" itself carries deliberate meaning — unlike Nassim Taleb's "Black Swan" (completely unforeseeable events), a "Gray Swan" refers to risks that are theoretically foreseeable but often overlooked in practice, which maps precisely to the many known but inadequately addressed threat categories in AI safety.

Fredrikson emphasizes that effective AI red-teaming requires three things:

Automated exploration: Using AI itself to discover AI's weaknesses, building automated adversarial testing pipelines. This approach embodies the idea of "fighting fire with fire" — using a specially trained "attack model" to systematically explore the target model's safety boundaries, achieving efficiency and coverage far beyond manual testing.
Systematic coverage: Rather than relying on individual testers' creativity, establishing reproducible and quantifiable evaluation frameworks. This requires defining a clear risk taxonomy covering multiple dimensions — from harmful content generation, privacy leakage, and bias amplification to tool misuse — with quantitative metrics for each dimension.
Continuous monitoring: Security evaluation is not a one-time event but a continuous process spanning the model's entire lifecycle. After deployment, models face distribution shift — real user input patterns may differ significantly from training and test data, meaning pre-deployment safety evaluations cannot cover all real-world risks.

A Two-Way Channel Between Academic Frontiers and Product Deployment

Kolter serves as a Carnegie Mellon University professor, an OpenAI board member, and a co-founder of Gray Swan. Kolter joined OpenAI's restructured board in 2024, an appointment that itself reflects the AI industry's elevated focus on safety governance. In late 2023, OpenAI experienced a board crisis that shook the global tech world — CEO Sam Altman was briefly fired and then quickly reinstated, with one of the core disputes centering on the tension between AI safety and the pace of commercialization. The restructured board deliberately brought in more members with technical safety backgrounds, and Kolter's appointment exemplifies this strategy. His dual role as both an AI safety company founder and a governor of one of the most cutting-edge AI labs provides a unique perspective, though it has also sparked discussions about potential conflicts of interest.

His multiple roles enable Gray Swan to rapidly translate cutting-edge academic research into practical safety tools. The company's core products include an automated red-teaming platform and model safety evaluation tools, serving both AI labs and enterprise AI deployment scenarios. This tight coupling of academia and industry is especially critical in AI safety — because new attack methods are often first discovered and published in academic papers, and whoever can engineer research findings into products fastest gains the upper hand in the security arms race.

Industry Implications: Redefining the Boundaries of AI Safety

Moving Beyond the "Find a Vulnerability, Apply a Patch" Mindset

Both experts call on the industry to transcend the traditional patch-based security mindset. AI system security needs to be considered from the design phase, not remediated after the fact. This philosophy is known as "Shift Left Security" in software engineering, but its implementation in AI is far more complex than in traditional software — because AI model behavior is not determined by explicitly written rules but implicitly learned from massive datasets. Specifically:

The model training process itself needs to incorporate safety constraints, including safety preference labeling in RLHF (Reinforcement Learning from Human Feedback), adversarial training to enhance model robustness against malicious inputs, and self-constraint mechanisms like Constitutional AI
Deployment environments need multi-layered defense mechanisms, including input filtering, output moderation, rate limiting, and dynamic safety policies based on intent classification
Safety evaluation standards need to evolve in lockstep with model capabilities — current evaluation benchmarks may become completely ineffective against next-generation models

Practical Advice for AI Application Developers

For developers building AI applications, this conversation sends a clear signal: Do not assume that the foundation model you're using is "safe enough." Every specific application scenario can introduce new attack vectors that require targeted security evaluation. For example, an AI customer service assistant connected to an enterprise's internal knowledge base could be manipulated by attackers through carefully crafted prompts to leak sensitive business data — such "Indirect Prompt Injection" attacks have already been validated in multiple real-world deployment scenarios. Integrating security testing into every stage of the development process, rather than running a single check before launch, is the right approach to managing AI safety risks.

Conclusion

AI safety as an independent discipline is maturing rapidly. Kolter and Fredrikson's perspectives remind us that transplanting traditional cybersecurity experience into this domain may not only be ineffective but actively harmful — because it creates a false sense of security. True AI safety requires new tools, new methodologies, and a deep understanding of the fundamental nature of AI systems. Currently, AI safety regulatory frameworks are also taking shape at an accelerating pace worldwide — the EU AI Act officially took effect in 2024, the U.S. White House issued an executive order on AI safety, and China has introduced regulations on generative AI. These policy signals further demonstrate that AI safety is no longer a niche academic topic but an infrastructure-level issue critical to the healthy development of the entire technology ecosystem. For every AI practitioner, now is the pivotal moment to reassess your security strategy.