SIRFEX
An open-source simulation robustness testing framework for stress-testing autonomous controllers, discovering reproducible failures and evaluating improvements. It supports deterministic simulation, scenario search, failure replay and physical feasibility analysis.
Timeline (last 90 days)
Developers are building an open-source framework called SIRFEX (Simulation-based Integrity, Robustness & Failure Exploration), aimed at simulation-based stress testing of autonomous controllers, finding reproducible failures, and evaluating whether improvements truly enhance robustness.
The SIRFEX framework supports PPO training, compute-matched continuation experiments, failure-driven curricula with validation stress exposure, and an optional LangGraph/Groq research agent for proposing bounded training intervention suggestions.
In a 51,200-interaction development study of SIRFEX, increased stress exposure improved tracking performance under hard conditions but frequently caused tracking degradation under normal conditions.
In the seed 3 experiment, the PPO policy trained on normal data only achieved a nominal MAE of 0.0177 and a stress MAE of 0.1166; the 5% stress policy achieved a nominal MAE of 0.0296 and a stress MAE of 0.0654; the 10% stress policy achieved a nominal MAE of 0.0501 and a stress MAE of 0.0555.
The controller trained only on normal data achieved zero physical violations on the development stress set.
Developers tried using GPT-OSS 120B to propose a retraining curriculum; in a matched development pilot on one seed, this approach performed worse than a simple fixed 5% stress policy, and neither passed the nominal performance preservation guardrail.
As stress exposure increased, upper-part action saturation of the controller intensified, suggesting the policy may drift toward more aggressive pumping behavior.
Failure-driven PPO fine-tuning may be inferior to plain continuation training, with possible causes including distribution shift, reward-shaping side effects, and curriculum order sensitivity.
Developers have not yet determined whether the nominal performance degradation is caused by catastrophic forgetting, reward incentives, curriculum order, or other reasons.
The SIRFEX framework runs on a local CPU and emphasizes reproducibility and scientific honesty.
2 more timeline events
All Facts (12)
Developers are building an open-source framework called SIRFEX (Simulation-based Integrity, Robustness & Failure Exploration), aimed at simulation-based stress testing of autonomous controllers, finding reproducible failures, and evaluating whether improvements truly enhance robustness.
50%UnverifiedThe SIRFEX framework supports PPO training, compute-matched continuation experiments, failure-driven curricula with validation stress exposure, and an optional LangGraph/Groq research agent for proposing bounded training intervention suggestions.
50%UnverifiedIn a 51,200-interaction development study of SIRFEX, increased stress exposure improved tracking performance under hard conditions but frequently caused tracking degradation under normal conditions.
50%UnverifiedIn the seed 3 experiment, the PPO policy trained on normal data only achieved a nominal MAE of 0.0177 and a stress MAE of 0.1166; the 5% stress policy achieved a nominal MAE of 0.0296 and a stress MAE of 0.0654; the 10% stress policy achieved a nominal MAE of 0.0501 and a stress MAE of 0.0555.
50%UnverifiedThe controller trained only on normal data achieved zero physical violations on the development stress set.
50%UnverifiedDevelopers tried using GPT-OSS 120B to propose a retraining curriculum; in a matched development pilot on one seed, this approach performed worse than a simple fixed 5% stress policy, and neither passed the nominal performance preservation guardrail.
50%UnverifiedAs stress exposure increased, upper-part action saturation of the controller intensified, suggesting the policy may drift toward more aggressive pumping behavior.
50%UnverifiedFailure-driven PPO fine-tuning may be inferior to plain continuation training, with possible causes including distribution shift, reward-shaping side effects, and curriculum order sensitivity.
50%UnverifiedDevelopers have not yet determined whether the nominal performance degradation is caused by catastrophic forgetting, reward incentives, curriculum order, or other reasons.
50%UnverifiedThe SIRFEX framework runs on a local CPU and emphasizes reproducibility and scientific honesty.
50%UnverifiedSuggestions put forward by the developers include: nominal performance guardrails should be a hard constraint; failure scenario sampling needs to be balanced with normal scenarios; robustness metrics should go beyond violation frequency; negative results are worth publishing.
50%UnverifiedThe SIRFEX code is open-sourced on GitHub, and the developers explicitly state that criticism, related papers, implementation suggestions, and contributions are welcome.
50%