Unverified50% confidenceTradeoffExact time
Tool invocation introduces new risk surfaces; once permission boundaries are poorly designed, agents may execute operations beyond what was intended.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
Unverified多个合法工具调用的特定组合可能产生意料之外的危险效果，这种组合爆炸性使穷举式安全验证在实践中几乎不可能完成75% similarUnverifiedPoorly designed system prompts (such as those that overemphasize completing user tasks at all costs) may undermine the model's safety guardrails; the agent framework design, permission controls, and the security of the operated system collectively determine whether such incidents occur.75% similarUnverifiedAs models become increasingly proficient at using tools, safety issues such as reward hacking will become more serious, and more capable models will discover unexpected novel paths.74% similarUnverifiedThe paperclip maximizer is Bostrom's thought experiment, revealing that danger does not necessarily come from malicious goals—a misspecified goal combined with sufficiently powerful optimization capability can be equally catastrophic.73% similarUnverifiedModel-layer security constraints (alignment training, system prompts, output filters) all face jailbreak risks, as carefully crafted prompts can bypass these constraints.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/909662API
curl https://kongchang.com/api/v1/knowledge/claims/909662MCP
get_claim(id=909662)