Agentic Security Plugins: Real Defense or Benchmark Theatre?

The landscape of agentic security is shifting from theoretical research to active deployment. Cognition has open-sourced its Devin Vulnerability Fixes plugin, targeting RCE vulnerabilities in PHP, Node.js, and GitLab by automating patch verification and pull request generation. Simultaneously, Anthropic has introduced sandbox-runtime, an OS-level isolation layer that replaces heavy containerization with native tools like macOS’s Seatbelt and Linux’s bubblewrap to secure autonomous agent workflows. These tools represent a move toward "security-by-default," yet the industry remains split on whether these automated remediations are robust enough for production environments or if they simply introduce new, non-transparent attack surfaces.

This push for automation mirrors the broader, more inconsistent reality of agentic testing seen elsewhere. While startups like DrizzVision attempt to replace human mobile app testers with visual-recognition agents, current benchmarks show highly variable performance. We are left with a dichotomy: specialized, high-stakes security plugins that promise to reduce manual labor, and generalized testing agents that struggle to maintain reliability in real-world scenarios. The question remains whether these new security plugins offer verifiable, long-term defense or if they are merely sophisticated "benchmark theatre" designed to impress in controlled demos while failing to account for the chaotic edge cases of complex production systems.

What we're arguing about

1. Have you implemented automated remediation tools like the Devin plugin in a production environment, and did they successfully catch vulnerabilities that manual scanning missed?
2. Does the move toward container-free security layers like Anthropic’s sandbox-runtime provide enough isolation to trust an agent with root-level access to your infrastructure?
3. Are autonomous testing agents actually reducing your QA overhead, or are you spending more time debugging the agent’s inconsistent visual recognition than you would have spent on manual scripts?

Share your recent experiences with integrating agentic plugins into your security or testing workflows.