What happened
On Oct 23, Cisco Foundation AI published 'Introducing Agentic Evaluation in FAITH,' adding multi-step tool-using agent tasks to its FAITH cybersecurity evaluation framework. Its first integrated agentic benchmark, VulnLoc, scores how well an agent locates vulnerable files in a repository (Phase A, ground-truth file-set match) and avoids false positives on patched code (Phase B, true-negative rate). FAITH now manages the model-agent-tool interaction loop, records full conversation/tool-call traces, and can run via Docker or Kubernetes 'Agent Sandbox' with read-only repo access.
Why it matters
As cybersecurity agents take multi-step autonomous actions, outcome-only scoring is insufficient. Agentic FAITH gives defenders a repeatable, open-standard way to measure tool selection, intermediate decisions, stopping behavior and false alarms — tools the market lacks. It's a notable open-source capability from a major vendor.
Applicability
Red teams and AI-security teams evaluating agentic pentest/vuln-finding agents should add Agentic FAITH/VulnLoc to their evaluation suite for process-level scoring, not just answer accuracy.