An assessment that claims to see everything gets audited for what it missed. One that claims a single number gets audited for whether the number is right.
Why the AI Security Assessment Matters Now
Every CISO is being asked some version of the same question by a board, an auditor, or a regulator: what AI is running in our environment, and what can it reach? The usual response is a questionnaire, a vendor risk spreadsheet and a slide with a heat map. It produces a document. It rarely produces a position you can mark to market.
The reason is structural. A traditional assessment inventories what was procured. Agents are not procured; they are installed, pulled from GitHub, enabled by a vendor release, or built by a citizen developer on a Tuesday. Ospiri’s published research puts the scale of the gap in plain numbers.
| Signal | Figure | Source |
|---|---|---|
| Enterprises with agents in use that IT did not sanction | 88% | Ospiri published research |
| Added breach cost attributed to shadow AI | +$670K | Ospiri published research, directionally consistent with IBM Cost of a Data Breach |
| Typical time to put agent governance in place | 12-18 months | Ospiri published research |
If the unsanctioned population is the majority, an assessment built on the sanctioned list is measuring the wrong book.
Five KPIs vs One: The Comparison
The instinct in security is to be comprehensive. Five dashboards, twelve scores, a maturity radar. Let’s step back and ask what a CISO does with that on Monday morning. Usually nothing, because nobody can say which number to move first.
| Dimension | Multi-KPI assessment | Single-snapshot assessment |
|---|---|---|
| Question answered | How mature are we, across everything? | How much local agent risk do we hold today? |
| Typical output | Radar chart, scorecards, 40-page report | One number with three drivers |
| Defensibility | Challenged on every axis | Challenged on one definition |
| Time to first result | Quarter | Days |
| Repeatability | Hard to re-run | Re-run weekly, trend the delta |
| Board readability | Low | High |
A single KPI is not a simplification of the problem. It is a decision about which question you will be held to.
What the Snapshot Actually Measures
The workable unit is the local agent footprint: what is installed on endpoints, what it can reach, and what it can load. Three components, each observable without reading anyone’s prompts.
- Credentials reachable. Tokens, keys and cloud profiles an agent process can read from the machine it runs on. This is the blast radius in its most literal form.
- Data sources touched. The repositories, drives, databases and SaaS tenants that agent activity actually reaches, as opposed to what a policy says it should reach.
- Skills and MCP inventory. Which skills files are loaded, which MCP servers are configured, whether they are signed or approved, and where they came from.
The reason to stop there is discipline. Each component is a count with an owner, and each count can be re-taken next week.
The Local Agent Risk Score
Frequency times severity, the way a risk desk would size any position:
Local Agent Risk = (Credentials Reachable × Data Sources Touched) + (Unapproved Skills and MCP Servers × Egress Paths)
| Factor | What you measure | Where it comes from |
|---|---|---|
| Credentials Reachable | Distinct secrets readable by agent processes per endpoint | Endpoint inventory of agent runtimes and their file access |
| Data Sources Touched | Distinct systems agents reached in the window | Path, host and tool-call metadata derived on device |
| Unapproved Skills and MCP Servers | Loaded capabilities with no approval or signature | Skills file and MCP configuration scan |
| Egress Paths | Destinations an agent can push output to, including personal storage | Network and process observation |
Notice what is absent: a verdict on intent. That is deliberate.
Defer Malice Classification
Here’s the bet. The first assessment should not try to say which agents are malicious. Classifying malice requires a threat feed, a false-positive process and an argument about definitions, and it invites the worst possible follow-up question: why did you miss the one you never claimed to look for?
A first-phase assessment states what it measures and what it does not. Phase one counts exposure. Phase two classifies behavior. The separation keeps the scope honest, which is the property that lets the result survive a skeptical read. For more on stating coverage limits up front, see the gap report test.
| Phase | Claims | Does not claim | Artifact an auditor can hold |
|---|---|---|---|
| 1. Snapshot | Reachable credentials, data sources, skills and MCP inventory | Which agents are hostile | Dated inventory with counts |
| 2. Classification | Verdicts on skills, servers and behavior | Prediction of unseen threats | Verdict log with evidence |
| 3. Enforcement | Policy applied at runtime | Zero residual risk | Allow and block record |
What CISOs Should Do This Quarter
| Step | Action | Output | Effort |
|---|---|---|---|
| 1 | Pick the single KPI and write its three-line definition | Signed-off metric definition | 1 day |
| 2 | Run a local agent footprint snapshot on a representative endpoint cohort | Dated baseline with counts | 1 week |
| 3 | Assign an owner to every unapproved skill and MCP server found | Named ownership list | 1 week |
| 4 | Re-run weekly and report the delta to the risk committee | Trend line, not a one-off | Ongoing, low |
Map the result to the frameworks your auditors already use, such as NIST AI RMF and ISO 27001 controls, rather than inventing a new vocabulary. The deployment sequencing is covered in the 90-day runbook.
The Bottom Line
An AI security assessment earns its keep when it reduces the estate to one number a CISO can defend and move. Count credentials reachable, data sources touched, and the skills and MCP inventory, then leave malice scoring for phase two. If your team is sizing this for the next quarter’s risk committee, request a working session. We will walk through your environment, scope the endpoint cohort and the three-driver snapshot, and map the output to your audit calendar. Expect a dated baseline within the first two weeks.