.png)
Your Incident Response Team Has an AI Problem to Rehearse
Security teams already know how to investigate compromised accounts and suspicious endpoints. AI systems introduce evidence, failure modes and containment decisions that many teams haven’t practised yet.
ITSEC Asia
An AI assistant connected to internal data starts returning information it shouldn’t reveal.
What does the incident responder collect first?
Traditional evidence still matters, but the investigation may also require prompts, model outputs, retrieval sources, system instructions, access permissions and records of actions taken through connected tools. Disabling an endpoint won’t necessarily answer what the model accessed or what information it produced.
Recent workforce data suggests many teams haven’t rehearsed that situation.
ISACA’s 2026 State of Cybersecurity research, based on more than 1,800 cybersecurity professionals globally, found that only 8% of organizations regularly conduct AI specific response exercises. The same research found that 45% see LLM SecOps as a skills gap, up 12 percentage points from 2025 and 21 points from 2024. ISACA
AI security is becoming operational work rather quickly.
AI Changes the Evidence
NIST convened an AI Incident Management Workshop in May because a new class of incidents is emerging as AI systems become embedded in critical infrastructure, cybersecurity and other operational environments. Its work covers incident definitions, lifecycles, taxonomies, existing playbooks and gaps in current cybersecurity guidance. NIST
For security practitioners, that means knowing what evidence exists around an AI system.
Depending on the implementation, responders may need to examine:
- Prompts, outputs and conversation histories
- Retrieval sources and data accessed by the system
- Model and application configuration
- Identity and permission records
- Connected APIs, tools and automated actions
- Monitoring data before and after abnormal behaviour
- Human approvals or interventions
The investigation may involve cybersecurity, AI engineering, data, privacy and business teams simultaneously.
That’s quite a crowd for an incident nobody has rehearsed.
Containment Gets More Complicated Too
Suppose an AI agent can read documents, send messages and update a business application. Suspicious behaviour appears.
Should the organization disable the entire system? Remove one tool permission? Restrict access to certain data? Roll back a configuration? Keep the system running in a reduced mode so investigators can observe it?
Those are technical and business decisions.
ISACA reports that cybersecurity teams are already becoming more involved in AI itself. Fifty-one percent of respondents said they or their teams were involved in developing, onboarding or implementing AI solutions, compared with 40% in 2025 and 29% in 2024. ISACA
The people responsible for security therefore need experience responding to failures in systems they increasingly help deploy.
Put the AI Incident Into the Exercise
A useful exercise could start with a simulated AI assistant that suddenly retrieves confidential information outside its expected scope.
Participants receive logs, prompts, access records and system configurations. They must determine what happened, establish the possible blast radius, preserve evidence and decide how to contain the system without unnecessarily stopping the business process.
Then complicate it.
Perhaps the AI has already called another application. Maybe its output was copied by a user before the alert appeared. Perhaps the original behaviour can’t immediately be reproduced.
Exercises like these fit naturally into hands on learning at ITSEC Cyber & AI Academy, where AI and cybersecurity skills can be developed through scenarios that require investigation and decisions rather than passive familiarity with terminology.
If the first AI incident exercise happens during the first real AI incident, the training schedule has slipped rather badly.
Explore practical cybersecurity and AI training at ITSEC Cyber & AI Academy.
