Glossary — Secure AI

Access package

A Microsoft Entra ID Governance bundle that grants time-bound, auditable, least-privilege access to a defined set of resources — security groups, Entra roles, and application permissions. For AI agents, access packages hand out scoped, expiring entitlements that clean themselves up, rather than standing access that lingers, and they standardize across whole fleets of similar agents.

Agent identity blueprint blueprint

In Microsoft Entra Agent ID, the reusable template that defines a class of agents and holds their credentials, minting individual agent identities on demand. Because the blueprint holds the credentials for the whole class, it — not the individual running agent — is the thing you protect and the blast radius if it is compromised, which is why recovery rotates the blueprint's credentials.

Agent registry

The tenant-wide operating inventory of AI agents in Microsoft Agent 365 — Microsoft-built, organization-built, external, ownerless, unmanaged, and at-risk agents — with aggregated risk signals. Visibility in the registry is the floor every other governance control builds on, though being listed is not the same as being under identity-based enforcement.

Agent risk

The low, medium, or high risk level Microsoft Entra ID Protection assigns to an AI agent by baselining its normal behaviour and flagging anomalies. Risk-based Conditional Access can consume the signal to block a compromised agent automatically, with no human in the critical path.

Agent sprawl

The uncontrolled proliferation of unmanaged or over-permissioned AI agents across an organization. Sprawl widens the attack surface, weakens least privilege, and erodes accountability — which is why agents need inventory, ownership, and lifecycle governance rather than ad hoc creation. Learn more

Agent user

In Microsoft Entra Agent ID, an optional user object added to an agent when it must also appear and act as a person — for example to hold a mailbox or a license. It is layered onto the agent identity only when needed, keeping most agents as pure workload identities.

Agentic agentic loop · agentic AI

Describes AI that does not just generate text but acts — reasoning toward a goal, calling tools, and taking multi-step actions with some autonomy, often called the agentic loop. This behaviour is what separates an agent from a chatbot, and why agents need their own identity, tool and MCP governance, and runtime protection rather than content controls alone.

AI Bill of Materials AI BOM · AIBOM

An inventory of the components that make up an AI workload — models, datasets, libraries, and services — used to understand provenance, exposure, and risk across the AI estate. It is the AI equivalent of a software bill of materials, and underpins posture management and supply-chain governance for AI. Learn more

AI gateway LLM gateway · generative AI gateway

A control point placed in front of model and agent traffic that authenticates callers, enforces token and rate limits, and governs which tools an agent may reach. It gives a central place to apply security, quota, and observability policy instead of trusting each client to behave — often implemented with Azure API Management. Learn more

AI management system AIMS

The set of policies, roles, objectives, controls, risk treatment, monitoring, and continual-improvement processes an organization uses to govern AI responsibly. It is the thing ISO/IEC 42001 certifies — an operating system for AI governance, not a technical control on any single model. Learn more

AI red teaming

The practice of deliberately attacking an AI system — with prompt injection, jailbreaks, and misuse scenarios — to find weaknesses before adversaries do. It is how you validate that guardrails, content filters, and runtime protections actually hold under pressure rather than only on paper.

Attack Success Rate ASR

A metric that expresses how often adversarial attempts succeed against an AI system, as a percentage of total attempts. Red-teaming tools use it to quantify a system's risk posture and to compare it before and after mitigations. Learn more

Attack surface

The full set of points where an attacker could try to enter, manipulate, or extract data from a system. Every tool, connector, data source, and agent-to-agent link an AI system adds expands its attack surface, which is why scoping and inventory matter.

Blast radius

The extent of damage that follows if a single identity, resource, or system is compromised — how far an attacker can reach from that entry point. Security teams shrink blast radius through least privilege, segmentation, and scoped access, and analysts map it during an incident to understand the true scope of a compromise.

Conditional Access

Microsoft Entra's policy engine for real-time, if-then access decisions. It evaluates signals — user or agent identity, device compliance, risk level, location — and then allows, blocks, or steps up access to a resource. For AI agents it only applies when the agent authenticates with an Entra identity; API-key and other non-Entra paths bypass it, so it is powerful but not a universal choke point.

Content filtering content filter · content safety

Guardrails that inspect AI inputs and outputs and block harmful, unsafe, or policy-violating content — categories such as hate, violence, and self-harm. In Microsoft's stack this is delivered by Azure AI Content Safety, layered in front of models and agents so unsafe generations never reach the user.

Critic agent critic agents

A secondary agent that audits another agent's inputs, outputs, or actions in real time to catch unsafe or manipulated behaviour. Critic agents are one layer in a defense-in-depth approach to agentic systems, especially in multi-agent workflows. Learn more

Customer-managed keys CMK · customer-managed encryption keys

Encryption keys that you create and control in your own key vault, used to encrypt a service's data instead of relying solely on Microsoft-managed keys. They give an organization the ability to govern, rotate, and revoke the keys protecting its AI data, keeping control of that data in its own hands. Learn more

Data classification

The practice of categorising data by how sensitive it is so the right protection can follow — for example public, internal, confidential, or highly confidential. Classification is the foundation that labels and data-loss controls depend on, because AI can only be restricted from data that has first been identified.

Data exfiltration data exfil

The unauthorised movement of data out of an organisation's control — copied, uploaded, pasted, or transmitted to a destination it should never reach. In an AI context, prompts and file uploads to unsanctioned AI apps are a common exfiltration path, which is why data-loss controls extend across browsers, endpoints, and network traffic.

Data Loss Prevention DLP

A control discipline that identifies sensitive data and stops users from sharing it inappropriately — warning, blocking, or restricting actions such as copying, uploading, or pasting. For AI, DLP boundaries prevent sensitive content from reaching Copilot, agents, and third-party AI apps.

Data poisoning model poisoning

Corrupting the data an AI model learns from — training, fine-tuning, or grounding data — so the model behaves in ways the attacker chooses. Poisoning can be targeted, to skew specific outputs, or indiscriminate, to degrade overall quality, and it often exploits untrusted or public datasets that are hard to audit.

Data retention retention

The policy-driven decision about how long information is kept before it is archived or deleted, balancing legal, regulatory, and business needs against risk. For AI, retention governs how long prompts, responses, and their source content persist and stay available for investigation.

Data Security Posture Management DSPM · DSPM for AI

A continuous, discovery-driven approach to finding sensitive data, assessing how it is exposed, and prioritising the risks that matter. DSPM for AI extends this to generative AI usage — surfacing which prompts and responses touch sensitive data so you can apply protection where AI actually creates exposure.

Data spillage

The exposure of sensitive information into a location or to an audience not authorised to hold it — for example a confidential file surfaced in an AI response or copied into an unmanaged app. Responding to spillage means finding every copy and preserving, purging, or remediating it.

Declarative agent

A customized version of Microsoft 365 Copilot, defined by declaring instructions, knowledge sources, and actions rather than writing the model or orchestration yourself. Because it runs on Copilot's own orchestrator, foundation models, and trusted AI services, it inherits Microsoft 365 Copilot's security, compliance, and Responsible AI protections — so securing it is mostly a data-access question, not an application-security one.

Defense in depth defence in depth

A security strategy that layers multiple independent controls so no single failure leads to compromise. For AI systems it means combining model, runtime, identity, and monitoring safeguards — assuming any one layer, such as a prompt filter, will eventually be bypassed.

Denial of Wallet DoW · wallet attack

An attack that drives excessive, costly requests at a metered AI service — directly or through an application — to run up the target's bill rather than to steal data. Because generative AI is billed per token or per call, sustained or anomalous request volume can inflict real financial harm without ever breaching the system.

eDiscovery electronic discovery · e-discovery

The process of identifying, preserving, collecting, reviewing, and exporting electronic information as evidence for legal cases, investigations, or regulatory requests. Applied to AI, eDiscovery covers the prompts and responses of Copilot and agents so they can be held, reviewed, and produced like any other record.

EU AI Act AI Act

The European Union's law governing artificial intelligence. It regulates uses of AI rather than models in the abstract, sorting systems into risk tiers — from prohibited practices through high-risk uses to minimal-risk applications — and attaching obligations accordingly. Like GDPR, it can apply to organizations outside the EU when their AI systems or outputs reach the EU market. Learn more

Excessive agency

The risk that an AI system can do too much, too autonomously, with too little control — for example an agent granted broad tool access, standing permissions, or the ability to act without approval. Listed as LLM06 in the OWASP Top 10 for LLM Applications, it is contained by least privilege, scoped tools, and human approval for high-impact actions. Learn more

Fine-tuning

Further training a base model on domain- or task-specific data to specialize its behaviour. Because fine-tuning bakes that data into the model, it is a data-governance and data-residency decision — sensitive or regulated information used for tuning effectively becomes part of the model.

Groundedness

A measure of how well an AI response is actually supported by its grounding data rather than invented. Groundedness checks — part of AI safety evaluation — flag responses that drift from the source material, helping catch hallucination before a user or an agent acts on a wrong answer.

Grounding grounding data

The practice of supplying an AI model with authoritative, context-specific data at query time so its answers are based on your organisation's real information rather than only its training data. Because grounding pipelines pull from live data sources, the sensitivity and access controls of that data flow directly into the model's responses.

Guardrails AI guardrails · agent guardrails

The layered safety and security controls placed around a model or agent to keep its inputs, outputs, and actions within policy — content filters, prompt attack detection, protected-material checks, and task-alignment controls. In Microsoft Foundry these are grouped as guardrails and controls applied across the model and the agent runtime. Learn more

Hallucination

When a generative AI model produces confident but false or fabricated information because it predicts plausible text rather than retrieving verified fact. Grounding and groundedness checks reduce it — and it becomes a security problem, not just a quality one, the moment an agent acts on a wrong answer.

High-risk AI system

Under the EU AI Act, a use of AI that is permitted but carries the heaviest obligations — such as risk management, data governance, logging, human oversight, robustness, and cybersecurity — because it affects areas like employment, education, credit, critical infrastructure, or law enforcement. The use case, not the model, decides whether a system is high risk, so classification comes before architecture. Learn more

Human-in-the-loop HITL

A control that requires a person to review, approve, or veto a high-risk or irreversible action before an AI system carries it out. It is the last line of defense when inputs are ambiguous or an agent could be manipulated, trading some automation for accountability on the actions that matter most.

Information Flow Control IFC

A technique that tags data with its trust level and enforces policy on where it can flow, so untrusted content cannot influence a sensitive decision or action. In agentic systems it keeps injected instructions in retrieved data from reaching planning or tool execution. Learn more

Insider risk

The risk that people inside an organisation — employees, contractors, or partners — cause harm through their access, whether by mistake or intent, such as leaking, stealing, or mishandling data. AI raises the stakes because a single prompt can expose or exfiltrate sensitive content, so insider-risk signals help spot risky AI behaviour.

Intelligibility system intelligibility

The degree to which an AI system makes clear what it plans to do, is doing, and has already done, including which tools and data it used. Without intelligibility, users cannot review, undo, or investigate an agent's actions. Learn more

ISO/IEC 27001 ISO 27001 · ISMS

The international standard for an information security management system (ISMS): the auditable framework of policies, controls, and risk management an organization uses to protect information. It is not AI-specific, but it is the security-management foundation many AI controls sit on, and the model ISO/IEC 42001 mirrors for AI governance. Learn more

ISO/IEC 42001 ISO 42001 · ISO/IEC 42001:2023

The international standard for an AI management system — the auditable policies, roles, controls, and continual-improvement process an organization uses to govern AI responsibly. It plays the same role for AI governance that ISO/IEC 27001 plays for information security: it certifies that a management system exists and works, rather than testing any single model. Learn more

Jailbreak jailbreak attack

An attack that manipulates an AI model into bypassing its safety guardrails or system instructions to produce restricted content or take disallowed actions. It differs from prompt injection — which smuggles in new instructions, often through grounding data or tool output — but the two are frequently combined, and both are why runtime protection sits in front of AI agents.

Just-in-time access JIT · JIT access · short-lived privileges

Granting an identity elevated permissions only at the moment they are needed and revoking them immediately after, rather than standing access. For agents it shrinks the window in which a compromised identity is useful to an attacker.

Large language model LLM

A neural network trained on vast amounts of text that predicts and generates language — the engine behind chatbots and most AI agents. Because an LLM reflects its training data and whatever context it is handed at runtime, its output is only as trustworthy as the data grounding it and the guardrails wrapped around it.

Least privilege principle of least privilege · least action

The principle of granting an identity only the minimum permissions, tools, and data access needed for its task, and nothing more. Applied to agents it means scoped, auditable identities and just-enough access, so a compromised or misbehaving agent can reach as little as possible.

Mean time to detect and respond MTTD · MTTR

Two operational metrics: how long it takes to detect an incident (MTTD) and how long to respond and contain it (MTTR). Purpose-built AI telemetry and AI-assisted investigation aim to shorten both for agent and model incidents.

MITRE ATLAS

A public knowledge base of adversary tactics and techniques against AI and machine-learning systems, modeled on the MITRE ATT&CK framework. It gives defenders a structured way to describe and simulate AI-specific attacks. Learn more

Model Context Protocol MCP

An open standard that defines how AI applications and agents connect to external tools, data sources, and services through a common interface. It lets a model discover and call tools at runtime instead of having integrations hard-coded, which is why securing MCP servers and the tools they expose is a core part of governing agents. Learn more

Model drift drift · model decay

The gradual change in an AI system's behavior or output quality over time as models are updated, prompts are refined, or the data it sees shifts away from what it was validated against. Because drift is silent, it is a governance risk: an agent that was safe and accurate at launch can degrade without anyone noticing unless it is continuously monitored.

Model evasion adversarial evasion

An attack that crafts inputs to make a model produce a wrong or attacker-chosen output — misclassifying content, bypassing a safety filter, or slipping past a detector — without changing the model itself. It is a recognized technique in MITRE ATLAS and a reason model decisions on untrusted input should not be treated as inherently reliable. Learn more

Model theft model extraction · model stealing

An attack that reconstructs a model or its behavior without authorized access — typically by querying it heavily and using the responses to train a copy, or by exfiltrating the model weights directly. Catalogued in MITRE ATLAS, it threatens intellectual property and can expose training data or reveal weaknesses for further attack. Learn more

NIST AI RMF NIST AI Risk Management Framework · AI RMF

A voluntary framework from the US National Institute of Standards and Technology for managing AI risk across the lifecycle. It organizes the work around four functions — Govern, Map, Measure, and Manage — giving security, legal, data science, and engineering teams a shared language for the same system. A companion Generative AI Profile applies the same structure to generative-AI-specific risks. Learn more

Non-determinism non-deterministic · probabilistic

The property that an AI system can return different outputs for the same input, because its behaviour is probabilistic rather than a fixed code path. It is why AI systems cannot be verified with a single test pass, and why threat modeling and incident response must assume a range of possible behaviours.

Non-human identity NHI · machine identity

An identity that belongs to software rather than a person — service principals, managed identities, workload identities, and now AI agents. Non-human identities increasingly outnumber human ones and often carry broad standing access, which is why agents need scoped, governed identities of their own.

Observability

The ability to understand what a system is doing from the signals it emits — logs, metrics, and traces. For AI, observability extends to AI-native signals such as prompts, retrieval sources, tool calls, and evaluation scores, so agent behaviour can be reconstructed, monitored, and investigated. Learn more

OpenTelemetry OTel

An open, vendor-neutral standard for generating and collecting telemetry — traces, metrics, and logs — from applications and services. It defines a common format so signals can flow to any compatible backend; AI agents increasingly emit OpenTelemetry spans that record each invocation, output, and tool call. Learn more

Overreliance

When users trust an AI system's output more than its accuracy warrants, acting on persuasive but potentially wrong answers. It is a human-centered risk that design controls — uncertainty signaling, disclosure, and human review — aim to reduce.

Oversharing

The condition where files, sites, or data are accessible to more people than they should be. It is a latent risk that becomes acute the moment Microsoft 365 Copilot or an agent can surface that content in a response, so reducing oversharing is a prerequisite for safely grounding AI on enterprise data.

OWASP Top 10 for LLMs OWASP LLM Top 10

A community-maintained list of the most critical security risks for applications built on large language models, such as prompt injection, insecure output handling, and data leakage. It gives teams a shared vocabulary for AI-specific threats when threat modeling and testing. Learn more

Plan drift detection plan drift

A runtime check that watches an agent's multi-step reasoning for deviations from its intended task flow. Catching plan drift early is a defense against indirect prompt injection, where hidden instructions try to steer an agent off its original goal. Learn more

Policy template

In Microsoft Agent 365, a reusable bundle of predefined Entra, Purview, SharePoint, and Defender controls applied automatically when an agent is activated — so governance is the default state for new agents rather than a manual checklist. Default templates set the baseline; custom templates add scenario-specific controls.

Prompt injection indirect prompt injection · cross-prompt injection · XPIA

An attack in which crafted text — supplied directly by a user or hidden inside content the model retrieves — manipulates an AI system into ignoring its instructions or performing unintended actions. Grounding and tool-using agents are especially exposed, since untrusted data can carry the injected instructions.

Prompt Shields

A Microsoft Azure AI Content Safety capability that detects and blocks prompt injection and jailbreak attempts — both direct user attacks and indirect ones hidden in documents, tool output, or grounding data — before they reach the model and steer it off course.

Responsible AI

An approach to developing, assessing, and deploying AI systems safely, ethically, and with trust. Microsoft's Responsible AI Standard frames the practice around six principles: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. These are design choices made from the first architecture sketch, not a review bolted on before launch. Learn more

Retrieval-Augmented Generation RAG

A pattern that retrieves relevant documents from a data source and feeds them to a language model as context before it generates an answer. RAG is the most common way to ground an AI application in enterprise data, which makes the security of the retrieval index and its source content a first-class concern.

Secure Access Service Edge SASE · SSE

A cloud-delivered architecture that combines network access with security controls — such as secure web gateway, cloud access security broker, and firewall-as-a-service — into one service applied at the edge. It gives an organisation a single point to inspect and control traffic to cloud and AI apps, wherever users work.

Secure Web Gateway SWG

A security service that sits between users and the web, inspecting outbound traffic to enforce policy — blocking risky sites, filtering content, and preventing sensitive data from leaving. It is a common enforcement point for detecting and blocking data shared with unsanctioned AI apps.

Security Information and Event Management SIEM

A platform that collects and correlates security signals — logs, alerts, and telemetry — from across an organisation's systems so a security team can detect, investigate, and respond to threats from one place. Modern SIEMs layer analytics, threat intelligence, and automated response on top of that centralised data.

Security Operations Center SOC

The team and function responsible for monitoring, detecting, investigating, and responding to security threats across an organisation. A SOC combines people, process, and technology — analysts, threat hunters, and incident responders working from centralised telemetry to limit the time and reach an attacker has.

Semantic index

The index that lets Microsoft 365 Copilot retrieve organizational content from Microsoft Graph using semantic, not just keyword, understanding of the data. It honors the user's identity-based access boundary — indexing never changes permissions, and the grounding process only returns content the person asking is already authorized to open — which is why keeping permissions clean is the primary control against Copilot oversharing.

Sensitivity label

A Microsoft Purview Information Protection label applied to content — and now to AI prompts and responses — that classifies how sensitive it is and can enforce protection such as encryption, access limits, and DLP. Labels let AI governance reason about data sensitivity, for example stopping Microsoft 365 Copilot or an agent from processing labeled files when grounding on data.

Shadow AI shadow IT for AI

The use of AI applications and agents that have not been reviewed, approved, or governed by the organisation. Like shadow IT before it, the risk is not the tools themselves but the sensitive data flowing into them with no visibility or policy — which is why discovery and governance come before blocking.

Shared responsibility model

The division of security duties between a cloud provider and its customer — the provider secures the platform, the customer secures how they configure and use it. For AI, it clarifies that Microsoft secures the service while the organization stays accountable for its agents, data, and human oversight. Learn more

Small language model SLM

A compact language model — such as Microsoft's Phi family — that trades some breadth for lower cost, faster inference, and the ability to run on-device or at the edge. SLMs make it practical to keep AI processing, and the sensitive data it touches, inside a controlled boundary.

In Microsoft Entra Agent ID, the human or group accountable for an agent's lifecycle and access decisions, required when the agent is created. The sponsor provides business accountability — who answers for the agent — separate from the technical owner who manages it day to day.

Spotlighting

A prompt-injection defense that marks untrusted content — retrieved documents, tool output, pasted text — so the model can tell it apart from its own trusted instructions and refuse to follow commands hidden inside it. It reduces the chance that data the agent reads can hijack what the agent does. Learn more

System prompt metaprompt · system message

The instructions given to a model that set its role, rules, and boundaries ahead of any user input. It is a primary control surface for AI safety — and a primary target for prompt injection, which tries to override or leak it.

Tabletop exercise

A rehearsed, discussion-based simulation of an incident that tests whether people, playbooks, and coordination hold up before a real event. For AI, tabletops should include novel harm types, ambiguous root cause, and high-volume output scenarios that traditional drills miss.

Task adherence

How reliably an agent stays within the task, plan, or objective a user actually intended, rather than pursuing an inferred goal or skipping required steps. Weak task adherence is a security concern, not just a quality one, because a drifting agent can take actions no one authorized. Learn more

Threat hunting hunting

The proactive search through security telemetry for threats that have evaded automated detection — forming a hypothesis and querying data to confirm or rule out malicious activity. For AI, hunting correlates prompt, agent, and identity signals to surface attacks that no single alert reveals.

Tool-chain analysis

Assessing and constraining the sequences of tool calls an agent can make, to block risky combinations even when each individual call looks benign. It limits the damage an injected or misaligned agent can do through its tools. Learn more

Trust boundary

A line in a system's architecture where data or instructions cross from a less-trusted zone into a more-trusted one and must be validated. AI raises the stakes because models can treat untrusted retrieved content as executable instructions, so separating instructions from data across trust boundaries is a core defense.