Security review checklist for an AI feature

There is a data-flow diagram showing every place prompts, retrieved content, outputs and logs travel, including the model provider.

Most findings in a review come from a flow nobody drew.

Every data source the feature reads is listed with its sensitivity label and its owner.

Restricted data is either excluded from indexing or protected by per-user filters, decided and written down.

You know what the model provider stores, for how long, and whether it is used for training. The contract says so.

The feature acts as the signed-in user (on-behalf-of), not as a shared service account, wherever it reads or writes user data.

A shared robot account sees everything, so the assistant can leak across permission boundaries.

Retrieval filters documents by the user's permissions before ranking, not after the answer is written.

Each tool has its own scoped credential with the least privilege it needs.

Admin and configuration endpoints are separate from user endpoints and require stronger authentication.

Retrieved documents, web pages, emails and tool results are treated as data, never as instructions.

Indirect prompt injection arrives through content, not through the chat box.

The system prompt does not contain secrets, internal URLs or anything you would not want printed.

There is a test set of injection attempts (direct and through documents) that runs on every prompt or model change.

Model output is encoded before it is rendered, so it cannot inject HTML, scripts or markdown links that exfiltrate data.

Every tool with side effects is listed, with what it can change and who approves it.

Irreversible or external actions (send, pay, delete, publish) require human approval.

Agents have hard limits on steps, spend and time, and stop cleanly when they hit them.

Tool calls with side effects carry idempotency keys so a retry cannot repeat the action.

Model and tool keys live in a vault, are injected at runtime and are never in the repo, the prompt or client code.

Keys can be rotated without a deploy, and you know who to call when one leaks.

Model, search and tool endpoints are reached over private networking where the platform supports it.

Egress from the agent runtime is allow-listed, so a manipulated agent cannot call arbitrary URLs.

Every tool invocation is logged with the user, arguments, result and time, in a store the feature cannot edit.

Logs that contain prompts or outputs follow the same retention and access rules as the source data.

Prompt logs quietly become the most sensitive dataset you own.

You can reconstruct any answer from its trace, including the retrieved passages.

Model versions are pinned, and a version change goes through the same evaluation as a code change.

The provider's region, data handling and sub-processors have been checked against your requirements.

Rate limits exist per user and per tenant, and cost alerts fire before the monthly budget is gone.

Content filters are configured for your use case and their false positives have been looked at.

There is a kill switch that disables the feature, or just its actions, within minutes.

A runbook covers prompt injection, data leakage and provider outage, with an owner for each.