AI governance defines how AI decisions and actions are controlled, evaluated, and owned. Product teams make it tangible through understandable permissions, visible evidence, meaningful confirmations, and recovery paths.
From policy to behavior
NIST’s AI Risk Management Framework provides a voluntary foundation for managing AI risks. It is a useful reference for organizing the conversation. My recommendation is to translate that conversation into specific user journeys and observable system behavior.
A policy saying that an agent must protect customer data is incomplete as a product requirement. The team needs to define which users can request which actions, what information the agent can access, and how denial or uncertainty is communicated.
Design around consequential actions
Make a distinction between reading information, preparing a change, and committing a change. Those actions may require different permissions even when they happen in the same conversation.
Imagine an assistant preparing a refund. It should make the proposed amount, account, and rationale clear before the commitment step. The required approval should follow the organization’s actual policy. This example illustrates an interaction pattern, not a universal permission rule.
Make trust inspectable
Show relevant sources, the scope of access, and the status of an action. Avoid a generic message that says everything is done when the agent only prepared a draft. A user should be able to understand what happened and what remains.
Product teams should also make the unhappy paths usable. Explain permission denials without exposing sensitive information. Provide a path to the right owner. Keep evidence that supports investigation when an action fails.
Measure the governed experience
Track task completion alongside unauthorized attempts, recoverability, and user confusion. Review where confirmations add safety and where they create repeated interruptions without reducing risk.
Governance should help people act with confidence. A blocked user is not automatically a protected user; a successful action is not automatically an authorized one. Test both the control and the experience around it.
October 2026: examine the oversight system, too
METR’s October 14, 2026 control-assessment proposal asks whether internal agents could cause serious harm without being stopped. It separates monitoring coverage, vulnerabilities in the monitoring system, evasion, human response and potential impact. This is a proposed assessment approach for frontier AI developers, not a certification or evidence that a particular product is safe.
On October 15, Anthropic described granting a small METR team employee-like access for evaluation. Its October 16 clarification describes publication rights alongside predefined redaction categories, with Anthropic deciding what falls into those categories. The company’s announcement describes its arrangement; it does not independently establish that its safeguards work.
The product implication is a question your team can investigate: what evidence would let a reviewer challenge an agent’s behavior? A dashboard showing alerts is insufficient if some actions never reach it, or if nobody can stop an action after an alert. Treat access to evidence, response ownership and the ability to intervene as explicit acceptance criteria.
A 15-minute mentoring exercise: follow one risky action
Choose a real workflow such as an agent proposing a production configuration change. Work with a peer from engineering, security or operations. This exercise is an editorial adaptation for product teams, not METR’s assessment methodology.
Minutes 0–5: draw the path from request to tool call to committed change. Name the accountable human, the permitted actions and the point beyond which recovery becomes costly. Mark which steps are prevented by permissions, which require approval and which are only observed afterward.
Minutes 5–10: write the evidence needed at each step: requester identity, authorization decision, proposed change, executed result and escalation record. Ask whether a subagent or alternate tool could bypass that evidence path. Use a synthetic test environment; do not disable production controls to demonstrate a gap.
Minutes 10–15: rehearse one denied action, one monitoring outage and one unanswered escalation. Decide what the system does in each case, who can stop it, and what evidence proves that behavior. Agree one acceptance test and an owner before increasing autonomy.
Bring the map to a mentoring conversation. The useful output is one unresolved tradeoff and a testable next step. For example: a simulated unapproved write must be denied and recorded; if the record cannot be created, the write remains blocked. Choose response-time targets from your actual risk and operational constraints.
Questions worth asking
Who owns AI governance in a product team?
Ownership is shared across relevant functions. The product team should name accountable owners for each consequential workflow and make the controls visible in the experience.
Should every AI action ask for confirmation?
No. Match approval requirements to permissions, consequences, reversibility, and organizational policy.
Does human oversight mean every action needs approval?
No. Design permission boundaries, evidence and escalation around the consequence and reversibility of each action. Test what happens when monitoring or a reviewer is unavailable.
Sources & further reading
- NIST — AI Risk Management Framework
- METR — Control and monitoring assessment proposal (14 October 2026)
- Anthropic — Independent evaluators (15 October; clarified 16 October 2026)
These sources inform the context. Recommendations and frameworks are editorial analysis, not claims of proven results. AI-assisted draft prepared for Yaron’s review; personal experience claims require his confirmation.