Situation

Four services and an agent run on the platform, each in your tenant's namespace, each registered, each reached through the model gateway. Northwind's support desk does not care that they run. It cares what a resolved ticket costs, whether a bad release can reach every customer and how fast it comes back, who approved what went live, and whether it can show a regulator where an EU customer's data went. The capstone puts Project 1 in front of the agent as a cheap pre-filter and measures cost per resolved ticket before and after; promotes an approved model and an image to the live target behind a canary and takes one back on purpose; sets the objectives and alarms that decide when to stop; hands the operator's controls to a person; and puts the governance documents beside the code they describe.

How this part runs

Stage	Steps	What you leave with
Orient	The whole system, where you stand on it, and what it costs	The system read end to end, your platform state checked (and recovered), three cost numbers
Build	Route cheap first; cost per resolved ticket, before and after	The router, and a measured cost per resolution both ways
Promote	The promotion drill; images through the delivery pipeline, and the rollback drill	A model live behind a canary, an image through delivery, one canary taken back
Operate	SLOs, burn rates, quality signals and the dashboards; the operator's controls	An objective with its burn rate, an approval executed once, a key rotation read, the kill switch
Govern	The DPIA, the risk register, residency and retention	Answers a privacy office and a buyer ask, each with its document
Scale out	Lower and higher environments	The path to a second environment, drawn (and on Local, run)
Close	Part check, demos, debrief	Green checks, a pushed commit, `NOTES.md` as your runbook

sessions/session-06/NOTES.md is the file every step writes into. Its template has the sections for cost, promotion, rollback and the SLO; you add "Operations" and "Governance".

Modes

	Cohort mode	Solo mode	Local track
Who deploys the platform	The instructor, once per track, before the pre-work	You, in the pre-work, into your own account, project or subscription	You, `make local-up`
What you are	A tenant: your handle is the prefix of every resource; your own API key, gateway key and budget	The tenant `solo` on a platform you own	The tenant in `.env`, `solo` unless you are practising a cohort
The live target	Shared; the instructor promotes a volunteer's approved version while the room watches (on Azure tenants hold no role on it at all)	Yours	Yours, plus the `higher` tier
The delivery pipeline	The instructor's; you read its state	Yours if you set up the CodeConnections connection, the Cloud Build GitHub connection or the Azure DevOps service connections; otherwise the laptop commands	`make local-up`, `make local-canary`, `make local-promote`
Approvals	Your tenant's proposals are yours to approve; `live` belongs to the approvers	All yours	All yours, no second enforcement
Stop and destroy	The instructor, after each part and after the cohort	You, before you leave	`make local-down`
Cost	The credit to request per learner on the cost sheet	The solo line on the cost sheet	No cloud money; 32 GB of memory on a Mac for gpt-oss:20b, or 16 GB with `SMALL=1` and qwen3:4b; about 60 GB of Docker disk

Route cheap first

The router, nw/agent/router.py, is an agent with its own use case, uc-router, its own card listing both model roles, its own DPIA section. A confident P0 goes straight to a proposed escalation with no model call. A P3 question goes to the Economy role with a small step cap and budget. Everything else runs the agent loop on the Workhorse. The tools are bound to the request's account for the whole route, and the P0 proposal is approved with the same command as any other. The same twenty tickets, before and after, give a number; the course's split on its triage artifacts is 16 Workhorse, 2 Economy, 2 P0 direct. When the meter and the gateway disagree, the gateway bills and the meter estimates.

Promotion and rollback, per track

What moves a triage model or an image to the live target, and what takes it back.

	Approval	Live target and canary	Judged by	Rolls back
Model, AWS	`set_stage(..., APPROVED)`; the approval event deploys your serverless endpoint	`set_stage(..., LIVE)` re-registers into `northwind-live-triage`; SageMaker blue/green, 10 percent of capacity for 5 minutes	`northwind-live-triage-5xx`, `-p95`, `northwind-triage-quality-level`; the `-drift` alarm and the other quality alarms page only	Automatically on those three alarms; after a finished promotion, `set_stage(previous, LIVE, reason)`
Model, Google Cloud	the alias `approved` in Model Registry; your Cloud Run service rolls a revision	Vertex endpoint `northwind-live-triage`: canary at `canary_percent`, then `promote` undeploys the old model	the dashboard's endpoint panels, the Model Monitoring anomaly alert	A person: `endpoints.rollback`; after a finished promotion, deploy the previous version again
Model, Azure	the `stage` tag; your endpoint's `blue` deployment	`nw-live-triage-<scope>`: the colour that is not stable at `canary_percent`, then `promote` deletes the old colour	`northwind-live-5xx`, `northwind-live-p95`	A person: `endpoints.rollback`; after a finished promotion, deploy the previous version again
Model, Local	the alias `approved` in MLflow	`serving-canary` at the proxy weight, then `live` on `serving-stable`	Grafana, read by you	A person: `make local-canary WEIGHT=0`
Images, AWS	the `Approve` stage (action `Promote`) of `northwind-delivery`, `make approve-aws`	signed with AWS Signer, verified with Notation; the policy alias through CodeDeploy, 10 percent for 15 minutes; the live resolver on the new digest with no canary	`northwind-policy-errors`, `-p95`, `northwind-policy-quality-level`	Automatically on those three; the resolver's alarms page
Images, Google Cloud	`make approve-gcp` before any traffic moves, and again to advance	signed and verified with cosign keyless in Cloud Build; Cloud Deploy canary on the private live Cloud Run services	the server errors alert, the quality and drift alerts	A person: `gcloud deploy targets rollback`
Images, Azure	`make approve-azure`, which refuses while a live alert fires, or the Approvals check on `northwind-live`	signed with cosign and a Key Vault key, verified before release; a new revision of the live policy and agent apps at 10 percent	`northwind-live-policy-5xx`, `-agent-5xx`, `northwind-live-quality-level`	A person: traffic back to the stable revision
Prompts	`set_stage(..., APPROVED)` in the prompt store	`Stage.LIVE`; the hash in every answer proves which version served	the policy regression gate	a new stage change with a reason
Agents	the card approved in the registry (on AWS by the approvers role as curator)	the live runtime takes the approved digest through delivery	the agent's alarms, sampled Judge scores	the kill switch, `NW_AGENT_DISABLED=1`

Automatic rollback exists only on AWS, and only on errors, latency and the quality level: never on drift, because an hourly monitor cannot judge a five-minute canary, and never on the other quality signals, which page with the value that moved. On Google Cloud and Azure the alerts tell a person, and on Azure they also block the approval. The Local track has no alerting. Making a version live retires the previous live one on every track.

Recovery. The solutions branches restore code, not platform state. When a registry holds nothing to promote (promotion_candidate raises LookupError naming what it holds), make bootstrap-aws, bootstrap-gcp or bootstrap-azure (NAMES=triage, FORCE=1 when only a live version exists) registers artifacts/<name>/latest with source=bootstrap and the pipeline's lineage tags and approves it with a reason saying it is a recovery. On the Local track, make pipeline-submit PIPELINE=triage ARGS="--wait" registers a pipeline candidate, and make local-bootstrap makes latest live on a fresh stack.

The system and the DevOps figures

The system figure above (one per track) is the whole Northwind support intelligence system in five rows: the control plane with governance; the pipeline a ticket takes behind keys and the rate limiter; knowledge and state, the ops store among them; operations with the canary; the runtime and security underneath. The DevOps figure is how a change reaches the live target on your track: pull requests checked in GitHub Actions, a merge built once and pushed by digest, a signature, a manual approval, a canary behind the alarms, one environment, and how the same pipeline reaches a higher one.

Lower and higher environments

The course builds one environment per track and describes the rest. An organisation deploys the same code once per environment, parameterised by the environment name (NW_ENV in the CDK stack, environment in Terraform, NW_ENVIRONMENT for the Bicep, the higher profile in compose), and changes three things:

A second target. The delivery pipeline lives in the lowest environment or a tooling account and gains one stage per higher environment, each with its own approval and canary: a Deploy action per environment in CodePipeline, a second google_clouddeploy_target promoted with gcloud deploy releases promote, a stage per environment in Azure Pipelines, the higher serving tier locally.
A deployment role. Each higher environment holds a deployer (northwind-<env>-deployer on AWS, a deployer service account on Google Cloud, a <env>-deployer managed identity with a federated credential on Azure) that trusts exactly one principal: the pipeline's role in the lower environment. Nothing else in the lower environment can write to the higher one.
Promotion by reference, never by rebuild. Images move by digest (ECR replication, a shared Artifact Registry, az acr import) and are verified against the same signing key; model versions are copied with the same artifact URI and carry where they came from (promoted_from locally). The artifact that passed the gate is the one that goes live.

The shared services (registry, gateway, observability, identity) move to a shared-services account or project; tenants stay in dev; production holds only live.

The platform components this part used

Contract or box	Local	AWS	Google Cloud	Azure
`ModelRegistry`	MLflow registry, aliases `candidate`, `approved`, `live`, `retired`	SageMaker Model Registry, approval status and `nw:stage`	Model Registry with aliases and a stage log	Azure Machine Learning registry, the `stage` tag
`EndpointClient`	MLflow model serving, two replicas behind nginx with weights	Serverless Inference per tenant; a real-time endpoint with blue/green for `live`	Cloud Run per tenant; a Vertex endpoint with a stable and a canary model for `live`	a managed online endpoint per tenant; blue and green with tags for `live`
`AgentRuntime`	the `resolver` container	AgentCore Runtime in VPC mode	Agent Runtime (formerly Agent Engine), tools through the private MCP service; the live agent on Cloud Run	Foundry Agent Service hosted agents; the live agent on Container Apps
Ops store (`NW_OPS_STORE`)	files under `artifacts/`	the ops bucket	the artifacts bucket, managed folders	the lake's `artifacts` container
Model gateway	LiteLLM with a key and budget per tenant	LiteLLM on ECS Fargate behind CloudFront	LiteLLM on Cloud Run	API Management's AI gateway
Image registry	`registry:3` on :5050	ECR, tags immutable	Artifact Registry, tags immutable	Azure Container Registry
Delivery	`make local-up`, `make local-canary`, `make local-promote`, the `higher` profile	CodePipeline, CodeBuild, Signer, CodeDeploy	Cloud Build, cosign, Cloud Deploy	Azure Pipelines (or GitHub Actions with OIDC), cosign, Container Apps revisions
Monitoring	Evidently reports over the capture files	Model Monitor (for accounts that already use it), the services' own signals	Model Monitoring, the services' own signals	the services' own signals; Azure ML monitoring has nothing to read yet
Tracing and metrics	OpenTelemetry collector, Jaeger, Prometheus, Grafana	X-Ray, CloudWatch with EMF	Cloud Trace, Cloud Monitoring with log-based metrics	Application Insights, Log Analytics, the workbook
Pull-request checks	GitHub Actions in your fork	GitHub Actions	GitHub Actions	GitHub Actions

DevOps: the ops loop

What is versioned. The image by digest, signed; the model version in the registry with its stage, reason and lineage tags (source, git_sha, image_digest, the lock and source hashes); the prompt version with its hash; the agent version on its card; the dataset version in data/MANIFEST.json; the OpenAPI snapshots in docs/openapi/ as the contract the tools depend on.

What is measured. Per service: requests, errors, the latency histogram, cost, drift level and the quality gauges; for the agent, runs by termination, tool errors and a sampled Judge score. Exported once a minute on the cloud tracks, scraped locally. Per tenant: spend on the gateway. Per release: the canary's alarms.

What gates a release. On a pull request, in GitHub Actions: tests, lint, the free retrieval gate, the generation gate on prompt changes, the offline tiered agent gate, the catalog and registry checks, Trivy, the SBOM, pip-audit, gitleaks, cosign signing, the OpenAPI and data manifest checks. On delivery, in the platform's pipeline: a verified signature, a human approval and a canary with its alarms. While the error budget in deploy/SLO.md is spent, nothing promotes.

What an operator can do. Roll the canary back; set the weight or the split to the stable version; approve or refuse proposals from the ops store; rotate a key (make rotate-key TRACK=aws|gcp|azure TENANT=<handle>|live|all, with DRY_RUN=1 first; it refuses during a traffic split); NW_AGENT_DISABLED=1; the concurrency cap; the spend cap and the gateway budget; make stop-* for the idle meter.

Service level objectives and alerting

deploy/SLO.md is the platform's promise on all four tracks, in three kinds: service (availability and warm latency as ratios of good events to valid events over 28 days, and cost per resolution against the track baseline), quality at release (the gates, with counts, Wilson intervals, tolerances in cases and paired tests) and quality in production (the canary signals every service exports, each waiting for its window). It specifies multiwindow burn-rate alerting (page at 14.4 over one hour and five minutes, ticket at 6 over six hours and thirty minutes, nothing below 100 valid events), bake time sized to traffic, and the error budget policy: while the budget is spent, promotions freeze, gates stay, and the operator's controls are used in order. The alarms deployed today are threshold alarms on counts, p95 and the quality level; the burn-rate alerts are specified, not yet deployed, and the capstone computes one by hand.

Quality alarms per service, named <environment>-<service>-quality-{level,shadow,p0-high,p0-low,refusal,judge} on AWS and Azure and "quality ... past its bar" on Google Cloud, page with the signal that moved. Only the level, the monitor's own verdict with its minimum sample and interval, stops a canary: it rolls one back on AWS and blocks make approve-azure on Azure (northwind-live-quality-level).

MLOps maturity, stated plainly

Google's MLOps article (https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) names three levels: 0, a manual process; 1, ML pipeline automation, where training runs as a pipeline that validates, trains, evaluates and gates, a registry holds the result and retraining can run without a person; 2, CI/CD pipeline automation, where the pipeline itself is built, tested and delivered automatically, triggered by monitoring. The course builds the Level 1 pieces for Projects 1 and 2: the pipelines and their gates, the registries, a human approval and monitoring. Continuous training is triggered by new data today; drift and label-volume triggers exist in code and fire once an operator schedules the central drift and quality jobs. The retraining schedules ship off on every cloud (the EventBridge rules disabled, the Cloud Scheduler jobs paused, the Azure schedules not enabled), and retrain-triage.yml runs weekly but skips when nothing changed. Promotion always needs a person. The course does not claim Level 2, and a team on the full platform that promotes by hand is still describing how it works, not which services it bought.

Governance

docs/governance/ holds the documents a regulated buyer asks for, written for this system: the DPIA (dpia.md, GDPR Article 35, a template filled per use case, checked by the catalog), the risk register mapped to the twelve risks of NIST AI 600-1 (risk-register.md), the datasheets for every dataset (datasheets.md), the retention classes and the erasure runbook (retention-and-erasure.md) and the residency and subprocessor table (residency-and-subprocessors.md). docs/SECURITY.md has the threat model, what is and is not redacted, and the limits of tenant isolation per cloud. make governance-check runs the catalog (EU AI Act class, DPIA section, privacy approver, data reach derived from the tools), the registry and the data manifest checks.

Residency in one line per track: AWS has EU options for all three roles; Google Cloud has an EU Judge only, so EU Workhorse and Economy calls are refused; Azure routes EU Workhorse and Economy calls to Mistral-Large-3 on the EU Data Zone and refuses the Judge; Local keeps everything on your machine. A missing EU option raises ResidencyError before any network call, never a silent fallback. EU AI Act dates: Article 50 transparency from 2 August 2026; stand-alone Annex III obligations from 2 December 2027 after the Digital Omnibus.

Framing: Gartner AI TRiSM

Gartner's AI trust, risk and security management (https://www.gartner.com/en/articles/ai-trust-and-ai-risk) is organised in four layers, and every control in this course sits in one of them. AI governance: the use case catalog, the agent registry and its approval, the model and prompt registries with stages and reasons, the calibrated Judge, the tiered gates and the governance documents. AI runtime inspection and enforcement: the screener in front (Bedrock Guardrails, Model Armor, Prompt Shields, Llama Guard), the Cedar policy on the AWS tool gateway, the approval gate in code and in the ops store's permissions, drift and quality alarms, the kill switch and the spend caps. Information governance: redaction before prompts, logs and traces, the audience decided by the caller's key, residency routing, retention by lifecycle rule. Infrastructure and stack: least-privilege identities per tenant with the per-cloud limits in docs/SECURITY.md, secrets by reference, KMS, image scanning, SBOM and signing, private services where the track has them. Gartner sells the framework as a market map; the course uses it only as a checklist for whether a layer has nothing in it.

The cost sheet

deploy/COSTS-platform.md has every rate with the date it was fetched, per track and per mode: the credit to request per learner, the solo cost, the daily idle cost of a deployed platform, what stop leaves running and what destroy leaves behind. Read it before the first deploy and again before you leave.

Who does what in the capstone

From make roles, the rows this part exercises, and the privacy approver each use case names in the catalog.

Role	Stage	Owns
Product owner	Test and release	reads the system tier of the gate report and signs the approval state
QA engineer	Test and release	runs the tiered gate and keeps the tier bars in the baselines under `data/golden/`
Platform engineer	Deploy	registers the card with the agent runtime (`make agent-cards PUSH=1`) and runs the canary
Reviewer	Deploy	approves or rejects proposed escalations (`make approve RUN=<run_id>`), and the promotion
Platform engineer	Maintain	owns the drift and quality alarms and the kill switch (`NW_AGENT_DISABLED`)
Data engineer	Maintain	runs the capture and backtest loop and refreshes the baseline
Reviewer	Maintain	reviews the approval log and guardrail interventions each month
Privacy approver	Govern	signs each use case's DPIA section and re-reads it on every trigger

Questions worth asking

Of every failure you saw across the six parts, which would have reached production in your current setup, and which of the four TRiSM layers would have caught it?
Your router saves money on P0 and P3 tickets. What happens to the saving when Project 1's P0 recall drops, and which signal tells you first: the shadow agreement, the P0 share ratio or a person?
On your track, who rolls a bad canary back, and how long does it take them? Would you let an alarm do it for a model, for a prompt, for an agent?
Which identity in your organisation can write to production today without going through a pipeline?
Which of your stores would stop you meeting a 30-day erasure promise?
Which component would you replace first if the model vendor changed tomorrow, and what in this repository would you have to touch?

What you built

A router that cuts the cost per resolved ticket with the cheapest path that suffices, measured before and after through the model gateway; an approved model promoted to the live target behind a canary, and a canary taken back; images through the platform's delivery pipeline with a signature, an approval and a canary; objectives with burn rates and an error budget policy, and the quality signals that stop a canary; approvals executed once from the ops store by a person the platform names, key rotation and the kill switch; governance documents a buyer can read against the code; and the path, drawn and on the Local track run, by which the same artifact moves into a higher environment without being rebuilt.