Monitoring, how you know it broke (and what it costs)

Launch week went well, and you have been posting the proof, a rising sign-up graph, to the chat your pilot customers watch. Then, at 3:02 one morning, the email service behind your sign-up form starts rejecting requests. Nothing visible changes: the homepage loads, the demo runs, and every visitor who tries to join watches a spinner until they give up. Two days later a stranger's email arrives: "Tried to sign up three times this week. Is this still happening?" It has been happening for 41 hours. That email is your entire monitoring system, and it reported the incident two days late, through the one visitor patient enough to write instead of leave.

Things break in every product; the real failure is that a user knew before you did.

Monitoring is the discipline of finding out about a problem before your users do, and in an AI product it also watches the one signal users cannot report: what every answer costs.

Logs, metrics, and error tracking: the three layers you need

The part of monitoring you need collapses into three layers, all visible in the log, the diary your server writes as one timestamped line per request. Here are three lines from the night of the outage: a normal request, a slow one, the first failure.

03:01:57 INFO   POST /api/signup 200 in 412ms
03:02:09 INFO   POST /api/signup 200 in 2967ms
03:02:14 ERROR  POST /api/signup 502 in 30004ms  email-service: connection refused

Each line carries the time, the action, the verdict code from the journey of a request, and the wait.

Logs answer "what happened." Read the lines back: a healthy sign-up, a slow one at almost three seconds, then a 502 (a machine this server depends on failed) after the 30-second spinner your visitor stared at, and the note naming the culprit, your email service.
Metrics answer "how much." A metric is one number made by counting log lines over time: requests per minute, the share ending in error, and in an AI product, the day's spend.
Error tracking answers "what broke, and who it hit." An error tracker (Sentry is the common first pick) collects every ERROR line, groups identical failures onto one card, and counts the users each hit, so you fix the worst thing first.

Little of it needs building: hosts write the log and draw basic metrics by default, and an error tracker is quick to add. You will read your own build's log in After the ship: watch real use and decide the next move.

The missing piece at 3 a.m. was an alert

Run the scene back through the layers: the ERROR line hit your log at 03:02:14, and your sign-up success metric fell to zero and stayed there for 41 hours. Everything was recorded while you slept, and you still heard it from a stranger, because a record answers only when someone asks, and at 3 a.m. nobody asks.

The missing piece is an alert: an automatic message that fires when a number you chose crosses a line you chose (the threshold), sent somewhere that interrupts you. "If sign-up success drops below 90 percent for fifteen minutes, message our shared channel" would have buzzed your phone at 03:17. You write yours before this chapter ends.

The objection: the dashboard already shows this

Your host ships a dashboard of requests and errors, so the natural objection is that monitoring comes with the rent, and with no users yet there is nothing to watch. But a dashboard answers only when you open it, and the stranger who emailed was your first real user, met by a dead form. And the two failures that matter most in an AI product never reach that dashboard at all.

The AI bill climbs without anything failing

Every model call is metered in tokens, the same receipt you priced in APIs, how systems talk to each other. The bill is a monitoring problem because it climbs while everything succeeds: a retry bug calls the model three times per question, a prompt edit doubles the context, or one user pastes a whole contract where you expected a paragraph; each returns success, and only the cost meter moves.

The defense, the provider's spending alert, is a step on the pre-flight checklist in Run the pre-flight: secrets, money, data, and the law; for now, treat spend per day as one more metric your spec can put a line under.

A request can succeed and still answer badly

Everything so far catches what machines detect: a crash, a timeout, an error code. A model feature can fail with none of those: the model returns a fluent, confident, wrong answer and the request completes cleanly. A support bot can misstate your refund policy a hundred times before the first angry email, because to the infrastructure every conversation worked.

Count what users do next. Track thumbs-down if your product has the buttons, retries (the same user asking the same question within a minute), and how often a user quits halfway through a task. These crude stand-ins turn "answered badly" into a number that can trend and alert; measuring quality properly is its own discipline, evaluation, taught later in the course.

A status page shows users what your monitoring sees

Providers face the same 3 a.m. problem, and their answer is public: a status page showing current health and a history of resolved incidents. When your AI feature degrades, check the provider's status page before debugging your code: it answers "is this us or them" fastest. Once people rely on your build, a small status page of your own shows them you noticed first.

Write your one-sentence monitoring spec

You do not have to design any of this; you have to specify it.

A complete monitoring spec is one sentence with three slots: the worst silent failure, the signal that would catch it, and where the alert lands. You write the sentence; your build tool proposes the wiring.

The worst silent failure is the thing users would feel before you did, usually sign-up, checkout, or the AI feature itself. The sign-up sentence above is one complete spec; "if daily token spend doubles compared with the week before, email both of us" is another. If you cannot finish yours, the failure is not specific enough; keep sharpening.

Try it now

This drill takes about ten minutes and spends nothing. You arrive with the cost sentence from Hosting, renting a computer on the internet and leave with your one-sentence monitoring spec, which Write your Build Plan and start the build collects.

No setup: Write the spec by hand for the idea you have been circling: its worst silent failure, the number that moves when it breaks, and a landing place that interrupts you. If the signal is hard to name, borrow one: sign-up success rate, or spend per day compared with the week before. Add the finished sentence to the idea file you started in Decode the technical questions that stop you.

With your tools: Describe the same idea to Claude Code and ask for the three most likely silent failures, ranked by how much each would hurt. Pick the worst, finish the sentence yourself, and copy it into your idea file; once a build exists, handing the sentence back gets the alert wired. In Codex or Cursor: ask the sidebar chat for candidate silent failures, then complete the spec by hand. If your tools are not installed yet, the Setup Clinic gets you there in one sitting.

The whole stack in one everyday order

This chapter closes the stack, so run the whole part through one everyday product: a food delivery app. The menu screen that answers the instant you tap is the frontend, running on your phone. Place the order and the request crosses to the company's servers, where the backend checks the restaurant is open and charges the card once. Phone and server speak in the fixed request-and-response format of an API.

Your address, the order, and the driver's position live in a database, so closing the app loses nothing. All of it runs on rented, hosted computers that stay up around the clock. You see only your own orders because auth checks who you are at sign-in and what you may do on every request. And when the payment provider starts refusing at 3 a.m., a phone at that company buzzes within minutes, because a sentence very like yours is wired into its monitoring. Every product you use is built this way, including the one you will build.

Chapter Summary

Monitoring means finding out about a problem before your users have to tell you about it.
Logs record what happened (a timestamped diary of every request), metrics count how much (one number tracked over time), and error tracking groups what broke by how many users it hit.
The 3 a.m. outage was recorded by all three layers while you slept; a record answers only when you ask, so recording alone told nobody.
The missing piece was an alert: an automatic message that fires when a number you chose crosses a line you chose, sent somewhere that interrupts you.
A host dashboard cannot replace an alert, because a dashboard requires you to open it, and failures arrive after you have stopped checking.
An AI product adds two signals the standard layers will not watch: the bill, which climbs while every request succeeds, and answer quality, where counting what users do next (thumbs-down, retries, quitting mid-task) stands in until you measure it properly.
Setting the provider's spending alert is a pre-flight step you will run before your build meets strangers; this chapter's job is to name spend per day as a signal worth a threshold.
Check the provider's status page before debugging your own code, and run a small one of your own once people depend on your build.
Your whole monitoring job is one sentence naming the worst silent failure, the signal that would catch it, and where the alert lands; force every alert to fire once before you trust it.
The stack is complete; next, Find your build and give the model a job turns the course toward the product only you can scope.

Sources

Anthropic status page, status.anthropic.com (last verified July 2026).
OpenAI status page, status.openai.com (last verified July 2026).
Vercel runtime logs documentation (last verified July 2026).
Sentry error monitoring documentation (last verified July 2026).