0%
Forward Deployed Engineering
The Role
Discovery and Scoping
Customer Data and Systems
Building in the Field
Monitoring Customer Deployments
A deployment that passed acceptance can still fail the next morning. A source system changes its export, a nightly job runs out of memory, a certificate expires or traffic doubles after a new team joins. Monitoring is how the team finds out before the customer does. For forward deployed engineers the job is harder than for an internal service, because the system runs inside or beside an environment owned by someone else, depends on their data and serves their users. This lesson covers what to monitor, how to set alerts that matter, how to monitor inside customer environments, how to build dashboards for different audiences and how to keep monitoring healthy over time.
The customer finds out first
The worst way to learn about a problem is from the customer sponsor. When users notice before the team does, trust drops twice: once for the failure and again for not knowing about it. Good monitoring reverses this. The team sees the problem, starts working on it and tells the customer what is happening before most users are affected. That one difference often decides whether an incident damages the relationship or strengthens it.
Four layers of signals
Think about monitoring in layers. Infrastructure signals cover machines, containers, memory, disk and network. Service signals cover request rates, error rates and response times for each part of the system. Data signals cover whether data arrived, how much, how fresh it is and whether it looks right. Outcome signals cover what the customer actually cares about, such as orders routed, tickets resolved or forecasts delivered on time. Most teams monitor the first two well and the last two poorly, yet customers feel the last two first.
Golden signals for services
For each service, track four basic measures: how many requests it handles, how many fail, how long they take and how close it is to its limits. Response times should be tracked as percentiles, such as the median and the ninety ninth percentile, rather than averages, because a slow tail can hurt many users while the average looks fine. These four measures catch most service problems and are a sensible starting point for any new deployment.
Data freshness and volume
Data pipelines fail quietly. A job that runs but loads nothing looks healthy to infrastructure monitoring. Track when each source last delivered data, how many rows arrived compared with the usual range and how long each step took. Alert when a source is late beyond its agreed window or when volume drops or spikes sharply. Freshness is often the first signal that a customer system changed without warning.
Data quality checks
Beyond arrival, check that data makes sense. Watch the share of empty values in key fields, the number of records that failed validation, duplicate keys, values outside expected ranges and new categories that the system does not recognise. Run these checks at each stage of the pipeline and store the results, so that a slow decline is visible as a trend rather than discovered only when a report looks wrong.
Outcome measures
Agree a few outcome measures with the customer during scoping and monitor them from day one. For a routing system it might be the share of orders routed automatically and the share sent to manual review. For a forecasting system it might be whether forecasts were published by six each morning and how accurate last week turned out. Outcome measures tie monitoring to the reason the system exists and catch problems that every technical measure misses.
Model and AI signals
For AI features, add signals specific to them: refusal rates, the share of questions with no answer found, grader scores on sampled outputs, token usage and cost per request, and how often the provider returns errors or slow responses. These connect to the production evaluation described in the previous lesson. A sudden change in any of them is worth investigating even when nothing else looks wrong.
Dependencies outside your control
Customer deployments depend on systems the team does not own: the identity provider, the customer data warehouse, a partner API, a model provider and network links between sites. Monitor each dependency from the outside, with simple checks that call it on a schedule and record whether it answered and how fast. When something breaks, these checks show quickly whether the fault is in your system or upstream.
Synthetic checks
A synthetic check is a scripted request that behaves like a user: sign in, search for a known record, ask a known question or submit a test order in a safe sandbox. Run it every few minutes from where users sit. Synthetic checks catch failures that internal measures miss, such as an expired certificate on the customer proxy or a broken sign in flow, and they work even at night when there is no real traffic.
Logs, traces and their limits
Logs and traces are essential for finding why something failed, but they are a poor way to find out that something failed. Use measures and synthetic checks to detect problems and logs and traces to investigate them. Make sure logs carry the request identifier, customer site, version and step, and agree with the customer what may be logged, since logs often contain sensitive data.
Starting small
A new deployment does not need hundreds of measures. Start with the golden signals for each service, freshness and volume for each source, one or two outcome measures and a synthetic check for the main user flow. Add more as incidents reveal gaps. Monitoring that grows from real problems stays focused on what matters.
Baselines before launch
Collect at least a week or two of measures during the pilot before switching on alerts. Baselines show what normal looks like for this customer: when traffic peaks, how long the nightly run usually takes, which sources tend to arrive late and how much the error rate moves on an ordinary day. Without a baseline, every threshold is a guess, and the first weeks of production turn into a stream of false alarms.
Monitoring per site and per tenant
When the same system runs for several customer sites, regions or business units, break every measure down by site. A problem at one small site can vanish inside healthy totals. Label every measure with the site, environment and version, so dashboards and alerts can slice by them without extra work.
Cost as a signal
Cloud and model costs belong on the monitoring list too. A sudden rise in spend often points to a bug, such as a retry loop, a job processing the full history every night or a prompt that pulls in far too many passages. Track cost per day by component and alert on sharp changes, so that a bill is never the first sign of a problem.
Watching the customer side
Some of the most important signals live in customer systems: the export job that feeds your pipeline, the queue where your system posts results and the screen where users see your output. Where possible, agree access to simple health views of these points, or ask the customer team to share their own alerts with yours. Problems at the boundary between two teams are the ones most often noticed late.
Cover infrastructure, services, data and outcomes, track percentiles not averages, watch freshness, volume and quality at every pipeline stage and agree outcome measures with the customer. Add AI signals, check dependencies and run synthetic checks from where users sit, use logs to investigate rather than detect and start small.