0%
Forward Deployed Engineering
The Role
Discovery and Scoping
Customer Data and Systems
Building in the Field
Incidents on Customer Sites
An incident is any unplanned event that harms or threatens the service the customer relies on: an outage, wrong results, late data, a security concern or a sharp drop in quality. Every engineering team has incidents. For forward deployed engineers they carry extra weight, because the system runs in the customer world, often inside their environment, touching their data and their users, and the relationship with the customer is on the line along with the system. This lesson covers why customer incidents are different, how to respond in the first hour, how to communicate, how to fix and recover and how to learn afterwards.
The relationship is part of the incident
When an internal service fails, the team fixes it and moves on. When a customer deployment fails, the sponsor may have to explain the failure to their own leadership, their users may lose a day of work and the renewal may be at risk. How the team behaves during the incident, how quickly it tells the customer, how honest it is and how calmly it works, often matters as much as how fast the fix arrives. A well handled incident can leave the customer trusting the team more than before.
Shared ownership of the system
Customer deployments are rarely owned by one side. The FDE team owns its code and configuration, the customer owns the source systems, network and identity provider, and third parties own model APIs and partner services. Many incidents start in a part the FDE team does not control. The team still has to lead the response for its own system, work out where the fault lies and coordinate with the owners of the other parts without blaming them.
Limited access and visibility
Inside customer environments, the team may not be able to log in freely, read every log or restart services. Access may need an approval, a customer engineer at the keyboard or a change ticket. These limits slow investigation and fixes. Planning for them before the first incident, by agreeing emergency access routes and who can approve them at night, is one of the most valuable things an FDE team can do during setup.
Wrong answers can be worse than outages
An outage is obvious: users see an error and stop. Wrong results are quieter and often more harmful. A routing system that sends orders to the wrong warehouse, a forecast that is half the real value or an assistant that quotes the wrong policy can cause damage for hours before anyone notices. Treat silent correctness failures as incidents as serious as outages, and be ready to stop the system rather than let it keep producing bad output.
Severity levels agreed with the customer
Define severity levels with the customer before launch. A common scheme has a critical level for a full outage or widespread wrong results affecting core work, a high level for a major feature broken or one site down, a medium level for degraded service with a workaround and a low level for minor issues. For each level, agree who is told, how fast the team must respond and how often updates are sent. Agreed levels stop arguments in the middle of an incident about how serious it is.
Contracts and response targets
Many customer contracts set response and update times for each severity. Know them before the first incident. If the contract says critical incidents get a response within thirty minutes at any hour, the on call and paging setup must be able to deliver that. Track response times for every incident, since they are often reviewed at renewal.
Customer users are part of the response
Customer users may be the first to notice, the source of crucial details and the people who need workarounds. Make it easy for them to report problems through a known channel and make sure reports reach the on call quickly. Their description of what they see, such as which screen, which site and since when, often narrows the search faster than any log.
Time zones and working hours
Customer sites may be in other time zones, so their busiest hour may be the middle of the night for the team. An incident during their morning peak hurts far more than one at their midnight. Know when each site is busiest and treat incidents during those hours with extra urgency.
Regulated and sensitive settings
In healthcare, finance, government and similar settings, incidents may have legal reporting duties, especially when personal data or safety is involved. Know the customer obligations and your own, agree how possible data incidents are escalated to their security and legal teams and never delay that escalation while investigating. Security incidents have their own rules, covered in a later lesson, and follow the customer process.
Preparing before the first incident
Most of a good incident response is prepared in advance: severity levels, contacts on both sides with phone numbers, emergency access routes, runbooks, a way to switch off or roll back each feature and templates for customer updates. Write these down during setup and test them in a practice incident with the customer. The first real incident is the wrong time to discover that nobody knows who can approve access at night.
Incidents during the pilot
Incidents during a pilot are common and often the first real test of trust. Handle them with the same care as production incidents: tell the champion quickly, explain plainly and fix carefully. A pilot incident handled well can be the moment the customer decides the team is worth working with long term.
Partial failures
Many customer incidents are partial: one site down while others work, one category of documents misread, one team unable to sign in or slow responses only at peak hours. Partial failures are easy to dismiss because most measures look fine. Break measures down by site, team and category during triage, and take reports from a single group seriously until they are explained.
Incidents caused by change
A large share of incidents follow a change: a release, a configuration edit, an index refresh, a provider update or a change on the customer side such as a firewall rule or a new export format. When an incident starts, the first question should be what changed recently, on either side. Keeping a shared change log with the customer, and annotations on dashboards, makes that question quick to answer.
The cost of waiting
Teams sometimes hesitate to declare an incident because they are not sure it is real or serious. Waiting costs more than a false alarm. Declaring early gets the right people looking, starts the clock on customer updates and can be downgraded easily. Make it normal for anyone on the team to declare an incident without asking permission, and thank people who declare one that turns out to be minor.
Blame and defensiveness
Under pressure, both sides may look for someone to blame. The customer may assume the new system is at fault, and the team may suspect the customer source. Blame slows the response, because people defend rather than investigate. Focus on facts, share what each side sees and keep the question on how to restore service. Root causes can be discussed calmly once users are back to work.
Multiple customers at once
If the same platform serves several customers, one fault may affect many of them. Check quickly whether other deployments show the same symptoms, coordinate the response across them and make sure each customer hears from its own FDE contact rather than discovering the problem from a status page alone.
Expect incidents to start in parts you do not own, treat silent wrong results as seriously as outages and agree severity levels and response targets with the customer before launch. Plan emergency access, know busy hours and legal duties, make it easy for users to report and test the whole response in a practice incident.