Ingesting Customer Data

Topics Covered

Finding and Accessing the Data

Starting from the workflow

Inventorying sources

Systems of record

Access methods

Read replicas and load on sources

Requesting access

Least privilege

Sample data first

Data owners and their concerns

Undocumented sources

Data contracts

Planning the timeline around access

Reading the source schema

Talking to the people who use the data

Personal and sensitive data

Data residency

A worked inventory

Access that expires

Confirming access works end to end

Almost every forward deployed engagement depends on customer data. Before any model, dashboard or workflow can help, the data has to be found, accessed, moved and loaded reliably. Ingestion is often the longest and least glamorous part of a deployment, and it is where many deployments slip. This lesson covers how to find data, choose ingestion patterns, sync incrementally, validate loads and run pipelines that customers can trust.

Starting from the workflow

Start from the workflow, not the data catalogue. Ask what decisions the users make and what information they look at to make them. Then trace each piece of information back to its source system. This keeps ingestion focused on data that matters. Customers often offer access to everything, but loading everything wastes weeks and creates risk without adding value.

Inventorying sources

List every source the deployment needs: the system, its owner, the tables or endpoints, the volume, the refresh rate and how access will be granted. A planning deployment might need orders from an enterprise resource planning system, stock levels from a warehouse system and carrier events from a partner feed. Each source has its own owner, format and quirks.

Systems of record

For each piece of data, identify the system of record, the place the business treats as the truth. Customer addresses may live in several systems, but only one is authoritative. Loading from the wrong system leads to subtle errors that users spot quickly. Ask the data owner directly which system wins when two disagree.

Access methods

Data can be accessed in many ways: direct database reads, replica databases, application programming interfaces, file drops on a secure server, message streams or exports from a reporting tool. Each has trade offs in freshness, load on the source and ease of approval. Prefer methods the customer already uses for other integrations, since they are easier to approve and support.

Read replicas and load on sources

Reading directly from a production database can slow systems that users depend on. Ask whether a read replica or reporting copy exists. If not, agree a schedule that avoids busy periods and limit the size of each query. Being careful with source load builds trust with database administrators, who are important gatekeepers for any data work.

Requesting access

Access requests should be specific: which system, which tables or fields, read or write, from which network location and for what purpose. Specific requests are approved faster and reduce security concerns. Include how the data will be stored and protected. Vague requests for full access tend to sit in queues for weeks while security teams ask questions.

Least privilege

Ask only for the access the deployment needs. If the workflow uses five fields from a customer table, request those five, not the whole table. Least privilege reduces the damage if credentials leak and makes security reviews easier. It also shows the customer that the vendor takes their data seriously, which helps with later requests.

Sample data first

Before building pipelines, ask for a sample: a few thousand rows from each source, or a recent export. Explore the sample to understand fields, formats, null patterns and quirks. Samples often reveal problems such as dates stored as text, duplicated records or missing keys. Finding these early lets you plan cleaning work and set expectations with the customer.

Ingestion starts from the workflow, traces each need to its system of record and asks for exactly the access it needs.

Data owners and their concerns

Each data owner worries about something: load on their system, data leaving their control, changes they did not approve or being blamed for bad data. Meet them early, explain what you need and why and ask what worries them. Addressing their concerns directly turns data owners from blockers into allies who help when something breaks.

Undocumented sources

Some important data lives in places nobody documents: spreadsheets on shared drives, email attachments or a database built by one engineer years ago. These sources often hold the corrections and overrides that make the official data usable. Find them through observation and ask whether they should become formal sources or be replaced by the deployment.

Data contracts

A data contract is an agreement about what a source will provide: fields, types, meanings, freshness and how changes will be announced. Even an informal contract written in a shared document helps. It gives both sides a reference when something changes and reduces the chance that a schema change silently breaks the pipeline.

Planning the timeline around access

Access is usually on the critical path. Request it in the first week, track each request as a milestone and escalate delays early. While waiting, build with sample data or synthetic data that matches the real schema. Never let the whole deployment wait idly for access, but never assume samples will behave exactly like production.

Reading the source schema

Once you have a sample, read the source schema closely. Look at primary keys, foreign keys, indexes, timestamp columns and any status or flag fields. Ask what each unclear column means, since names often mislead: a field called ship date may hold the promised date, not the actual one. Write the meanings down in a shared glossary as you learn them.

Talking to the people who use the data

The data owner knows the system, but the people who use the data daily know its traps. Ask analysts and power users which fields they trust, which they ignore and which reports they reconcile by hand. A finance analyst may tell you that returns are booked as negative orders, which no schema document mentions. These conversations save days of confusion.

Personal and sensitive data

Identify personal and sensitive fields early: names, addresses, contact details, health information or payment details. Ask whether the deployment needs them at all. Often an identifier is enough, and the sensitive fields can stay in the source. If they are needed, agree how they will be protected, masked in lower environments and deleted when no longer required.

Data residency

Some customers require data to stay in a particular country or region. Ask about residency during scoping, since it affects where pipelines run and where data is stored. A deployment that copies data to the wrong region can fail a security review late in the project. Residency requirements are far easier to meet when planned from the start.

A worked inventory

A planning deployment inventory might list: orders from the resource planning system, two million rows, owner the finance systems team, read replica, hourly; stock levels from the warehouse system, two hundred thousand rows, owner the warehouse systems manager, nightly file drop; carrier events from a partner interface, fifty thousand events a day, owner logistics, webhook with nightly reconciliation; products from a master data service, thirty thousand rows, weekly full load. Each line names a source, a size, an owner, a method and a refresh rate.

Customer data comes from many systems of very different sizes, and asking for only the fields needed speeds every approval.

Access that expires

Credentials often expire: temporary accounts, certificates and tokens all have end dates. Record each expiry and renew well before it, since an expired credential at five in the morning stops the whole pipeline.

Confirming access works end to end

When access is granted, test it end to end the same day: connect from the real network location, run a small query and land the result in staging. Many grants turn out to be incomplete, missing a firewall rule or a table permission, and finding that out immediately saves another week in the queue.

Start from decisions, ask precisely

Trace each user decision to its system of record, inventory sources with owners and volumes and choose access methods the customer already supports. Request least privilege access in week one, explore samples early, protect source systems from load and write simple data contracts so changes are announced, not discovered.