Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
The 10 data quality management best practices that run a program: table owners, a weekly cadence, a severity ladder, five numbers, and what to stop doing.
Updated on
Data quality management is the practice of keeping data accurate, complete and fit for the decision it feeds. That is the whole definition, and our guide to what data quality management is, the six dimensions and how to score a table covers the measurement side properly. This page is about the other half of the job: what the work actually looks like on a Tuesday morning.
Most best practice articles list activities. Establish governance. Run regular audits. Add validation rules. Train the team. Buy a tool. Every one of those is correct, and not one of them tells you who is contacted at seven in the morning when the revenue table lands short, or what to do about the check that has fired every Monday for six months and been closed unread every time.
So this page skips the vocabulary. Below are ten practices that describe a data quality program as an operating routine, with the people, the hours, the response times and the numbers written down. They are ordered by dependency, which means the earlier ones are the ones that make the later ones possible, and the last section says which three to build first if you are starting from nothing.
Before the practices, the shape. A data quality program that is genuinely running has five visible parts, and you can audit your own in about ten minutes by asking whether each one exists in writing.
The roles below are the smallest set that makes those five parts work. The hours are what we would budget for a data platform of a few hundred tables with a handful of tier 1 products; scale them with the tier 1 count, not with the total table count.
| Role | What they own | What reaches them | Time it costs |
|---|---|---|---|
| Table owner (the engineer who builds it) | Every check on their tier 1 tables, and the fix when one of them breaks. | Any level 1 or level 2 incident on a table they own, routed automatically. | 2 to 4 hours a week |
| Data steward (a domain analyst, part time) | The business rule behind each check and the tier each table sits in. | Arguments about whether a value is wrong or merely unusual. The steward decides. | 2 hours a week |
| Quality lead (one person, never a committee) | The cadence, the severity ladder, and the five program numbers. | Nothing operational. They chair the weekly review and publish the numbers. | 4 hours a week |
| On call data engineer (a rota, shared with pipeline on call) | First response on any level 1 outside the owner working hours. | Paged by the alert route, never by a person tapping them on the shoulder. | One week in four or more |
| Executive sponsor | The tier 1 list and the budget behind it. | The quarterly recertification, and any tier change that adds cost. | 1 hour a quarter |
Notice what is not in that table. There is no data quality team. Quality that lives in a separate team becomes a service the rest of the organization orders from, and the producers who create most of the problems never see the consequence. The question of where quality sits relative to governance, and which of the two comes first, is answered in detail in our piece on the difference between data quality and data governance.
This is first because everything else fails without it. An alert that routes to a shared inbox is an alert that gets read by whoever is least busy, which is whoever knows least about the table. Within a month the channel is muted.
Do it in this order. Pull the list of tables that feed a board report, a customer facing product, a regulatory filing or a machine learning model in production. That is tier 1, and for most companies it is between twenty and eighty tables, not two thousand. Tier 2 is everything a named internal team uses daily. Tier 3 is everything else, and tier 3 gets no checks at all until it earns promotion.
Then put one name against every tier 1 row. Not a team, not a squad, a person, with a named deputy for their leave. If a table cannot get a name in a week, that is the finding: either nobody depends on it, in which case it is not tier 1, or it is orphaned, in which case you have found the thing most likely to break next.
The test that this practice is real: pick any three tier 1 tables at random and ask a new joiner to name the owner in under a minute using only the tools they already have. If they cannot, ownership exists in a spreadsheet rather than in the system.
A data quality standard that lives in a document is a wish. The same standard expressed as tests that run on a schedule is a control, and the difference between the two is whether anyone finds out when it is broken.
The translation is more mechanical than it sounds. "Customer records must be complete" becomes a not null test on the four fields the downstream report actually reads. "Orders must be unique" becomes a uniqueness test on the order identifier. "Every order must belong to a customer" becomes a referential test between the two tables. If you already run dbt, four generic data tests ship built in and cover most of that, and each one returns the rows that failed rather than a pass or fail flag, which is what makes a failure investigable. Great Expectations GX Core (docs.greatexpectations.io) is the open source option if your transformations are Python rather than SQL, and a platform such as Decube adds the checks nobody writes by hand, freshness, volume and schema drift, which are the ones that catch a broken pipeline rather than a broken record.
Two rules keep the check set honest. Every check names the table and column it applies to, so there is no such thing as an organization wide rule that nobody has implemented anywhere. And every check has a one line reason recorded with it saying what breaks downstream if it fails, because that sentence is what the on call engineer reads at three in the morning and what you will use later to decide whether to retire it.
Cadence is the practice teams skip, and it is the one that turns a set of monitors into a program. Without it the work happens when someone complains, which means it happens after the damage.
Four rituals are enough. They are deliberately short, and the time budget is part of the design: a data quality meeting that runs long gets cancelled within two months.
| Ritual | Frequency | Who | Input and output | Time budget |
|---|---|---|---|---|
| Morning triage | Every working day | On call engineer alone | In: overnight incidents. Out: each one closed, assigned to an owner, or escalated. | 15 minutes |
| Quality review | Weekly, same slot | Quality lead, table owners, one steward | In: the five program numbers and every incident older than seven days. Out: a decision on each stale incident and a list of checks to add or retire. | 45 minutes |
| Producer review | Monthly | Quality lead and the upstream engineering or operations team | In: the incidents whose root cause was upstream. Out: fixes committed in the producing system, not in the warehouse. | 30 minutes |
| Recertification | Quarterly | Quality lead, executive sponsor, stewards | In: the tier list, the owner list and the retired check log. Out: a tier 1 list signed again and confirmed owners. | 60 minutes |
The recertification is the one people question and the one that pays for itself. Table tiers rot faster than anything else in a data platform: a table promoted to tier 1 for a launch stays tier 1 for three years after the launch is forgotten, and the alert budget goes with it. Sixty minutes a quarter spent demoting tables is the cheapest noise reduction available.
Without a written ladder, severity is decided by whoever is looking, and people are consistent in only one direction: everything becomes low priority. The ladder below is a starting point to argue with, not a standard. What matters is that your version is written down before the next incident and that the response time is a commitment rather than an aspiration.
| Level | What it means | Response time | Who is contacted | What stops |
|---|---|---|---|---|
| 1. Critical | Wrong data has reached a customer, a regulator or a board report, or is about to within the hour. | Acknowledge in 15 minutes, contain in 1 hour | On call engineer paged, table owner and quality lead notified, sponsor told within the hour | The affected pipeline is paused and the dashboard or feed is marked as unavailable |
| 2. High | A tier 1 table is broken but nothing has left the building yet. | Acknowledge in 1 hour, fix by end of the working day | Table owner directly, quality lead on the thread | Downstream jobs that read the table are held |
| 3. Medium | A tier 2 table is broken, or a tier 1 table has a fault a consumer would not notice today. | Triaged next working day, fixed within the week | Table owner, in the daily triage | Nothing stops; the incident is logged and scheduled |
| 4. Low | A rule failed on data nobody currently reads, or a threshold was set too tightly. | Reviewed at the weekly meeting | Nobody is contacted out of band | Nothing stops; a candidate for retirement under practice 8 |
One discipline makes the ladder hold. The level is set by the consequence to the consumer, never by how alarming the alert looks. A null rate jumping from nought to ninety percent on a column no report reads is a level 4. A single wrong row in the table behind a customer statement is a level 1. Teams that set severity by the size of the anomaly end up paging engineers about staging tables.
The value of a runbook is not that the steps are clever, it is that they are the same every time, so nothing is skipped when everyone is tired. Five steps, in this order.
The fifth step is where the program improves, and it is the step that decides whether the incident count falls next quarter. The incidents nobody had a check for are the list that tells you what to build next. This is also the part of the work a platform can genuinely shorten, because most of the elapsed time in an incident goes on working out what the broken table feeds. The short walkthrough below shows that loop in Decube: opening a volume incident to see the unexpected row count and its history, running the debug query that gets to the cause, and reading the impacted areas and lineage panels that list every downstream table, dashboard and chart the issue touches.
A note on where this sits relative to monitoring. Detecting that a table changed is the observability half of the job, and judging whether the records inside it are right is the quality half. The boundary matters when you are buying, and we set it out properly in data quality versus data observability. The incident workflow above needs both halves: the monitor tells you something moved, the check tells you whether it mattered, and Decube data observability runs them against the same lineage graph so the impact question is answered without leaving the incident.
Cleaning a bad value in the warehouse repairs one report. Fixing the form field, the API validation or the upstream job that produced it repairs every consumer of that field, including the ones you do not know about.
This is easy to agree with and hard to do, because the warehouse fix takes ten minutes and the source fix takes a sprint in a team that does not report to you. Two things make it happen. First, the monthly producer review in the cadence table exists specifically to carry these fixes, so they have a standing route rather than needing a favour each time. Second, every incident write up records where the value first went wrong, so after a quarter you can show the producing team a ranked list of the faults their system creates rather than an anecdote.
Where the source genuinely cannot be fixed, a third party feed you have no influence over, a legacy system in a freeze, then clean it in one place, at the earliest layer you control, and record the cleaning rule as an owned artefact with the same review cycle as a check. What you must not do is let each downstream team invent its own correction, because that is how two dashboards start disagreeing about revenue.
Almost every team measures the data and almost none measures the program. The count of checks, the count of tables monitored and the count of alerts sent are all activity metrics: they go up when you work harder and say nothing about whether the data is getting better. Five numbers do say that.
| Number | What it measures | A starting target | Where it comes from |
|---|---|---|---|
| Tier 1 coverage | The share of tier 1 tables with at least one check per applicable dimension and a named owner. | 100 percent within one quarter. Anything less means the tier list is wrong or the tier is too big. | The tier list joined to the check inventory |
| Time to detect | Median elapsed time from a value going wrong to an alert firing. | Under one pipeline cycle. If your loads are hourly, under an hour. | Incident timestamps against the data timestamp of the bad record |
| Time to resolve | Median elapsed time from alert to the check passing on corrected data. | Level 1 within a working day, level 2 within two. | Incident open and close times |
| Escaped defects | The number of incidents reported by a consumer before any check caught them. | Falling every quarter, and each one produces a new check. | Incidents tagged by who raised them |
| Alert precision | The share of alerts that led to an action. The Site Reliability Workbook definition: the proportion of detected events that were significant. | Above 70 percent. Below 50 percent and people have already stopped reading. | Incident outcomes, counted at the weekly review |
Alert precision is the one to add first if you only add one. Alerting on service level objectives is a solved problem in software operations, and the Google Site Reliability Workbook chapter on alerting defines precision as the proportion of the events you detected that were actually worth detecting. Data teams rarely measure it and it is usually the number that explains why a well instrumented platform still misses incidents: the alerts fire, and nobody reads them any more.
Report all five on the same weekday, to the same audience, whether the week was good or bad. A quality report that appears only when the numbers improved is not a measurement system, it is marketing, and the audience works that out inside a quarter.
This practice is missing from every competing article on the subject and it is the one most likely to save a program that is already failing. Adding checks is treated as progress and removing them as retreat, so check sets only ever grow, and a growing check set with a fixed amount of human attention produces a falling response rate.
The rule is arithmetic rather than judgment. At the weekly review, take any check that has fired three or more times in the last quarter and ask what action followed each firing. If the answer is that every firing was closed without a change to data, code or pipeline, the check has three possible futures and no fourth: retune the threshold so it fires on the condition that actually matters, downgrade it to level 4 so it stops interrupting anyone, or delete it. Leaving it alone is a decision to keep training your team to ignore the channel.
Keep a retired check log with the reason. It costs nothing, it stops the same check being reintroduced by the next joiner, and at recertification it is the evidence that the program is being pruned rather than merely grown.
Most quality disputes are expectation mismatches rather than defects. The analyst thought the table was updated hourly, it is updated at 06:00, and the eight o clock meeting used yesterday numbers. No check would have caught that, because nothing was broken.
A data contract fixes it by writing the commitment down where the consumer sees it. Keep it to six lines: what the table contains, who owns it, how often it updates and by what time, which fields are guaranteed present and unique, what happens when it breaks and where that is announced, and how much history is retained. Put it in the catalog entry the consumer already opens, not in a document they will never find.
The discipline part is that a contract is a promise you have to be able to keep. Do not publish an 06:00 freshness commitment on a pipeline that misses it one morning in five. Publish the number you actually hit, then improve it. A contract that is quietly wrong is worse than no contract, because the consumer stops checking.
The last practice is the one that changes the trend rather than the response. Most bad data is created upstream by people who never see what it breaks: a field made optional in a form, a status value added to an enumeration, a column renamed in a migration.
Two changes move more incidents than any monitoring work. First, a schema change on a tier 1 source needs the consuming owner to be told before it ships, not after, which is a one line addition to a pull request template rather than a governance program. Second, a new field on a tier 1 table does not count as done until it has a check and a line in the contract, exactly the way a new endpoint does not count as done without a test.
Training belongs here too, and it is worth being specific about what kind. A general data quality awareness session teaches nothing that survives the week. A thirty minute session showing the producing team the three most recent incidents their own system caused, and what it cost the people downstream, changes behavior, because it is the first time most of them have seen the consequence.
Data quality issues arise from a small number of repeating causes: human entry error, system limits, integration faults, and business requirements that changed without anyone telling the data team. Five patterns account for most of them, and each has a check type that catches it and a practice above that prevents it recurring.
The most common issue of all, produced by manual entry, repeated imports or a synchronisation fault between two systems. Duplicates distort every count in a report and quietly raise storage cost. Deduplication is the remedy, identifying the repeated records and consolidating them into one accurate record, either by hand or with automated matching. The check that catches it is a uniqueness test on the business key, and the practice that prevents it is a standard applied at the point of entry, so that a new record is validated against existing records before it is written rather than cleaned up afterwards.
Records missing information the analysis needs, caused by entry error, a source system that does not collect the field, or an integration that silently drops it. The remedy is a completeness standard enforced before the data enters the system, so required fields cannot be left empty, backed by tooling that flags what got through anyway. The check is a not null test on the specific fields a downstream consumer reads, which is narrower and far more useful than a rule demanding every column be populated.
Values that are simply wrong, or right but in the wrong format. The causes are the same three: entry error, source limits, integration faults. The remedy is the same combination of a written standard and validation rules that reject a value outside the allowed set, with automated tooling identifying and correcting what slips past. The check is a range or accepted values test, and this is the category where a check needs a steward behind it, because deciding whether a value is wrong or merely surprising is a business judgment rather than a technical one.
Data that is no longer current, produced by a changed business requirement, a system limit or a source that stopped updating. It is the most dangerous of the five because nothing looks broken: the report renders, the numbers are plausible, and they are last week. The remedy is not only having standards but updating them as requirements change. The check is a freshness test on every tier 1 table, and the contract in practice 9 is what stops the mismatch between when a consumer thinks the table updates and when it actually does.
The same fact recorded in different formats or structures across systems, so it cannot be compared or aggregated. Entry error and system limits both produce it. The remedy is a consistency standard applied before data enters, plus validation and automated correction for what already exists. The check is a format or referential test between the systems that hold the same entity, and the practice that prevents recurrence is practice 6, because inconsistency almost always means two producers were never told the same rule.
Four categories of tool sit behind the practices above, and they are frequently confused in procurement because their marketing overlaps. The distinction that matters is which question each one answers, and none of them is a substitute for the ownership and cadence described above. Vendor names below are plain text and are not endorsements.
| Category | What it does | Examples | When you actually need it |
|---|---|---|---|
| Data quality and observability platforms | Profiles data, runs the checks, detects the anomalies nobody wrote a rule for, and carries the incident through to root cause and downstream impact. | Decube, Trifacta, Talend, Informatica | As soon as you have a tier 1 list. This is the category that makes practices 2 to 5 possible and it is where the incident workflow lives. |
| Master data management | Keeps one agreed record for an entity, a customer or a product, across every system that holds it, with the stewardship workflow to resolve conflicts. | IBM InfoSphere MDM, Informatica MDM, Talend MDM | When the same entity exists in three or more systems and they disagree. Before that, a uniqueness check on one warehouse table is cheaper and enough. |
| Data integration | Moves and reshapes data between systems, with mapping, transformation and loading. | Apache NiFi, IBM InfoSphere DataStage, Talend | Always present in some form. It matters here because it is where most inconsistency and most silently dropped fields are introduced. |
| Business intelligence | Presents the data to a human as a report, a dashboard or a chart. | Tableau, Power BI, QlikView | Already in place everywhere. It matters to quality only as the consumer: the dashboard is what you mark unavailable at step 1 of the runbook. |
Two open source options are worth knowing before you buy anything, because for a small check set they are free and sufficient. If your transformations run in SQL through dbt, its four built in generic data tests cover not null, uniqueness, referential integrity and accepted values, and each returns the failing rows. If your pipeline is Python, Great Expectations GX Core (docs.greatexpectations.io) does the equivalent as a library. Both stop short at the same place: they run where you invoke them, so they catch a broken record and not a pipeline that never ran, and neither carries an incident, an owner or a severity. That gap is what a platform is for. For the integration layer, Apache NiFi remains the common open source choice.
Choose by the smallest set that covers your practices, not by feature count. A team with forty tier 1 tables, an owner against each and a weekly review will get more from a platform they configure in a fortnight than from a suite that takes two quarters to deploy and answers questions they have not asked yet.
Four habits do active harm, and stopping them is faster than any of the ten practices above.
If you are in banking, insurance or another supervised sector, the ten practices are the same and two things change: the evidence you keep, and who signs. Supervisors do not ask whether you have a data quality tool. They ask you to show that a named person is accountable, that the control ran on the day in question, and that the failure was followed by a fix. The Basel Committee Principles for effective risk data aggregation and risk reporting, published in January 2013 and still current, is the clearest statement of the expectation: fourteen principles of which principles 3 to 6 cover the data itself, and principle 5 on timeliness explicitly ties itself back to accuracy and integrity, completeness and adaptability rather than treating any of them separately.
Practically that means three additions. Retain the check results, not just the incidents, so you can show a control ran on a given date. Keep the recertification minutes with the sponsor signature, because the tier list is the scope of your control and a supervisor will ask how it was set. And record lineage from the reported figure back to its sources, since the question is rarely whether a number is right today, it is whether you can demonstrate where it came from. Decube customers in supervised markets face this from OJK in Indonesia, APRA in Australia, MAS in Singapore and the NAIC insurance framework in the United States, and in every case the evidence requirement is the same shape.
Ten practices is too many to start with and they are not equal. If you have none of this today, build three, in this order, and leave the rest until they are habits.
Checks, tooling and measurement follow naturally once those three exist, and they follow badly if they do not. A team that buys a platform first ends up with monitors on two thousand tables, alerts arriving in a channel with no owner, and a quarterly report counting how many checks it has. That is the failure this article is written against.
The reason to take the sequence seriously is that overlooking data quality does not produce a data problem, it produces a decision problem: reports that are believed and wrong, revenue recognized against duplicates, and models trained on a field that stopped updating in March. Decube brings the catalog, the lineage, the checks and the incident workflow into one platform so that ownership, severity and impact live in the same place rather than in three tools and a spreadsheet. If you want to see the incident and scorecard workflow against your own tables, request a Decube demo.
Ten practices describe a data quality program as an operating routine rather than a list of activities: give every tier 1 table a named owner before writing any check; write the standard as checks against named tables rather than as a policy document; run a fixed cadence of daily triage, a weekly review, a monthly producer review and a quarterly recertification; set a four level severity ladder with a response time for each level; handle every incident with the same five step runbook; fix the source rather than the warehouse copy; measure the program with five numbers rather than counting checks; retire any check that fires and is ignored; publish a data contract the consumer can read; and put data quality in the definition of done for the teams that produce the data.
By making the standard executable and giving it an owner. A rule written in a document is a wish; the same rule expressed as a test that runs on a schedule and returns the rows that failed is a control. So translate each business rule into a check against a named table and column, attach every tier 1 table to a named person rather than a team alias, route alerts to that person by severity, and review the results in a fixed weekly slot. The three components that make it hold are ownership, a schedule the checks run on, and a severity ladder that decides who responds and how quickly.
For an incident there are five, always in the same order: contain the problem by pausing the pipeline and marking the affected dashboard as unavailable; tell the consumers what is wrong and when the next update will come; find the cause by reading the failing rows and walking the lineage upstream to the first place the value was wrong; fix the source and backfill the affected window, confirming the check passes before declaring it resolved; and write it up in fifteen minutes, recording why the existing checks did not catch it and the one check that would have. The last step is where the program improves, because the incidents nobody had a check for are the list of what to build next.
Improve the trend by fixing where the data is produced, not where it is consumed. Cleaning a value in the warehouse repairs one report, while fixing the form field, the API validation or the upstream job repairs every consumer of that field at once. Give those fixes a standing route through a monthly review with the producing team, record in every incident write up where the value first went wrong so you can show a ranked list rather than an anecdote, and add two rules to the producing team definition of done: a schema change on a tier 1 source tells the consuming owner before it ships, and a new field is not finished until it has a check.
A data quality procedure is the written routine a team follows, and it needs four parts to be usable. A tier list classifying every table by what depends on it, with a named owner against each tier 1 table. A check inventory naming the table and column each rule applies to, with a one line reason saying what breaks downstream if it fails. A severity ladder of four levels, each with a response time and a route. And a meeting cadence with time budgets: fifteen minutes of daily triage, forty five minutes of weekly review, thirty minutes monthly with the producers, and sixty minutes of quarterly recertification.
Five numbers answer that, and none of them is the count of checks. Tier 1 coverage, the share of tier 1 tables with a named owner and at least one check per applicable dimension, which should reach 100 percent within a quarter. Time to detect, the median time from a value going wrong to an alert firing, which should be under one pipeline cycle. Time to resolve, from alert to the check passing on corrected data. Escaped defects, the incidents a consumer reported before any check caught them, which should fall every quarter. And alert precision, the share of alerts that led to an action, which should stay above 70 percent, because below about 50 percent people have already stopped reading the channel.
The same ten practices plus evidence and a signature. A supervisor does not ask whether you own a data quality tool; they ask you to show that a named person was accountable, that the control ran on the date in question, and that a failure was followed by a fix. That means retaining the check results rather than only the incidents, keeping the quarterly recertification minutes with the sponsor signature because the tier list is the scope of your control, and recording lineage from a reported figure back to its sources. The Basel Committee Principles for effective risk data aggregation and risk reporting, published in January 2013 and still current, set the expectation for banks in fourteen principles, of which principles 3 to 6 cover the data itself.
Treat it as a certification gate rather than a project. Before an agent is allowed to read a table, that table should be tier 1 or tier 2 with a named owner, carry freshness, volume and schema drift monitoring so a silent stall is detected, have its sensitive columns classified so the agent cannot surface what a person could not, and publish a contract stating update time and guaranteed fields. Add one rule specific to agents: an agent reads only tables that pass their checks on the most recent run, so a failing check withdraws access rather than merely raising an alert, because an agent will not notice that a number looks wrong.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)