As I was reading How Will You Measure Your Life by Chris Christensen, it occurred to me with greater clarity that data integrity is merely a reflection of human integrity. Nobody wants data to contain errors – and yet it happens all the time despite our best intentions.
In organizations with a lot of data, there are often many people involved in data integrity: from the users who provide information to the people who interpret the reports – and everyone in between. But ask any person in the chain of data if they have integrity and they will always answer in the affirmative. However, the data is a reflection of the truth about our own integrity and it is rarely (never) flawless.
What I’ve observed is that data integrity issues are rarely (if ever) a result of a deliberate decision to enter incorrect data. Instead, it is often the result of rushed, tired, or stressed employees who are pressured to finish something quickly. This is true not only for the data entry clerks but for the people building the software, the business systems, and the reports as well.
Marginal thinking is a principle that is taught in most finance and economics courses. It tells us that in evaluating alternative investments, we should ignore costs that have already incurred and instead, base decisions on the new costs and new revenues that each alternative entails.
When faced with intense pressure to finish a project on time, the marginal cost of skipping a step just this once always seems negligible. We say to ourselves, “look, I know that as a general rule, most people shouldn’t do this – but in this particular extenuating circumstance, just this once, it’s OK.” The price of doing something wrong just this once usually appears alluringly low.
For example, Sam is a cook who was told that the Prime Minister has entered the restaurant and needs to finish his meal in one hour. Sam works swiftly to prepare the meal and, just before the meal is served, receives a phone call from one of the suppliers informing him that there’s a small chance the lettuce might have been on a truck that is suspected to have been contaminated by a poisonous substance.
As soon as Sam gets off the phone, the waitress hounds him about the status of the meal. To make matters more difficult, Sam recalls a time when his peers have taken a shortcut and received praise from the head chef for delivering on time versus another time when his peers have been hung up by an issue and received rash treatment for causing a delay.
In extenuating circumstances, most of us have been lulled into the “just this once” mindset. We justify making a small decision to break our own personal rules just one time. In the moment, it never feels like a major decision since the marginal costs are almost always low. But since life is just one unending stream of extenuating circumstances, each of those decisions aggregate and can turn us into someone we never aspired to be.
If the cook had crossed the line just that one time, he would’ve continued crossing it over and over again in various different ways. “It’s easier to hold to your principles 100% of the time than it is to hold to them 98% of the time. The first step down the wrong path is taken with a small decision. You justify all the small decisions that lead up to the big one and then you get to the big one and it doesn’t seem so enormous anymore.” – Chris Christensen in How Will You Measure Your Life
Data integrity is in constant battle with time. What makes it difficult is that the cost of skipping a step often seems lower than the cost of missing a tight timeline. Nobody will appreciate the correctness of data and everyone will point fingers if a deliverable is late. By the time any mistakes are found, it might be many months (or years) later. Few stakeholders appreciate the time required to check and verify aggregations. It’s necessary but yields no immediately tangible value that is easily measured.