There are two things that will make your team look like amateurs: someone pinging you on Slack, telling you your data is bad/wrong/weird, or — worse — you and your team not even knowing about it — that last part keeps me up at night.
I’d down tools for half an hour and hide in the bathroom in shame. If you have the word “Data” or “Engineer” in your title, you have no excuse for bad data quality.
None.
In some ways, maybe data quality isn’t even the issue. There are so many tools and frameworks out there now that you have no excuse not to be checking things along the way. Yet, data quality is still front and centre — maybe it’s just one of those things that will always be there, like mistakes.
You will make mistakes. Things will go wrong, no matter how experienced you are. With data quality, there are no silver bullets, no quick fixes — just constant pain. In my opinion, it’s one of the biggest issues plaguing the industry.
And it’s 2025! Yes, 2–0–2–5!
Keep reading with a 7-day free trial
Subscribe to Art of Data Engineering to keep reading this post and get 7 days of free access to the full post archives.