The Reality of Mistakes in Data Engineering (Tips from the Trenches)
How to Avoid the What-the-Hell-Have-I-Just-Done Moments
You clicked go, enter, execute, run, or whatever the magic button is that makes things happen. The problem is, you realized you made a huge mistake. You’ve just dropped a table, deleted some rows (and forgotten the where clause), shut down the wrong server, written over some config, or the classic — you thought you were working in development.
You get it picture.
No words can describe the feeling of making a big mistake. The sinking feeling in the pit of your stomach like you just swallowed a lead ball. Your stomach slams shut like a book. Hands start shaking; you can barely move the mouse, let alone type, mind racing a million miles a second. Time stands still. You’re truly in the moment. A moment you just created.
The first thought is normally sh*t!
Second thought, hide.
Third thought, this can’t be happening, can it? — It can, and it has.
You try to reverse it, backtrack, undo, restore, but it’s too late. You’re out of luck, and you know it. The transaction is in rollback. The query won’t stop. The table’s gone — bye-bye, and you, dear reader, are about to learn a valuable life lesson, and this lesson is going to be a hard one. From this day on, you will always be known as that ‘guy’ who nuked the payment gateway, the dude who dropped the finance tables, or woke up the entire development team on Christmas Eve.
Working in data means if you mess up, half the company will know about it in record time. Everything you touch is likely plugged into every monitoring, alerting, and logging system in the world. No hiding for you — sorry. The positive side is you probably won’t make the evening news (I said probably).
That said, a mistake is only a small event on a long road that is your career.
Two Types of People
There are two types of data people out there: those who have made mistakes, and those who are going to make a mistake. There is no getting around that part. Sooner or later, your luck will run out.
What can you do about it? Well, preparation helps. Experience too. Half the job is getting your sh*t together, but the clock is always ticking, and eventually your time will come (as it does for all of us).
I still mess up from time to time, just not as epically as I used to. I know not to panic, to keep calm and figure it out. I’m also crazy meticulous when it comes to planning. So if something does go wrong, it’s because I didn’t see it coming and it was likely not something I could have anticipated. I always have a plan B, a rollback, a safety net. In the event of failure, I can say, “See, look here, I did all this to make sure X would not happen.”
In times of crisis, I’ve learned early on that you have to take these kinds of moments on the chin. Hold up your hand. Ask for help, and work the problem, backtrack, and fix. Most things are figure-out-able one way or another.
Why do Mistakes Happen?
Overconfidence. Rushing. Cutting corners. Not planning. Winging it. Spinning too many plates. Distraction. Those are a few of the reasons that probably got you into this mess. The only way out now? Going slow. Slow like you should have done in the first place. People won’t want to hear that, but that’s the truth of it. Slow is the only way out of a mess.
I’ve had a lot of what-the-hell-am-I-doing moments in my career.
I’ve also had a lot of what-the-hell-have-I-just-done moments too.
I will have many more of them, I’m sure.
That’s the honest truth.
You see, the internet is littered with hero stories, of pipelines built, systems created, projects delivered, mind-blowing ideas made real, and success stories galore, but rarely do people ever share their mess-ups.
Experience Accelerators
Mistakes are your ‘experience accelerators,’ as I like to call them.
The greats out there have made tons of mistakes. A lot of them, I’d wager. But people seem to forget that part. Why? Because we like winners — plain and simple. We don’t like hearing that good things take time, that the road one takes in their career will be littered with mistakes and wrong turns. People want to win now and not have to work and make mistakes. It’s chip-brain logic:
Failure = bad, Success = good.
It’s dumb, but that’s how it is.
To me, Failure = Learning.
Everyone I’ve worked with has a story.
One they can recall like it happened yesterday. The rookie mistake. The epic fail that taught them the importance of caution. You will be surprised to know that some people never learn.
I admire these stories. Why? Because they are real. True-to-life in the trenches kind of real. These are the stories of adventures in data. I admire the resilience, the steadiness, the ability to get it together, to take a problem and still make something of themselves.
At the time, these failures suck. I’m the first to admit that. They ain’t fun. Looking back, I’m glad for them, though. The mistakes were hard to swallow, but I learned some valuable lessons that no course, no tutorial, no guru could have taught me. I earned experience points and a lot of them fast.
I’ve deleted files I thought weren’t in use.
Locked up transaction tables.
Shut down a production server halfway across the country (and the only way to turn it on was to drive to the data center and push a button).
I’ve built things that have worked in testing, but failed miserably in production.
Did I cry about these? Nope. Did I pick myself up and carry on? Yes. Did I learn? Definitely! I’m a better Data Engineer for all of it.
My Advice to Prevent Mistakes
When you are starting out, a little knowledge can be dangerous. Very dangerous. Tread carefully. You should learn from the mistakes of others because you can’t make them all yourself. Get good at learning and watching others.
See what people did wrong, why it happened, and how it could have been avoided. Everything is obvious after the fact, like when you break up with a girlfriend and look back to see all the signposts you missed because you were too love drunk or stupid.
Learning from other people’s errors over the years has saved me a lot of pain. It’s also made me a well-rounded person and a half-decent Data Engineer. I had a colleague once tell me, “Tim, try hard not to make mistakes, but don’t be scared of them either.” That one sentence has stuck with me throughout my career and has given me the freedom to do some pretty cool things.
Pro tips:
Get organized and be meticulous with your planning.
Work clean — That means if you have a thousand Google tabs open, notes sprawled all over the place, and a mail/Slack inbox full to the brim, you are an accident waiting to happen.
Limit distractions — That means no Instagram and Facebook checking every 5 minutes.
Do not assume. If you do not know, do not push the button. Get clarity first; understand what you are doing before moving any further with your task.
Check, recheck, test, double-check, make backups, and have someone double-check what you are doing. Do everything you can to avoid disaster.
Reading this list to me is just plain common sense, but we all know common sense is not common. I have enough experience in this field to know that. I see young kids nowadays jump headfirst into brick walls all because they didn’t stop for 5 seconds to assess the situation and plan.
Newbies want to run off and ‘code.’ Coding is lame. Planning is where it’s at. That’s the part that will get you out of jail and save your ass — maybe even your job.
So, what have been some of your epic fails? I want to hear your story.
📩 Think this is valuable? Share it with someone who might benefit.
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.