How to Ruin a Data Engineers Day (in 20 Easy Steps)
A Step-by-Step Guide to Derailing a Data Engineer's Workflow
There are three things I need in my day-to-day life.
I need organisation.
I need lists.
I need a plan.
Note the above is a list (wink wink).
Sadly, when you are a data engineer, things don’t always go to plan. You have good days, and you have bad days. It's more good than bad if you play the game right.
When things are going right, the pipelines are running smoothly, you’re cranking out work at a good pace and life seems good. Something will come along and ruin your day, maybe even your week. Make no mistake, the traffic lights of data engineering life are not always green, and flowers are not always rosy.
In this article, I reached out to a handful of fellow data engineers to ask them what sort of things ruin data engineers’ days. I had a good laugh at some of the feedback, and I could relate to a lot of the responses (actually, all of them). This article is a bit of fun and a little less serious than my usual type of article.
I hope the up-and-coming data engineers out there read this and have a good chuckle to see what’s in store for them. I’m sure there will be those out there (my fellow data engineers in the trenches) who could relate to a few of the things on this list too.
Here are the top 20 things that could ruin a data engineer’s day,
How to Ruin a Data Engineer’s Day:
Merge conflict hell — This was top of the list and one of the most mentioned things Data Engineers hate. Nothing like a good merge conflict to ruin a good day.
Seeing a LOWER case (select * from) — This is my addition. This doesn’t ruin my day, but it makes me real sad. It’s SELECT not select!
Adhoc requests that are needed by the end of the week. Say goodbye to whatever plans you had that day or week. This very “important” ad-hoc request will take priority.
Internet connectivity issues, like IT deciding the tried and tested VPN just isn’t cool enough let’s upgrade on Monday or your internet supplier decides to break down in the middle of a storm. Goodbye, day.
Any kind of hardware problems (or the hardware problems that disappear after a restart, then come back randomly) — Goodbye day, hello Google.
Software upgrades that get pushed to your laptop (you don’t have a say). Goodbye day, hello Google and IT support.
A rogue schema change — somewhere up the chain some dude decided it would be a good idea to make a schema change and not let anyone know (or not let enough people know). Time to play detective and figure out where it all went wrong.
Slack message saying “Quick question” — There is no way it will be quick — never. Time to practice those deep breathing exercises.
Zero requirement tickets. Not understanding where to start, now you have to ask around and find out what actually needs to happen.
The scope creep monster. He lurks in every nook and cranny, disguised as a simple request that turns into a monster of a task.
Subqueries in production code — Again, not a day ruiner, but this makes SQL fairies out there die and makes me want to cry.
Getting invited last minute to meetings that just started. If you weren’t invited in the first place and get invited last minute, you know you’re in for a world of pain.
Zero documentation — I’ll set the scene: You ask: How do you do this? Is there documentation? Reply: No. Well — you’re about to reverse engineer the crap out of something to figure it out.
Your estimates (or guesstimate) just became a deadline.
Too many meetings equals not enough done and low impact day. A single meeting can wreck an entire afternoon. It does this by splitting it into two parts, each too short to tackle any challenging tasks. Your day is as good as gone, see you tomorrow.
Someone logging a ticket, sending me a Slack message (and or email) and coming past my desk to tell me they raised a ticket. I call this the three-pronged attack. If you’re in this situation, you’re in trouble — big trouble.
You’re just about to wrap up a long project, and the end user decides the goalposts need to be moved because he thinks this report will look better with a different kind of metric. Time to go on a little walk and cry.
Constant never-ending interruptions — ping, ping, ping. Nothing says “game over” quite like constant never-ending notifications.
A good old-fashioned power outage — Queue the rapid pairing of your phone as a hotspot, and you constantly look at your MacBook battery life. This is modern-day survival.
The vanishing colleague — That moment you realize the only person who understood that one crucial part of the code, process, or pipeline has left for a two-week vacation, leaving no notes and documentation behind (see number 13). Good luck!
Wrapping Up
Your days as a data engineer will be filled with many challenges. For me, challenges equal experience and make you a more rounded data engineer. If things always went to plan, life would be dull, and data engineering would be boring. It’s the problem-solving, quick thinking, and ability to adapt that will make you a great data engineer.
The above list is an example of all the types of things you will have to navigate in your journey to being a good data engineer. The trick here is to take it all in your stride, plan, prioritize, and get your sh*t together. Doing so means any of the above won’t knock you or your confidence to get stuff done. Would love to know the sorts of things that ruin your day? Fire them off in the comments section. Let’s laugh together.
Thanks for reading! I send this email out twice a week. If you would like to receive it, join your fellow data peeps and subscribe below to get my latest articles.
👉 If you enjoy reading this post, feel free to share it! Or feel free to click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.