Data Engineering Success Starts with One Simple Habit - Organization
Key lessons for improving efficiency and staying organized
I work better under pressure.
I’ve heard that time and again over my career. Hell, I’ve said it a few times myself, back when I was young and, let’s face it — stupid. No one works better under pressure. They just don’t. If you think you do, I’m afraid you are wrong.
I don’t want anyone who’s under pressure calling the shots, making decisions, or pushing buttons on a computer. Nor do I want this person handling data / heavy machinery of any sort. This screams problems to me — big problems all over the shop. It’s a one-way ticket off a cliff.
You would not get on a plane if the pilot announced at takeoff,
“Oh, by the way, I’m feeling really under pressure today, but don’t worry, I got this.” That would leave passengers scrambling for the door.
I come from database land, where one wrong move means goodbye data and/or your job. Safe hands, folks. Slow and steady. That is what it’s all about. Data engineering, to me, is no different. It calls for an organized mind and clear thinking.
You do not have clear thinking if you are under pressure. Period.
A great Data Engineer is organized and efficient. No last-minute flapping around, thank you very much. I want more time thinking, planning, and organizing my day, week, or task. I can then proceed in an orderly fashion to get shit done — usually pretty quickly too (I said usually).
I don’t know why I see so many disorganized engineers out in the wild. Maybe I’m missing something? Folks seem to mess about for days on end and then, last minute, expect everything to fall into place — it never does (if you were wondering).
This, to the outside person, screams u-n-p-r-o-f-e-s-s-i-o-n-a-l.
Therein begs the question: Do you want this person poking around your data? Building things that your business is making decisions on? Hmmm…
How do you become a more efficient Data Engineer? And more organized? Here are some tips on the subject from yours truly.
Plan before you code
Actually, plan before anything. This is just good old common sense. It’s so obvious, but it’s a step often missed. Sit down, think through the task at hand, and map out what you are going to do from top to bottom.
How I do this is: I get out the old notebook and pen and make notes (yes, caveman style). Then proceed to dig into code bases, investigate anything that is prerequisite, and examine source data, files, you name it. This in turn leads to more questions and more notes. At which point I then try to map out a flow of the steps involved and things I need to do or get clarity on. Which leads to the next point.
Which way am I going? Clear Directions
The biggest waste of time you will encounter is unclear requirements. This is public enemy number one. If you can nip that problem in the bud from the get-go or early on, you’re halfway to being a more efficient Data Engineer. If you don’t, you are digging your own grave, and you have only yourself to blame.
If you don’t know, ASK! It’s so simple, yet Data Engineers plow on headfirst into walls. If you learn to ask better questions, you will get better guidance and clarity. The best way to ask better questions is to try to map out the things you are trying to do.
Keep it clean
This goes for your code and your workspace. If I see a cluttered desk or cluttered desktop (both physical and virtual), this screams cluttered mind. It tells me you are spinning plates and are therefore (in my mind) distracted.
Work clean.
Fewer distractions and less messiness mean more productivity. Working clean also means trying to get solid chunks of focused time. No Slack, no phones, no messages, no meetings. No pinging from annoying notifications. The zone — that is what it means to keep it clean. Answer messages and emails (if possible) twice a day: once in the morning and again in the afternoon.
Build Simple Things
Try to build simple things. Simple is the way to go — clean and efficient. I see complex solutions for simple problems all the time. I see over engineering constantly. I don’t know why, but people like to overcomplicate basic things. When you write code or build something, always assume the next person that has to work on it will not understand what you were trying to do.
More often than not, the person coming along is going to be you, 6 months later. If you write it clean and simple, you will be able to get things done quicker the next time you come around. If I can’t explain what this piece of code or pipeline is doing in 5 minutes or less, then what will it be like in 5 months when you’ve not touched it? Simple is always better.
Build your experience, Keep improving
If you have to investigate a schema or database, you will be less productive.
If you have to keep looking for answers about who does this or that, you will be less efficient.
If you have to copy-paste code from ChatGPT or StackOverflow instead of knowing how to write it yourself, you will be less efficient.
If you are writing code that already exists somewhere and trying to reinvent the wheel, then you are going to be less productive.
If you don’t know your infrastructure or system setup, you will go slower.
If you can’t find the time to do the job right the first time, what makes you think the second time will be any faster?
If you keep learning, keep improving, and keep pushing yourself to find ways to be better, you will gradually become a more efficient data engineer. That takes time. It’s up to you how much time.
Final Thoughts
No one cares how many commits you made, what tool you used to get the job done, or whether you copied code or wrote it from scratch.
They don’t care.
What matter is that the job got done.
What you should care about is the easiest way to get quality work done. Note the word quality there. No one wants to go back and refix or plug holes because you didn’t get it right the first time.
Getting the right things done (easily) is the name of the game. This is the difference between an average data engineer and a great data engineer. The question should never be how much can I get done (because you can be busy and have a lot going on, but it doesn’t mean you’re getting anywhere).
It should be, What is the easiest way to get things done? Organisation is the only way to become efficient.
🔥 Last Week's Most Read Article 🔥
📩 Think this is valuable? Share it with someone who might benefit.
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.