To the Data Engineer: You Don't Need to Master It All
Why Being a Data Engineer Doesn't Mean Knowing Everything
A few years back, I found myself standing at the bottom of this Data Engineering mountain, thinking to myself, “Where the hell do I start?”
The outlook seemed bleak, to say the least.
Everywhere I seemed to look, an endless maze of advice: the good and the bad. Every turn filled with someone or other telling me to learn this, to do that, and follow this path. Around and round it would go.
I would see people signing up for bootcamps (dropping hundreds of dollars on them in the process), or the classic “this year’s roadmap to data engineering,” all with the promise of solving your problems.
If only it was that easy. It’s not that easy.
I felt overwhelmed and lost. I came into this thing with a good amount of time in tech under my belt, too. I can’t imagine how it must feel for newbies coming into this thing fresh nowadays.
I don’t envy you, but I want to help.
Here is My Advice to You.
Get this one thing clear in your mind from the get-go. Write it down. Pin it on your wall. Do whatever it takes to remember this…
Being a data engineer is not about knowing it all.
You can’t know it all.
I don’t know it all.
The greats out there definitely don’t know it all, even if it seems they do. They don’t. They know the things they need to know, and they do them well.
Everyone is learning as they go.
Here are facts: you will have bad days, for sure. You will most definitely struggle, but you will have good days too — loads of them.
Will you learn? Yes. Will you get to grips with the things you need to know? For sure, eventually.
You can’t just click a button (although I wish I could) or sign up for a course and know everything. It takes time, loads of time, to feel confident. You learn that in the trenches.
Companies are not one size fits all. Companies have different tech stacks, different problems, different tools, different setups, and ways of doing things. Some use Spark; some don’t. Some stream data; most don’t. These tutorials, courses, bootcamps don’t teach you that. They scare you into thinking you need to know it all.
What can you do?
Well, it’s simple, really.
It’s about being aware of what you do and don’t know.
It’s about focusing on the fundamentals.
If you don’t know something.
It’s a clear sign you need to fill in the gaps. Example: If you’re running into problems that are SQL heavy and you aren’t strong with SQL. Well, you need to ride that wave and throw yourself into the deep end and learn.
If you don’t know something.
Get good at figuring things out and finding the answers. Put on that detective hat of yours and figure it out. One of the best skills you can improve on is problem-solving. Learn to tackle problems. Find a way to solve problems, learn to think outside the box. In my experience, most problems are figureoutable, most of the time.
If you don’t know something.
It means you’re out of your comfort zone, and in my world, that’s the place to be. If you want to grow, you need to do the things that are uncomfortable. Exiting your comfort zone has far too many upsides to ignore. Remember, the scary will become normal in time. In a year, you will look back at something that scared you and see how much you’ve learned.
If you don’t know something.
It means knowing when to ask for help and when to plow on alone. I’ve seen Data Engineers suffering in silence and struggling alone. When all they need to do is ask for help. Sure, take the time to try to make a dent alone, try things, research, but if you truly are lost, ask for help (sooner rather than later). This is a great skill to learn — to know when you need help. Asking for help has helped me level up my skills and broaden my perspective. It’s tough to do, but striking the right balance between independence and collaboration is key.
Remember
This data engineering thing is tough. If someone is telling you it’s a walk in the park, they are living in cuckoo land. This thing is complicated, but also rewarding as hell. I love it.
Remember, every struggle is a lesson. Every lesson makes you stronger. You can’t know it all, but you can learn enough to get good.
Keep going.
Thanks for reading! I send this email out twice a week. If you would like to receive it, join your fellow data peeps and subscribe below to get my latest articles.
👉 If you enjoy reading this post, feel free to share it! Or feel free to click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.