5 Key Mindset Shifts to Succeed in Data Engineering
Forget the Roadmaps You've Seen - Create Your Own Path in Data Engineering
Here is something I wish I knew from day one.
There are no shortcuts in data engineering — none.
Not one.
No secrets. No hidden backdoor. No cheat codes.
I know this because I wasted a lot of time trying to find them.
There are no roadmaps either, although the internet is littered with them. Your road, the one you end up taking, is not going to look anything like some roadmap you find on the internet. Roadmaps are, at best, a list of things you may come across in your career.
Sure, some stops will look familiar. Sure, it will probably point you vaguely in a half-decent direction, but the road isn’t going to be the same. No way. It’s not a straight path or clear shot. Why? Because you are not in the same boat as the person who wrote the roadmap.
If (and it’s a big if) by some miracle you actually do complete the roadmap (you won’t), there is no pot of gold waiting for you. The homework does not guarantee anything. I wish it was that simple. Tick these boxes, then become a rockstar Data Engineer? Nope, doesn’t work that way.
Fact is, ticking off items on someone else’s perceived path to Data Engineering is not going to work. You will likely spend a lot of time learning things you may or may not use. The industry is looking for a unicorn, and these roadmaps are part of the problem.
It’s a mountain…
It’s also hella daunting looking at someone’s roadmap and thinking to yourself, “How am I going to learn all of this?” Nine times out of ten, you will struggle through it, lose motivation, or worse, lose your confidence. It’s also very possible that you will forget everything that you learned. Why? Because you don’t do it every day!
So, what should you do? The first thing you need to do is get honest with yourself. That means realizing there are no shortcuts! Just hard work, dedication, and consistency. Do all three of those things, and you will be on the right path.
I’ve spent some time thinking about all the advice I’ve been given over the years by Data people doing various things in various roles. I’ve tried to condense it down for you and for me here.
1. Get Smart with the Basics
I preach fundamentals every chance I get. Why? Because if you are starting out, you need a fundamental grasp on the basics of data. You need to know how the basics work, and you need to know them backwards.
It’s been my experience that any given discipline relies upon the fundamentals.
You need to be able to fall back on a solid understanding of the most basic skills in data, then build on them.
SQL — Learn it: It’s been the backbone of my career in data, and it is the number one method for slicing and dicing data. It is in almost every job that involves data. SQL is the number one foundational skill you need in data.
Data Modeling — Data modeling — real, data modeling is a dying art. I’m generalizing here, but I’m starting to feel like it is. Understanding how data is structured and how to structure data efficiently is critical. Learning how to model data will level you up faster than any roadmap you stumble upon.
Problem Solving — There are plenty of problems out in the wild. It amazes me how many different and unique problems I’ve come across in my time. Some will be the same, others will put you in positions to learn and grow. If you can get good at analyzing a business problem and solving it with data, you’re going to be printing a golden ticket to Willy Wonka’s chocolate factory.
2. Don’t rely on AI
What a time to be alive, eh?
We live (for better or worse) in this make-believe world where ChatGPT makes things ‘easy.’ It is slowly spoon-feeding us all to the point where we cannot do even basic things without our AI fix. This is it now, this is the new ‘normal,’ but there is a difference between using AI and relying on AI.
Like all good tools, you should use it when you need it, and then put it down. Leverage it in a collaborative way to enhance your abilities rather than let it slow your growth. AI is a powerful tool, but it will never replace the need to deeply understand the fundamentals. Develop your skills, then let AI amplify what you already know, not mask what you don’t know.
If you are starting out in data, you are going to gravitate toward it for all your problems. But if you do that, then it will become a crutch. It leads people to think this data thing has got to be simple, right? It’s not.
I love Procreates take on this whole AI thing - “AI is not our future”
3. Hard = Growth
It’s the old saying — “Easy choices, hard life. Hard choices, easy life.” It’s that simple — put the work in (the hard work), get the rewards.
Normally, what happens is when you put the work in, nothing happens quickly enough, so the newbies out there give up. It’s a consistency game. It’s showing up day in, day out, working, learning, and growing your skillset every day, bit by bit. I wish there was a silver bullet, but there isn’t one.
Like everything in life, if you are willing to make the short-term sacrifice, you’ll have the long-term benefit. No one wants to hear that, though.
4. Make Time
“I don’t have time” is code for it’s not a priority. I used to tell myself this all the time like it was an acceptable excuse to have. “I don’t have time, I have kids.” It’s all a mindset shift. If you want something bad enough, like learning Python or SQL or writing online, then you will make the time. I push myself hard because I know there is always another level; there is always something to learn. I do this for me, and I do this for my family. I make time. If you want something bad enough, you will find a way. Look at your schedule honestly, and you will see areas where you could find time to upskill and learn. People are masters at wasting time (gaming, Netflix, Social Media). There is time in your schedule — you just need to make it a priority.
5. Learning is failing
Sometimes you have to get it wrong to get it right. The best way to learn is to fail. To get something wrong and then get it right the second or hundredth time. No one gets anything 100% perfect the first time. It’s all part of the game. Every new bit of knowledge you acquire builds on what you already learned. I think of them like bricks in a building. Foundations first, then things you learn (bricks) — this is how you build experience.
I struggled for years to pick up things because I just wasn’t into it. The best way I’ve found to learn is when I’m fully engaged in the content. For me, this means getting hands-on. Trial and error. Picking problems or parts of something I don’t know apart to the basics and then building it up.
Learning something new requires active engagement. We expect learning to be easy, but it’s not. Learning is hard, and it’s supposed to be like that, but it doesn’t mean it can’t be fun. Find ways to make it fun. Collaboration works for me. Investigation too — I love reading up about things and then trying them out myself. Find what works for you.
That’s that..
Look, everyone is making it up as they go along. You have to find your own way, your own path. YOUR roadmap. Find the things that work for you. Throw away what doesn’t. This data engineering game is a long, long road — one with many paths. You need to figure it all out yourself and just do the work.
🔥 Last Week's Most Read Article 🔥
📩 Think this is valuable? Share it with someone who might benefit.
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.
Pure fire, brother! I forwarded this to my mentees. Thank you!! :{>