My Advice for Aspiring Data Engineers (If You're New to This)
5 Essential Lessons for New Data Engineers
I’ve worked with many brilliant people.
All had varying skills and talents, but it was those who were great at a particular tool, product, or software that made everything look easy AND made me feel as if I still had my training wheels on.
These people woke me up and changed my approach to learning.
Here’s the type.
They know the ins and the outs of things. They don’t Google; they just seem to know — everything. They have all the answers. When things break, they often solve it and move on — just another day at the office.
Everything seems easy for them.
I remember one particular guy I worked with; he was on another planet when it came to SQL Server. He knew anything and everything about it. His only problem was he wasn’t interested in doing anything else. He didn’t care about learning another programming language and certainly wanted nothing to do with anything cloud-related. If it was not SQL Server, he wanted nothing to do with it. He wanted to be left alone to tinker away and do his thing.
You see where I am going with this, right?
He was happy with what he knew and loved being the go-to guy. His little bubble was a safe place where he was the “master.” At the time, I thought to myself, this is awesome. He’s got it all figured out.
Here’s the but though, things changed when the company decided to migrate to the cloud. I remember his face when the news dropped. He was fuming. He was pissed.
He fought the decision, moaned, and complained to all who would listen, but SQL Server’s days were numbered. SQL Server was on the way out, and just like that, the company’s priorities changed, and they started heading in that direction.
The shift didn’t happen overnight (obviously), and the company tried hard to push this guy to learn, upskill, and embrace the change. They wanted him along for the ride (his experience would have been useful), but he was having none of it. He dug his heels in, and that was that.
Long story short, the company changed, and he didn’t want to. He moved on after time and I guess found a gig that suited him — somewhere SQL Server heavy (which is fine, I guess…).
Here is the lesson I learned from this guy. Don’t build your career on mastering a specific technology, branch out, explore, and build up your skillset. Be open to change and learning new things. Don’t be the guy who gets left behind.
I got asked recently what advice I have for up-and-coming data engineers.
Here’s what I’d tell you if you were up and coming…
1. Fundamentals
If there’s one thing that I know, it’s this — tools come and go. Every other day, there’s some new silver bullet that’s going to solve all our data problems. These tools normally come with a great marketing team and an ‘influencer’, ‘advocate’, or ‘Top Voice’ telling you this is the way to go. It’s rarely the way to go. What you, as the up-and-coming rock star, need to do is focus on the skills you KNOW will stick around, like SQL; it’s not going anywhere, and those who say it will are wrong.
A programming language (Python, or dare I say Rust?) Right now, I’d stick with Python.
A cloud service? Probably AWS, but I’m biased and will tell you GCP. Version control?
Learn GIT, learn it well. You will thank yourself for knowing your way around the terminal. It has the added benefit of making you look like a pro.
2. Data Modeling
Data modeling is the unicorn of the data world. It feels to me like data modeling has become a dying art. If I were an up-and-coming Data Engineer, I would learn to data model and learn to model data well. How do you do that? Well, you pull your sleeves up and do it. Sorry, you’ve got to do it — reading about data modeling is like trying to learn how to paint by reading. You’re not going to learn unless you do it. Find a resource, something like the Kimball Data Warehouse Toolkit, and read and apply it. If you actually get hands-on and know your way around basic data modeling, you’re going to be ahead of the game, a cut above 80% of folks out there.
3. Problems Solving
Problem-solving is an art in itself. Being good at figuring things out when no one else can is a secret weapon and an unfair advantage. It’s fundamental to nearly every aspect of the job, maybe even any job in tech. The best thing about problem-solving and getting good at it is this — it transcends specific technologies or programming languages. If you know how to look at a problem, analyze the options, and come up with a plan on how to solve it, it is a skill you can apply again and again to many different situations in life, making you not just a problem solver but a valuable asset in any team or organization.
4. Effective Communication
Early in my career, I figured out that listening to people gave me an edge. It’s a common misconception that people listen. The truth is, they don’t — they’re distracted. They’re thinking ahead, or planning their response, they’re dying to say something, instead of just listening to what’s being said. As a newbie, it’s easy to think it’s all about talking. Nope, it’s about listening, and listening well. Paying close attention, tuning into the underlying message, and genuinely understanding others can set you apart from the crowd. Remember, there’s a huge difference between hearing words and actively listening.
Pro tip: Take notes (with an actual pen and paper — I know right old school) in meetings; this keeps you engaged and gives you something to look back at.
5. Daily Routine
When you’re starting your career, you need to get organized, and fast. How? Establish a daily routine! Going into your day with a plan will help you make the most of it. What are you planning to get done? What are you aiming to learn? If you don’t know what you should be doing, a routine gives you something to do; it gives you a purpose. It stops you from wasting time and helps you make the most of the time you have. Routines prevent you from procrastinating and getting distracted. Building a solid routine takes time, so take time to study your habits, your moods, and your schedule. If you can get into a good routine early in your career, you’ll make things easier for yourself, trust me.
Thanks for reading! I send this email out twice a week. If you would like to receive it, join your fellow data peeps and subscribe below to get my latest articles.
👉 If you enjoy reading this post, feel free to share it! Or feel free to click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.