Notes From A Data Engineers Journal
Lessons Learned from a Career in Data Engineering
Anything worthwhile comes at a price, and being a good data engineer is no different.
The obvious cost is the constant, never-ending learning and managing complex data systems. But the hidden price people don’t talk about is the fear, doubt, uncertainty, and phases of imposter syndrome that come with it — all of this is overlooked until you’re in the trenches dealing with it day in and day out.
I tell you this because everyone experiences these feelings at varying levels throughout their career. I’ve been working in data for well over 20 years, and I still feel like a newbie sometimes. Best way to put it is we are all in the same boat. Some folks just handle it better than others.
The best way I’ve found to deal with it is time and curiosity. If you don’t find what you’re doing fascinating, then I’m sorry to say the next few years are going to suck, big time. You’ll either burn out or, like many of the new kids like to do, quietly quit.
You need to find the joy in all this data stuff. You also need to realise that learning all of this takes time — a lot of it. Time is your secret weapon. If you figure that out, you can play the long game. It’s all about consistency. If you can consistently show up, even when you don’t feel like it, you’ll slowly (I said slowly) go from beginner to pretty good, maybe even great. I wish I’d learned that earlier in my career, instead of wasting time on things that didn’t improve my skill set or myself, but hey ho — live and learn.
For you data engineers coming up, here’s a list of things I’ve learned the hard way. I hope some of this resonates with you, and you don’t have to suffer like I did to figure it out.
Mindset & Learning
Be curious, learn fast.
It’s okay to not know it all.
Build a learning habit.
Don’t wait for someone to tell you what to learn — take charge of it yourself.
If you don’t know, ask — go find out about it.
Sometimes you have to get it wrong to get it right.
The best way to learn is to fail.
Some lessons have to be experienced before they can be understood.
Books don’t teach you real-world data engineering — doing it first-hand with real-world data does.
Waiting for the right time never works. You’re never ready.
Time is your secret weapon.
Technical Skills & Best Practices
Automate yourself.
SQL isn’t going anywhere.
Data quality is everything.
Filter early, aggregate late.
Most pipelines can be simplified.
Break down pipelines into small tasks.
Protect your focus time at all costs.
Good enough is better than perfect.
Always take a backup.
Don’t reinvent the wheel with every pipeline.
You don’t need big data solutions for simple problems.
Build a fundamental grasp of the basics of data.
Always optimise for scale, because it WILL need to scale.
Include testing in every piece of code you write.
The best way to optimise a pipeline is by figuring out where it will break first.
“These numbers don’t match” are some of the worst words you will hear in data engineering.
Project & Task Management
Plan first, do second.
Document everything you touch.
Share the things you’ve learnt.
Plan on the plan not going to plan.
Know how you will spend your day at work.
Look everything up, write everything down.
Cost-effective.
Build trust by delivering quality data.
Figure out the problem you are trying to solve first, then execute.
Better to take a little caution than have big regrets.
Soft Skills & Personal Development
Leave your ego at the door.
Keep working on your soft skills.
Fewer distractions and less messiness mean more productivity.
You can’t control what happens, so instead of worrying about what will happen, focus on the things you can control.
You do not have clear thinking if you are under pressure. Period.
Getting the right things done (easily) is the name of the game.
Career & Long-Term Perspective
The hardest task in data engineering is stopping the goalpost from moving.
Build a fundamental grasp of the basics of data.
Experiment with new tools, tech, and techniques.
SQL isn’t going anywhere.
Data modelling — real data modelling — is a dying art.
When you make a mistake, the only way out is slow.
Everyone loves new stuff, but it’s the tried and tested tools that stick around.
I love learning about how others work! If you have any tips or advice from your own experience, please drop a line in the comments. I’d love to learn from you and hear your story.
📩 Think this is valuable? Share it with someone who might benefit.
🔥 Last Week's Most Read Article 🔥
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.
Another winner, Tim! :{>