6 Core Lessons for Data Engineers (No Fluff)
Skills You Need (And Ones You Don't)
People think you need to master it all to get into Data Engineering. You don’t.
You can also check the word master at the door. I haven’t met a master of anything, let alone data engineering. Anyone who is brilliant at what they do will be the first to say they aren’t. So, forget mastering anything and get to work learning and improving yourself.
I look at it this way: if a burger joint has to say they are the best in town, then chances are they aren’t. The best burger joints are the ones everyone knows about without them banging a drum about it. So, be the data engineer who gets their sh*t done without worrying if they are mastering everything they ‘think’ they need to know.
It’s called being realistic. You won’t know it all.
SQL, Python, Java, Data Modeling, Snowflake, Databricks, RDBMS, AWS, GCP, Azure, Airflow, Spark, Docker, Kubernetes — the list goes on. It’s a joke. Don’t be scared into thinking you need all of this to be a Data Engineer.
You don’t… you need a few of these, yes, but not all of it. Sure, some people know all this stuff (some better than others), and it took them years to learn. Most have an okay understanding and they get by; they are no masters — whatever that means. So why can’t you?
The way I see it, if you think you need all this to land a job or be a Data Engineer, you’ll overlook the little day-to-day things that fascinate you. Eventually, you WILL burn out… because no one can learn all that. I would rather be great at a handful than bad at all of it.
My north star is this: you know you are on the right path when it feels like common sense.
Here are six core principles I stick to as a data engineer.
1 —Make Value your Priority
You can build all the pipelines you like and knock out projects left and right, but if it’s not bringing the business any value, you may as well pack it in.
Value to the business is all that matters.
Value, to me, means time, money, and energy. If you can save or bring in money to the business, you’re golden; if you can save teams, people, or the company time, then that equals money (ding ding — value!), and if you can do it all without breaking your back, well, the triad is complete.
Train yourself to see what can be, not just what is.
2 — The Plan Is Everything
There’s a lot of “step-by-step” in data engineering, and it doesn’t lend itself well to “jumping all over the place.”
Everything you do as a data engineer should have some form of a plan. This goes for your learning plan as well as the work you actually do in the trenches. If you don’t have a plan, then you’re winging it, and winging it is the quickest way to, one, mess up, and two, take twice as long as it should to get something done. If you don’t have a plan, then something or someone will plan it for you, and 9 times out of 10, that plan is going to cost you time and give you a headache.
The plan is everything, so plan.
3 — When in Doubt, Over-Communicate
In an ideal world, everyone is on the same page. The plan is clear, the requirements are golden, and everyone knows what is needed. Reality couldn’t be further from that.
So to avoid doubt, communicate — if you don’t know, ask. If requirements aren’t clear, say so. If you go down a path and it’s murky as mud, speak up.
It cannot be any easier than that. I’ve seen engineers spend weeks on things that a 5-second Slack message could have solved. So when in doubt, over-communicate and get everyone on the same page. That, sadly, is much like herding cats into a box — near impossible, but the good data engineer tries anyway.
4 — Prepare for Failure Before It Happens
Every bit of code that is written is a ticking time bomb. Everything you build is waiting for its time to fail — and it will fail for some reason.
In effect, everything written or built is, in some way, technical debt that you will have to pay. Keep that in mind when building anything, changing anything, or planning anything. I like to think ahead and figure out all the ways something could go south. I also tend to double-check and recheck anything that changes so that, one, my ass is covered, and two, I don’t make a rookie mistake and nuke something I shouldn’t have.
The best part of failure, though, is that every time something fails, it makes the chances of it happening again less likely.
5 — Take Ownership
In my first job in data when I was 18, my manager at the time told me this: “It may not be our fault, but it is always our responsibility,” and that little nugget of wisdom has served me well throughout my career.
Most people take the “it’s not my problem” approach, waiting for someone else to either jump in and sort it out or to be spoon-fed instructions to fix it. Then there are the folks who are quick to say, “I didn’t touch it,” and wash their hands of the problem.
If you want to be good — really good — then take responsibility for all the problems your team has to deal with, whether you caused it or not. Doing so will set you apart and shine a light on you as a person and a professional.
You can teach someone tech, but you can’t teach character.
6 — Everything is solvable
I will rephrase that: most problems are solvable — with enough time.
There are, however, some things you need to get straight. The solution might not be the one you think it will be, and the new solution might well cause other problems not yet seen. I like to keep an open mind when it comes to data problems. I’ve seen some big issues in my time — ones I thought we wouldn’t be able to figure out — but we did. Whenever I run into a problem, I like to keep the mindset that somewhere out there, some guy has had the same problem I’m facing and nailed it. The odds are in your favor to find a solution.
With enough time, most things are solvable.
📩 Think this is valuable? Share it with someone who might benefit.
🔥 Last Week's Most Read Article 🔥
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.