17 Non-Code Lessons Every Data Engineer Should Remember (Especially Me)
No-Cost Secrets to Thriving as a Data Engineer
Many paths lead to the same castle.
I’ve never met a Data Engineer who’s followed the same path. Some have been moms-to-be and then took up coding. Others were software engineers, got interested in data, and made the switch; a few take the conventional route, study something related to data, get a degree in university, intern, and land a job. Most of the time, something happened. Something else happens then bada bing and boom, your path leads you to Data Engineering.
I guess, what I’m getting at here is, the events that lead you to data engineering are not the same. I get asked a lot, “How do you become a data engineer?” I don’t really have an answer. Everyone’s path is different. It’s not following some roadmap, I’ll give you that for free. It’s a combination of things, a passion for the job, dedication to learning and solving problems and, yes some luck I suppose.
Did I grow up wanting to be a Data Engineer — Nope. I just found my way into the field and found it to be rewarding. I’m not going to lie to you, it’s been a long hard slog at times. Tough for sure. I’ve had a lot of downs (tons of them) and a fair amount of ups. I’ve doubted myself many times — still do, but I keep at it and try to be a little bit better every day.
I’m not a rock star data engineer, but I work hard and I like figuring things out. Somehow it works for me.
This post is a summary of all the things I like to remind myself about from time to time. Things I forget along the way. The things I’ve learned. The things I’ve had to relearn.
I hope it helps someone coming up or those of you in the trenches with me.
1 — Resilience
Be okay with making mistakes and looking dumb. Mistakes happen. Such is life. When they do happen (because they will), own them and fix them. Your best learning comes from making mistakes. The dumb mistakes are even better because you won’t be making that mistake again, not anytime soon.
2 — Determination
A strong work ethic builds confidence. Do the work, put the time in, do your best. Hard work breeds confidence. We live in a world where people look for shortcuts, the easy path. There is no easy path, only hard work. Why? Hard work works.
3 — Education
Read books and apply what you learn. That means reading documentation too, no getting around that I’m afraid. If you want to be a half-decent data engineer, the official documentation should be your go-to. Read articles by other good data people, learn from them. Keep up to date with the data world. It’s a no-brainer, but lots of people don’t do this.
4 — Adaptability
Not everything has to be so hard. When things start becoming complicated (the code, the pipeline, etc.) and seem hard, you’re doing it wrong. Hit the brakes and find another approach — there’s always another way to look at problems.
5 — Strategy
Break down the complex problems into small things you can understand; doing so lets you see the big picture. Problem solving is an art and art is creative; you need to be creative in your problem-solving.
6 — Investment
Invest in yourself. In the long run, investing in yourself has massive gains. If you want to go on a course, just pay for it (and actually do it). You want that book — buy it, read it. Too many times I’ve talked myself out of getting something that would’ve improved my skill set. Don’t do it, just go for it.
7 — Completion
Finish strong and finish what you start. Random hustling is not productive; find a flow state and ride that wave. Begin something and work until it’s done. Find a productive way of working and get the job done. Having 5 tickets on the go is not productive.
8 — Curiosity
Search for the things that interest you and gravitate to it. They are stepping stones to other interesting things. I’ve learned a hell of a lot by following my curiosity; it’s made me a well-rounded data person.
9 — Reflection
Reflect on your journey. People like the winner at the end of the story, but the struggles and middle are where the learning and growth happen. Remember those moments, all the things you had to go through to get to where you are. Celebrate the journey.
10 — Initiative
Do the work no one wants to do. I’ve learned a lot by doing the crappy tasks, by digging into tasks no one else wants to do. It’s also opened a lot of doors for me, one I’d have never found if I didn’t suck it up and got my hands dirty.
11 — Accountability
Just because it’s not your fault doesn’t mean it’s not your responsibility. I’ve seen data engineers turn blind eyes to issues that were problems that they didn’t cause, so it’s not their problem. Wrong. If you want to be a solid data engineer, you need to take responsibility for any and all problems.
12 — Focus
Comparing yourself to Mr. LinkedIn who seemingly knows it all is not healthy. Focus on the things you can control like working hard and persevering.
13 — Calm
Calm is contagious. Be calm to help others. When things are blowing up all around you, be the calm and slow down. Work the problem and fix it.
14 — Attention
Any skilled work requires you to give attention to what you are doing. I love that data engineering requires a high level of concentration at times. When you work with data, attention to detail is where it’s at.
15 — Mentoring
Be more generous than necessary. New starter coming on board? Someone shows an interest in learning? Spend time helping them grow and learn. “Each one, teach one.” Helping those around you level up is the best way to lift yourself up.
16 — Prioritize
Sweep the shop. Tend to the small things. The small things will overwhelm you if you don’t keep them in check. Hit the small things first so the big things get more of your attention.
17 — Insight
Every person you meet knows an amazing lot about something you know nothing about. It won’t be obvious at first, or it just might well be, but most people I work with have amazing skills that I get to learn firsthand from.
Thanks for reading! I send this email out twice a week. If you would like to receive it, join your fellow data peeps and subscribe below to get my latest articles.
👉 If you enjoy reading this post, feel free to share it! Or feel free to click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.