How to Stay Sane and Productive in Data Engineering
Smarter Habits for Data Engineers
Someone once told me, “You can never have enough help.”
I’ve said it before, and I’ll say it again: Data Engineering is a team sport — if you want to do it well, you need a strong team.
I will say this, though: Data Engineering is not all-day pairing sessions and collaboration. A lot of the time, it’s also a solo grind. It has to be. You NEED thinking time. You will spend large chunks of time alone — planning, tweaking, learning, building, and coding to make things work.
It’s the nature of the job.
Not all companies are the same, but in my experience, Data Engineering often involves a lot of organised work, and much of the time, you need to be methodical in what you are doing.
No mess-ups. No mistakes. No cutting corners.
That requires concentration and uninterrupted time to get in the zone (to get things done).
If you like to work by “jumping” all over the place and you’re easily distracted — i.e., picking up your phone every chance you get — Data Engineering is going to suck — big time.
What I’m saying here is that you have a lot of responsibility when working with data (more than you think). If a company has a data engineer on staff, then that company values data, and therefore data is critical.
You need to be in the zone a lot of the time. Spinning plates is not an option. Data needs safe hands, especially when you’re juggling pipelines, ETL jobs, data modelling, and monitoring everything going on.
You need to have your shit together. Here’s how I try to do that and stay in the zone.
As always, take what works and apply it to your way of working, and toss away what doesn’t.
1 — Automation, A Blessing and a Curse
We have a rule in my team that we live by: First time manual, second time scripted, third time automated.
Automation is a lifeline, but it can also be a trap if you sink time into automating the wrong things — things that don’t need to be automated. You can end up blowing loads of time fixing problems caused by unnecessary automation.
Before you start automating, you should:
Make sure it needs to be automated (most things don’t need to be).
Make sure the process is stable and repeatable enough for automation.
If you don’t, you’ll waste time automating (and fixing) instead of solving the original problem and bringing value (fast). When you do automate, focus on letting the automation do the heavy lifting for you:
CI/CD for pipelines: Use GitHub Actions for deployments and testing your code.
Monitoring and alerting: Integrate alerts into Slack or your favourite communication tool to catch errors early.
Write self-healing scripts: Create scripts that retry failed jobs, handle schema issues, check if source data is ready, etc.
Automation should save time, not create more work.
2 — Standardise Everything
Folks like to reinvent the wheel any chance they get. If you want to survive and not have to put out fires day in and out. Make wheels and then use them all the time. This will save you time.
Stick to conventions.
Write boilerplate code for common tasks (ingestion, transformations, queries, exports).
Reuse code wherever possible to save time and effort — rinse and repeat.
Keep repos, buckets, permissions, and datasets organised so (future you) or new you can pick up and follow on easily.
Think modular: create reusable functions for tasks like API extractions or BigQuery DDL changes.
Centralise reusable functions in a shared utilities repository.
Keep things simple, don’t reinvent wheels. Make wheels then use them.
3 — Document Like a Future Engineer Is Watching
When you work solo, there’s a tendency to fly under the cover of darkness like Batman (warning: there are a lot of Batmans out there). Somehow, things get done, and no one knows how. Until three months later, someone asks, “Hey, how does this work?” or “Who wrote this?” I call it the fairy effect.
Document everything, either through team documentation or solid resources that are accessible for everyone. If you track work through JIRA, comment the crap out of your tickets and leave an audit trail.
Write playbooks. Document how to troubleshoot common pipeline issues.
Data dictionary. Use tools like GCP Data Catalog or dbt docs to make data assets discoverable.
Pipeline diagrams. Tools like dbt DAGs or Airflow itself can help.
Comment your code. A few lines explaining logic or assumptions can save hours of reverse engineering later.
Document for your future teammates (or yourself six months from now).
4— Protect Your Time
Being in data means you get a lot of questions, and questions can sink your day. You’ll be pulled in every direction if you’re not careful with your time. Say no strategically to stay focused when you’re busy.
Batch requests. Block out time for answering email, Slack, and general ticket work. This helps you avoid constant interruptions.
Prioritise ROI. Work on tasks that deliver the most value. Tackle the low-hanging fruit first — 80/20 rule, anyone?
Set boundaries. Define clear work hours and stick to them. It’s okay to be unavailable sometimes.
Hemingway once said, “The only thing that could spoil a day was people, and if you could keep from making engagements, each day had no limits.”
You only have a few short hours in a day, you need all the focus time you can get. Make sure you protect your time as best you can.
5 — Know When to Ask for Help
Every engineer will hit a brick wall now and then. That’s just a fact. The best engineers out there know when to ask for help rather than waste weeks trying to figure it out alone. They recognise when they need outside help and support, so know when it’s time to call in reinforcements.
Get over it — For some reason, there’s a real reluctance to ask for help in this industry. Get over it and just ask. Whether it’s clarity, more details, or guidance, people are often more willing to help than you think.
Ask for help in person —The best way to ask for help is in person. Remote work makes this tricky, but a quick Slack message asking someone to jump on a call is a great alternative.
Don’t let ego or hesitation hold you back — asking for help is a strength, not a weakness. It will free you up so you can move on and get back to your focus time.
📩 Think this is valuable? Share it with someone who might benefit.
🔥 Last Week's Most Read Article 🔥
Thanks for reading! I send this email weekly. If you would like to receive it, join other readers and subscribe below to get my latest articles.
👉 If you enjoyed reading this post, and think this is valuable? Share it with someone who might benefit. Or click the ❤️ button on this post so more people can discover it on Substack 🙏
🚀 Want to get in touch? feel free to find me on LinkedIn.
🎯 Want to read more of my articles, find me on Medium.
✍🏼 Write Content for Art of Data Engineering Publication.