
Data can be a battle zone. Cloud data management is hard.
If you’re not careful, things will get out of control — fast. This goes for BigQuery, Snowflake, and Databricks. Treat them like dumping grounds, and you’ll get buried in cost, performance, and all-around chaos — yes, a mess!
Once that happens, you’re on fire watch. Putting out fires will be your number one priority for a long, long time.
What’s worse, you and your team will only have yourselves to blame. All these systems are amazing and do some incredible things — things I could only dream of when I got into data — but you don’t see the hidden side of dealing with them until it’s too late: cost, clutter, and chaos, in that order.
I like to joke that cloud providers are like casinos — the house always wins. They are not endless or bottomless (as much as they want you to believe they are); there’s a price to pay. You will pay one way or another — with time (when you have to clean up) or money (when you mess up). And when you let the wheels come off, you’re going to get nailed.
Keep reading with a 7-day free trial
Subscribe to Art of Data Engineering to keep reading this post and get 7 days of free access to the full post archives.