Scaling Metrics at Spotify: The Story Behind Data Engineering Challenges

A recap of Spotify’s keynote presentation at the Data & AI Stockholm event.

In September 2025, Spotify’s engineers, Annica Ivert and Erisa Dervishi, lifted the curtain on one of the toughest challenges faced by their teams for our Data & AI Stockholm community: calculating Company Key Metrics (CKMs) at a scale few companies in the world ever reach.

These metrics, from high-level indicators like Daily Active Users (DAU), Monthly Active Users (MAU), churn, retention, and subscribers, to more granular measures of product success such as engagement, content hours, and feature-level data, may appear as simple numbers on a dashboard. But inside Spotify, they are the rhythm that guides strategy, aligning thousands of employees and shaping decisions about growth, engagement, and business direction.

Yet, producing them is far from simple.

Every day, Spotify ingests over 11 terabytes of streaming data, reflecting billions of user interactions. CKMs must be 100% accurate! There is no room for approximation when they influence the company’s most important choices. They must also represent real user behaviour, which means bots and artificial streaming must be filtered out. And they must be delivered on time, as they feed countless downstream systems that depend on them.

The scale of the problem became even clearer when Spotify hit a wall with its cloud provider a few years back. On Google Cloud, the costly process of redistributing data between workers, known as shuffling, had an upper bound of 200 terabytes per day. Spotify’s CKM pipelines had already exceeded that limit. To continue scaling, the team had to invent more innovative ways of working with data rather than simply throwing more hardware at the problem.

This is where optimisation became not just a matter of efficiency, but a matter of survival.

The engineers walked us through the process of how they tackled the challenge. They built systems to filter data as early as possible, reducing unnecessary load downstream. They optimised joins, using techniques like Bloom Filters and Sort-Merge-Bucket joins, which colocate keys during write so later computations don’t require expensive shuffle operations. They introduced smarter rollups, carefully pre-aggregating data to avoid double-counting and prevent the need for entire re-computations of DAU across hundreds of days.

One clever solution addressed the problem of fraud. When fraudulent activity was discovered in the system, the naive approach would have been to recompute metrics across the entire dataset, an impossible task at Spotify’s scale. Instead, the team developed a method to recalculate metrics only for the fraudulent users, leaving the rest unchanged. This single idea dramatically reduced cost and computation time.

At its core, the system is orchestrated by Flyte, Spotify’s backbone for reproducible and scalable data workflows. But the true innovation lies not in the tools, but in the philosophy: build pipelines that can handle growth not by consuming infinite compute, but by making more intelligent trade-offs at every stage.

Spotify’s journey shows that scaling data is not about brute force. It is about the engineering discipline of balancing accuracy, efficiency, and speed. For every DAU or MAU figure reported, there is a story of distributed joins, incremental updates, fraud detection, and a fight against shuffle costs at the petabyte scale.

And in the end, it is this hidden machinery of data engineering that keeps Spotify’s metrics not only alive but also trustworthy, enabling the company to keep pace with its mission and its millions of listeners worldwide.

> This is written by Vanessa Andersson for Data AI Stockholm. More about DAIS. If you would like to contribute or write to us, please reach out to me or DAIS on Linkedin