Streaming at Scale
From Hands-On Flink to Real-Time at King
Our latest Data & AI Stockholm gathering wasn’t just a meetup; it was a full day of learning, building, and deep discussion.
We started the day with a hands-on workshop on Apache Flink, where participants gained practical experience working directly with the framework. It was incredibly valuable to move beyond slides and actually experiment, test, and learn by doing.
In the evening, together with Apache Flink and King, we hosted a meetup on streaming at scale, diving into real-world systems that process billions of events per day.
Paris Carbone – Rethinking State in Flink 2.0
Paris Carbone took us into the architectural evolution of Apache Flink 2.0, focusing on one of the most fundamental aspects of stream processing: state.
By decoupling state from operators, Flink enables faster recovery, greater elasticity, and more cloud-native execution while preserving the semantics and guarantees developers rely on.
It was a powerful reminder that large systems can evolve internally without breaking what developers depend on externally.
Anis Nasir – Building Kingestor for 100B+ Events per Day
Anis Nasir shared how King built Kingestor, their real-time ingestion platform processing over 100 billion events per day.
From Kafka ingestion to Avro files in Google Cloud Storage and micro-batching into BigQuery, the architecture balances speed, integrity, and cost. One key challenge discussed was the skewed distribution of events and how dynamic partitioning helped restore balance across subtasks.
The outcome:
~10 minute ingestion latency
Strong reconciliation and data integrity
Disaster recovery capabilities
10x cost reduction
anis_kingestor_flink_meetup
Real engineering at real scale.
Reza Sadraei – Making Real-Time Accessible with RBEA
Reza Sadraei introduced RBEA (Rule-Based Event Aggregator), King’s internal self-service platform for real-time analytics. He highlighted the hidden complexity of real-time applications, including scaling, checkpointing, monitoring, and output guarantees, and how RBEA abstracts this through a scripting model built on Flink’s broadcast and keyed state patterns.
The result is a system that empowers teams to define streaming logic safely, without managing infrastructure.
Ending the Night with Discussion
We wrapped up the evening the way we always aim to at DAIS, with group discussions.
Breaking into smaller groups, we reflected on:
State and scalability
Handling skew and operational trade-offs
Democratizing streaming across teams
The room was buzzing. Conversations sparked new ideas, new questions, and new connections.
That’s what makes this community special. Not just the speakers. Not just the technology. But the space to think together.
A huge thank you to Apache Flink, King, Paris Carbone, Anis Nasir, and Reza Sadraei, and to everyone who joined both the workshop and the evening meetup.💙
This is how we grow: hands-on learning, real-world insights, and honest discussions!
> This is written by Vanessa Andersson, for Data AI Stockholm. More about Data & AI Stockholm. If you would like to contribute or write to us, please reach out to me or DAIS on linkedin