The stateful nature of WebSockets carries inherent complexities regarding horizontal scaling. Kubernetes is a natural fit for stateless workloads, but operating scalable stateful applications can be challenging. We discuss these challenges in the context of an EV charging IoT platform. A concrete problem we faced revolves around WebSocket reconnection storms: if thousands of charging stations lose their connection, e.g., due to regional network carrier outages, they all try to reconnect simultaneously once the network issue is resolved, thus causing large load spikes. These load spikes propagate across our systems, potentially causing positive feedback loops and introducing instability. In this talk, we dive into the intricacies of scaling out WebSockets and how we evolved the system architecture towards event-driven communication. We discuss how introducing a Redis-based circuit-breaker component in the connection flow helped us alleviate bottlenecks and improve system stability.

Saadi Myftija

Software Engineer - Architecture & Cloud