Redis Streams is a new data structure, introduced in Redis 5.0, that allows you to create and manage data streams. In a previous article, I showed how to add data to a stream, and how to read the data in multiple ways. In this article, I’ll explain how to use consumer groups in Redis Streams. A consumer group is a way to split a stream of messages among multiple clients to speed up processing or lighten the load for slower consumers.
In a perfect world, both data producers and consumers work at the same pace, and there’s no data loss or data backlog. Unfortunately, that’s not the case in the real world. In nearly all real-time data stream processing use cases, producers and consumers work at different speeds. In addition, there is more than one type of consumer, each with its own requirements and processing pace. Redis Streams addresses this need with a feature set that gravitates heavily towards supporting the consumers. One of its most important features is the consumer group.
When to use a Redis Streams consumer group
The purpose of consumer groups is to scale out your data consumption process. Let’s consider one example — an image processing application. The solution requires three main components:
