Redis, the in-memory multi-model database, is popular for many use cases. These include content caching, session stores, real-time analytics, message brokering, and data streaming. Last year I wrote about how to use Redis Pub/Sub, Lists, and Sorted Sets for real-time stream processing. Now, with the arrival of Redis 5.0, Redis has a brand-new data structure designed to managing streams.
With the Redis Streams data structure, you can do a lot more than what was possible with Pub/Sub, Lists, and Sorted Sets. Among the many benefits, Redis Streams enables you to do the following:
- Collect large volumes of data arriving in high velocity (the only bottleneck is your network I/O);
- Create a data channel between many producers and many consumers;
- Effectively manage your consumption of data even when producers and consumers don’t operate at the same rate;
- Persist data when your consumers are offline or disconnected;
- Communicate between producers and consumers asynchronously;
- Scale your number of consumers;
- Implement transaction-like data safety when consumers fail in the midst of consuming data; and
- Use your main memory efficiently.
The best part of Redis Streams is that it’s built into Redis, so there are no extra steps required to deploy or manage Redis Streams. In this article, I’ll walk you through the basics of using Redis Streams. We’ll look at how we can add data to a stream, and how we can read that data (all at once, asynchronously, as it arrives, etc.) to satisfy different consumer use cases.
