Redis Streams is a new Redis data structure that allows you to manage data channels between producers and consumers. In two previous articles here, I covered the basics and benefits of Redis Streams and how to use Redis Streams consumer groups to scale out data stream consumers. In this article, I will demonstrate how to develop a data stream processing application using Redis Streams.
In order to bring Redis Streams to life, let’s build a real-world solution for identifying the top influencers on Twitter. This application, which I’ll call TopSocialForce, will gather social messages from Twitter, store them in Redis Streams, and analyze them in real time. TopSocialForce will discover the top social forces by tagging any Twitter handle with more than 10,000 followers as an “influencer” and maintaining a running catalog.
TopSocialForce will collect Twitter stream data through a third-party service called PubNub. (GNIP is another option.) The tweets follow the JSON format as described by Twitter, so every tweet contains the Twitter handle and its follower count, which we’ll use for influencer classification.
Redis Labs
Figure 1. How TopSocialForce works.
