Today, there are many data sources—such at IoT devices, user interaction events from mobile applications, financial service transactions, and health monitoring systems—that broadcast critical information in real time. Developers working with these data sources need to think about the architecture to capture real time streaming data at varying scales and complexities.
It used to be that processing real time information at significant scale was hard to implement. Hardware architectures needed to be engineered for low latency while software needed more advanced programming techniques that combined receiving data, processing it, and shipping it efficiently.
The paradigm shift in data processing
I recently attended the Strata Data Conference and discovered a paradigm shift: There are multiple frameworks (both open source and commercial) that let developers handle data streaming or real time data-processing payloads. There are also commercial tools that simplify the programming, scaling, monitoring, and data management of data streams.
The world isn’t batch anymore, and the tools to process data streams is a lot more accessible today than just two or three years ago.
