There are two ways to solve any problem: Accurately or approximately. Accurate data structures has its disadvantages – too much memory usage and unscalable for real-time nature of data. In this talk I explained how to take advantage of the newly release Redis 4.0 with pluggable modules to build a data pipeline which uses probabilistic data structures to get real-time insights.
There are different insights and metrics that could be obtained from log events data. Processing the data in real-time and getting accurate results are possible in theory. In practice, not so easy.