Kafka Streams based application

A Kafka Streams sample application is available on Github… This is a microservice (packaged in form of an Uber JAR) which uses the Kafka Streams Processor (low level) API to calculate the Cumulative Moving Average of the CPU metrics of each machine in a distributed manner

  • A producer application continuously emits CPU usage metrics into a Kafka topic (cpu-metrics-topic) and consumer application (instances) do the computation
  • Consumers can be horizontally scaled – the processing work is distributed amongst many nodes and the process is elastic and flexible thanks to Kafka Streams (and the fact that it leverages Kafka for fault tolerance etc.)
  • Each instance has its own (local) state for the calculated average. A custom REST API has been (using Jersey JAX-RS implementation) to tap into this state and provide a unified view of the entire system (moving averages of CPU usage of all machines)

….. more in the project README

Cheers!

Advertisements

About Abhishek

Loves Java EE, distributed KV stores and messaging systems. Frequently blogs at abhirockzz.wordpress.com as well as simplydistributed.wordpress.com. Oh, I have also authored a few (mini) books, articles, Refcards etc. :-)
This entry was posted in Distributed systems, Kafka and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s