Accessing Kafka on host machine from minikube pods

There are times when you want to access processes running on your host machine e.g. databases etc. from your minikube Kubernetes cluster in Virtual Box (or any other supported provider)

For details, continue reading this blog on Medium

Cheers!

Posted in Distributed systems, Kafka, Kubernetes | Tagged , , | Leave a comment

Kafka Streams Interactive Queries

This blog post explores the Interactive Queries feature in Kafka Streams with help of a practical example. It covers the DSL API and how the state store information was exposed via a REST service

For details, check out the blog post on Medium and the source code is available on GitHub

Everything is setup using Docker including Kafka, Zookeeper, the stream processing services as well as the producer app

Here is a bird’s eye view of the overall solution looks

Cheers!

Posted in Distributed systems | Leave a comment

Kafka Go client: No Producer error during connection ?

Although its rare, but there are times when you actually want to see errors in your code – more importantly, at the right time !

Kafka Go Producer behaviour

You need to be mindful of this while using the Kafka Go client producer. For e.g. if you were to supply an incorrect value for the Kafka broker …

no error

… assuming you don’t have a Kafka broker at foo:9092, you would expect that the above code will respond with producer creation failed along with the specific error details. Instead, the flow carries on and ends by printing done. This is because the error returned by Produce is only in case message does not get enqueued to the internal librdkafka queue

You should…

Hook up to the delivery producer reports channel (using Producer.Events()) in order to catch this error – better late than never right !

Now, the error is rightfully returned – producer error foo:9092/bootstrap: Failed to resolve 'foo:9092': nodename nor servname provided, or not known (after 1543569582778ms in state INIT)

You can also provide your own channel to the Produce method to receive delivery events. Only the error is handled and the other possible event i.e. *kafka.Message is ignored under the default case umbrella

Cheers!

sdfsdfsdfsdfsdf

Posted in Distributed systems, go, Kafka, messaging | Tagged , , | Leave a comment

Kafka Go client quick start

Here is a Docker based example for Kafka using Go client

Overview

  • Producer
    • uses the traditional API for pushing messages to Kafka (channel is another option)
    • receives notifications on the (default) Events (custom channel is another option or you can choose not to get notified)
  • Consumer uses (Events) channel – Poll()ing is another alternative
  • Docker Compose to bootstrap the sample app (producer and consumer) as a single unit
  • BYOK (bring your own Kafka) for trying this out

To run

Check out the README – it’s super easy to get going, thanks to Docker Compose!

References

Cheers!

 

Posted in Distributed systems, Kafka, messaging | Leave a comment

Redis 5 – bootstrapping a Redis Cluster with Docker

With Redis 5 (in RC state at time of writing), cluster creation utility is now available as part of redis-cli which is easier as compared to the (previous) ruby way of doing it (using redis-trib)

check out the release notes for more info

Here is the command list – docker run -i --rm redis:5.0-rc redis-cli --cluster help

cluster-commands

redis-cli –cluster help

It’s super simple to create a 6-node Redis Cluster (3 masters and corresponding slaves) on Docker

Once executed, you should see a similar output

To check cluster, you can run

  • ​​​​​docker run -i --rm --net redis_cluster_net redis:5.0-rc redis-cli --cluster check localhost:6380
  • docker run -i --rm --net redis_cluster_net redis:5.0-rc redis-cli --cluster info localhost:6380

Cheers!

Posted in Distributed systems, nosql, redis | Tagged , , , , | 6 Comments

(eBook) Practical Redis: first few chapters released!

I am happy to announce that the first few chapters of Practical Redis are now available

About Practical Redis

cover5

 

It is a hands-on, code-driven guide to Redis where each chapter is based on an application (simple to medium complexity) which demonstrates the usage of Redis and its capabilities (data structures, commands etc.).

The applications in the book are based on Java and Golang. The Java based Redis client libraries which have been used in this book include JedisRedisson and Lettuce while go-redis is used as the client library for Golang. Docker and Docker Compose are used to deploy the application stack (along with Redis)

 

Here is a quick outline of the book contents

  • Hello Redis – quick tour of Redis capabilities including its versatile data structures
  • Redis: the basic data structures – intro to core Redis data structures with a Java based news sharing app using the Jedisclient
  • Extending Redis with Redis Modules – learn about the basics of Redis Modules and make use of ReBloom in a Go based recommendation service
  • Tweet analysis service – keep track of relevant tweets using a Java and Go based application which ingests tweets, consumes/processes them and produces queryable info in real time using reliable queues (Redis LISTs), SETs and HASHes

Coming soon …

The below chapters are work in progress on and will be made available in subsequent releases of this book

  • Pipelines and Transactions in Redis
  • Scalable chat application – Use Redis PubSub and Websocket to create a chat service
  • Stream processing with Redis – watch Redis Streams in action (new data structure in Redis 5.0)
  • Redis based Tracking service – thanks to combination of the Geo data type and Lua scripting capability in Redis
  • Redis for distributed computing – explore interesting use cases made possible by Redisson
  • Data partitioning in Redis – practical examples highlighting data sharding strategies in Redis
  • Redis high availability

Go check it out & stay tuned for more updates

Cheers!

Posted in Distributed systems, go, nosql, redis | Tagged , , , , , , , , | Leave a comment

Redis geo.lua example using Go

I stumbled upon geo.lua which seemed to be an interesting library

It’s described as – “… a Lua library containing miscellaneous geospatial helper routines for use with Redis

Here is an example of using it with the Go Redis client (go-redis). This is what it does in a nutshell

To run, refer README – its super simple

Cheers!

Posted in go, nosql, redis | Tagged , , , , , , | Leave a comment

certificate error with Go HTTP client in alpine Docker

Using the Go HTTP client from a alpine docker image will result in this error – x509: failed to load system roots and no roots provided

Solution: alpine is a minimal image, hence CA certificates are required. You add that in the Dockerfile as per below example

FROM alpine 
RUN apk add --no-cache ca-certificates 
COPY my-app .
CMD ["./my-app"]

Got this hint from the golang alpine Dockerfile
Cheers!
Posted in go | Tagged , , , | Leave a comment

etcd Watch example using Go client

etcd is a distributed, highly available key-value store which serves as the persistent back end for Kubernetes

etcd

Here is an example to demonstrate its Watch feature. For details, check out the README (and the sample code) on Github

It uses

Summary

Cheers!

Posted in Distributed systems | Tagged , , , | Leave a comment

NATS on Kubernetes example

In a previous blog, you saw an example of NATS along with producer and consumer – based on Docker Compose. This post uses the same app, but runs on Kubernetes

nats-k8s.jpg

For how to run, follow the README on Github

High level overview

  • uses the NATS Kubernetes operator
    • it seeds a CRD …
    • … using which we can spawn a NATS cluster on Kubernetes (with a single command!)
    • A Service object is created automatically – uses ClusterIP by default

nats-cluster-service.JPG

NATS Kubernetes Service object

  • The environment variable corresponding to the Serviceis used in the application logic to locate NATS server. Note its usage here

Cheers!

nats-logo.png

Posted in Distributed systems, Kubernetes, nats | Tagged , , , , , , , , | Leave a comment