Redis geo.lua example using Go

I stumbled upon geo.lua which seemed to be an interesting library

It’s described as – “… a Lua library containing miscellaneous geospatial helper routines for use with Redis

Here is an example of using it with the Go Redis client (go-redis). This is what it does in a nutshell

To run, refer README – its super simple

Cheers!

Advertisements
Posted in go, nosql, redis | Tagged , , , , , , | Leave a comment

certificate error with Go HTTP client in alpine Docker

Using the Go HTTP client from a alpine docker image will result in this error – x509: failed to load system roots and no roots provided

Solution: alpine is a minimal image, hence CA certificates are required. You add that in the Dockerfile as per below example

FROM alpine 
RUN apk add --no-cache ca-certificates 
COPY my-app .
CMD ["./my-app"]

Got this hint from the golang alpine Dockerfile
Cheers!
Posted in go | Tagged , , , | Leave a comment

etcd Watch example using Go client

etcd is a distributed, highly available key-value store which serves as the persistent back end for Kubernetes

etcd

Here is an example to demonstrate its Watch feature. For details, check out the README (and the sample code) on Github

It uses

Summary

Cheers!

Posted in Distributed systems | Tagged , , , | Leave a comment

NATS on Kubernetes example

In a previous blog, you saw an example of NATS along with producer and consumer – based on Docker Compose. This post uses the same app, but runs on Kubernetes

nats-k8s.jpg

For how to run, follow the README on Github

High level overview

  • uses the NATS Kubernetes operator
    • it seeds a CRD …
    • … using which we can spawn a NATS cluster on Kubernetes (with a single command!)
    • A Service object is created automatically – uses ClusterIP by default
nats-cluster-service.JPG

NATS Kubernetes Service object

  • The environment variable corresponding to the Serviceis used in the application logic to locate NATS server. Note its usage here

Cheers!

nats-logo.png

Posted in Distributed systems, Kubernetes, nats | Tagged , , , , , , , , | Leave a comment

Redis tech tip #1

In order to delete members from your Redis Geo-spatial collection, you can use ZREM (underneath the hood, the data is stored in a sorted set)

Add a few (Indian) cities

GEOADD IndianCities 28.6108127 77.2060241 NewDelhi

​GEOADD IndianCities 12.9715987 77.5945627 Bangalore

GEOADD IndianCities 13.0826802 80.2707184 Chennai

The weather in Chennai humid – not my kind.. let’s remove it

ZREM IndianCities Chennai

You can also delete the entire collection using DEL IndianCities (usual stuff…)

Cheers!

Posted in nosql, redis | Tagged , , , | Leave a comment

NATS & Kafka: random notes

This is not NATS vs Kafka by any means – just jotting down some info. NATS recently joined CNCF (which host projects like Kubernetes, Prometheus etc. – look at the dominance of Golang here!) and that’s when it caught my attention

I have been taken aback by its simplicity and performance (yet to test drive it fully – but the numbers are out there)

  • Kafka runs on JVM (Scala to be specific), NATS is written in Go
  • Protocol – Kafka is binary over TCP as opposed to NATS being simple text (also over TCP)
  • Messaging patterns – Both support pub-sub and queues, but NATS supports request-reply as well (sync and async)
  • For (horizontally) scalable processing,
    • NATS has a concept of queue (with a unique name of course) and all the subscribers hooked on same queue end up being a part of the same queue group. Only one of the (potentially multiple) subscribers gets the message. Multiple such queue groups would also receive the same set of messages. This makes it a hybrid pub-sub (one-to-many) and queue (point-to-point)
    • Same thing is supported in Kafka via consumer groups​ which can pull data from one or more topics
  • Stream processing – NATS does not support stream processing as a first class feature like Kafka does with Kafka Streams
  • Kafka clients use a poll-based technique in order to extract messages as opposed to NATS where the server itself routes messages to clients (maintains an interest-graph internally)
  • NATS can act pretty sensitive in the sense that it has the ability to cut off consumers who are not keeping pace with the rate of production as well as clients who don’t respond to heartbeat requests.
    • The consumer liveness check is executed by Kafka as well. From what I recall, this is done/initiated from the client itself & there are complex situations that can arise due to this (e.g. when you’re in a message processing loop and don’t poll ). There are a bunch of configuration parameter/knobs to tune this behavior (on client side)
  • Delivery semantic – NATS supports at-most once (and at-least-once with NATS streaming) as opposed to Kafka which also supports exactly-once (tough!)
  • NATS doesn’t seem to have a notion of partitioning/sharding messages like Kafka does
  • No external dependency in case of NATS. Kafka requires Zookeeper

… work in progress…

 

Posted in Distributed systems, Kafka, messaging | Tagged , , , | Leave a comment

NATS – quick start with Docker

Here is a hello-world example (on Github) for getting started with natsio – a distributed messaging server written in Go

It consists of the following components which are managed via Docker Compose

Cheers!

Posted in Distributed systems, messaging | Tagged , , , , | 1 Comment

Session @ Oracle Code San Francisco 2017

Here are the details for my talk @ Oracle Code (a track at JavaOne 2017) – Streaming Solutions for Real time problems (Stream Processing solutions using Apache Kafka, Kafka Streams and Redis)

Code (Github) – https://github.com/abhirockzz/accs-ehcs-stream-processing

Video

Slides

Cheers!

Posted in Distributed systems, Kafka | Tagged , , , , , , , , , | Leave a comment

Debezium test drive

Debezium is an open source, distributed change data capture system built on top of Apache Kafka. I tried it out and the project is available on Github

Setup

Details are in the README. It uses the Debezium tutorial as a background but the setup is very simple since it uses Docker Compose – one command (docker-compose up --build is all it takes to get going !

Components

  • MySQL – the database
  • Debezium – the CDC platform which tails MySQL binlog and pushes its logs to Kafka using its MySQL Kafka Connect connector
  • Consumer – its a Java EE application which consumes the (DB change) events from Kafka (topic), derserializes and parses it out and logs it

results.jpg

Cheers!

 

Posted in Distributed systems, Kafka | Tagged , , , , , , | 1 Comment

Kafka & Websocket

For those who are interested in an example of Kafka working with the (Java EE) Websocket API, please check out this blog . There is an associated Github project as well

Cheers!

Posted in Distributed systems, Kafka | Tagged , , | Leave a comment