Designing a production service environment around Apache Kafka that delivers low latency and zero-data loss at scale is non-trivial. Indeed, it’s the holy grail of messaging systems. In this blog post, I’ll outline some of the fundamental service design considerations that you’ll need to take into account in order to get your service architecture to measure up.
Let’s start with the basics.
Foundation services for Apache Kafka solutions
Foundation services are things like Network Time Protocol (NTP) services, Domain Name Services (DNS) as well as network routing, firewalling and zoning. You’ll need to get this right before even considering moving onto the application deployment itself. Now, I know what you’re saying – “I’m deploying Kafka on the cloud, this stuff is all taken care of, nothing to see here”. Well, not quite.
Even on the cloud, you’ll need to ensure that your VPC environment has network routing configured correctly, that firewall ingress…
