Kafka, End to End on Daniel - Minimalist | Tinkerer | Believer

Making Your Kafka Cluster Production-Ready with Strimzi: Best Practices

daniel.mudanjara@gmail.com (Dan) — Tue, 22 Oct 2024 17:45:52 +0530

Apache Kafka is widely used for building distributed, real-time data pipelines. However, running Kafka in a production environment requires a solid setup to ensure performance, reliability, and scalability. Strimzi simplifies deploying and managing Kafka clusters on Kubernetes or OpenShift, but to be truly production-ready, several factors need to be considered. In this article, we will cover essential best practices and optimizations to make your Kafka cluster ready for production, based on insights from the Strimzi Production Ready series and additional knowledge.

Kafka Explained: The Post Office of Modern Data Streaming

daniel.mudanjara@gmail.com (Dan) — Tue, 24 Sep 2024 13:47:01 +0530

Joshua had been waiting anxiously for weeks. As an aspiring clerk, he knew that the government official’s appointment order could arrive any day. This appointment was crucial—it would determine his future and career. Every morning, Joshua checked his local post office, hoping that today would be the day the letter arrived, confirming his new position.

In many ways, Joshua’s experience mirrors how Kafka, a powerful distributed streaming platform, manages and delivers data. Just like how Joshua relies on the postal system to receive his appointment letter, modern data systems depend on Kafka to ensure that critical information reaches its intended recipients quickly and reliably.

Installing Kafka and Kafka Connect on Seperate Servers

daniel.mudanjara@gmail.com (Dan) — Thu, 12 Sep 2024 13:46:20 +0530

Installing Kafka and Kafka Connect on separate servers allows for better resource management, especially in production environments where Kafka brokers and Connectors may need dedicated hardware. This guide will walk you through the steps to set up Kafka and Kafka Connect on separate Linux servers, using Ubuntu 24.04.

Prerequisites

Two Linux machines (Ubuntu 24.04)
Inbound ports opened on both servers:
- Kafka broker ports: 9092, 9093
- Kafka Connect REST port: 8083
Java 11 (OpenJDK) installed on both servers

Step 1: Set Up Kafka on the First Server

Set the hostname for the Kafka server: