Big Data Hadoop & Spark

Guide to Install Apache Kafka

What is the use of Kafka?

Kafka was originally developed by Linkedin to solve its problem of real-time analytics as at that time there were not many Big Data solutions available which could handle the large flow of data originating from its existing infrastructure and website.

There were good batch processing framework available but that could not handle its real-time data and as a result, Kafka was developed to solve the problem of real-time data ingestion.

Kafka Architecture and Components:

Let’s see the architecture of Kafka and try to understand its various components.
Topic : Kafka messages are classified into multiple categories which may be called as topics.
Producer : Producer is any Kafka client or application which publishes or push messages to topics.
Consumer : It is an application which pulls messages from Topics.
Broker : Since Kafka is a distributed framework so each node can be termed as Kafka broker.
Since we have understood the underlying importance of Kafka,we can start our discussion on installation steps.

Installing Kafka:

Step-by-Step Installation of Apache Kafka in Single Node Hadoop Cluster:

Step 1: Check if Hadoop and Zookeeper are installed and running.

Step 2: Download the Kafka tar file from below link and extract the file.


Step 3: To get recognized by the system, open bashrc and add the home and path for the Kafka.

Step 4: Execute the source command for the changes made in bashrc file to get affected. Also, make a directory named logs where Kafka will be writing all its logs process.

You can refer to the below screenshot and find a file named

Step 5: Edit the property in file, inside conf directory in the extracted folder of Kafka and save the file.

Step 6: In the terminal, run the command for starting Kafka, using the below syntax:

Nohup <path to file> <path to>

Step 7: Use the jps command to check if the Kafka daemon has started.

Hope this post has been helpful in understanding the steps to install Kafka. In case of any queries, do drop us your question in the comments section below and we will get back to you at the earliest.

Stay tuned for more updates on Big Data and other technologies.
Keep visiting our site for more updates on Hadoop and other technologies.



An alumnus of the NIE-Institute Of Technology, Mysore, Prateek is an ardent Data Science enthusiast. He has been working at Acadgild as a Data Engineer for the past 3 years. He is a Subject-matter expert in the field of Big Data, Hadoop ecosystem, and Spark.

One Comment

  1. Pingback: Kafka installation Guide by | Hadoop Reference Links

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles