Kafka was originally developed by Linkedin to solve its problem of real-time analytics as at that time there were not many Big Data solutions available which could handle the large flow of data originating from its existing infrastructure and website.
There were good batch processing framework available but that could not handle its real-time data and as a result, Kafka was developed to solve the problem of real-time data ingestion.
Let’s see the architecture of Kafka and try to understand its various components.
Topic : Kafka messages are classified into multiple categories which may be called as topics.
Producer : Producer is any Kafka client or application which publishes or push messages to topics.
Consumer : It is an application which pulls messages from Topics.
Broker : Since Kafka is a distributed framework so each node can be termed as Kafka broker.
Since we have understood the underlying importance of Kafka,we can start our discussion on installation steps.
Step-by-Step Installation of Apache Kafka in Single Node Hadoop Cluster:
Step 1: Check if Hadoop and Zookeeper are installed and running.
Step 2: Download the Kafka tar file from below link and extract the file.
Step 3: To get recognized by the system, open bashrc and add the home and path for the Kafka.
Step 4: Execute the source command for the changes made in bashrc file to get affected. Also, make a directory named logs where Kafka will be writing all its logs process.
You can refer to the below screenshot and find a file named server.properties.
Step 5: Edit the property in server.property file, inside conf directory in the extracted folder of Kafka and save the file.
Step 6: In the terminal, run the command for starting Kafka, using the below syntax:
Nohup <path to kafka-server-start.sh file> <path to server.properties>
Step 7: Use the jps command to check if the Kafka daemon has started.
Hope this post has been helpful in understanding the steps to install Kafka. In case of any queries, do drop us your question in the comments section below and we will get back to you at the earliest.