Big Data Hadoop & Spark

Configuring Rack Awareness on Hadoop Using Centos

This blog gives you information about Rack Awareness in Apache Hadoop. HDFS block placement uses rack awareness for fault tolerance by placing one block replica on a different rack. This ensures data availability during network switch failure or a partition within the cluster. Rack awareness is very helpful in making an appropriate replication factor. Configuring rack awareness provides the information to Hadoop as to which Datanode is on which rack.
Note: For rack awareness configuration, all  the changes/modifications have to be made on NameNode (Masternode) only.

You can configure rack awareness in 3 steps:

1. Create a topology data file anywhere in Master node (NameNode)

vi topology.data

Next, mention your slave nodes ( DataNodes) relative to their rack into topology.data-

100% Free Course On Big Data Essentials

Subscribe to our blog and get access to this course ABSOLUTELY FREE.

192.168.10.101 /rack1
192.168.10.102 /rack2
192.168.10.103 /rack2

topology.data

2. Create a topology.sh  script file (Also called as rack awareness script file)

vi topology.sh
HADOOP_CONF=/home/hadoop/address of topology.data
while [ $# -gt 0 ] ; do
nodeArg=$1
exec< ${HADOOP_CONF}/topology.data
result=""
while read line ; do
ar=( $line )
if [ "${ar[0]}" = "$nodeArg" ] ; then
result="${ar[1]}"
fi
done
shift
if [ -z "$result" ] ; then
echo -n "/default/rack "
else
echo -n "$result "
fi
done

topology.sh

3. Add this property into core-site.xml of Master node only

<property>
<name>topology.script.file.name</name>
<value>/home/hadoop/topology.sh</value>
</property>

core-site.xml

Next, start your cluster.

start-dfs.sh

start-dfs.sh

Check the Hadoop admin report to see if the cluster is aware of the rack.

hadoop dfsadmin -report

hadoop dfsadmin -report
hadoop dfsadmin -report
hadoop dfsadmin -report
hadoop dfsadmin -report

This cluster is now rack aware !

Hope this post was helpful in understanding about the Commissioning and Decommissioning of the datanodes in Hadoop.
In case of any queries, feel free to write to us at [email protected] or comment below, and we will get back to you at the earliest. Keep visiting our website Acadgild for more updates on Big Data and other technologies. Click here to learn Big Data Hadoop Development.

Hadoop

Tags

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles

Close
Close