In this article, we will go through the ZooKeeper Fundamentals.
Let’s start with why you want to use ZooKeeper.
ZooKeeper is a one of the building block for distributed systems. While designing a distributed system, there is typically a need for designing and developing some coordination services.
- Name service— It is a service that maps a name to some information associated with that name. A Bank statement is a name service that maps the name of a person to his/her account number. In the same way, a DNS service is a name service that maps a domain name to an IP address. In any distributed system, you may want to keep a track of which servers or services are up and running and want to look up their status by name. ZooKeeper gives a simple interface to do the same. A name service can also act as a group membership service by means of which you can obtain information given to the group associated with the entity whose name is being looked up.
- Locking— The serialized access to a shared resource in any distributed system, you may need to implement distributed mutexes(a program object that allows multiple program threads to share the same resource, such as file access). ZooKeeper provides an easy way for you to implement them.
- Synchronization—Distributed mutexes with hand in hand synchronizing access is the need for shared resources. Whether implementing a producer-consumer queue or a barrier, ZooKeeper is the solution which provides for a simple interface to implement that.
- Configuration management— Using ZooKeeper to centrally store and manage the configuration of your distributed system can be done. This means any new nodes joining can pick up the up-to-date centralized configuration from ZooKeeper as soon as they join the system. This also allows you to centrally modify the state of your distributed system by changing the centralized configuration through one of the ZooKeeper clients.
- Leader election— The distributed system may have to deal with problem of nodes going down, also you may want to implement an automatic fail-over strategy. ZooKeeper provides off-the-shelf support for doing so through leader election.
Although it’s possible to design and implement all of these services from scratch, it comes with extra work and difficulty to debug any problems, race conditions, or deadlocks. There was need that people should not have to write their own leader election services or name services from scratch every time they need it. This led to the development and open sourcing of Apache ZooKeeper, a reliable, scalable, and high-performance coordination service for distributed systems.
ZooKeeper being a coordination service for distributed systems, also is a distributed application on its own. ZooKeeper follows a simple model of client-server where clients are nodes that make use of the service, and servers are nodes that provide the service.
At any given time, one ZooKeeper client is connected to one ZooKeeper server. Each ZooKeeper server can handle a large number of client connections at the same time. Each client periodically sends pings(also know as heartbeat ) to the ZooKeeper server it is connected to let it know that it is alive and connected. The ZooKeeper server revert with an acknowledgment of the ping, indicating the server is alive as well. When the client does not receive an acknowledgment from the server within the specified time, the client connects to another server in the cluster, and the client session is transparently transferred over to the new ZooKeeper server.
Figure below shows the client-server architecture of ZooKeeper.
Figure 1. Client-server architecture of ZooKeeper
Applications of ZooKeeper
ZooKeeper has variety of practical applications. We will list some of those applications here. Most of these uses have been taken from the Apache ZooKeeper wiki.
- ZooKeeper helps for automatic fail-over of Hadoop HDFS Namenode and for the high availability of YARN ResourceManager.
- Apache HBase is a distributed database built on Hadoop, uses ZooKeeper for master election, lease management and other communication between region servers.
- Apache Accumulo, another sorted distributed key/value store is built on top of Apache ZooKeeper (and Apache Hadoop).
- For leader election and centralized configuration Apache Solr uses ZooKeeper.
- Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications. It uses ZooKeeper for fault-tolerant replicated master.
- Neo4j is a distributed graph database that uses ZooKeeper for write master selection and read slave coordination.
- Cloudera Search integrates search functionality (via Apache Solr) with Apache Hadoop using ZooKeeper for centralized configuration management.
Apache ZooKeeper is a high-performance coordination server for distributed applications. It gives a simple interface to expose common services — such as naming and configuration management, synchronization, and group services,relieving the user from the need to program from scratch. Zookeeper comes with off-the-shelf support for implementing consensus, leader election,group management, and presence protocols.