Free Shipping

Secure Payment

easy returns

24/7 support

Troubleshooting Hive Metadata

 July 20  | 0 Comments

All the metadata for Hive tables and partitions are accessed through the Hive Metastore. Metadata is persisted using JPOX ORM solution (Datanucleus) so any database that is supported by it can be used by Hive. Most of the commercial relational databases and many open source databases are supported.

By default, Hive uses derby as metastore storage engine but, in production, we always use an external database like Mysql. Although, we can use any database server, but Mysql is the most commonly used server integrated with Hive.

Let’s look at some common metastore errors.

Case 1:  Mysql Jdbc driver is missing. The property javax.jdo.option.ConnectionDriverName” talks about the driver to connect to Mysql. The Mysql driver “mysql-connector-java-5.1.39-bin.jar”, must be present under the directory “hive/lib”.

Solution: To resolve the above error, we need to copy “mysql-connector-java-5.1.39-bin.jar” to “hive/lib”

Case 2: The Hive prompt just hangs and waits for the timeout before giving any error.

Reason/Solution: Not able to reach the mysql server due to Firewall. Test it by using Telnet as shown below.

Notice that the port is wrong as correct port is “3306”

Once these corrections are made, we can see the Hive prompt.

Case 4: Mysql has incorrect grants for the hive user and metastore table.

Solution: Make sure the user “Hadoop” or whichever is running hive has permission to modify the HiveDB.

Case 5: Hive Error “Too many counters” Error”

Solution: This is when there are too many counters emitted by Hive, which is set to 120 by default. We can increase this by changing “mapreduce.job.counters.max” in mapred-site.xml .

Case 6: Sometimes when we set our classpath in “.bashrc” it might conflict with the other jars.