All CategoriesBig Data Hadoop & Spark

Building Hadoop Application Using Gradle

In our previous blogs, we have seen how to build Hadoop applications using Maven and SBT. Now in this blog, we will show a demo on building Hadoop application using Gradle. But before we begin, let us see what is Gradle.

Gradle is an open source build automation system that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language (DSL) instead of the XML form used by Apache Maven of declaring the project configuration.

100% Free Course On Big Data Essentials

Subscribe to our blog and get access to this course ABSOLUTELY FREE.

Building Hadoop Application Using Gradle

Installing Gradle in Linux

For Debian based systems like Ubuntu, use the below command to install Gradle.

Sudo apt-get install Gradle

Installing Gradle in Eclipse

Gradle can also be integrated with Eclipse. Let us now see the procedure to integrate Gradle with Eclipse.

If you are using Eclipse Neon, then click on Help–> Install New Software–> Here give the below link.

http://download.eclipse.org/buildship/updates/e46/milestones/2.x

Now click on Next as shown in the below screenshot.

In the Next page select the plugin and Click on Next again. In the Next page select the option

I accept the terms of license agreement option and click Finish.

Now all the files that are required for Gradle will be downloaded and installed automatically. After successful installation, your eclipse will ask for a restart. Please restart to affect the changes.

Now you can create Gradle projects directly from eclipse by clicking on File–>New–>Other–>Gradle project

Now we will create a normal project outside eclipse and we will import that project into eclipse.

First, you need to create one Project folder. Here, inside src/main/java directory you need to write your code otherwise, Gradle could not find your main class file.

Now we have created one file called Hadoop_app_gradle and inside src/main/java we have pasted the code of Hadoop word count program.

Inside the root directory i.e., Hadoop_app_gradle you need to create one more file called build.gradle. Here, you need to provide all the specifications and library dependencies of your project. Here are our specifications of Gradle build.

apply plugin: 'java' //necessary for any java project
// Set up group and version info for the artifact
group = "com.acadgild.hadoop_wc_gradle"
version = "1.0"
repositories {
mavenCentral() //repositories for downloading your dependencies
}
//dependencies for hadoop application
dependencies {
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version: '2.7.1'
}
dependencies {
compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '2.7.1'
}
jar {
from configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
}
ext.hadoopVersion = "2.7.1"
//eclipse plugin
apply plugin: "eclipse"
eclipse {
// Ensure Eclipse build output appears in build directory
classpath {
defaultOutputDir = file("${buildDir}/eclipse-classes")
}
}

Above are the specifications of our Gradle file. Now we will build the Gradle project using the command Gradle build. This command will compile your code and will build a jar file automatically.

After successful compilation and building of the jar, you can see a success message and in the project_root_folder/build/libs directory, you can see the jar file.

To make this project eclipse ready, you need to use the command sbt eclipse to build the project and classpath that are required for eclipse. You can see the same in the below screen shot.

Now your project is ready to import into eclipse. For importing the project into eclipse, follow the below procedure.

Open Eclipse –> Click on File–> Click on Import–> Click on General –> Click on Existing Projects into Workspace — >Select root directory–>Click on Browse–>Select the project file which you have created–>Click on ok

In the next screen, select all the projects present and click on Finish.

Hadoop

Now you can browse this project as a normal eclipse project.

Let us run the jar file which we have built using Gradle with the Hadoop jar command. Here is the command we have used to run the jar file.

hadoop jar /home/kiran/Hadoop_app_gradle/build/libs/Hadoop_app_gradle-1.0.jar WordCount /word_count_input /wc_output_gradle

The jar file successfully ran. Let us see the output in the output folder.

In the above screenshot, you can see the output of the word count program. So we have successfully built the Hadoop application using Gradle.

We hope this blog helped you in understand building Hadoop application using Gradle. Keep visiting our site www.acadgild.com for more updates on Big Data and other technologies.

Hadoop

Tags

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Articles

Close
Close