How To Fix: Hadoop command not found error

Posted by Marta on October 6, 2021 Viewed 5983 times

In this article you will learn how to fix the Hadoop command not found error, so you can fix your Hadoop installation and start working on challenging big data projects.

Hadoop is an open source software optimised for reliable and scalable distributed computing written in Java, and therefore to use Hadoop you will need java installed in your machine, and the JAVA_HOME environment variable. If you haven’t installed Java or download Hadoop, you can find some help here.

Let’s dive into your problem. You just downloaded Hadoop and unzip the file. Then you check if you installed Hadoop correctly by running the following command:

>>hadoop version

And instead of a version number, you got the following message:

-bash: hadoop: command not found

Table of Contents

What’s the meaning of Hadoop command not found error?

Let’s start with some background about how commands work. When you run a command in your terminal, you are actually executing a binary file. For example, when you execute command ls, or cp, this command corresponds with binary files locate in the /bin folder.

How can your machine find those command files? Using the PATH environment variable. Every time you type a command in your terminal, the kernel will look for the corresponding binary file in all folders included in the PATH variable.

Therefore if you like to use the Hadoop command, after downloading it , you will need to update the PATH environment variable to include the Hadoop binaries, so the terminal knows where to find Hadoop.

How to fix it?

To fix this problem you will need to create the HADOOP_HOME environment variable, which will point to the directory where you uncompressed the Hadoop file. You should also include this folder in your PATH variable. See below the command you should execute:

>> export HADOOP_HOME=~/sw/hadoop-x.y.z
>> export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

If you run these commands from your terminal, Hadoop will be available through the terminal. Unfortunately this change will last only during the terminal session.

As a result, next time you restart the laptop, or close the terminal and reopen it, you will need to run these commands again. How can you avoid this? Adding the above commands to the .bash_profile file, located in your user home folder. The content of this file get executed every time you open the terminal.

Once you update the .bash_profile file, make sure you close and reopen the terminal so your change get applied. Or run the following command to reload the .bash_profile file:

>> source ~/.bash_profile

Once you have corrected the PATH environment variable, and run the command: hadoop version from your terminal, you should use something similar to the message below:

Hadoop 2.10.0
Subversion ssh://git.corp.linkedin.com:29418/hadoop/hadoop.git -r e2f1f118e465e787d8567dfa6e2f3b72a0eb9194
Compiled by jhung on 2019-10-22T19:10Z
Compiled with protoc 2.5.0
From source with checksum 7b2d8877c5ce8c9a2cca5c7e81aa4026
This command was run using /Users/martarey/dev/hadoop-2.10.0/share/hadoop/common/hadoop-common-2.10.0.jar

How to display PATH in terminal

Here is the command you need to run from your terminal to visualise the value of the PATH environment variable:

>> echo $PATH

And the output should be similar to the following message, where the different folder paths are separated by a colon : symbol.

/Library/Frameworks/Python.framework/Versions/3.7/bin:/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin

Conclusion

To summarise, we have seen what the error command not found means, and how you can repair your Hadoop installation and fix this error Hadoop command not found.

In case you would like to learn more about Hadoop and the Hadoop ecosystem and how to run it locally, please feel free to check this tutorial. You will find useful information about the several components that make up Hadoop like HDFS and MapReduce which makes the platform reliable and scalable.