Saturday, 9 April 2022

Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

Error While running Hadoop on Window 

 Exception in thread "main" java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:737)

at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:272)

at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:288)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:777)

at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:522)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:562)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:561)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)

at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:705)

at com.nokia.cemod.ice.rest.controller.HDFSDemo.createDir(HDFSDemo.java:42)

at com.nokia.cemod.ice.rest.controller.HDFSDemo.main(HDFSDemo.java:24)

Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:549)

Solution

Hadoop requires native libraries on Windows to work properly -that includes to access the file:// filesystem, where Hadoop uses some Windows APIs to implement posix-like file access permissions.

This is implemented in HADOOP.DLL and WINUTILS.EXE.

In particular, %HADOOP_HOME%\BIN\WINUTILS.EXE must be locatable.

If it is not, Hadoop or an application built on top of Hadoop will fail.

How to fix a missing WINUTILS.EXE

You can fix this problem in two ways

  1. Install a full native windows Hadoop version. The ASF does not currently (September 2015) release such a version; releases are available externally.
  2. Or: get the WINUTILS.EXE binary from a Hadoop redistribution. There is a repository of this for some Hadoop versions on github.

Then

  1. Set the environment variable %HADOOP_HOME% to point to the directory above the BIN dir containing WINUTILS.EXE.
  2. Or: run the Java process with the system property hadoop.home.dir set to the home directory.
  3. In Eclipse/Studio Job configuration, open the Run > Advanced settings tab. In the JVM Setting section, select the Use specific JVM arguments check box, click the New button, and add a new argument like this:
  4. Dhadoop.home.dir=C:\hadoop\bin
      5.Also in Development Environment set HADOOP_HOME=C:\hadoop\bin

9 comments: