2

I'm working on a Hadoop installation done by someone else. 

Ubuntu 18.04.1 LTS
Hadoop version 3.0.3
Java open jdk 11 (auto mode) and 8 (manual mode)

Currently Hadoop runs with Java 11 but I want to configure Hadoop to run with Java 8 because Hive doesn't work with Java 11.

Following is the Java configuration in my virtual machine. 

hadoop@hadoop-VirtualBox:~$ whereis java
java: /usr/bin/java /usr/share/java /usr/share/man/man1/java.1.gz

hadoop@hadoop-VirtualBox:~$ update-alternatives --list java
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

hadoop@hadoop-VirtualBox:~$ java -version
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1, mixed mode)

hadoop@hadoop-VirtualBox:~$ sudo update-alternatives --config java

There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      auto mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode


hadoop@hadoop-VirtualBox:~$ ls -lh /usr/lib/jvm/
total 8.0K
lrwxrwxrwx 1 root root   25 Apr  8 14:16 default-java -> java-1.11.0-openjdk-amd64
lrwxrwxrwx 1 root root   21 Apr 21 14:41 java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64
drwxr-xr-x 9 root root 4.0K Aug 23 12:28 java-11-openjdk-amd64
lrwxrwxrwx 1 root root   20 Apr 26 16:59 java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64
drwxr-xr-x 7 root root 4.0K Aug 17 08:24 java-8-openjdk-amd64

I was looking at this thread to configure Java however in the .bashrc or /etc/environment file there is no entry on JAVA_HOME. The only place JAVA_HOME is specified is in the /usr/local/hadoop/etc/hadoop/hadoop-env.sh file. In the file JAVA_HOME is specified as follows:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

With the above configuration Hadoop runs without any problem. I tried changing the JAVA_HOME in the hadoop-env.sh file to /usr/lib/jvm/java-8-openjdk-amd64 and tried running Hadoop. But this gives an error

Unrecognized option: --add-modules Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I tried changing the priority of the Java version using sudo update-alternatives --config java along with the hadoop-env.sh alteration but it still gives the same error.

In order to change the Java version do I need to add the path to .bashrc and /etc/environment as well? I would like to know what changed needs to be done.

SriniShine
  • 183
  • 1
  • 2
  • 7

1 Answers1

0

Make sure that you have a verified backup first.

A lot of the hadoop ecosystem only supports Java 8. Scala, Spark, etc. However, when installing Hadoop, JAVA_HOME needs to be setup first.

You will need to uninstall Java 11 before installing Java 8. For a Hadoop system, it will be better to receive an error message to help identify where the issue is, then to have different JVMs running (Java 11 and Java 8). Note: I am assuming that Hadoop is running automatically at system startup, so you will definitely need to remove Java 11.

Then you will need to go thru each of the Hadoop configurations files to double check that JAVA_HOME has not been configured. For Ubuntu this is generally /usr/local/hadoop/etc/hadoop but can be anything that the previous admin wanted ($HADOOP_HOME/etc/hadoop). A grep of then entire directly will speed this up. This goes for each add-on product such as Hive, HBase, etc. You may be lucky in that only hadoop-env.sh has JAVA_HOME configured.

Also check the init startup scripts in /etc/init.d

After rebooting Hadoop, review the log files. I would also shutdown and restart each service one at a time and review the output.

John Hanley
  • 150
  • I got Hadoop working with java. Hadoop path settings were in in .profile instead on .bashrc. Still I've got some problem in running hive. I've checked several videos some people include the settings in the .bashrc or .profile or both. Is having settings in the .profile is sufficient? – SriniShine Sep 01 '18 at 07:54
  • What problems running Hive? Did you remove the other Java version? Have you reviewed the log files for Hive? When you restart the Hive service what errors are logged / displayed? Where you place the environment variables depends on how Hadoop is starting uip (as a service on boot or manually after you login and by which shell). Read a document on how Hive is configured and the verify your installation. – John Hanley Sep 01 '18 at 16:03
  • About java - Yes I completely removed java (all versions) using purge and installed Java 8 I run hadoop manually - the environment variables were placed in the .profile by the person who installed it first. I think because I run hadoop manually it needs to be specified in the .bashrc. However dind't move them to .bashrc Hive - I guess one problem with hive was that Hive 2.3 was installed with Hadoop 3.1. So I installed the latest hive version (3.1). But still has an error with the metastore – SriniShine Sep 07 '18 at 09:49
  • Sorry about the late reply. I'm new to Ubuntu environment hence have quite a lot to learn. – SriniShine Sep 07 '18 at 09:49
  • This is the hive error - https://askubuntu.com/questions/1073035/failed-hiveexception-java-lang-runtimeexception-unable-to-instantiate-org-apac – SriniShine Sep 07 '18 at 17:17