ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 26591: hive command line fails to start after setting hive.execution.engine to tez in HDP 2.2.1
Date Sat, 11 Oct 2014 02:43:33 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/
-----------------------------------------------------------

(Updated Oct. 11, 2014, 2:43 a.m.)


Review request for Ambari, Andrew Onischuk, Dmytro Sen, and Sumit Mohanty.


Bugs: AMBARI-7739
    https://issues.apache.org/jira/browse/AMBARI-7739


Repository: ambari


Description
-------

Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.

After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell
generates an error.

Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning:
TezSession has already shutdown
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
	at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
	... 8 more
    
    
***********************************************************
# This needs two +1s in order to be committed to branch-1.7.0


Diffs
-----

  ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6

  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
93ad95d 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml
c1e8fa6 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
dfedb6b 

Diff: https://reviews.apache.org/r/26591/diff/


Testing (updated)
-------

Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
Then changed the following properties to coincide with the changed files.

HDFS => hadoop-env = 
{
...

if [[ -d "/usr/hdp/current/tez-client" &&  -d "/etc/tez/conf/" ]]; then
  # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez
in HDP.
  export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
fi

# Setting path to hdfs command line
export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}

# Mostly required for hadoop 2.0
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64

...
}


Hive => hive-env =
{
...

if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
  export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
  export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
fi

...
}

Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz


After restarting the services, ran the following commands to simulate the behavior of the
scripts during the installation of Hadoop.

export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH

Then ran
su - hive
hive

And was able to get the hive shell to start.
To test this on a brand new cluster install, would like for this to be committed first.

Also ran unit tests,
----------------------------------------------------------------------
Total run:641
Total errors:0
Total failures:0
OK

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:21.678s
[INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
[INFO] Final Memory: 63M/1540M
[INFO] ------------------------------------------------------------------------


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message