ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 26591: hive command line fails to start after setting hive.execution.engine to tez in HDP 2.2.1
Date Sat, 11 Oct 2014 19:37:01 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56295
-----------------------------------------------------------



ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
<https://reviews.apache.org/r/26591/#comment96608>

    Yes it does.
    And the test fails if the symlink exists but its target doesn't.


- Alejandro Fernandez


On Oct. 11, 2014, 3:23 a.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
> 
> (Updated Oct. 11, 2014, 3:23 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-7739
>     https://issues.apache.org/jira/browse/AMBARI-7739
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
> 
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive
shell generates an error.
> 
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning:
TezSession has already shutdown
> 	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> 	at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> 	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> 	... 8 more
>     
>     
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
62e3cd6 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
93ad95d 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml
PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml
c1e8fa6 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
dfedb6b 
> 
> Diff: https://reviews.apache.org/r/26591/diff/
> 
> 
> Testing
> -------
> 
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
> 
> HDFS => hadoop-env = 
> {
> ...
> 
> if [[ -d "/usr/hdp/current/tez-client" &&  -d "/etc/tez/conf/" ]]; then
>   # When using versioned RPMs, the tez-client will be a symlink to the current folder
of tez in HDP.
>   export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
> 
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
> 
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> 
> ...
> }
> 
> 
> Hive => hive-env =
> {
> ...
> 
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
>   export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
>   export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
> 
> ...
> }
> 
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
> 
> 
> After restarting the services, ran the following commands to simulate the behavior of
the scripts during the installation of Hadoop.
> 
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
> 
> Then ran
> su - hive
> hive
> 
> And was able to get the hive shell to start.
> To test this on a brand new cluster install, would like for this to be committed first.
> 
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message