hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Harsch <>
Subject Re: Turning on Tez for Hive
Date Thu, 17 Jul 2014 23:29:46 GMT
Hi Lefty,
I came across those documents as well.  They gave me some good hints, but
were in some places too specific to Horton Works.  For the record, I did
get Hive 0.13.1 and Tez 0.4.1 working together.  (and I immediately saw a
200% speed up on my corpus of queries).  For future travelers here are my
(somewhat raw) notes:

Get the source distro at:

explode it, and run 'mvm package -DskipTests -Dtar'

explode the tar from tez-dist/target/tez-0.4.1-incubating.tar.gz to a
location which will become TEZ_INSTALL_DIR.

Tez needs a user dir for to match unix user, or you will get File does not exist: hdfs:/user/tharsch:
% hdfs dfs -mkdir /user/tharsch; hdfs dfs -chmod g+w /user/tharsch

# NOTE: set HIVE_AUX_JARS_PATH using if -z test or get
java.lang.IllegalArgumentException: Can not create a Path from an empty
# NOTE: TEZ_JARS cannot include "*" or you will get error
" File
file:/home/users/tharsch/apps/tez/tez-0.4.1-incubating/* does not exist"

export TEZ_INSTALL_DIR=/home/users/tharsch/apps/tez/tez-0.4.1-incubating
export TEZ_JARS=$(echo "$TEZ_INSTALL_DIR"/*.jar | tr ' ' ':'):$(echo
"$TEZ_INSTALL_DIR"/lib/*.jar | tr ' ' ':')
if [ -z "$HIVE_AUX_JARS_PATH" ]; then

NOTE:  Be sure to copy TEZ jars to HDFS and set HADOOP_CLASSPATH or you
will get:
org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez
jars, tez.lib.uris is not defined in the configurartion


% hdfs dfs -mkdir -p /apps/tez-0.4.1; hdfs dfs -chmod g+w /apps/tez-0.4.1
% hdfs dfs -copyFromLocal $TEZ_INSTALL_DIR/* /apps/tez-0.4.1
% hdfs dfs -copyFromLocal $TEZ_INSTALL_DIR/lib/* /apps/tez-0.4.1

NOTE: Set mapred-site.xml

NOTE: create tez-site.xml in $TEZ_CONF_DIR

NOTE: HIVE Settings:
set hive.execution.engine=tez;
set hive.use.tez.natively=true;
set hive.enable.mrr=true;

View raw message