hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Anyone successfully deployed Hive on TEZ engine?
Date Mon, 30 May 2016 11:55:12 GMT
Hi Gopal,

please see my correspondence about Tez in tez user group. I forwarded to
hive user group.

thanks

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 30 May 2016 at 12:30, Gopal Vijayaraghavan <gopalv@apache.org> wrote:

> > I do not use any vendor's product., All my own set up, build and
> >configure.
>
> My autobuild scripts should serve as readable documentation for this,
> since nearly everything's in a single Makefile with an install: target.
>
> Or take the easy route with
>
> $ make dist install
>
> In case you use the llap branch, just do "set
> hive.llap.execution.mode=none;" to use Tez.
>
> > java version "1.8.0_77"
> > Hadoop 2.6.0
> ...
> > https://tez.apache.org/install.html
>
> Looks good so far.
>
> > Ok I just need to make it work as I have hive on spark engine as well.
>
> You're missing 3 things approximately - if you read through the Makefile
> in github.
>
> First, a good tez-site.xml in the classpath (remember, tez.lib.uris needs
> to be an HDFS path - for the rest, see the base file from autobuild).
>
> I usually update Tez to ${fs.default.name}/user/gopal/tez/tez.tar.gz and I
> do not use the minimal tarball, but the full dist tarball.
>
> The fixed tarball means it hits all the good localization characteristics
> of YARN, which can add up to minutes on a >250+ node cluster.
>
> Second, put that in the classpath for Hive (append to
> $INSTALL_ROOT/hive/bin/hive-config.sh)
>
> > export
> >HADOOP_CLASSPATH="$INSTALL_ROOT/tez/*:$INSTALL_ROOT/tez/lib/*:$INSTALL_ROO
> >T/tez/conf/:$HADOOP_CLASSPATH"
>
> > export HADOOP_USER_CLASSPATH_FIRST=true
>
>
> Replace $INSTALL_ROOT with wherever Tez is located.
>
> Third, disable the hive-1.x jars coming from SparkSQL (append/create in
> $INSTALL_ROOT/hive/conf/hive-env.sh)
>
> > export HIVE_SKIP_SPARK_ASSEMBLY=true
>
>
> After that, you can do
>
> > hive --hiveconf hive.execution.engine=tez
>
> to get Tez working (add --hiveconf tez.queue.name=<queue> to use queues).
>
> Cheers,
> Gopal
>
>
>

Mime
View raw message