hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: Hive on TEZ + LLAP
Date Tue, 19 Jul 2016 01:17:12 GMT

> These looks pretty impressive. What execution mode were you running
>these? Yarn client may be?

There is no other mode - everything runs on YARN.

> 53 times


The factor is actually bigger in actual execution.

The MRv2 version takes 2.47s to prep a query, while the LLAP version takes
1.64s.

The MRv2 version takes 200.319s to execute the query, while the LLAP
version takes 1.02s.

The execution factor is nearly ~200x, but the compile becomes significant
as you scale down the latencies.

> My calculations on Hive 2 on Spark 1.3.1

Not sure where Hive2-on-Spark is going - the last commit to SparkCompiler
was late last year, before there was a Hive2.

On the speed front, I'm pretty sure you have got most of the Hive2
optimizations disabled, even the most basic of the Stinger optimizations
might be missing for you.

Check if you have

set hive.vectorized.execution.enabled=true;


Some of these new optimizations don't work on H-o-S, because Hive-on-Spark
does not implement a true broadcast join - instead it uses a
SparkHashTableSinkOperatorwhich actually writes to HDFS instead of sending
it directy to the downstream task.


I don't understand why that is the case instead of RDD brodcast, but that
prevents the JOIN optimizations which convert the 34 sec query into a 3.8
sec query from applying to Spark execution.

A couple of examples would be

set hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled=true;
set hive.vectorized.execution.mapjoin.minmax.enabled=true;

Those two make easy work of joins in LLAP, particularly semi-joins which
are common in BI queries.


Once LLAP is out of tech preview, we can enable most of them by default
for Tez+LLAP, but that would not mean all of it applies to
Hive-on-(Spark/MR).

Getting these new features onto another engine takes active effort from
the engine's devs.

Cheers,
Gopal











Mime
View raw message