hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jia, Ke A" <ke.a....@intel.com>
Subject RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez
Date Sat, 25 Nov 2017 10:40:51 GMT
Hi Gopal ,
>I still do not understand the "only" mode. In the "only" mode, where the query fragment
run, LLAP daemon or tez container? 
>I change the execution mode from "all" to "only" and disable the HybridGraceHashJoin.
The execution time of q1 from 76s to 456s.
Now, in "all" mode , we set the llap container size to 210g(llap java heap size:180g, cache
size:20g). If we change to "only" mode, Whether we change the llap container size? And do
you have some recommended value? 

Regards,
Ke
-----Original Message-----
From: Gopal Vijayaraghavan [mailto:gopalv@apache.org] 
Sent: Saturday, November 25, 2017 3:39 AM
To: user@hive.apache.org
Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

Hi,
 
> In our test, we found the shuffle stage of LLAP is very slow. Whether need to configure
some related shuffle value or not? 

Shuffle is the one hit by the 2nd, 3rd and 4th resource starvation issues listed earlier (FDs,
somaxconn & DNS UDP packet loss).

> And we get the following log from the LLAP daemon in shuffle stage:
> 2017-11-23T12:48:40,718 INFO  [New I/O worker #128 ()] org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler:
Setting connection close header...

This is from a Tez setting (& is unexpected with LLAP). Enable keep-alive on the Tez client-side


https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/tez-site.xml.frag#L171

FYI, the whole autobuild repo was made to allow easy rebuilding of Hive-LLAP with some sort
of basic settings for a machine with 256Gb RAM  & 32 HT cores.

> Now " hive.llap.execution.mode" have "auto,none,all,map,only" mode. About the four mode,
do you have some suggestions? Whether the "all" mode can gain the best performance or not?
And  how the "auto" and "only" mode work?

The fastest mode is "only" - "auto" was designed for hybrid execution of ETL queries (i.e
small tasks run in LLAP, large tasks run in Tez) & slows down BI queries (i.e slower queries,
higher throughput, assuming bulk ETL).

The "all" model was designed to throw as much work into LLAP as possible, without failing
queries. The trouble with this mode is that the performance of queries will be unpredictable,
if some part of them falls out of LLAP. The "only" mode fails those queries, so that you can
report a bug or at least, know that the performance loss is because LLAP is not active.

Once you switch to the "only" mode, it is a good idea to disable the HybridGraceHashJoin,
to get much more performance out of the engine. Because the HybridGrace is a stateful hashtable,
it produces 1 hashtable per-thread to avoid race conditions, while disabling it produces 1
read-only stateless hashtable which ends up being interleaved across all NUMA zones, but built
exactly once per LLAP daemon.

This will cause a reduction in CPU usage (which is a good thing, unlike the lock starvation
problems).

Cheers,
Gopal  


Mime
View raw message