hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carter Shanklin <car...@hortonworks.com>
Subject Re: Hive and engine performance tez vs mr
Date Tue, 07 Apr 2015 01:22:42 GMT
Erwan,

Faced with a similar situation last week I found that decreasing

mapred.max.split.size

Increased my parallelism by 6x. Yes mapred even though it was a Tez job. I
reduced it to 10mb from 256mb which I believe is the default.

The other variables to try are:
tez.grouping.min-size (make it smaller)
tez.grouping.max-size (smaller as well)


Good luck.


On 4/6/15, 2:57 PM, "Erwan MAS" <erwan@mas.nom.fr> wrote:

>On Mon, Apr 06, 2015 at 12:15:05PM -0500, max scalf wrote:
>> Try setting the below in Hive and see what happens..btw what are you
>> configs in hive if any?
>> 
>> set mapred.map.tasks = 20;
>> 
>
>Does not change the behavior :(
>
>--
>     ____________________________________________________________
>    / Erwan MAS                                                 /\
>   | mailto:erwan@mas.nom.fr                                   |_/
>___|________________________________________________________   |
>\___________________________________________________________\__/


Mime
View raw message