hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jitendra Yadav <jeetuyadav200...@gmail.com>
Subject Re: skewjoin problem
Date Mon, 11 May 2015 07:55:23 GMT
May be your one reducer is overloaded due to groupby keys. If you are using
groupby then try below property and see if reducer data distributed.

set hive.groupby.skewindata=true;

Thanks
Jitendra

On Mon, May 11, 2015 at 12:35 PM, r7raul1984@163.com <r7raul1984@163.com>
wrote:

> Status: Running (Executing on YARN cluster with App id
> application_1419300485749_1493279)
>
> --------------------------------------------------------------------------------
>
> VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
> --------------------------------------------------------------------------------
>
> Map 1 .......... SUCCEEDED 200 200 0 0 0 0
> Map 4 .......... SUCCEEDED 3 3 0 0 0 0
> Map 5 .......... SUCCEEDED 152 152 0 0 0 0
> Reducer 2 ..... RUNNING 20 19 1 0 0 0
> Reducer 3 RUNNING 23 0 23 0 0 0
> --------------------------------------------------------------------------------
>
> VERTICES: 03/05 [========================>>--] 93% ELAPSED TIME: 791.14 s
>
>
> A reduce run for a long time.
>
> I try set hive.exec.reducers.bytes.per.reducer = 4000000000;
> set hive.skewjoin.key = 1000000000;
> set hive.optimize.skewjoin =true;
>
> but nothing helped. Only the reduce num decrease....
>
>
> ------------------------------
> r7raul1984@163.com
>

Mime
View raw message