hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Bruce <dbse...@gmail.com>
Subject Re: In reduce task,i have a join operation ,and i found "org.apache.hadoop.mapred.FileInputFormat: Total input paths to process : 1" cast much long
Date Fri, 20 Oct 2017 06:35:59 GMT
Hi Gopal,

Thanks for your input! In my case I'm using MapReduce not Tez. I figured
I'd better be more specific so as to provide you more details.

For this job there are 298 maps and 74 reduces. All the maps completed real
fast within 1 minute, and 73 reduces completed in about 2 minutes.

Now there is only 1 reduce task running (forever). Here's a screenshot for
the job details: https://ibb.co/eBDj6R

I noticed one thing interesting: MAP_OUTPUT_RECORDS and
REDUCE_INPUT_RECORDS don't match for the whole job (99,073,863 vs.
98,105,913).

Here's a screenshot for the counters of the dangling reduce task:
https://ibb.co/dHgyY6

The ratio of REDUCE_INPUT_RECORDS and REDUCE_INPUT_GROUPS is 1. What does
it mean?

For comparison, here's a screenshot for the counters of a different reduce
task which completed within 1 minute. It also has the ratio of 1:
https://ibb.co/mzoHRR

Another comparison I did is for the tasklog. Below No. 1 is for the
dangling reduce task, and No. 2 and 3 are for two completed reduce tasks.
1. https://ibb.co/caKVfm
2. https://ibb.co/earJ0m
3. https://ibb.co/edQiY6

I don't understanding what the running reduce task is doing. Any other logs
that could be helpful?

Regards,
Daniel

On Thu, Oct 19, 2017 at 9:45 PM, Gopal Vijayaraghavan <gopalv@apache.org>
wrote:

> > . I didn't see data skew for that reducer. It has similar amount of
> REDUCE_INPUT_RECORDS as other reducers.
> …
> > org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 8000
> rows for join key [4092813312923569]
>
>
> The ratio of REDUCE_INPUT_RECORDS and REDUCE_INPUT_GROUPS is what is
> relevant.
>
>
>
> The row containers being spilled to disk means that at least 1 key in the
> join has > 10000 values.
>
> If you have Tez, this comes up when you run the SkewAnalyzer.
>
> https://github.com/apache/tez/blob/master/tez-tools/
> analyzers/job-analyzer/src/main/java/org/apache/tez/
> analyzer/plugins/SkewAnalyzer.java#L41
>
>
>
> Cheers,
>
> Gopal
>

Mime
View raw message