hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mawata <chris.maw...@gmail.com>
Subject Re: issue about how to assiging map output to reducer?
Date Wed, 08 Jan 2014 09:56:05 GMT
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers.  (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <justlooks@gmail.com> wrote:

> hi,maillist:
>             i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>

Mime
View raw message