hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhengguo 'Mike' SUN <zhengguo...@yahoo.com>
Subject The behavior of HashPartitioner
Date Fri, 12 Jun 2009 18:25:20 GMT
Hi,

The intermediate key generated by my Mappers is IntWritable. I tested with different number
of Reducers. When the number of Reducers is the same as the number of different keys of intermediate
output. It partitions perfectly. Each Reducer receives one input group. When these two numbers
are different, the partitioning function becomes difficult to understand. For example, when
the number of keys is less than the number of Reducers, I am expecting that each Reducer at
most receive one input group. But it turns out that many Reducers receive more than one input
group. On the other hand, when the number of keys is larger than the number of Reducers, I
am expecting that each Reducer at least receive one input group. But it turns out that some
Reducers receive nothing to process. The expectation I had is from the implementation of HashPartitioner
class, which just uses modulo operator with the number of Reducers to generate partitions.

Anyone has any insights into this?



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message