Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=O1cjDYykMREPwJTwooY8UIQeG4KAVKk6AAZZaBrxYHhrGfJ1akGArWCp+2rpt5mnjbOtDFyC4r/UPx00nDjopuKx9prP3C4LFa1bv8NwkZ8GF3JHDr9Ic/l5G9vrG/cE4ndRqVMYTWtGMlX/a6c+gY1s896sjuJpTvWgT3rc7bI=;
Message-ID: <32285.60871.qm@web36101.mail.mud.yahoo.com>
Date: Fri, 12 Jun 2009 11:25:20 -0700 (PDT)
From: Zhengguo 'Mike' SUN <zhengguosun@yahoo.com>
Subject: The behavior of HashPartitioner
To: core-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-399386578-1244831120=:60871"

--0-399386578-1244831120=:60871
Content-Type: text/plain; charset=us-ascii

Hi,

The intermediate key generated by my Mappers is IntWritable. I tested with different number of Reducers. When the number of Reducers is the same as the number of different keys of intermediate output. It partitions perfectly. Each Reducer receives one input group. When these two numbers are different, the partitioning function becomes difficult to understand. For example, when the number of keys is less than the number of Reducers, I am expecting that each Reducer at most receive one input group. But it turns out that many Reducers receive more than one input group. On the other hand, when the number of keys is larger than the number of Reducers, I am expecting that each Reducer at least receive one input group. But it turns out that some Reducers receive nothing to process. The expectation I had is from the implementation of HashPartitioner class, which just uses modulo operator with the number of Reducers to generate partitions.

Anyone has any insights into this?


--0-399386578-1244831120=:60871--