hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stan Rosenberg <stan.rosenb...@gmail.com>
Subject Re: custom partitioner
Date Sun, 25 Mar 2012 16:51:05 GMT
Typically, numPartitons is used as a modulus in orde to derive a value that
is between zero and strictly less than numPartitons.  That is, key.get() %
numPartitions would yield such a value.

stan
On Mar 25, 2012 11:25 AM, "Harun Ra┼čit ER" <harunrasiter@gmail.com> wrote:

> public int getPartition(IntWritable key, Chromosome value, int
> numOfPartitions)
>  {
>                int partition = key.get();
>                if (partition < 0 || partition >= numOfPartitions)
>                {
>                        partition = numOfPartitions-1;
>                }
>                System.out.println("partition "+partition );
>                return partition;
> }
>
> I wrote the custom partitioner above. But the problem is about the third
> parameter, numOfPartitions.
>
> It is always "1" in pseudo-distributed mode. I have 4 mappers and 4
> reducers, but only one of the reducers uses the real values. The others
> yield nothing, just empty files.
>
> When I remove the if statement, hadoop complains about the partition number
> as "illegal partition for ...".
>
> How can i set the number of partitions in pseudo-distributed mode?
>
> Thanks.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message