hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: custom partitioner
Date Sun, 25 Mar 2012 17:17:58 GMT
Harun,

Does your map task stdout logs show varying values for "partition"?
Seems to me like all your keys are somehow outside of [0,
numOfPartitions), and hence go to the last partition, per your logic.

2012/3/25 Harun Raşit ER <harunrasiter@gmail.com>:
> public int getPartition(IntWritable key, Chromosome value, int
> numOfPartitions)
>  {
>                int partition = key.get();
>                if (partition < 0 || partition >= numOfPartitions)
>                {
>                        partition = numOfPartitions-1;
>                }
>                System.out.println("partition "+partition );
>                return partition;
> }
>
> I wrote the custom partitioner above. But the problem is about the third
> parameter, numOfPartitions.
>
> It is always "1" in pseudo-distributed mode. I have 4 mappers and 4
> reducers, but only one of the reducers uses the real values. The others
> yield nothing, just empty files.
>
> When I remove the if statement, hadoop complains about the partition number
> as "illegal partition for ...".
>
> How can i set the number of partitions in pseudo-distributed mode?
>
> Thanks.



-- 
Harsh J

Mime
View raw message