hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Custom Partitioner is now working in CDH4
Date Wed, 23 Jan 2013 17:03:37 GMT
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.

My response below:

>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?


On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:

> Hi  All
>    Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3  same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
>         static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
>         }
>         @Override
>         public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
>             String [] nameAgeScore = value.toString().split("\t");
>             String age = nameAgeScore[1];
>             int ageInt = Integer.parseInt(age);
>
>             //this is done to avoid performing mod with 0
>             if(numReduceTasks == 0)
>                 return 0;
>
>             //if the age is <20, assign partition 0
>             if(ageInt <=20){
>                 return 0;
>             }
>             //else if the age is between 20 and 50, assign partition 1
>             if(ageInt >20 && ageInt <=50){
>
>                 return 1 % numReduceTasks;
>             }
>             //otherwise assign partition 2
>             else
>                 return 2 % numReduceTasks;
>
>         }
>     }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>


-- 
Harsh J

Mime
View raw message