Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D932510EAB for ; Fri, 30 Aug 2013 00:56:16 +0000 (UTC) Received: (qmail 31503 invoked by uid 500); 30 Aug 2013 00:56:12 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 31236 invoked by uid 500); 30 Aug 2013 00:56:11 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 31229 invoked by uid 99); 30 Aug 2013 00:56:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Aug 2013 00:56:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of adeelmahmood@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Aug 2013 00:56:07 +0000 Received: by mail-we0-f172.google.com with SMTP id t60so1069379wes.3 for ; Thu, 29 Aug 2013 17:55:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=k9fhO02UrHusgzbS5aG+KmavvKzCMCVYRQ0vlMH7mVc=; b=OuF1ptn+wVlvpL1viAfhKQM6rJX7JW5RSJZQ4pIxuA280kCfjrMQ4MXdRCD1swuwhP QL3jZh6p1JQ6rJ2rxR/UWCAQnkHX0BcJ+HtD+vGIa1bMz1BN+/1IOerl9MhaPZPXDoen rl+JDcn2GKiCD5c5rrTiDR+4ZI1PI210kGNX5ZrhtHRDBMBZI3pd/fPAjiTubvKK4izW qTAE9/GLFCpY+Szn7Gq4q0qPavNy0fUePUyWMrmjxHYUHvfNzxEd93i4GZVwMMK/JFxG aYOkQVDNqTuEW2dPmZr+8pce+2lSutbEOIaj1BrgsLT2l8yDH47BHVeRagy/YC54tOjW 1kHw== MIME-Version: 1.0 X-Received: by 10.194.122.129 with SMTP id ls1mr120843wjb.37.1377824146425; Thu, 29 Aug 2013 17:55:46 -0700 (PDT) Received: by 10.180.72.101 with HTTP; Thu, 29 Aug 2013 17:55:46 -0700 (PDT) In-Reply-To: References: <6CC784D5-D3E0-45BC-916C-D9865AA4F27B@cloudera.com> Date: Thu, 29 Aug 2013 20:55:46 -0400 Message-ID: Subject: Re: secondary sort - number of reducers From: Adeel Qureshi To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e012299c8cf8ee004e51fae32 X-Virus-Checked: Checked by ClamAV on apache.org --089e012299c8cf8ee004e51fae32 Content-Type: text/plain; charset=ISO-8859-1 okay so when i specify the number of reducers e.g. in my example i m using 4 (for a much smaller data set) it works if I use a single column in my composite key .. but if I add multiple columns in the composite key separated by a delimi .. it then throws the illegal partition error (keys before the pipe are group keys and after the pipe are the sort keys and my partioner only uses the group keys java.io.IOException: Illegal partition for *Atlanta:GA|Atlanta:GA:1:Adeel*(-1) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1073) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at com.att.hadoop.hivesort.HSMapper.map(HSMapper.java:39) at com.att.hadoop.hivesort.HSMapper.map(HSMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.Child.main(Child.java:249) public int getPartition(Text key, HCatRecord record, int numParts) { //extract the group key from composite key String groupKey = key.toString().split("\\|")[0]; return groupKey.hashCode() % numParts; } On Thu, Aug 29, 2013 at 8:31 PM, Shekhar Sharma wrote: > No...partitionr decides which keys should go to which reducer...and > number of reducers you need to decide...No of reducers depends on > factors like number of key value pair, use case etc > Regards, > Som Shekhar Sharma > +91-8197243810 > > > On Fri, Aug 30, 2013 at 5:54 AM, Adeel Qureshi > wrote: > > so it cant figure out an appropriate number of reducers as it does for > > mappers .. in my case hadoop is using 2100+ mappers and then only 1 > reducer > > .. since im overriding the partitioner class shouldnt that decide how > > manyredeucers there should be based on how many different partition > values > > being returned by the custom partiotioner > > > > > > On Thu, Aug 29, 2013 at 7:38 PM, Ian Wrigley wrote: > >> > >> If you don't specify the number of Reducers, Hadoop will use the default > >> -- which, unless you've changed it, is 1. > >> > >> Regards > >> > >> Ian. > >> > >> On Aug 29, 2013, at 4:23 PM, Adeel Qureshi > wrote: > >> > >> I have implemented secondary sort in my MR job and for some reason if i > >> dont specify the number of reducers it uses 1 which doesnt seems right > >> because im working with 800M+ records and one reducer slows things down > >> significantly. Is this some kind of limitation with the secondary sort > that > >> it has to use a single reducer .. that kind of would defeat the purpose > of > >> having a scalable solution such as secondary sort. I would appreciate > any > >> help. > >> > >> Thanks > >> Adeel > >> > >> > >> > >> --- > >> Ian Wrigley > >> Sr. Curriculum Manager > >> Cloudera, Inc > >> Cell: (323) 819 4075 > >> > > > --089e012299c8cf8ee004e51fae32 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
okay so when i specify the number of reducers e.g. in my e= xample i m using 4 (for a much smaller data set) it works if I use a single= column in my composite key .. but if I add multiple columns in the composi= te key separated by a delimi .. it then throws the illegal partition error = (keys before the pipe are group keys and after the pipe are the sort keys a= nd my partioner only uses the group keys

java.io.IOException: Illegal partition for Atlanta:G= A|Atlanta:GA:1:Adeel (-1)
=A0 =A0 =A0 =A0 at org.apache.hadoo= p.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1073)
=A0 = =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(Ma= pTask.java:691)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapreduce.TaskInputOutputContext.= write(TaskInputOutputContext.java:80)
=A0 =A0 =A0 =A0 at com.att.= hadoop.hivesort.HSMapper.map(HSMapper.java:39)
=A0 =A0 =A0 =A0 at= com.att.hadoop.hivesort.HSMapper.map(HSMapper.java:1)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:= 144)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.runNewMa= pper(MapTask.java:764)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapre= d.MapTask.run(MapTask.java:370)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child$4.run(Child.java:255= )
=A0 =A0 =A0 =A0 at java.security.AccessController.doPrivileged(= Native Method)
=A0 =A0 =A0 =A0 at javax.security.auth.Subject.doA= s(Subject.java:396)
=A0 =A0 =A0 =A0 at org.apache.hadoop.security.UserGroupInformation.doA= s(UserGroupInformation.java:1136)
=A0 =A0 =A0 =A0 at org.apache.h= adoop.mapred.Child.main(Child.java:249)


public int getPartition(Text key, HCatRecord record, int numParts) {
<= /span>//extract the group key from composite key
String groupKey =3D key.toString().split("\\|")[0];
= return groupKey.hashCod= e() % numParts;
}


On Thu, Aug 2= 9, 2013 at 8:31 PM, Shekhar Sharma <shekhar2581@gmail.com> wrote:
No...partitionr decides which keys should go to which redu= cer...and
number of reducers you need to decide...No of reducers depends on
factors like number of key value pair, use case etc
Regards,
Som Shekhar Sharma
+91-8197243810=


On Fri, Aug 30, 2013 at 5:54 AM, Adeel Qureshi <adeelmahmood@gmail.com> wrote:
> so it cant figure out an appropriate number of reducers as it does for=
> mappers .. in my case hadoop is using 2100+ mappers and then only 1 re= ducer
> .. since im overriding the partitioner class shouldnt that decide how<= br> > manyredeucers there should be based on how many different partition va= lues
> being returned by the custom partiotioner
>
>
> On Thu, Aug 29, 2013 at 7:38 PM, Ian Wrigley <ian@cloudera.com> wrote:
>>
>> If you don't specify the number of Reducers, Hadoop will use t= he default
>> -- which, unless you've changed it, is 1.
>>
>> Regards
>>
>> Ian.
>>
>> On Aug 29, 2013, at 4:23 PM, Adeel Qureshi <adeelmahmood@gmail.com> wrote:
>>
>> I have implemented secondary sort in my MR job and for some reason= if i
>> dont specify the number of reducers it uses 1 which doesnt seems r= ight
>> because im working with 800M+ records and one reducer slows things= down
>> significantly. Is this some kind of limitation with the secondary = sort that
>> it has to use a single reducer .. that kind of would defeat the pu= rpose of
>> having a scalable solution such as secondary sort. I would appreci= ate any
>> help.
>>
>> Thanks
>> Adeel
>>
>>
>>
>> ---
>> Ian Wrigley
>> Sr. Curriculum Manager
>> Cloudera, Inc
>> Cell: (323) 819 4075
>>
>

--089e012299c8cf8ee004e51fae32--