Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 48868D6D9 for ; Sun, 4 Nov 2012 21:15:46 +0000 (UTC) Received: (qmail 75948 invoked by uid 500); 4 Nov 2012 21:15:41 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 75864 invoked by uid 500); 4 Nov 2012 21:15:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 75857 invoked by uid 99); 4 Nov 2012 21:15:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Nov 2012 21:15:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of aseem.iiith@gmail.com designates 209.85.215.176 as permitted sender) Received: from [209.85.215.176] (HELO mail-ea0-f176.google.com) (209.85.215.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Nov 2012 21:15:36 +0000 Received: by mail-ea0-f176.google.com with SMTP id n12so2358526eaa.35 for ; Sun, 04 Nov 2012 13:15:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vUeY07GqQl0Z0t2yxhaqedGAGuu3rfjvSb8Q1EHnZro=; b=mdSed7XIlxmoHcnRRyhfC37RPbNqIQfZfPWY72zO9NIO+F2fxoCFQ5syQ0gzRPrQUi sqv9Cbh7PQZV3SH3JkKd9QDM6SSXcx9HWkja4NiSa/OxFTJlYxmdSpkwiqMKB59HNW3a iQ6hIuBe9DWdDWdwG/fcxwVyAXfIqS/7iLFahJpkimJTrdOg+0D3E13+h1G1NtVoTkJw /Uv1su6cQH9tY/4O4he0VHSLJHn+l/93Dut1wQPrctUWNU84GbWwob68GPCrxJLcoTk4 2kYSVzzTS0bAL6Q0c6DZl1mOyjDjrCuM2W8Ws7GjQYoxmIUxNnOL0ke37n9TKUt/w6wk f6xQ== MIME-Version: 1.0 Received: by 10.14.225.71 with SMTP id y47mr29627386eep.0.1352063715054; Sun, 04 Nov 2012 13:15:15 -0800 (PST) Received: by 10.14.134.11 with HTTP; Sun, 4 Nov 2012 13:15:15 -0800 (PST) In-Reply-To: References: Date: Mon, 5 Nov 2012 02:45:15 +0530 Message-ID: Subject: Re: Task does not enter reduce function after secondary sort From: Aseem Anand To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b66fa5173153304cdb1dd56 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b66fa5173153304cdb1dd56 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hey, Here are code snippets. In the driver class : job.setMapperClass(SkyzKnnMapperT.class); job.setReducerClass(SkyzKnnReducer.class); job.setGroupingComparatorClass(GroupComparator.class); job.setPartitionerClass(MyPartitioner.class); job.setSortComparatorClass(KeyComparator.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(NullWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); public class GroupComparator extends WritableComparator { protected GroupComparator() { super(Text.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { //consider only zone and day part of the key Text t1 =3D (Text) w1; Text t2 =3D (Text) w2; String[] t1Items =3D t1.toString().split(":"); String[] t2Items =3D t2.toString().split(":"); int comp =3D t1Items[0].compareTo(t2Items[0]); System.out.println("GROUP" + comp); return comp; } } public class SkyzKnnReducer extends Reducer { public void reduce(Text key, Iterable values, Context output, Reporter reporter) throws IOException, InterruptedException { String t =3D key.toString(); t =3D "HELLO" + t; output.write(new Text(t),new Text(t)); } } The composite key is of the form A:Rest_of_text where A is the natural key. Override annotation to this reduce method shows an error in Eclipse. What else could be going wrong ? Thanks, Aseem On Mon, Nov 5, 2012 at 2:33 AM, Harsh J wrote: > Sounds like an override issue to me. If you can share your code, we > can take a quick look - otherwise, try annotating your reduce(=85) > method with @Override and recompiling to see if it really is the right > signature Java expects. > > On Mon, Nov 5, 2012 at 1:48 AM, Aseem Anand wrote= : > > Hi, > > I am using a Secondary Sort for my Hadoop program. My map function emit= s > > (Text,NullWritable) where Text contains the composite key and appropria= te > > comparison functions are made and a custom Partitioner . These seem to = be > > working fine. > > > > I have been struggling with the problem that these values are not being > > received by the reduce function and instead automatically get written t= o > the > > hdfs in x number of files where x is the number of reducers. I have mad= e > > sure the reduce function is set to my Reduce function and not identity > > reduce. > > > > Can someone please explain this behavior and what could be possibly > wrong ? > > > > Thanks & Regards, > > Aseem > > > > -- > Harsh J > --047d7b66fa5173153304cdb1dd56 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hey,
Here are code snippets.

In the driver cla= ss :=A0
=A0 =A0 =A0 =A0 job.setMapperClass(SkyzKnnMapperT.cl= ass); =A0 =A0 =A0
=A0 =A0 =A0 =A0 job.setReducerClass(SkyzKnnRedu= cer.class);
=A0 =A0 =A0 =A0 job.setGroupingComparatorClass(GroupComparator.class);
=A0 =A0 =A0 =A0 job.setPartitionerClass(MyPartitioner.class);
=A0 =A0 =A0 =A0 job.setSortComparatorClass(KeyComparator.class);
=A0 =A0 =A0 =A0 job.setMapOutputKeyClass(Text.class);
=A0 =A0 =A0 =A0 job.setMapOutputValueClass(NullWritable.class);
<= div>=A0 =A0 =A0 =A0 job.setOutputKeyClass(Text.class);
=A0 =A0 = =A0 =A0 job.setOutputValueClass(Text.class);

=
public class GroupComparator extends WritableComparator {

protected GroupComparator() {
super(Text.class, true);
}

@Override
public int compare(WritableComparable w1, WritableCompa= rable w2) {

//consider only zone and day part of the key
Text t1 =3D (Text)= w1;
Text= t2 =3D (Text) w2;
String[] t1Items =3D t1.toString().split(":")= ;
String[] = t2Items =3D t2.toString().split(":");
int comp =3D t1Items[0].co= mpareTo(t2Items[0]);
Syst= em.out.println("GROUP" + comp);
return comp;

}
}
public class SkyzKnnReducer extends Reducer<Text,Ite= rable,Text,Text> {
public void reduce(Text key, Iterable= <NullWritable> values,
Context output, Reporter reporter)
throws IOException, InterruptedExcep= tion {
=A0 =A0 =A0 =A0 =A0 =A0 String t =3D key.toString();
=A0 =A0= =A0 =A0 =A0 =A0 t =3D "HELLO" + t;
=A0 =A0 =A0 =A0 =A0= =A0 output.write(new Text(t),new Text(t));

}
}

The composite key is of the form A:Rest_of_= text where A is the natural key.

Override annotati= on to this reduce method shows an error in Eclipse. What else could be goin= g wrong ?=A0

Thanks,
Aseem=A0
On Mon, Nov 5, 2012 at 2:33 AM, Harsh J <harsh@cloudera.com= > wrote:
Sounds like an override issue to me. If you = can share your code, we
can take a quick look - otherwise, try annotating your reduce(=85)
method with @Override and recompiling to see if it really is the right
signature Java expects.

On Mon, Nov 5, 2012 at 1:48 AM, Aseem Anand <aseem.iiith@gmail.com> wrote:
> Hi,
> I am using a Secondary Sort for my Hadoop program. My map function emi= ts
> (Text,NullWritable) where Text contains the composite key and appropri= ate
> comparison functions are made and a custom Partitioner . These seem to= be
> working fine.
>
> I have been struggling with the problem that these values are not bein= g
> received by the reduce function and instead automatically get written = to the
> hdfs in x number of files where x is the number of reducers. I have ma= de
> sure the reduce function is set to my Reduce function and not identity=
> reduce.
>
> Can someone please explain this behavior and what could be possibly wr= ong ?
>
> Thanks & Regards,
> Aseem



--
Harsh J

--047d7b66fa5173153304cdb1dd56--