Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 42189EF86 for ; Thu, 7 Feb 2013 19:00:51 +0000 (UTC) Received: (qmail 45281 invoked by uid 500); 7 Feb 2013 19:00:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 45051 invoked by uid 500); 7 Feb 2013 19:00:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 45044 invoked by uid 99); 7 Feb 2013 19:00:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Feb 2013 19:00:45 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ravichandran777@gmail.com designates 209.85.214.174 as permitted sender) Received: from [209.85.214.174] (HELO mail-ob0-f174.google.com) (209.85.214.174) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Feb 2013 19:00:40 +0000 Received: by mail-ob0-f174.google.com with SMTP id 16so3086523obc.33 for ; Thu, 07 Feb 2013 11:00:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=n0BDRBk9myJlRE1DcFkWBhWW5OWxmxPcksJ6J+OuxkI=; b=inz+AuWZMA8hJ5XX7bfpOJC7FM/aFj7ZR8SPAyFk5nfHZhx78aTsEfFG9QflI/bA2i IylNHHSYxVirc28qNDqIGp8/0xmLrKQPRq6v+BoV+4QYLiMrVSNg2gxWvfPTG90lCUWl ZJRTKiQz125wT7rVlF1cr8It4Bg6yPiB/rd2bCnwBF2nht+HGQp6WR6Bp6RXrCoEeGBc i5biIkisYhVtmIYEOYARB2aGabzsBCiBVclOt6k1GJp222CTiwENrikuXscEKg9anA3l fFC/tODbZja3uDmlxZRQOaz9sd/4BjdNhthsv0vgQddZA/kjXRA1Wqv98tAA5zGLg5Ib +AUQ== MIME-Version: 1.0 X-Received: by 10.182.40.2 with SMTP id t2mr2053814obk.51.1360263619732; Thu, 07 Feb 2013 11:00:19 -0800 (PST) Received: by 10.182.18.195 with HTTP; Thu, 7 Feb 2013 11:00:19 -0800 (PST) In-Reply-To: References: Date: Fri, 8 Feb 2013 00:30:19 +0530 Message-ID: Subject: Re: Secondary Sort example error From: Ravi Chandran To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04462c38dae64404d5270d0d X-Virus-Checked: Checked by ClamAV on apache.org --f46d04462c38dae64404d5270d0d Content-Type: text/plain; charset=ISO-8859-1 Thanks for replying. . I believe writable and writablecomparable handle the serialization.. I found the required class public class Name implements WritableComparable{ private String fName; private String lName; static { // register this comparator WritableComparator.define(Name.class, new NameSorterComparator()); } public Name() { } public Name(String first, String last) { set(first, last); } public void set(String first, String last) { this.fName = first; this.lName = last; } public String getfName() { return fName; } public void setfName(String fName) { this.fName = fName; } public String getlName() { return lName; } public void setlName(String lName) { this.lName = lName; } public String toString() { return this.getfName()+" "+this.getlName(); } @Override public void write(DataOutput out) throws IOException { // TODO Auto-generated method stub out.writeUTF(fName); out.writeUTF(lName); } public boolean equals(Name o) { Name other = o; if(this.fName.toString().equalsIgnoreCase(other.fName.toString())) { if(this.lName.toString().equalsIgnoreCase(other.lName.toString())) { return true; } } return false; } @Override public void readFields(DataInput in) throws IOException { // TODO Auto-generated method stub this.fName = in.readUTF(); this.lName = in.readUTF(); } @Override public int hashCode() { return fName.hashCode() * 514 + lName.hashCode(); } @Override public int compareTo(Name tp) { int cmp = fName.compareTo(tp.fName); if (cmp != 0) { return cmp; } return lName.compareTo(tp.lName); } } On Fri, Feb 8, 2013 at 12:16 AM, Harsh J wrote: > Thanks, I managed to correlate proper line numbers. > > Are you using some form of custom serialization in your job code? That > is, are your keys non-Writable types and are of some other type? The > specific NPE is arising from the SerializationFactory not being able > to find a serializer for your Map-Output key class. You may want to > look into that direction, or share your code for the list to spot it > instead. > > On Fri, Feb 8, 2013 at 12:11 AM, Ravi Chandran > wrote: > > hi, > > > > it is Hadoop 2.0.0-cdh4.1.1. the whole output is given below: > > > > Hadoop 2.0.0-cdh4.1.1 > > Subversion > > > file:///data/1/jenkins/workspace/generic-package-centos32-6/topdir/BUILD/hadoop-2.0.0-cdh4.1.1/src/hadoop-common-project/hadoop-common > > -r 581959ba23e4af85afd8db98b7687662fe9c5f20 > > > > > > > > On Fri, Feb 8, 2013 at 12:04 AM, Harsh J wrote: > >> > >> Hey Ravi, > >> > >> What version of Hadoop is this exactly? (Type and send output of > >> "hadoop version" if unsure) > >> > >> On Thu, Feb 7, 2013 at 11:55 PM, Ravi Chandran > >> wrote: > >> > Hi, > >> > > >> > I am trying to do a name sorting using secondary sort. I have a > working > >> > example, which I am taking as a reference. But I am getting a null > >> > pointer > >> > error in the MapTask class. I am not able to locate the reason. as the > >> > logic > >> > to create the Custom Object from a given file has been tested through > a > >> > java > >> > class.. > >> > I am getting this error: > >> > > >> > 13/02/07 12:23:42 WARN snappy.LoadSnappy: Snappy native library is > >> > available > >> > 13/02/07 12:23:42 INFO snappy.LoadSnappy: Snappy native library loaded > >> > 13/02/07 12:23:42 INFO mapred.FileInputFormat: Total input paths to > >> > process > >> > : 1 > >> > 13/02/07 12:23:43 INFO mapred.JobClient: Running job: > >> > job_201301301056_0014 > >> > 13/02/07 12:23:44 INFO mapred.JobClient: map 0% reduce 0% > >> > 13/02/07 12:23:56 INFO mapred.JobClient: Task Id : > >> > attempt_201301301056_0014_m_000000_0, Status : FAILED > >> > java.lang.NullPointerException > >> > at > >> > > >> > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814) > >> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385) > >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) > >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > at javax.security.auth.Subject.doAs(Subject.java:396) > >> > at > >> > > >> > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > >> > at org.apache.hadoop.mapred.Child.main(Child.java:262) > >> > 13/02/07 12:23:57 INFO mapred.JobClient: Task Id : > >> > attempt_201301301056_0014_m_000001_0, Status : FAILED > >> > java.lang.NullPointerException > >> > at > >> > > >> > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:814) > >> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:385) > >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) > >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > >> > at java.security.AccessController.doPrivileged(Native Method) > >> > at javax.security.auth.Subject.doAs(Subject.java:396) > >> > at > >> > > >> > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > >> > at org.apache.hadoop.mapred.Child.main(Child.java:262) > >> > > >> > I am giving the Mapper code below: > >> > > >> > import java.io.IOException; > >> > import org.apache.hadoop.io.LongWritable; > >> > import org.apache.hadoop.io.Text; > >> > import org.apache.hadoop.mapred.MapReduceBase; > >> > import org.apache.hadoop.mapred.Mapper; > >> > import org.apache.hadoop.mapred.OutputCollector; > >> > import org.apache.hadoop.mapred.Reporter; > >> > import org.apache.log4j.Logger; > >> > import com.pom.Name; > >> > > >> > public class StubMapper extends MapReduceBase implements > >> > Mapper { > >> > > >> > private static Logger logger = > >> > Logger.getLogger(StubMapper.class.getName()); > >> > > >> > StringBuffer readLine = new StringBuffer(); > >> > private final Name name = new Name(); > >> > @Override > >> > public void map(LongWritable key, Text value, > >> > OutputCollector output, Reporter reporter) > >> > throws IOException { > >> > String line = value.toString(); > >> > String[] packer = null; > >> > > >> > packer = line.split(" "); > >> > > >> > // create the object > >> > if(packer.length>2) > >> > { > >> > // take everything except last name > >> > for (int i = 0; i < packer.length-1; i++) { > >> > readLine.append(packer[i]+" "); > >> > } > >> > > >> > name.setfName(readLine.toString()); > >> > name.setlName(packer[packer.length-1]); > >> > > >> > //clear the variable > >> > readLine.delete(0, readLine.length()); > >> > } > >> > else if(packer.length>0) > >> > { > >> > name.setfName(packer[0]); > >> > name.setlName(packer[1]); > >> > } > >> > > >> > output.collect(name, new Text(name.getlName())); > >> > > >> > } > >> > } > >> > > >> > I am not able to figure out the possible cause.. > >> > > >> > -- > >> > Thanks & Regards > >> > Ravi > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > > > -- > > Thanks & Regards > > Ravi > > > > -- > Harsh J > -- Thanks & Regards Ravi --f46d04462c38dae64404d5270d0d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for replying.=A0 . I believe writable and writ= ablecomparable handle the serialization..=A0=A0I found the required class
=A0
public class Name implements WritableComparable<N= ame>{
=A0
=A0private=A0String fName;
=A0private=A0String lName;
= =A0 static {=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 // register this comparator=A0=A0=A0=A0=A0=A0 WritableComparator.define(Name.class, new NameSorterCo= mparator());
=A0=A0=A0=A0 }
=A0
=A0
=A0public Name()
=A0{
=A0=A0
=A0}
=A0
=A0public Name(String first, String last)<= br>=A0{
=A0=A0set(first, last);
=A0}
=A0
=A0 public void set(St= ring first, String last) {
=A0=A0 this.fName =3D first;
=A0 this.lNa= me =3D last;
=A0 }
=A0public String getfName() {
=A0=A0return fName;
=A0}
=A0public v= oid setfName(String fName) {
=A0=A0this.fName =3D fName;
=A0}
=A0p= ublic String getlName() {
=A0=A0return lName;
=A0}
=A0public void = setlName(String lName) {
=A0=A0this.lName =3D lName;
=A0}
=A0public String toString()=A0{
=A0=A0return this.getfName()+" "+this.getlName();=A0=A0=
=A0}
=A0
=A0@Override
=A0public void write(DataOutput out) thr= ows IOException {
=A0=A0// TODO Auto-generated method stub
=A0=A0out.writeUTF(fName);
=A0=A0out.writeUTF(lName);
=A0}
= =A0public boolean equals(Name=A0 o) {
=A0=A0Name other =3D o;
= =A0=A0if(this.fName.toString().equalsIgnoreCase(other.fName.toString()))=A0=A0{
=A0=A0=A0if(this.lName.toString().equalsIgnoreCase(other.lName.= toString()))
=A0=A0=A0{
=A0=A0=A0=A0return true;
=A0=A0=A0}
=A0=A0}
=A0=A0re= turn false;
=A0}
=A0@Override
=A0public void readFields(Dat= aInput in) throws IOException {
=A0=A0// TODO Auto-generated method stub=
=A0=A0this.fName =3D in.readUTF();
=A0=A0this.lName =3D in.readUTF();
=A0}
=A0@Override
=A0pub= lic int hashCode() {
=A0=A0return fName.hashCode() * 514 + lName.hashCod= e();
=A0}
=A0@Override
=A0public int compareTo(Name tp) {=A0=A0int cmp =3D fName.compareTo(tp.fName);
=A0=A0if (cmp !=3D 0) {
=A0=A0=A0return cmp;
=A0=A0}
=A0=A0return = lName.compareTo(tp.lName);
=A0}
}=A0


On Fri, Feb 8, 2013 at 12:1= 6 AM, Harsh J <harsh@cloudera.com> wrote:
Thanks, I managed to correlate proper line n= umbers.

Are you using some form of custom serialization in your job code? That
is, are your keys non-Writable types and are of some other type? The
specific NPE is arising from the SerializationFactory not being able
to find a serializer for your Map-Output key class. You may want to
look into that direction, or share your code for the list to spot it
instead.

On Fri, Feb 8, 2013 at 12:11 AM, Ravi Chandran
<ravichandran777@gmail.com<= /a>> wrote:
> hi,
>
> it is Hadoop 2.0.0-cdh4.1.1. the whole output is given below:
>
> Hadoop 2.0.0-cdh4.1.1
> Subversion
> file:///data/1/jenkins/workspace/generic-package-centos32-6/topdir/BUI= LD/hadoop-2.0.0-cdh4.1.1/src/hadoop-common-project/hadoop-common
> -r 581959ba23e4af85afd8db98b7687662fe9c5f20
>
>
>
> On Fri, Feb 8, 2013 at 12:04 AM, Harsh J <
harsh@cloudera.com> wrote:
>>
>> Hey Ravi,
>>
>> What version of Hadoop is this exactly? (Type and send output of >> "hadoop version" if unsure)
>>
>> On Thu, Feb 7, 2013 at 11:55 PM, Ravi Chandran
>> <ravichandran777@g= mail.com> wrote:
>> > Hi,
>> >
>> > I am trying to do a name sorting using secondary sort. I have= a working
>> > example, which I am taking as a reference. But I am getting a= null
>> > pointer
>> > error in the MapTask class. I am not able to locate the reaso= n. as the
>> > logic
>> > to create the Custom Object from a given file has been tested= through a
>> > java
>> > class..
>> > I am getting this error:
>> >
>> > 13/02/07 12:23:42 WARN snappy.LoadSnappy: Snappy native libra= ry is
>> > available
>> > 13/02/07 12:23:42 INFO snappy.LoadSnappy: Snappy native libra= ry loaded
>> > 13/02/07 12:23:42 INFO mapred.FileInputFormat: Total input pa= ths to
>> > process
>> > : 1
>> > 13/02/07 12:23:43 INFO mapred.JobClient: Running job:
>> > job_201301301056_0014
>> > 13/02/07 12:23:44 INFO mapred.JobClient: =A0map 0% reduce 0%<= br> >> > 13/02/07 12:23:56 INFO mapred.JobClient: Task Id :
>> > attempt_201301301056_0014_m_000000_0, Status : FAILED
>> > java.lang.NullPointerException
>> > =A0at
>> >
>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>= (MapTask.java:814)
>> > =A0at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.j= ava:385)
>> > =A0at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)<= br> >> > =A0at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> > =A0at java.security.AccessController.doPrivileged(Native Meth= od)
>> > =A0at javax.security.auth.Subject.doAs(Subject.java:396)
>> > =A0at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGrou= pInformation.java:1332)
>> > =A0at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> > 13/02/07 12:23:57 INFO mapred.JobClient: Task Id :
>> > attempt_201301301056_0014_m_000001_0, Status : FAILED
>> > java.lang.NullPointerException
>> > =A0at
>> >
>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>= (MapTask.java:814)
>> > =A0at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.j= ava:385)
>> > =A0at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)<= br> >> > =A0at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> > =A0at java.security.AccessController.doPrivileged(Native Meth= od)
>> > =A0at javax.security.auth.Subject.doAs(Subject.java:396)
>> > =A0at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGrou= pInformation.java:1332)
>> > =A0at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> >
>> > I am giving the Mapper code below:
>> >
>> > import java.io.IOException;
>> > import org.apache.hadoop.io.LongWritable;
>> > import org.apache.hadoop.io.Text;
>> > import org.apache.hadoop.mapred.MapReduceBase;
>> > import org.apache.hadoop.mapred.Mapper;
>> > import org.apache.hadoop.mapred.OutputCollector;
>> > import org.apache.hadoop.mapred.Reporter;
>> > import org.apache.log4j.Logger;
>> > import com.= pom.Name;
>> >
>> > public class StubMapper extends MapReduceBase implements
>> > =A0 =A0 Mapper<LongWritable, Text, Name, Text> {
>> >
>> > =A0 private static Logger logger =3D
>> > =A0 =A0 Logger.getLogger(StubMapper.class.getName());
>> >
>> > =A0StringBuffer readLine =3D new StringBuffer();
>> > =A0 private final Name name =3D new Name();
>> > =A0 @Override
>> > =A0 public void map(LongWritable key, Text value,
>> > =A0 =A0 =A0 OutputCollector<Name, Text> output, Reporte= r reporter)
>> > =A0 =A0 =A0 throws IOException {
>> > =A0 =A0 String line =3D value.toString();
>> > =A0 =A0 String[] packer =3D null;
>> >
>> > =A0 =A0 packer =3D line.split(" ");
>> >
>> > =A0 =A0 // create the object
>> > =A0if(packer.length>2)
>> > =A0{
>> > =A0 // take everything except last name
>> > =A0 for (int i =3D 0; i < packer.length-1; i++) {
>> > =A0 =A0readLine.append(packer[i]+" ");
>> > =A0 }
>> >
>> > =A0 name.setfName(readLine.toString());
>> > =A0 name.setlName(packer[packer.length-1]);
>> >
>> > =A0 //clear the variable
>> > =A0 readLine.delete(0, readLine.length());
>> > =A0}
>> > =A0else if(packer.length>0)
>> > =A0{
>> > =A0 name.setfName(packer[0]);
>> > =A0 =A0 =A0name.setlName(packer[1]);
>> > =A0 =A0}
>> >
>> > =A0output.collect(name, new Text(name.getlName()));
>> >
>> > =A0 }
>> > }
>> >
>> > I am not able to figure out the possible cause..
>> >
>> > --
>> > Thanks & Regards
>> > Ravi
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Thanks & Regards
> Ravi



--
Harsh J



--
Thanks & RegardsRavi
--f46d04462c38dae64404d5270d0d--