hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Shraibman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-2564) NullPointerException in WritableComparator
Date Fri, 03 Jun 2011 15:37:50 GMT
NullPointerException in WritableComparator
------------------------------------------

                 Key: MAPREDUCE-2564
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: task
    Affects Versions: 0.20.203.0
         Environment: java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)


hadoop-0.20.203.0rc1
            Reporter: Joseph Shraibman
            Priority: Blocker


java.lang.NullPointerException
        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

It is easy to see why this is happening.  The WritableComparator is created in JobConf line
776:
{code:title=JobConf.java}
   return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
}
{code}
which calls
{code:title=WritableComparator.java|borderStyle=solid}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
    this(keyClass, false);
  }

  protected WritableComparator(Class<? extends WritableComparable> keyClass,
      boolean createInstances) {
    this.keyClass = keyClass;
    if (createInstances) {
      key1 = newKey();
      key2 = newKey();
      buffer = new DataInputBuffer();
    } else {
      key1 = key2 = null;
      buffer = null;
    }
  }
{code}

key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because
buffer is null

{code:title=WritableComparator.java|borderStyle=solid}
 public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
    try {
      buffer.reset(b1, s1, l1);                   // parse key1
      key1.readFields(buffer);
      
      buffer.reset(b2, s2, l2);                   // parse key2
      key2.readFields(buffer);
      
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
    
    return compare(key1, key2);                   // compare them
  }
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message