hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Berry, Matt" <mwbe...@amazon.com>
Subject RE: Null Pointer Excpetion while writing Key on Mapper emit
Date Wed, 04 Jul 2012 19:32:14 GMT
Thanks for the response Anand,
The NullPointerException was by design, I wanted to illustrate that it was not null. Your
suspicion about having multiple threads was correct. I traced the error down to the threadpool
terminating prior to the last mapper completing its operations.

Matthew Berry

-----Original Message-----
From: Anand Srivastava [mailto:Anand.Srivastava@guavus.com] 
Sent: Tuesday, July 03, 2012 8:31 PM
To: <mapreduce-user@hadoop.apache.org>
Subject: Re: Null Pointer Excpetion while writing Key on Mapper emit

Hi Matt,
	You StringOne is clearly not null else it would have thrown an exception in the if. You should
not be calling equals on String unless you are sure that it is not null; use == instead for
nullity checks. This means that one of 'kvbuffer' or 'b' (byte[] argument) to be null.

Mind telling us the hadoop version you are using? 
Any chance that there are two or more threads in your mapper class?

On 04-Jul-2012, at 2:46 AM, Berry, Matt wrote:

> While emitting a record from my mapper, I am receiving a NullPointerException. The stack
trace seems to indicate there is a problem serializing the Key. The key contains a few strings
and a few longs. It is a string that it is having trouble serializing. I've checked the contents
of the string post-mortem and it looks like a valid string (at least, other strings like it
work just fine). Could this be a problem not with the string itself but the destination of
the arraycopy()?
> The Key.write(DataOutput out) function:
>    @Override
>    public void write(DataOutput out) throws IOException {
>        if(StringOne.equals(null) | StringTwo.equals(null)) {
>            System.out.println("This is not hit");
>        }
>        out.writeUTF(StringOne); //Line 125
>        out.writeUTF(StringTwo);
>        out.writeLong(LongOne);
>        out.writeLong(LongTwo);
>    }
> The Stack Trace:
> java.lang.NullPointerException
>        at java.lang.System.arraycopy(Native Method)
>        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1108)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
>        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
>        at xxxxxx.xxxxxxxxxxxx.xxxx.xxxxx.xxxxxKey.write(xxxxxKey.java:125)
>        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
>        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
>        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:892)
>        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
>        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:82)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:63)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxx(xxxxxxxxxxxxxxxxxxxxxxxxxxxx.java:131)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.xxxxxxxxxxxxxxx(xxxxxxxxxxxxxx.java:282)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.access$000(xxxxxxxxxxxxxx.java:42)
>        at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx$xxxxxxxxxxxx.run(xxxxxxxxxxxxx.java:74)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)

View raw message