hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: QuickSort array out of bound exception during Secondary Sort
Date Thu, 07 Feb 2013 17:03:23 GMT
Without access to org.skyz.basic.KeyComparator.compare it is hard to say why this is happening.
It looks like you got an array with 0 entries and the given function could not handle that.

I don't really know why.


From: Aseem Anand <aseem.iiith@gmail.com<mailto:aseem.iiith@gmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, February 5, 2013 7:11 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: QuickSort array out of bound exception during Secondary Sort

I'm facing a problem with hadoop's secondary sort such that it is displaying the following
error message. The code I have used has been used by me previously and had not given any issues
for a large number of intermediate output pairs. The number of mapper output values is around
175,000. Could someone please guide me on this .

13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

Hoping for a positive response.
Thanks & Regards,
Aseem Anand

View raw message