hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4915) Out of Memory error in reduce shuffling phase when compression is turned on
Date Thu, 18 Dec 2008 21:17:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657906#action_12657906
] 

Zheng Shao commented on HADOOP-4915:
------------------------------------

Error log:

2008-12-18 11:42:46,319 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
done copying task_200812161126_7976_m_000462_0 output from hadoop5355.snc1.facebook.com..
2008-12-18 11:42:46,593 WARN org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError: Direct
buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:633)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
	at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompressor.java:108)
	at org.apache.hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.java:188)
	at org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(SequenceFile.java:1458)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1564)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1363)
	at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:2989)
	at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2804)
	at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2556)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:1632)

2008-12-18 11:42:47,272 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1:
Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from
previous failures
2008-12-18 11:42:47,272 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Got 384 known map output location(s); scheduling...
2008-12-18 11:42:47,273 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Scheduled 4 of 384 known outputs (0 slow hosts and 380 dup hosts)
2008-12-18 11:42:47,273 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Copying task_200812161126_7976_m_002498_0 output from hadoop5373.snc1.facebook.com..
2008-12-18 11:42:47,274 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Copying task_200812161126_7976_m_000879_0 output from hadoop5158.snc1.facebook.com..
2008-12-18 11:42:47,274 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Copying task_200812161126_7976_m_005415_0 output from hadoop5354.snc1.facebook.com..
2008-12-18 11:42:47,274 INFO org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1
Copying task_200812161126_7976_m_006464_0 output from hadoop5323.snc1.facebook.com..
2008-12-18 11:42:47,276 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: task_200812161126_7976_r_000272_1The reduce copier failed
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:329)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2139)
2008-12-18 11:42:47,277 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure:
java.lang.NullPointerException
	at org.apache.hadoop.fs.InMemoryFileSystem$RawInMemoryFileSystem.reserveSpace(InMemoryFileSystem.java:324)
	at org.apache.hadoop.fs.InMemoryFileSystem.reserveSpaceWithCheckSum(InMemoryFileSystem.java:436)
	at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:153)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:818)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:767)

2008-12-18 11:42:47,279 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure:
java.lang.NullPointerException
	at org.apache.hadoop.fs.InMemoryFileSystem$RawInMemoryFileSystem.reserveSpace(InMemoryFileSystem.java:324)
	at org.apache.hadoop.fs.InMemoryFileSystem.reserveSpaceWithCheckSum(InMemoryFileSystem.java:436)
	at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:153)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:818)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:767)


> Out of Memory error in reduce shuffling phase when compression is turned on
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4915
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4915
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.2
>            Reporter: Zheng Shao
>
> mapred.compress.map.output is set to true, and the job has 6860 mappers and 300 reducers.
> Several reducers failed because:out of memory error in the shuffling phase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message