hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Ratnaparkhi <abhay.ratnapar...@gmail.com>
Subject error in shuffle in InMemoryMerger
Date Tue, 28 Aug 2012 09:22:27 GMT
Hello,

I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies"  property is set to 5.

org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
java.security.AccessController.doPrivileged(AccessController.java:284) at
javax.security.auth.Subject.doAs(Subject.java:573) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/map_128.out at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
java.security.AccessController.doPrivileged(AccessController.java:284) at
javax.security.auth.Subject.doAs(Subject.java:573) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/map_119.out at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF

Regards,
Abhay

Mime
View raw message