hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: No space left on device during merge.
Date Fri, 24 Jan 2014 18:29:04 GMT
That's a lot of data to process for a single reducer. You should try
increasing the number of reducers to achieve more parallelism and also try
modifying your logic to avoid significant skew in the reducers.

Unfortunately this means rethinking about your app, but that's the only way
about it. It will also help you scale smoothly into the future if you have
adjustable parallelism and more balanced data processing.

+Vinod
Hortonworks Inc.
http://hortonworks.com/


On Fri, Jan 24, 2014 at 6:47 AM, Tim Potter <tep@yahoo-inc.com> wrote:

>  Hi,
>   I'm getting the below error while trying to sort a lot of data with Hadoop.
>
> I strongly suspect the node the merge is on is running out of local disk space. Assuming
this is the case, is there any way
> to get around this limitation considering I can't increase the local disk space available
on the nodes?  Like specify sort/merge parameters or similar.
>
> Thanks,
>   Tim.
>
> 2014-01-24 10:02:36,267 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new
decompressor [.lzo_deflate]
> 2014-01-24 10:02:36,280 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last
merge-pass, with 100 segments left of total size: 642610678884 bytes
> 2014-01-24 10:02:36,281 ERROR [main] org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:XXXXXX (auth:XXXXXX) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
error in shuffle in OnDiskMerger - Thread to merge on-disk map-outputs
> 2014-01-24 10:02:36,282 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in
OnDiskMerger - Thread to merge on-disk map-outputs
> 	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:167)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> Caused by: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> 	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:213)
> 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> 	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:107)
> 	at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:88)
> 	at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:150)
> 	at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:140)
> 	at org.apache.hadoop.io.compress.BlockCompressorStream.write(BlockCompressorStream.java:99)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:107)
> 	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:249)
> 	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:200)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeManager$OnDiskMerger.merge(MergeManager.java:572)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94)
> Caused by: java.io.IOException: No space left on device
> 	at java.io.FileOutputStream.writeBytes(Native Method)
> 	at java.io.FileOutputStream.write(FileOutputStream.java:318)
> 	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:211)
> 	... 14 more
>
> 2014-01-24 10:02:36,284 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for
the task
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message