hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aitor Cedres <aced...@pivotal.io>
Subject Re: Datanode volume full, but not moving to free volume
Date Wed, 08 Oct 2014 08:44:58 GMT
Hi Brian,

Hadoop does not balance the disks within a DataNode. If you ran out of
space and then add additional disks, you should shutdown the DataNode and
move manually a few files to the new disk.

Regards,

Aitor Cedrés


On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com>
wrote:

> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
> per node.  Lately some of the volumes have been filling, but instead of
> moving to other configured volumes that *have* free space, it's giving
> errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>  operation  src: /172.17.1.3:35412 dst: /172.17.1.2:50010
> java.io.IOException: No space left on device
>     at java.io.FileOutputStream.writeBytes(Native Method)
>     at java.io.FileOutputStream.write(FileOutputStream.java:345)
>     at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receivePacket(BlockReceiver.java:592)
>     at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receiveBlock(BlockReceiver.java:734)
>     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
> writeBlock(DataXceiver.java:741)
>     at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(
> Receiver.java:124)
>     at org.apache.hadoop.hdfs.protocol.datatransfer.
> Receiver.processOp(Receiver.java:71)
>     at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
> DataXceiver.java:234)
>     at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails, it's
> passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should move to
> the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS data on
> those volumes?
>
> Thanks,
> Brian
>
>

Mime
View raw message