hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian C. Huffman" <bhuff...@etinternational.com>
Subject Re: Datanode volume full, but not moving to free volume
Date Wed, 08 Oct 2014 11:39:03 GMT
Hmmm..  It seems that there's only one block pool per disk.  So that 
won't help me.  :-(

Also, I see the blockpool directory names are all the same.  Is that 
expected?  So even if I put a larger disk in, I couldn't consolidate the 
smaller disk's blockpool directories?
[hadoop@thor1 current]$ ls 
/data/data1/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207  VERSION
[hadoop@thor1 current]$ ls 
/data/data2/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207  VERSION
[hadoop@thor1 current]$ ls 
/data/data3/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207  VERSION

Regards,
Brian

On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> I would try to move the Block Pools directories 
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your 
> DataNode process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman 
> <bhuffman@etinternational.com <mailto:bhuffman@etinternational.com>> 
> wrote:
>
>     Can I move a whole subdir?  Or does it have to be individual block
>     files / metadata?
>
>     For example, I see this:
>     [hadoop@thor1 finalized]$ pwd
>     /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>     [hadoop@thor1 finalized]$ du -sh subdir10/
>     80G    subdir10/
>
>     So could I move subdir10 to the same location under /data/data3?
>
>     Thanks,
>     Brian
>
>
>     Brian C. Huffman System Administrator ET International, Inc.On
>     10/8/14, 4:44 AM, Aitor Cedres wrote:
>>
>>     Hi Brian,
>>
>>     Hadoop does not balance the disks within a DataNode. If you ran
>>     out of space and then add additional disks, you should shutdown
>>     the DataNode and move manually a few files to the new disk.
>>
>>     Regards,
>>
>>     Aitor Cedrés
>>
>>     On 6 October 2014 14:46, Brian C. Huffman
>>     <bhuffman@etinternational.com
>>     <mailto:bhuffman@etinternational.com>> wrote:
>>
>>         All,
>>
>>         I have a small hadoop cluster (2.5.0) with 4 datanodes and 3
>>         data disks per node. Lately some of the volumes have been
>>         filling, but instead of moving to other configured volumes
>>         that *have* free space, it's giving errors in the datanode logs:
>>         2014-10-03 11:52:44,989 ERROR
>>         org.apache.hadoop.hdfs.server.datanode.DataNode:
>>         thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>          operation  src: /172.17.1.3:35412 <http://172.17.1.3:35412>
>>         dst: /172.17.1.2:50010 <http://172.17.1.2:50010>
>>         java.io.IOException: No space left on device
>>             at java.io.FileOutputStream.writeBytes(Native Method)
>>             at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>             at
>>         org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>             at
>>         org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>             at
>>         org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>             at
>>         org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>             at
>>         org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>             at
>>         org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>             at java.lang.Thread.run(Thread.java:745)
>>
>>         Unfortunately it's continuing to try to write and when it
>>         fails, it's passing the exception to the client.
>>
>>         I did a restart and then it seemed to figure out that it
>>         should move to the next volume.
>>
>>         Any suggestions to keep this from happening in the future?
>>
>>         Also - could it be an issue that I have a small amount of
>>         non-HDFS data on those volumes?
>>
>>         Thanks,
>>         Brian
>>
>>
>
>


Mime
View raw message