hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Datanode volume full, but not moving to free volume
Date Thu, 09 Oct 2014 01:15:02 GMT
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.


On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:

>  Hmmm..  It seems that there's only one block pool per disk.  So that
> won't help me.  :-(
>
> Also, I see the blockpool directory names are all the same.  Is that
> expected?  So even if I put a larger disk in, I couldn't consolidate the
> smaller disk's blockpool directories?
> [hadoop@thor1 current]$ ls
> /data/data1/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207  VERSION
> [hadoop@thor1 current]$ ls
> /data/data2/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207  VERSION
> [hadoop@thor1 current]$ ls
> /data/data3/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207  VERSION
>
> Regards,
> Brian
>
>
> On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
>  I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
> process before doing this operation.
>
>  Regards,
>
>  Aitor Cedrés
>
>   On 8 October 2014 11:46, Brian C. Huffman <bhuffman@etinternational.com>
> wrote:
>
>>  Can I move a whole subdir?  Or does it have to be individual block
>> files / metadata?
>>
>> For example, I see this:
>> [hadoop@thor1 finalized]$ pwd
>>
>> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>> [hadoop@thor1 finalized]$ du -sh subdir10/
>> 80G    subdir10/
>>
>> So could I move subdir10 to the same location under /data/data3?
>>
>> Thanks,
>> Brian
>>
>>
>> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
>> 4:44 AM, Aitor Cedres wrote:
>>
>>
>> Hi Brian,
>>
>>  Hadoop does not balance the disks within a DataNode. If you ran out of
>> space and then add additional disks, you should shutdown the DataNode and
>> move manually a few files to the new disk.
>>
>>  Regards,
>>
>>  Aitor Cedrés
>>
>>   On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com
>> > wrote:
>>
>>> All,
>>>
>>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>>> per node.  Lately some of the volumes have been filling, but instead of
>>> moving to other configured volumes that *have* free space, it's giving
>>> errors in the datanode logs:
>>> 2014-10-03 11:52:44,989 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>>  operation  src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>>> java.io.IOException: No space left on device
>>>     at java.io.FileOutputStream.writeBytes(Native Method)
>>>     at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>>     at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>>     at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>>     at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>>     at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>>     at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>>     at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>>     at java.lang.Thread.run(Thread.java:745)
>>>
>>> Unfortunately it's continuing to try to write and when it fails, it's
>>> passing the exception to the client.
>>>
>>> I did a restart and then it seemed to figure out that it should move to
>>> the next volume.
>>>
>>> Any suggestions to keep this from happening in the future?
>>>
>>> Also - could it be an issue that I have a small amount of non-HDFS data
>>> on those volumes?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>

Mime
View raw message