hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakumar...@huawei.com>
Subject RE: Trash data after upgrade from 2.7.1 to 2.7.2
Date Mon, 15 Feb 2016 05:46:51 GMT
Hi Chef,

   Can you confirm the below points?


1)      Did you upgrade all datanodes to 2.7.2?

2)      Did you finalized the upgrade using the following command?
Run "hdfs dfsadmin -rollingUpgrade finalize<https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade>"
to finalize the rolling upgrade.
If the finalize is not executed, all the blocks which were present before upgrade, will be
moved to trash on deletion.
 So to save the space, if you are trying to delete old files on upgraded ( but not finalized)
cluster, will not actually save anything on disk.
-vinay

From: Chef Win2er [mailto:win2erchef@gmail.com]
Sent: 12 February 2016 11:31
To: user@hadoop.apache.org
Subject: Trash data after upgrade from 2.7.1 to 2.7.2

Hi Hadoop users,

I have hadoop-2.7.1 installed on my cluster with HA, 4 data nodes and 3 journal nodes.
I upgraded it to hadoop2.7.2 a a few days ago following the steps below.

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Upgrade_without_Downtime
But today I realized that there's trash fold created in data node's data directory and took
a lot of space.

$ hdfs dfs -du -s -h /
11.5 G  /

I set replication 2 so the disk usage may be 30G or 40G.
But actually it is 144GB.

$ hdfs dfsadmin -report
Configured Capacity: 422185762816 (393.19 GB)
Present Capacity: 415469745432 (386.94 GB)
DFS Remaining: 260712565164 (242.81 GB)
DFS Used: 154757180268 (144.13 GB)
DFS Used%: 37.25%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

By 'du -h' commnand I got the result below.

......
11G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/current/finalized/subdir0
11G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/current/finalized
11G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/current
...
38G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/trash/finalized/subdir0
38G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/trash/finalized
38G     ./datanode/current/BP-606697376-<datanode ip>-1452599640542/trash
...
Could anyone help me with this?

Thanks
MA
Mime
View raw message