hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogini Gulkotwar <yogini.gulkot...@flutura.com>
Subject Discrepancy in the values of consumed disk space by hadoop
Date Sun, 11 Aug 2013 17:14:50 GMT
Hi All,

I have a CDH4 hadoop cluster setup with 3 datanodes and a data replication
factor of 2.

When I try to check the consumed dfs space, I get different values using
the "hdfs dfsadmin -report" and "hdfs fsck" command.
Could anyone please help me understand the reason behind the discrepancy in
the values?

 I get the following output:

*# sudo -u hdfs hdfs dfsadmin -report*


Configured Capacity: 321252989337600 (292.18 TB)
Present Capacity: 264896108259328 (240.92 TB)
DFS Remaining: 264665811648512 (240.71 TB)
DFS Used: 230296610816 (214.48 GB)
DFS Used%: 0.09%
Under replicated blocks: 19
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Live datanodes:
Name: (slave1)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 107084329779200 (97.39 TB)
DFS Used: 77728510976 (72.39 GB)
Non DFS Used: 18784664751104 (17.08 TB)
DFS Remaining: 88221936517120 (80.24 TB)
DFS Used%: 0.07%
DFS Remaining%: 82.39%
Last contact: Fri Aug 09 13:26:38 IST 2013


Name: (slave3)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 107084329779200 (97.39 TB)
DFS Used: 76206287872 (70.97 GB)
Non DFS Used: 18786185925632 (17.09 TB)
DFS Remaining: 88221937565696 (80.24 TB)
DFS Used%: 0.07%
DFS Remaining%: 82.39%
Last contact: Fri Aug 09 13:26:37 IST 2013


Name:(slave2)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 107084329779200 (97.39 TB)
DFS Used: 76361811968 (71.12 GB)
Non DFS Used: 18786030401536 (17.09 TB)
DFS Remaining: 88221937565696 (80.24 TB)
DFS Used%: 0.07%
DFS Remaining%: 82.39%

--------------------------------------------------------------------------------------------------------------------------
*# sudo -u hdfs hadoop fsck /*


Connecting to namenode via http://master1:50070


Status: HEALTHY
 Total size: 75245213337 B
 Total dirs: 3203
 Total files: 7893
 Total blocks (validated): 7642 (avg. block size 9846272 B)
 Minimally replicated blocks: 7642 (100.0 %)
 Over-replicated blocks: 0 (0.0 %)
 Under-replicated blocks: 19 (0.24862601 %)
 Mis-replicated blocks: 0 (0.0 %)
 Default replication factor: 2
 Average block replication: 2.0024862
 Corrupt blocks: 0
 Missing replicas: 133 (0.86162215 %)
 Number of data-nodes: 3
 Number of racks: 1
FSCK ended at Fri Aug 09 14:01:47 IST 2013 in 266 milliseconds


The filesystem under path '/' is HEALTHY

----------------------------------------------------------------------------------------------------------------------------------------------------


*# sudo -u hdfs hadoop fs -count -q /*
  2147483647      2147472547            none             inf         3203
      7897        75245470999 /



Thanks & Regards,
*Yogini Gulkotwar*
*Flutura Decision Sciences & Analytics, Bangalore*
*Email*: yogini.gulkotwar@flutura.com<yogini.gulkotwar@fluturasolutions.com>
*Website*: www.fluturasolutions.com

Mime
View raw message