hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander C.H. Lorenz" <wget.n...@googlemail.com>
Subject Re: HDFS reporting varying number of under-replicated blocks; MISSING blocks
Date Sat, 05 Nov 2011 09:56:22 GMT
Hi,

could be that the balancer is running, I tried to reproduce and see some
output like this with a running job. Just an idea.

- alex

On Fri, Nov 4, 2011 at 2:43 PM, <shawn.higgins@thomsonreuters.com> wrote:

>  Hello all,****
>
> I am getting wildly varying reports of under-replicated blocks from
> dfsadmin and fsck. I am wondering what's causing this. hadoop dfsadmin
> -metasave reports ~1,000 actual blocks awaiting replication and about
> ~404,000 MISSING blocks awaiting replication. How do I fix this? Why are
> there so many MISSING blocks? All data nodes have checked in and none are
> dead. The number of missing blocks has been increasing by about 50,000 to
> 100,000 each day for the past few days. Jobs run just fine and there does
> not appear to be data missing. ****
>
> Please see output from the following commands below: hadoop fsck /, hadoop
> dfsadmin -report, hadoop dfsadmin -metasave, and the namenode web GUI.****
>
> ** **
>
> *DFSadmin report:*
>
> Configured Capacity: 59070545264640 (53.72 TB)****
>
> Present Capacity: 56104851291863 (51.03 TB)****
>
> DFS Remaining: 35451062198272 (32.24 TB)****
>
> DFS Used: 20653789093591 (18.78 TB)****
>
> DFS Used%: 36.81%****
>
> *Under replicated blocks: 405571*
>
> Blocks with corrupt replicas: 110****
>
> Missing blocks: 0****
>
> ** **
>
> *Hadoop fsck /:*
>
> Total size:    6793151991960 B (Total open files size: 46894626752 B)****
>
> Total dirs:    1748****
>
> Total files:   229033 (Files currently being written: 486)****
>
> Total blocks (validated):      244648 (avg. block size 27767044 B) (Total
> open file blocks (not validated): 675)****
>
> Minimally replicated blocks:   244648 (100.0 %)****
>
> Over-replicated blocks:        0 (0.0 %)****
>
> *Under-replicated blocks:       931 (0.38054675 %)*
>
> Mis-replicated blocks:         0 (0.0 %)****
>
> Default replication factor:    3****
>
> Average block replication:     2.9945922****
>
> Corrupt blocks:                0****
>
> Missing replicas:              1844 (0.25169903 %)****
>
> Number of data-nodes:          20****
>
> Number of racks:               1****
>
> FSCK ended at Fri Nov 04 08:26:25 CDT 2011 in 7222 milliseconds****
>
> The filesystem under path '/' is HEALTHY****
>
> ** **
>
> *Hadoop dfsadmin –metasave excerpt:*
>
> 231268 files and directories, 245326 blocks = 476594 total****
>
> Live Datanodes: 20****
>
> Dead Datanodes: 0****
>
> Metasave: Blocks waiting for replication: 404829****
>
> /file/logs/prod/A/20111031/kj9384-91-prod-oidu.20111028.213210:
> blk_-9102337703676304543_9404421 (replicas: l: 1 d: 0 c: 0 e: 0)
> 33.112.130.196:50010 :****
>
> ..........****
>
> : blk_9223167009736432600_10568409 *MISSING* (replicas: l: 0 d: 0 c: 0 e:
> 0)****
>
> : blk_9223167009736432600_10574235 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10579879 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10582788 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10588269 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10595646 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10598074 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10600859 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10606703 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10612156 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10614776 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10624115 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10630240 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10634047 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10637486 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10644655 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10653784 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10657780 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10661247 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> : blk_9223167009736432600_10664704 MISSING (replicas: l: 0 d: 0 c: 0 e: 0)
> ****
>
> Metasave: Blocks being replicated: 0****
>
> Metasave: Blocks 0 waiting deletion from 0 datanodes.****
>
> Metasave: Number of datanodes: 20****
>
> ** **
>
> *Web GUI:*
>
> Cluster Summary****
>
> 231267 files and directories, 245316 blocks = 476583 total. Heap Size is
> 626.94 MB / 8.68 GB (7%)****
>
> Configured Capacity:      53.72 TB****
>
> DFS Used             :               19.05 TB****
>
> Non DFS Used   :               2.43 TB****
>
> DFS Remaining  :               32.25 TB****
>
> DFS Used%         :               35.46 %****
>
> DFS Remaining%:             60.02 %****
>
> Live Nodes          :               20****
>
> Dead Nodes       :               0****
>
> Decommissioning Nodes :            0****
>
> *Number of Under-Replicated Blocks:*     *405943*****
>
> ** **
>
> *Shawn Higgins*****
>
> *Thomson Reuters*
>
> shawn.higgins@thomsonreuters.com
> thomsonreuters.com**
>



-- 
Alexander Lorenz
http://mapredit.blogspot.com

*P **Think of the environment: please don't print this email unless you
really need to.*

Mime
View raw message