hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 권병창 <magnu...@navercorp.com>
Subject mismatch corrupt blocks from fsck and dfadmin report.
Date Tue, 12 Jan 2016 10:20:06 GMT
Hi. hadooper!
 
I use hadoop-2.7.1 and  my cluster has 130 nodes.
 
recently I am facing a problem.
 
I have found corrupt block by nagios.
 
nagios request http://namenode01:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem.

 
below is result.  notice CorruptBlocks is 1. 

 {
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=FSNamesystem",
    "modelerType" : "FSNamesystem",
    "tag.Context" : "dfs",
    "tag.HAState" : "active",
    "tag.Hostname" : "css0700.nhnsystem.com",
    "MissingBlocks" : 0,
    "MissingReplOneBlocks" : 0,
    "ExpiredHeartbeats" : 10,
    "TransactionsSinceLastCheckpoint" : 820630,
    "TransactionsSinceLastLogRoll" : 1916,
    "LastWrittenTransactionId" : 376685578,
    "LastCheckpointTime" : 1452583950883,
    "CapacityTotal" : 1650893130075660,
    "CapacityTotalGB" : 1537514.0,
    "CapacityUsed" : 1237079990848257,
    "CapacityUsedGB" : 1152121.0,
    "CapacityRemaining" : 410189981364473,
    "CapacityRemainingGB" : 382019.0,
    "CapacityUsedNonDFS" : 3623157862930,
    "TotalLoad" : 6717,
    "SnapshottableDirectories" : 0,
    "Snapshots" : 0,
    "BlocksTotal" : 4034155,
    "FilesTotal" : 2866690,
    "PendingReplicationBlocks" : 0,
    "UnderReplicatedBlocks" : 0,
    "CorruptBlocks" : 1,
    "ScheduledReplicationBlocks" : 0,
    "PendingDeletionBlocks" : 0,
    "ExcessBlocks" : 87,
    "PostponedMisreplicatedBlocks" : 0,
    "PendingDataNodeMessageCount" : 0,
    "MillisSinceLastLoadedEdits" : 0,
    "BlockCapacity" : 67108864,
    "StaleDataNodes" : 0,
    "TotalFiles" : 2866690
  } ]
}


 
 
however  'hdfs fsck / -list-corruptfileblocks' does not found.
 
below is fsck result.

 The filesystem under path '/' has 0 CORRUPT files



 
'hdfs dfsadmin -report'  is similar first result.

 Configured Capacity: 1650892318461452 (1.47 PB)
Present Capacity: 1647353181258422 (1.46 PB)
DFS Remaining: 408711410865856 (371.72 TB)
DFS Used: 1238641770392566 (1.10 PB)
DFS Used%: 75.19%
Under replicated blocks: 0
Blocks with corrupt replicas: 1
Missing blocks: 0
Missing blocks (with replication factor 1): 0



 
My question is
 
1. why different result?
2. How do I find corrupt filename? I wonder which file is corrupt.
 
Thank you.
 
Mime
View raw message