hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C G <parallel...@yahoo.com>
Subject When is HDFS really "corrupt"...(and can I upgrade a corrupt FS?)
Date Thu, 15 May 2008 19:15:22 GMT
Earlier this week I wrote about a master node crash and our efforts to recover from the crash.
 We recovered from the crash and all systems are normal.  However, I have a concern about
what fsck is reporting and what it really means for a filesystem to be marked "corrupt."
   
  With the mapred engine shut down, I ran fsck / -files -blocks -locations to inspect the
file system.  The output looks clean with the exception of this at the end of the output:
   
  Status: CORRUPT
 Total size:    5113667836544 B
 Total blocks:  1070996 (avg. block size 4774684 B)
 Total dirs:    50012
 Total files:   1027089
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Target replication factor:     3
 Real replication factor:       3.0
  
The filesystem under path '/' is CORRUPT
   
  In reviewing the fsck output, there are no obvious errors being reported.  I see tons of
output like this:
   
  /foo/bar/part-00005 3387058, 6308 block(s):  OK
0. 4958936159429948772 len=3387058 repl=3 [10.2.14.5:50010, 10.2.14.20:50010, 10.2.14.8:50010]
   
  and the only status ever reported is "OK."
   
  So this begs the question about what causes HDFS to declare the FS is "corrupt" and how
do I clear this up?
   
  The second question, assuming that I can't make the "corrupt" state go away, concerns running
an upgrade.  If every file in HDFS reports "OK" but the FS reports "corrupt", is it safe to
undertake an upgrade from 0.15.x to 0.16.4 ?
   
  Thanks for any help....
  C G
   

       
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message