hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne <wav...@gmail.com>
Subject Re: hbck -fix
Date Sun, 03 Jul 2011 02:27:14 GMT
Like most problems we brought it on ourselves. To me the bigger issue is how
to get out. Since region definitions are the core of what hbase does, it
would be great to have a bullet proof recovery process that we can invoke to
get us out. Bugs and human error will bring on problems and nothing will
ever change that, but not having tools to help recover out of the hole is
where I think it is lacking. HDFS is very stable. The hbase .META. table
(and -ROOT-?) are the core how HBase manages things. If this gets out of
whack all is lost. I think it would be great to have automatic backup of the
meta table and the ability to recover everything based on the HDFS data out
there and the backup. Something like a recovery mode that goes through and
sees what is out there and rebuilds the meta based on it. With corrupted
data and lost regions etc. etc. like any relational database there should be
one or more recovery modes that goes through everything and rebuilds it
consistently. Data may be lost but at least the cluster will be left in a
100% consistent/clean state. Manual editing of .META. is not something
anyone should do (especially me). It is prone to human error...it should be
easy to have well tested recover tools that can do the hard work for us.

Below is an attempt at the play by play in case it helps. It all started
with the root partition of the namenode/hmaster filling up due to a table
export.

When I restarted hadoop this error was in the namenode log;
"java.io.IOException: Incorrect data format. logVersion is -18 but
writables.length is 0"

So i found this<https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/e35ee876da1a3bbc>,
which mentioned editing the namenode log files after verifying our namenode
log files seem to have the same symptom. So I copied each namenode "name"
file to root's home directory and followed their advice.
That allowed the namenode to start, but then HDFS wouldn't come up. It kept
hanging in safe-mode with the repeated error;
"The ratio of reported blocks 0.9925 has not reached the threshold 0.9990.
Safe mode will be turned off automatically."
So i turned safe-mode off with; "hadoop dfsadmin -safemode leave" and I
tried to run "hadoop fsck" a few times and it still showed HDFS as
"corrupt", so i did "hadoop fsck -move" and this is the last part of the
output;
....................................................................................Status:
CORRUPT
 Total size: 1423140871890 B (Total open files size: 668770828 B)
 Total dirs: 3172
 Total files: 2584 (Files currently being written: 11)
 Total blocks (validated): 23095 (avg. block size 61621167 B) (Total open
file blocks (not validated): 10)
  ********************************
  CORRUPT FILES: 65
  MISSING BLOCKS: 173
  MISSING SIZE: 8560948988 B
  CORRUPT BLOCKS: 173
  ********************************
 Minimally replicated blocks: 22922 (99.25092 %)
 Over-replicated blocks: 0 (0.0 %)
 Under-replicated blocks: 0 (0.0 %)
 Mis-replicated blocks: 0 (0.0 %)
 Default replication factor: 3
 Average block replication: 2.9775276
 Corrupt blocks: 173
 Missing replicas: 0 (0.0 %)
 Number of data-nodes: 10
 Number of racks: 1

I ran it again and got this;
.Status: HEALTHY
 Total size: 1414579922902 B (Total open files size: 668770828 B)
 Total dirs: 3272
 Total files: 2519 (Files currently being written: 11)
 Total blocks (validated): 22922 (avg. block size 61712761 B) (Total open
file blocks (not validated): 10)
 Minimally replicated blocks: 22922 (100.0 %)
 Over-replicated blocks: 0 (0.0 %)
 Under-replicated blocks: 0 (0.0 %)
 Mis-replicated blocks: 0 (0.0 %)
 Default replication factor: 3
 Average block replication: 3.0
 Corrupt blocks: 0
 Missing replicas: 0 (0.0 %)
 Number of data-nodes: 10
 Number of racks: 1


The filesystem under path '/' is HEALTHY

So i started everything and it seemed to be superficially functional.

I then shutdown hadoop and restarted. hadoop came up in a matter of a few
minutes, then hbase took about ten minutes of seeming to copy files around,
based on the hbase master logs.

After this we saw region not found client errors on some tables. I ran hbase
hbck to look for problems and saw the errors I reported in the original
post. Add in the ganglia problems and a botched attempt to edit the .META.
table which brought us even further into the rabbit hole. I then decided to
drop the affected tables but lo and behold one can not disable a table that
has messed up regions...so I manually deleted the data but some of the
.META. table entries were still there. Finally this afternoon we reformatted
the entire cluster.

Thanks.



On Sat, Jul 2, 2011 at 5:25 PM, Stack <stack@duboce.net> wrote:

> On Sat, Jul 2, 2011 at 9:55 AM, Wayne <wav100@gmail.com> wrote:
> > It just returns a ton of errors (import: command not found). Our cluster
> is
> > hosed anyway. I am waiting to get it completely re-installed from
> scratch.
> > Hope has long since flown out the window. I just changed my opinion of
> what
> > it takes to manage hbase. A Java engineer is required on staff. I also
> > realized now a backup strategy is more important than for a RDBMS. Having
> > RF=3 in HDFS offers no insurance against hbase lossing its shirt and
> having
> > .META. getting corrupted. I think I just found the achilles heel.
> >
> >
>
> Yeah, stability is primary but I do not know how you got into the
> circumstance you find yourself in.  All I can offer is to try and do
> diagnostics since avoiding hitting this situation again is of utmost
> importance.
>
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message