hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject Re: The whole key space is not covered by the .META.
Date Tue, 12 Apr 2011 04:46:19 GMT
the results of ./bin/hbase hbck show lots of 'inconsistence status' like

====
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on HDFS,
but not listed in META or deployed on any region server.
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on HDFS,
but not listed in META or deployed on any region server.
ERROR: Region
hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on HDFS,
but not listed in META or deployed on any region server.
====

How can I match the names of the directories from the .META.?

I checked the help in hbase shell, but looks like it only has close_region,
no open_region.

2011/4/12 Stack <stack@duboce.net>

> Can you open the region again? (See shell commands for opening regions).
>
> What does hbck say: ./bin/hbase hbck.
>
> Add the -details flag.
>
> It might tell you a story about an offlined region.
>
> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
> on the 14th, has most of them bundled).
>
> St.Ack
>
> 2011/4/11 茅旭峰 <m9suns@gmail.com>:
> > From the output of scan '.META.' I pasted before, we can see there are
> two
> > key ranges
> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They are
> >
> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
> >
> > The output has less info about the second key range. And I also found
> some
> > log like
> >
> > ====
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG
> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
> Processing
> > close of
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
> > disabling compactions & flushes
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
> > disabled for region
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> >
> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
> > 09:48:36,953 DEBUG
> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
> > region
> >
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
> > ====
> >
> > It looks like the second had already been closed for some reason. Is it
> > possile that the info of that region
> > is not clean clearly? How can I check this in detail?
> >
> > I'm guessing maybe 'getClosestRowBefore' returns the closed region in
> > HConnectionManager.locateRegionInMeta,
> > which is closed, so this leads to the problem.
> >
> > 2011/4/12 茅旭峰 <m9suns@gmail.com>
> >
> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check the
> >> issue further, but my understanding is the meta info and the root
> >> region are saved by zookeeper, right? Do I need to check them there?
> >>
> >> m9suns
> >>
> >> 在 2011-4-12,0:40,Jean-Daniel Cryans <jdcryans@apache.org> 写道:
> >>
> >> > It's possible under some bugs, which HBase version are you using?
> >> >
> >> > J-D
> >> >
> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9suns@gmail.com> wrote:
> >> >> Hi,
> >> >>
> >> >> Is it possible that some table cannot cover the whole key space. What
> I
> >> saw
> >> >> was like
> >> >>
> >> >> ====
> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'
> >> >>
> >> >> 0 row(s) in 0.3030 seconds
> >> >>
> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> >> 'cfEStore:dasd', '123'
> >> >>
> >> >> ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
> >> >>
> >>
> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
> >> >>
> >>
> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
> >> >>
> >> >> Here is some help for this command:
> >> >> Put a cell 'value' at specified table/row/column and optionally
> >> >> timestamp coordinates.  To put a cell value into table 't1' at
> >> >> row 'r1' under column 'c1' marked with the time 'ts1', do:
> >> >>
> >> >>  hbase> put 't1', 'r1', 'c1', 'value', ts1
> >> >>
> >> >> ====
> >> >>
> >> >> I guess this means our table has lost some key range for the specific
> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
> >> >> is this reasonable? I wonder what might cause this issue? How can I
> >> check
> >> >> it?
> >> >>
> >> >> One weird thing, I can see such lines like
> >> >>
> >> >> ====
> >> >>
> >>
> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
> >> >> ====
> >> >>
> >> >> from the web UI, which I think means the table has holds the key
> range
> >> for
> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
> >> >>
> >> >> I also have not seen any errors/warnings in the log file. Any
> >> suggestions?
> >> >>
> >> >> Thanks in advance.
> >> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message