hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject Re: The whole key space is not covered by the .META.
Date Wed, 13 Apr 2011 15:11:16 GMT
The hbck output lists several empty REGIONINFO_QUALIFIER rows in .META.
I was not able to insert keys around these regions.  I removed those records
in .META.
Insertion still does not work. Then we have some holes in .META.
I tried bin/hbase org.jruby.Main bin/check_meta.rb but it complained
==
11/04/13 22:57:50 INFO zookeeper.ClientCnxn: Session establishment complete
on server cloud135/10.241.67.22:2181, sessionid = 0x12edcaa37f9005f,
negotiated timeout = 40000
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@50b2fb1e;
hsa=cloud140:60020
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is cloud136:60020
11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Creating scanner over
.META. starting at key ''
11/04/13 22:57:50 DEBUG client.HTable$ClientScanner: Advancing internal
scanner to startKey at ''
11/04/13 22:57:50 DEBUG client.HConnectionManager$HConnectionImplementation:
Cache hit for row <> in tableName .META.: location server cloud136:60020,
location region name .META.,,1.1028785192
Writables.java:75:in `org.apache.hadoop.hbase.util.Writables.getWritable':
java.lang.NullPointerException: null (NativeException)
from Writables.java:119:in
`org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
from null:-1:in `sun.reflect.GeneratedMethodAccessor6.invoke'
from DelegatingMethodAccessorImpl.java:25:in
`sun.reflect.DelegatingMethodAccessorImpl.invoke'
from Method.java:597:in `java.lang.reflect.Method.invoke'
from JavaMethod.java:196:in
`org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
from JavaMethod.java:182:in `org.jruby.javasupport.JavaMethod.invoke_static'
from JavaClass.java:371:in
`org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
from SimpleCallbackMethod.java:81:in
`org.jruby.internal.runtime.methods.SimpleCallbackMethod.call'
 ... 16 levels...
from Main.java:183:in `org.jruby.Main.runInterpreter'
from Main.java:120:in `org.jruby.Main.run'
from Main.java:95:in `org.jruby.Main.main'
Complete Java stackTrace
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
at org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:196)
at org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
at
org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.java:371)
at
org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:81)
at org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:207)
at
org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1254)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:286)
at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:387)
at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
at org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628)
at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:356)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
at org.jruby.Ruby.eval(Ruby.java:278)
at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
at org.jruby.Main.runInterpreter(Main.java:238)
at org.jruby.Main.runInterpreter(Main.java:183)
at org.jruby.Main.run(Main.java:120)
at org.jruby.Main.main(Main.java:95)
==

>From a previous thread, I saw some one insert some records into .META.
manually,
to fill the holes, it seems like we need to generate encoded region name
with
tablename, start_key, id and md5-hash, I do not know how to generate id and
start_key.
Any tips on fill those region holes?

Thanks and regards,

Mao Xu-Feng

2011/4/13 茅旭峰 <m9suns@gmail.com>

> Hi Stack,
>
> Sorry for bothering you in private.
>
> Regarding this issue, so far I still can not put specific key into the
> hbase?
>
> I attached the output of './bin/hbase shell' with this message.
>
> No matter what causes this issue, is there any operation I can perform to
> cover this scenario, at least we need chance to put those key-values.
>
> And then, I'm trying to find the root cause by debugging.
>
> Will keep you update. Thanks!
>
>
> 2011/4/13 Stack <stack@duboce.net>
>
>> You can also cat the .regioninfo file that is under the region
>> directory to learn more about the region -- its HRegionInfo (The file
>> format is serialized HRegionInfo followed by a String version so
>> intelligible to non-machines).
>> St.Ack
>>
>> 2011/4/12 Stack <stack@duboce.net>:
>> > These regions that are in hdfs but not in .META. and not on any region
>> > server are probably harmless.  Would be interesting to trace how they
>> > got to this state.  My guess is that they are old let go regions that
>> > were not cleaned up.
>> >
>> > The region directory name is its encoded name which is the tail part
>> > of the row name in the .META. table.
>> >
>> > St.Ack
>> >
>> >
>> > 2011/4/11 茅旭峰 <m9suns@gmail.com>:
>> >> the results of ./bin/hbase hbck show lots of 'inconsistence status'
>> like
>> >>
>> >> ====
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01c80f8b54523ad6c242c5f695544f16 on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01ce4e2f72baa0df51b7b2010c9d1ef0 on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ERROR: Region
>> >> hdfs://cloud137:9000/hbase/table1/01d0855af5e646823bdff82fddadd81e on
>> HDFS,
>> >> but not listed in META or deployed on any region server.
>> >> ====
>> >>
>> >> How can I match the names of the directories from the .META.?
>> >>
>> >> I checked the help in hbase shell, but looks like it only has
>> close_region,
>> >> no open_region.
>> >>
>> >> 2011/4/12 Stack <stack@duboce.net>
>> >>>
>> >>> Can you open the region again? (See shell commands for opening
>> regions).
>> >>>
>> >>> What does hbck say: ./bin/hbase hbck.
>> >>>
>> >>> Add the -details flag.
>> >>>
>> >>> It might tell you a story about an offlined region.
>> >>>
>> >>> 0.90.2 has some fixes for issues in and around here (CDH3 release, out
>> >>> on the 14th, has most of them bundled).
>> >>>
>> >>> St.Ack
>> >>>
>> >>> 2011/4/11 茅旭峰 <m9suns@gmail.com>:
>> >>> > From the output of scan '.META.' I pasted before, we can see there
>> are
>> >>> > two
>> >>> > key ranges
>> >>> > which might cover the put key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='. They
>> are
>> >>> >
>> >>> > #1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
>> >>> > #2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key'
>> >>> >
>> >>> > The output has less info about the second key range. And I also
>> found
>> >>> > some
>> >>> > log like
>> >>> >
>> >>> > ====
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>> >>> > Processing
>> >>> > close of
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Closing
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
>> >>> > disabling compactions & flushes
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Updates
>> >>> > disabled for region
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Closed
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> >
>> >>> >
>> essential/hbase-0.90.1-CDH3B4/logs/hbase-hadoop-regionserver-cloud138.log.2011-03-21:2011-03-21
>> >>> > 09:48:36,953 DEBUG
>> >>> > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
>> Closed
>> >>> > region
>> >>> >
>> >>> >
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
>> >>> > ====
>> >>> >
>> >>> > It looks like the second had already been closed for some reason.
Is
>> it
>> >>> > possile that the info of that region
>> >>> > is not clean clearly? How can I check this in detail?
>> >>> >
>> >>> > I'm guessing maybe 'getClosestRowBefore' returns the closed region
>> in
>> >>> > HConnectionManager.locateRegionInMeta,
>> >>> > which is closed, so this leads to the problem.
>> >>> >
>> >>> > 2011/4/12 茅旭峰 <m9suns@gmail.com>
>> >>> >
>> >>> >> We are using hadoop-CDH3B4 and hbase0.90.1-CDH3B4. I'll check
the
>> >>> >> issue further, but my understanding is the meta info and the
root
>> >>> >> region are saved by zookeeper, right? Do I need to check them
>> there?
>> >>> >>
>> >>> >> m9suns
>> >>> >>
>> >>> >> 在 2011-4-12,0:40,Jean-Daniel Cryans <jdcryans@apache.org>
写道:
>> >>> >>
>> >>> >> > It's possible under some bugs, which HBase version are
you using?
>> >>> >> >
>> >>> >> > J-D
>> >>> >> >
>> >>> >> > On Mon, Apr 11, 2011 at 4:50 AM, 茅旭峰 <m9suns@gmail.com>
wrote:
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> Is it possible that some table cannot cover the whole
key space.
>> >>> >> >> What I
>> >>> >> saw
>> >>> >> >> was like
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >> hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd',
'123'
>> >>> >> >>
>> >>> >> >> 0 row(s) in 0.3030 seconds
>> >>> >> >>
>> >>> >> >> hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >>> >> >> 'cfEStore:dasd', '123'
>> >>> >> >>
>> >>> >> >> ERROR: java.io.IOException: HRegionInfo was null or
empty in
>> .META.,
>> >>> >> >>
>> >>> >>
>> >>> >>
>> row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
>> >>> >> >>
>> >>> >>
>> >>> >>
>> table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}
>> >>> >> >>
>> >>> >> >> Here is some help for this command:
>> >>> >> >> Put a cell 'value' at specified table/row/column and
optionally
>> >>> >> >> timestamp coordinates.  To put a cell value into table
't1' at
>> >>> >> >> row 'r1' under column 'c1' marked with the time 'ts1',
do:
>> >>> >> >>
>> >>> >> >>  hbase> put 't1', 'r1', 'c1', 'value', ts1
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >>
>> >>> >> >> I guess this means our table has lost some key range
for the
>> >>> >> >> specific
>> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
>> >>> >> >> is this reasonable? I wonder what might cause this
issue? How
>> can I
>> >>> >> check
>> >>> >> >> it?
>> >>> >> >>
>> >>> >> >> One weird thing, I can see such lines like
>> >>> >> >>
>> >>> >> >> ====
>> >>> >> >>
>> >>> >>
>> >>> >>
>> table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
>> >>> >> >> LC3MILeAUy8HmRFgU5-ESE-9T7w= LD4jOJWFyt4m7A3KGFST6d-uj3A=
>> >>> >> >> ====
>> >>> >> >>
>> >>> >> >> from the web UI, which I think means the table has
holds the key
>> >>> >> >> range
>> >>> >> for
>> >>> >> >> key 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
>> >>> >> >>
>> >>> >> >> I also have not seen any errors/warnings in the log
file. Any
>> >>> >> suggestions?
>> >>> >> >>
>> >>> >> >> Thanks in advance.
>> >>> >> >>
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message