hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mike anderson <saidthero...@gmail.com>
Subject Re: hbase crashed, table missing
Date Thu, 03 Dec 2009 20:36:28 GMT
This was a table that had been around for almost two months now and had many
regions. The web UI reports 231 regions, and I am certain that the tables
being reported don't have nearly that many regions, so perhaps this count
includes those from the missing table.

In the folder: /hbase/cached_web_pages/1102708773/http is a single 130MB
file full of rows/columns. We are caching the full html of websites into the
columns so copying and pasting some of the rows won't be very useful, but
the chunk starts with this:

"DATABLK*f#ŸRhttp%3A%2F%2Fwww.informaworld.com
%2Fsmpp%2Ftitle%7Edb%3Dall%7Econtent%3Dg903750466httpdata$íó"

I tried to enable a region, but get:

from (hbase):3hbase(main):003:0> enable_region
'cached_web_pages,metapress_ris_120417,1257429337740'
NativeException: java.lang.NullPointerException: null
from org/apache/hadoop/hbase/util/Writables.java:74:in `getWritable'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:278:in `invoke_static'
from org/jruby/java/invokers/StaticMethodInvoker.java:57:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:150:in `call'
from org/jruby/ast/CallTwoArgNode.java:59:in `interpret'
from org/jruby/ast/LocalAsgnNode.java:123:in `interpret'
from org/jruby/ast/NewlineNode.java:104:in `interpret'
from org/jruby/ast/BlockNode.java:71:in `interpret'
from org/jruby/internal/runtime/methods/InterpretedMethod.java:201:in `call'
from org/jruby/internal/runtime/methods/DefaultMethod.java:162:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:150:in `call'
... 112 levels...
from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:253:in `cacheAndCall'
from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
from usr/local/hbase/bin/$_dot_dot_/bin/hirb.rb:487:in `__file__'
from usr/local/hbase/bin/$_dot_dot_/bin/hirb.rb:-1:in `load'
from org/jruby/Ruby.java:577:in `runScript'
from org/jruby/Ruby.java:480:in `runNormally'
from org/jruby/Ruby.java:354:in `runFromMain'
from org/jruby/Main.java:229:in `run'
from org/jruby/Main.java:110:in `run'
from org/jruby/Main.java:94:in `main'
from /usr/local/hbase/bin/../bin/HBase.rb:138:in `enable_region'
from /usr/local/hbase/bin/../bin/hirb.rb:350:in `enable_region'
from (hbase):4hbase(main):004:0>

Thanks again.

-mike

On Thu, Dec 3, 2009 at 3:21 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> What's in the HDFS folder of that table? Here I see that you should
> have something like:
>
> /hbase/cached_web_pages/1325672518/http/  stuff...
>
> Was there only this one region?
>
> Also are you able to enable a region in the shell? Take one of the row
> key from .META. and do
>
> > enable_region 'region name'
>
> J-D
>
> On Thu, Dec 3, 2009 at 12:11 PM, mike anderson <saidtherobot@gmail.com>
> wrote:
> > Here's a snippit from the meta table (I can send you the whole thing, but
> > it's quite large),
> >
> > cached_web_pages,http%3A%2F column=info:serverstartcode,
> > timestamp=1259853027975, value=1259852967063
> >  %2Fdx.doi.org%2F10.1002%252
> >
> >  Fajpa.21214,1259739437144
> >
> >  cached_web_pages,http%3A%2F column=historian:assignment,
> > timestamp=1259807436758, value=Region assigned to se
> >  %2Fdx.doi.org%2F10.1002%252 rver
> > ghetto169.projectlounge.com,60020,1256139356112
> >
> >  Fejoc.200900768,12555040994
> >
> >  35
> >
> >  cached_web_pages,http%3A%2F column=historian:open,
> timestamp=1259807436723,
> > value=Region opened on server : g
> >  %2Fdx.doi.org%2F10.1002%252 hetto169.projectlounge.com
> >
> >  Fejoc.200900768,12555040994
> >
> >  35
> >
> >  cached_web_pages,http%3A%2F column=historian:assignment,
> > timestamp=1259853024917, value=Region assigned to se
> >  %2Fdx.doi.org%2F10.1002%252 rver
> > ghetto167.projectlounge.com,60020,1259852967063
> >
> >  Fsmi.1285,1258589376676
> >
> >  cached_web_pages,http%3A%2F column=historian:open,
> timestamp=1259853027984,
> > value=Region opened on server : g
> >  %2Fdx.doi.org%2F10.1002%252 hetto167.projectlounge.com
> >
> >  Fsmi.1285,1258589376676
> >
> >  cached_web_pages,http%3A%2F column=info:regioninfo,
> > timestamp=1258589203875, value=REGION => {NAME => 'cached
> >  %2Fdx.doi.org%2F10.1002%252 _web_pages,http\\x253A\\x252F\\
> x252Fdx.doi.org
> > \\x252F10.1002\\x25252Fsmi.1285,125
> >  Fsmi.1285,1258589376676     8589376676', STARTKEY =>
> 'http\\x253A\\x252F\\
> > x252Fdx.doi.org\\x252F10.1002\\x252
> >                             52Fsmi.1285', ENDKEY => 'http\\x253A\\x252F\\
> > x252Fdx.doi.org\\x252F10.1016\\x252F
> >                             j.apergo.2009.09.005', ENCODED => 1325672518,
> > TABLE => {{NAME => 'cached_web_page
> >                             s', FAMILIES => [{NAME => 'http', VERSIONS =>
> > '1', COMPRESSION => 'NONE', TTL =>
> >                             '2147483647', BLOCKSIZE => '65536', IN_MEMORY
> > => 'false', BLOCKCACHE => 'true'}]}
> >                             }
> >
> >
> > and you can see the table which has gone missing 'cached_web_pages' in
> the
> > key spot. The crash over the weekend was pretty traumatic. Complete power
> > outage to the entire cluster except(!) for the master.  The data is
> > definitely still on HDFS, I will take a look at the add_table script and
> > upgrade to 0.20.2.
> >
> >
> > Cheers and thanks a lot.
> >
> > mike
> >
> >
> > On Thu, Dec 3, 2009 at 2:51 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> This is weird if the table is in .META. and still not showing up...
> >> could you pastebin the .META. rows?
> >>
> >> Also was it a new table that was just created or has it been there for
> >> some time?
> >>
> >> What kind of crash did you get this weekend?
> >>
> >> The best way to recover your data, if it's still on HDFS, will be to
> >> upgrade to 0.20.2 and use the script bin/add_table.rb to rebuild
> >> .META.
> >>
> >> J-D
> >>
> >> On Thu, Dec 3, 2009 at 11:29 AM, mike anderson <saidtherobot@gmail.com>
> >> wrote:
> >> > From the web UI and from calling 'list' in the shell I can't see the
> >> table
> >> > name.
> >> >
> >> > Hadoop/Hbase 0.20/0.20.1, distributed setup, 10 nodes.
> >> >
> >> > -mike
> >> >
> >> > On Thu, Dec 3, 2009 at 1:54 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> Mike,
> >> >>
> >> >> So if you looked in .META. and the rows are there, how did you figure
> >> >> that the table is missing?
> >> >>
> >> >> Also the usuals: which version of Hadoop/HBase, what kind of setup,
> etc
> >> >>
> >> >> J-D
> >> >>
> >> >> On Thu, Dec 3, 2009 at 7:29 AM, mike anderson <
> saidtherobot@gmail.com>
> >> >> wrote:
> >> >> > Hbase crashed on me this weekend, and upon restarting one of the
> >> tables
> >> >> is
> >> >> > just completely gone. All of the table data is still in HDFS and
my
> >> >> missing
> >> >> > table is still mentioned in .META.. I tried restarting hbase a
few
> >> times,
> >> >> > but the table didn't show up. What else can I do to debug this?
I
> >> looked
> >> >> > through the logs, but nothing really jumped out at me. Is there
> >> something
> >> >> I
> >> >> > should look for?
> >> >> >
> >> >> > I took a look at this ticket,
> >> >> > http://issues.apache.org/jira/browse/HBASE-1342, but don't know
> >> enough
> >> >> about
> >> >> > the inner workings of hbase to make sense of it.
> >> >> >
> >> >> >
> >> >> > thanks in advance.
> >> >> >
> >> >>
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message