hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane O'Donnell" <sha...@knownormal.com>
Subject Re: Hbase master not starting
Date Mon, 04 Jan 2016 17:28:45 GMT
And the error was (no surprise) not finding 'namespace':

2016-01-04 16:28:45,323 INFO
[PriorityRpcServer.handler=7,queue=1,port=60020]
regionserver.RSRpcServices: Open
hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.

2016-01-04 16:28:45,326 ERROR [RS_OPEN_REGION-ip-10-0-1-151:60020-1]
handler.OpenRegionHandler: Failed open of
region=hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.,
starting to roll back the global memstore size.

java.lang.IllegalStateException: Could not instantiate a region instance.

        at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5666)

        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5973)

        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5945)

        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5901)

        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5852)

        at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)

        at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)

        at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc

========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9de88312-9164-4f22-c6ca-b4c867938fc3>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!

On Mon, Jan 4, 2016 at 12:26 PM, Shane O'Donnell <shaneo@knownormal.com>
wrote:

> I looked for region 82432aca9ede964943b40753cb64e808 on each of my region
> servers and none of them had it.  They all had identical failures in the
> log of attempting to open it, but none had it (or at least were successful
> in opening it).
>
> My solution to "finding it" was grepping for
> "82432aca9ede964943b40753cb64e808" in the logs.  If there's a better or
> more reliable way of searching for this, let me know.
>
> Thanks -
>
> Shane O.
>
> ========================
> Shane O'Donnell
>
> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=27fa6676-ade7-41f1-f304-76654be07bdd>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 12:03 PM, Shane O'Donnell <shaneo@knownormal.com>
> wrote:
>
>> It's not there.  The directory listing was for the right directory, but
>> the namespace directory is not there.
>>
>> Here it is one from directory level up:
>>
>> [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
>> -R /hbase/data
>>
>> drwxr-xr-x   - hdfs hadoop          0 2016-01-04 14:47 /hbase/data/hbase
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> /hbase/data/hbase/meta
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tabledesc
>>
>> -rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tmp
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740
>>
>> -rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/.regioninfo
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
>> /hbase/data/hbase/meta/1588230740/info
>>
>> drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/recovered.edits
>>
>> -rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>>
>> ========================
>> Shane O'Donnell
>>
>> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
>> 4819 Emperor Blvd., Ste 400
>> Durham, North Carolina 27703
>> tel: +1.424.262.KNOW x703
>> skype: shaneodonnell
>> email: shaneo@knownormal.com
>> ========================
>> :wq!
>>
>> On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> In your listing, meta table directory structure was shown.
>>>
>>> Please look under /hbase/data/hbase for namespace table.
>>>
>>> Cheers
>>>
>>> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <shaneo@knownormal.com>
>>> wrote:
>>>
>>> > I attempted to restore a backed-up "meta" directory from a copy made by
>>> > 'hbase hbck', so my directory structure may be messed up:
>>> >
>>> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs
>>> -ls -R
>>> > /hbase/data/hbase
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tabledesc
>>> >
>>> > -rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tmp
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740
>>> >
>>> > -rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/.regioninfo
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
>>> > /hbase/data/hbase/meta/1588230740/info
>>> >
>>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/recovered.edits
>>> >
>>> > -rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>>> >
>>> > And I'm not seeing any "namespace" directory.
>>> >
>>> > Will check out the server now...
>>> >
>>> > Shane O.
>>> >
>>> > ========================
>>> > Shane O'Donnell
>>> > <
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
>>> > >
>>> > 4819 Emperor Blvd., Ste 400
>>> > Durham, North Carolina 27703
>>> > tel: +1.424.262.KNOW x703
>>> > skype: shaneodonnell
>>> > email: shaneo@knownormal.com
>>> > ========================
>>> > :wq!
>>> >
>>> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> >
>>> > > Can you log onto the server hosting region
>>> > 82432aca9ede964943b40753cb64e808
>>> > > and see what happened ?
>>> > >
>>> > > See if the namespace table can be found under rootdir.
>>> > > e.g. assuming /apps/hbase/data is the rootdir, you should see
>>> something
>>> > > similar to the following:
>>> > >
>>> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
>>> > > Found 3 items
>>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
>>> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
>>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
>>> > > /apps/hbase/data/data/hbase/namespace/.tmp
>>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:51
>>> > >
>>> /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>>> > >
>>> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <
>>> shaneo@knownormal.com>
>>> > > wrote:
>>> > >
>>> > > > Some progress...
>>> > > >
>>> > > > /hbase did NOT have either the hbase.id or hbase.version files
so
>>> I
>>> > > > temporarily changed hbase.rootdir and started the master so the
>>> files
>>> > > would
>>> > > > be recreated elsewhere and copied them in.
>>> > > >
>>> > > > Now it starts fine, but my hbase tables are gone.  Specifically,
>>> I'm
>>> > > > getting this error:
>>> > > >
>>> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
>>> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
>>> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>>> > > >
>>> > > > Is my existing data toast, or is there a crafty way out of this?
>>> > > >
>>> > > > Thanks -
>>> > > >
>>> > > > Shane O.
>>> > > >
>>> > > > ========================
>>> > > > Shane O'Donnell
>>> > > > <
>>> > > >
>>> > >
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
>>> > > > >
>>> > > > 4819 Emperor Blvd., Ste 400
>>> > > > Durham, North Carolina 27703
>>> > > > tel: +1.424.262.KNOW x703
>>> > > > skype: shaneodonnell
>>> > > > email: shaneo@knownormal.com
>>> > > > ========================
>>> > > > :wq!
>>> > > >
>>> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
>>> > shaneo@knownormal.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi -
>>> > > > >
>>> > > > > My cluster has been running perfectly until the other day
when I
>>> > found
>>> > > it
>>> > > > > down.
>>> > > > >
>>> > > > > The error seems to be related not being able to get the ClusterID
>>> > from
>>> > > > > zookeeper, but I'm stumped as to what to do about it.  This
>>> seems to
>>> > be
>>> > > > the
>>> > > > > relevant part of the master's log:
>>> > > > >
>>> > > > >      http://pastebin.com/C3iaxM3p
>>> > > > >
>>> > > > > starting at line 235 (also highlighted).
>>> > > > >
>>> > > > > Any help is appreciated!
>>> > > > >
>>> > > > > Thanks -
>>> > > > >
>>> > > > > Shane O.
>>> > > > > ========================
>>> > > > > Shane O'Donnell
>>> > > > >
>>> > > > > <
>>> > > >
>>> > >
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
>>> > > > >
>>> > > > > 4819 Emperor Blvd., Ste 400
>>> > > > > Durham, North Carolina 27703
>>> > > > > tel: +1.424.262.KNOW x703
>>> > > > > skype: shaneodonnell
>>> > > > > email: shaneo@knownormal.com
>>> > > > > ========================
>>> > > > > :wq!
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message