hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mukund murrali <mukundmurra...@gmail.com>
Subject Re: New Region Server failing to open region and subsequent cluster hang
Date Fri, 21 Aug 2015 11:52:57 GMT
Sorry guys, the new regionservers added, pointed to some other hadoop
cluster. And the hbase:meta got registered with new node. And subsequently
all region opening failed as the meta searched in other dfs cluster. Once
we removed the node, the meta got registered to the existing RS which
pointed to correct hadoop cluster and started working fine.

On Thu, Aug 20, 2015 at 9:43 PM, mukund murrali <mukundmurrali9@gmail.com>
wrote:

> Hi Ted
>
> No didn't run hbck. Once we removed the newly added node things became
> fine. The same table was compacted. The above failed open logs happened in
> new region server only. Existing region servers were fine.
>
> Thanks
>
>
> On Thursday 20 August 2015, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Have you run hbck on testnamespace:testtable ?
>>
>> bq. major compaction was running for a table.
>>
>> Which table was being compacted ?
>>
>> Is it possible for you to share the region server log of the new server ?
>>
>> Cheers
>>
>> On Thu, Aug 20, 2015 at 4:58 AM, mukund murrali <mukundmurrali9@gmail.com
>> >
>> wrote:
>>
>> > Hi
>> >
>> > Recently we tried adding a new regionserver to the cluster. The region
>> > server had the following logs for all the regions it tried to open.
>> >
>> > 2015-08-14 01:23:52,175 ERROR [RS_OPEN_REGION-localhost:16020-0]
>> > handler.OpenRegionHandler: Failed open of
>> >
>> >
>> region=testnamespace:testtable,test_14390754745,1439165529250.7aaa895f2ec3ca602c50fcc69120c01e.,
>> > starting to roll back the global memstore size.
>> > java.lang.IllegalStateException: Could not instantiate a region
>> instance.
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4566)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4873)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
>> >         at
>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >         at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >         at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.lang.reflect.InvocationTargetException
>> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> >         at
>> >
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> >         at
>> >
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >         at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4563)
>> >         ... 10 more
>> > Caused by: java.lang.IllegalArgumentException: Need table descriptor
>> >         at
>> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:601)
>> >         at
>> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:578)
>> >         ... 15 more
>> >
>> >
>> > In addition to this, all other region server began to stop serving
>> request.
>> >
>> > One thing to note is during this time a major compaction was running
>> for a
>> > table.
>> >
>> > Also the hbase:meta table was in the new rs.
>> >
>> > The clients had the exception
>> >
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
>> > location
>> >
>> >
>> > and jstack of clients were waiting at locateRegionInMeta.
>> >
>> > On decommissioning the new node, the existing regionserver started
>> serving
>> > request. What would have caused this problem?  We are using hbase-1.0.0.
>> >
>> > Thanks
>> >
>> > Regards
>> > Mukund Murrali
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message