Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALte62wPtRifh_BnOsoszZxVO+BNjSZX3fVrUiJvUNeN58Hf=Q@mail.gmail.com>
References: 
 <CABYEoojP_KVHeMBzOhyD42+YgtRZaaOvO_+n0gbFjODAbzm7xQ@mail.gmail.com>
	<CALte62xTwuuXByTDy7F8=A2jYDCmVE4_SG=2PgKGTuAdQ8-L+Q@mail.gmail.com>
	<CABYEoojvRMt8x0VDfp_FkdNQY92wxw=BULTERV4hBUfa5jQLbQ@mail.gmail.com>
	<CALte62wPtRifh_BnOsoszZxVO+BNjSZX3fVrUiJvUNeN58Hf=Q@mail.gmail.com>
Date: Wed, 2 Dec 2015 13:03:21 -0800
Message-ID: 
 <CABYEoogPn+hpKGrdCbShcyu_7mrW6AM0UERwW=ZsocpZieAd7Q@mail.gmail.com>
Subject: Re: Issues with HBase Regions in Transition
From: Amanda Moran <amanda.moran@esgyn.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a113cd2b0b550370525f09a51

--001a113cd2b0b550370525f09a51
Content-Type: text/plain; charset=UTF-8

Hi there Ted-

Thanks for the info. Very interesting...I will be trying this for sure.

Thanks alot, great information.

Amanda

On Wed, Dec 2, 2015 at 12:57 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> From log snippet:
>
> Class
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not found
>
> I assume the above class is in the trafodion jar (just a guess)
> This can explain why regions would be stuck in transition.
>
> For upgrade, consider replacing the trafodion jar instead of removing it.
>
> Cheers
>
> On Wed, Dec 2, 2015 at 12:51 PM, Amanda Moran <amanda.moran@esgyn.com>
> wrote:
>
> > Hi there Ted-
> >
> > Thanks for response. Update: I copied my trafodion.jar files back to
> > ../hbase/lib , did a restart and then hbase is back online... I then
> > removed them again (just to make sure) did a restart .. and then its back
> > in the region in transition. Seems like we should be able to remove our
> > files and do a restart and have hbase be back online... Is this
> incorrect?
> >
> > Log snippet:
> >
> > 2015-12-02 19:48:41,614 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open
> > of region=hbase:meta,,1.1588230740, starting to roll back the global
> > memstore size.
> > java.lang.IllegalStateException: Could not instantiate a region instance.
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
> >         at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >         at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > Class
> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not
> > found
> >         at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
> >         ... 10 more
> > Caused by: java.lang.ClassNotFoundException: Class
> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not
> > found
> >         at
> >
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
> >         at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
> >         ... 11 more
> > 2015-12-02 19:48:41,627 INFO
> > org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
> > region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '',
> > ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
> > expecting version 4
> > 2015-12-02 19:48:41,673 INFO
> > org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
> > hbase:meta,,1.1588230740
> > 2015-12-02 19:48:41,679 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open
> > of region=hbase:meta,,1.1588230740, starting to roll back the global
> > memstore size.
> > java.lang.IllegalStateException: Could not instantiate a region instance.
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
> >         at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >         at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > Class
> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not
> > found
> >         at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
> >         ... 10 more
> > Caused by: java.lang.ClassNotFoundException: Class
> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not
> > found
> >         at
> >
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
> >         at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
> >         ... 11 more
> >
> >
> > Let me know if you need more from the logs.
> >
> > THANKS!
> >
> > On Wed, Dec 2, 2015 at 12:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Do you mind pastebin snippet of region server log where the region
> stuck
> > in
> > > transition was hosted ?
> > >
> > > This would give us some clue.
> > >
> > > Cheers
> > >
> > > On Wed, Dec 2, 2015 at 12:14 PM, Amanda Moran <amanda.moran@esgyn.com>
> > > wrote:
> > >
> > > > Hi there All-
> > > >
> > > > I apologize if this issue has been raised before... I have done ALOT
> of
> > > > googling on this issues... and finally decided to just write here to
> > the
> > > > source :)
> > > >
> > > > Background:
> > > >
> > > > I work on the incubating Apache Trafodion project.
> > > >
> > > > I am currently using HBase 1.0 with Cloudera 5.4.5.
> > > >
> > > > I have tried many times to upgrade my cloudera cluster (to get the
> > latest
> > > > version of HBase) and have always failed due to my HBase master
> failing
> > > to
> > > > come backup do to a Region suck in Transition. It is always the
> > > hbase:meta.
> > > >
> > > > In the past I just assumed that the upgrade feature just didn't
> > actually
> > > > work... and so even though frustrating... I just let it go.
> > > >
> > > > But now I have come across this issue again, in a slightly different
> > > way...
> > > > and this makes me think it is an HBase problem.
> > > >
> > > > I am preparing for an upgrade, and because of this I decided to
> remove
> > > the
> > > > "Trafodion".jar files that are located in the ../hbase/lib
> directory. I
> > > > thought this might be the cause of not being able to upgrade,
> because I
> > > was
> > > > leaving the trafodion files in this directory. I removed them, then
> > did a
> > > > deploy client config (since some configs were stale), and then did a
> > > > restart. Now I have a region stuck in transition, even before the
> > > upgrade!
> > > >
> > > > Everything I have seen on the web, has not worked in the past. How
> do I
> > > get
> > > > pass this issue? Do I have no choice but to do a full
> > uninstall/reinstall
> > > > and lose my data?
> > > >
> > > > Thanks all for any help!
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > >
> > > > Amanda Moran
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> >
> > Amanda Moran
> >
>


-- 
Thanks,

Amanda Moran

--001a113cd2b0b550370525f09a51--