Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of kevin.odell@cloudera.com
 designates 209.85.216.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPQV63V9kWKjsk3t0sL+bvhEmsBFMRG+rJqk9o3uuRy1i2KsPg@mail.gmail.com>
References: 
 <95B9CD55CC75E24AAE1CBC366CB9C3802BA3D61D@NDHEP50004.na.corp.mckesson.com>
	<CAGngS9f4yVvV5CA8vKV_UnNo13pW_cKhWDvxHhsHECB04j9AOQ@mail.gmail.com>
	<CAPQV63V9kWKjsk3t0sL+bvhEmsBFMRG+rJqk9o3uuRy1i2KsPg@mail.gmail.com>
Date: Tue, 10 Dec 2013 13:45:52 -0500
Message-ID: 
 <CAGngS9eBPW8qEcV89_p-FuVb0F11PmMmiZLsRXoea1vY=C23OQ@mail.gmail.com>
Subject: Re: Table state
From: "Kevin O'dell" <kevin.odell@cloudera.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2bd389afb7d04ed328510

--001a11c2bd389afb7d04ed328510
Content-Type: text/plain; charset=ISO-8859-1

Just to close the loop, the previous recommended steps help to get us back
up, but one of the HMasters is not happy now.  I will update with a final
analysis shortly.


On Tue, Dec 10, 2013 at 1:10 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Also, might be interesting to look in the RS logs to see what this region
> can not come back online...
>
> JM
>
>
> 2013/12/10 Kevin O'dell <kevin.odell@cloudera.com>
>
> > Hey Raheem,
> >
> >   You can sideline the table into tmp(mv /hbase/table /tmp/table, then
> > bring HBase back online.  Once HBase is back you can use HBCK to repair
> > your META -fixMeta -fixAssignments.  Once HBase is consistent again, you
> > can move the table back out of tmp and use HBCK to reupdate META.  If the
> > issue reoccurs let us know.
> >
> >
> > On Tue, Dec 10, 2013 at 11:50 AM, Daya, Raheem
> > <Raheem.Daya@relayhealth.com>wrote:
> >
> > > I have a distributed Hbase cluster that will not start.  It looks like
> > > there is a table that is an inconsistent state:
> > > 2013-12-10 07:40:50,447 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > Unexpected state :
> > >
> >
> ct_claims,204845|81V6SO4EF56DD1TKOIU7AS4L5D,1386050670937.6d138b97cde8bc3e49ff34639913109c.
> > > state=PENDING_OPEN, ts=1386690050445,
> server=rhf-045,60020,1386689069486
> > ..
> > > Cannot transit it to OFFLINE.
> > >
> > > Is there a way to manually set the table to OFFLINE?  I have tried
> > > deleting the /hbase node in zookeeper.  I tried bringing up the master
> > and
> > > then a region server and vice versa.  In the case of bringing the
> master
> > up
> > > first, the master starts.  As soon as I bring up a region server the
> > master
> > > goes down.  My thought is to move the tables to OFFLINE, (assuming it
> is
> > > possible), and try bringing up the cluster again.  hbck will not work
> as
> > > none of the region servers are up.  Any one have any other ideas?
> > > Thanks,
> > > Raheem
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Kevin O'Dell
> > Systems Engineer, Cloudera
> >
>


-- 
Kevin O'Dell
Systems Engineer, Cloudera

--001a11c2bd389afb7d04ed328510--