Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of
 ramkrishna.vasudevan@huawei.com designates 119.145.14.65 as permitted sender)
From: "Ramkrishna.S.Vasudevan" <ramkrishna.vasudevan@huawei.com>
To: <user@hbase.apache.org>
References: 
 <CAOT3TWom=AsCtPLCVtiM3=on9SzuEtEL0G5g6ZY6o3bOCrXvGQ@mail.gmail.com>
	<CAMVC6RMUdrBrqitHcUu-Ld_FdqFdg9tFZ1ta=7w_JDGQ6zxrRA@mail.gmail.com>
	<CAOT3TWrCtbwryr6vM96EvnirB3dkYN57U=q=wuugTVsBZhksJQ@mail.gmail.com>
	<CAMVC6RP+-cwJV13=XyXm5UcMS7F9KVWZY9MSEOQHRbHEYSL_SA@mail.gmail.com>
	<CAMVC6RMsWQSfJDaQVeQhdkRoWyjJ4wdaPY=1Nc8+P9LHq+hkig@mail.gmail.com>
	<CAOT3TWqVj4W1mMXnCwjppT9Zsr2XtDvhyJ-fvJL9048eXLj2Yg@mail.gmail.com>
	<CAMVC6RPsc6+aJ37bSU5FRGZ+6SbqrFt6-MO1NpwqmTF7DH_MHQ@mail.gmail.com>
	<CAOT3TWqfjv2AfC9RsGtGXrKJ6uMRsfV=tXAmH7cdaHourA+a6g@mail.gmail.com>
 <CAOT3TWoyUot1_Us0S9jDLv1hS50wdGytmFzmk6z2s--tO=v51g@mail.gmail.com>
In-Reply-To: 
 <CAOT3TWoyUot1_Us0S9jDLv1hS50wdGytmFzmk6z2s--tO=v51g@mail.gmail.com>
Subject: RE: disable table
Date: Thu, 27 Sep 2012 09:59:21 +0530
Message-ID: <000001cd9c68$ab5a0df0$020e29d0$@vasudevan@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: Ac2cQEAMOSc5PpAiRVOA858hwXayDAAJ6ZMw
Content-Language: en-us

Hi Mohith
First of all thanks to Tariq for his replies.

Just to add on,
Basically HBase  uses the Zookeeper to know the status of the cluster like
the no of tables enabled, disabled and deleted.
Enabled and deleted states are handled bit different in the 0.94 version.

ZK is used for various region assignments.

Also the ZK is used to track the Active master and standby master.

As you understand correctly that the master is responsible for the overall
maintenance of the no of tables and their respective states, it seeks the
help of ZK to do it and that is where the states are persisted.

Also there are few cases where the enable and disable table are having some
issues due to some race conditions in the 0.92 versions, In the latest
version we are trying to resolve them.
You can attach the master and RS logs to identify exactly what caused this
problem in your case which will be really help ful so that I can be fixed in
the kernel.

Regards
Ram

> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Thursday, September 27, 2012 5:09 AM
> To: user@hbase.apache.org
> Subject: Re: disable table
> 
> I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> restart
> hbase and see if it works.
> 
> One thing I don't understand is why is zookeeper holding information
> about
> this table if it is enabled or disabled? Wouldn't this information be
> with
> master?
> 
> On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> <mohitanchlia@gmail.com>wrote:
> 
> > I don't see path like /hbase/SESSIONID_TIMELINE
> > This is what I see
> >
> > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > [SESSIONID_TIMELINE]
> > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> >
> > cZxid = 0x1000000fe
> > ctime = Mon Sep 10 15:31:45 PDT 2012
> > mZxid = 0x1000000fe
> > mtime = Mon Sep 10 15:31:45 PDT 2012
> > pZxid = 0x5000008f1
> > cversion = 3
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x0
> > dataLength = 0
> > numChildren = 1
> >
> >  On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> <dontariq@gmail.com>wrote:
> >
> >> In order to delete a znode you have to go to the ZK shell and issue
> the
> >> "delete" command along with the required path. For example :
> >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit
> the ZK
> >> homepage at : zookeeper.apache.org
> >>
> >> Actually when we try to fetch data from an Hbase table, the client
> or app
> >> first contacts the ZK to get the location of server holding the
> >> -ROOT- table. From this we come to know about the server hosting the
> >> .META.
> >> table. This tells us the location of the server which actually holds
> the
> >> rows of interest. Because of some reasons the znode which was
> holding this
> >> info has either faced some catastrophe or lost the info associated
> with
> >> this particular table. Or sometimes the znode remains unable to keep
> >> itself
> >> updated with the latest changes. That could also be a probable
> reason. We
> >> should always keep in mind that ZK is the centralized service that
> >> actually
> >> coordinating everything behind the scene. As a result, any problem
> to the
> >> ZK quorum means problem with Hbase custer.
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >>
> >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> <mohitanchlia@gmail.com
> >> >wrote:
> >>
> >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> also
> >> could
> >> > you please help me understand how this might have happened?
> >> >
> >> >
> >> > ERROR: Region
> >> >
> SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> >> > deployed on any region server.
> >> >
> >> >
> >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> <dontariq@gmail.com>
> >> > wrote:
> >> >
> >> > > A possible reason could be that the znode associated with this
> >> particular
> >> > > table is not behaving properly. In such case, you can try the
> >> following:
> >> > >
> >> > > Stop Hbase
> >> > > Stop ZK
> >> > > Take a backup of ZK data
> >> > > Restart ZK
> >> > > Remove the znode
> >> > > Start Hbase again
> >> > >
> >> > > After this hopefully your table would be enabled.
> >> > >
> >> > > Regards,
> >> > >     Mohammad Tariq
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> <dontariq@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Yes. Also have a look at the logs of the problematic region if
> hbck
> >> > shows
> >> > > > any inconsistency.
> >> > > >
> >> > > > Regards,
> >> > > >     Mohammad Tariq
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> >> mohitanchlia@gmail.com
> >> > > >wrote:
> >> > > >
> >> > > >> Which node should I look at for logs? Is this the master
> node? I'll
> >> > try
> >> > > >> hbck.
> >> > > >>
> >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> >> dontariq@gmail.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Hello Mohit,
> >> > > >> >
> >> > > >> >     Try hbck once and see if it shows any inconsistency.
> Also,
> >> you
> >> > can
> >> > > >> try
> >> > > >> > restarting your cluster and deleting the table again.
> Having a
> >> look
> >> > at
> >> > > >> the
> >> > > >> > logs could also be useful.
> >> > > >> >
> >> > > >> > Regards,
> >> > > >> >     Mohammad Tariq
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> >> > > mohitanchlia@gmail.com
> >> > > >> > >wrote:
> >> > > >> >
> >> > > >> > > When I try to disable table I get:
> >> > > >> > >
> >> > > >> > >  hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> >> > SESSIONID_TIMELINE
> >> > > >> > > Here is some help for this command:
> >> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> >> > > >> > >
> >> > > >> > > But then I try to enable I get:
> >> > > >> > >
> >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> >> > > SESSIONID_TIMELINE
> >> > > >> > > Here is some help for this command:
> >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> >> > > >> > >
> >> > > >> > > I've tried flush, major_compaction also. I tseems it's
> stuck in
> >> > > >> > > inconsistent state. Could someone point me to correct
> >> direction? I
> >> > > am
> >> > > >> > using
> >> > > >> > > 92.1
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >