hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: disable table
Date Thu, 27 Sep 2012 16:24:41 GMT
I did restart entire cluster and still that didn't help. Looks like once I
get in this Race condition there is no way to come out of it?

On Thu, Sep 27, 2012 at 8:00 AM, rajesh babu chintaguntla <
chrajeshbabu32@gmail.com> wrote:

> Hi Mohit,
>
> We should not delete znode's manually which will cause inconsistencies like
> region may be shown as online on master, but it wont be on region server.
> That's put is failing in your case. Master restart will bring back your
> cluster to normal state(recovery any failures in enable/disable). Even hbck
> also wont solve this problem.
>
> FYI,
> Presently discussion is going on this issue. You can follow jira associated
> with this issue at
> https://issues.apache.org/jira/browse/HBASE-6469
>
>  On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >wrote:
>
> > Thanks everyone for the input, it's helpful. I did remove the znode from
> > /hbase/table/SESSIONID_TIMELINE and after that I was able to list the
> > table. At that point I tried to do a "put" but when I did a put I got a
> > message NoRegionServer online. I looked in the logs and it says the
> Failed
> > to open region server at nodexxx. When I went to nodexxx it complains
> > something about unable to run testcompression.
> >
> > I setup SNAPPY compression on my table and I also ran SNAPPY compression
> > test which was successful. Not sure what's going on in the cluster.
> > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <dontariq@gmail.com>
> > wrote:
> >
> > > Hello Mohit,
> > >
> > >     It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for
> the
> > > typo. For rest of the things, I feel Ramkrishna sir has provided a good
> > and
> > > proper explanation. Please let us know if you still have any doubt or
> > > question.
> > >
> > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share
> > > space with you people.
> > >
> > > Regards,
> > >     Mohammad Tariq
> > >
> > >
> > >
> > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
> > > ramkrishna.vasudevan@huawei.com> wrote:
> > >
> > > > Hi Mohith
> > > > First of all thanks to Tariq for his replies.
> > > >
> > > > Just to add on,
> > > > Basically HBase  uses the Zookeeper to know the status of the cluster
> > > like
> > > > the no of tables enabled, disabled and deleted.
> > > > Enabled and deleted states are handled bit different in the 0.94
> > version.
> > > >
> > > > ZK is used for various region assignments.
> > > >
> > > > Also the ZK is used to track the Active master and standby master.
> > > >
> > > > As you understand correctly that the master is responsible for the
> > > overall
> > > > maintenance of the no of tables and their respective states, it seeks
> > the
> > > > help of ZK to do it and that is where the states are persisted.
> > > >
> > > > Also there are few cases where the enable and disable table are
> having
> > > some
> > > > issues due to some race conditions in the 0.92 versions, In the
> latest
> > > > version we are trying to resolve them.
> > > > You can attach the master and RS logs to identify exactly what caused
> > > this
> > > > problem in your case which will be really help ful so that I can be
> > fixed
> > > > in
> > > > the kernel.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > > -----Original Message-----
> > > > > From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> > > > > Sent: Thursday, September 27, 2012 5:09 AM
> > > > > To: user@hbase.apache.org
> > > > > Subject: Re: disable table
> > > > >
> > >  > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> > > > > restart
> > > > > hbase and see if it works.
> > > > >
> > > > > One thing I don't understand is why is zookeeper holding
> information
> > > > > about
> > > > > this table if it is enabled or disabled? Wouldn't this information
> be
> > > > > with
> > > > > master?
> > > > >
> > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > > > > <mohitanchlia@gmail.com>wrote:
> > > > >
> > > > > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > > > > This is what I see
> > > > > >
> > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > > > > [SESSIONID_TIMELINE]
> > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > > > > >
> > > > > > cZxid = 0x1000000fe
> > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > mZxid = 0x1000000fe
> > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > pZxid = 0x5000008f1
> > > > > > cversion = 3
> > > > > > dataVersion = 0
> > > > > > aclVersion = 0
> > > > > > ephemeralOwner = 0x0
> > > > > > dataLength = 0
> > > > > > numChildren = 1
> > > > > >
> > > > > >  On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > > > > <dontariq@gmail.com>wrote:
> > > > > >
> > > > > >> In order to delete a znode you have to go to the ZK shell
and
> > issue
> > > > > the
> > > > > >> "delete" command along with the required path. For example
:
> > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you
can
> visit
> > > > > the ZK
> > > > > >> homepage at : zookeeper.apache.org
> > > > > >>
> > > > > >> Actually when we try to fetch data from an Hbase table,
the
> client
> > > > > or app
> > > > > >> first contacts the ZK to get the location of server holding
the
> > > > > >> -ROOT- table. From this we come to know about the server
hosting
> > the
> > > > > >> .META.
> > > > > >> table. This tells us the location of the server which actually
> > holds
> > > > > the
> > > > > >> rows of interest. Because of some reasons the znode which
was
> > > > > holding this
> > > > > >> info has either faced some catastrophe or lost the info
> associated
> > > > > with
> > > > > >> this particular table. Or sometimes the znode remains unable
to
> > keep
> > > > > >> itself
> > > > > >> updated with the latest changes. That could also be a probable
> > > > > reason. We
> > > > > >> should always keep in mind that ZK is the centralized service
> that
> > > > > >> actually
> > > > > >> coordinating everything behind the scene. As a result, any
> problem
> > > > > to the
> > > > > >> ZK quorum means problem with Hbase custer.
> > > > > >>
> > > > > >> Regards,
> > > > > >>     Mohammad Tariq
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > > > > <mohitanchlia@gmail.com
> > > > > >> >wrote:
> > > > > >>
> > > > > >> > Thanks! I do see Inconsistency. How do I remove the
znode. And
> > > > > also
> > > > > >> could
> > > > > >> > you please help me understand how this might have happened?
> > > > > >> >
> > > > > >> >
> > > > > >> > ERROR: Region
> > > > > >> >
> > > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050.
> > not
> > > > > >> > deployed on any region server.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > > > > <dontariq@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > A possible reason could be that the znode associated
with
> this
> > > > > >> particular
> > > > > >> > > table is not behaving properly. In such case,
you can try
> the
> > > > > >> following:
> > > > > >> > >
> > > > > >> > > Stop Hbase
> > > > > >> > > Stop ZK
> > > > > >> > > Take a backup of ZK data
> > > > > >> > > Restart ZK
> > > > > >> > > Remove the znode
> > > > > >> > > Start Hbase again
> > > > > >> > >
> > > > > >> > > After this hopefully your table would be enabled.
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > >     Mohammad Tariq
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > > > > <dontariq@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Yes. Also have a look at the logs of the
problematic
> region
> > if
> > > > > hbck
> > > > > >> > shows
> > > > > >> > > > any inconsistency.
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > >     Mohammad Tariq
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia
<
> > > > > >> mohitanchlia@gmail.com
> > > > > >> > > >wrote:
> > > > > >> > > >
> > > > > >> > > >> Which node should I look at for logs?
Is this the master
> > > > > node? I'll
> > > > > >> > try
> > > > > >> > > >> hbck.
> > > > > >> > > >>
> > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad
Tariq <
> > > > > >> dontariq@gmail.com>
> > > > > >> > > >> wrote:
> > > > > >> > > >>
> > > > > >> > > >> > Hello Mohit,
> > > > > >> > > >> >
> > > > > >> > > >> >     Try hbck once and see if it
shows any
> inconsistency.
> > > > > Also,
> > > > > >> you
> > > > > >> > can
> > > > > >> > > >> try
> > > > > >> > > >> > restarting your cluster and deleting
the table again.
> > > > > Having a
> > > > > >> look
> > > > > >> > at
> > > > > >> > > >> the
> > > > > >> > > >> > logs could also be useful.
> > > > > >> > > >> >
> > > > > >> > > >> > Regards,
> > > > > >> > > >> >     Mohammad Tariq
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM,
Mohit Anchlia <
> > > > > >> > > mohitanchlia@gmail.com
> > > > > >> > > >> > >wrote:
> > > > > >> > > >> >
> > > > > >> > > >> > > When I try to disable table
I get:
> > > > > >> > > >> > >
> > > > > >> > > >> > >  hbase(main):011:0> disable
'SESSIONID_TIMELINE'
> > > > > >> > > >> > > ERROR:
> > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > >> > SESSIONID_TIMELINE
> > > > > >> > > >> > > Here is some help for this
command:
> > > > > >> > > >> > > Start disable of named table:
e.g. "hbase> disable
> > 't1'"
> > > > > >> > > >> > >
> > > > > >> > > >> > > But then I try to enable I
get:
> > > > > >> > > >> > >
> > > > > >> > > >> > > hbase(main):012:0> enable
'SESSIONID_TIMELINE'
> > > > > >> > > >> > > ERROR:
> > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > >> > > SESSIONID_TIMELINE
> > > > > >> > > >> > > Here is some help for this
command:
> > > > > >> > > >> > > Start enable of named table:
e.g. "hbase> enable
> 't1'"
> > > > > >> > > >> > >
> > > > > >> > > >> > > I've tried flush, major_compaction
also. I tseems
> it's
> > > > > stuck in
> > > > > >> > > >> > > inconsistent state. Could someone
point me to correct
> > > > > >> direction? I
> > > > > >> > > am
> > > > > >> > > >> > using
> > > > > >> > > >> > > 92.1
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message