hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: Re: Re: Wrong Configuration lead to a failure when enabling table
Date Wed, 04 Feb 2015 08:40:37 GMT
Raised this https://issues.apache.org/jira/browse/HBASE-12967.

On Wed, Feb 4, 2015 at 1:36 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> distcp should work.  Try online snapshot case - I think it might work. But
> ideally here the table is in the DISABLING state. so it is again tricky.
>
> Take a raw back up of the table data in the HDFS -  and again put it back
> after restoring the table?
>
> On Wed, Feb 4, 2015 at 12:41 PM, Weichen YE <yeweichen2010@gmail.com>
> wrote:
>
>> Hi,Ted, Ram,
>>
>>      Thank you for your attemtion for this bug.
>>
>>      I meet this bug in production environment and the table contains
>> important data. If we are not able to enable this table in current
>> cluster,
>> do you have any idea to get the table data back in some other way? Maybe
>> export, snapshot, copytable, distcp all table files to another cluster ?
>>
>> 2015-02-04 13:17 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>
>> > Looks like the NPE was caused by the following method in
>> BaseLoadBalancer
>> > returning null:
>> >
>> >   protected Map<ServerName, List<HRegionInfo>> assignMasterRegions(
>> >
>> >       Collection<HRegionInfo> regions, List<ServerName> servers)
{
>> >
>> >     if (servers == null || regions == null || regions.isEmpty()) {
>> >
>> >       return null;
>> >
>> > Since bulkPlan is null, calling BulkAssigner seems unnecessary.
>> >
>> >
>> >
>> > On Tue, Feb 3, 2015 at 9:01 PM, ramkrishna vasudevan <
>> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >
>> >> It is not only about the state on the table descriptor but also the in
>> >> memory state in the AM.  I remember some time back Rajeshbabu worked
>> on a
>> >> HBCK like tool which will forcefully change the state of these tables
>> in
>> >> such cases. I don't remember the JIRA now.I thought of restarting the
>> >> master thinking the in memory state would change and I got this
>> >>
>> >> java.lang.NullPointerException
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:210)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:142)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingState(AssignmentManager.java:1695)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:416)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:720)
>> >>         at
>> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170)
>> >>         at
>> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459)
>> >>         at java.lang.Thread.run(Thread.java:745)
>> >> 2015-02-04 16:11:45,932 FATAL [stobdtserver3:16040.activeMasterManager]
>> >> master.HMaster: Master server abort: loaded coprocessors are:
>> >> [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
>> >> 2015-02-04 16:11:45,933 FATAL [stobdtserver3:16040.activeMasterManager]
>> >> master.HMaster: Unhandled exception. Starting shutdown.
>> >> java.lang.NullPointerException
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:210)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:142)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingState(AssignmentManager.java:1695)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:416)
>> >>         at
>> >>
>> >>
>> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:720)
>> >>         at
>> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170)
>> >>         at
>> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459)
>> >>
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Wed, Feb 4, 2015 at 10:25 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >>
>> >> > What about creating an offline tool which can modify the table
>> >> descriptor
>> >> > so that table goes to designated state ?
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Tue, Feb 3, 2015 at 8:51 PM, ramkrishna vasudevan <
>> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >
>> >> > > I tried reproducing this scenario on trunk. The same problem
>> exists.
>> >> > > Currently in the master the table state is noted in the Table
>> >> descriptor
>> >> > > and not on the ZK.  In 0.98.XX version it should be on the zk.
>> >> > >
>> >> > > When we tried to enable the table the region assignment failed
due
>> to
>> >> > > ClassNotFound and already the state is in ENABLING.  But doing
a
>> >> describe
>> >> > > table still shows it in DISABLED.
>> >> > >
>> >> > > Thought we could alter the correct Configuration but specifying
>> >> another
>> >> > > alter Table command we are still not able to enable the table.
>> >> > >
>> >> > > Moving this to dev to see if there is any workaround for this
>> issue.
>> >> If
>> >> > > not we may have to solve this issue across branches until we have
>> the
>> >> > > Procedure V2 implemenation ready on trunk.
>> >> > >
>> >> > > Any suggestions?
>> >> > >
>> >> > > Regards
>> >> > > Ram
>> >> > >
>> >> > > On Wed, Feb 4, 2015 at 4:05 AM, 叶炜晨 <yeweichen@qiyi.com>
wrote:
>> >> > >
>> >> > > >  my version is 0.98.6-cdh5.2.0, the problem in my production
>> >> > environment.
>> >> > > >
>> >> > > > So should I first delete znode? And then how to distable
this
>> >> table?my
>> >> > > > goal is to fix the wrong table configuration to get my data.
>> >> > > >
>> >> > > >
>> >> > > >  from my mobile phone.
>> >> > > >
>> >> > > >  在 2015-2-4 上午12:46,ramkrishna vasudevan <
>> >> > > ramkrishna.s.vasudevan@gmail.com
>> >> > > > >写道:
>> >> > > >
>> >> > > > >
>> >> > > > > I think the only way out here is to clear the zookeeper
node.
>> >> But am
>> >> > > > not sure on the ramifications of that.
>> >> > > > >
>> >> > > >
>> >> > > > > Which version are you using?  The newer versions are
>> >> 'protobuf'fed.
>> >> > > > >
>> >> > > >
>> >> > > > > Are you running this in production?
>> >> > > > >
>> >> > > >
>> >> > > > > Regards
>> >> > > > > Ram
>> >> > > > >
>> >> > > >
>> >> > > > > On Tue, Feb 3, 2015 at 5:00 PM,yeweichen2010@gmail.com<
>> >> > > > yeweichen2010@gmail.com>wrote:
>> >> > > >
>> >> > > > >>
>> >> > > > >> I tried HBCK, but it doesn`t help.
>> >> > > >
>> >> > > > I want to disable the table, so that I can use "alter" to
fix the
>> >> wrong
>> >> > > > configuration. But now the table keep in the status that
no
>> matter I
>> >> > use
>> >> > > > "is_enabled" or "is_disabled", it return false.
>> >> > > > >>
>> >> > > > >> ________________________________
>> >> > > > >> yeweichen2010@gmail.com
>> >> > > > >>>
>> >> > > > >>>
>> >> > > > >>> From: ramkrishna vasudevan
>> >> > > > >>> Date: 2015-02-03 19:55
>> >> > > > >>> To: user@hbase.apache.org
>> >> > > > >>> CC: yeweichen
>> >> > > > >>> Subject: Re: Wrong Configuration lead to a failure
when
>> enabling
>> >> > > table
>> >> > > > >>> Can you try HBCK? Did it help in anyway?  Remember
something
>> was
>> >> > done
>> >> > > > >>> related to failure in ENABLE/DISABLE table some
time back.
>> >> > > > >>>
>> >> > > > >>> Regards
>> >> > > > >>> Ram
>> >> > > > >>>
>> >> > > > >>> On Tue, Feb 3, 2015 at 3:38 PM,yeweichen2010@gmail.com<
>> >> > > > >>> yeweichen2010@gmail.com> wrote:
>> >> > > > >>>
>> >> > > > >>> > Hi, all,
>> >> > > > >>> >
>> >> > > > >>> > II did the following command in hbase shell:
>> >> > > > >>> >
>> >> > > > >>> > disable 'TestTable'
>> >> > > > >>> > alter 'TestTable', CONFIGURATION =>
>> >> > > > >>> > {'hbase.regionserver.region.split.policy'
=> 'xxxxxxxxx'}
>> >> > > > >>> > enable 'TestTable'
>> >> > > > >>> >
>> >> > > > >>> > At first I want to put
>> >> > > > >>> >
>> >> > >
>> "org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy"
>> >> > > > to the
>> >> > > > >>> > place "xxxxxxxxx", but because a spelling
error, now is
>> >> something
>> >> > > > wrong in
>> >> > > > >>> > this configuration. After I enable the
table, it failed
>> >> bacause
>> >> > of
>> >> > > > >>> > ClassNotFound.
>> >> > > > >>> >
>> >> > > > >>> > Now is the problem: the table failed to
enable and stay in
>> a
>> >> > middle
>> >> > > > >>> > status. The table is neither enabled nor
disabled now. How
>> >> can I
>> >> > > > save my
>> >> > > > >>> > table and fix the wrong configuration?
>> >> > > > >>> >
>> >> > > > >>> >
>> >> > > > >>> >
>> >> > > > >>> >
>> >> > > > >>> >
>> >> > > > >>> >yeweichen2010@gmail.com
>> >> > > > >>> >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message