hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: Re: Re: Wrong Configuration lead to a failure when enabling table
Date Wed, 04 Feb 2015 08:06:48 GMT
distcp should work.  Try online snapshot case - I think it might work. But
ideally here the table is in the DISABLING state. so it is again tricky.

Take a raw back up of the table data in the HDFS -  and again put it back
after restoring the table?

On Wed, Feb 4, 2015 at 12:41 PM, Weichen YE <yeweichen2010@gmail.com> wrote:

> Hi,Ted, Ram,
>
>      Thank you for your attemtion for this bug.
>
>      I meet this bug in production environment and the table contains
> important data. If we are not able to enable this table in current cluster,
> do you have any idea to get the table data back in some other way? Maybe
> export, snapshot, copytable, distcp all table files to another cluster ?
>
> 2015-02-04 13:17 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>
> > Looks like the NPE was caused by the following method in BaseLoadBalancer
> > returning null:
> >
> >   protected Map<ServerName, List<HRegionInfo>> assignMasterRegions(
> >
> >       Collection<HRegionInfo> regions, List<ServerName> servers) {
> >
> >     if (servers == null || regions == null || regions.isEmpty()) {
> >
> >       return null;
> >
> > Since bulkPlan is null, calling BulkAssigner seems unnecessary.
> >
> >
> >
> > On Tue, Feb 3, 2015 at 9:01 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> >> It is not only about the state on the table descriptor but also the in
> >> memory state in the AM.  I remember some time back Rajeshbabu worked on
> a
> >> HBCK like tool which will forcefully change the state of these tables in
> >> such cases. I don't remember the JIRA now.I thought of restarting the
> >> master thinking the in memory state would change and I got this
> >>
> >> java.lang.NullPointerException
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:210)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:142)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingState(AssignmentManager.java:1695)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:416)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:720)
> >>         at
> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170)
> >>         at
> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459)
> >>         at java.lang.Thread.run(Thread.java:745)
> >> 2015-02-04 16:11:45,932 FATAL [stobdtserver3:16040.activeMasterManager]
> >> master.HMaster: Master server abort: loaded coprocessors are:
> >> [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
> >> 2015-02-04 16:11:45,933 FATAL [stobdtserver3:16040.activeMasterManager]
> >> master.HMaster: Unhandled exception. Starting shutdown.
> >> java.lang.NullPointerException
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:210)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:142)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingState(AssignmentManager.java:1695)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:416)
> >>         at
> >>
> >>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:720)
> >>         at
> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170)
> >>         at
> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459)
> >>
> >>
> >> Regards
> >> Ram
> >>
> >> On Wed, Feb 4, 2015 at 10:25 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> > What about creating an offline tool which can modify the table
> >> descriptor
> >> > so that table goes to designated state ?
> >> >
> >> > Cheers
> >> >
> >> > On Tue, Feb 3, 2015 at 8:51 PM, ramkrishna vasudevan <
> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >
> >> > > I tried reproducing this scenario on trunk. The same problem exists.
> >> > > Currently in the master the table state is noted in the Table
> >> descriptor
> >> > > and not on the ZK.  In 0.98.XX version it should be on the zk.
> >> > >
> >> > > When we tried to enable the table the region assignment failed due
> to
> >> > > ClassNotFound and already the state is in ENABLING.  But doing a
> >> describe
> >> > > table still shows it in DISABLED.
> >> > >
> >> > > Thought we could alter the correct Configuration but specifying
> >> another
> >> > > alter Table command we are still not able to enable the table.
> >> > >
> >> > > Moving this to dev to see if there is any workaround for this issue.
> >> If
> >> > > not we may have to solve this issue across branches until we have
> the
> >> > > Procedure V2 implemenation ready on trunk.
> >> > >
> >> > > Any suggestions?
> >> > >
> >> > > Regards
> >> > > Ram
> >> > >
> >> > > On Wed, Feb 4, 2015 at 4:05 AM, 叶炜晨 <yeweichen@qiyi.com>
wrote:
> >> > >
> >> > > >  my version is 0.98.6-cdh5.2.0, the problem in my production
> >> > environment.
> >> > > >
> >> > > > So should I first delete znode? And then how to distable this
> >> table?my
> >> > > > goal is to fix the wrong table configuration to get my data.
> >> > > >
> >> > > >
> >> > > >  from my mobile phone.
> >> > > >
> >> > > >  在 2015-2-4 上午12:46,ramkrishna vasudevan <
> >> > > ramkrishna.s.vasudevan@gmail.com
> >> > > > >写道:
> >> > > >
> >> > > > >
> >> > > > > I think the only way out here is to clear the zookeeper
node.
> >> But am
> >> > > > not sure on the ramifications of that.
> >> > > > >
> >> > > >
> >> > > > > Which version are you using?  The newer versions are
> >> 'protobuf'fed.
> >> > > > >
> >> > > >
> >> > > > > Are you running this in production?
> >> > > > >
> >> > > >
> >> > > > > Regards
> >> > > > > Ram
> >> > > > >
> >> > > >
> >> > > > > On Tue, Feb 3, 2015 at 5:00 PM,yeweichen2010@gmail.com<
> >> > > > yeweichen2010@gmail.com>wrote:
> >> > > >
> >> > > > >>
> >> > > > >> I tried HBCK, but it doesn`t help.
> >> > > >
> >> > > > I want to disable the table, so that I can use "alter" to fix
the
> >> wrong
> >> > > > configuration. But now the table keep in the status that no
> matter I
> >> > use
> >> > > > "is_enabled" or "is_disabled", it return false.
> >> > > > >>
> >> > > > >> ________________________________
> >> > > > >> yeweichen2010@gmail.com
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> From: ramkrishna vasudevan
> >> > > > >>> Date: 2015-02-03 19:55
> >> > > > >>> To: user@hbase.apache.org
> >> > > > >>> CC: yeweichen
> >> > > > >>> Subject: Re: Wrong Configuration lead to a failure
when
> enabling
> >> > > table
> >> > > > >>> Can you try HBCK? Did it help in anyway?  Remember
something
> was
> >> > done
> >> > > > >>> related to failure in ENABLE/DISABLE table some
time back.
> >> > > > >>>
> >> > > > >>> Regards
> >> > > > >>> Ram
> >> > > > >>>
> >> > > > >>> On Tue, Feb 3, 2015 at 3:38 PM,yeweichen2010@gmail.com<
> >> > > > >>> yeweichen2010@gmail.com> wrote:
> >> > > > >>>
> >> > > > >>> > Hi, all,
> >> > > > >>> >
> >> > > > >>> > II did the following command in hbase shell:
> >> > > > >>> >
> >> > > > >>> > disable 'TestTable'
> >> > > > >>> > alter 'TestTable', CONFIGURATION =>
> >> > > > >>> > {'hbase.regionserver.region.split.policy' =>
'xxxxxxxxx'}
> >> > > > >>> > enable 'TestTable'
> >> > > > >>> >
> >> > > > >>> > At first I want to put
> >> > > > >>> >
> >> > > "org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy"
> >> > > > to the
> >> > > > >>> > place "xxxxxxxxx", but because a spelling error,
now is
> >> something
> >> > > > wrong in
> >> > > > >>> > this configuration. After I enable the table,
it failed
> >> bacause
> >> > of
> >> > > > >>> > ClassNotFound.
> >> > > > >>> >
> >> > > > >>> > Now is the problem: the table failed to enable
and stay in a
> >> > middle
> >> > > > >>> > status. The table is neither enabled nor disabled
now. How
> >> can I
> >> > > > save my
> >> > > > >>> > table and fix the wrong configuration?
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> >yeweichen2010@gmail.com
> >> > > > >>> >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message