Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5A271105DD for ; Wed, 4 Feb 2015 08:41:11 +0000 (UTC) Received: (qmail 84667 invoked by uid 500); 4 Feb 2015 08:41:10 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 84600 invoked by uid 500); 4 Feb 2015 08:41:10 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 84573 invoked by uid 99); 4 Feb 2015 08:41:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 08:41:10 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ramkrishna.s.vasudevan@gmail.com designates 209.85.223.173 as permitted sender) Received: from [209.85.223.173] (HELO mail-ie0-f173.google.com) (209.85.223.173) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 08:40:44 +0000 Received: by mail-ie0-f173.google.com with SMTP id tr6so385100ieb.4; Wed, 04 Feb 2015 00:40:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UhlHW4VE1xNbh9D7GG2weHOs84zIFQ+Md1uTGY6gGs4=; b=iSRQyVYve4qufQKKg++UmISWQ3NQg8Guc9IO2xFCSBymsUcCGvTFLzm5Rt0/RpSo9y zgGxBbdEJ/ZEeerH2tJAwX6Svzs740YhsXWLl+9tvfJ/EXIBb9qkTKw/05sJxv7YIboJ 4K8j4dAkcZXeshbPjlqoBXEUo6f8DRp1M0u/szcx3IiQpFOt8Vug2snpHkBThVMIVA/H 8t7YGMhTquEaaBh51N0LLbSJDN7ADPh5+Zz5MBlDAj9p5q7sSPuvkQmB47LgYGdiJuCj zT0khkjxsyUphdylL6b+ursHEjCMYetBGxbRP+9qcPoGnHf2n37Hs0qNlURSMnVTPywm VSOQ== MIME-Version: 1.0 X-Received: by 10.42.44.197 with SMTP id c5mr727528icf.35.1423039237937; Wed, 04 Feb 2015 00:40:37 -0800 (PST) Received: by 10.107.47.158 with HTTP; Wed, 4 Feb 2015 00:40:37 -0800 (PST) In-Reply-To: References: <1liaiali9gs6i79k229ic496.1423002921954@email.android.com> Date: Wed, 4 Feb 2015 14:10:37 +0530 Message-ID: Subject: Re: Re: Re: Wrong Configuration lead to a failure when enabling table From: ramkrishna vasudevan To: "user@hbase.apache.org" Cc: Ted Yu , "dev@hbase.apache.org" , =?UTF-8?B?5Y+254Kc5pmo?= Content-Type: multipart/alternative; boundary=bcaec51968cf476930050e3f2431 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51968cf476930050e3f2431 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Raised this https://issues.apache.org/jira/browse/HBASE-12967. On Wed, Feb 4, 2015 at 1:36 PM, ramkrishna vasudevan < ramkrishna.s.vasudevan@gmail.com> wrote: > distcp should work. Try online snapshot case - I think it might work. Bu= t > ideally here the table is in the DISABLING state. so it is again tricky. > > Take a raw back up of the table data in the HDFS - and again put it back > after restoring the table? > > On Wed, Feb 4, 2015 at 12:41 PM, Weichen YE > wrote: > >> Hi=EF=BC=8CTed, Ram, >> >> Thank you for your attemtion for this bug. >> >> I meet this bug in production environment and the table contains >> important data. If we are not able to enable this table in current >> cluster, >> do you have any idea to get the table data back in some other way? Maybe >> export, snapshot, copytable, distcp all table files to another cluster ? >> >> 2015-02-04 13:17 GMT+08:00 Ted Yu : >> >> > Looks like the NPE was caused by the following method in >> BaseLoadBalancer >> > returning null: >> > >> > protected Map> assignMasterRegions( >> > >> > Collection regions, List servers) { >> > >> > if (servers =3D=3D null || regions =3D=3D null || regions.isEmpty(= )) { >> > >> > return null; >> > >> > Since bulkPlan is null, calling BulkAssigner seems unnecessary. >> > >> > >> > >> > On Tue, Feb 3, 2015 at 9:01 PM, ramkrishna vasudevan < >> > ramkrishna.s.vasudevan@gmail.com> wrote: >> > >> >> It is not only about the state on the table descriptor but also the i= n >> >> memory state in the AM. I remember some time back Rajeshbabu worked >> on a >> >> HBCK like tool which will forcefully change the state of these tables >> in >> >> such cases. I don't remember the JIRA now.I thought of restarting the >> >> master thinking the in memory state would change and I got this >> >> >> >> java.lang.NullPointerException >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTa= ble(EnableTableHandler.java:210) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(Enable= TableHandler.java:142) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingS= tate(AssignmentManager.java:1695) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentM= anager.java:416) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(= HMaster.java:720) >> >> at >> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170) >> >> at >> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 2015-02-04 16:11:45,932 FATAL [stobdtserver3:16040.activeMasterManage= r] >> >> master.HMaster: Master server abort: loaded coprocessors are: >> >> [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] >> >> 2015-02-04 16:11:45,933 FATAL [stobdtserver3:16040.activeMasterManage= r] >> >> master.HMaster: Unhandled exception. Starting shutdown. >> >> java.lang.NullPointerException >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTa= ble(EnableTableHandler.java:210) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(Enable= TableHandler.java:142) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager.recoverTableInEnablingS= tate(AssignmentManager.java:1695) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentM= anager.java:416) >> >> at >> >> >> >> >> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(= HMaster.java:720) >> >> at >> >> org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:170) >> >> at >> org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1459) >> >> >> >> >> >> Regards >> >> Ram >> >> >> >> On Wed, Feb 4, 2015 at 10:25 AM, Ted Yu wrote: >> >> >> >> > What about creating an offline tool which can modify the table >> >> descriptor >> >> > so that table goes to designated state ? >> >> > >> >> > Cheers >> >> > >> >> > On Tue, Feb 3, 2015 at 8:51 PM, ramkrishna vasudevan < >> >> > ramkrishna.s.vasudevan@gmail.com> wrote: >> >> > >> >> > > I tried reproducing this scenario on trunk. The same problem >> exists. >> >> > > Currently in the master the table state is noted in the Table >> >> descriptor >> >> > > and not on the ZK. In 0.98.XX version it should be on the zk. >> >> > > >> >> > > When we tried to enable the table the region assignment failed du= e >> to >> >> > > ClassNotFound and already the state is in ENABLING. But doing a >> >> describe >> >> > > table still shows it in DISABLED. >> >> > > >> >> > > Thought we could alter the correct Configuration but specifying >> >> another >> >> > > alter Table command we are still not able to enable the table. >> >> > > >> >> > > Moving this to dev to see if there is any workaround for this >> issue. >> >> If >> >> > > not we may have to solve this issue across branches until we have >> the >> >> > > Procedure V2 implemenation ready on trunk. >> >> > > >> >> > > Any suggestions? >> >> > > >> >> > > Regards >> >> > > Ram >> >> > > >> >> > > On Wed, Feb 4, 2015 at 4:05 AM, =E5=8F=B6=E7=82=9C=E6=99=A8 wrote: >> >> > > >> >> > > > my version is 0.98.6-cdh5.2.0, the problem in my production >> >> > environment. >> >> > > > >> >> > > > So should I first delete znode? And then how to distable this >> >> table=EF=BC=9Fmy >> >> > > > goal is to fix the wrong table configuration to get my data. >> >> > > > >> >> > > > >> >> > > > from my mobile phone. >> >> > > > >> >> > > > =E5=9C=A8 2015-2-4 =E4=B8=8A=E5=8D=8812:46=EF=BC=8Cramkrishna = vasudevan < >> >> > > ramkrishna.s.vasudevan@gmail.com >> >> > > > >=E5=86=99=E9=81=93=EF=BC=9A >> >> > > > >> >> > > > > >> >> > > > > I think the only way out here is to clear the zookeeper node. >> >> But am >> >> > > > not sure on the ramifications of that. >> >> > > > > >> >> > > > >> >> > > > > Which version are you using? The newer versions are >> >> 'protobuf'fed. >> >> > > > > >> >> > > > >> >> > > > > Are you running this in production? >> >> > > > > >> >> > > > >> >> > > > > Regards >> >> > > > > Ram >> >> > > > > >> >> > > > >> >> > > > > On Tue, Feb 3, 2015 at 5:00 PM,yeweichen2010@gmail.com< >> >> > > > yeweichen2010@gmail.com>wrote: >> >> > > > >> >> > > > >> >> >> > > > >> I tried HBCK, but it doesn`t help. >> >> > > > >> >> > > > I want to disable the table, so that I can use "alter" to fix t= he >> >> wrong >> >> > > > configuration. But now the table keep in the status that no >> matter I >> >> > use >> >> > > > "is_enabled" or "is_disabled", it return false. >> >> > > > >> >> >> > > > >> ________________________________ >> >> > > > >> yeweichen2010@gmail.com >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> From: ramkrishna vasudevan >> >> > > > >>> Date: 2015-02-03 19:55 >> >> > > > >>> To: user@hbase.apache.org >> >> > > > >>> CC: yeweichen >> >> > > > >>> Subject: Re: Wrong Configuration lead to a failure when >> enabling >> >> > > table >> >> > > > >>> Can you try HBCK? Did it help in anyway? Remember somethin= g >> was >> >> > done >> >> > > > >>> related to failure in ENABLE/DISABLE table some time back. >> >> > > > >>> >> >> > > > >>> Regards >> >> > > > >>> Ram >> >> > > > >>> >> >> > > > >>> On Tue, Feb 3, 2015 at 3:38 PM,yeweichen2010@gmail.com< >> >> > > > >>> yeweichen2010@gmail.com> wrote: >> >> > > > >>> >> >> > > > >>> > Hi, all, >> >> > > > >>> > >> >> > > > >>> > II did the following command in hbase shell: >> >> > > > >>> > >> >> > > > >>> > disable 'TestTable' >> >> > > > >>> > alter 'TestTable', CONFIGURATION =3D> >> >> > > > >>> > {'hbase.regionserver.region.split.policy' =3D> 'xxxxxxxxx= '} >> >> > > > >>> > enable 'TestTable' >> >> > > > >>> > >> >> > > > >>> > At first I want to put >> >> > > > >>> > >> >> > > >> "org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy" >> >> > > > to the >> >> > > > >>> > place "xxxxxxxxx", but because a spelling error, now is >> >> something >> >> > > > wrong in >> >> > > > >>> > this configuration. After I enable the table, it failed >> >> bacause >> >> > of >> >> > > > >>> > ClassNotFound. >> >> > > > >>> > >> >> > > > >>> > Now is the problem: the table failed to enable and stay i= n >> a >> >> > middle >> >> > > > >>> > status. The table is neither enabled nor disabled now. Ho= w >> >> can I >> >> > > > save my >> >> > > > >>> > table and fix the wrong configuration? >> >> > > > >>> > >> >> > > > >>> > >> >> > > > >>> > >> >> > > > >>> > >> >> > > > >>> > >> >> > > > >>> >yeweichen2010@gmail.com >> >> > > > >>> > >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > > --bcaec51968cf476930050e3f2431--