From dev-return-73069-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Tue Mar 12 22:16:35 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 53758180600 for ; Tue, 12 Mar 2019 23:16:35 +0100 (CET) Received: (qmail 5829 invoked by uid 500); 12 Mar 2019 22:16:34 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 5818 invoked by uid 99); 12 Mar 2019 22:16:34 -0000 Received: from mail-relay.apache.org (HELO mailrelay2-lw-us.apache.org) (207.244.88.137) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Mar 2019 22:16:34 +0000 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id 46B383593 for ; Tue, 12 Mar 2019 22:16:33 +0000 (UTC) Received: by mail-qt1-f173.google.com with SMTP id h39so4557114qte.2 for ; Tue, 12 Mar 2019 15:16:33 -0700 (PDT) X-Gm-Message-State: APjAAAXfkXl/Esf1liBZ7pi4kqG0N/+e3jX+8zZSHvFeroc7tAi9WWDS A3iUXm9nHsRqUx9VFxuJZ2AtNrySR/R3kaBbUc8= X-Google-Smtp-Source: APXvYqwZ8sv4IYtvH/5vJPVQAj9x5smReBOJgGUyDf4YvmXwAiP2B/IYvoG/Q1gL5HRp+fRI5SGedZHNmHc0rA3BcOc= X-Received: by 2002:ac8:2486:: with SMTP id s6mr32838882qts.251.1552428992617; Tue, 12 Mar 2019 15:16:32 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Sean Busbey Date: Tue, 12 Mar 2019 17:17:49 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: 2.0.4 to 2.2.0 testing To: dev Content-Type: text/plain; charset="UTF-8" On Tue, Mar 12, 2019 at 2:21 PM Jean-Marc Spaggiari wrote: > > > The important bit will be that you get a currently working HBCK2 instance > > I did. I downloaded all 2.2.0 artifacts locally, and build with the > parameters you gave. All went well. However, HBCK2 can't talk to my HBase > server... > > I tried this: > java org.apache.hbase.HBCK2 assigns 1431292037213 > > But I got this: > PleaseHoldException: Master is initializing > That's not how HBCK2 is invoked. Let's get the correct invocation working and then we can work out what's going on with master. > The hbase hbck -j ~/hbase-hbck2-for-2.2.0.jar unassigns 1588230740 command > doesn.t seems to be correct. "-j" is not a valid parameter, should be -jar. > And fixing it displayes the previous HBCK version help. > Okay, something is still wrong then. The cli option is definitely "-j" and not "-jar". Which "hbase" command is being run? Can you confirm it's the one in the bin tarball? what does the version command output? What does the mapredcp command output? What does the cli help for the hbase command show? > Because the META is now assigned, I changed 1588230740 to 1431292037213 to > get the namespace table assigned. No idea how the meta got assigned... I > removed /hbase from ZK, maybe it helped. > It's not good that we're bouncing around states without knowing why those states are changing. Let's focus on getting hbck2 invocations to work as expected before continuing with cluster health. > Cluster is still running in maintenance mode: > > hbase.master.maintenance_mode > true > > I don't recall suggesting maintenance mode and would suggest not being in it for now. > That's why I'm very surprised it tries to still assign some regions to > other servers (master is node2) > 2019-03-12 10:59:12,286 WARN [master/node2:60000:becomeActiveMaster] > master.HMaster: > hbase:namespace,,1431292037213.7f4a480f47f98300185d1ae2ff663295. is NOT > online; state={7f4a480f47f98300185d1ae2ff663295 state=OPENING, > ts=1552402749342, server=node3,16020,1551984565705}; > ServerCrashProcedures=true. Master startup cannot progress, in > holding-pattern until region onlined. > Yes, that is unusual. Let's maybe come back to that if necessary, but for now please stop trying to use maintenance mode.