From user-return-55642-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Thu Dec 6 18:56:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7A45C180674 for ; Thu, 6 Dec 2018 18:56:08 +0100 (CET) Received: (qmail 93330 invoked by uid 500); 6 Dec 2018 17:56:07 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 93319 invoked by uid 99); 6 Dec 2018 17:56:07 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Dec 2018 17:56:07 +0000 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 3B89F146F for ; Thu, 6 Dec 2018 17:56:06 +0000 (UTC) Received: by mail-ot1-f49.google.com with SMTP id e12so1236365otl.5 for ; Thu, 06 Dec 2018 09:56:06 -0800 (PST) X-Gm-Message-State: AA+aEWZo98nACfzvsAQLVqQUNXs6lppXrIhDx0z6YJeYVw1JkA0phF5h wH1hrbcgF5lxxw6BAK3D8g8gqIYmtsrLm0Kgzzo= X-Google-Smtp-Source: AFSGD/WPqBUOLqfipM8+qstXQh1PqyXKNZd6oLE31Jysw1btj8usWMLNYugQH/HT2ndaADSh3NRpxOVVWvqYnwIEKf0= X-Received: by 2002:a9d:7502:: with SMTP id r2mr19190393otk.273.1544118965649; Thu, 06 Dec 2018 09:56:05 -0800 (PST) MIME-Version: 1.0 References: <2FAD35A9-C124-44B5-9116-D6BA1F265412@gmail.com> In-Reply-To: From: Andrew Purtell Date: Thu, 6 Dec 2018 09:55:29 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: How should I be getting the set of regions in transition? To: Hbase-User Content-Type: multipart/alternative; boundary="000000000000704bb3057c5e361d" --000000000000704bb3057c5e361d Content-Type: text/plain; charset="UTF-8" > In the second one the master was still initializing so it'd depend on if master will respond to cluster status requests in that state. HBASE-21521 proposes to change the order of master initialization so the UI is up immediately and able to show the current initialization step and status. I'm thinking about taking on that issue but a change of that nature will introduce a bunch of new corner cases so we will see how far I get. On Thu, Dec 6, 2018 at 9:29 AM Sean Busbey wrote: > Excellent! that definitely would have been perfect for the first case. > In the second one the master was still initializing so it'd depend on > if master will respond to cluster status requests in that state. > > I'll go grab the patch and get ready to use it the next time I hit > this. If it works I'm going to add some docs to the ref guide and a > release note. > On Thu, Dec 6, 2018 at 11:05 AM Andrew Purtell > wrote: > > > > HBASE-21283 > > > > > On Dec 6, 2018, at 8:55 AM, Andrew Purtell > wrote: > > > > > > I recently added a shell command "rit" that displays the list of > current RIT. Would that have worked? It does require that the master is > responsive to a GetClusterStatus request. > > > > > > > > >> On Dec 6, 2018, at 7:45 AM, Sean Busbey wrote: > > >> > > >> This week I've run into two cases where I needed the set of regions in > > >> transition so I could recover them and I ran into what I think is a > > >> gap in our operator tooling. I'm hoping folks will have some ideas > > >> I've missed. > > >> > > >> Depending on how this thread goes, I'll make some follow-on on the > > >> dev@hbase list for implementing changes and documentation. > > >> > > >> Case 1: HBase 1.2-ish RIT following RS crash > > >> > > >> Cluster had a handful of region servers fail and for whatever reason a > > >> few regions were stuck in transition. The operator I was helping > > >> already is used to dealing with the occasional manual recovery. Their > > >> normal process looks like this: > > >> > > >> 1) Got to Master UI website > > >> 2) Scroll down to Regions in Transition list > > >> 3) Find a RIT in FAILED_CLOSE / FAILED_OPEN / PENDING_OPEN > > >> 4) confirm on RS logs that the RS associated in the above is now in > > >> good health and doesn't expect to do anything with said region > > >> 5) run "assign" in the hbase shell for the region > > >> > > >> Unfortunately, the cluster's HDFS was under duress and so listing > > >> snapshot information was super slow. This caused the Master UI website > > >> to hang prior to displaying the RIT list. > > >> > > >> We ended up looking at the master log file. > > >> > > >> Case 2: HBase 2.1-ish RIT following cluster wide crash > > >> > > >> AFAICT cluster had experienced a failure of all RS and masters. Upon > > >> coming back up Master was left with ~10% of ~10K regions in a state of > > >> PENDING_OPEN or OPENING all with a RS that had no idea it was involved > > >> with those regions. I'm pretty sure this is a bug; I'm still triaging > > >> it and I don't think it's relevant to the current question. > > >> > > >> Once I confirmed the given RS was not currently doing anything for any > > >> of those regions I figured I'd use HBCK2 to run an assigns to get > > >> things fixed. However, since there were like 900 RITs, the Master UI > > >> was unusable for getting a complete list. Also with that many all in > > >> the same state I want to be able to automate running against each of > > >> them. > > >> > > >> I ended up greping the master log file and pulling out the WARN > > >> messages about RIT to tease out the list of regions, then passed those > > >> to hbck2. > > >> > > >> ---- > > >> > > >> Am I missing some obvious place where I can use a CLI tool to get a > > >> list of RIT? I don't see anything in the ref guide. I looked through > > >> the help of HBCK 1 and the shell and couldn't find anything. > > >> > > >> I think I can use Admin.getClusterStatus() and getClusterMetrics() to > > >> get this info from the Java API. That means there's some way to get it > > >> in the hbase shell, but it'll probably be ugly. If there's not already > > >> an easier way I'll want to wrap that so it's a simple command. > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk --000000000000704bb3057c5e361d--