Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8693718DC1 for ; Mon, 4 Jan 2016 17:26:50 +0000 (UTC) Received: (qmail 70722 invoked by uid 500); 4 Jan 2016 17:26:49 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 70641 invoked by uid 500); 4 Jan 2016 17:26:48 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 70624 invoked by uid 99); 4 Jan 2016 17:26:48 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2016 17:26:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 10928C78DE for ; Mon, 4 Jan 2016 17:26:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.413 X-Spam-Level: **** X-Spam-Status: No, score=4.413 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, JMQ_TRACKER=0.5, KAM_LOTSOFHASH=0.25, SPF_NEUTRAL=0.652, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=knownormal-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 84evREpAbcqW for ; Mon, 4 Jan 2016 17:26:36 +0000 (UTC) Received: from mail-ob0-f179.google.com (mail-ob0-f179.google.com [209.85.214.179]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 0E36020515 for ; Mon, 4 Jan 2016 17:26:36 +0000 (UTC) Received: by mail-ob0-f179.google.com with SMTP id bx1so227140667obb.0 for ; Mon, 04 Jan 2016 09:26:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=knownormal-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=zsXaqy/p4LesBh+W5UyNseUNu9Aagy6Xi2vYR3fA1B0=; b=O/aY+kZHY4x1o+/LUuCyi+cy8V+0JsaOx86dXJ/g1kfqDbsKwELvzlZCmIVUBI6rfG 5ptkFaYEIKoWP3ueWHmtzYu/8n/XkDuyr1pg8pkCP1HXXyuXHfz9FuOAK21pRo6v+SUz umYCEtOi5t48pfpiYbT62S8hM027FNXmiCKXbWt/q/Y+ZGRHQg6Jlen3bFNhRnL2it4Y DFvy3q0dLz+7TKSfyd2ZvvaSEdCtYdfTaGdfbsM7iqNtyKE3ghGpCcnvvDfzrfFUt6Wh /DzrxnXuOMPb4WYhHV5VCn5xsM9dX3WAlBZ0UKnrWFrt3SOHJLWfLH5grX9yOIq+rLsq wQoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=zsXaqy/p4LesBh+W5UyNseUNu9Aagy6Xi2vYR3fA1B0=; b=fVA9qvbVLZmfJDGpfXLGNG/0mm8OUer2NuQgdbvH+Tev2KlWNkAliTCLr79tq5GTXi 2CMtR7XxwGn1o2AuvTn+IuVtCUh+vWAtIjJRah4gq8yyu5zdj44a68OP3xtOIYBiF0GR j/UsShmG5Wt2pF0qwekJqMjQPed4TeR8S91oyeBxRfgIPBJtyBLL4YHf4LoS4DDb3MDQ B4GVsCL0yQJmn1I9954RFMILlG/Bh4b9fAPfVVY1H4V8w5m4iVpf/IHRZBviU/1p3wPD eN3w5aS+Xvg6Kef8v9BgP/OHaSAgUd6h3a2bTyncL68xDn/Wb03mBKqEQiQYfJhmOXaR S52A== X-Gm-Message-State: ALoCoQlcmVbYIf4P3ud52vh4usD5av6aMuoj6eBGWABi4WnKQ9UCsY5/BEQ5vErpw7om4todutwnMrZ2UW+VOV5z74KKZMj7ig== X-Received: by 10.60.54.133 with SMTP id j5mr63062026oep.2.1451928389204; Mon, 04 Jan 2016 09:26:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.91.135 with HTTP; Mon, 4 Jan 2016 09:26:09 -0800 (PST) In-Reply-To: References: From: "Shane O'Donnell" Date: Mon, 4 Jan 2016 12:26:09 -0500 Message-ID: Subject: Re: Hbase master not starting To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=089e0115f906e0d75e0528856bc4 --089e0115f906e0d75e0528856bc4 Content-Type: text/plain; charset=UTF-8 I looked for region 82432aca9ede964943b40753cb64e808 on each of my region servers and none of them had it. They all had identical failures in the log of attempting to open it, but none had it (or at least were successful in opening it). My solution to "finding it" was grepping for "82432aca9ede964943b40753cb64e808" in the logs. If there's a better or more reliable way of searching for this, let me know. Thanks - Shane O. ======================== Shane O'Donnell 4819 Emperor Blvd., Ste 400 Durham, North Carolina 27703 tel: +1.424.262.KNOW x703 skype: shaneodonnell email: shaneo@knownormal.com ======================== :wq! On Mon, Jan 4, 2016 at 12:03 PM, Shane O'Donnell wrote: > It's not there. The directory listing was for the right directory, but > the namespace directory is not there. > > Here it is one from directory level up: > > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R > /hbase/data > > drwxr-xr-x - hdfs hadoop 0 2016-01-04 14:47 /hbase/data/hbase > > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 > /hbase/data/hbase/meta > > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 > /hbase/data/hbase/meta/.tabledesc > > -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48 > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 > > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 > /hbase/data/hbase/meta/.tmp > > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 > /hbase/data/hbase/meta/1588230740 > > -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28 > /hbase/data/hbase/meta/1588230740/.regioninfo > > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47 > /hbase/data/hbase/meta/1588230740/info > > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 > /hbase/data/hbase/meta/1588230740/recovered.edits > > -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28 > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid > > ======================== > Shane O'Donnell > > > 4819 Emperor Blvd., Ste 400 > Durham, North Carolina 27703 > tel: +1.424.262.KNOW x703 > skype: shaneodonnell > email: shaneo@knownormal.com > ======================== > :wq! > > On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu wrote: > >> In your listing, meta table directory structure was shown. >> >> Please look under /hbase/data/hbase for namespace table. >> >> Cheers >> >> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell >> wrote: >> >> > I attempted to restore a backed-up "meta" directory from a copy made by >> > 'hbase hbck', so my directory structure may be messed up: >> > >> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls >> -R >> > /hbase/data/hbase >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 >> > /hbase/data/hbase/meta >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 >> > /hbase/data/hbase/meta/.tabledesc >> > >> > -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48 >> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 >> > /hbase/data/hbase/meta/.tmp >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 >> > /hbase/data/hbase/meta/1588230740 >> > >> > -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28 >> > /hbase/data/hbase/meta/1588230740/.regioninfo >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47 >> > /hbase/data/hbase/meta/1588230740/info >> > >> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 >> > /hbase/data/hbase/meta/1588230740/recovered.edits >> > >> > -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28 >> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid >> > >> > And I'm not seeing any "namespace" directory. >> > >> > Will check out the server now... >> > >> > Shane O. >> > >> > ======================== >> > Shane O'Donnell >> > < >> > >> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69 >> > > >> > 4819 Emperor Blvd., Ste 400 >> > Durham, North Carolina 27703 >> > tel: +1.424.262.KNOW x703 >> > skype: shaneodonnell >> > email: shaneo@knownormal.com >> > ======================== >> > :wq! >> > >> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu wrote: >> > >> > > Can you log onto the server hosting region >> > 82432aca9ede964943b40753cb64e808 >> > > and see what happened ? >> > > >> > > See if the namespace table can be found under rootdir. >> > > e.g. assuming /apps/hbase/data is the rootdir, you should see >> something >> > > similar to the following: >> > > >> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace >> > > Found 3 items >> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15 >> > > /apps/hbase/data/data/hbase/namespace/.tabledesc >> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15 >> > > /apps/hbase/data/data/hbase/namespace/.tmp >> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51 >> > > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8 >> > > >> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell < >> shaneo@knownormal.com> >> > > wrote: >> > > >> > > > Some progress... >> > > > >> > > > /hbase did NOT have either the hbase.id or hbase.version files so I >> > > > temporarily changed hbase.rootdir and started the master so the >> files >> > > would >> > > > be recreated elsewhere and copied them in. >> > > > >> > > > Now it starts fine, but my hbase tables are gone. Specifically, I'm >> > > > getting this error: >> > > > >> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808. >> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago), >> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749 >> > > > >> > > > Is my existing data toast, or is there a crafty way out of this? >> > > > >> > > > Thanks - >> > > > >> > > > Shane O. >> > > > >> > > > ======================== >> > > > Shane O'Donnell >> > > > < >> > > > >> > > >> > >> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960 >> > > > > >> > > > 4819 Emperor Blvd., Ste 400 >> > > > Durham, North Carolina 27703 >> > > > tel: +1.424.262.KNOW x703 >> > > > skype: shaneodonnell >> > > > email: shaneo@knownormal.com >> > > > ======================== >> > > > :wq! >> > > > >> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell < >> > shaneo@knownormal.com> >> > > > wrote: >> > > > >> > > > > Hi - >> > > > > >> > > > > My cluster has been running perfectly until the other day when I >> > found >> > > it >> > > > > down. >> > > > > >> > > > > The error seems to be related not being able to get the ClusterID >> > from >> > > > > zookeeper, but I'm stumped as to what to do about it. This seems >> to >> > be >> > > > the >> > > > > relevant part of the master's log: >> > > > > >> > > > > http://pastebin.com/C3iaxM3p >> > > > > >> > > > > starting at line 235 (also highlighted). >> > > > > >> > > > > Any help is appreciated! >> > > > > >> > > > > Thanks - >> > > > > >> > > > > Shane O. >> > > > > ======================== >> > > > > Shane O'Donnell >> > > > > >> > > > > < >> > > > >> > > >> > >> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac >> > > > > >> > > > > 4819 Emperor Blvd., Ste 400 >> > > > > Durham, North Carolina 27703 >> > > > > tel: +1.424.262.KNOW x703 >> > > > > skype: shaneodonnell >> > > > > email: shaneo@knownormal.com >> > > > > ======================== >> > > > > :wq! >> > > > > >> > > > >> > > >> > >> > > --089e0115f906e0d75e0528856bc4--