Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E3D95C173 for ; Sun, 20 May 2012 15:02:55 +0000 (UTC) Received: (qmail 14084 invoked by uid 500); 20 May 2012 15:02:54 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 14044 invoked by uid 500); 20 May 2012 15:02:53 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 14036 invoked by uid 99); 20 May 2012 15:02:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 15:02:53 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.111.87 as permitted sender) Received: from [65.55.111.87] (HELO blu0-omc2-s12.blu0.hotmail.com) (65.55.111.87) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 15:02:45 +0000 Received: from BLU0-SMTP390 ([65.55.111.71]) by blu0-omc2-s12.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Sun, 20 May 2012 08:02:24 -0700 X-Originating-IP: [173.15.87.37] X-Originating-Email: [michael_segel@hotmail.com] Message-ID: Received: from [192.168.0.100] ([173.15.87.37]) by BLU0-SMTP390.phx.gbl over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Sun, 20 May 2012 08:02:22 -0700 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 (Apple Message framework v1278) Subject: Re: forcing offline From: Michael Segel In-Reply-To: <8A076CC6-3AB4-4F65-AD4D-B6157968499D@ymail.com> Date: Sun, 20 May 2012 10:02:21 -0500 Content-Transfer-Encoding: quoted-printable References: <8A076CC6-3AB4-4F65-AD4D-B6157968499D@ymail.com> To: user@hbase.apache.org X-Mailer: Apple Mail (2.1278) X-OriginalArrivalTime: 20 May 2012 15:02:22.0746 (UTC) FILETIME=[8F67E7A0:01CD3699] X-Virus-Checked: Checked by ClamAV on apache.org What did you see when you ran the HBase shell's status?=20 Did you run status w higher details? (see status help) On May 20, 2012, at 2:12 AM, Ben Cuthbert wrote: > All >=20 > We run a load test and after about 3 hours our application stopped. = Check the logs I see this in the hbase-master log >=20 > 2012-05-20 08:08:17,251 INFO = org.apache.hadoop.hbase.master.AssignmentManager: Region has been = OFFLINE for too long, reassigning -ROOT-,,0.70236052 to a random server > 2012-05-20 08:08:17,252 DEBUG = org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; = was=3D.META.,,1.1028785192 state=3DOFFLINE, ts=3D1337497517243 > 2012-05-20 08:08:17,252 DEBUG = org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; = was=3D-ROOT-,,0.70236052 state=3DOFFLINE, ts=3D1337497517243 > 2012-05-20 08:10:10,309 INFO = org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection = from /0:0:0:0:0:0:0:1%0:62747 > 2012-05-20 08:10:10,315 INFO = org.apache.zookeeper.server.NIOServerCnxn: Client attempting to = establish new session at /0:0:0:0:0:0:0:1%0:62747 > 2012-05-20 08:10:10,316 INFO = org.apache.zookeeper.server.NIOServerCnxn: Established session = 0x137653a0e8e02fa with negotiated timeout 40000 for client = /0:0:0:0:0:0:0:1%0:62747 > 2012-05-20 08:10:10,316 INFO = org.apache.zookeeper.server.PrepRequestProcessor: Got user-level = KeeperException when processing sessionid:0x137653a0e8e02fa type:create = cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error = Path:/hbase Error:KeeperErrorCode =3D NodeExists for /hbase > 2012-05-20 08:10:10,329 INFO = org.apache.zookeeper.server.PrepRequestProcessor: Got user-level = KeeperException when processing sessionid:0x137653a0e8e02fa type:create = cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error = Path:/hbase/unassigned Error:KeeperErrorCode =3D NodeExists for = /hbase/unassigned > 2012-05-20 08:10:10,329 INFO = org.apache.zookeeper.server.PrepRequestProcessor: Got user-level = KeeperException when processing sessionid:0x137653a0e8e02fa type:create = cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error = Path:/hbase/rs Error:KeeperErrorCode =3D NodeExists for /hbase/rs > 2012-05-20 08:10:10,330 INFO = org.apache.zookeeper.server.PrepRequestProcessor: Got user-level = KeeperException when processing sessionid:0x137653a0e8e02fa type:create = cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error = Path:/hbase/table Error:KeeperErrorCode =3D NodeExists for /hbase/table >=20 >=20 > Hadoop seems to be up and running. >=20 > last log in the datanode is >=20 > 12/05/20 06:15:25 INFO datanode.DataBlockScanner: Verification = succeeded for blk_-3639294708473848144_3329 > 12/05/20 06:26:20 INFO datanode.DataBlockScanner: Verification = succeeded for blk_2502932128500788221_3413 > 12/05/20 06:26:20 INFO datanode.DataBlockScanner: Verification = succeeded for blk_3390059684225099859_3440 > 12/05/20 06:59:32 INFO datanode.DataNode: BlockReport of 157 blocks = took 19 msec to generate and 3 msecs for RPC and NN processing > 12/05/20 07:24:51 INFO datanode.DataBlockScanner: Verification = succeeded for blk_8954400942867609419_3363 > 12/05/20 07:55:51 INFO datanode.DataBlockScanner: Verification = succeeded for blk_-3650918785526360502_3387 > 12/05/20 07:59:33 INFO datanode.DataNode: BlockReport of 157 blocks = took 20 msec to generate and 3 msecs for RPC and NN processing > 12/05/20 08:07:25 INFO datanode.DataBlockScanner: Verification = succeeded for blk_786514597978592338_3336 >=20 > I tried using hbase-explorer to view the tables but they all seem to = down.