Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50244FE0E for ; Tue, 16 Apr 2013 01:55:28 +0000 (UTC) Received: (qmail 87004 invoked by uid 500); 16 Apr 2013 01:55:21 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 86679 invoked by uid 500); 16 Apr 2013 01:55:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 86672 invoked by uid 99); 16 Apr 2013 01:55:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 01:55:20 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.217.169 as permitted sender) Received: from [209.85.217.169] (HELO mail-lb0-f169.google.com) (209.85.217.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 01:55:15 +0000 Received: by mail-lb0-f169.google.com with SMTP id p11so74240lbi.0 for ; Mon, 15 Apr 2013 18:54:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=uxN38Ah39SuUB3TRtRn7cLPkGp9Qz59VQNTW4HwIkvM=; b=ZhKtNudL8J7QW4oRMRBzCQRZp32DNJWxsNminDTN2UlL/xb/ccyZBhZlNFDLQR/f4q atTbl7sp8IrgepIkenTe8hyfHvaeoI5129kc/xrx5SLnvwzR8eJidOXCZonJztBBTQvP Pw8foIYpdNdCY6fdDqnXHHe+/WxTmO2Wdiz5ueZOQrdRJzeaEB8u0gu1cwZOxP1Y72K/ V1ryCsgF38J/OwnCv3PfumDoWNrN6TzsVkk7UxS5eI3fQVqhNNrnzasTq2QWYxYqGVjS X2uoT1al/Btnr4cwXS/LMckuqf9YufKYyDOA8mXGj5Y0hXU1+H6aEyyZ65+QKgsv//kb bQGQ== MIME-Version: 1.0 X-Received: by 10.112.147.229 with SMTP id tn5mr307535lbb.112.1366077294202; Mon, 15 Apr 2013 18:54:54 -0700 (PDT) Received: by 10.112.5.101 with HTTP; Mon, 15 Apr 2013 18:54:54 -0700 (PDT) In-Reply-To: <26da01ce3a44$ef929030$ceb7b090$@gmail.com> References: <26da01ce3a44$ef929030$ceb7b090$@gmail.com> Date: Mon, 15 Apr 2013 18:54:54 -0700 Message-ID: Subject: Re: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again From: Ted Yu To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=047d7b3a897adb916704da70a700 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b3a897adb916704da70a700 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I think this question would be more appropriate for HBase user mailing list= . Moving hadoop user to bcc. Please tell us the HBase version you are using. Thanks On Mon, Apr 15, 2013 at 6:51 PM, dylan wrote: > Hi**** > > ** ** > > I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes > for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN node= . > ), 3 DN nodes with zookeepers, It works fine. When I reboot one data > node machine which includes zookeeper, after that , restart all processes= . > The hadoop works fine, but hbase not. I cannot disable tables and drop > tables.**** > > ** ** > > The logs an follows:**** > > The Hbase HMaster log:**** > > DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to > unassign region -ROOT-,,0.70236052 but it is not currently assigned anywh= ere > **** > > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in > transition timed out: -ROOT-,,0.70236052 state=3DCLOSING, ts=3D136600155= 8865, > server=3DMaster,60000,1366001238313**** > > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has > been CLOSING for too long, this should eventually complete or the server > will expire, send RPC again**** > > 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting > unassignment of region -ROOT-,,0.70236052 (offlining)**** > > ** ** > > The Hbase HRegionServer log:**** > > ** ** > > DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D7.44 > MB, free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0= , > hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, cachingHitsRatio=3D0, > evictions=3D0, evicted=3D0, evictedPerRun=3DNaN**** > > ** ** > > The Hbase Web show=EF=BC=9A**** > > Region State**** > > 70236052 -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 = CST > 2013 (75440s ago), server=3DMaster,60000,1366001238313**** > > ** ** > > How fix it?**** > > ** ** > > Thanks.**** > --047d7b3a897adb916704da70a700 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I think this question would be more appropriate for HBase user mailing list= .

Moving hadoop user to bcc.

Pl= ease tell us the HBase version you are using.

Than= ks

On Mon, Apr 15, 2013 at 6:51 PM, dylan <dw= ld0425@gmail.com> wrote:

Hi

= =C2=A0

= I am a newer for hadoop, and set up hadoop with tarbal= l . I have 5 nodes for cluster, 2 NN nodes with QJM (3 Journal Nodes, one o= f them on DN node. =C2=A0), 3 DN nodes with zookeepers, =C2=A0It works fine= . =C2=A0When I reboot one data node machine which includes zookeeper, after= that , restart all proc= esses. The hadoop works fine, but hbase n= ot. I cannot disable tables and drop tables.

=C2=A0

The logs an follows= :

The Hbase HMaster log:=

DEB= UG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign = region -ROOT-,,0.70236052 but it is not currently assigned anywhere<= u>

,683 INFO org.apache.hado= op.hbase.master.AssignmentManager: Regions in transition timed out:=C2=A0 -= ROOT-,,0.70236052 state=3DCLOSING, ts=3D1366001558865, server=3DMaster,6000= 0,1366001238313

,683 INFO org.apache.hado= op.hbase.master.AssignmentManager: Region has been CLOSING for too long, th= is should eventually complete or the server will expire, send RPC again<= /u>

10,684 DEBUG org.apache.h= adoop.hbase.master.AssignmentManager: Starting unassignment of region -ROOT= -,,0.70236052 (offlining)

=C2=A0

The Hbase HRegionSe= rver log:

=C2=A0

DEBUG org.apache.ha= doop.hbase.io.hfile.LruBlockCache: Stats: total=3D7.44 MB, free=3D898.81 MB= , max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0, hitRatio=3D0, cachin= gAccesses=3D0, cachingHits=3D0, cachingHitsRatio=3D0, evictions=3D0, evicte= d=3D0, evictedPerRun=3DNaN

=C2=A0

The Hbase Web show<= /span>=EF=BC=9A<= span lang=3D"EN-US" style=3D"font-size:10.0pt;font-family:"Tahoma"= ;,"sans-serif"">

Region=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 State

70236052=C2=A0=C2=A0=C2=A0 -ROOT-= ,,0.70236052 state=3DCLOSING, ts=3DMon Apr= 15 12:52:38 CST 2013 (75440s ago), server=3DMaster,60000,1366001238313

=C2=A0

How fix it?<= u>

=C2=A0

Thanks.


--047d7b3a897adb916704da70a700--