Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4616C10A64 for ; Tue, 16 Apr 2013 02:42:54 +0000 (UTC) Received: (qmail 7897 invoked by uid 500); 16 Apr 2013 02:42:49 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 7800 invoked by uid 500); 16 Apr 2013 02:42:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 7791 invoked by uid 99); 16 Apr 2013 02:42:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 02:42:49 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of azuryyyu@gmail.com designates 209.85.128.172 as permitted sender) Received: from [209.85.128.172] (HELO mail-ve0-f172.google.com) (209.85.128.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 02:42:42 +0000 Received: by mail-ve0-f172.google.com with SMTP id db10so20677veb.31 for ; Mon, 15 Apr 2013 19:42:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=wdvaL/nbvmDU9b+g8e4z8a+35lzvP+6CfDSNGku3fDo=; b=zl4EWcCtXJLMRdNcHoN2LEw4RKrASykZMERskb+a9NlYJU1QCIGEsyImxfsr7ixVO2 f3MYsQaxp611PPJAQ0RFmZQh7QVMB9Thz0rSMVhr2+FlIGmQP1Mad/mXd6wy+4oxTI/v ZLAD+C7A3frjXNnHq3/RjEjID3oaNencMuDt8yJZX5ezB23435LB85xBd0UI7jYdvvzY cxmhKIghigJPD/UZJZKIcPH8CEvipJvZCsbXfF+O/bofcrWEFtip9ZfLE6qwGAaJWAC0 wckO85HjSFQxWgDk/Nl5o29qcrhk0rpORoIQWl4T1fiwl3YxrzTE2K+94bhEbIgpI4wO Jgjg== MIME-Version: 1.0 X-Received: by 10.52.243.196 with SMTP id xa4mr225033vdc.22.1366080140944; Mon, 15 Apr 2013 19:42:20 -0700 (PDT) Received: by 10.220.72.200 with HTTP; Mon, 15 Apr 2013 19:42:20 -0700 (PDT) In-Reply-To: <293801ce3a4b$6a446af0$3ecd40d0$@gmail.com> References: <26da01ce3a44$ef929030$ceb7b090$@gmail.com> <27e401ce3a45$f45552f0$dcfff8d0$@gmail.com> <293801ce3a4b$6a446af0$3ecd40d0$@gmail.com> Date: Tue, 16 Apr 2013 10:42:20 +0800 Message-ID: Subject: =?GB2312?Q?Re=3A_=B4=F0=B8=B4=3A_=B4=F0=B8=B4=3A_Region_has_been_CLOSING_for_too_?= =?GB2312?Q?long=2C_this_should_eventually_complete_or_the_server_will_exp?= =?GB2312?Q?ire=2C_send_RPC_again?= From: Azuryy Yu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c1c476892a8504da715177 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c1c476892a8504da715177 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable it located under hbase-home/logs/ if your zookeeper is managed by hbase. but I noticed you configured QJM, then did your QJM and Hbase share the same ZK cluster? if so, then just paste your QJM zk configuration in the hdfs-site.xml and hbase zk configuration in the hbase-site.xml. On Tue, Apr 16, 2013 at 10:37 AM, dylan wrote: > How to check zookeeper log?? It is the binary files, how to transform it > to normal log? ****** **I find the =A1=B0 > org.apache.zookeeper.server.LogFormatter=A1=B1, how to run?****** ** > > ** ** > > *=B7=A2=BC=FE=C8=CB:* Azuryy Yu [mailto:azuryyyu@gmail.com] > *=B7=A2=CB=CD=CA=B1=BC=E4:* 2013=C4=EA4=D4=C216=C8=D5 10:01 > *=CA=D5=BC=FE=C8=CB:* user@hadoop.apache.org > *=D6=F7=CC=E2:* Re: =B4=F0=B8=B4: Region has been CLOSING for too long, t= his should > eventually complete or the server will expire, send RPC again**** > > ** ** > > This is zookeeper issue.**** > > ** ** > > please paste zookeeper log here. thanks.**** > > ** ** > > On Tue, Apr 16, 2013 at 9:58 AM, dylan wrote:**** > > It is hbase-0.94.2-cdh4.2.0.**** > > **** > > *=B7=A2=BC=FE=C8=CB:* Ted Yu [mailto:yuzhihong@gmail.com] > *=B7=A2=CB=CD=CA=B1=BC=E4:* 2013=C4=EA4=D4=C216=C8=D5 9:55 > *=CA=D5=BC=FE=C8=CB:* user@hbase.apache.org > *=D6=F7=CC=E2:* Re: Region has been CLOSING for too long, this should eve= ntually > complete or the server will expire, send RPC again**** > > **** > > I think this question would be more appropriate for HBase user mailing > list.**** > > **** > > Moving hadoop user to bcc.**** > > **** > > Please tell us the HBase version you are using.**** > > **** > > Thanks**** > > On Mon, Apr 15, 2013 at 6:51 PM, dylan wrote:**** > > Hi**** > > **** > > I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes > for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN node= . > ), 3 DN nodes with zookeepers, It works fine. When I reboot one data > node machine which includes zookeeper, after that , restart all processes= . > The hadoop works fine, but hbase not. I cannot disable tables and drop > tables.**** > > **** > > The logs an follows:**** > > The Hbase HMaster log:**** > > DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to > unassign region -ROOT-,,0.70236052 but it is not currently assigned anywh= ere > **** > > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in > transition timed out: -ROOT-,,0.70236052 state=3DCLOSING, ts=3D136600155= 8865, > server=3DMaster,60000,1366001238313**** > > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has > been CLOSING for too long, this should eventually complete or the server > will expire, send RPC again**** > > 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting > unassignment of region -ROOT-,,0.70236052 (offlining)**** > > **** > > The Hbase HRegionServer log:**** > > **** > > DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D7.44 > MB, free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0= , > hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, cachingHitsRatio=3D0, > evictions=3D0, evicted=3D0, evictedPerRun=3DNaN**** > > **** > > The Hbase Web show=A3=BA**** > > Region State**** > > 70236052 -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 = CST > 2013 (75440s ago), server=3DMaster,60000,1366001238313**** > > **** > > How fix it?**** > > **** > > Thanks.**** > > **** > > ** ** > --001a11c1c476892a8504da715177 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable
it located under hbase-home/logs/  if your zookeeper = is managed by hbase.

but I noticed you configured = QJM, then did your QJM and Hbase share the same ZK cluster? if so, then jus= t paste your QJM zk configuration in the hdfs-site.xml and hbase zk configu= ration in the hbase-site.xml.


On Tue,= Apr 16, 2013 at 10:37 AM, dylan <dwld0425@gmail.com> wrote= :

How to ch= eck zookeeper log?? It is the binary files, how to transform = it to normal log?

 

I find the “= org.apache.zookeeper.server.LogFormatter”= , how to run?

 

 

=B7=A2=BC=FE=C8= =CB: Azuryy Yu [mailto:azuryyyu@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 1= 0:01
=CA=D5=BC=FE=C8=CB: user@ha= doop.apache.org
=D6=F7=CC=E2:<= /b> Re: =B4=F0=B8=B4: Regi= on has been CLOSING for too long, this should eventually complete or the se= rver will expire, send RPC again

&= nbsp;

Thi= s is zookeeper issue.

<= span lang=3D"EN-US"> 

please paste zookeep= er log here. thanks.

 =

On Tue, Apr 16, 2013 at 9:= 58 AM, dylan <dw= ld0425@gmail.com> wrote:

It is hbase-0.94.2-cdh4.2.0.

<= p class=3D"MsoNormal"> 

=B7=A2=BC=FE=C8= =CB: Ted Yu [mailto:yuzhihong@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 9= :55
=CA=D5=BC=FE=C8=CB: user@hba= se.apache.org
=D6=F7=CC=E2: Re: Region has been CLOSING for too long, this shoul= d eventually complete or the server will expire, send RPC again

 <= /span>

I think this question= would be more appropriate for HBase user mailing list.

 

=

Moving hadoop user t= o bcc.

 

Please tell us the H= Base version you are using.

 

<= p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"> Thanks

On Mon, Apr 15, 2013 at 6:51 PM, dylan <dwld0425@gmail.com>= ; wrote:

Hi

 

I am a newer for hadoop, = and set up hadoop with tarball . I have 5 nodes for cluster, 2 NN nodes wit= h QJM (3 Journal Nodes, one of them on DN node.  ), 3 DN nodes with zo= okeepers,  It works fine.  When I reboot one data node machine wh= ich includes zookeeper, after that , restart all processes. The h= adoop works fine, but hbase not. I cannot disable tables and drop tables.

 

The logs an follows:

The Hbase HMaster log:

DEBUG org.apache.hadoop.hbase.master.AssignmentMan= ager: Attempted to unassign region -ROOT-,,0.70236052 but it is not current= ly assigned anywhere

,683 INFO org.apache.hado= op.hbase.master.AssignmentManager: Regions in transition timed out:  -= ROOT-,,0.70236052 state=3DCLOSING, ts=3D1366001558865, server=3DMaster,6000= 0,1366001238313

,683 INFO org.apache.hado= op.hbase.master.AssignmentManager: Region has been CLOSING for too long, th= is should eventually complete or the server will expire, send RPC again

10,684 DEBUG org.apache.h= adoop.hbase.master.AssignmentManager: Starting unassignment of region -ROOT= -,,0.70236052 (offlining)<= /p>

 

The Hbase HRegionServer log:<= u>

 

DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: tot= al=3D7.44 MB, free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, = hits=3D0, hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, cachingHitsRa= tio=3D0, evictions=3D0, evicted=3D0, evictedPerRun=3DNaN

 

The Hbase Web show=A3=BA=

Region = ;            &n= bsp;            = ;            &n= bsp;       State<= u>

7023605= 2    -ROOT-,,0.70236052 state=3DCL= OSING, ts=3DMon Apr 15 12:52:38 CST 2013 (75440s ago), server=3DMast= er,60000,1366001238313

 

How fix it?

 

Thanks.

 =

 


--001a11c1c476892a8504da715177--