Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 336DD10AE2 for ; Tue, 16 Apr 2013 03:00:53 +0000 (UTC) Received: (qmail 78122 invoked by uid 500); 16 Apr 2013 03:00:47 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 78015 invoked by uid 500); 16 Apr 2013 03:00:47 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 78007 invoked by uid 99); 16 Apr 2013 03:00:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 03:00:47 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dwld0425@gmail.com designates 209.85.160.50 as permitted sender) Received: from [209.85.160.50] (HELO mail-pb0-f50.google.com) (209.85.160.50) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 03:00:41 +0000 Received: by mail-pb0-f50.google.com with SMTP id jt11so29000pbb.9 for ; Mon, 15 Apr 2013 20:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-type:x-mailer:thread-index:content-language; bh=CeQ06Uz9sR9Z0gRVMw6Phq5bG7IAR1DZ0YibB3DdSVk=; b=EtPmJprEX8CPvTizwX7UQnHhEWeFyvmMNVgvl3EI4UZAmY+gC7CM2EAIEYn0fHh7SI ccUSRzF2h4M676hvcRc0d/C2CR8tyWvMCkIC1hIdxuhfInFnVQamkYIxKLo9o95d23a5 2YebPk3ukop2jwC7VBHG7ytio791DqsH8NOUvmcsGb0Uj9NpuHpIl1KwJQfg/DTJuAhF ZLJQBNyfHrM4/L5u9Si4zLz7zhFCsp+INtp8t6rtVr1txwRF3x7GBzcseLLrPNbKG46y ELkG62UpDLvDR6+9eV+oi/3m6XGFzgbeBzYFE6v0BEYBtS4X3qGS6quw6K7Vfb8LmhPU 9LgA== X-Received: by 10.68.7.70 with SMTP id h6mr534993pba.77.1366081220875; Mon, 15 Apr 2013 20:00:20 -0700 (PDT) Received: from fangbob04581d4 ([218.94.153.146]) by mx.google.com with ESMTPS id ew5sm110021pbc.9.2013.04.15.20.00.17 (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 15 Apr 2013 20:00:19 -0700 (PDT) From: "dylan" To: References: <26da01ce3a44$ef929030$ceb7b090$@gmail.com> <27e401ce3a45$f45552f0$dcfff8d0$@gmail.com> <293801ce3a4b$6a446af0$3ecd40d0$@gmail.com> In-Reply-To: Subject: =?gb2312?B?tPC4tDogtPC4tDogtPC4tDogUmVnaW9uIGhhcyBiZWVuIENMT1NJTg==?= =?gb2312?B?RyBmb3IgdG9vIGxvbmcsIHRoaXMgc2hvdWxkIGV2ZW50dWFsbHkgYw==?= =?gb2312?B?b21wbGV0ZSBvciB0aGUgc2VydmVyIHdpbGwgZXhwaXJlLCBzZW5kIFJQQyA=?= =?gb2312?B?YWdhaW4=?= Date: Tue, 16 Apr 2013 11:00:14 +0800 Message-ID: <296501ce3a4e$86dfd840$949f88c0$@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_2966_01CE3A91.9507D330" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHLtJjdoMOrX1da9dMMgCEZGBb8GwIXstlaAVfpxfEBj1ev4AGal63yAZY5lzoCc5eT3ZiIGLYA Content-Language: zh-cn X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_2966_01CE3A91.9507D330 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable I use hbase hbck =A8Cfix to fix hbase . It show : RecoverableZooKeeper: The identifier of this process is 11286@Master 13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Opening socket connection = to server Slave02/192.168.75.243:2181. Will not attempt to authenticate = using SASL (Unable to locate a login configuration) 13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Socket connection = established to Slave02/192.168.75.243:2181, initiating session 13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Session establishment = complete on server Slave02/192.168.75.243:2181, sessionid =3D 0x23e0dc5a333000b, negotiated timeout =3D 40000 =20 =B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 10:45 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: =B4=F0=B8=B4: =B4=F0=B8=B4: Region has been CLOSING = for too long, this should eventually complete or the server will expire, send RPC again =20 and paste ZK configuration in the zookeerp_home/conf/zoo.cfg =20 On Tue, Apr 16, 2013 at 10:42 AM, Azuryy Yu wrote: it located under hbase-home/logs/ if your zookeeper is managed by = hbase. =20 but I noticed you configured QJM, then did your QJM and Hbase share the = same ZK cluster? if so, then just paste your QJM zk configuration in the hdfs-site.xml and hbase zk configuration in the hbase-site.xml. =20 On Tue, Apr 16, 2013 at 10:37 AM, dylan wrote: How to check zookeeper log?? It is the binary files, how to transform it = to normal log?=20 =20 I find the =A1=B0org.apache.zookeeper.server.LogFormatter=A1=B1, how to = run? =20 =20 =B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 10:01 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: =B4=F0=B8=B4: Region has been CLOSING for too long, = this should eventually complete or the server will expire, send RPC again =20 This is zookeeper issue. =20 please paste zookeeper log here. thanks. =20 On Tue, Apr 16, 2013 at 9:58 AM, dylan wrote: It is hbase-0.94.2-cdh4.2.0. =20 =B7=A2=BC=FE=C8=CB: Ted Yu [mailto:yuzhihong@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 9:55 =CA=D5=BC=FE=C8=CB: user@hbase.apache.org =D6=F7=CC=E2: Re: Region has been CLOSING for too long, this should = eventually complete or the server will expire, send RPC again =20 I think this question would be more appropriate for HBase user mailing = list. =20 Moving hadoop user to bcc. =20 Please tell us the HBase version you are using. =20 Thanks On Mon, Apr 15, 2013 at 6:51 PM, dylan wrote: Hi =20 I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes = for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN node. = ), 3 DN nodes with zookeepers, It works fine. When I reboot one data node machine which includes zookeeper, after that , restart all processes. = The hadoop works fine, but hbase not. I cannot disable tables and drop = tables. =20 The logs an follows: The Hbase HMaster log: DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign region -ROOT-,,0.70236052 but it is not currently assigned = anywhere ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,,0.70236052 state=3DCLOSING, = ts=3D1366001558865, server=3DMaster,60000,1366001238313 ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has = been CLOSING for too long, this should eventually complete or the server will expire, send RPC again 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region -ROOT-,,0.70236052 (offlining) =20 The Hbase HRegionServer log: =20 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: = total=3D7.44 MB, free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0, = hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, cachingHitsRatio=3D0, = evictions=3D0, evicted=3D0, evictedPerRun=3DNaN =20 The Hbase Web show=A3=BA Region State 70236052 -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 = CST 2013 (75440s ago), server=3DMaster,60000,1366001238313 =20 How fix it? =20 Thanks. =20 =20 =20 =20 ------=_NextPart_000_2966_01CE3A91.9507D330 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

I use hbase hbck =A8Cfix to fix hbase .

It show :

RecoverableZooKeeper: The identifier of this process is = 11286@Master

13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Opening socket = connection to server Slave02/192.168.75.243:2181. Will not attempt to = authenticate using SASL (Unable to locate a login = configuration)

13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Socket connection = established to Slave02/192.168.75.243:2181, initiating = session

13/04/16 10:58:34 INFO zookeeper.ClientCnxn: Session establishment = complete on server Slave02/192.168.75.243:2181, sessionid =3D = 0x23e0dc5a333000b, negotiated timeout =3D 40000

 

=B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com] =
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 10:45
=CA=D5=BC=FE=C8=CB: = user@hadoop.apache.org
=D6=F7=CC=E2: Re: = =B4=F0=B8=B4: =B4=F0=B8=B4: Region has been CLOSING for too long, this should = eventually complete or the server will expire, send RPC = again

 

and paste ZK configuration in the = zookeerp_home/conf/zoo.cfg

 

On Tue, Apr 16, 2013 at 10:42 AM, Azuryy Yu <azuryyyu@gmail.com> = wrote:

it located under hbase-home/logs/  if your zookeeper = is managed by hbase.

 

but I noticed you configured QJM, = then did your QJM and Hbase share the same ZK cluster? if so, then just = paste your QJM zk configuration in the hdfs-site.xml and hbase zk = configuration in the = hbase-site.xml.

 

On Tue, Apr 16, 2013 at 10:37 AM, dylan <dwld0425@gmail.com> = wrote:

How to check = zookeeper log?? It is the = binary files, how to transform it = to normal log?

 

I find the =A1=B0org.apache.zookeeper.server.LogF= ormatter=A1=B1, how to run?

 <= span lang=3DEN-US>

 

=B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 10:01
=CA=D5=BC=FE=C8=CB: user@hadoop.apache.org
=D6=F7=CC=E2: Re: = =B4=F0=B8=B4: Region has been CLOSING for too = long, this should eventually complete or the server will expire, send = RPC again

 

This is zookeeper issue.

 

please paste zookeeper log here. = thanks.

 

On Tue, Apr 16, 2013 at 9:58 AM, dylan <dwld0425@gmail.com> = wrote:

It is hbase-0.94.2-cdh4.2.0.

 

=B7=A2=BC=FE=C8=CB: Ted Yu [mailto:yuzhihong@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 9:55
=CA=D5=BC=FE=C8=CB: user@hbase.apache.org
=D6=F7=CC=E2: Re: Region has been = CLOSING for too long, this should eventually complete or the server will = expire, send RPC again

 

I think this question would be more appropriate for HBase = user mailing list.

 

Moving hadoop user to = bcc.

 

Please tell us the HBase version you are = using.

 

Thanks

On Mon, Apr 15, 2013 at 6:51 PM, dylan <dwld0425@gmail.com> = wrote:

Hi

 

I am a newer for hadoop, and set up hadoop with tarball . I = have 5 nodes for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of = them on DN node.  ), 3 DN nodes with zookeepers,  It works = fine.  When I reboot one data node machine which includes = zookeeper, after that , restart all processes. The = hadoop works fine, but hbase not. I cannot disable tables and drop = tables.

 

The logs an = follows:

The Hbase = HMaster log:

DEBUG = org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign = region -ROOT-,,0.70236052 but it is not currently assigned = anywhere

,683 INFO = org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition = timed out:  -ROOT-,,0.70236052 state=3DCLOSING, ts=3D1366001558865, = server=3DMaster,60000,1366001238313

,683 INFO = org.apache.hadoop.hbase.master.AssignmentManager: Region has been = CLOSING for too long, this should eventually complete or the server will = expire, send RPC again

10,684 = DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting = unassignment of region -ROOT-,,0.70236052 (offlining)

 

The Hbase = HRegionServer log:

 

DEBUG = org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D7.44 MB, = free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0, = hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, = cachingHitsRatio=3D0, evictions=3D0, evicted=3D0, = evictedPerRun=3DNaN

 

The Hbase = Web show=A3=BA

Region        = ;            =             &= nbsp;           &n= bsp; State

70236052    -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 CST = 2013 (75440s ago), = server=3DMaster,60000,1366001238313

 

How fix = it?

 

Thanks.

 

 

 

 

------=_NextPart_000_2966_01CE3A91.9507D330--