Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5FAB810B1F for ; Tue, 16 Apr 2013 03:05:32 +0000 (UTC) Received: (qmail 98644 invoked by uid 500); 16 Apr 2013 03:05:27 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 98274 invoked by uid 500); 16 Apr 2013 03:05:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98253 invoked by uid 99); 16 Apr 2013 03:05:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 03:05:26 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dwld0425@gmail.com designates 209.85.160.44 as permitted sender) Received: from [209.85.160.44] (HELO mail-pb0-f44.google.com) (209.85.160.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2013 03:05:19 +0000 Received: by mail-pb0-f44.google.com with SMTP id wz12so30860pbc.17 for ; Mon, 15 Apr 2013 20:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-type:x-mailer:thread-index:content-language; bh=WaLH6UH4ciw2ws1xZRhhKXB2MsnPRIctoNwCXaZtjW8=; b=a1TUo2snbA4pwDs4N7cO8braX/IdNk22iH3NJ9cqoUp+k0e9zuiEiYJvQ+PInZ5D21 Xc5tZgNEP30MXQ1cIVIfMQxBDDzS+3rK1WjpHEtewKZDk8C+pE07xAOyneNcy5n8QPT0 LotFxhx7HxTz9OTTuiuwZxM/vzzLbED/NxpzutDbamb9SCekdi9eSitTIXT8A39PznVo avlR5apKekYBVofDDBqc/qvto1j48joOiX3uADoCE2JYAte9uZ1zab0XuC8OJH7q7ggv cnHpiQG3TfNrM629FfLolkj8sF6w+8IWyfDtv5J1aVd4Bd/5XaKegZu71bfabk66AlWT duhQ== X-Received: by 10.66.254.136 with SMTP id ai8mr1135390pad.26.1366081499363; Mon, 15 Apr 2013 20:04:59 -0700 (PDT) Received: from fangbob04581d4 ([218.94.153.146]) by mx.google.com with ESMTPS id ew5sm125558pbc.9.2013.04.15.20.04.54 (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 15 Apr 2013 20:04:58 -0700 (PDT) From: "dylan" To: References: <26da01ce3a44$ef929030$ceb7b090$@gmail.com> <27e401ce3a45$f45552f0$dcfff8d0$@gmail.com> <293801ce3a4b$6a446af0$3ecd40d0$@gmail.com> <295801ce3a4d$de3aeea0$9ab0cbe0$@gmail.com> In-Reply-To: Subject: =?gb2312?B?tPC4tDogtPC4tDogtPC4tDogtPC4tDogUmVnaW9uIGhhcyBiZWVuIA==?= =?gb2312?B?Q0xPU0lORyBmb3IgdG9vIGxvbmcsIHRoaXMgc2hvdWxkIGV2ZW50dQ==?= =?gb2312?B?YWxseSBjb21wbGV0ZSBvciB0aGUgc2VydmVyIHdpbGwgZXhwaXJlLCBzZW4=?= =?gb2312?B?ZCBSUEMgYWdhaW4=?= Date: Tue, 16 Apr 2013 11:04:50 +0800 Message-ID: <29c401ce3a4f$2beea2d0$83cbe870$@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_29C5_01CE3A92.3A1B7FC0" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHLtJjdoMOrX1da9dMMgCEZGBb8GwIXstlaAVfpxfEBj1ev4AGal63yAZY5lzoCc5eT3QFemvSDAivFfpWYa8cKMA== Content-Language: zh-cn X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_29C5_01CE3A92.3A1B7FC0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable I use hbase shell=20 =20 I always show : ERROR: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing =20 =B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 10:59 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: =B4=F0=B8=B4: =B4=F0=B8=B4: =B4=F0=B8=B4: Region has = been CLOSING for too long, this should eventually complete or the server will expire, send RPC again =20 did your hbase managed zookeeper? or did you set export HBASE_MANAGES_ZK=3Dfalse in the hbase-env.sh? =20 if not, then that's zookeeper port conflicted. =20 On Tue, Apr 16, 2013 at 10:55 AM, dylan wrote: # The number of milliseconds of each tick tickTime=3D2000 # The number of ticks that the initial=20 # synchronization phase can take initLimit=3D10 # The number of ticks that can pass between=20 # sending a request and getting an acknowledgement syncLimit=3D5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just=20 # example sakes. dataDir=3D/usr/cdh4/zookeeper/data # the port at which the clients will connect clientPort=3D2181 =20 server.1=3DSlave01:2888:3888 server.2=3DSlave02:2888:3888 server.3=3DSlave03:2888:3888 =20 =B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 10:45 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: =B4=F0=B8=B4: =B4=F0=B8=B4: Region has been CLOSING = for too long, this should eventually complete or the server will expire, send RPC again =20 and paste ZK configuration in the zookeerp_home/conf/zoo.cfg =20 On Tue, Apr 16, 2013 at 10:42 AM, Azuryy Yu wrote: it located under hbase-home/logs/ if your zookeeper is managed by = hbase. =20 but I noticed you configured QJM, then did your QJM and Hbase share the = same ZK cluster? if so, then just paste your QJM zk configuration in the hdfs-site.xml and hbase zk configuration in the hbase-site.xml. =20 On Tue, Apr 16, 2013 at 10:37 AM, dylan wrote: How to check zookeeper log?? It is the binary files, how to transform it = to normal log?=20 =20 I find the =A1=B0org.apache.zookeeper.server.LogFormatter=A1=B1, how to = run? =20 =20 =B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 10:01 =CA=D5=BC=FE=C8=CB: user@hadoop.apache.org =D6=F7=CC=E2: Re: =B4=F0=B8=B4: Region has been CLOSING for too long, = this should eventually complete or the server will expire, send RPC again =20 This is zookeeper issue. =20 please paste zookeeper log here. thanks. =20 On Tue, Apr 16, 2013 at 9:58 AM, dylan wrote: It is hbase-0.94.2-cdh4.2.0. =20 =B7=A2=BC=FE=C8=CB: Ted Yu [mailto:yuzhihong@gmail.com]=20 =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 9:55 =CA=D5=BC=FE=C8=CB: user@hbase.apache.org =D6=F7=CC=E2: Re: Region has been CLOSING for too long, this should = eventually complete or the server will expire, send RPC again =20 I think this question would be more appropriate for HBase user mailing = list. =20 Moving hadoop user to bcc. =20 Please tell us the HBase version you are using. =20 Thanks On Mon, Apr 15, 2013 at 6:51 PM, dylan wrote: Hi =20 I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes = for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN node. = ), 3 DN nodes with zookeepers, It works fine. When I reboot one data node machine which includes zookeeper, after that , restart all processes. = The hadoop works fine, but hbase not. I cannot disable tables and drop = tables. =20 The logs an follows: The Hbase HMaster log: DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign region -ROOT-,,0.70236052 but it is not currently assigned = anywhere ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: -ROOT-,,0.70236052 state=3DCLOSING, = ts=3D1366001558865, server=3DMaster,60000,1366001238313 ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has = been CLOSING for too long, this should eventually complete or the server will expire, send RPC again 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region -ROOT-,,0.70236052 (offlining) =20 The Hbase HRegionServer log: =20 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: = total=3D7.44 MB, free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0, = hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, cachingHitsRatio=3D0, = evictions=3D0, evicted=3D0, evictedPerRun=3DNaN =20 The Hbase Web show=A3=BA Region State 70236052 -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 = CST 2013 (75440s ago), server=3DMaster,60000,1366001238313 =20 How fix it? =20 Thanks. =20 =20 =20 =20 =20 ------=_NextPart_000_29C5_01CE3A92.3A1B7FC0 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

I use  hbase shell

 

I always show :

ERROR: org.apache.hadoop.ipc.RemoteException: = org.apache.hadoop.hbase.PleaseHoldException: Master is = initializing

 

=B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com] =
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 10:59
=CA=D5=BC=FE=C8=CB: = user@hadoop.apache.org
=D6=F7=CC=E2: Re: = =B4=F0=B8=B4: =B4=F0=B8=B4: =B4=F0=B8=B4: Region has been = CLOSING for too long, this should eventually complete or the server will = expire, send RPC again

 

did your hbase managed zookeeper? = or did you set export HBASE_MANAGES_ZK=3Dfalse in the = hbase-env.sh?

 

if not, then that's zookeeper port = conflicted.

 

On Tue, Apr 16, 2013 at 10:55 AM, dylan <dwld0425@gmail.com> = wrote:

# The number of milliseconds of each tick

tickTime=3D2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=3D10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=3D5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=3D/usr/cdh4/zookeeper/data

# the port at which the clients will connect

clientPort=3D2181

 

server.1=3DSlave01:2888:3888

server.2=3DSlave02:2888:3888

server.3=3DSlave03:2888:3888

 

=B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]

=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 10:45
=CA=D5=BC=FE=C8=CB: user@hadoop.apache.org
=D6=F7=CC=E2: Re: = =B4=F0=B8=B4: =B4=F0=B8=B4: Region has been CLOSING for too long, this should = eventually complete or the server will expire, send RPC = again

 

and paste ZK configuration in the = zookeerp_home/conf/zoo.cfg

 

On Tue, Apr 16, 2013 at 10:42 AM, Azuryy Yu <azuryyyu@gmail.com> = wrote:

it located under hbase-home/logs/  if your zookeeper = is managed by hbase.

 

but I noticed you configured QJM, then did your QJM and = Hbase share the same ZK cluster? if so, then just paste your QJM zk = configuration in the hdfs-site.xml and hbase zk configuration in the = hbase-site.xml.

 

On Tue, Apr 16, 2013 at 10:37 AM, dylan <dwld0425@gmail.com> = wrote:

How to check = zookeeper log?? It is the = binary files, how to transform it = to normal log?

 

I find the =A1=B0org.apache.zookeeper.server.LogF= ormatter=A1=B1, how to run?

 <= span lang=3DEN-US>

 

=B7=A2=BC=FE=C8=CB: Azuryy Yu [mailto:azuryyyu@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 10:01
=CA=D5=BC=FE=C8=CB: user@hadoop.apache.org
=D6=F7=CC=E2: Re: = =B4=F0=B8=B4: Region has been CLOSING for too = long, this should eventually complete or the server will expire, send = RPC again

 

This is zookeeper issue.

 

please paste zookeeper log here. = thanks.

 

On Tue, Apr 16, 2013 at 9:58 AM, dylan <dwld0425@gmail.com> = wrote:

It is hbase-0.94.2-cdh4.2.0.

 

=B7=A2=BC=FE=C8=CB: Ted Yu [mailto:yuzhihong@gmail.com]
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA4=D4=C216=C8=D5 = 9:55
=CA=D5=BC=FE=C8=CB: user@hbase.apache.org
=D6=F7=CC=E2: Re: Region has been = CLOSING for too long, this should eventually complete or the server will = expire, send RPC again

 

I think this question would be more appropriate for HBase = user mailing list.

 

Moving hadoop user to = bcc.

 

Please tell us the HBase version you are = using.

 

Thanks

On Mon, Apr 15, 2013 at 6:51 PM, dylan <dwld0425@gmail.com> = wrote:

Hi

 

I am a newer for hadoop, and set up hadoop with tarball . I = have 5 nodes for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of = them on DN node.  ), 3 DN nodes with zookeepers,  It works = fine.  When I reboot one data node machine which includes = zookeeper, after that , restart all processes. The = hadoop works fine, but hbase not. I cannot disable tables and drop = tables.

 

The logs an = follows:

The Hbase = HMaster log:

DEBUG = org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign = region -ROOT-,,0.70236052 but it is not currently assigned = anywhere

,683 INFO = org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition = timed out:  -ROOT-,,0.70236052 state=3DCLOSING, ts=3D1366001558865, = server=3DMaster,60000,1366001238313

,683 INFO = org.apache.hadoop.hbase.master.AssignmentManager: Region has been = CLOSING for too long, this should eventually complete or the server will = expire, send RPC again

10,684 = DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting = unassignment of region -ROOT-,,0.70236052 (offlining)

 

The Hbase = HRegionServer log:

 

DEBUG = org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D7.44 MB, = free=3D898.81 MB, max=3D906.24 MB, blocks=3D0, accesses=3D0, hits=3D0, = hitRatio=3D0, cachingAccesses=3D0, cachingHits=3D0, = cachingHitsRatio=3D0, evictions=3D0, evicted=3D0, = evictedPerRun=3DNaN

 

The Hbase = Web show=A3=BA

Region        = ;            =             &= nbsp;           &n= bsp; State

70236052    -ROOT-,,0.70236052 state=3DCLOSING, ts=3DMon Apr 15 12:52:38 CST = 2013 (75440s ago), = server=3DMaster,60000,1366001238313

 

How fix = it?

 

Thanks.

 

 

 

 

 

------=_NextPart_000_29C5_01CE3A92.3A1B7FC0--