Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of julian.zhou@me.com designates
 17.158.161.2 as permitted sender)
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_IMpu4JFXhw1QtpGzUy6TYw)"
To: user@hbase.apache.org
Cc: dev@hbase.apache.org, user@hbase.apache.org, issues@hbase.apache.org
From: Julian Zhou <julian.zhou@me.com>
Subject: Re: HConnectionImplementation.listTables and list table exception
Date: Wed, 23 Oct 2013 13:16:18 +0000 (GMT)
Message-id: <d487ac7c-8ab0-4632-87e3-a2185b1e1fc8@me.com>
In-reply-to: 
 <CAC0sRsXgpUJAnGJG4pdFtXGrv=he-wrCmS12ZkBELz6p8sF4NA@mail.gmail.com>

--Boundary_(ID_IMpu4JFXhw1QtpGzUy6TYw)
Content-type: text/plain; charset=utf-8; format=flowed
Content-transfer-encoding: quoted-printable

Hello Michelle,
   How many regions totally are there in your 600 nodes cluster? Looks lik=
e many of them are pending for open and being assigned to region servers.
Can you see many items under zookeeper dir /hbase/unassigned?
=EF=BB=BF
You would like to refer http://blog.sina.com.cn/s/blog_4a1f59bf01018tu4.ht=
ml?

Best Regards, Julian


On Oct 23, 2013, at 01:46 PM, =E5=BC=A0=E8=8E=89=E8=8B=B9 <zlpmichelle@gma=
il.com> wrote:

> Dear HBase dev and users,
>
> Did you meet this
> "org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa=
tion.listTables"
> issue?
>
> We setup a 600 nodes cluster, 9 zookeeper nodes to load data into hbase,
> but it seemed hbase master was busy handling transition with zookeeper, =
and
> hbase =E2=80=9Clist=E2=80=9D could not get response. The hbase table was=
 created but it
> didn't do any insert.
>
> Do you have any idea of the root cause and how to fix it? :)Highly
> appreciate for your answers!
>
>
>
> Here is the exception stack:
> ---------------------------------------------------
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy7.getHTableDescriptors(Unknown Source)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementat=
ion.listTables(HConnectionManager.java:2237)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:317=
)
>
>
>
>
> hbase master log:
>
> -----------------------------
>
> 2013-10-18 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign=
:
> master:60000-0x341be88202300ab* Deleting existing unassigned node* for
> 0ec3308bd1e2bdd9576b2d60d2eee68e that is in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,279 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager:* Handling
> transition=3DRS_ZK_REGION_OPENING*, s*erver=3Dnode0878*.
> ic.analyticsworkbench.com,60020,1381883086785,
> region=3D15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds=
 late
>
> 2013-10-18 06:19:41,280 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: *Handling
> transition=3DRS_ZK_REGION_OPENING,
> server=3Dnode0898*.ic.analyticsworkbench.com,60020,1381883200494,
> region=3D1a4c929534e6828c85f22b062f949304, which is more than 15 seconds=
 late
>
> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign=
:
> master:60000-0x341be88202300ab Successfully *deleted unassigned node *fo=
r
> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected state
> RS_ZK_REGION_OPENED
>
> 2013-10-18 06:19:41,289 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=3DRS_ZK_REGION_OPENING,
> server=3Dnode0693.ic.analyticsworkbench.com,60020,1381881773670,
> region=3Dd47bfe1af0051c405de295a51c1c6e63, which is more than 15 seconds=
 late
>
>
>
> We also try to "list" in hbase shell,it also failed:
>
> The hbase =E2=80=9Clist=E2=80=9D got error as:
>
> ------------------------------------------
>
>
>
> hbase(main):001:0> list
>
> TABLE
>
>
>
>
> ERROR: java.lang.reflect.UndeclaredThrowableException: Call to
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000 failed on socket
> timeout exception: java.net.SocketTimeoutException: 120000 millis timeou=
t
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=3D/10.1.50.15:45726 remo=
te=3D
> node0997.ic.analyticsworkbench.com/10.1.50.17:60000]
>
>
>
>
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (Li Ping Zhang)

--Boundary_(ID_IMpu4JFXhw1QtpGzUy6TYw)
Content-type: multipart/related;
 boundary="Boundary_(ID_Nnw8Y7S/jYlGQbVIhg94nA)"; type="text/html"


--Boundary_(ID_Nnw8Y7S/jYlGQbVIhg94nA)
Content-type: text/html; charset=utf-8
Content-transfer-encoding: quoted-printable

<html><body><div>Hello Michelle,<br>&nbsp;&nbsp; How many regions totally are there in=
 your 600 nodes cluster? Looks like many of them are pending for open and =
being assigned to region servers.<br>Can you see many items under zookeepe=
r dir /hbase/unassigned?<br><br>You would like to refer http://blog.sina.c=
om.cn/s/blog_4a1f59bf01018tu4.html?<br></div><div><pre style=3D"font-famil=
y: HelveticaNeue, Helvetica, sans-serif; font-size: 13px; white-space: pre=
-wrap" data-mce-style=3D"font-family: HelveticaNeue, Helvetica, sans-serif=
; font-size: 13px; white-space: pre-wrap;">Best Regards, Julian</pre></div=
><div><br>On Oct 23, 2013, at 01:46 PM, =E5=BC=A0=E8=8E=89=E8=8B=B9 &lt;zl=
pmichelle@gmail.com&gt; wrote:<br><br><div><blockquote type=3D"cite"><div =
class=3D"msg-quote"><div class=3D"_stretch">Dear HBase dev and users,<br> =
<br> Did you meet this<br> "org.apache.hadoop.hbase.client.HConnectionMana=
ger$HConnectionImplementation.listTables"<br> issue?<br> <br> We setup a 6=
00 nodes cluster, 9 zookeeper nodes to load data into hbase,<br> but it se=
emed hbase master was busy handling transition with zookeeper, and<br> hba=
se =E2=80=9Clist=E2=80=9D could not get response. The hbase table was crea=
ted but it<br> didn't do any insert.<br> <br> Do you have any idea of the =
root cause and how to fix it? :)Highly<br> appreciate for your answers!<br=
> <br> <br> <br> Here is the exception stack:<br> ------------------------=
---------------------------<br> java.lang.reflect.UndeclaredThrowableExcep=
tion<br> at $Proxy7.getHTableDescriptors(Unknown Source)<br> at<br> org.ap=
ache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.list=
Tables(HConnectionManager.java:2237)<br> at<br> org.apache.hadoop.hbase.cl=
ient.HBaseAdmin.listTables(HBaseAdmin.java:317)<br> <br> <br> <br> <br> hb=
ase master log:<br> <br> -----------------------------<br> <br> 2013-10-18=
 06:19:41,279 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:<br> master=
:60000-0x341be88202300ab* Deleting existing unassigned node* for<br> 0ec33=
08bd1e2bdd9576b2d60d2eee68e that is in expected state<br> RS_ZK_REGION_OPE=
NED<br> <br> 2013-10-18 06:19:41,279 DEBUG<br> org.apache.hadoop.hbase.mas=
ter.AssignmentManager:* Handling<br> transition=3DRS_ZK_REGION_OPENING*, s=
*erver=3Dnode0878*.<br> ic.analyticsworkbench.com,60020,1381883086785,<br>=
 region=3D15a4fb29aa1d905b13f33594e50bc8de, which is more than 15 seconds =
late<br> <br> 2013-10-18 06:19:41,280 DEBUG<br> org.apache.hadoop.hbase.ma=
ster.AssignmentManager: *Handling<br> transition=3DRS_ZK_REGION_OPENING,<b=
r> server=3Dnode0898*.ic.analyticsworkbench.com,60020,1381883200494,<br> r=
egion=3D1a4c929534e6828c85f22b062f949304, which is more than 15 seconds la=
te<br> <br> 2013-10-18 06:19:41,289 DEBUG org.apache.hadoop.hbase.zookeepe=
r.ZKAssign:<br> master:60000-0x341be88202300ab Successfully *deleted unass=
igned node *for<br> region 0ec3308bd1e2bdd9576b2d60d2eee68e in expected st=
ate<br> RS_ZK_REGION_OPENED<br> <br> 2013-10-18 06:19:41,289 DEBUG<br> org=
.apache.hadoop.hbase.master.AssignmentManager: Handling<br> transition=3DR=
S_ZK_REGION_OPENING,<br> server=3Dnode0693.ic.analyticsworkbench.com,60020=
,1381881773670,<br> region=3Dd47bfe1af0051c405de295a51c1c6e63, which is mo=
re than 15 seconds late<br> <br> <br> <br> We also try to "list" in hbase =
shell,it also failed:<br> <br> The hbase =E2=80=9Clist=E2=80=9D got error =
as:<br> <br> ------------------------------------------<br> <br> <br> <br>=
 hbase(main):001:0&gt; list<br> <br> TABLE<br> <br> <br> <br> <br> ERROR: =
java.lang.reflect.UndeclaredThrowableException: Call to<br> node0997.ic.an=
alyticsworkbench.com/10.1.50.17:60000 failed on socket<br> timeout excepti=
on: <a href=3D"http://java.net" data-mce-href=3D"http://java.net">java.net=
</a>.SocketTimeoutException: 120000 millis timeout<br> while waiting for c=
hannel to be ready for read. ch :<br> java.nio.channels.SocketChannel[conn=
ected local=3D/10.1.50.15:45726 remote=3D<br> node0997.ic.analyticsworkben=
ch.com/10.1.50.17:60000]<br> <br> <br> <br> <br> Cheers,<br> -----<br> Big=
 Data - Big Wisdom - Big Value<br> --------------<br> Michelle Zhang (Li P=
ing Zhang)<br></div></div></blockquote></div></div></body></html>=

--Boundary_(ID_Nnw8Y7S/jYlGQbVIhg94nA)--

--Boundary_(ID_IMpu4JFXhw1QtpGzUy6TYw)--