hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Demai Ni <nid...@gmail.com>
Subject Re: HBase UI and Shell not responsive (java.net.SocketTimeoutException: 600000 millis timeout)
Date Mon, 26 May 2014 17:20:16 GMT
hi, Esteban,

thanks for the quick response.

We have a load job running at the time, I believe the task was both high
CPU and IO, and may cause some hotspot. Well, it wasn't my cluster, so I
don't have the first hand info. But from the past experience, we may have
one hot regionServer with quite some small regions on it.

"how many RPC handlers have you configured for the master.."
is it for hbase.regionserver.handler.count? it was set as 30.

I will get a jstack the next time

Demai






On Mon, May 26, 2014 at 10:08 AM, Esteban Gutierrez <esteban@cloudera.com>wrote:

> Hello Demai,
>
> Have you seen any thing else going on the master, e.g. high cpu load? Can
> you try to get a jstack of the HBase master next time you experience this
> issue? Also how many regions you have in this cluster and how many RPC
> handlers have you configured for the master?
>
> regards,
> esteban.
>
>
> --
> Cloudera, Inc.
>
>
>
> On Mon, May 26, 2014 at 9:41 AM, Demai Ni <nidmgg@gmail.com> wrote:
>
> > hi,
> >
> > HBase version is 0.96.
> > Occasionally, a large cluster got the time out(stacktrace at the end of
> the
> > email) when access UI or Hbase shell. HBase Master seems working fine as
> a
> > load job was running and eventually completed successfully.
> >
> > I did some google and found a suggestion to increase hbase.rpc.timeout.
> So
> > my question is, besides change hbase.rpc.time to 120 sec or longer, any
> > other possiblity to cause this issue. or any other configuration to tune
> >
> > BTW, I found a good blog here:
> > http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html. Any
> > detailed suggestion of best practice around these configuration? thanks
> >
> > Demai
> >
> >
> > Time out stack  in pastin : http://pastebin.com/edtATYrw
> >
> > HTTP ERROR 500
> > Problem accessing /master-status. Reason:
> >     Call to hdperf001.svl.ibm.com/9.30.75.10:60000 failed because
> > java.net.SocketTimeoutException: 600000 millis timeout while waiting for
> > channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/9.30.75.10:1047 remote=
> > hdperf001.svl.ibm.com/9.30.75.10:60000]
> > Caused by:
> > java.net.SocketTimeoutException: Call to
> > hdperf001.svl.ibm.com/9.30.75.10:60000 failed because
> > java.net.SocketTimeoutException: 600000 millis timeout while waiting for
> > channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/9.30.75.10:1047 remote=
> > hdperf001.svl.ibm.com/9.30.75.10:60000]
> > at
> org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1478)
> > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1453)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message