hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: load balancing considerations
Date Wed, 11 Aug 2010 10:09:58 GMT
Here is client side stack trace:

java.io.IOException: Call to
us01-ciqps1-grid01.carrieriq.com/10.32.42.233:60020 failed on local
exception: java.io.EOFException
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused

        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1222)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1144)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230)
        at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
        at
com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flushAllStripesNew(StripedHBaseTable.java:300)

On Tue, Aug 10, 2010 at 11:01 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> Use a tool like Yourkit to grovel that heap, the open source tools are
> not really there yet.
>
> But your stack trace tells a lot.... the fatal allocation is in the
> RPC layer.  Either a client is sending a massive value, or you have a
> semi-hostile network client sending bytes to your open socket which
> are being interpreted as the buffer size to allocate.  If you look at
> the actual RPC code (any RPC code really) there is often a 'length'
> field which is then used to allocate a dynamic buffer.
>
> -ryan
>
> On Tue, Aug 10, 2010 at 10:55 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > The compressed file is still big:
> > -rw-r--r-- 1 hadoop users  809768340 Aug 11 05:49 java_pid26972.hprof.gz
> >
> > If you can tell me specific things to look for in the dump, I would
> collect
> > it (through jhat) and publish.
> >
> > Thanks
> >
> > On Tue, Aug 10, 2010 at 10:29 PM, Stack <stack@duboce.net> wrote:
> >
> >> On Tue, Aug 10, 2010 at 9:52 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> > Here are GC-related parameters:
> >> > /usr/java/jdk1.6/bin/java -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError
> >> > -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> >> >
> >>
> >> You have > 2 CPUs per machine I take it?  You could probably drop the
> >> conservative XX:+CMSIncrementalMode.
> >>
> >> > The heap dump is big:
> >> > -rw------- 1 hadoop users 4146551927 Aug 11 03:59 java_pid26972.hprof
> >> >
> >> > Do you have ftp server where I can upload it ?
> >> >
> >>
> >> Not really.  I was hoping you could put a compressed version under an
> >> http server somewhere that I could pull from.  You might as well
> >> include the GC log while you are at it.
> >>
> >> Thanks Ted,
> >>
> >> St.Ack
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message