Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of azuryyyu@gmail.com designates
 209.85.223.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKmPmZ-k1GkttCnHzLSkP8jTjC1ouEeWfMg-K1Y71aDTU-5bFQ@mail.gmail.com>
References: 
 <CAKmPmZ-gCtrBVK75pGytW0mRMxrEAbt_eVbbrACCETWWrNUP1g@mail.gmail.com>
	<CALte62xsd1Yf72D0Aae7s4gwuHQMxPD9NH7ji9gvi47OuEGaWQ@mail.gmail.com>
	<CAKmPmZ-ktQJTOA+V68jbyhiXPucjCeqjvsAEkbY_KFPtGiEMvA@mail.gmail.com>
	<CAKmPmZ8S2-hDYsH1J8HwZ+CjHPo0uWVhfJNKvVyZLwNnJC1tYg@mail.gmail.com>
	<CALr1C9rrAWu8f7P33yNutiC14FFcPEOyFx9976oR-7ptjNN-Wg@mail.gmail.com>
	<CAKmPmZ-k1GkttCnHzLSkP8jTjC1ouEeWfMg-K1Y71aDTU-5bFQ@mail.gmail.com>
Date: Fri, 12 Apr 2013 22:33:11 +0800
Message-ID: 
 <CALr1C9p+EFDC5LV4FxzyjWUBMBagA-U33TSjnYcuUimz7e+VPQ@mail.gmail.com>
Subject: Re: Error while doing multi get from HBase
From: Azuryy Yu <azuryyyu@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b10c847578bb004da2ac814

--047d7b10c847578bb004da2ac814
Content-Type: text/plain; charset=ISO-8859-1

your CMS is not tuned, please find how tune CMS on java web site.

then Xmn is too small, this is not suitable for frequent multi get. please
change Xmn=1g

--Send from my Sony mobile.
On Apr 12, 2013 10:29 PM, "anand nalya" <a.nalya@computer.org> wrote:

> the block cache size is 0.25
>
> Each row holds around 2KB data, so size should not be an issue at least
> till the number of records is less than 1000.
>
> Also,
> HBASE_HEAPSIZE=8000
> HBASE_OPTS="-XX:+UseConcMarkSweepGC"
> HBASE_REGIONSERVER_OPTS="-Xmx4g -Xms4g -Xmn256m -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> On 12 April 2013 19:30, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
> > and whats your block cache size?
> >
> > there are two possible reasons:
> > 1. result is too big
> > 2. GC options are not optimized.
> >
> > can you paste your gc options here?
> >
> > --Send from my Sony mobile.
> > On Apr 12, 2013 9:53 PM, "anand nalya" <a.nalya@computer.org> wrote:
> >
> > > Hi Azuryy,
> > >
> > > I'm using the default cache size of 100 for scanner. For mutigets, I've
> > > tried with 1 (13ms), 10(356ms), 100(1135ms), 1000(4330ms), and
> > > 10000(17744ms) keys. Normal workload will be around 10000 keys at a
> time.
> > >
> > > Are there any optimization that can be done for multigets. Is HBase a
> > good
> > > candidate for usecase?
> > >
> > > Thanks,
> > > Anand
> > >
> > >
> > > On 12 April 2013 19:17, anand nalya <a.nalya@computer.org> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > The region servers are not loaded. It is showing 5% CPU usage. The
> > > > datanode is showing around 50% CPU utilization. disk IO is aroung
> > 7Mbps.
> > > >
> > > > There is nothing noticeable in GC log.
> > > >
> > > > Thanks,
> > > > Anand
> > > >
> > > >
> > > > On 12 April 2013 02:56, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > >> How loaded were the region servers when the query was running ?
> > > >>
> > > >> Did you check GC log ?
> > > >>
> > > >> Thanks
> > > >>
> > > >> On Thu, Apr 11, 2013 at 8:23 AM, anand nalya <anand.nalya@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I'm using HBase 0.94.5 with thrift server. I'm trying to get the
> > rows
> > > >> from
> > > >> > HBase using
> > > >> >
> > > >>
> > >
> org.apache.hadoop.hbase.thrift.generated.Hbase.Client.getRows(ByteBuffer,
> > > >> > List<ByteBuffer>, Map<ByteBuffer, ByteBuffer>) but it is giving
> > > results
> > > >> > very slowly (around 2 mins for 100 rows). For larger number of
> > > records,
> > > >> > there is no response.
> > > >> >
> > > >> > I've two region server and a total of 128 regions. Total data size
> > is
> > > >> > around 250GB (250 million records) uniformly distributed across
> > > regions.
> > > >> >
> > > >> > Regionserver only show the following in its log:
> > > >> >
> > > >> > 2013-04-11 19:53:44,535 ERROR
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting
> > call
> > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc
> > > >> version=1,
> > > >> > client version=29, methodsFingerPrint=-1368823753 from
> > > >> > 192.168.145.195:52277after 74994 ms, since caller disconnected
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516)
> > > >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > >> >     at
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > > >> > 2013-04-11 19:53:46,121 ERROR
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting
> > call
> > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc
> > > >> version=1,
> > > >> > client version=29, methodsFingerPrint=-1368823753 from
> > > >> > 192.168.145.195:52277after 76580 ms, since caller disconnected
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516)
> > > >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > >> >     at
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > > >> >
> > > >> >
> > > >> > Any idea what might be wrong here?
> > > >> >
> > > >> > Thanks,
> > > >> > Anand
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

--047d7b10c847578bb004da2ac814--