hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Error while doing multi get from HBase
Date Fri, 12 Apr 2013 14:33:11 GMT
your CMS is not tuned, please find how tune CMS on java web site.

then Xmn is too small, this is not suitable for frequent multi get. please
change Xmn=1g

--Send from my Sony mobile.
On Apr 12, 2013 10:29 PM, "anand nalya" <a.nalya@computer.org> wrote:

> the block cache size is 0.25
>
> Each row holds around 2KB data, so size should not be an issue at least
> till the number of records is less than 1000.
>
> Also,
> HBASE_HEAPSIZE=8000
> HBASE_OPTS="-XX:+UseConcMarkSweepGC"
> HBASE_REGIONSERVER_OPTS="-Xmx4g -Xms4g -Xmn256m -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> On 12 April 2013 19:30, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
> > and whats your block cache size?
> >
> > there are two possible reasons:
> > 1. result is too big
> > 2. GC options are not optimized.
> >
> > can you paste your gc options here?
> >
> > --Send from my Sony mobile.
> > On Apr 12, 2013 9:53 PM, "anand nalya" <a.nalya@computer.org> wrote:
> >
> > > Hi Azuryy,
> > >
> > > I'm using the default cache size of 100 for scanner. For mutigets, I've
> > > tried with 1 (13ms), 10(356ms), 100(1135ms), 1000(4330ms), and
> > > 10000(17744ms) keys. Normal workload will be around 10000 keys at a
> time.
> > >
> > > Are there any optimization that can be done for multigets. Is HBase a
> > good
> > > candidate for usecase?
> > >
> > > Thanks,
> > > Anand
> > >
> > >
> > > On 12 April 2013 19:17, anand nalya <a.nalya@computer.org> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > The region servers are not loaded. It is showing 5% CPU usage. The
> > > > datanode is showing around 50% CPU utilization. disk IO is aroung
> > 7Mbps.
> > > >
> > > > There is nothing noticeable in GC log.
> > > >
> > > > Thanks,
> > > > Anand
> > > >
> > > >
> > > > On 12 April 2013 02:56, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > >> How loaded were the region servers when the query was running ?
> > > >>
> > > >> Did you check GC log ?
> > > >>
> > > >> Thanks
> > > >>
> > > >> On Thu, Apr 11, 2013 at 8:23 AM, anand nalya <anand.nalya@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I'm using HBase 0.94.5 with thrift server. I'm trying to get
the
> > rows
> > > >> from
> > > >> > HBase using
> > > >> >
> > > >>
> > >
> org.apache.hadoop.hbase.thrift.generated.Hbase.Client.getRows(ByteBuffer,
> > > >> > List<ByteBuffer>, Map<ByteBuffer, ByteBuffer>) but
it is giving
> > > results
> > > >> > very slowly (around 2 mins for 100 rows). For larger number of
> > > records,
> > > >> > there is no response.
> > > >> >
> > > >> > I've two region server and a total of 128 regions. Total data
size
> > is
> > > >> > around 250GB (250 million records) uniformly distributed across
> > > regions.
> > > >> >
> > > >> > Regionserver only show the following in its log:
> > > >> >
> > > >> > 2013-04-11 19:53:44,535 ERROR
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting
> > call
> > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc
> > > >> version=1,
> > > >> > client version=29, methodsFingerPrint=-1368823753 from
> > > >> > 192.168.145.195:52277after 74994 ms, since caller disconnected
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516)
> > > >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > >> >     at
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > > >> > 2013-04-11 19:53:46,121 ERROR
> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting
> > call
> > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc
> > > >> version=1,
> > > >> > client version=29, methodsFingerPrint=-1368823753 from
> > > >> > 192.168.145.195:52277after 76580 ms, since caller disconnected
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576)
> > > >> >     at
> > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516)
> > > >> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > >> >     at java.lang.reflect.Method.invoke(Method.java:597)
> > > >> >     at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > >> >     at
> > > >> >
> > > >>
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > > >> >
> > > >> >
> > > >> > Any idea what might be wrong here?
> > > >> >
> > > >> > Thanks,
> > > >> > Anand
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message