Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.xkwang1@gmail.com
 designates 209.85.215.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <007701cdb5cc$ebbb9f30$c332dd90$@com>
References: <002c01cdb586$19509910$4bf1cb30$@com>
	<F4FBBA80-7EDA-4C67-AEAC-0953657D08FF@gmail.com>
	<005301cdb5aa$59a4e920$0ceebb60$@com>
	<007701cdb5cc$ebbb9f30$c332dd90$@com>
Date: Tue, 30 Oct 2012 09:52:28 +0800
Message-ID: 
 <CAAmGf0v9wQ0ZVb+Hj4BjxfPW74fLhxerP4Nc1axt_8K64HT5CQ@mail.gmail.com>
Subject: 
 =?GB2312?Q?Re=3A_=B4=F0=B8=B4=3A_How_to_adjust_hbase_settings_when_too_ma?=
	=?GB2312?Q?ny_store_files=3F?=
From: xkwang bruce <bruce.xkwang1@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae9d717a8d306f004cd3d0950

--14dae9d717a8d306f004cd3d0950
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Hi,=CB=D5=EE=F1.

U may need presplit you htable when the load is heavy or there should be
some problem in your client code.
Just a suggestion.

bruce


2012/10/29 =CB=D5=EE=F1 <sucheng@lietou.com>

> Hi, everyone.
>
> I changed the max size of hbase store file and increased region servers.
> The
> former exception doesn't happen again.
> But there is another exception at the client side.
>
> 2012-10-29 19:06:27:758 WARN [pool-2-thread-2]
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on
> | Failed all from
>
> region=3Dstatistic_visit_detail1,,1351508069797.3272dd30817191d9d393d1d6e=
1b99d
> 1b., hostname=3Dhadoop02, port=3D60020
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.NullPointerException
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.
> processBatchCallback(HConnectionManager.java:1557)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.
> processBatch(HConnectionManager.java:1409)
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:773)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
>         at
> com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter$1.
> process(HBaseImporter.java:150)
>         at
> com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter$1.
> process(HBaseImporter.java:133)
>         at
>
> com.lietou.datawarehouse.common.range.Repeater.rangeRepeat(Repeater.java:=
48)
>         at
>
> com.lietou.datawarehouse.common.range.Repeater.rangeRepeat(Repeater.java:=
30)
>         at
>
> com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter.run(HBaseI=
mpo
> rter.java:162)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.ja
> va:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:9
> 08)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.
> getRegionServerWithoutRetries(HConnectionManager.java:1371)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on$
> 3.call(HConnectionManager.java:1383)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on$
> 3.call(HConnectionManager.java:1381)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         ... 3 more
> Caused by: java.lang.NullPointerException
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcE=
ngi
> ne.java:158)
>         at $Proxy10.multi(Unknown Source)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on$
> 3$1.call(HConnectionManager.java:1386)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on$
> 3$1.call(HConnectionManager.java:1384)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.
> getRegionServerWithoutRetries(HConnectionManager.java:1365)
>         ... 7 more
>
>
> This error happens quite offen. And on the server side, here are some
> warnings
>
> 2012-10-29 19:50:39,748 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
>
> {"processingtimems":17476,"call":"multi(org.apache.hadoop.hbase.client.Mu=
lti
> Action@62aaeb8e), rpc version=3D1, client version=3D29,
> methodsFingerPrint=3D54742778","client":"192.168.1.70:3237
> ","starttimems":1351
>
> 511422270,"queuetimems":0,"class":"HRegionServer","responsesize":0,"metho=
d":
> "multi"}
>
>
> I have only 5 threads to execute put actions at the same time.
> The machine load is not very high.
>
> top - 19:55:02 up  7:56,  4 users,  load average: 1.62, 1.36, 1.11
>
> Anyone met this error before? Please help me.
>
> Thanks.
>
>
> -----=D3=CA=BC=FE=D4=AD=BC=FE-----
> =B7=A2=BC=FE=C8=CB: =CB=D5=EE=F1 [mailto:sucheng@lietou.com]
> =B7=A2=CB=CD=CA=B1=BC=E4: 2012=C4=EA10=D4=C229=C8=D5 15:53
> =CA=D5=BC=FE=C8=CB: user@hbase.apache.org
> =D6=F7=CC=E2: =B4=F0=B8=B4: How to adjust hbase settings when too many st=
ore files?
>
> I checked the region server log again, and I found something below:
>
> 2012-10-28 06:24:24,811 ERROR
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> Compaction failed regionName=3Dstati
>
> stic_visit_detail,20120922|13984|451728,1351376659451.9b2bfae5d77109693a1=
53e
> b16fcb7793., storeName=3Dcf1, fileCount=3D7, fileSize=3D1.1g (
> 681.7m, 168.6m, 139.9m, 36.2m, 53.0m, 26.5m, 5.9m), priority=3D0,
> time=3D469259302083252
> java.io.IOException: java.io.IOException: File
>
> /hbase/statistic_visit_detail/9b2bfae5d77109693a153eb16fcb7793/.tmp/3a3e6=
ee8
> 8a524659b
> 9f9716e5ca21a74 could only be replicated to 0 nodes, instead of 1
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FS=
Nam
> esystem.java:1531)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:68=
5)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source=
)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>
> It seems that, the fileSize already exceed the max size, which is 1G as
> default.
>
> So I add more region servers and enlarge the max file size, to see if it
> works.
>
> Thanks a lot.
>
> -----=D3=CA=BC=FE=D4=AD=BC=FE-----
> =B7=A2=BC=FE=C8=CB: Ramkrishna.S.Vasudevan [mailto:ramkrishna.vasudevan@h=
uawei.com]
> =B7=A2=CB=CD=CA=B1=BC=E4: 2012=C4=EA10=D4=C229=C8=D5 14:06
> =CA=D5=BC=FE=C8=CB: user@hbase.apache.org
> =D6=F7=CC=E2: RE: How to adjust hbase settings when too many store files?
>
> Also check what is your heap size of RS?
> When you say hTable.put(), how many such threads are there?
>
> What is your region size?  Is your regions splitting continuously do to
> heavy load?
>
> Regards
> Ram
>
>
> > -----Original Message-----
> > From: yuzhihong@gmail.com [mailto:yuzhihong@gmail.com]
> > Sent: Monday, October 29, 2012 10:01 AM
> > To: user@hbase.apache.org
> > Cc: <user@hbase.apache.org>
> > Subject: Re: How to adjust hbase settings when too many store files?
> >
> > What version of hbase were you using ?
> > Did you pre split the table before loading ?
> >
> > Thanks
> >
> >
> >
> > On Oct 28, 2012, at 8:33 PM, =CB=D5=EE=F1 <sucheng@lietou.com> wrote:
> >
> > > Hello. I encounter a region server error when I try to put bulk data
> > from a
> > > java client.
> > >
> > > The java client extracts data from a relational database and puts
> > those data
> > > into hbase.
> > >
> > > When I try to extract data from a large table(say, 1 billion
> > records), the
> > > error happens.
> > >
> > >
> > >
> > > The region server's log says:
> > >
> > >
> > >
> > >> 2012-10-28 00:00:02,169 WARN
> > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region
> > > statistic_visit_detail,20120804|72495|8549
> > >
> > > 56,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. has too many store
> > files;
> > > delaying flush up to 90000ms
> > >
> > >> 2012-10-28 00:00:02,791 DEBUG
> > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread
> > woke up
> > > because memory above low wa
> > >
> > > ter=3D347.1m
> > >
> > >> 2012-10-28 00:00:02,791 DEBUG
> > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Under global
> > heap
> > > pressure: Region statistic_vis
> > >
> > >
> > it_detail,20120804|72495|854956,1351353594195.ad2592ee7a3610c60c47cf8be
> > 77496
> > > c8. has too many store files, but is 141.5m vs best flus
> > >
> > > hable region's 46.8m. Choosing the bigger.
> > >
> > >> 2012-10-28 00:00:02,791 INFO
> > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
> > > statistic_visit_detail,20120804|7
> > >
> > > 2495|854956,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. due to
> > global
> > > heap pressure
> > >
> > > ...
> > >
> > >
> > >
> > > And finally,
> > >
> > >
> > >
> > >> 2012-10-28 00:00:43,511 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > > compaction interrupted by user
> > >
> > >> java.io.InterruptedIOException: Aborting compaction of store cf1 in
> > region
> > > statistic_visit_detail,20120804|72495|854956,135135359419
> > >
> > > 5.ad2592ee7a3610c60c47cf8be77496c8. because user requested stop.
> > >
> > >        at
> > >
> > org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1275
> > )
> > >
> > >        at
> > > org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:765)
> > >
> > >        at
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1023)
> > >
> > >        at
> > >
> > org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(
> > Compa
> > > ctionRequest.java:177)
> > >
> > >        at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut
> > or.ja
> > > va:886)
> > >
> > >        at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> > ava:9
> > > 08)
> > >
> > >        at java.lang.Thread.run(Thread.java:662)
> > >
> > >
> > >
> > > Then the region server shuts down.
> > >
> > >
> > >
> > > It seems that too many store files(due to too many records from
> > > relational-db) consumed too many memories, if I'm right.
> > >
> > > I'm new to hbase, what settings should I adjust? Or even increase
> > region
> > > servers?
> > >
> > > I'm going to do some research by myself, and any advise will be
> > appreciated.
> > >
> > > Best regards,
> > >
> > >
> > >
> > > Su
> > >
> > >
> > >
>
>
>
>

--14dae9d717a8d306f004cd3d0950--