hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZhouShuaifeng 00100568 <zhoushuaif...@huawei.com>
Subject Region server shutdown after putting millions of rows
Date Tue, 14 Sep 2010 02:08:00 GMT
Hi All,

I encounted some problem when doing putting data test on hbase. Please help. Thanks a lot.

After putting about millions of rows, the 2 region servers of 3 were stopped. 
Server 1 stopped when putting about 50 million rows.
Server 2 stopped when putting about 100 million rows.

Some exceptions are throwed.
The client exception info is below:
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META.
for region xxx.
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:833)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:677)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfPuts(HConnectionManager.java:1419)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:664)
	at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:549)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)

The server exception is below:
org.apache.hadoop.hbase.NotServingRegionException: xxx. is closed
	at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2122)
	at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2211)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1493)
	at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1447)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1703)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2361)
	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)

region server 1 stoped at about 13:59
2010-09-12 13:59:01,017 INFO org.apache.hadoop.hbase.master.ServerManager: 2 region servers,
1 dead, average load 99.5[md-prod04,60020,1284169880500]

the last 2 logs of this regionserver before it stoped is:
2010-09-12 13:57:46,170 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Caches flushed,
doing commit now (which includes update scanners)
2010-09-12 13:57:46,172 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
flush of ~21.0m for region percontent_hr,2000-01-03#http#001#url18#s#states44#0,1284322397591.8a557b61c9eb4b117368051b98e8d1d1.
in 290ms, sequence id=155719842, compaction requested=false

region server 2 stoped at about 19:36:
2010-09-12 19:37:01,104 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers,
1 dead, average load 356.0[md-prod01,60020,1284169861364]

the last logs of this regionserver before it stoped is:
2010-09-12 19:36:02,398 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction
of 3 file(s) in visitors of xxx.; new storefile is hdfs://xxx; store size is 201.3m
2010-09-12 19:36:02,398 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region xxx. in 9sec
2010-09-12 19:36:04,604 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache
LRU eviction started.  Attempting to free 20845272 bytes
2010-09-12 19:36:04,609 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache
LRU eviction completed. Freed 19920968 bytes.  Priority Sizes: Single=43.940674MB (46075136),
Multi=74.515015MB (78134656),Memory=49.535378MB (51941608)

******************************************************************************************
 This email and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
here in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this email in error, please notify the sender by phone or email
 immediately and delete it!
 *****************************************************************************************

Mime
View raw message