hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Angus He <angu...@gmail.com>
Subject Many NotServingRegionException exceptions when inserting data to HBase
Date Mon, 27 Jul 2009 09:49:30 GMT
Hi All,

We have a cluster with 4 nodes, one for the Master server, and others
for Region Server.

In our case, there are 3 Downloaders crawling some specific web pages
which will be saved in HBase(0.19.3) then.
After running the Downloaders for a while,  we found that there are
tons of Exceptions in the hbase log files like this:

2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
of ~1.8m for region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
in 316ms, sequence id=4774267, compaction requested=true
2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_INFORMATION
2009-07-27 12:12:25,834 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_CONTENT
2009-07-27 12:12:25,834 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
        at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
        at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
        at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
        at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


And from then ,  the Downloaders got a big drop in the speed of page crawling.

Any ideas?


-- 
Regards
Angus

Mime
View raw message