hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Angus He <angu...@gmail.com>
Subject Re: Many NotServingRegionException exceptions when inserting data to HBase
Date Tue, 28 Jul 2009 04:22:36 GMT
Yup ,  I found this in the Master log file.

2009-07-27 12:13:10,880 INFO
org.apache.hadoop.hbase.master.ServerManager: Received
MSG_REPORT_SPLIT:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
split; daughters:
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248667945476,
webpage,http://www.tianya.cn/techforum/content/50/565730.shtml,1248667945476:
safeMode=false from 192.168.33.5:60020

BTW, There is about a 50-second time lag between Region server and
Master server.


On Tue, Jul 28, 2009 at 11:45 AM, stack<stack@duboce.net> wrote:
> I was looking at this today.
>
> When you start to get the NotServingRegionException, has the region just
> split?  You can tell by looking for "webpage,
> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868" in the
> master log.  If it has a message about splitting just before you get the
> below NSRE exception, then it seems like its a case of HBASE-1671.
> Hopefully will have a fix this evening.
>
> St.Ack
>
> On Mon, Jul 27, 2009 at 2:49 AM, Angus He <angushe@gmail.com> wrote:
>
>> Hi All,
>>
>> We have a cluster with 4 nodes, one for the Master server, and others
>> for Region Server.
>>
>> In our case, there are 3 Downloaders crawling some specific web pages
>> which will be saved in HBase(0.19.3) then.
>> After running the Downloaders for a while,  we found that there are
>> tons of Exceptions in the hbase log files like this:
>>
>> 2009-07-27 12:12:25,833 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
>> of ~1.8m for region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> in 316ms, sequence id=4774267, compaction requested=true
>> 2009-07-27 12:12:25,833 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed
>> 1734514356/CF_INFORMATION
>> 2009-07-27 12:12:25,834 DEBUG
>> org.apache.hadoop.hbase.regionserver.HStore: closed
>> 1734514356/CF_CONTENT
>> 2009-07-27 12:12:25,834 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> 2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837
ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> 2009-07-27 12:12:25,837 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
>> 9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
>> 9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>> org.apache.hadoop.hbase.NotServingRegionException: Region
>> webpage,
>> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
>> closed
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
>>        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>>
>>
>> And from then ,  the Downloaders got a big drop in the speed of page
>> crawling.
>>
>> Any ideas?
>>
>>
>> --
>> Regards
>> Angus
>>
>



-- 
Regards
Angus

Mime
View raw message