hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Cannot open filename exception in 0.20.5
Date Fri, 25 Jun 2010 21:52:32 GMT
And nothing after that? Because this would be the first assignment we
see in the region server log.

J-D

On Fri, Jun 25, 2010 at 2:49 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> 10.32.56.159 and sjc9-flash-grid05 are the same server:
>
> 2010-06-25 09:56:52,308 INFO org.apache.hadoop.hbase.master.ServerManager: 2
> region servers, 0 dead, average load 22.52010-06-25 09:56:52,490 INFO
> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
> scanning meta region {server: 10.32.56.159:60020, regionname: -ROOT-,,0,
> startKey: <>}
> 2010-06-25 09:56:52,498 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
> Current assignment of .META.,,1 is not valid;  serverAddress=
> 10.32.56.160:60020, startCode=1277484838693 unknown.
> 2010-06-25 09:56:52,499 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 10.32.56.159:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-06-25 09:56:52,847 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning for address: 10.32.56.159:60020, startcode: 1277484792423, load:
> (requests=5, regions=23, usedHeap=30, maxHeap=3991): total nregions to
> assign=0, regions to give other servers than this=1, isMetaAssign=true
> 2010-06-25 09:56:52,847 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning address: 10.32.56.159:60020, startcode: 1277484792423, load:
> (requests=5, regions=23, usedHeap=30, maxHeap=3991) 0 regions
> 2010-06-25 09:56:52,847 INFO org.apache.hadoop.hbase.master.RegionManager:
> Assigning region .META.,,1 to sjc9-flash-grid05.carrieriq.com
> ,60020,1277484792423
> 2010-06-25 09:56:52,849 INFO org.apache.hadoop.hbase.master.ServerManager:
> Processing MSG_REPORT_OPEN: .META.,,1 from
> sjc9-flash-grid05.carrieriq.com,60020,1277484792423;
> 1 of 1
> 2010-06-25 09:56:52,850 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Processing todo: PendingOpenOperation from sjc9-flash-grid05.carrieriq.com
> ,60020,1277484792423
> 2010-06-25 09:56:52,850 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: .META.,,1 open on
> 10.32.56.159:60020
> 2010-06-25 09:56:52,852 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: Updated row .META.,,1
> in region -ROOT-,,0 with startcode=1277484792423, server=10.32.56.159:60020
> 2010-06-25 09:56:52,852 DEBUG
> org.apache.hadoop.hbase.master.RegionServerOperation: Adding to
> onlineMetaRegions: {server: 10.32.56.159:60020, regionname: .META.,,1,
> startKey: <>}
> 2010-06-25 09:56:52,853 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 10.32.56.159:60020,
> regionname: .META.,,1, startKey: <>}
>
> On Fri, Jun 25, 2010 at 2:33 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> While taking a second look, I see that actually your region server is
>> the one who is opening it over and over again:
>>
>> 2010-06-25 09:56:03,918 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
>> MSG_REGION_OPEN: .META.,,1
>> 2010-06-25 09:56:03,919 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>> .META.,,1, encoded=1028785192
>> 2010-06-25 09:56:03,933 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: loaded
>> /hbase/.META./1028785192/info/7628076653206028272, isReference=false,
>> sequence id=2991, length=51941, majorCompaction=false
>> ...
>> 2010-06-25 10:00:03,447 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
>> .META.,,1
>> 2010-06-25 10:00:03,448 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
>> MSG_REGION_OPEN: .META.,,1
>> ...
>> 2010-06-25 10:02:03,130 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
>> .META.,,1
>> 2010-06-25 10:02:03,130 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
>> MSG_REGION_OPEN: .META.,,1
>>
>> These messages come from the master, look for "Assigning region
>> .META.,,1" not the encoded name.
>>
>> J-D
>>
>>
>> On Fri, Jun 25, 2010 at 2:22 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> > Hi,
>> > 2010-06-25 09:56:03,936 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> > region .META.,,1/1028785192 available; sequence id is 2992
>> >
>> > I searched for 1028785192 in master log but didn't find any occurrence.
>> > The only log about compaction I see in master log is something like this:
>> >
>> > 2010-06-25 09:59:39,898 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> > Closing TED-PACKAGESUMMARY-1277485167424-0,,1277485173774: disabling
>> > compactions & flushes
>> >
>> > Any insight ?
>> >
>> > Thanks
>> >
>> > On Fri, Jun 25, 2010 at 11:31 AM, Jean-Daniel Cryans <
>> jdcryans@apache.org>
>> > wrote:
>> >>
>> >> At first glance it looks like a double assignment of .META., and the
>> >> file it's trying to get was probably already compacted by another
>> >> region server. Check the master log to see why/how it happened.
>> >>
>> >> J-D
>> >>
>> >> On Fri, Jun 25, 2010 at 11:13 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >> > Hi,
>> >> > I upgraded a 3 node (1 master and 2 RS) cluster to 0.20.5 this
>> morning.
>> >> > hadoop version is Cloudera 0.20.2+228
>> >> > I saw the following in
>> >> > hbase-hadoop-regionserver-sjc9-flash-grid05.ciq.com.log
>> >> > (/hbase/.META./1028785192/info/7628076653206028272 was not on hdfs):
>> >> >
>> >> > http://pastebin.com/0rH56Vzb
>> >> >
>> >> > This is from hadoop-hadoop-datanode-sjc9-flash-grid05.ciq.com.log:
>> >> > http://pastebin.com/eYvq6FEV
>> >> >
>> >> > Has anyone seen similar issue ?
>> >> >
>> >> > Thanks
>> >> >
>> >
>> >
>>
>

Mime
View raw message