Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <BEC82105-6225-45A8-AC1D-F532A76AE834@adobe.com>
References: <BEC82105-6225-45A8-AC1D-F532A76AE834@adobe.com>
Date: Thu, 31 Jan 2013 06:48:10 -0500
Message-ID: 
 <CAPQV63XMLsAjhRg8_kHR7wQS+brXd6YAYqajKfMPpGs0eOcUjw@mail.gmail.com>
Subject: Re: HRegionInfo was null or empty in Meta errors
From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=UTF-8

Hi Brandon,

I faced the same issue for "HRegionInfo was null or empty" on January
24th and Ted replied:

"Encountered problems when prefetch META table:

You can ignore the warning."

So I think you should focus on the last one "not listed in META or
deployed on any region server".

Have you tried hbck to see if it can fix it?

JM

2013/1/31, Brandon Peskin <bpeskin@adobe.com>:
> hadoop 0.20.2-cdh3u2
> hbase 0.90.4-cdh3u2
>
> On January 8th I had a network event where I lost three region servers.
>
> When they came back I had unassigned regions/regions not being served errors
> which I fixed with the hbck -fix
>
>
>
> Since then, however I have been getting an increasing number of these when I
> have clients trying to write to specific tables:
>
>
> java.io.IOException: HRegionInfo was null or empty in Meta for
> algol_profile_training_record,
> row=algol_profile_training_record,clientcode:49128:abce6d9f-1ee2-434a-8a82-a151b7dc183f,99999999999999
> 	at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:142)
> ~[hbase-0.90.4-cdh3u2.jar:na]
> 	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
> ~[hbase-0.90.4-cdh3u2.jar:na]
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:649)
> [hbase-0.90.4-cdh3u2.jar:na]
>
>
>
> ....to the point now where I seemingly can't even write to that table.
>
> This also coincides with the following, seen in the the hbase master log:
>
> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is empty
> in
> keyvalues={algol_tmp_client_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:server/1351785275123/Put/vlen=32,
> algol_tmp_clientcode_39691_20121101053400_6,,1351783324210.b703b203d6189cad224764853cbd88f7./info:serverstartcode/1351785275123/Put/vlen=8}
> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is empty
> in
> keyvalues={algol_tmp_client_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:server/1351785275128/Put/vlen=32,
> algol_tmp_clientcode_39691_20121101053400_6,\x7F\xD6\xA5Pm\xF2E\x9D\x81\xB1b\xD1'\xD3\xE5\xA4,1351783324210.507bdf342aaae45644d597c272c52e02./info:serverstartcode/1351785275128/Put/vlen=8}
> 13/01/31 02:49:55 WARN master.CatalogJanitor: REGIONINFO_QUALIFIER is empty
> in
> keyvalues={algol_tmp_client_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:server/1349461385375/Put/vlen=32,
> algol_tmp_overstock_34785_20121004102759_0,,1349383219155.a45774fa4c8af765df87d3373d30cfdc./info:serverstartcode/1349461385375/Put/vlen=8}
>
>
> ...and there's 58 of these I cannot fix (presumably I could with the hbck
> provided with CDH4):
>
> ERROR: Region
> hdfs://namenode:9000/hbase/algol_profile_training_record/fdaa1024d1b725b4997d2283640f0fa4
> on HDFS, but not listed in META or deployed on any region server
>
>
> I'm absolutely stumped. I've done some poking around and I can't find any
> sort of data surrounding this issue with the exception of similar symptoms
> in an exchange on this list of March this year (though the inquirer had
> different questions). Any help would be appreciated, though I suspect I will
> be told 'Upgrade to CDH4' or 'Drop and re-create the table'.
>
> Thanks in advance.
>
> --
> Brandon Peskin
> Senior Systems Administrator
> Adobe Systems
> bpeskin@adobe.com
>
>
>
>
>