hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bijieshan <bijies...@huawei.com>
Subject Re: HRegionInfo was null or empty in -ROOT-
Date Fri, 29 Apr 2011 08:23:17 GMT
I could't see the two regions in the UI.
After I restarted hbase, this issue was still exist. I scaned the -ROOT- table and .META.
table, saw the same exception as "HRegionInfo was null or empty in -ROOT-".
Before the exception occurred, someone else deleted some log files:

/hbase/.logs/Linux-1,60020,1303815506931/Linux-1%3A60020.1303822717531
/hbase/.logs/Linux-2,60020,1303815512433/Linux-2%3A60020.1303826323694
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-1,60020,1303822864554/Linux-1%3A60020.1303822864894
/hbase/.logs/Linux-3,60020,1303819206339/Linux-3%3A60020.1303819216692
/hbase/.logs/Linux-3,60020,1303842736457/Linux-3%3A60020.1303842860921
/hbase/.logs/Linux-2,60020,1303842932959/Linux-2%3A60020.1303842943315
/hbase/.logs/Linux-1,60020,1303842785259/Linux-1%3A60020.1303846459356
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-1,60020,1303896597830/Linux-1%3A60020.1303900208368
/hbase/.logs/Linux-2,60020,1303896592421/Linux-2%3A60020.1303896602802
/hbase/.logs/Linux-3,60020,1303896579647/Linux-3%3A60020.1303896590052

I don't know whether these operations are the direct causation.

【Error info from the page of master.jsp 】

Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
null for region , row '', but failed after 10 attempts.
Exceptions:
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.
org.apache.hadoop.hbase.TableNotFoundException: .META.

【Error info from hbase shell】
hbase(main):001:0> list
TABLE
11/04/29 15:24:59 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but
got as null.Hence setting to default value 1048575

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region
server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13,
.META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304060872102/Put/vlen=13,
.META.,,1/info:serverstartcode/1304060872102/Put/vlen=8}


hbase(main):002:0>
hbase(main):003:0* scan '-ROOT-'
ROW                                           COLUMN+CELL
 .META.,,1                                    column=info:server, timestamp=1304062076821,
value=Linux-2:60020
 .META.,,1                                    column=info:serverstartcode, timestamp=1304062076821,
value=1304060518330
1 row(s) in 0.0830 seconds

hbase(main):004:0> scan '.META.'
ROW                                           COLUMN+CELL

ERROR: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304062076821/Put/vlen=13,
.META.,,1/info:serverstartcode/1304062076821/Put/vlen=8}


All the logs I have attached in the email.

Thanks!

Jieshan Bean



-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月29日 12:10
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: HRegionInfo was null or empty in -ROOT-

Can you see your two regions in the UI?

What if you restart hbase?  Does this fix the issue?

Going by your master log, it seems happy enough, almost as though it
were a different cluser altogether.

You could try scanning '.META.' to see if you got your two regions back.

You could try restarting the regionserver that is currently hosting .META.?

St.Ack

2011/4/28 bijieshan <bijieshan@huawei.com>:
> Thanks Stack!
>
> When I scan the -ROOT- table, it's not empty. So I delete all the records in -ROOT- using
as you told:
>
> Hbase > delateall '-ROOT-','.META.,,1'
>
> After that, I scan the -ROOT again , the records was indeed deleted.
>
> Hbase > scan '-ROOT-'
> 0 row(s) in 0.0210 seconds.
>
> And then, I unassign the META region:
>
> Hbase > unassign ".META.,,1"
>
> After a long while, I executed list:
>
> hbase(main):002:0> list
> TABLE
>
> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region
server null for region , row '', but failed after 7 attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
> org.apache.hadoop.hbase.TableNotFoundException: .META.
>
> Here is some help for this command:
> List all tables in hbase. Optional regular expression parameter could
> be used to filter the output. Examples:
>
>  hbase> list
>  hbase> list 'abc.*'
>
> In HMaster logs, there's no effect happened:
>
> 2011-04-29 11:41:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load
balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:46:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned
0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:46:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load
balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:51:29,609 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned
0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:51:29,610 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load
balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 11:56:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned
0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 11:56:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load
balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
> 2011-04-29 12:01:29,607 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned
0 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-04-29 12:01:29,609 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load
balancing.  servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=1
>
> Any problem exist in my operations?
>
> Jieshan Bean
>
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月29日 11:41
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: HRegionInfo was null or empty in -ROOT-
>
> In general, its usually best to let hbase manage all under the /hbase
> dir in hdfs.
>
> What happens if you scan the -ROOT- table in shell?
>
> hbase> scan "-ROOT-"
>
> Is it empty there?
>
> If so, thats not the real issue.  The real issue is why .META. is not
> being deployed.
>
> Try removing the entry in -ROOT-
>
> hbase> deleteall ".META.,,1"
>
> Then, try doing something like unassigning .META.
>
> hbase> unassign .META.
>
> Tail the master log.  See if the above has any effect.
>
> St.Ack
>
> 2011/4/28 bijieshan <bijieshan@huawei.com>:
>> Thanks J-D.
>> I think it's not this problem. I have checked the logs, there's no DNS exception.
>> Some one deleted some hdfs files, use the following method:
>>
>> ./hadoop fs -rm /hbase/.logs/181-6-30-103,60020,1303774895876/181-6-30-103%3A60020.1303774909239
>>
>> I don't know whether these operations can bring any effects.
>>
>> You can check the log files from the attachments.
>>
>> Jieshan Bean
>>
>> -----邮件原件-----
>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
>> 发送时间: 2011年4月29日 10:53
>> 收件人: user@hbase.apache.org
>> 抄送: Chenjian
>> 主题: Re: HRegionInfo was null or empty in -ROOT-
>>
>> Last time I heard about this issue, it was because of a DNS problem.
>> Look at your master and region server logs.
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 7:49 PM, bijieshan <bijieshan@huawei.com> wrote:
>>> Hi,
>>>
>>> When I execute command of 'list' in hbase shell, it told me that the HRegioninfo
was null or empty in -ROOT-.
>>> Did the -ROOT- table value got corrupted or any other reasons cause this problem?
>>>
>>> Here's the info from the 'list' and 'hbck' tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> hbase(main):001:0> list
>>> TABLE
>>> 11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as
an integer but got as null.Hence setting to default value 1048575
>>>
>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server null for region , row '', but failed after 7 attempts.
>>> Exceptions:
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13,
.META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> --------------------------------------------------------------------------------------------------------------------
>>> HBase hbck tool:
>>> --------------------------------------------------------------------------------------------------------------------
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052,
attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052,
attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052,
attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052,
attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052,
attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
>>> 11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation:
Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a;
hsa=Linux-1:60020
>>> --------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks,
>>>
>>> Regards,
>>> Jieshan bean
>>>
>>>
>>
>
Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message