hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bluemetrix Development <bmdevelopm...@gmail.com>
Subject Re: hbase shell count crashes
Date Wed, 17 Feb 2010 15:50:59 GMT
Hi,
So after a few more attempts and crashes from trying the shell count,
I ran the MR rowcounter and noticed that the number of rows were less
than they should have been - even on smaller test tables.
This led me to start looking through the logs and perform a few
compacts on META and restarts of hbase. Unfortunately, now two tables
are entirely missing - no longer show up under the shell list command.

I'm not entirely sure what to look for in the logs, but I've noticed a
lot of this in the master log.

2010-02-16 23:59:25,856 WARN org.apache.hadoop.hbase.master.HMaster:
info:regioninfo is empty for row:
UserData_0209,e834d76faddee14b,1266316478685; has keys: info:server,
info:serverstartcode

Came across this in the regionserver log:
2010-02-16 23:58:33,851 WARN
org.apache.hadoop.hbase.regionserver.Store: Skipping
hdfs://upp1.bmeu.com:50001/hbase/.META./1028785192/info/4080287239754005013
because its empty. HBASE-646 DATA LOSS?

Any ideas if the tables are recoverable? Its not a big deal for me to
re-insert from scratch as this is still in testing phase,
but would be curious to find out what has led to these issues in order
to possibly fix or at least not repeat.

Thanks

On Tue, Feb 16, 2010 at 2:43 PM, Bluemetrix Development
<bmdevelopment@gmail.com> wrote:
> Hi, Thanks for the explanation.
>
> Yes, I was able to cat the file from all three of my region servers:
> hadoop fs -cat /hbase/.META./1028785192/info/8254845156484129698 > tmp.out
>
> I have never came across this before, but this is the first time I've
> had 7M rows in the db.
> Is there anything going on that would bog down the network and cause
> this file to be unreachable?
>
> I have 3 servers. The master is running the jobtracker, namenode and hmaster.
> And all 3 are running datanodes, regionservers and zookeeper.
>
> Appreciate the help.
>
> On Tue, Feb 16, 2010 at 2:11 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
>> This line
>> java.io.IOException: java.io.IOException: Could not obtain block:
>> blk_-6288142015045035704_88516
>> file=/hbase/.META./1028785192/info/8254845156484129698
>>
>> Means that the region server wasn't able to fetch a block for the .META.
>> table (the table where all region addresses are). Are you able to open that
>> file using the bin/hadoop command line utility?
>>
>> J-D
>>
>> On Tue, Feb 16, 2010 at 11:08 AM, Bluemetrix Development <
>> bmdevelopment@gmail.com> wrote:
>>
>>> Hi,
>>> I'm currently trying to run a count in hbase shell and it crashes
>>> right towards the end.
>>> This is turn seems to crash hbase or at least causes the regionservers
>>> to become unavailable.
>>>
>>> Here's the tail end of the count output:
>>> http://pastebin.com/m465346d0
>>>
>>> I'm on version 0.20.2 and running this command:
>>> > count 'table', 1000000
>>>
>>> Anyone with similar issues or ideas on this?
>>> Please let me know if you need further info.
>>> Thanks
>>>
>>
>

Mime
View raw message