hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkrishna S Vasudevan <ramakrish...@huawei.com>
Subject RE: Region has been OPENING for too long
Date Mon, 31 Oct 2011 12:26:50 GMT
Hi 

What I would suggest is try doing a forcefull assign of that region that is
showing this log using the shell.(If the logs still continues to appear).

Regards
Ram

 



-----Original Message-----
From: tovbinm@gmail.com [mailto:tovbinm@gmail.com] On Behalf Of Matthew
Tovbin
Sent: Monday, October 31, 2011 3:38 PM
To: HBase User
Subject: Re: Region has been OPENING for too long

Ted,  thanks for such a rapid response.

You're right, we use hbase 0.90.3 from cdh3u1.

So, I suppose I need to make bulk loading in smaller bulks then. Any other
suggestions?


Best regards,
    Matthew Tovbin =)

>
>
>I assume you're using HBase 0.90.x where HBASE-4015 isn't available.
>
>>> 5. And so on, till some of Slaves fail with "java.net.SocketException:
>Too many open files".
>Do you have some monitoring setup so that you can know the number of 
>open file handles ?
>
>Cheers
>
>On Sun, Oct 30, 2011 at 7:21 AM, Matthew Tovbin <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>>   I've bulkloaded a solid amount of data (650GB, ~14000 files) into 
>> Hbase (1master + 3regions) and now enabling the table results the 
>> following behavior on the cluster:
>>
>>   1. Master says that opening started  -
>>    "org.apache.hadoop.hbase.master.AssignmentManager: Handling
>>   transition=RS_ZK_REGION_OPENING, server=slave..."
>>   2. Slaves report about opening files in progress -
>>   "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
>>   3. Then after ~10 mins the following error occurs on hmaster -
>>    "org.apache.hadoop.hbase.master.AssignmentManager: Regions in
transition
>>   timed out / Region has been OPENING for too long, reassigning
region=..."
>>   4. More slaves report about opening files in progress -
>>   "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
>>   5. And so on, till some of Slaves fail with "java.net.SocketException:
>>   Too many open files".
>>
>>
>> What I've done already to solve the issue (which DID NOT help though):
>>
>>   1. Set 'ulimit -n 65536' for hbase user
>>   2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in
hbase-site.xml
>>
>>
>>  What else can I try?!
>>
>>
>> Best regards,
>>    Matthew Tovbin =)
>>


Mime
View raw message