hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Tovbin <matt...@tovbin.com>
Subject Region has been OPENING for too long
Date Sun, 30 Oct 2011 14:21:19 GMT
Hi guys,

   I've bulkloaded a solid amount of data (650GB, ~14000 files) into Hbase
(1master + 3regions) and now enabling the table results the
following behavior on the cluster:

   1. Master says that opening started  -
    "org.apache.hadoop.hbase.master.AssignmentManager: Handling
   transition=RS_ZK_REGION_OPENING, server=slave..."
   2. Slaves report about opening files in progress -
   "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
   3. Then after ~10 mins the following error occurs on hmaster -
    "org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
   timed out / Region has been OPENING for too long, reassigning region=..."
   4. More slaves report about opening files in progress -
   "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
   5. And so on, till some of Slaves fail with "java.net.SocketException:
   Too many open files".


What I've done already to solve the issue (which DID NOT help though):

   1. Set 'ulimit -n 65536' for hbase user
   2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in hbase-site.xml


 What else can I try?!


Best regards,
   Matthew Tovbin =)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message