hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vidhyashankar Venkataraman <vidhy...@yahoo-inc.com>
Subject Effect of turning major compactions off..
Date Wed, 26 May 2010 16:07:53 GMT
Are there any side effects to turning major compactions off, other than just a hit in the read
performance?

I was trying to merge a 120 Gig update (modify/insert/delete operations) into a 2 TB fully
compacted Hbase table with 5 region servers using a map reduce job.. Each RS was serving around
2000 regions (256 MB max size)... Major compactions were turned off before the job started
(by setting the compaction period very high to around 4 or 5 days)..

As the job was going on, the region servers just shut down after the table reached near-100%
fragmentation (as shown in the web interface)..  On looking at the RS logs, I saw that there
were compaction checks for each region which obviously didn't clear, and the RS's shut down
soon after the checks..  I tried restarting the database after killing the map reduce job
(still, with major compactions turned off).. The RS's shut down soon after booting up..

   Is this expected? Even if the update files (the additional StoreFiles) per region get huge,
won't the region get split on its own?

Thank you
Vidhya

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message