hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vito <liyumin...@gmail.com>
Subject Will compaction block memstore flush to disk
Date Mon, 14 Jul 2014 03:57:27 GMT
Hi ,

  Will compaction(minor or major) block memstore flush to disk(or HDFS)? 
That's, when region server is doing compaction,  is it allowed to flush
memstore to HDFS?  Let's base the answer in HBase 0.94. 

Here are some info I dig out, but I'm not sure of the anwser:

hbase.hstore.compactionThreshold:(0.89)

    If more than this number of HStoreFiles in any one HStore (one
HStoreFile is written per flush of memstore) then a compaction is run to
rewrite all HStoreFiles files as one.  Larger numbers put off compaction but
when it runs, it takes longer to complete.  *During a compaction, updates
cannot be flushed to disk. * Long compactions require memory sufficient to
carry the logging of all updates across the duration of the compaction.   If
too large, clients timeout during compaction.

hbase.hstore.compactionThreshold:(0.94)
If more than this number of HStoreFiles in any one HStore (one HStoreFile is
written per flush of memstore) then a compaction is run to rewrite all
HStoreFiles files as one. Larger numbers put off compaction but when it
runs, it takes longer to complete.


In the source code of 0.94, the comment says new storeFiles can be written
from memstore. 
org.apache.hadoop.hbase.regionserver.Store.java:compact()
 
 
//////////////////////////////////////////////////////////////////////////////
  // Compaction
 
//////////////////////////////////////////////////////////////////////////////

  /**
   * Compact the StoreFiles.  This method may take some time, so the calling
   * thread must be able to block for long periods.
   *
   * <p>During this time, the Store can work as usual, getting values from
   * StoreFiles and writing new StoreFiles from the memstore.
   *
   * Existing StoreFiles are not destroyed until the new compacted StoreFile
is
   * completely written-out to disk.
   ...


Regards, 
Vito Lee




--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Will-compaction-block-memstore-flush-to-disk-tp4061302.html
Sent from the HBase User mailing list archive at Nabble.com.

Mime
View raw message