hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Vissapragada <bhara...@cloudera.com>
Subject Re: Will compaction block memstore flush to disk
Date Mon, 14 Jul 2014 05:50:34 GMT
If there are too many store files, flush will wait till compaction clears
them [1] or hbase.hstore.blockingWaitTime is over! Do you see similar logs?

[1]
https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java#L363


On Mon, Jul 14, 2014 at 9:27 AM, vito <liyuming05@gmail.com> wrote:

> Hi ,
>
>   Will compaction(minor or major) block memstore flush to disk(or HDFS)?
> That's, when region server is doing compaction,  is it allowed to flush
> memstore to HDFS?  Let's base the answer in HBase 0.94.
>
> Here are some info I dig out, but I'm not sure of the anwser:
>
> hbase.hstore.compactionThreshold:(0.89)
>
>     If more than this number of HStoreFiles in any one HStore (one
> HStoreFile is written per flush of memstore) then a compaction is run to
> rewrite all HStoreFiles files as one.  Larger numbers put off compaction
> but
> when it runs, it takes longer to complete.  *During a compaction, updates
> cannot be flushed to disk. * Long compactions require memory sufficient to
> carry the logging of all updates across the duration of the compaction.
> If
> too large, clients timeout during compaction.
>
> hbase.hstore.compactionThreshold:(0.94)
> If more than this number of HStoreFiles in any one HStore (one HStoreFile
> is
> written per flush of memstore) then a compaction is run to rewrite all
> HStoreFiles files as one. Larger numbers put off compaction but when it
> runs, it takes longer to complete.
>
>
> In the source code of 0.94, the comment says new storeFiles can be written
> from memstore.
> org.apache.hadoop.hbase.regionserver.Store.java:compact()
>
>
>
> //////////////////////////////////////////////////////////////////////////////
>   // Compaction
>
>
> //////////////////////////////////////////////////////////////////////////////
>
>   /**
>    * Compact the StoreFiles.  This method may take some time, so the
> calling
>    * thread must be able to block for long periods.
>    *
>    * <p>During this time, the Store can work as usual, getting values from
>    * StoreFiles and writing new StoreFiles from the memstore.
>    *
>    * Existing StoreFiles are not destroyed until the new compacted
> StoreFile
> is
>    * completely written-out to disk.
>    ...
>
>
> Regards,
> Vito Lee
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Will-compaction-block-memstore-flush-to-disk-tp4061302.html
> Sent from the HBase User mailing list archive at Nabble.com.
>



-- 
Bharath Vissapragada
<http://www.cloudera.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message