hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Fox <r...@connexity.com>
Subject Re: Minor compacts seem blocked on major compacts
Date Thu, 08 Oct 2015 14:49:32 GMT
Thanks Ted, looks the right change.  We are using CDH and it does not have that fix.  Easy
enough to put in, though I tried to build hbase and ran into many issues.
Is there a doc somewhere on the prereqs for building hbase (1.0.0) in particular and the mvn
flags?

Thanks,

Randy



On 10/7/15, 1:34 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:

>Have you seen the following code ?
>
>   ThreadPoolExecutor pool = (selectNow &&
>s.throttleCompaction(compaction.getRequest().getSize()))
>      ? longCompactions : shortCompactions;
>
>Looks like throttleCompaction() returned false.
>Please see the following method in RatioBasedCompactionPolicy :
>
>  public boolean throttleCompaction(long compactionSize) {
>
>    return compactionSize > comConf.getThrottlePoint();
>
>FYI
>
>On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <rfox@connexity.com> wrote:
>
>> Looking at the source I don’t see how a manual major compact could get in
>> the large compaction pool:
>> If selectNow Is True the pool selector has to be shortCompactions.  From
>> the logs it sure seems that selectNow has to be true, else it would have
>> logged “system” instead of
>> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
>> ;”
>>
>>
>> Sure seems like an hbase bug.
>>
>> Am I missing something?
>>
>> Thanks,
>>
>> Randy
>>
>>   /**
>>    * @param r HRegion store belongs to
>>    * @param s Store to request compaction on
>>    * @param why Why compaction requested -- used in debug messages
>>    * @param priority override the default priority (NO_PRIORITY == decide)
>>    * @param request custom compaction request. Can be <tt>null</tt> in
>> which case a simple
>>    *          compaction will be used.
>>    */
>>   private synchronized CompactionRequest requestCompactionInternal(final
>> HRegion r, final Store s,
>>       final String why, int priority, CompactionRequest request, boolean
>> selectNow)
>>           throws IOException {
>>     if (this.server.isStopped()
>>         || (r.getTableDesc() != null &&
>> !r.getTableDesc().isCompactionEnabled())) {
>>       return null;
>>     }
>>
>>     CompactionContext compaction = null;
>>     if (selectNow) {
>>       compaction = selectCompaction(r, s, priority, request);
>>       if (compaction == null) return null; // message logged inside
>>     }
>>
>>     // We assume that most compactions are small. So, put system
>> compactions into small
>>     // pool; we will do selection there, and move to large pool if
>> necessary.
>>     long size = selectNow ? compaction.getRequest().getSize() : 0;
>>     ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
>>       ? longCompactions : shortCompactions;
>>     pool.execute(new CompactionRunner(s, r, compaction, pool));
>>     if (LOG.isDebugEnabled()) {
>>       String type = (pool == shortCompactions) ? "Small " : "Large ";
>>       LOG.debug(type + "Compaction requested: " + (selectNow ?
>> compaction.toString() : "system")
>>           + (why != null && !why.isEmpty() ? "; Because: " + why : "") +
>> "; " + this);
>>     }
>>     return selectNow ? compaction.getRequest() : null;
>>   }
>>
>>
>>
>>
>>
>> On 10/6/15, 9:58 PM, "Randy Fox" <rfox@connexity.com> wrote:
>>
>> >Sorry, I do not follow. I thought there was one thread per pool, one for
>> large and one for small.  Thus my major compact should have been in the
>> large pool and not blocking the small pool.  That is how it worked in 0.94.
>> >From the logs it demoted my compact to the small pool.
>> >
>> >-randy
>> >
>> >
>> >
>> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vladrodionov@gmail.com> wrote:
>> >
>> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G
so it
>> >>should have been put into the Large compaction pool.
>> >>
>> >>You answered your question yourself. Minor compaction in the same pool (1
>> >>thread default) will be waiting until major is finished.
>> >>
>> >>-Vlad
>> >>
>> >>
>> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rfox@connexity.com> wrote:
>> >>
>> >>> 2015-10-06 14:50:35,644 INFO
>> org.apache.hadoop.hbase.regionserver.HStore:
>> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
>> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
>> >>>
>> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
>> >>> totalSize=4.8 G
>> >>> 2015-10-06 14:50:35,646 DEBUG
>> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
>> Compaction
>> >>> requested:
>> >>>
>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
>> ;
>> >>> Because: User-triggered major compaction; compaction_queue=(0:0),
>> >>> split_queue=0, merge_queue=0
>> >>>
>> >>>
>> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so
it
>> >>> should have been put into the Large compaction pool.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On 10/6/15, 3:07 PM, "Randy Fox" <rfox@connexity.com> wrote:
>> >>>
>> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
>> >>> compacts manually every night. One of the first things we noticed is
>> that
>> >>> when a major compact runs, no minors run.  The compaction queue grows
>> and
>> >>> when the major compact finishes the minors then run.   I have not
>> found any
>> >>> new knobs we should be setting.  Any ideas?
>> >>> >Our config is:
>> >>> >
>> >>> ><property>
>> >>> >  <name>hbase.hregion.majorcompaction</name>
>> >>> >  <value>0</value>
>> >>> >  <final>true</final>
>> >>> ></property>
>> >>> ><property>
>> >>> >    <name>hbase.hstore.compactionThreshold</name>
>> >>> >    <value>3</value>
>> >>> ></property>
>> >>> >
>> >>> ><property>
>> >>> >    <name>hbase.hstore.compaction.max</name>
>> >>> >    <value>9</value>
>> >>> ></property>
>> >>> ><property>
>> >>> >    <name>hbase.regionserver.thread.compaction.throttle</name>
>> >>> >    <value>2684354560</value>
>> >>> ></property>
>> >>> >
>> >>> >
>> >>> >Thanks in advance,
>> >>> >
>> >>> >Randy
>> >>>
>>
Mime
View raw message