Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <9680FF4F-6AA1-4016-8707-489CE209AC3E@connexity.com>
References: <AEB329F1-DBC9-411C-85A4-DA963124E11F@connexity.com>
	<57258DD8-8440-4F2A-9D51-C1ADC28ED7A9@connexity.com>
	<CAAg3a2rOjD4CSqim+J0wSg9aypZNDyja52hea6pQcFzZ7b-PbA@mail.gmail.com>
	<BE724292-7E23-4DA7-904E-4EC397DB12C3@connexity.com>
	<9680FF4F-6AA1-4016-8707-489CE209AC3E@connexity.com>
Date: Wed, 7 Oct 2015 13:34:01 -0700
Message-ID: 
 <CALte62zssEf3_xbJcdzZOt7RnB2+wKJuhuXbnAsAZW4Cv8m5sQ@mail.gmail.com>
Subject: Re: Minor compacts seem blocked on major compacts
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c0844aab627ae052189aaa3

--94eb2c0844aab627ae052189aaa3
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Have you seen the following code ?

   ThreadPoolExecutor pool =3D (selectNow &&
s.throttleCompaction(compaction.getRequest().getSize()))
      ? longCompactions : shortCompactions;

Looks like throttleCompaction() returned false.
Please see the following method in RatioBasedCompactionPolicy :

  public boolean throttleCompaction(long compactionSize) {

    return compactionSize > comConf.getThrottlePoint();

FYI

On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <rfox@connexity.com> wrote:

> Looking at the source I don=E2=80=99t see how a manual major compact coul=
d get in
> the large compaction pool:
> If selectNow Is True the pool selector has to be shortCompactions.  From
> the logs it sure seems that selectNow has to be true, else it would have
> logged =E2=80=9Csystem=E2=80=9D instead of
> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactio=
nContext@7f236eb8
> ;=E2=80=9D
>
>
> Sure seems like an hbase bug.
>
> Am I missing something?
>
> Thanks,
>
> Randy
>
>   /**
>    * @param r HRegion store belongs to
>    * @param s Store to request compaction on
>    * @param why Why compaction requested -- used in debug messages
>    * @param priority override the default priority (NO_PRIORITY =3D=3D de=
cide)
>    * @param request custom compaction request. Can be <tt>null</tt> in
> which case a simple
>    *          compaction will be used.
>    */
>   private synchronized CompactionRequest requestCompactionInternal(final
> HRegion r, final Store s,
>       final String why, int priority, CompactionRequest request, boolean
> selectNow)
>           throws IOException {
>     if (this.server.isStopped()
>         || (r.getTableDesc() !=3D null &&
> !r.getTableDesc().isCompactionEnabled())) {
>       return null;
>     }
>
>     CompactionContext compaction =3D null;
>     if (selectNow) {
>       compaction =3D selectCompaction(r, s, priority, request);
>       if (compaction =3D=3D null) return null; // message logged inside
>     }
>
>     // We assume that most compactions are small. So, put system
> compactions into small
>     // pool; we will do selection there, and move to large pool if
> necessary.
>     long size =3D selectNow ? compaction.getRequest().getSize() : 0;
>     ThreadPoolExecutor pool =3D (!selectNow && s.throttleCompaction(size)=
)
>       ? longCompactions : shortCompactions;
>     pool.execute(new CompactionRunner(s, r, compaction, pool));
>     if (LOG.isDebugEnabled()) {
>       String type =3D (pool =3D=3D shortCompactions) ? "Small " : "Large =
";
>       LOG.debug(type + "Compaction requested: " + (selectNow ?
> compaction.toString() : "system")
>           + (why !=3D null && !why.isEmpty() ? "; Because: " + why : "") =
+
> "; " + this);
>     }
>     return selectNow ? compaction.getRequest() : null;
>   }
>
>
>
>
>
> On 10/6/15, 9:58 PM, "Randy Fox" <rfox@connexity.com> wrote:
>
> >Sorry, I do not follow. I thought there was one thread per pool, one for
> large and one for small.  Thus my major compact should have been in the
> large pool and not blocking the small pool.  That is how it worked in 0.9=
4.
> >From the logs it demoted my compact to the small pool.
> >
> >-randy
> >
> >
> >
> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vladrodionov@gmail.com> wrote:
> >
> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G so =
it
> >>should have been put into the Large compaction pool.
> >>
> >>You answered your question yourself. Minor compaction in the same pool =
(1
> >>thread default) will be waiting until major is finished.
> >>
> >>-Vlad
> >>
> >>
> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <rfox@connexity.com> wrote:
> >>
> >>> 2015-10-06 14:50:35,644 INFO
> org.apache.hadoop.hbase.regionserver.HStore:
> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. in=
to
> >>>
> tmpdir=3Dhdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a935=
0a95f6c288e49/.tmp,
> >>> totalSize=3D4.8 G
> >>> 2015-10-06 14:50:35,646 DEBUG
> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
> Compaction
> >>> requested:
> >>>
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompaction=
Context@7f236eb8
> ;
> >>> Because: User-triggered major compaction; compaction_queue=3D(0:0),
> >>> split_queue=3D0, merge_queue=3D0
> >>>
> >>>
> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so i=
t
> >>> should have been put into the Large compaction pool.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 10/6/15, 3:07 PM, "Randy Fox" <rfox@connexity.com> wrote:
> >>>
> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
> >>> compacts manually every night. One of the first things we noticed is
> that
> >>> when a major compact runs, no minors run.  The compaction queue grows
> and
> >>> when the major compact finishes the minors then run.   I have not
> found any
> >>> new knobs we should be setting.  Any ideas?
> >>> >Our config is:
> >>> >
> >>> ><property>
> >>> >  <name>hbase.hregion.majorcompaction</name>
> >>> >  <value>0</value>
> >>> >  <final>true</final>
> >>> ></property>
> >>> ><property>
> >>> >    <name>hbase.hstore.compactionThreshold</name>
> >>> >    <value>3</value>
> >>> ></property>
> >>> >
> >>> ><property>
> >>> >    <name>hbase.hstore.compaction.max</name>
> >>> >    <value>9</value>
> >>> ></property>
> >>> ><property>
> >>> >    <name>hbase.regionserver.thread.compaction.throttle</name>
> >>> >    <value>2684354560</value>
> >>> ></property>
> >>> >
> >>> >
> >>> >Thanks in advance,
> >>> >
> >>> >Randy
> >>>
>

--94eb2c0844aab627ae052189aaa3--