hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Split issue for multiple ColumnFamilies
Date Wed, 01 Jun 2011 04:10:56 GMT
Schubert:
Are you able to upgrade to 0.90.3 ?

On Tue, May 31, 2011 at 7:06 PM, Schubert Zhang <zsongbo@gmail.com> wrote:

> Another issue:
>
> in Store.compct()
>
>      if (!majorcompaction && !references &&
>          (forceSplit || (filesToCompact.size() < compactionThreshold))) {
>        return checkSplit(forceSplit);
>      }
>
> In some cases, (filesToCompact.size() >= compactionThreshold), but the
> following compaction logic still cannot compact to make number of
> storefiles
> lower, since the file-selection-policy of compaction is very complicated.
>
>
>
> On Wed, Jun 1, 2011 at 9:42 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
>
> > Thanks Ted.
> >
> > We are using 0.90.1 now.
> >
> > It's good news that new 0.90.3 and trunk resolved this issue.
> >
> >
> >
> > On Wed, Jun 1, 2011 at 1:35 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> >> Which version of HBase are you using ?
> >> In 0.90.3, I see this in Store.compact():
> >>    boolean forceSplit = this.region.shouldForceSplit();
> >> For trunk, the code has been rewritten.
> >>
> >> Cheers
> >>
> >> On Mon, May 30, 2011 at 11:27 PM, Schubert Zhang <zsongbo@gmail.com>
> >> wrote:
> >>
> >> > We found a issue of forceSplit.
> >> >
> >> > 1. My schema:
> >> >    We have a table with multiple column families: A, B, C, D
> >> >    After put many data, but there may be no data (no storeFile) for
> some
> >> > stores.
> >> >    for example: store A and B is empty, but C and D have many data
> now.
> >> >
> >> > 2. Then I request a split from hbase shell or the web GUI, but nothing
> >> > happend.
> >> >
> >> > 3. Then we check the code, it may be a bug for multiple CFs.
> >> >    (1) split request.
> >> >         RegionServer:   region.shouldSplit(true)
> >> >         and enqueue this request
> >> >    (2) HRegion.compactStores(false)
> >> >         for (Store store: stores.values()) {
> >> >            final Store.StoreSize ss = store.compact(majorCompaction);
> >> >            lastCompactSize += store.getLastCompactSize();
> >> >            if (ss != null && ss.getSize() > maxSize) {
> >> >              maxSize = ss.getSize();
> >> >              splitRow = ss.getSplitRow();
> >> >            }
> >> >          }
> >> >         but, for store A and store B, store.compact(false) returns
> null.
> >> >     (3) Store.compact(false)
> >> >          boolean forceSplit = this.region.shouldSplit(false);
> >> >          for the first store A and second store B, it set
> >> > this.splitRequest=false. This is a bug.
> >> >     (4) Then even for store C and D which have many data, forceSplit
> is
> >> > always false.
> >> >
> >> >
> >> > btw: seems the current code of compaction/spliting is so disordered
> and
> >> > confused.
> >> >
> >> > Schubert
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message