hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mukund murrali <mukundmurra...@gmail.com>
Subject Re: Major compaction skipping for older regions
Date Tue, 11 Aug 2015 11:39:02 GMT
Yes. We have min version to be 1, hence I suppose there wont be any sort of
time based compaction. After triggering major compaction we had only the
following logs

compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store
files, 0 compacting, 3 eligible, 50 blocking

Thanks

On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> bq. oldestTime -9223370597787064221ms
>
> This is due to minTimestamp missing from store file:
>
>         Long minTimestamp = sf.getMinimumTimestamp();
>
>         long oldest = (minTimestamp == null)
>
>             ? Long.MIN_VALUE
>
>             : now - minTimestamp.longValue();
>
> Can you pastebin log from RatioBasedCompactionPolicy in region server log
> after the manual compaction ?
>
> Thanks
>
> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <mukundmurrali9@gmail.com>
> wrote:
>
> > We are using hbase-1.0.0. The following logs appears for all regions in
> the
> > regionserver
> >
> > 2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of
> >
> >
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8
> > because one (major) compacted file only, oldestTime
> -9223370597787064221ms
> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
> >
> >
> > Yes after manual triggering the deletes purged. But we don't want to have
> > it manual. Any other config to avoid such scenario?
> >
> > Thanks
> >
> >
> >
> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > What release of hbase are you using ?
> > >
> > > Can you pastebin region server log with DEBUG logging ?
> > >
> > > I guess you have tried issuing manual command. Did it work ?
> > >
> > > Thanks
> > >
> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali <
> > mukundmurrali9@gmail.com>
> > > wrote:
> > >
> > > > Any one help us in this :(  Are we missing somewhere in the use case?
> > > None
> > > > of the deleted cells are undergoing major compaction.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali <
> > > mukundmurrali9@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We wanted to have deleted data for a week. So we configured
> > > > >
> > > > > MIN_VERSIONS => 1
> > > > > KEEP_DELETED_CELLS => TTL
> > > > > TTL => 1 week.
> > > > >
> > > > > As per our understanding, after 1 week the deleted data becomes
> > > available
> > > > > for major compaction and should be purged (correct if wrong). Since
> > we
> > > > have
> > > > > time series data, we don't have any write operations in those
> regions
> > > > after
> > > > > a week . But major compaction never took place for any regions and
> > our
> > > > > overall size grew drastically though we have deletes happening.
> After
> > > > > analyzing, we found that major compaction takes place if any one
of
> > > the 2
> > > > > condition is satisfied.
> > > > >
> > > > > 1. If the time interval between major compaction is greater than
a
> > week
> > > > > (default config).
> > > > > 2. if the block locality index falls below a threshold.
> > > > >
> > > > > In our case, since we have min_versions to be 1, the first case
> > > condition
> > > > > fails. Time to verify is set to Long.Max value, if min versions is
> > not
> > > 0.
> > > > >
> > > > > Second is block locality.  To check the block locality index we
> > enabled
> > > > > fine logs. And we found the  block locality is always 1, and we got
> > > logs
> > > > > stating "Skipping major compaction......".
> > > > >
> > > > > So, in this case is manually triggering major compaction the only
> > > choice?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message