Return-Path: X-Original-To: apmail-hbase-commits-archive@www.apache.org Delivered-To: apmail-hbase-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3C9C61852B for ; Mon, 29 Feb 2016 19:08:33 +0000 (UTC) Received: (qmail 35173 invoked by uid 500); 29 Feb 2016 19:08:12 -0000 Delivered-To: apmail-hbase-commits-archive@hbase.apache.org Received: (qmail 35086 invoked by uid 500); 29 Feb 2016 19:08:12 -0000 Mailing-List: contact commits-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list commits@hbase.apache.org Received: (qmail 34118 invoked by uid 99); 29 Feb 2016 19:08:11 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Feb 2016 19:08:11 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 4DA33E788F; Mon, 29 Feb 2016 19:08:11 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: misty@apache.org To: commits@hbase.apache.org Date: Mon, 29 Feb 2016 19:08:36 -0000 Message-Id: <3d0cf66d92b247ae9a7a07927c97a607@git.apache.org> In-Reply-To: <8a6679a68b71466c96c944b62248c43f@git.apache.org> References: <8a6679a68b71466c96c944b62248c43f@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [27/51] [partial] hbase-site git commit: Published site at 88f775996b3b52d784ad13ab07515134619316ba. http://git-wip-us.apache.org/repos/asf/hbase-site/blob/bdd1f3f0/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.html ---------------------------------------------------------------------- diff --git a/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.html b/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.html new file mode 100644 index 0000000..6ed6731 --- /dev/null +++ b/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DateTieredCompactionPolicy.html @@ -0,0 +1,592 @@ + + + + + +DateTieredCompactionPolicy (Apache HBase 2.0.0-SNAPSHOT API) + + + + + + +
+ + + + + +
+
+ + + +
+ +
+
+ + +
+ + +
+ + +
+
org.apache.hadoop.hbase.regionserver.compactions
+

Class DateTieredCompactionPolicy

+
+
+ +
+
    +
  • +
    +
    +
    @InterfaceAudience.LimitedPrivate(value="Configuration")
+public class DateTieredCompactionPolicy
+extends RatioBasedCompactionPolicy
    +
    HBASE-15181 This is a simple implementation of date-based tiered compaction similar to + Cassandra's for the following benefits: + 1. Improve date-range-based scan by structuring store files in date-based tiered layout. + 2. Reduce compaction overhead. + 3. Improve TTL efficiency. + Perfect fit for the use cases that: + 1. has mostly date-based data write and scan and a focus on the most recent data. + 2. never or rarely deletes data. Out-of-order writes are handled gracefully. Time range + overlapping among store files is tolerated and the performance impact is minimized. Configuration + can be set at hbase-site or overriden at per-table or per-column-famly level by hbase shell. + Design spec is at + https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/
    +
    • +
+
+
+ +
+
+
    +
  • + + + + + +
      +
    • + + +

      Method Detail

      + + + + + + + + + + + + + + + +
        +
      • +

        applyCompactionPolicy

        +
        public ArrayList<StoreFile> applyCompactionPolicy(ArrayList<StoreFile> candidates,
+                                         boolean mayUseOffPeak,
+                                         boolean mayBeStuck)
+                                           throws IOException
        +
        Could return null if no candidates are found
        +
        +
        Overrides:
        +
        applyCompactionPolicy in class RatioBasedCompactionPolicy
        +
        Parameters:
        candidates - pre-filtrate
        +
        Returns:
        filtered subset + -- Default minor compaction selection algorithm: + choose CompactSelection from candidates -- + First exclude bulk-load files if indicated in configuration. + Start at the oldest file and stop when you find the first file that + meets compaction criteria: + (1) a recently-flushed, small file (i.e. <= minCompactSize) + OR + (2) within the compactRatio of sum(newer_files) + Given normal skew, any newer files will also meet this criteria +

        + Additional Note: + If fileSizes.size() >> maxFilesToCompact, we will recurse on + compact(). Consider the oldest files first to avoid a + situation where we always compact [end-threshold,end). Then, the + last file becomes an aggregate of the previous compactions. + + normal skew: + + older ----> newer (increasing seqID) + _ + | | _ + | | | | _ + --|-|- |-|- |-|---_-------_------- minCompactSize + | | | | | | | | _ | | + | | | | | | | | | | | | + | | | | | | | | | | | |

        +
        Throws:
        +
        IOException
        +
        • +
      + + + + + + + +
        +
      • +

        newestBucket

        +
        private ArrayList<StoreFile> newestBucket(List<ArrayList<StoreFile>> buckets,
+                                int incomingWindowThreshold,
+                                long now,
+                                long baseWindowMillis,
+                                boolean mayUseOffPeak)
+                                   throws IOException
        +
        Parameters:
        buckets - the list of buckets, sorted from newest to oldest, from which to return the + newest bucket within thresholds.
        incomingWindowThreshold - minimum number of storeFiles in a bucket to qualify.
        maxThreshold - maximum number of storeFiles to compact at once (the returned bucket will + be trimmed down to this).
        +
        Returns:
        a bucket (a list of store files within a window to be compacted).
        +
        Throws:
        +
        IOException - error
        +
        • +
      + + + +
        +
      • +

        partitionFilesToBuckets

        +
        private static List<ArrayList<StoreFile>> partitionFilesToBuckets(Iterable<StoreFile> storeFiles,
+                                                 long baseWindowSizeMillis,
+                                                 int windowsPerTier,
+                                                 long now)
        +
        We receive store files sorted in ascending order by seqId then scan the list of files. If the + current file has a maxTimestamp older than last known maximum, treat this file as it carries + the last known maximum. This way both seqId and timestamp are in the same order. If files carry + the same maxTimestamps, they are ordered by seqId. We then reverse the list so they are ordered + by seqId and maxTimestamp in decending order and build the time windows. All the out-of-order + data into the same compaction windows, guaranteeing contiguous compaction based on sequence id.
        +
        • +
      + + + +
        +
      • +

        filterOldStoreFiles

        +
        private static Iterable<StoreFile> filterOldStoreFiles(List<StoreFile> storeFiles,
+                                      long maxAge,
+                                      long now)
        +
        Removes all store files with max timestamp older than (current - maxAge).
        +
        Parameters:
        storeFiles - all store files to consider
        maxAge - the age in milliseconds when a store file stops participating in compaction.
        now - current time. store files with max timestamp less than (now - maxAge) are filtered.
        +
        Returns:
        a list of storeFiles with the store file older than maxAge excluded
        +
        • +
      + + + + +
      • +
    +
    • +
+
+
+ + +
+ + + + + +
+
+ + + +
+ +
+
+ + +
+ + +
+ +

Copyright © 2007–2016 The Apache Software Foundation. All rights reserved.

+ + http://git-wip-us.apache.org/repos/asf/hbase-site/blob/bdd1f3f0/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.html ---------------------------------------------------------------------- diff --git a/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.html b/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.html index 9a46fea..a84db12 100644 --- a/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.html +++ b/devapidocs/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.html @@ -35,7 +35,7 @@