Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 54AFD1788D for ; Fri, 2 Oct 2015 06:33:27 +0000 (UTC) Received: (qmail 38637 invoked by uid 500); 2 Oct 2015 06:33:27 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 38586 invoked by uid 500); 2 Oct 2015 06:33:27 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 38575 invoked by uid 99); 2 Oct 2015 06:33:27 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2015 06:33:27 +0000 Date: Fri, 2 Oct 2015 06:33:27 +0000 (UTC) From: "Vladimir Rodionov (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940816#comment-14940816 ] Vladimir Rodionov commented on HBASE-14468: ------------------------------------------- HBASE-14477 - now its DateTieredCompaction :) > Compaction improvements: FIFO compaction policy > ----------------------------------------------- > > Key: HBASE-14468 > URL: https://issues.apache.org/jira/browse/HBASE-14468 > Project: HBase > Issue Type: Improvement > Reporter: Vladimir Rodionov > Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14468-v1.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch, HBASE-14468-v4.patch > > > h2. FIFO Compaction > h3. Introduction > FIFO compaction policy selects only files which have all cells expired. The column family MUST have non-default TTL. > Essentially, FIFO compactor does only one job: collects expired store files. I see many applications for this policy: > # use it for very high volume raw data which has low TTL and which is the source of another data (after additional processing). Example: Raw time-series vs. time-based rollup aggregates and compacted time-series. We collect raw time-series and store them into CF with FIFO compaction policy, periodically we run task which creates rollup aggregates and compacts time-series, the original raw data can be discarded after that. > # use it for data which can be kept entirely in a a block cache (RAM/SSD). Say we have local SSD (1TB) which we can use as a block cache. No need for compaction of a raw data at all. > Because we do not do any real compaction, we do not use CPU and IO (disk and network), we do not evict hot data from a block cache. The result: improved throughput and latency both write and read. > See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style > h3. To enable FIFO compaction policy > For table: > {code} > HTableDescriptor desc = new HTableDescriptor(tableName); > desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, > FIFOCompactionPolicy.class.getName()); > {code} > For CF: > {code} > HColumnDescriptor desc = new HColumnDescriptor(family); > desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, > FIFOCompactionPolicy.class.getName()); > {code} > h3. Limitations > Do not use FIFO compaction if : > * Table/CF has MIN_VERSION > 0 > * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL) > * Table/CF is MOB -- This message was sent by Atlassian JIRA (v6.3.4#6332)