Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 6 Mar 2016 10:29:41 +0000 (UTC)
From: "Duo Zhang (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12947415.1457155227000.14995.1457260181199@Atlassian.JIRA>
In-Reply-To: <JIRA.12947415.1457155227000@Atlassian.JIRA>
References: <JIRA.12947415.1457155227000@Atlassian.JIRA>
 <JIRA.12947415.1457155227935@arcas>
Subject: [jira] [Commented] (HBASE-15400) Using Multiple Output for Date
 Tiered Compaction
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182080#comment-15182080 ] 

Duo Zhang commented on HBASE-15400:
-----------------------------------

For minor compaction, we can only assign different seqIds which count is less than the input files. But for major compaction, we do not need to reuse the seqIds of the input files I think? If we have N output files, we could just use {{seqId - N + 1}} ~ {{seqId}} as the seqIds of the output files? And if we can write out N files, it means that we should have more than N entries so the seqId must be greater than N?

Thanks.

> Using Multiple Output for Date Tiered Compaction
> ------------------------------------------------
>
>                 Key: HBASE-15400
>                 URL: https://issues.apache.org/jira/browse/HBASE-15400
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Compaction
>            Reporter: Clara Xiong
>            Assignee: Clara Xiong
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15400.patch
>
>
> When we compact, we can output multiple files along the current window boundaries. There are two use cases:
> 1. Major compaction: We want to output date tiered store files.
> 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP.
> Pros: 
> 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout.
> 2. The best way to fix a skewed layout.
>  
> This work is based on a prototype of date tiered file writer from HBASE-15389. I have to call out a few design decisions:
> 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the sizes of the maximum tier window size determined by base window size, windows per tier and max age.
> 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve.
> 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 
> 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting other compaction policies. 
> 5. We need some cleanup of current design of StoreEngine and CompactionPolicy.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)