hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clara Xiong (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-15400) Multiple Output for Date Tiered Compaction
Date Sat, 05 Mar 2016 05:20:40 GMT
Clara Xiong created HBASE-15400:
-----------------------------------

             Summary: Multiple Output for Date Tiered Compaction
                 Key: HBASE-15400
                 URL: https://issues.apache.org/jira/browse/HBASE-15400
             Project: HBase
          Issue Type: Improvement
          Components: Compaction
            Reporter: Clara Xiong
            Assignee: Clara Xiong
             Fix For: 2.0.0


When we compact, we can output multiple files along the current window boundaries. There are
two use cases:

1. Major compaction: We want to output date tiered store files.
2. Bulk load files and the old file generated by major compaction before upgrading to DTCP.

Pros: 
1. Restore locality, process versioning, updates and deletes while maintaining the tiered
layout.
2. The best way to fix a skewed layout.
 
I am starting on a prototype of date tiered file writer from HBASE-15389 and will upload a
patch soon. I have to call out a few design decisions:

1. We only want to output the files along all windows for major compaction. 

2. For minor compaction, we don't want to output too many files, which will remain around
because of current restriction of contiguous compaction by seq id. I will only output two
files if all the files in the windows are being combined, one for the data within window and
the other for the out-of-window tail. If there is any file in the window excluded from compaction,
only one file will be output from compaction. When the windows are promoted, the situation
of out of order data will gradually improve.

3. We have to pass the boundaries with the list of store file as a complete time snapshot
instead of two separate calls because window layout is determined by the time the computation
is called. So we will need new type of compaction request. 
4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp
subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which
sort by seq id and other criteria. I will use this order for DTCP only, to avoid impacting
other compaction policies. 
5. We need some cleanup of current design of StoreEngine and CompactionPolicy.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message