hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clara Xiong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
Date Thu, 10 Mar 2016 17:33:41 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189601#comment-15189601
] 

Clara Xiong commented on HBASE-15181:
-------------------------------------

There are a few tweaks made in this patch from Cassandra's design:
1. Use maxTimestamp instead of minTimestamp to favor late arriving data.
2. Plug in a compaction policy (default to exploring compaction)per window to reduce wasteful
compaction.
3. Normalize timestamp to sequence id to guarantee contiguous compaction by seq id for correctness
while paying some performance penalty for out-of-order data.

The subsequent work HBASE-15400 so far is to deal with major compaction and very large bulk
load files so we can still maintain a date tiered layer. 

> A simple implementation of date based tiered compaction
> -------------------------------------------------------
>
>                 Key: HBASE-15181
>                 URL: https://issues.apache.org/jira/browse/HBASE-15181
>             Project: HBase
>          Issue Type: New Feature
>          Components: Compaction
>            Reporter: Clara Xiong
>            Assignee: Clara Xiong
>             Fix For: 2.0.0, 1.3.0, 0.98.18
>
>         Attachments: HBASE-15181-0.98-ADD.patch, HBASE-15181-0.98.patch, HBASE-15181-0.98.v4.patch,
HBASE-15181-98.patch, HBASE-15181-ADD.patch, HBASE-15181-branch-1.patch, HBASE-15181-master-v1.patch,
HBASE-15181-master-v2.patch, HBASE-15181-master-v3.patch, HBASE-15181-master-v4.patch, HBASE-15181-v1.patch,
HBASE-15181-v2.patch
>
>
> This is a simple implementation of date-based tiered compaction similar to Cassandra's
for the following benefits:
> 1. Improve date-range-based scan by structuring store files in date-based tiered layout.
> 2. Reduce compaction overhead.
> 3. Improve TTL efficiency.
> Perfect fit for the use cases that:
> 1. has mostly date-based date write and scan and a focus on the most recent data. 
> 2. never or rarely deletes data.
> Out-of-order writes are handled gracefully. Time range overlapping among store files
is tolerated and the performance impact is minimized.
> Configuration can be set at hbase-site.xml or overriden at per-table or per-column-famly
level by hbase shell.
> Design spec is at https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing
> Results in our production is at https://docs.google.com/document/d/1GqRtQZMMkTEWOijZc8UCTqhACNmdxBSjtAQSYIWsmGU/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message