hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15339) Add archive tiers for date based tiered compaction
Date Fri, 26 Feb 2016 23:23:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170083#comment-15170083

Duo Zhang commented on HBASE-15339:

Oh, I see the picture in the design doc, it is not the same with Cassandra's DTCS. Fixed windows
is enough although aligned by calendar looks better and has some benefits on statistic. And
for our needs, we have replication, so we need to make sure the boundary is same on all the
replication connected clusters. This makes it easier to verify the data consistency between
these clusters.

Thanks [~davelatham], helps a lot. Let me learn the patch carefully and change the description
here. And for the config, for our service we only need to have calendar aligned boundaries
in the max tier. we do not need to archive data by days, then merge days to week, then month,
quarter and year. We still have a hot data tier which is relative to now(typically, one week).

> Add archive tiers for date based tiered compaction
> --------------------------------------------------
>                 Key: HBASE-15339
>                 URL: https://issues.apache.org/jira/browse/HBASE-15339
>             Project: HBase
>          Issue Type: Improvement
>          Components: Compaction
>            Reporter: Duo Zhang
> For our MiCloud service, the old data is rarely touched but we still need to keep it,
so we want to put the data on inexpensive device and reduce redundancy using EC to cut down
the cost.
> With date based tiered compaction introduced in HBASE-15181, new data and old data can
be placed in different tier. But the tier boundary moves as time lapse so it is still possible
that we do compaction on old tier which breaks our block moving and EC work.
> So here we want to introduce an "archive tier" to better fit our scenario. Add an configuration
called "archive unit", for example, year. That means, if we find that the tier boundary is
already in the previous year, then we reset the boundary to the start of year and end of year,
and if we want to do compaction in this tier, just compact all files into one file. The file
will never be changed unless we force a major compaction so it is safe to apply EC and other
cost reducing approach on the file. And we make more tiers before this tier year by year.

This message was sent by Atlassian JIRA

View raw message