hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
Date Tue, 15 Nov 2016 10:50:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15666823#comment-15666823
] 

Anoop Sam John commented on HBASE-16981:
----------------------------------------

Oh I see. Thanks.
Say we have compaction frequency as weekly once and as of now it will make 7 files as it is
daily grouping.  And as per monthly policy this would have been grouped as one file. Correct.
  And next week compaction will again compact the last week's file + this week's to make one
file (as the goal is one file per month)  So the IO will be more than old way?  Just trying
to understand the full diff.. Sorry if am missing some obvious things.  Did not check MOB
area from a long time now and I might have forgot many things.

> Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
> -------------------------------------------------------------------------------
>
>                 Key: HBASE-16981
>                 URL: https://issues.apache.org/jira/browse/HBASE-16981
>             Project: HBase
>          Issue Type: New Feature
>          Components: mob
>    Affects Versions: 2.0.0
>            Reporter: huaxiang sun
>            Assignee: huaxiang sun
>         Attachments: HBASE-16981.master.001.patch, HBASE-16981.master.002.patch, Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily partition mob compaction
policy, after major mob compaction, there is still one file per region daily. Given there
is 365 days in one year, at least 365 files per region. Since HDFS has limitation for number
of files under one folder, this is not going to scale if there are lots of regions. To reduce
mob file number,  we want to introduce other partition policies such as weekly, monthly to
compact mob files within one week or month into one file. This jira is create to track this
effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message