hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
Date Fri, 17 Feb 2017 04:35:41 GMT

     [ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Anoop Sam John updated HBASE-16981:
-----------------------------------
    Release Note: 
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY
=> 'weekly'}
 
or 

hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY
=> 'monthly'}

Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and "monthly", the default
is "daily".

When it is "weekly" policy, the mob compaction will try to compact files within one calendar
week into one for a specific partition, similar for "daily" and "monthly".

With "weekly" policy, one mob file normally is compacted twice during its lifetime (that is
first on daily basis and then all such daily based compacted files belonging to a week at
the weekly interval), for one region, there normally are 52 files for one year. With "Monthly"
policy, one mob file normally is compacted 3 times during its lifetime (First daily and then
weekly followed by monthly at end of every month) and normally there are 12 files for one
year.


  was:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY
=> 'weekly'}
 
or 

hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY
=> 'monthly'}

Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and "monthly", the default
is "daily".

When it is "weekly" policy, the mob compaction will try to compact files within one calendar
week into one for a specific partition, similar for "daily" and "monthly".

With "weekly" policy, one mob file normally is compacted twice during its lifetime, for one
region, there normally are 52 files for one year. With "Monthly" policy, one mob file normally
is compacted 3 times during its lifetime and normally there are 12 files for one year.



> Expand Mob Compaction Partition policy from daily to weekly, monthly
> --------------------------------------------------------------------
>
>                 Key: HBASE-16981
>                 URL: https://issues.apache.org/jira/browse/HBASE-16981
>             Project: HBase
>          Issue Type: New Feature
>          Components: mob
>    Affects Versions: 2.0.0
>            Reporter: huaxiang sun
>            Assignee: huaxiang sun
>             Fix For: 2.0.0
>
>         Attachments: HBASE-16981.master.001.patch, HBASE-16981.master.002.patch, HBASE-16981.master.003.patch,
HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, HBASE-16981.master.006.patch,
HBASE-16981.master.007.patch, HBASE-16981.master.008.patch, Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily partition mob compaction
policy, after major mob compaction, there is still one file per region daily. Given there
is 365 days in one year, at least 365 files per region. Since HDFS has limitation for number
of files under one folder, this is not going to scale if there are lots of regions. To reduce
mob file number,  we want to introduce other partition policies such as weekly, monthly to
compact mob files within one week or month into one file. This jira is create to track this
effort.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message