hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Feng Honghua (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-9501) Provide throttling for replication
Date Fri, 31 Jan 2014 14:52:12 GMT

     [ https://issues.apache.org/jira/browse/HBASE-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Feng Honghua updated HBASE-9501:
--------------------------------

    Attachment: HBASE-9501-trunk_v2.patch

[~jdcryans] : Thanks very much for the review. New patch attached according to your review
feedback. The throttling logic is refactored out to be a helper method which computes the
sleep time(>0 if needed) according to the current context which are passed as parameters,
the real sleep occurs in shipEdits after calling this helper function. Now the newly added
unit test testApplyThrottling in TestReplicationSmallTests just checks the sleep time computed
by this helper function, but without real sleep operation/call. The original added testThrottling
is also run several times to verify this refactor is fine before being removed.

> Provide throttling for replication
> ----------------------------------
>
>                 Key: HBASE-9501
>                 URL: https://issues.apache.org/jira/browse/HBASE-9501
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>            Reporter: Feng Honghua
>            Assignee: Feng Honghua
>         Attachments: HBASE-9501-trunk_v0.patch, HBASE-9501-trunk_v1.patch, HBASE-9501-trunk_v2.patch
>
>
> When we disable a peer for a time of period, and then enable it, the ReplicationSource
in master cluster will push the accumulated hlog entries during the disabled interval to the
re-enabled peer cluster at full speed.
> If the bandwidth of the two clusters is shared by different applications, the push at
full speed for replication can use all the bandwidth and severely influence other applications.
> Though there are two config replication.source.size.capacity and replication.source.nb.capacity
to tweak the batch size each time a push delivers, but if decrease these two configs, the
number of pushes increase, and all these pushes proceed continuously without pause. And no
obvious help for the bandwidth throttling.
> From bandwidth-sharing and push-speed perspective, it's more reasonable to provide a
bandwidth up limit for each peer push channel, and within that limit, peer can choose a big
batch size for each push for bandwidth efficiency.
> Any opinion?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message