hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
Date Sun, 14 Dec 2014 05:14:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245824#comment-14245824
] 

Lars Hofhansl commented on HBASE-12636:
---------------------------------------

Was thinking that if both cluster are active a write that is repeated much later will might
cause issues.

We can also drive down the frequency of the checks... Maybe? I.e. if the frequency is too
high, just check a little less often and let more data accumulate.


> Avoid too many write operations on zookeeper in replication
> -----------------------------------------------------------
>
>                 Key: HBASE-12636
>                 URL: https://issues.apache.org/jira/browse/HBASE-12636
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.94.11
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>              Labels: replication
>             Fix For: 1.0.0
>
>         Attachments: HBASE-12635-v2.diff, HBASE-12636-v1.diff
>
>
> In our production cluster, we found there are about over 1k write operations per second
on zookeeper from hbase replication. The reason is that the replication source will write
the log position to zookeeper for every edit shipping. If the current replicating WAL is just
the WAL that regionserver is writing to,  each skipping will be very small but the frequency
is very high, which causes many write operations on zookeeper.
> A simple solution is that writing log position to zookeeper when position diff or skipped
edit number is larger than a threshold, not every  edit shipping.
> Suggestions are welcomed, thx~



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message