hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Singhi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13153) enable bulkload to support replication
Date Wed, 02 Sep 2015 17:42:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727711#comment-14727711
] 

Ashish Singhi commented on HBASE-13153:
---------------------------------------

bq. There seem to be two types of notifications from the peer cluster
No, only one. Sorry if I confused you.
At a very high level the sequence of operation will be something like below for a positive
cycle,
(1) doBulkLoad(HFiles, source cluster IDs)
(2) notify(BulkLoadListeners)
(3) ZK(add HFile entries only if no cyclic loop) \[return to the client\]
(4) ReplicateEntries(entries, source cluster ids) --> Back to step (1).

c1->c2
Complete bulk load process at cluster c1,
(1) doBulkLoad(HFiles)
(2) notify(BulkLoadListeners)
(3) ZK(add HFile entries, no cyclic loop -> replicate to c2) \[return to the client\]
(4) ReplicateEntries(entries, source cluster ids)

Replication process at cluster c2,
(1) doBulkLoad(HFiles,{c1\})
(2) notify(BulkLoadListeners) \[return to the client, client will delete the znode\]

For cyclic replication case,
c1->c2->c3->c1
Complete bulk load process at cluster c1,
(1) doBulkLoad(HFiles)
(2) notify(BulkLoadListeners)
(3) ZK(add HFile entries, no cyclic loop -> replicate to c2) \[return to the client\]
(4) ReplicateEntries(entries, source cluster ids)

Replication process at cluster c2,
(1) doBulkLoad(HFiles,{c1\})
(2) notify(BulkLoadListeners)
(3) ZK(add HFile entries, no cyclic loop -> replicate to c3) \[return to the client, client
will delete the znode\]
(4) ReplicateEntries(entries, source cluster ids) --> Back to step (1).

Replication process at cluster c3,
(1) doBulkLoad(HFiles,{c1,c2\})
(2) notify(BulkLoadListeners)
(3) ZK(add HFile entries, cyclic loop detected) \[return to the client, client will delete
the znode\]
Cyclic loop ends here

Hope I am able to clear your questions.

> enable bulkload to support replication
> --------------------------------------
>
>                 Key: HBASE-13153
>                 URL: https://issues.apache.org/jira/browse/HBASE-13153
>             Project: HBase
>          Issue Type: New Feature
>          Components: Replication
>            Reporter: sunhaitao
>            Assignee: Ashish Singhi
>             Fix For: 2.0.0
>
>         Attachments: HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster tolerance scenario.But
we encounter an issue that we will use bulkload very frequently,because bulkload bypass write
path, and will not generate WAL, so the data will not be replicated to backup cluster. It's
inappropriate to bukload twice both on active cluster and backup cluster. So i advise do some
modification to bulkload feature to enable bukload to both active cluster and backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message