hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Singhi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
Date Wed, 28 Oct 2015 07:42:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14977904#comment-14977904
] 

Ashish Singhi commented on HBASE-13153:
---------------------------------------

Following problems were faced in our internal testing,

*Problem 1:*
Replication for bulk loaded data was not working HDFS HA cluster.

*Solution:*
Sink cluster will need the source cluster hdfs client configuration to be able to perform
its operation. So we have taken the below approach,

a) Configure unique replication cluster ID <hbase.replication.cluster.id> for each Source
cluster.  Example: dc1, dc2
b) Each source cluster needs to place the HDFS client configurations in peer cluster, in <hbase.replication.cluster.id>
of source cluster under <hbase.replication.conf.dir> directory.
c) During replication, source cluster will send its unique replication id to peer cluster
in the request.  Peer cluster will identify this source cluster hdfs client configurations
based on this id and configured replication configuration directory and will perform replication.

!https://imageshack.com/i/p3Ik2xanp!

This approach requires two more configurations,
i. At source cluster, hbase.replication.cluster.id (mandatory when hbase.replication.bulkload.enabled
is set to true)
ii. At sink cluster, hbase.replication.conf.dir (default hbase configuration directory)

*Problem 2:* 
If source and sink cluster are sharing same HDFS then the hfile from source cluster is moved
to sink cluster, instead of copy as per the logic in {{SecureBulkLoadEndPoint}}

*Solution:*
To avoid this we will copy the hfiles from source cluster into sink cluster staging directory
parallely in the {{HFileReplicationCallable}} class itself before sending the request to {{LoadIncrementalHFiles}}
and will ensure to avoid copy/rename again in {{SecureBulkLoadEndPoint}} to staging directory.
This will also solve the problem "the bulk load to peer cluster needs a split, we will do
split by reading each cell from remote src cluster HFile. This will be a costly op" found
during a offline discussion with [~anoopsamjohn].

*Problem 3:*
If add_peer is done before enabling replication for bulk load data, then to that peer, bulk
loaded data was not getting replicated.

*Solution:*
As part RS startup, we init {{ReplicationSourceManager}} there we will check if all the peers
in the cluster have their node in hfile-refs znode.



Please review the above solutions, if it seems ok then I will post a new patch based on above
solutions and update the doc.

> Bulk Loaded HFile Replication
> -----------------------------
>
>                 Key: HBASE-13153
>                 URL: https://issues.apache.org/jira/browse/HBASE-13153
>             Project: HBase
>          Issue Type: New Feature
>          Components: Replication
>            Reporter: sunhaitao
>            Assignee: Ashish Singhi
>             Fix For: 2.0.0
>
>         Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, HBASE-13153-v11.patch,
HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase
Bulk Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster tolerance scenario.But
we encounter an issue that we will use bulkload very frequently,because bulkload bypass write
path, and will not generate WAL, so the data will not be replicated to backup cluster. It's
inappropriate to bukload twice both on active cluster and backup cluster. So i advise do some
modification to bulkload feature to enable bukload to both active cluster and backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message