hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Singhi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
Date Tue, 20 Oct 2015 09:03:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964824#comment-14964824
] 

Ashish Singhi commented on HBASE-13153:
---------------------------------------

During a offline discussion with Anoop on this, we found that when the source hfiles are in
a different FS and if the hfile requires a split then LoadIncrementalHFiles will open a remote
reader to source hfile, scan the file and append the data to each of the file split. 
Since we anyway copy the hfiles to the local FS if the source hfiles are in remote FS later,
so we thought we can optimize this by copying the hfiles to a temp directory in local FS if
source hfiles are in a different FS first and then do a local read and write.

This is related to LoadIncrementalHFiles, when ever the source hfiles are in a different FS
so I will handle this as part of another jira which will be subtask of this.
So in this jira there will be no change in the patch or doc related to this.

Any further review comments on the patch will be really appreciated.
Thanks Ted, Ram, Anoop and Matteo for the reviews till now.

> Bulk Loaded HFile Replication
> -----------------------------
>
>                 Key: HBASE-13153
>                 URL: https://issues.apache.org/jira/browse/HBASE-13153
>             Project: HBase
>          Issue Type: New Feature
>          Components: Replication
>            Reporter: sunhaitao
>            Assignee: Ashish Singhi
>             Fix For: 2.0.0
>
>         Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, HBASE-13153-v11.patch,
HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase
Bulk Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster tolerance scenario.But
we encounter an issue that we will use bulkload very frequently,because bulkload bypass write
path, and will not generate WAL, so the data will not be replicated to backup cluster. It's
inappropriate to bukload twice both on active cluster and backup cluster. So i advise do some
modification to bulkload feature to enable bukload to both active cluster and backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message