hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: [question about replication] how to apply delta from Master to Slave after crash ?
Date Fri, 26 Jul 2013 20:08:19 GMT

On Fri, Jul 26, 2013 at 12:59 PM, Jean-Daniel Cryans
<jdcryans@apache.org> wrote:
> I guess I didn't explain my ideas clearly.
> So, first, replication in HBase is master-push, so you don't want to
> reverse the process. It means that this tool needs to run on the
> master cluster.
> Then I don't think you need to specify a timestamp since the
> replication state is in ZK. Basically that tool we're talking about
> would be able to read the replication state of each master region
> server, finish replicating what's missing, and then clear that state
> in zookeeper.
> The code that handles replication does most of that already. Check
> ReplicationSourceManager and ReplicationSource. Basically when
> ReplicationSourceManager.init() is called, it will check all the
> queues in ZK and try to grab those that aren't attached to a region
> server. If the whole cluster is down, it will grab all of them.
> The beautiful thing here is that you could start that tool on all your
> machines and the load will be spread out, but that might not be a big
> concern if replication wasn't lagging since it would take a few
> seconds to finish replicating the missing data for each region server.
> I'll open a jira.
> J-D
> On Fri, Jul 26, 2013 at 11:50 AM, Demai Ni <nidmgg@gmail.com> wrote:
>> JD,
>> yeah. that sounds what I will need to do. a tool like this
>> [slave_cluster]$tool_to_syncup master_ZKquorum table_name start_timestamp
>> so two tasks for me:
>> 1) identify the start_timestamp
>> 2) write the tool_to_syncup which will reach to master_ZK, copy the HLOGs
>> from makster, replay the HLOGs on Slave.
>> are you aware of some example code for the 2) task that I can leverage?
>> thanks
>> Demai

View raw message