hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ryan rawson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2223) Handle 10min+ network partitions between clusters
Date Fri, 12 Feb 2010 22:26:28 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833203#action_12833203

ryan rawson commented on HBASE-2223:

the previous plan was to NOT delete the log files if replication needed them... Sounds like
you changed that?

If replication needs a log file and the server crashed and they are set to be split, we need
to NOT delete the logfile, move them to a holding area perhaps and then get someone to pick
the up and send them along.

A 2 hour outage isnt that much, I'd say that we should buffer logs until someone decides it's
not worth the disk space.  Ie: make it a top level admin action/alert and give the administrator
an option to drop the log retention and then do alternative catch up later.  

It would be better to hold on to a few TB of replication logs then replay that after 24-48
hours of downtime than to mess with the map-reduce stuff, since you'd have to be careful to
hopefully avoid duplicating keyvalues.

> Handle 10min+ network partitions between clusters
> -------------------------------------------------
>                 Key: HBASE-2223
>                 URL: https://issues.apache.org/jira/browse/HBASE-2223
>             Project: Hadoop HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.21.0
> We need a nice way of handling long network partitions without impacting a master cluster
(which pushes the data). Currently it will just retry over and over again.
> I think we could:
>  - Stop replication to a slave cluster if it didn't respond for more than 10 minutes
>  - Keep track of the duration of the partition
>  - When the slave cluster comes back, initiate a MR job like HBASE-2221 
> Maybe we want less than 10 minutes, maybe we want this to be all automatic or just the
first 2 parts. Discuss.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message