hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2223) Handle 10min+ network partitions between clusters
Date Fri, 12 Feb 2010 22:36:28 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833209#action_12833209
] 

Jean-Daniel Cryans commented on HBASE-2223:
-------------------------------------------

{quote}
the previous plan was to NOT delete the log files if replication needed them... Sounds like
you changed that?
If replication needs a log file and the server crashed and they are set to be split, we need
to NOT delete the logfile, move them to a holding area perhaps and then get someone to pick
the up and send them along.
{quote}

That's HBASE-2070.

{quote}
A 2 hour outage isnt that much, I'd say that we should buffer logs until someone decides it's
not worth the disk space. Ie: make it a top level admin action/alert and give the administrator
an option to drop the log retention and then do alternative catch up later.

It would be better to hold on to a few TB of replication logs then replay that after 24-48
hours of downtime than to mess with the map-reduce stuff, since you'd have to be careful to
hopefully avoid duplicating keyvalues.
{quote}

Yes, that's how I see we could do it if we don't do the MR path.

> Handle 10min+ network partitions between clusters
> -------------------------------------------------
>
>                 Key: HBASE-2223
>                 URL: https://issues.apache.org/jira/browse/HBASE-2223
>             Project: Hadoop HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.21.0
>
>
> We need a nice way of handling long network partitions without impacting a master cluster
(which pushes the data). Currently it will just retry over and over again.
> I think we could:
>  - Stop replication to a slave cluster if it didn't respond for more than 10 minutes
>  - Keep track of the duration of the partition
>  - When the slave cluster comes back, initiate a MR job like HBASE-2221 
> Maybe we want less than 10 minutes, maybe we want this to be all automatic or just the
first 2 parts. Discuss.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message