Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Fri, 24 Jun 2016 03:00:20 +0000 (UTC)
From: "Shawn Walker (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12982434.1466709542000.17648.1466737220259@Atlassian.JIRA>
In-Reply-To: <JIRA.12982434.1466709542000@Atlassian.JIRA>
References: <JIRA.12982434.1466709542000@Atlassian.JIRA> <JIRA.12982434.1466709542206@arcas>
Subject: [jira] [Commented] (ACCUMULO-4353) Stabilize tablet assignment
 during transient failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 24 Jun 2016 03:00:22 -0000


    [ https://issues.apache.org/jira/browse/ACCUMULO-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347595#comment-15347595 ] 

Shawn Walker commented on ACCUMULO-4353:
----------------------------------------

I did have rolling restarts as a primary motivation for doing this work, though a few other scenarios did come to mind as potential applications:
* tserver loses lock (possibly due to load), dies, and is restarted quickly via some external infrastructure, e.g. Puppet
* temporary network connectivity loss

I was thinking a `table.suspend.duration` on the order of 2-3 minutes might make sense for general purposes in a large cluster.  Long enough to catch most truly transient problems, sufficiently short that many applications wouldn't be unduly impacted.  Particularly seeing as any application already has to deal with a ~30 second wait before the master really notices a tablet server gone anyways.

I hadn't seen ACCUMULO-1454, I'll take a closer look in the morning. One concern that I had with some rolling-restart ideas was a matter of ops complexity.  In my (admittedly limited) experience, orchestrating a rolling restart that needs to do much more than "kill daemon, restart daemon" over a large cluster can be a huge headache.


> Stabilize tablet assignment during transient failure
> ----------------------------------------------------
>
>                 Key: ACCUMULO-4353
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4353
>             Project: Accumulo
>          Issue Type: Improvement
>            Reporter: Shawn Walker
>            Assignee: Shawn Walker
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When a tablet server dies, Accumulo attempts to reassign the tablets it was hosting as quickly as possible to maintain availability.  If multiple tablet servers die in quick succession, such as from a rolling restart of the Accumulo cluster or a network partition, this behavior can cause a storm of reassignment and rebalancing, placing significant load on the master.
> To avert such load, Accumulo should be capable of maintaining a steady tablet assignment state in the face of transient tablet server loss.  Instead of reassigning tablets as quickly as possible, Accumulo should be await the return of a temporarily downed tablet server (for some configurable duration) before assigning its tablets to other tablet servers.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)