accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Walker (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-4353) Stabilize tablet assignment during transient failure
Date Fri, 24 Jun 2016 03:00:20 GMT


Shawn Walker commented on ACCUMULO-4353:

I did have rolling restarts as a primary motivation for doing this work, though a few other
scenarios did come to mind as potential applications:
* tserver loses lock (possibly due to load), dies, and is restarted quickly via some external
infrastructure, e.g. Puppet
* temporary network connectivity loss

I was thinking a `table.suspend.duration` on the order of 2-3 minutes might make sense for
general purposes in a large cluster.  Long enough to catch most truly transient problems,
sufficiently short that many applications wouldn't be unduly impacted.  Particularly seeing
as any application already has to deal with a ~30 second wait before the master really notices
a tablet server gone anyways.

I hadn't seen ACCUMULO-1454, I'll take a closer look in the morning. One concern that I had
with some rolling-restart ideas was a matter of ops complexity.  In my (admittedly limited)
experience, orchestrating a rolling restart that needs to do much more than "kill daemon,
restart daemon" over a large cluster can be a huge headache.

> Stabilize tablet assignment during transient failure
> ----------------------------------------------------
>                 Key: ACCUMULO-4353
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>            Reporter: Shawn Walker
>            Assignee: Shawn Walker
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
> When a tablet server dies, Accumulo attempts to reassign the tablets it was hosting as
quickly as possible to maintain availability.  If multiple tablet servers die in quick succession,
such as from a rolling restart of the Accumulo cluster or a network partition, this behavior
can cause a storm of reassignment and rebalancing, placing significant load on the master.
> To avert such load, Accumulo should be capable of maintaining a steady tablet assignment
state in the face of transient tablet server loss.  Instead of reassigning tablets as quickly
as possible, Accumulo should be await the return of a temporarily downed tablet server (for
some configurable duration) before assigning its tablets to other tablet servers.

This message was sent by Atlassian JIRA

View raw message