accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4353) Stabilize tablet assignment during transient failure
Date Thu, 23 Jun 2016 21:32:16 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347238#comment-15347238
] 

Josh Elser commented on ACCUMULO-4353:
--------------------------------------

bq. If multiple tablet servers die in quick succession, such as from a rolling restart of
the Accumulo cluster or a network partition, this behavior can cause a storm of reassignment
and rebalancing, placing significant load on the master.

In the case of a rolling-restart, isn't that more of an operational issue to solve (give Accumulo
time to process the reassignments before restarting more nodes)?

Can you expand a bit more on the specifics behind a network partition that you're trying to
solve with this? Say, you lose a node? A rack? Half of your racks? Are we talking about a
5second interruption/delay? 30s? Minutes?

bq. To avert such load, Accumulo should be capable of maintaining a steady tablet assignment
state in the face of transient tablet server loss. Instead of reassigning tablets as quickly
as possible, Accumulo should be await the return of a temporarily downed tablet server (for
some configurable duration) before assigning its tablets to other tablet servers.

I'm a little worried about this as a configuration knob -- I feel like it kind of goes against
the highly-available distributed database which we expect Accumulo to be. When we don't reassign
tablets fast, that is a direct lack of availability for clients to read data.

bq. placing significant load on the master

Can you expand on this some more? Given that assignment is arguably the most important thing
for the Master to do, why are we concerned about letting the master do that as fast as it
can (for the aforementioned reason)? Do we need to come up with a more efficient way for the
master to handle the reassignment of many tablets? For example, I know that HBase has special
logic to batch assignments together in one RPC call to regionservers in order to bring things
online more quickly (I'm not sure if we have logic like that -- instead we just spam requests
to a tabletserver to load a tablet until it does so).

While seeing a pull request accompanying the issue reported, It seems a bit premature to me
to see code without some discussion on what the problems are and how best to solve them.

> Stabilize tablet assignment during transient failure
> ----------------------------------------------------
>
>                 Key: ACCUMULO-4353
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4353
>             Project: Accumulo
>          Issue Type: Improvement
>            Reporter: Shawn Walker
>            Assignee: Shawn Walker
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When a tablet server dies, Accumulo attempts to reassign the tablets it was hosting as
quickly as possible to maintain availability.  If multiple tablet servers die in quick succession,
such as from a rolling restart of the Accumulo cluster or a network partition, this behavior
can cause a storm of reassignment and rebalancing, placing significant load on the master.
> To avert such load, Accumulo should be capable of maintaining a steady tablet assignment
state in the face of transient tablet server loss.  Instead of reassigning tablets as quickly
as possible, Accumulo should be await the return of a temporarily downed tablet server (for
some configurable duration) before assigning its tablets to other tablet servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message