accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "C Drew Thornton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1572) single node zookeeper failure kills connected accumulo servers
Date Tue, 16 Jul 2013 14:48:48 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709824#comment-13709824
] 

C Drew Thornton commented on ACCUMULO-1572:
-------------------------------------------

There are two configurations where I have experienced this:

1) The Zookeeper leader is removed from the ensemble and a new leader is elected.
  Result: Master immediately goes down along with the tablet servers.

2) According to http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html, the leader should
not serve clients in 5+ node ensemble (zoo.cfg - leaderServes=no). Accumulo does not seem
to anticipate this situation, so perhaps when a node is removed, the clients attempt to reestablish
connection at the leader, who is not serving, and they fail. This may appear as two failures
in the ensemble.
  Result: Tablet servers die who were clients of the removed node. If the master was one of
the clients, then the cluster goes down.

Versions:
Zookeeper 3.4.5 - 5 nodes with 4 processors and 6GB system memory (-Xmx4096m -XX:ParallelGCThreads=8
-XX:+UseConcMarkSweepGC)
CDH 4.3 - MRv1, no YARN - no special configuration
Accumulo 1.5.0 - adjusted for system memory
                
> single node zookeeper failure kills connected accumulo servers
> --------------------------------------------------------------
>
>                 Key: ACCUMULO-1572
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1572
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master, tserver
>    Affects Versions: 1.5.0
>            Reporter: Eric Newton
>            Priority: Blocker
>             Fix For: 1.5.1
>
>
> Drew Thornton writes on the user mailing list:
> {quote}
> If one zookeeper node is shutdown/fails/whatever and the rest of the ensemble stays up,
the tablet servers attached as clients to the shutdown node immediately fail. If one of the
clients happens to be the master, the cluster goes down.
> Accumulo does not seem to be failing over to the remaining zookeeper nodes, and this
causes me to restart the individual tablet servers again.
> The zookeeper ensemble is very stable and has plenty of bandwidth/memory/processing,
so taking one node down out of five doesn't crash the zookeepers, just the tablet servers...
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message