Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Fri, 24 Apr 2015 18:50:38 +0000 (UTC)
From: "Josh Elser (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12823845.1429901398000.21602.1429901438800@Atlassian.JIRA>
In-Reply-To: <JIRA.12823845.1429901398000@Atlassian.JIRA>
References: <JIRA.12823845.1429901398000@Atlassian.JIRA>
 <JIRA.12823845.1429901398924@arcas>
Subject: [jira] [Created] (ACCUMULO-3750) Bad instance.secret causes master
 to repeatedly fail fast attempting to acquire lock
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Josh Elser created ACCUMULO-3750:
------------------------------------

             Summary: Bad instance.secret causes master to repeatedly fail fast attempting to acquire lock
                 Key: ACCUMULO-3750
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3750
             Project: Accumulo
          Issue Type: Bug
          Components: master
    Affects Versions: 1.6.2, 1.6.1, 1.6.0, 1.5.2, 1.5.1, 1.5.0
            Reporter: Josh Elser
            Assignee: Josh Elser
             Fix For: 1.7.0, 1.6.3


Accidentally restarted a small cluster with bad configuration (missing instance.secret). The tabletservers bailed out quickly, but the master sat in a tight loop trying to get the lock.

{noformat}
2015-04-23 11:48:12,356 [trace.DistributedTrace] INFO : SpanReceiver org.apache.accumulo.tracer.ZooTraceClient was loaded successfully.
2015-04-23 11:48:12,357 [master.Master] INFO : trying to get master lock
2015-04-23 11:48:12,395 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:13,043 [server.Accumulo] WARN : System swappiness setting is greater than ten (60) which can cause time-sensitive operations to be delayed.  Accumulo is time sensitive because it needs to maintain distributed lo
ck agreement.
2015-04-23 11:48:13,410 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:14,418 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:15,426 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:16,433 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:17,440 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
2015-04-23 11:48:18,449 [master.Master] WARN : Failed to get master lock org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /accumulo/dc25a857-19d8-4387-bec0-64b4dc17cafb/masters/lock/zlock-
{noformat}

Looks like the only case which exits the Master when the lock is failed to be acquired is an illegal state where the master thinks it already has the lock.

If we get a NoAuthException, we should not attempt to get the lock again.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)