Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Mon, 1 Jun 2015 21:54:18 +0000 (UTC)
From: "Jonathan Hurley (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12834391.1433195614000.87365.1433195658440@Atlassian.JIRA>
In-Reply-To: <JIRA.12834391.1433195614000@Atlassian.JIRA>
References: <JIRA.12834391.1433195614000@Atlassian.JIRA>
 <JIRA.12834391.1433195614721@arcas>
Subject: [jira] [Created] (ACCUMULO-3880) Malformed Configuration Causes
 tservers To Shutdown
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Jonathan Hurley created ACCUMULO-3880:
-----------------------------------------

             Summary: Malformed Configuration Causes tservers To Shutdown
                 Key: ACCUMULO-3880
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3880
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
         Environment: HDP 2.2.7.0 to HDP 2.3.0.0 Upgrade
            Reporter: Jonathan Hurley
            Priority: Critical


During a rolling upgrade from HDP 2.2 to HDP 2.3, Accumulo tracer fails to start because it is unable to find any tabletservers. The tabletserver were updated to HDP 2.3 earlier in the upgrade process and did come online briefly. 

The PID file still exist, but the tservers are definitely down:

{noformat}
[root@c6401 accumulo]# cat accumulo-accumulo-tserver.pid
6075
[root@c6401 accumulo]# ps -a | grep 6075
{noformat}

It seems like the problem might be located in the following piece of code:
{code}
    private void checkPermission(TCredentials credentials, String lock, final String request) throws ThriftSecurityException {
      boolean fatal = false;
      try {
        log.trace("Got " + request + " message from user: " + credentials.getPrincipal());
        if (!security.canPerformSystemActions(credentials)) {
          log.warn("Got " + request + " message from user: " + credentials.getPrincipal());
          throw new ThriftSecurityException(credentials.getPrincipal(), SecurityErrorCode.PERMISSION_DENIED);
        }
      } catch (ThriftSecurityException e) {
        log.warn("Got " + request + " message from unauthenticatable user: " + e.getUser());
        if (getCredentials().getToken().getClass().getName().equals(credentials.getTokenClassName())) {
          log.error("Got message from a service with a mismatched configuration. Please ensure a compatible configuration.", e);
          fatal = true;
        }
        throw e;
      } finally {
        if (fatal) {
          Halt.halt(1, new Runnable() {
            @Override
            public void run() {
              gcLogger.logGCInfo(TabletServer.this.getConfiguration());
            }
          });
        }
      }
{code}

Where a malformed principal causes a {{Halt}}.

>From the tserver logs:
{noformat}
2015-06-01 19:25:30,462 [rpc.TServerUtils] DEBUG: Instantiating default, unsecure custom half-async Thrift server
2015-06-01 19:25:30,468 [tserver.TabletServer] INFO : address = c6401.ambari.apache.org:9997
2015-06-01 19:25:30,510 [tserver.TabletServer] INFO : Waiting for tablet server lock
{noformat}

There is also no content in the *.out or *.err files for tserver.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)