ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMBARI-13566) After upgrade Namenode fails to start when Kerberos is enabled using HDP 2.2.8.0
Date Mon, 26 Oct 2015 17:26:27 GMT
Andrew Onischuk created AMBARI-13566:
----------------------------------------

             Summary: After upgrade Namenode fails to start when Kerberos is enabled using
HDP 2.2.8.0
                 Key: AMBARI-13566
                 URL: https://issues.apache.org/jira/browse/AMBARI-13566
             Project: Ambari
          Issue Type: Bug
            Reporter: Andrew Onischuk
            Assignee: Andrew Onischuk
             Fix For: 2.1.3


After upgrading Ambari 1.7.0 (or 2.0.x) to Ambari 2.1.3 and then either
enabling Kerberos or Kerberos was previously enabled, NameNode fails to start
with the following error:

    
    
    2015-10-16 13:46:04,499 ERROR namenode.NameNode (NameNode.java:main(1645)) - Failed to
start namenode.
    java.io.IOException: Login failure for nn/localhost@EXAMPLE.COM from keytab /etc/security/keytabs/nn.service.keytab:
javax.security.auth.login.LoginException: Unable to obtain password from user
    
            at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
            at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:243)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:637)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:656)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:896)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:880)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1574)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1640)
    Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    
            at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
            at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
            at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
            at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
            at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
            at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
            at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
            at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
            ... 7 more
    2015-10-16 13:46:04,517 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with
status 1
    2015-10-16 13:46:04,532 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
    

I am using HDP 2.2.8.0-3150

    
    
    # hdp-select status
    accumulo-client - None
    accumulo-gc - None
    accumulo-master - None
    accumulo-monitor - None
    accumulo-tablet - None
    accumulo-tracer - None
    falcon-client - None
    falcon-server - None
    flume-server - None
    hadoop-client - 2.2.8.0-3150
    hadoop-hdfs-datanode - 2.2.8.0-3150
    hadoop-hdfs-journalnode - 2.2.8.0-3150
    hadoop-hdfs-namenode - 2.2.8.0-3150
    hadoop-hdfs-nfs3 - 2.2.8.0-3150
    hadoop-hdfs-portmap - 2.2.8.0-3150
    hadoop-hdfs-secondarynamenode - 2.2.8.0-3150
    hadoop-httpfs - 2.2.8.0-3150
    hadoop-mapreduce-historyserver - 2.2.8.0-3150
    hadoop-yarn-nodemanager - 2.2.8.0-3150
    hadoop-yarn-resourcemanager - 2.2.8.0-3150
    hadoop-yarn-timelineserver - 2.2.8.0-3150
    hbase-client - None
    hbase-master - None
    hbase-regionserver - None
    hive-metastore - None
    hive-server2 - None
    hive-webhcat - None
    kafka-broker - None
    knox-server - None
    mahout-client - None
    oozie-client - None
    oozie-server - None
    phoenix-client - None
    ranger-admin - None
    ranger-usersync - None
    slider-client - None
    spark-client - None
    spark-historyserver - None
    sqoop-client - None
    sqoop-server - None
    storm-client - None
    storm-nimbus - None
    storm-slider-client - None
    storm-supervisor - None
    zookeeper-client - 2.2.8.0-3150
    zookeeper-server - 2.2.8.0-3150
    

This issue does not exist when using 2.3.4.0-3011.

**Steps to reproduce #1**

  1. Install Ambari 2.0.2 and HDP 2.2 (HDP-2.2.8.0)
  2. Upgrade Ambari to 2.1.3
  3. Restart HDFS
  4. Enable Kerberos
  5. See Failure

**Steps to reproduce #2**

  1. Install Ambari 2.0.2 and HDP 2.2 (HDP-2.2.8.0)
  2. Enable Kerberos
  3. Upgrade Ambari to 2.1.3
  4. Restart HDFS
  5. See Failure

**Cause**  
In `org.apache.ambari.server.upgrade.UpgradeCatalog210#updateHdfsConfigs`,
`dfs.namenode.rpc-address` is set to be updated to the proper value. However
the call to `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#updateCon
figurationPropertiesForCluster(org.apache.ambari.server.state.Cluster,
java.lang.String, java.util.Map<java.lang.String,java.lang.String>, boolean,
boolean)` is made with the `updateIfExists` flag set to **false**. Before
getting to this point, new configs have been added from the hdfs-site.xml file
via `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#addNewConfigurati
onsFromXml`. This added `dfs.namenode.rpc-address` to the hdfs-site config
with the value of "localhost:8020" and thus the calculated (correct) value was
ignored.

**Solution**   
Change the call to `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#up
dateConfigurationPropertiesForCluster(org.apache.ambari.server.state.Cluster,
java.lang.String, java.util.Map<java.lang.String,java.lang.String>, boolean,
boolean)` so that it is made with the `updateIfExists` flag set to **true**.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message