ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Sen" <d...@hortonworks.com>
Subject Re: Review Request 39661: After upgrade Namenode fails to start when Kerberos is enabled using HDP 2.2.8.0
Date Mon, 26 Oct 2015 17:27:45 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39661/#review104045
-----------------------------------------------------------

Ship it!


Ship It!

- Dmytro Sen


On Окт. 26, 2015, 5:26 п.п., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39661/
> -----------------------------------------------------------
> 
> (Updated Окт. 26, 2015, 5:26 п.п.)
> 
> 
> Review request for Ambari and Dmytro Sen.
> 
> 
> Bugs: AMBARI-13566
>     https://issues.apache.org/jira/browse/AMBARI-13566
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> After upgrading Ambari 1.7.0 (or 2.0.x) to Ambari 2.1.3 and then either
> enabling Kerberos or Kerberos was previously enabled, NameNode fails to start
> with the following error:
> 
>     
>     
>     2015-10-16 13:46:04,499 ERROR namenode.NameNode (NameNode.java:main(1645)) - Failed
to start namenode.
>     java.io.IOException: Login failure for nn/localhost@EXAMPLE.COM from keytab /etc/security/keytabs/nn.service.keytab:
javax.security.auth.login.LoginException: Unable to obtain password from user
>     
>             at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
>             at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:243)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:637)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:656)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:896)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:880)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1574)
>             at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1640)
>     Caused by: javax.security.auth.login.LoginException: Unable to obtain password from
user
>     
>             at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
>             at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
>             at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>             at java.lang.reflect.Method.invoke(Method.java:606)
>             at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
>             at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
>             at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
>             at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
>             at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
>             at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
>             ... 7 more
>     2015-10-16 13:46:04,517 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting
with status 1
>     2015-10-16 13:46:04,532 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
>     
> 
> I am using HDP 2.2.8.0-3150
> 
>     
>     
>     # hdp-select status
>     accumulo-client - None
>     accumulo-gc - None
>     accumulo-master - None
>     accumulo-monitor - None
>     accumulo-tablet - None
>     accumulo-tracer - None
>     falcon-client - None
>     falcon-server - None
>     flume-server - None
>     hadoop-client - 2.2.8.0-3150
>     hadoop-hdfs-datanode - 2.2.8.0-3150
>     hadoop-hdfs-journalnode - 2.2.8.0-3150
>     hadoop-hdfs-namenode - 2.2.8.0-3150
>     hadoop-hdfs-nfs3 - 2.2.8.0-3150
>     hadoop-hdfs-portmap - 2.2.8.0-3150
>     hadoop-hdfs-secondarynamenode - 2.2.8.0-3150
>     hadoop-httpfs - 2.2.8.0-3150
>     hadoop-mapreduce-historyserver - 2.2.8.0-3150
>     hadoop-yarn-nodemanager - 2.2.8.0-3150
>     hadoop-yarn-resourcemanager - 2.2.8.0-3150
>     hadoop-yarn-timelineserver - 2.2.8.0-3150
>     hbase-client - None
>     hbase-master - None
>     hbase-regionserver - None
>     hive-metastore - None
>     hive-server2 - None
>     hive-webhcat - None
>     kafka-broker - None
>     knox-server - None
>     mahout-client - None
>     oozie-client - None
>     oozie-server - None
>     phoenix-client - None
>     ranger-admin - None
>     ranger-usersync - None
>     slider-client - None
>     spark-client - None
>     spark-historyserver - None
>     sqoop-client - None
>     sqoop-server - None
>     storm-client - None
>     storm-nimbus - None
>     storm-slider-client - None
>     storm-supervisor - None
>     zookeeper-client - 2.2.8.0-3150
>     zookeeper-server - 2.2.8.0-3150
>     
> 
> This issue does not exist when using 2.3.4.0-3011.
> 
> **Steps to reproduce #1**
> 
>   1. Install Ambari 2.0.2 and HDP 2.2 (HDP-2.2.8.0)
>   2. Upgrade Ambari to 2.1.3
>   3. Restart HDFS
>   4. Enable Kerberos
>   5. See Failure
> 
> **Steps to reproduce #2**
> 
>   1. Install Ambari 2.0.2 and HDP 2.2 (HDP-2.2.8.0)
>   2. Enable Kerberos
>   3. Upgrade Ambari to 2.1.3
>   4. Restart HDFS
>   5. See Failure
> 
> **Cause**  
> In `org.apache.ambari.server.upgrade.UpgradeCatalog210#updateHdfsConfigs`,
> `dfs.namenode.rpc-address` is set to be updated to the proper value. However
> the call to `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#updateCon
> figurationPropertiesForCluster(org.apache.ambari.server.state.Cluster,
> java.lang.String, java.util.Map<java.lang.String,java.lang.String>, boolean,
> boolean)` is made with the `updateIfExists` flag set to **false**. Before
> getting to this point, new configs have been added from the hdfs-site.xml file
> via `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#addNewConfigurati
> onsFromXml`. This added `dfs.namenode.rpc-address` to the hdfs-site config
> with the value of "localhost:8020" and thus the calculated (correct) value was
> ignored.
> 
> **Solution**   
> Change the call to `org.apache.ambari.server.upgrade.AbstractUpgradeCatalog#up
> dateConfigurationPropertiesForCluster(org.apache.ambari.server.state.Cluster,
> java.lang.String, java.util.Map<java.lang.String,java.lang.String>, boolean,
> boolean)` so that it is made with the `updateIfExists` flag set to **true**.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java a6ef1d0

>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
92cc9e6 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java
41754d9 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hdfs-site.xml
e53de9c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml
cf79e43 
> 
> Diff: https://reviews.apache.org/r/39661/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message