hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ac@hsk.hk" ...@hsk.hk>
Subject Re: Failed To Start SecondaryNameNode in Secure Mode
Date Wed, 28 Nov 2012 15:03:51 GMT
Hi,

I have 'dfs.secondary.namenode.kerberos.internal.spnego.principal' in hdfs-site.xml

I used the following commands to add this principal:

1) kadmin: addprinc -randkey HTTP/m146
2) kadmin:  ktadd -k /etc/hadoop/hadoop.keytab -norandkey HTTP/m146
		kadmin: Principal -norandkey does not exist.
		Entry for principal ......
		Entry for principal ......
3) klist -e -k -t /etc/hadoop/hadoop.keytab
		4 28/11/2012 22:20 HTTP/m146@...... (aes256-cts-hmac-sha1-96) 
   		4 28/11/2012 22:20 HTTP/m146@...... (arcfour-hmac) 
   		4 28/11/2012 22:20 HTTP/m146@...... (des3-cbc-sha1) 
   		4 28/11/2012 22:20 HTTP/m146@...... (des-cbc-crc) 

4) try to start the SNN (same namenode server)  {$HADOOP_HOME}/bin/hadoop-daemon.sh start
secondarynamenode  && jps

	Warning: $HADOOP_HOME is deprecated.
	starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
	Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
	Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication

	at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	3315 QuorumPeerMain
	14942 NameNode
	15611 Jps
	15560 SecondaryNameNode


5) wait a few seconds, try JPS

	$ jps
	15625 Jps
	3315 QuorumPeerMain
	14942 NameNode

        The SNN gone

6) check the log:

	2012-11-28 22:27:26,035 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:

	/************************************************************
	STARTUP_MSG: Starting SecondaryNameNode
	STARTUP_MSG:   host = m146.......
	STARTUP_MSG:   args = []
	STARTUP_MSG:   version = 1.0.4
	STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0
-r 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
	************************************************************/
	2012-11-28 22:27:26,552 INFO org.apache.hadoop.security.UserGroupInformation: Login successful
for user ......
	2012-11-28 22:27:26,686 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:

	/************************************************************
	SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
	************************************************************/


version: hadoop-1.0.4
distributed Cluster: True







On the other hand, 

7) I added the following to hsfd-site.xml

<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>host/_HOST@......</value>
</property>

8) Tried to start the SNN (same namenode server)  {$HADOOP_HOME}/bin/hadoop-daemon.sh start
secondarynamenode  && jps

	Warning: $HADOOP_HOME is deprecated.
	starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out

	15860 SecondaryNameNode
	3315 QuorumPeerMain
	15909 Jps
	14942 NameNode

        i.e. the error message in 4) gone


9) Wait a few seconds, check JPS

	3315 QuorumPeerMain
	15925 Jps
	14942 NameNode

	i.e.  the 15860 SecondaryNameNode also gone


10) check the log again

	************************************************************/
	2012-11-28 22:43:00,695 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:

	/************************************************************
	STARTUP_MSG: Starting SecondaryNameNode
	STARTUP_MSG:   host = m146......
	STARTUP_MSG:   args = []
	STARTUP_MSG:   version = 1.0.4
	STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0
-r 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
	************************************************************/
	2012-11-28 22:43:01,206 INFO org.apache.hadoop.security.UserGroupInformation: Login successful
for user ......
	2012-11-28 22:43:01,447 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting
web server as: host/m146......
	2012-11-28 22:43:01,480 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
	2012-11-28 22:43:01,531 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
	2012-11-28 22:43:01,536 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:

	/************************************************************
	SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146........
	************************************************************/




Please help!
Thanks
ac





On 28 Nov 2012, at 12:57 AM, Arpit Gupta wrote:

> Hi AC,
> 
> Do you have the following property defined in your hdfs-site.xml
> 
> <property>
> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
> <value>HTTP/_HOST@REALM</value>
> </property>
> 
> and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the
logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
> 
> 
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
> 
> On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac@hsk.hk> wrote:
> 
>> Hi,
>> 
>> Please help!
>> 
>> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh
start secondarynamenode
>> 
>> 1) from the log, I saw "Login successful" 
>> 	************************************************************/
>> 	2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login
successful for user ......
>> 	2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
SHUTDOWN_MSG:
>> 	/************************************************************
>> 	SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
>> 	************************************************************/
>> 
>> 
>> 2) However, from the command line, I saw 
>> 	$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
>> 	Warning: $HADOOP_HOME is deprecated.
>> 	starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
>> 	Exception in thread "main" java.io.IOException: Login failure for null from keytab
/etc/hadoop/hadoop.keytab
>> 		at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
>> 		at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
>> 		at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> 		at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>> 	Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name
for authentication 
>> 		at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
>> 		at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
>> 		at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
>> 		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 
>> 
>> There is no secondarynamenode process if I use JPS to check 
>> 
>> QUESTION: Any idea where I am wrong?
>> 
>> 
>> Thanks
>> ac
>> 
>> 
>> 
> 


Mime
View raw message