hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3980) NPE in HttpURLConnection.java while starting SecondaryNameNode.
Date Thu, 18 Oct 2012 05:22:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478684#comment-13478684
] 

Brahma Reddy Battula commented on HDFS-3980:
--------------------------------------------

[~atm] 
Thanks a lot for reply..

{quote}
can you tell me exactly what you did to generate the keytab in question? It's unfortunately
quite easy to accidentally invalidate a keytab for a given principal if you later export another
keytab including entries for the same principal
{quote}
I have generated keytab using following

xst -norandkey -k /etc/hadoop/hdfs.keytab hdfs/(hostname of machine)@HADOOP.COM
xst -norandkey -k /etc/hadoop/hdfs.keytab HTTP/(hostname of machine)@HADOOP.COM
and these two only I had configured..Please check following link for same..

https://issues.apache.org/jira/browse/HDFS-4043?focusedCommentId=13478670&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13478670
{quote}
 I can't find the method KerberosUtil#getWebDfsPrincipalAndReset in any version of Hadoop
that I'm aware of. What exact version are you experiencing this with?
{quote}

Internally we had added this one.I removed that(getWebDfsPrincipalAndReset) one executed even
then checkpoint is failing by throwing followig exception..


{noformat}
2012-10-18 10:04:55,907 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening
connection to http://10.***.177:50070/getimage?getimage=1&txid=280&storageInfo=-40:85372811:0:CID-86a868d5-df3a-4a3c-b068-cc9a3bafec9b
2012-10-18 10:05:21,943 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hdfs/linux-177@HADOOP.COM (auth:KERBEROS) cause:java.io.IOException: Exception trying to
open authenticated connection to http://10.***.177:50070/getimage?getimage=1&txid=280&storageInfo=-40:85372811:0:CID-86a868d5-df3a-4a3c-b068-cc9a3bafec9b
2012-10-18 10:05:21,944 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception
in doCheckpoint
java.io.IOException: Exception trying to open authenticated connection to http://10.***.177:50070/getimage?getimage=1&txid=280&storageInfo=-40:85372811:0:CID-86a868d5-df3a-4a3c-b068-cc9a3bafec9b
	at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:510)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:229)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:222)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:86)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:399)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:385)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:384)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:477)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:343)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:310)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1347)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:450)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:306)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos database (7)
- UNKNOWN_SERVER)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:273)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:169)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
	at org.apache.hadoop.security.SecurityUtil.openSecureHttpConnection(SecurityUtil.java:508)
	... 18 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7) - UNKNOWN_SERVER)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:252)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:228)
	... 21 more
Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
	at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
	at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
	... 28 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
	at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
	at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
	at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
	... 33 more
{noformat}

Please correct me If I am wrong..


                
> NPE in HttpURLConnection.java while starting SecondaryNameNode.
> ---------------------------------------------------------------
>
>                 Key: HDFS-3980
>                 URL: https://issues.apache.org/jira/browse/HDFS-3980
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Brahma Reddy Battula
>            Priority: Critical
>         Attachments: core-site.xml, hdfs-site.xml
>
>
> Scenario:
> ========
> I started secure cluster by going thru following..
> https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide..
> Here SecondaryNamenode is getting shutdown by throwing NPE..
> Please correct me If I am wrong...
> Will attach conf and logs..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message