Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 59FF91748A for ; Tue, 3 Feb 2015 14:32:34 +0000 (UTC) Received: (qmail 61192 invoked by uid 500); 3 Feb 2015 14:32:35 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 61132 invoked by uid 500); 3 Feb 2015 14:32:34 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 60987 invoked by uid 99); 3 Feb 2015 14:32:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2015 14:32:34 +0000 Date: Tue, 3 Feb 2015 14:32:34 +0000 (UTC) From: "Kihwal Lee (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7731) Can not start HA namenode with security enabled MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303339#comment-14303339 ] Kihwal Lee commented on HDFS-7731: ---------------------------------- Check and compare kvno of QJM's SPN from a client and the key version in qjm node keytabs. Unlike plain Kerberos, SPNEGO forces the latest key to be used. E.g. if a SPNEGO client is getting a service ticket based on the latest key version, but the server's keytab does not contain the matching latest key, it won't work. In the future please use a mailing list. JIRA is for reporting a bug. > Can not start HA namenode with security enabled > ----------------------------------------------- > > Key: HDFS-7731 > URL: https://issues.apache.org/jira/browse/HDFS-7731 > Project: Hadoop HDFS > Issue Type: Task > Components: ha, journal-node, namenode, security > Affects Versions: 2.5.2 > Environment: Redhat6.2 Hadoop2.5.2 > Reporter: donhoff_h > Labels: hadoop, security > > I am converting a secure non-HA cluster into a secure HA cluster. After the configuration and started all the journalnodes, I executed the following commands on the original NameNode: > 1. hdfs name -initializeSharedEdits #this step succeeded > 2. hadoop-daemon.sh start namenode # this step failed. > So the namenode can not be started. I verified that my principals are right. And if I change back to the secure non-HA mode, the namenode can be started. > The namenode log just reported the following errors and I could not find the reason according to this log: > 2015-02-03 17:42:06,020 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3 > 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3' to transaction ID 68994 > 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3' to transaction ID 68994 > 2015-02-03 17:42:06,154 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception initializing http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3 > java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:444) > at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:438) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) > at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) > at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) > at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) > at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) > at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) > at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:184) > at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:137) > at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:816) > at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676) > at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700) > at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529) > at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:751) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:735) > at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473) -- This message was sent by Atlassian JIRA (v6.3.4#6332)