hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emma Lin <l...@vmware.com>
Subject RE: Issues during setting up hadoop security cluster
Date Fri, 20 Jan 2012 10:33:23 GMT
After remove the upper-case, the problem disappeared. Now I get node manager connected to resource
manager successfully.
Thank you Vinod.

But now, I get another issue to connect Name Node from Data Node. The log in Name Node is
as following:
2012-01-20 18:17:02,127 WARN  ipc.Server (Server.java:saslReadAndProcess(1070)) - Auth failed
for 10.112.127.14:60456:null
2012-01-20 18:17:02,128 INFO  ipc.Server (Server.java:doRead(572)) - IPC Server listener on
9000: readAndProcess threw exception javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption
type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)] from client 10.112.127.14.
Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified
at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not
supported/enabled)]
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
        at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1054)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1232)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:567)
        at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:366)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:341)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption
type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
        ... 5 more
Caused by: KrbException: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled
        at sun.security.krb5.EncryptionKey.findKey(EncryptionKey.java:481)
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:260)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
        ... 8 more

From the internet, someone said that it's because the Java support AES 128 by default. And
to support AES 256, we need to install unlimited JCE policy. But after install the JCE, node
manager can connect to resource manager, the data node still cannot connect to name node.
As the datanode is started through jsvc, I don't know if the java setting does not work after
executed through jsvc. But anyway, it still complain for the AES 256 is not supported. 

Any ideas?
Thanks
Emma


-----Original Message-----
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] 
Sent: 2012年1月20日 13:23
To: common-user@hadoop.apache.org
Subject: Re: Issues during setting up hadoop security cluster

Hi,

Just today evening, I happened to run into someone who had the same
issue. After some debugging, I cornered that to the hostnames having
upper-case characters. Somehow, when DataNode or NodeManager try to
get a service ticket for their corresponding services (NameNode and
ResourceManager respectively), the hostname were getting converted
into all lowercase. You can see if it is the same situation with you
by looking at krb5kdc logs.

If that is the case, changing the hostnames everywhere to be all
small-case may help. Please try that and let me know.

HTH,
+Vinod


On Thu, Jan 19, 2012 at 8:52 PM, Emma Lin <line@vmware.com> wrote:
> Gurus,
>
> I’m setting up a security cluster of hadoop .23. But now, the communication
> between Data Node and Name Node, Node Manager and Resource Manager have
> problem.
>
> When I start the Node Manager, it will report following error, and then
> shutdown itself. Did you ever see such issue? Do you have any idea on how to
> triage this issue?
>
>
>
> 2012-01-20 12:03:08,258 INFO  ipc.HadoopYarnRPC
> (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.server.api.ResourceTracker
>
> 2012-01-20 12:03:08,291 INFO  nodemanager.NodeStatusUpdaterImpl
> (NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to
> ResourceManager at hadoopRM.example.aurora:9003
>
> 2012-01-20 12:03:20,399 WARN  ipc.Client (Client.java:run(526)) - Couldn't
> setup connection for nm/hadoopNM.example.aurora@EXAMPLE.AURORA to
> rm/hadoopRM.example.aurora@EXAMPLE.AURORA
>
> 2012-01-20 12:03:20,405 ERROR service.CompositeService
> (CompositeService.java:start(72)) - Error starting services
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>
> org.apache.avro.AvroRuntimeException:
> java.lang.reflect.UndeclaredThrowableException
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
>
>         at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
>
> Caused by: java.lang.reflect.UndeclaredThrowableException
>
>         at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:161)
>
>         at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
>
>         ... 3 more
>
> Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed
> on local exception: java.io.IOException: Couldn't setup connection for
> nm/hadoopNM.example.aurora@EXAMPLE.AURORA to
> rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is:
> "hadoopNM/10.112.127.102"; destination host is:
> ""hadoopRM.example.aurora":9003;
>
>         at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
>
>         at $Proxy14.registerNodeManager(Unknown Source)
>
>         at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
>
>         ... 5 more
>
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: Couldn't setup connection for
> nm/hadoopNM.example.aurora@EXAMPLE.AURORA to
> rm/hadoopRM.example.aurora@EXAMPLE.AURORA; Host Details : local host is:
> "hadoopNM/10.112.127.102"; destination host is:
> ""hadoopRM.example.aurora":9003;
>
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:655)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1089)
>
>         at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
>
>         ... 7 more
>
> Caused by: java.io.IOException: Couldn't setup connection for
> nm/hadoopNM.example.aurora@EXAMPLE.AURORA to
> rm/hadoopRM.example.aurora@EXAMPLE.AURORA
>
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:527)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:499)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:205)
>
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1195)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1065)
>
>         ... 8 more
>
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)]
>
>         at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
>
>         at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:205)
>
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
>
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:573)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:572)
>
>         ... 11 more
>
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Server not found in Kerberos database (7) - UNKNOWN_SERVER)
>
>         at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
>
>         at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
>
>         at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
>
>         at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
>
>         ... 20 more
>
> Caused by: KrbException: Server not found in Kerberos database (7) -
> UNKNOWN_SERVER
>
>         at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
>
>         at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
>
>         at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
>
>         at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
>
>         at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
>
>         at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
>
>         ... 23 more
>
> Caused by: KrbException: Identifier doesn't match expected value (906)
>
>         at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
>
>         at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
>
>         at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
>
>         at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
>
>         ... 28 more
>
>
>
> The error said that no valid server credential, but I’ve add those
> credentials in Resource Manager node. The keytab result is as following:
>
> line@hadoopRM:~$ klist -k -e -t /etc/krb5.keytab
>
> Keytab name: WRFILE:/etc/krb5.keytab
>
> KVNO Timestamp         Principal
>
> ---- -----------------
> --------------------------------------------------------
>
>    2 01/20/12 10:55:02 rm/hadoopRM.example.aurora@EXAMPLE.AURORA
> (aes256-cts-hmac-sha1-96)
>
>    2 01/20/12 10:55:02 rm/hadoopRM.example.aurora@EXAMPLE.AURORA
> (arcfour-hmac)
>
>    2 01/20/12 10:55:02 rm/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des3-cbc-sha1)
>
>    2 01/20/12 10:55:02 rm/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des-cbc-crc)
>
>    2 01/19/12 11:19:11 host/hadoopRM.example.aurora@EXAMPLE.AURORA
> (aes256-cts-hmac-sha1-96)
>
>    2 01/19/12 11:19:11 host/hadoopRM.example.aurora@EXAMPLE.AURORA
> (arcfour-hmac)
>
>    2 01/19/12 11:19:11 host/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des3-cbc-sha1)
>
>    2 01/19/12 11:19:11 host/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des-cbc-crc)
>
>    2 01/19/12 11:20:15 jhs/hadoopRM.example.aurora@EXAMPLE.AURORA
> (aes256-cts-hmac-sha1-96)
>
>    2 01/19/12 11:20:15 jhs/hadoopRM.example.aurora@EXAMPLE.AURORA
> (arcfour-hmac)
>
>    2 01/19/12 11:20:15 jhs/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des3-cbc-sha1)
>
>    2 01/19/12 11:20:15 jhs/hadoopRM.example.aurora@EXAMPLE.AURORA
> (des-cbc-crc)
>
>
>
> The whole node manager log is attached.
>
>
>
> Any idea is appreciated.
>
> Thanks
>
> Emma
Mime
View raw message