hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: SaslException: No common protection layer between client and server ?
Date Thu, 09 Jun 2016 20:38:29 GMT
Hello Dmitry,

For a successful SASL authentication, both the client and server need to negotiate and come
to agreement on a quality of protection.  The Hadoop configuration property hadoop.rpc.protection
supports a comma-separate list of values that map to the SASL QoP values.

<property>
  <name>hadoop.rpc.protection</name>
  <value>authentication</value>
  <description>A comma-separated list of protection values for secured sasl
      connections. Possible values are authentication, integrity and privacy.
      authentication means authentication only and no integrity or privacy;
      integrity implies authentication and integrity are enabled; and privacy
      implies all of authentication, integrity and privacy are enabled.
      hadoop.security.saslproperties.resolver.class can be used to override
      the hadoop.rpc.protection for a connection at the server side.
  </description>
</property>

This means that for this application to work, both the client end and the server end must
have at least one value in common in that list.  Judging from the stack trace, it looks like
the client is trying to communicate with the HDFS NameNode, so I recommend that you review
the setting for this configuration property at the NameNode and the application.  If you change
the configuration property for the NameNode, then it will require a restart for the change
to take effect.

If this still isn't working, then it's possible that Java's SASL debugging might provide more
clues.  Details on how to enable that logging are here:

https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html#DEBUG

--Chris Nauroth

From: Dmitry Goldenberg <dgoldenberg123@gmail.com<mailto:dgoldenberg123@gmail.com>>
Date: Wednesday, June 8, 2016 at 4:30 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: SaslException: No common protection layer between client and server ?

I'm trying to communicate programmatically to a Hadoop cluster which is kerberized (CDH 5.3/HDFS
2.5.0).

I have a valid Kerberos token on the client side.  But I'm getting an error as below, "No
common protection layer between client and server".

What does this error mean and are there any ways to fix or work around it?

Is this something related to HDFS-5688<https://issues.apache.org/jira/browse/HDFS-5688>?
The ticket seems to imply that the property "hadoop.rpc.protection" must be set, presumably
to "authentication<https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>"
(also per e.g. this<https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>).

Would this need to be set on all servers in the cluster and then the cluster bounced?  I don't
have easy access to the cluster so I need to understand whether 'hadoop.rpc.protection' is
the actual cause. It seems that 'authentication' should be the value used by default, at least
according to the core-default.xml documentation<https://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/core-default.xml>.


java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection
for principal1/server1.acme.net@xxx.acme.net<mailto:bdgtr001a01i1d.nam.nsroot.net@NAMUXDEV.DYN.NSROOT.NET>
to server2.acme.net/10.XX.XXX.XXX:8020<http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>;
Host Details : local host is: “some-host.acme.net/168.XX.XXX.XX<http://sd-e13b-c041.nam.nsroot.net/168.72.192.72>”;
destination host is: “server2.acm<http://bdgtmaster02i1d.nam.nsroot.net/>e.net<http://e.net>”:8020;

        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)

        at org.apache.hadoop.ipc.Client.call(Client.java:1415)

        at org.apache.hadoop.ipc.Client.call(Client.java:1364)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707)

        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785)

        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068)

        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)

        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)

        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)

        ... 11 more

Caused by: java.io.IOException: Couldn't setup connection for principal1/server1.acme.net@xxx.acme.net<mailto:bdgtr001a01i1d.nam.nsroot.net@NAMUXDEV.DYN.NSROOT.NET>
to server2.acme.net/10.XX.XXX.XXX:8020<http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>;

        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)

        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)

        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)

        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)

        at org.apache.hadoop.ipc.Client.call(Client.java:1382)

        ... 31 more

Caused by: javax.security.sasl.SaslException: No common protection layer between client and
server

        at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)

        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)

        at org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)

        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427)

        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)

        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)

        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)

        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)

        ... 34 more

Mime
View raw message