hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Goldenberg <dgoldenberg...@gmail.com>
Subject Re: SaslException: No common protection layer between client and server ?
Date Fri, 10 Jun 2016 00:00:31 GMT
Thanks, Chris.

Basically, I've tried all 3 values for 'hadoop.rpc.authentication'. I was
getting the same error as before for 'authentication' and 'integrity'.  But
for 'privacy', I started getting a different error (quoted below).

The recommendation I've heard so far is this:

"The new error is cause your cluster has HA HDFS but you are passing
only a single
hostname for the NN which is currently not the active one. You will need to
use the entire HA config set, or for the moment pass in the other NN
hostname."

I'm looking at trying that out. I may need to also try the SASL debug
technique you pointed me at and hopefully I'll get some clientside debug
output out of it.  I don't have any serverside access, or any access to the
server configs nor the Cloudera Manager interface.


----------------------

Exception in thread "main"
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category READ is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyS
tate.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$Na
meNodeHAContext.checkOperation(NameNode.java:1775)
at org.apache.hadoop.hdfs.server.namenode.FSNamesyste
m.checkOperation(FSNamesystem.java:1402)
at org.apache.hadoop.hdfs.server.namenode.FSNamesyste
m.getFileInfo(FSNamesystem.java:4221)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpc
Server.getFileInfo(NameNodeRpcServer.java:881)
at org.apache.hadoop.hdfs.server.namenode.Authorizati
onProviderProxyClientProtocol.getFileInfo(Authoriz
ationProviderProxyClientProtocol.java:526)

On Thu, Jun 9, 2016 at 4:38 PM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> Hello Dmitry,
>
> For a successful SASL authentication, both the client and server need to
> negotiate and come to agreement on a quality of protection.  The Hadoop
> configuration property hadoop.rpc.protection supports a comma-separate list
> of values that map to the SASL QoP values.
>
> <property>
>   <name>hadoop.rpc.protection</name>
>   <value>authentication</value>
>   <description>A comma-separated list of protection values for secured
> sasl
>       connections. Possible values are authentication, integrity and
> privacy.
>       authentication means authentication only and no integrity or
> privacy;
>       integrity implies authentication and integrity are enabled; and
> privacy
>       implies all of authentication, integrity and privacy are enabled.
>       hadoop.security.saslproperties.resolver.class can be used to override
>       the hadoop.rpc.protection for a connection at the server side.
>   </description>
> </property>
>
> This means that for this application to work, both the client end and the
> server end must have at least one value in common in that list.  Judging
> from the stack trace, it looks like the client is trying to communicate
> with the HDFS NameNode, so I recommend that you review the setting for this
> configuration property at the NameNode and the application.  If you change
> the configuration property for the NameNode, then it will require a restart
> for the change to take effect.
>
> If this still isn't working, then it's possible that Java's SASL debugging
> might provide more clues.  Details on how to enable that logging are here:
>
>
> https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html#DEBUG
>
> --Chris Nauroth
>
> From: Dmitry Goldenberg <dgoldenberg123@gmail.com>
> Date: Wednesday, June 8, 2016 at 4:30 PM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: SaslException: No common protection layer between client and
> server ?
>
> I'm trying to communicate programmatically to a Hadoop cluster which is
> kerberized (CDH 5.3/HDFS 2.5.0).
>
> I have a valid Kerberos token on the client side.  But I'm getting an
> error as below, "No common protection layer between client and server".
>
> What does this error mean and are there any ways to fix or work around it?
>
> Is this something related to HDFS-5688
> <https://issues.apache.org/jira/browse/HDFS-5688>? The ticket seems to
> imply that the property "hadoop.rpc.protection" must be set, presumably to "
> authentication
> <https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>"
> (also per e.g. this
> <https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>
> ).
>
> Would this need to be set on all servers in the cluster and then the
> cluster bounced?  I don't have easy access to the cluster so I need to
> understand whether 'hadoop.rpc.protection' is the actual cause. It seems
> that 'authentication' should be the value used by default, at least
> according to the core-default.xml documentation
> <https://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/core-default.xml>
> .
>
> java.io.IOException: Failed on local exception: java.io.IOException:
> Couldn't setup connection for principal1/server1.acme.net@xxx.acme.net
> <bdgtr001a01i1d.nam.nsroot.net@NAMUXDEV.DYN.NSROOT.NET> to
> server2.acme.net/10.XX.XXX.XXX:8020
> <http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>; Host Details
> : local host is: “some-host.acme.net/168.XX.XXX.XX
> <http://sd-e13b-c041.nam.nsroot.net/168.72.192.72>”; destination host is:
> “server2.acm <http://bdgtmaster02i1d.nam.nsroot.net/>e.net”:8020;
>
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1415)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>
>         at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:498)
>
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>
>         at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
>
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707)
>
>         at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785)
>
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068)
>
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>
>         at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
>
>         ... 11 more
>
> Caused by: java.io.IOException: Couldn't setup connection for principal1/
> server1.acme.net@xxx.acme.net
> <bdgtr001a01i1d.nam.nsroot.net@NAMUXDEV.DYN.NSROOT.NET> to
> server2.acme.net/10.XX.XXX.XXX:8020
> <http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>;
>
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
>
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1382)
>
>         ... 31 more
>
> Caused by: javax.security.sasl.SaslException: No common protection layer
> between client and server
>
>         at
> com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)
>
>         at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)
>
>         at
> org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)
>
>         at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
>
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
>
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>
>         at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
>
>         ... 34 more
>

Mime
View raw message