spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Udit Mehrotra (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle
Date Fri, 21 Jul 2017 00:18:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Udit Mehrotra updated SPARK-21494:
----------------------------------
    Description: 
Spark’s new AES based authentication mechanism does not seem to work when configured with
external shuffle service on YARN. 

Here is the stack trace for the error we see in the driver logs:
ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal: Unable to create
executor due to Unable to register with external shuffle server due to: java.lang.IllegalArgumentException:
Authentication failed.
                at org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
                at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
                at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
                at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 
Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’:
spark.network.crypto.enabled true
spark.network.crypto.saslFallback false
spark.authenticate               true
 
Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both Spark and YARN
side is not giving much information either about why authentication fails. The driver and
node manager logs have been attached to the JIRA.

  was:
Spark’s new AES based authentication mechanism does not seem to work when configured with
external shuffle service on YARN. Here is the stack trace for the error we see in the driver
logs:
ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal: Unable to create
executor due to Unable to register with external shuffle server due to: java.lang.IllegalArgumentException:
Authentication failed.
                at org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
                at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
                at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
                at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 
Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’:
spark.network.crypto.enabled true
spark.network.crypto.saslFallback false
spark.authenticate               true
 
Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both Spark and YARN
side is not giving much information either about why authentication fails. The driver and
node manager logs have been attached to the JIRA.


> Spark 2.2.0 AES encryption not working with External shuffle
> ------------------------------------------------------------
>
>                 Key: SPARK-21494
>                 URL: https://issues.apache.org/jira/browse/SPARK-21494
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager, Shuffle
>    Affects Versions: 2.2.0
>         Environment: AWS EMR
>            Reporter: Udit Mehrotra
>
> Spark’s new AES based authentication mechanism does not seem to work when configured
with external shuffle service on YARN. 
> Here is the stack trace for the error we see in the driver logs:
> ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal: Unable to create
executor due to Unable to register with external shuffle server due to: java.lang.IllegalArgumentException:
Authentication failed.
>                 at org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
>                 at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
>                 at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
>                 at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
>                 at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
>                 at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
>                 at org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
>  
> Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’:
> spark.network.crypto.enabled true
> spark.network.crypto.saslFallback false
> spark.authenticate               true
>  
> Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both Spark and
YARN side is not giving much information either about why authentication fails. The driver
and node manager logs have been attached to the JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message