spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6436) io/netty missing from external shuffle service jars for yarn
Date Fri, 20 Mar 2015 17:20:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371663#comment-14371663
] 

Marcelo Vanzin commented on SPARK-6436:
---------------------------------------

Hmm, weird. I have the assembly built both against hadoop-2.5 and against CDH, with my patch
from SPARK-6070, and both contain that class. Which version of Hadoop are you compiling against?
Can you try "jar tf" on the different jars under network/yarn/target?

> io/netty missing from external shuffle service jars for yarn
> ------------------------------------------------------------
>
>                 Key: SPARK-6436
>                 URL: https://issues.apache.org/jira/browse/SPARK-6436
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, YARN
>    Affects Versions: 1.3.0
>            Reporter: Thomas Graves
>
> I was trying to use the external shuffle service on yarn but it appears that io/netty
isn't included in the network jars.  I loaded up network-common, network-yarn, and network-shuffle.
 If there is some other jar supposed to be included please let me know.
> 2015-03-20 14:25:07,142 [main] FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager:
Error starting NodeManager
> java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
>         at org.apache.spark.network.shuffle.ExternalShuffleBlockManager.<init>(ExternalShuffleBlockManager.java:64)
>         at org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:53)
>         at org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:105)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message