spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Error Invoking Spark on Yarn on using Spark Submit
Date Fri, 24 Jun 2016 11:35:09 GMT
Hi Punneet,

File does not exist:
hdfs://localhost:8020/user/opc/.sparkStaging/application_1466711725829_0033/pipeline-lib-0.1.0-SNAPSHOT.jar

indicates a YARN issue. It is trying to get that file from HDFS and copy it
across to /tmp directory.


   1. Check that the class is actually created at compile time vis sbn mvt
   or if the jar file exists
   2. check that the working directory in YARN have the correct permissions
   3.

In yarn-site.xml check the following parameter is set

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/tmp</value>
  </property>
</configuration>

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 24 June 2016 at 10:46, Jeff Zhang <zjffdu@gmail.com> wrote:

> You might have multiple java servlet jars on your classpath.
>
> On Fri, Jun 24, 2016 at 3:31 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> can you please check the yarn log files to see what they say (both the
>> nodemamager and resourcemanager)
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 24 June 2016 at 08:14, puneet kumar <puneetkumar.2705@gmail.com>
>> wrote:
>>
>>>
>>>
>>> I am getting below error thrown when I submit Spark Job using Spark
>>> Submit on Yarn. Need a quick help on what's going wrong here.
>>>
>>> 16/06/24 01:09:25 WARN AbstractLifeCycle: FAILED org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter-791eb5d5:
java.lang.IllegalStateException: class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
is not a javax.servlet.Filter
>>> java.lang.IllegalStateException: class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
is not a javax.servlet.Filter
>>> 	at org.spark-project.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
>>> 	at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
>>> 	at org.spark-project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:768)
>>> 	at org.spark-project.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:265)
>>> 	at org.spark-project.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717)
>>> 	at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
>>> 	at org.spark-project.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95)
>>> 	at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
>>> 	at org.spark-project.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229)
>>> 	at org.spark-project.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:172)
>>> 	at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
>>> 	at org.spark-project.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95)
>>> 	at org.spark-project.jetty.server.Server.doStart(Server.java:282)
>>> 	at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
>>> 	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)
>>> 	at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
>>> 	at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
>>> 	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1988)
>>> 	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
>>> 	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1979)
>>> 	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
>>> 	at org.apache.spark.ui.WebUI.bind(WebUI.scala:137)
>>> 	at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
>>> 	at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
>>> 	at scala.Option.foreach(Option.scala:236)
>>> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)
>>> 	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
>>>
>>>
>>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Mime
View raw message