spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Re: Exception adding resource files in latest Spark
Date Fri, 05 Dec 2014 05:51:51 GMT
I created a ticket for this:

  https://issues.apache.org/jira/browse/SPARK-4757


Jianshi

On Fri, Dec 5, 2014 at 1:31 PM, Jianshi Huang <jianshi.huang@gmail.com>
wrote:

> Correction:
>
> According to Liancheng, this hotfix might be the root cause:
>
>
> https://github.com/apache/spark/commit/38cb2c3a36a5c9ead4494cbc3dde008c2f0698ce
>
> Jianshi
>
> On Fri, Dec 5, 2014 at 12:45 PM, Jianshi Huang <jianshi.huang@gmail.com>
> wrote:
>
>> Looks like the datanucleus*.jar shouldn't appear in the hdfs path in
>> Yarn-client mode.
>>
>> Maybe this patch broke yarn-client.
>>
>>
>> https://github.com/apache/spark/commit/a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53
>>
>> Jianshi
>>
>> On Fri, Dec 5, 2014 at 12:02 PM, Jianshi Huang <jianshi.huang@gmail.com>
>> wrote:
>>
>>> Actually my HADOOP_CLASSPATH has already been set to include
>>> /etc/hadoop/conf/*
>>>
>>> export
>>> HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml:/usr/lib/hbase/lib/hbase-protocol.jar:$(hbase
>>> classpath)
>>>
>>> Jianshi
>>>
>>> On Fri, Dec 5, 2014 at 11:54 AM, Jianshi Huang <jianshi.huang@gmail.com>
>>> wrote:
>>>
>>>> Looks like somehow Spark failed to find the core-site.xml in
>>>> /et/hadoop/conf
>>>>
>>>> I've already set the following env variables:
>>>>
>>>> export YARN_CONF_DIR=/etc/hadoop/conf
>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>> export HBASE_CONF_DIR=/etc/hbase/conf
>>>>
>>>> Should I put $HADOOP_CONF_DIR/* to HADOOP_CLASSPATH?
>>>>
>>>> Jianshi
>>>>
>>>> On Fri, Dec 5, 2014 at 11:37 AM, Jianshi Huang <jianshi.huang@gmail.com
>>>> > wrote:
>>>>
>>>>> I got the following error during Spark startup (Yarn-client mode):
>>>>>
>>>>> 14/12/04 19:33:58 INFO Client: Uploading resource
>>>>> file:/x/home/jianshuang/spark/spark-latest/lib/datanucleus-api-jdo-3.2.6.jar
>>>>> ->
>>>>> hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar
>>>>> java.lang.IllegalArgumentException: Wrong FS:
>>>>> hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar,
>>>>> expected: file:///
>>>>>         at
>>>>> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
>>>>>         at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:79)
>>>>>         at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:506)
>>>>>         at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
>>>>>         at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
>>>>>         at
>>>>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.ClientDistributedCacheManager.addResource(ClientDistributedCacheManager.scala:67)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:257)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:242)
>>>>>         at scala.Option.foreach(Option.scala:236)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.ClientBase$class.prepareLocalResources(ClientBase.scala:242)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:35)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.ClientBase$class.createContainerLaunchContext(ClientBase.scala:350)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:35)
>>>>>         at
>>>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:80)
>>>>>         at
>>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
>>>>>         at
>>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140)
>>>>>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:335)
>>>>>         at
>>>>> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:986)
>>>>>         at $iwC$$iwC.<init>(<console>:9)
>>>>>         at $iwC.<init>(<console>:18)
>>>>>         at <init>(<console>:20)
>>>>>         at .<init>(<console>:24)
>>>>>
>>>>> I'm using latest Spark built from master HEAD yesterday. Is this a bug?
>>>>>
>>>>> --
>>>>> Jianshi Huang
>>>>>
>>>>> LinkedIn: jianshi
>>>>> Twitter: @jshuang
>>>>> Github & Blog: http://huangjs.github.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jianshi Huang
>>>>
>>>> LinkedIn: jianshi
>>>> Twitter: @jshuang
>>>> Github & Blog: http://huangjs.github.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Jianshi Huang
>>>
>>> LinkedIn: jianshi
>>> Twitter: @jshuang
>>> Github & Blog: http://huangjs.github.com/
>>>
>>
>>
>>
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>



-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message