hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd <bit1...@163.com>
Subject Re:Re: Unable to start container using hive on spark
Date Tue, 12 Jan 2016 02:54:24 GMT
Hi Sofia,
Thank you for the detailed guide. 
I confirmed that there are compatibility issues between Hive 1.2.1 and Spark 1.5.X. Hive 1.2.1
is by default built against spark 1.3.1 and compatible with spark 1.4.1,but doesn't work with
1.5.X.


Basically,I am using somewhat the same configuration as yours,but I am using yarn-client as
the spark.master.
Did you try to use yarn-client as the spark.master?


I am building spark1.3.1 without hive and hive-thriftserver. I will get back to you when I
do the test,


Thank you, Sofia. 





At 2016-01-11 23:56:32, "Sofia" <sofia.panagiotidi@taiger.com> wrote:

From what I have seen you need to use a Spark version that does not contain the Hive jars.
(maybe build one as described at the page).
Maybe someone can respond to the compatibility issue better than me. 


However, this is how I managed to make hive work with Spark


I had similar issues with the downloaded Spark 1.5.2 and in the end I used the prebuilt Spark
1.4.1 with my Hive 1.2.1 (and Hadoop 2.7.1).
When doing the setup I used the configurations below:


hive-env.sh


export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop


hive-site.xml


<property>
   <name>hive.execution.engine</name>
   <value>spark</value>
 </property>
 <property>
   <name>spark.master</name>
   <value>spark://spark-master:7077</value>
 </property>




spark-env.sh


export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export SPARK_HOME=/usr/local/spark


export SPARK_DIST_CLASSPATH=`hadoop classpath`
export SPARK_CLASSPATH=$CLASSPATH




I don’t know the answer to your issue exactly as at some point I faced the same, but I hope
this might help.
Cheers




On 11 Jan 2016, at 11:39, Todd <bit1129@163.com> wrote:




Thank you, Sofia.


From the log, looks it is java.lang.AbstractMethodError that leads to the job failure.
I am using Hive 1.2.1+Spark 1.5.2, is this a compatibility  issue?








16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=SparkBuildPlan from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=SparkCreateTran.Map 1 from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO Configuration.deprecation:
mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO exec.Utilities:
Processing alias t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO exec.Utilities:
Adding input file hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO exec.Utilities:
Serializing MapWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
</PERFLOG method=serializePlan start=1452505326307 end=1452505326332 duration=25 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO storage.MemoryStore:
ensureFreeSpace(505800) called with curMem=0, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO storage.MemoryStore:
Block broadcast_0 stored as values in memory (estimated size 493.9 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO storage.MemoryStore:
ensureFreeSpace(42143) called with curMem=505800, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO storage.MemoryStore:
Block broadcast_0_piece0 stored as bytes in memory (estimated size 41.2 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO storage.BlockManagerInfo:
Added broadcast_0_piece0 in memory on 10.12.167.42:38883 (size: 41.2 KB, free: 265.4 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 ERROR util.Utils:
uncaught error in thread SparkListenerBus, stopping SparkContext
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: java.lang.AbstractMethodError
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO spark.SparkContext:
Created broadcast 0 from hadoopRDD at SparkPlanGenerator.java:188
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/api,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/static,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/executors,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/environment,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/storage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/stages,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO handler.ContextHandler:
stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
</PERFLOG method=SparkCreateTran.Map 1 start=1452505326274 end=1452505326824 duration=550
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=SparkCreateTran.Reducer 2 from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO exec.Utilities:
Serializing ReduceWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO ui.SparkUI:
Stopped Spark web UI at http://10.12.167.42:4040
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO scheduler.DAGScheduler:
Stopping DAGScheduler
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO cluster.YarnClientSchedulerBackend:
Shutting down all executors
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO cluster.YarnClientSchedulerBackend:
Interrupting monitor thread
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO cluster.YarnClientSchedulerBackend:
Asking each executor to shut down
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO cluster.YarnClientSchedulerBackend:
Stopped
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
</PERFLOG method=serializePlan start=1452505326849 end=1452505326874 duration=25 from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
</PERFLOG method=SparkCreateTran.Reducer 2 start=1452505326824 end=1452505326936 duration=112
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
</PERFLOG method=SparkBuildPlan start=1452505326273 end=1452505326936 duration=663 from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:06 INFO log.PerfLogger:
<PERFLOG method=SparkBuildRDDGraph from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:07 INFO log.PerfLogger:
</PERFLOG method=SparkBuildRDDGraph start=1452505326936 end=1452505327000 duration=64 from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11 17:42:07 INFO client.RemoteDriver:
Failed to run job b5a8886e-8229-4773-b943-b3c6f3c710bf
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: java.lang.IllegalStateException:
Cannot call methods on a stopped SparkContext
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at java.lang.Thread.run(Thread.java:745)
16/01/11 17:42:07 [RPC-Handler-3]: INFO client.SparkClientImpl: Received result for b5a8886e-8229-4773-b943-b3c6f3c710bf
Status: Failed
16/01/11 17:42:07 [main]: ERROR status.SparkJobMonitor: Status: Failed





At 2016-01-11 17:17:51, "Sofia" <sofia.panagiotidi@taiger.com> wrote:

Check log files to see the exact error. Alternatively run hive like this, to get the exact
error


hive --hiveconf hive.root.logger=INFO,console 


There is a good chance that you are encountering problems between the Hive and Spark versions
and installation.
See https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started




On 11 Jan 2016, at 08:47, Todd <bit1129@163.com> wrote:


Hive,


I am trying out the Hive on Spark with hive 1.2.1 and spark 1.5.2. Could someone help me on
this? Thanks!


Following are my steps:
1. build spark 1.5.2 without Hive and Hive Thrift Server. At this point,  I can use it to
submit application using spark-submit --master yarn-client
2. And the built spark assembly jar into $HIVE_HOME/lib
3. start hive and add the following parameters
hive > set spark.master=yarn-client
hive > set spark.executor.memory=512M
hive > set spark.driver.memory=512M
hive > set  spark.executor.instances=1


4. Then I run a simple query : select count(1) from t1;


The job fails will following error:


===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
    SPARK_LOG_URL_STDERR -> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096
    SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1452320323183_0007
    SPARK_YARN_CACHE_FILES_FILE_SIZES -> 142746538
    SPARK_USER -> hadoop
    SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
    SPARK_YARN_MODE -> true
    SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1452496343550
    SPARK_LOG_URL_STDOUT -> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096
    SPARK_YARN_CACHE_FILES -> hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar


  command:
    {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m '-Dhive.spark.log.dir=/home/hadoop/software/bigdata/spark-1.5.2-bin-hadoop2.6.0-withouthive/logs/'
-Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=43675' -Dspark.yarn.app.container.log.dir=<LOG_DIR>
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler
--executor-id 2 --hostname hadoop-Aspire-TC-606 --cores 1 --app-id application_1452320323183_0007
--user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
===============================================================================
      
16/01/11 15:12:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy : hadoop-Aspire-TC-606:50804
16/01/11 15:12:40 INFO yarn.YarnAllocator: Completed container container_1452320323183_0007_01_000003
(state: COMPLETE, exit status: 1)
16/01/11 15:12:40 INFO yarn.YarnAllocator: Container marked as failed: container_1452320323183_0007_01_000003.
Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1452320323183_0007_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)




Container exited with a non-zero exit code 1



































Mime
View raw message