hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuefu Zhang <xzh...@cloudera.com>
Subject Re: Job aborted due to stage failure
Date Wed, 03 Dec 2014 03:03:23 GMT
When you build Spark, remove -Phive as well as -Pyarn. When you run hive
queries, you may need to run "set spark.home=/path/to/spark/dir";

Thanks,
Xuefu

On Tue, Dec 2, 2014 at 6:29 PM, yuemeng1 <yuemeng1@huawei.com> wrote:

>  hi,XueFu,thanks a lot for your help,now i will provide more detail to
> reproduce this ssue:
> 1),i checkout a spark branch from hive github(
> https://github.com/apache/hive/tree/spark on Nov 29,becasue of for
> version now it will give something wrong about:Caused by:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ),
> and built command:mvn clean package -DskipTests -Phadoop-2 -Pdist
> after built i get package from
> :/home/ym/hive-on-spark/hive1129/hive/packaging/target(apache-hive-0.15.0-SNAPSHOT-bin.tar.gz)
> 2)i checkout spark from
> https://github.com/apache/spark/tree/v1.2.0-snapshot0,becasue of spark
> branch-1.2 is with spark parent(1.2.1-SNAPSHOT),so i chose
> v1.2.0-snapshot0 and i compare this spark's pom.xml with
> spark-parent-1.2.0-SNAPSHOT.pom(get from
> http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and
> there is only difference is spark-parent name,and built command is :
>
> mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package
>
> 3)comand i execute in hive-shell:
> ./hive --auxpath
> /opt/hispark/spark/assembly/target/scala-2.10/spark-assembly-1.2.0-hadoop2.4.0.jar(copy
> this jar to hive dir lib already)
> create table student(sno int,sname string,sage int,ssex string) row format
> delimited FIELDS TERMINATED BY ',';
> create table score(sno int,cno int,sage int) row format delimited FIELDS
> TERMINATED BY ',';
> load data local inpath
> '/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/student.txt'
> into table student;
> load data local inpath
> '/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/score.txt'
> into table score;
> set hive.execution.engine=spark;
> set spark.master=spark://10.175.xxx.xxx:7077;
> set spark.eventLog.enabled=true;
> set spark.executor.memory=9086m;
> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
> select distinct st.sno,sname from student st join score sc
> on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;(work in mr)
> 4)
> studdent.txt file
> 1,rsh,27,female
> 2,kupo,28,male
> 3,astin,29,female
> 4,beike,30,male
> 5,aili,31,famle
>
> score.txt file
> 1,10,80
> 2,11,85
> 3,12,90
> 4,13,95
> 5,14,100
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 2014/12/2 23:28, Xuefu Zhang wrote:
>
>  Could you provide details on how to reproduce the issue? such as the
> exact spark branch, the command to build Spark, how you build Hive, and
> what queries/commands you run.
>
>  We are running Hive on Spark all the time. Our pre-commit test runs
> without any issue.
>
>  Thanks,
>  Xuefu
>
> On Tue, Dec 2, 2014 at 4:13 AM, yuemeng1 <yuemeng1@huawei.com> wrote:
>
>>  hi,XueFu
>> i checkout a spark branch from sparkgithub(tags:v1.2.0-snapshot0)and i
>> compare this spark's pom.xml with spark-parent-1.2.0-SNAPSHOT.pom(get from
>> http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and
>> there is only difference is follow:
>> in spark-parent-1.2.0-SNAPSHOT.pom
>>   <artifactId>spark-parent</artifactId>
>>   <version>1.2.0-SNAPSHOT</version>
>> and in v1.2.0-snapshot0
>> <artifactId>spark-parent</artifactId>
>>   <version>1.2.0</version>
>> i think there is no essence diff,and i built v1.2.0-snapshot0 and deploy
>> it as my spark clusters
>> when i run query about join two table ,it still give some error what i
>> show u earlier
>>
>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times,
>> most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18):
>> java.lang.NullPointerException+details
>>
>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent
failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): java.lang.NullPointerException
>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
>> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
>> 	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233)
>> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
>> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
>> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:56)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> 	at java.lang.Thread.run(Thread.java:722)
>>
>> Driver stacktrace:
>>
>>
>>
>>  i think my spark clusters did't had any problem,but why always give me
>> such error
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 2014/12/2 13:39, Xuefu Zhang wrote:
>>
>>  You need to build your spark assembly from spark 1.2 branch. this
>> should give your both a spark build as well as spark-assembly jar, which
>> you need to copy to Hive lib directory. Snapshot is fine, and spark 1.2
>> hasn't been released yet.
>>
>>  --Xuefu
>>
>> On Mon, Dec 1, 2014 at 7:41 PM, yuemeng1 <yuemeng1@huawei.com> wrote:
>>
>>>
>>>
>>> hi.XueFu,
>>> thanks a lot for your inforamtion,but as far as i know ,the latest spark
>>> version on github is spark-snapshot-1.3,but there is no spark-1.2,only have
>>> a branch-1.2 with spark-snapshot-1.2,can u tell me which spark version i
>>> should built,and for now,that's
>>> spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar produce error like that
>>>
>>>
>>> On 2014/12/2 11:03, Xuefu Zhang wrote:
>>>
>>>  It seems that wrong class, HiveInputFormat, is loaded. The stacktrace
>>> is way off the current Hive code. You need to build Spark 1.2 and copy
>>> spark-assembly jar to Hive's lib directory and that it.
>>>
>>>  --Xuefu
>>>
>>> On Mon, Dec 1, 2014 at 6:22 PM, yuemeng1 <yuemeng1@huawei.com> wrote:
>>>
>>>>  hi,i built a hive on spark package and my spark assembly jar is
>>>> spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar,when i run a query in hive
>>>> shell,before execute this query,
>>>> i set all the  require which hive need with  spark.and i execute a join
>>>> query :
>>>> select distinct st.sno,sname from student st join score sc
>>>> on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;
>>>> but it failed,
>>>> get follow error in spark webUI:
>>>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times,
>>>> most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18):
>>>> java.lang.NullPointerException+details
>>>>
>>>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): java.lang.NullPointerException
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
>>>> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
>>>> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
>>>> 	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233)
>>>> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
>>>> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
>>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
>>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
>>>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
>>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
>>>> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>>> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:56)
>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> 	at java.lang.Thread.run(Thread.java:722)
>>>>
>>>> Driver stacktrace:
>>>>
>>>>
>>>>  can u give me a help to deal this probelm,and i think my built was
>>>> succussed!
>>>>
>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message