Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3E4E9CE83 for ; Tue, 2 Dec 2014 08:58:17 +0000 (UTC) Received: (qmail 40750 invoked by uid 500); 2 Dec 2014 08:58:03 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 40693 invoked by uid 500); 2 Dec 2014 08:58:03 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 40678 invoked by uid 99); 2 Dec 2014 08:58:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2014 08:58:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuemeng1@huawei.com designates 119.145.14.65 as permitted sender) Received: from [119.145.14.65] (HELO szxga02-in.huawei.com) (119.145.14.65) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2014 08:57:58 +0000 Received: from 172.24.2.119 (EHLO szxeml463-hub.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id CDH69155; Tue, 02 Dec 2014 16:57:35 +0800 (CST) Received: from [127.0.0.1] (10.177.39.227) by szxeml463-hub.china.huawei.com (10.82.67.206) with Microsoft SMTP Server id 14.3.158.1; Tue, 2 Dec 2014 16:55:59 +0800 Message-ID: <547D7E9D.3010804@huawei.com> Date: Tue, 2 Dec 2014 16:55:57 +0800 From: yuemeng1 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Subject: Re: Job aborted due to stage failure References: <547D2263.9090104@huawei.com> <547D34DC.8070509@huawei.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------050006030609030507060402" X-Originating-IP: [10.177.39.227] X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org --------------050006030609030507060402 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit hi,i checkout a spark 1.2 branch from spark github,and built,then copy spark assembly jar into Hive lib directory,but when i run this qeury ,it still give me this error. i am very confused,how can i let hive on spark work! On 2014/12/2 13:39, Xuefu Zhang wrote: > You need to build your spark assembly from spark 1.2 branch. this > should give your both a spark build as well as spark-assembly jar, > which you need to copy to Hive lib directory. Snapshot is fine, and > spark 1.2 hasn't been released yet. > > --Xuefu > > On Mon, Dec 1, 2014 at 7:41 PM, yuemeng1 > wrote: > > > > hi.XueFu, > thanks a lot for your inforamtion,but as far as i know ,the latest > spark version on github is spark-snapshot-1.3,but there is no > spark-1.2,only have a branch-1.2 with spark-snapshot-1.2,can u > tell me which spark version i should built,and for now,that's > spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar produce error like that > > > On 2014/12/2 11:03, Xuefu Zhang wrote: >> It seems that wrong class, HiveInputFormat, is loaded. The >> stacktrace is way off the current Hive code. You need to build >> Spark 1.2 and copy spark-assembly jar to Hive's lib directory and >> that it. >> >> --Xuefu >> >> On Mon, Dec 1, 2014 at 6:22 PM, yuemeng1 > > wrote: >> >> hi,i built a hive on spark package and my spark assembly jar >> is spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar,when i run a >> query in hive shell,before execute this query, >> i set all the require which hive need with spark.and i >> execute a join query : >> select distinct st.sno,sname from student st join score sc >> on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28; >> but it failed, >> get follow error in spark webUI: >> Job aborted due to stage failure: Task 0 in stage 1.0 failed >> 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID >> 7, datasight18): java.lang.NullPointerException+details >> >> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): java.lang.NullPointerException >> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437) >> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430) >> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) >> at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:233) >> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210) >> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) >> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >> at org.apache.spark.scheduler.Task.run(Task.scala:56) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> Driver stacktrace: >> >> can u give me a help to deal this probelm,and i think my >> built was succussed! >> >> > > --------------050006030609030507060402 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit
hi,i checkout a spark 1.2 branch from spark github,and built,then copy spark assembly jar into Hive lib directory,but when i run this qeury ,it still give me this error.
i am very confused,how can i let hive on spark work!

On 2014/12/2 13:39, Xuefu Zhang wrote:
You need to build your spark assembly from spark 1.2 branch. this should give your both a spark build as well as spark-assembly jar, which you need to copy to Hive lib directory. Snapshot is fine, and spark 1.2 hasn't been released yet.

--Xuefu

On Mon, Dec 1, 2014 at 7:41 PM, yuemeng1 <yuemeng1@huawei.com> wrote:


hi.XueFu,
thanks a lot for your inforamtion,but as far as i know ,the latest spark version on github is spark-snapshot-1.3,but there is no spark-1.2,only have a branch-1.2 with spark-snapshot-1.2,can u tell me which spark version i should built,and for now,that's spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar produce error like that


On 2014/12/2 11:03, Xuefu Zhang wrote:
It seems that wrong class, HiveInputFormat, is loaded. The stacktrace is way off the current Hive code. You need to build Spark 1.2 and copy spark-assembly jar to Hive's lib directory and that it.

--Xuefu

On Mon, Dec 1, 2014 at 6:22 PM, yuemeng1 <yuemeng1@huawei.com> wrote:
hi,i built a hive on spark package and my spark assembly jar is spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar,when i run a query in hive shell,before execute this query,
i set all the  require which hive need with  spark.and i execute a join query :
select distinct st.sno,sname from student st join score sc on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;
but it failed,
get follow error in spark webUI:
Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): java.lang.NullPointerException+details 
Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:56)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

Driver stacktrace:
can u give me a help to deal this probelm,and i think my built was succussed!




--------------050006030609030507060402--