hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jone Zhang <joyoungzh...@gmail.com>
Subject Re: Hive on Spark NPE at org.apache.hadoop.hive.ql.io.HiveInputFormat
Date Mon, 02 Nov 2015 01:20:41 GMT
pls check and  attach the application master log.

2015-11-02 8:03 GMT+08:00 Jagat Singh <jagatsingh@gmail.com>:

> Hi,
>
> I am trying to run Hive on Spark on HDP Virtual machine 2.3
>
> Following wiki
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
>
> I have replaced all the occurrences of hdp.version with 2.3.0.0-2557
>
> I start hive with following
>
> set hive.execution.engine=spark;
> set spark.master=yarn-client;
> set spark.executor.memory=512m;
>
> I run the query
>
> select count(*) from sample_07;
>
> The query starts and fails with following error.
>
> In console
>
> Status: Running (Hive on Spark job[0])
> Job Progress Format
> CurrentTime StageId_StageAttemptId:
> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
> [StageCost]
> 2015-11-01 23:40:26,411 Stage-0_0: 0(+1)/1 Stage-1_0: 0/1
> state = FAILED
> Status: Failed
> FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> hive> select count(*) from sample_07;
>
> In the logs
>
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 2015-11-01 23:55:36,313
INFO  - [pool-1-thread-1:] ~ Failed to run job b8649c92-1504-43c7-8100-020b866e58da (RemoteDriver:389)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: java.util.concurrent.ExecutionException:
Exception thrown by job
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:311)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:316)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:382)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.lang.Thread.run(Thread.java:745)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: Caused by: org.apache.spark.SparkException:
Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 0.0 (TID 3, sandbox.hortonworks.com): java.lang.NullPointerException
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:236)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:212)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at java.lang.Thread.run(Thread.java:745)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl:
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: Driver stacktrace:
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at scala.Option.foreach(Option.scala:236)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> 15/11/01 23:55:36 [stdout-redir-1]: INFO client.SparkClientImpl: 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> 15/11/01 23:55:36 [RPC-Handler-3]: DEBUG rpc.KryoMessageCodec: Decoded message of type
org.apache.hive.spark.client.rpc.Rpc$MessageHeader (5 bytes)
> 15/11/01 23:55:36 [RPC-Handler-3]: DEBUG rpc.KryoMessageCodec: Decoded message of type
org.apache.hive.spark.client.BaseProtocol$JobResult (3851 bytes)
> 15/11/01 23:55:36 [RPC-Handler-3]: DEBUG rpc.RpcDispatcher: [ClientProtocol] Received
RPC message: type=CALL id=2 payload=org.apache.hive.spark.client.BaseProtocol$JobResult
> 15/11/01 23:55:36 [RPC-Handler-3]: INFO client.SparkClientImpl: Received result for b8649c92-1504-43c7-8100-020b866e58da
> 15/11/01 23:55:36 [RPC-Handler-3]: DEBUG rpc.KryoMessageCodec: Encoded message of type
org.apache.hive.spark.client.rpc.Rpc$MessageHeader (5 bytes)
> 15/11/01 23:55:36 [RPC-Handler-3]: DEBUG rpc.KryoMessageCodec: Encoded message of type
org.apache.hive.spark.client.rpc.Rpc$NullMessage (2 bytes)
> state = FAILED
> 15/11/01 23:55:36 [main]: INFO status.SparkJobMonitor: state = FAILED
> Status: Failed
> 15/11/01 23:55:36 [main]: ERROR status.SparkJobMonitor: Status: Failed
>
>
> In Resource manager i get succeeded
>
> [image: Inline image 1]
>
>
>
> How to debug this ?
>
> Thanks,
>
>
>
>
>

Mime
View raw message