hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-18289) Fix jar dependency when enable rdd cache in Hive on Spark
Date Mon, 18 Dec 2017 04:15:00 GMT
liyunzhang created HIVE-18289:
---------------------------------

             Summary: Fix jar dependency when enable rdd cache in Hive on Spark
                 Key: HIVE-18289
                 URL: https://issues.apache.org/jira/browse/HIVE-18289
             Project: Hive
          Issue Type: Bug
            Reporter: liyunzhang
            Assignee: liyunzhang


running DS/query28 when enabling HIVE-17486's 4th patch
on tpcds_bin_partitioned_orc_10 whether on spark local or yarn mode
command
{code}
set spark.local=yarn-client;
echo 'use tpcds_bin_partitioned_orc_10;source query28.sql;'|hive --hiveconf spark.app.name=query28.sql
 --hiveconf hive.spark.optimize.shared.work=true -i testbench.settings -i query28.sql.setting
{code}
the exception 
{code}
ava.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.orc.OrcStruct.<init>()
748678         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
~[hadoop-common-2.7.3.jar:?]
748679         at org.apache.hadoop.io.WritableUtils.clone(WritableUtils.java:217) ~[hadoop-common-2.7.3.jar:?]
748680         at org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:85)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.       0-SNAPSHOT]
748681         at org.apache.hadoop.hive.ql.exec.spark.MapInput$CopyFunction.call(MapInput.java:72)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.       0-SNAPSHOT]
748682         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031)
~[spark-core_2.11-2.       0.0.jar:2.0.0]
748683         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1031)
~[spark-core_2.11-2.       0.0.jar:2.0.0]
748684         at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) ~[scala-library-2.11.8.jar:?]
748685         at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:214)
~[spark-core_2.11-2.0.0.jar:2.       0.0]
748686         at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:919)
~[spark-core_2.11-2.0.0.       jar:2.0.0]
748687         at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
~[spark-core_2.11-2.0.0.       jar:2.0.0]
748688         at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866) ~[spark-core_2.11-2.0.0.jar:2.0.0]
748689         at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
~[spark-core_2.11-2.0.0.jar:2.0.0]
748690         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668)
~[spark-core_2.11-2.0.0.jar:2.0.0]
748691         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330) ~[spark-core_2.11-2.0.0.jar:2.0.0]
748692         at org.apache.spark.rdd.RDD.iterator(RDD.scala:281) ~[spark-core_2.11-2.0.0.jar:2.0.0]
748693         at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
~[spark-core_2.11-2.0.0.jar:2.0.0]
748694         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) ~[spark-core_2.11-2.0.0.jar:2.0.0]
748695         at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) ~[spark-core_2.11-2.0.0.jar:2.0.0]
748696         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
~[spark-core_2.11-2
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message