hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when vectorized execution is switched on
Date Wed, 16 Mar 2016 05:51:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196806#comment-15196806
] 

Rui Li commented on HIVE-13277:
-------------------------------

Pinging [~xuefuz]

> Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer'
" occurred during query execution on spark engine when vectorized execution is switched on
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13277
>                 URL: https://issues.apache.org/jira/browse/HIVE-13277
>             Project: Hive
>          Issue Type: Bug
>         Environment: Hive on Spark engine
> Hive Version: Apache Hive 2.0.0
> Spark Version: Apache Spark 1.6.0
>            Reporter: Xin Hao
>
> Found when executing TPCx-BB query2 for Hive on Spark engine, and switch on :
> Found during TPCx-BB query2 execution on spark engine when vectorized execution is switched
on:
> (1) set hive.vectorized.execution.enabled=true; 
> (2) set hive.vectorized.execution.reduce.enabled=true; (default value for Apache Hive
2.0.0)
> It's OK for spark engine when hive.vectorized.execution.enabled is switched off:
> (1) set hive.vectorized.execution.enabled=false;
> (2) set hive.vectorized.execution.reduce.enabled=true;
> For MR engine, the query could pass and no exception occurred when vectorized execution
is either switched on or switched off.
> Detail Error Message is below:
> {noformat}
> 2016-03-14T10:09:33,692 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- 16/03/14 10:09:33 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle
0 is 154 bytes
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- 16/03/14 10:09:33 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 4.0 (TID 25, bhx3):
java.lang.RuntimeException: Failed to load plan: hdfs://bhx3:8020/tmp/hive/root/40b90ebd-32d4-47bc-a5ab-12ff1c05d0d2/hive_2016-03-14_10-08-56_307_7692316402338632647-1/-mr-10002/ab0c0021-0c1a-496e-9703-87d5879353c8/reduce.xml:
org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer"
for class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- Serialization trace:
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorLimitOperator)
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
> 2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:306)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:117)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.scheduler.Task.run(Task.scala:89)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at java.lang.Thread.run(Thread.java:745)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer"
for class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- Serialization trace:
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorLimitOperator)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,820 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:438)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:347)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:324)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:424)
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        ... 17 more
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
- Caused by: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer"
for class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,821 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> 2016-03-14T10:09:33,822 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> 2016-03-14T10:09:33,822 INFO  [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(593))
-        at org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message