hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Sun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-16175) Possible race condition in InstanceCache
Date Fri, 10 Mar 2017 16:57:04 GMT

     [ https://issues.apache.org/jira/browse/HIVE-16175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chao Sun updated HIVE-16175:
----------------------------
    Status: Patch Available  (was: Open)

> Possible race condition in InstanceCache
> ----------------------------------------
>
>                 Key: HIVE-16175
>                 URL: https://issues.apache.org/jira/browse/HIVE-16175
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Chao Sun
>            Assignee: Chao Sun
>         Attachments: HIVE-16175.1.patch
>
>
> Currently the [InstanceCache | https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java]
class is not thread safe, but it is sometimes used as a static variable, for instance [here|https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java#L114].
This will be an issue for HoS, at which scenario it could be accessed by multiple threads
at the same time. We found this sometimes causes NPE:
> {code}
> ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
java.util.concurrent.ExecutionException: Exception thrown by job
>         at org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:311)
>         at org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:316)
>         at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:382)
>         at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 20
in stage 0.0 failed 4 times, most recent failure: Lost task 20.3 in stage 0.0 (TID 33, hadoopworker992-sjc1.prod.uber.internal):
java.lang.RuntimeException: Map operator initialization failed: org.apache.hadoop.hive.ql.metadata.Hive
> Exception: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:127)
>         at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:55)
>         at org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:30)
>         at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
>         at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
>         at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>         at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>         at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:325)
>         at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:388)
>         at org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:92)
>         ... 16 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories(AvroObjectInspectorGenerator.java:142)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:91)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:104)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:104)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:121)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspector(AvroObjectInspectorGenerator.java:83)
>         at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.<init>(AvroObjectInspectorGenerator.java:56)
>         at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:126)
>         at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:80)
>         at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:520)
>         at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:142)
>         at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:299)
>         ... 18 more
> {code}
> (The stack trace is from Hive 1.1.0 but the issue should also exist in upstream).
> This issue is also reported earlier: https://groups.google.com/forum/#!topic/presto-users/CpEDvLiUSX0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message