hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Commented] (HIVE-16395) ConcurrentModificationException on config object in HoS
Date Fri, 19 May 2017 00:21:04 GMT


Sahil Takiar commented on HIVE-16395:

Think I found the issue. By default, Spark gives Tasks the same {{JobConf}} object to use
(ref [HadoopRDD.scala|]).
In this case, each {{CombineHiveInputFormat.getRecordReader}} inside an executor is given
the same {{JobConf}} object. This can lead to a {{ConcurrentModificationException}} if one
of the tasks mutates the {{JobConf}} while another task is iterating over it.

Similar issues have been reported in Spark too: SPARK-2546

I dug through the code a bit and Hive modifies the {{JobConf}} object in at least one place:
{{ColumnProjectionUtils.appendReadColumns}}. Its probably modified in other places too, and
SPARK-2546 suggests that some {{RecordReader}} implementations mutate it too.

The solution to SPARK-2546 was to add a config called {{spark.hadoop.cloneConf}} that is set
to {{false}} by default. When set to {{true}} it clones a new {{JobConf}} object for each
Spark Task, which avoids any thread safety issues (the [PR|]
for SPARK-2546 has a good explanation of the change). It's set to {{false}} by default for
performance considerations.

So for this JIRA we could:

1: Close this and tell users if they hit this issue just set {{spark.hadoop.cloneConf}} to
2: Given that we know Hive will mutate {{JobConf}} objects, maybe we should set {{spark.hadoop.cloneConf}}
to {{true}} for HoS, we can profile the perf impact
3: ??

[~lirui], [~xuefuz] any thoughts on this?

> ConcurrentModificationException on config object in HoS
> -------------------------------------------------------
>                 Key: HIVE-16395
>                 URL:
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
> Looks like this is happening inside spark executors, looks to be some race condition
when modifying {{Configuration}} objects.
> Stack-Trace:
> {code}
> java.lang.reflect.InvocationTargetException
> 	at
> 	at
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(
> 	at
> 	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:240)
> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
> 	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> 	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 	at
> 	at org.apache.spark.executor.Executor$
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> 	at java.lang.reflect.Constructor.newInstance(
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(
> 	... 21 more
> Caused by: java.util.ConcurrentModificationException
> 	at java.util.Hashtable$
> 	at org.apache.hadoop.conf.Configuration.iterator(
> 	at org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(
> 	at org.apache.hadoop.fs.FileSystem.access$200(
> 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(
> 	at org.apache.hadoop.fs.FileSystem$Cache.get(
> 	at org.apache.hadoop.fs.FileSystem.get(
> 	at org.apache.hadoop.fs.Path.getFileSystem(
> 	at org.apache.hadoop.mapred.LineRecordReader.<init>(
> 	at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(
> 	at<init>(
> 	... 26 more
> {code}

This message was sent by Atlassian JIRA

View raw message