hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiangrui Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5893) CBZip2InputStream is not threadsafe
Date Sat, 17 May 2014 04:16:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000652#comment-14000652
] 

Xiangrui Meng commented on MAPREDUCE-5893:
------------------------------------------

Checked the code in the trunk. This class has a static boolean member `skipDecompression`,
which indicates whether it is decompressing or checking the next marker.

> CBZip2InputStream is not threadsafe
> -----------------------------------
>
>                 Key: MAPREDUCE-5893
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5893
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv1, mrv2
>    Affects Versions: 1.2.1, 2.2.0
>            Reporter: Xiangrui Meng
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the implementation is not
threadsafe. This is not a really problem for Hadoop MapReduce because Hadoop runs each task
in a separate JVM. But for other libraries that utilize multithreading and use Hadoop's InputFormat,
e.g., Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6 org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394) org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
java.io.InputStream.read(InputStream.java:101) org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176)
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198)
org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35) scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
org.apache.spark.scheduler.Task.run(Task.scala:51) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message