spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite
Date Sat, 14 Feb 2015 01:45:38 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 fccd38d2e -> 152147f5f

[SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite

This patch fixes two difficult-to-reproduce Jenkins test failures in InputOutputMetricsSuite
(SPARK-5227 and SPARK-5679).  The problem was that WholeTextFileRecordReaderSuite modifies
the `fs.local.block.size` Hadoop configuration and this change was affecting subsequent test
suites due to Hadoop's caching of FileSystem instances (see HADOOP-8490 for more details).

The fix implemented here is to disable FileSystem caching in WholeTextFileRecordReaderSuite.

Author: Josh Rosen <>

Closes #4599 from JoshRosen/inputoutputsuite-fix and squashes the following commits:

47dc447 [Josh Rosen] [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite

(cherry picked from commit d06d5ee9b33505774ef1e5becc01b47492f1a2dc)
Signed-off-by: Patrick Wendell <>


Branch: refs/heads/branch-1.3
Commit: 152147f5f884ae4eea3873f01719e6ab9bc7afd2
Parents: fccd38d
Author: Josh Rosen <>
Authored: Fri Feb 13 17:45:31 2015 -0800
Committer: Patrick Wendell <>
Committed: Fri Feb 13 17:45:36 2015 -0800

 .../spark/input/WholeTextFileRecordReaderSuite.scala    | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
index 98b0a16..2e58c15 100644
--- a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
+++ b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
@@ -28,7 +28,7 @@ import org.scalatest.FunSuite
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
 import org.apache.spark.util.Utils
 import{DefaultCodec, CompressionCodecFactory, GzipCodec}
@@ -42,7 +42,15 @@ class WholeTextFileRecordReaderSuite extends FunSuite with BeforeAndAfterAll
   private var factory: CompressionCodecFactory = _
   override def beforeAll() {
-    sc = new SparkContext("local", "test")
+    // Hadoop's FileSystem caching does not use the Configuration as part of its cache key,
+    // can cause Filesystem.get(Configuration) to return a cached instance created with a
+    // configuration than the one passed to get() (see HADOOP-8490 for more details). This
+    // hard-to-reproduce test failures, since any suites that were run after this one would
+    // the new value of "fs.local.block.size" (see SPARK-5227 and SPARK-5679). To work around
+    // we disable FileSystem caching in this suite.
+    val conf = new SparkConf().set("spark.hadoop.fs.file.impl.disable.cache", "true")
+    sc = new SparkContext("local", "test", conf)
     // Set the block size of local file system to test whether files are split right or not.
     sc.hadoopConfiguration.setLong("fs.local.block.size", 32)

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message