spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject spark git commit: [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite
Date Sat, 14 Feb 2015 01:45:38 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 fccd38d2e -> 152147f5f


[SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite

This patch fixes two difficult-to-reproduce Jenkins test failures in InputOutputMetricsSuite
(SPARK-5227 and SPARK-5679).  The problem was that WholeTextFileRecordReaderSuite modifies
the `fs.local.block.size` Hadoop configuration and this change was affecting subsequent test
suites due to Hadoop's caching of FileSystem instances (see HADOOP-8490 for more details).

The fix implemented here is to disable FileSystem caching in WholeTextFileRecordReaderSuite.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #4599 from JoshRosen/inputoutputsuite-fix and squashes the following commits:

47dc447 [Josh Rosen] [SPARK-5227] [SPARK-5679] Disable FileSystem cache in WholeTextFileRecordReaderSuite

(cherry picked from commit d06d5ee9b33505774ef1e5becc01b47492f1a2dc)
Signed-off-by: Patrick Wendell <patrick@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/152147f5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/152147f5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/152147f5

Branch: refs/heads/branch-1.3
Commit: 152147f5f884ae4eea3873f01719e6ab9bc7afd2
Parents: fccd38d
Author: Josh Rosen <joshrosen@databricks.com>
Authored: Fri Feb 13 17:45:31 2015 -0800
Committer: Patrick Wendell <patrick@databricks.com>
Committed: Fri Feb 13 17:45:36 2015 -0800

----------------------------------------------------------------------
 .../spark/input/WholeTextFileRecordReaderSuite.scala    | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/152147f5/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
index 98b0a16..2e58c15 100644
--- a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
+++ b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
@@ -28,7 +28,7 @@ import org.scalatest.FunSuite
 
 import org.apache.hadoop.io.Text
 
-import org.apache.spark.SparkContext
+import org.apache.spark.{SparkConf, SparkContext}
 import org.apache.spark.util.Utils
 import org.apache.hadoop.io.compress.{DefaultCodec, CompressionCodecFactory, GzipCodec}
 
@@ -42,7 +42,15 @@ class WholeTextFileRecordReaderSuite extends FunSuite with BeforeAndAfterAll
{
   private var factory: CompressionCodecFactory = _
 
   override def beforeAll() {
-    sc = new SparkContext("local", "test")
+    // Hadoop's FileSystem caching does not use the Configuration as part of its cache key,
which
+    // can cause Filesystem.get(Configuration) to return a cached instance created with a
different
+    // configuration than the one passed to get() (see HADOOP-8490 for more details). This
caused
+    // hard-to-reproduce test failures, since any suites that were run after this one would
inherit
+    // the new value of "fs.local.block.size" (see SPARK-5227 and SPARK-5679). To work around
this,
+    // we disable FileSystem caching in this suite.
+    val conf = new SparkConf().set("spark.hadoop.fs.file.impl.disable.cache", "true")
+
+    sc = new SparkContext("local", "test", conf)
 
     // Set the block size of local file system to test whether files are split right or not.
     sc.hadoopConfiguration.setLong("fs.local.block.size", 32)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message