spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwendell <...@git.apache.org>
Subject [GitHub] incubator-spark pull request: For outputformats that are Configura...
Date Mon, 24 Feb 2014 04:07:13 GMT
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/incubator-spark/pull/638#discussion_r9980751
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -330,4 +335,74 @@ class PairRDDFunctionsSuite extends FunSuite with SharedSparkContext
{
           (1, ArrayBuffer(1)),
           (2, ArrayBuffer(1))))
       }
    +
    +  test("saveNewAPIHadoopFile should call setConf if format is configurable") {
    +    val pairs = sc.parallelize(Array((new Integer(1), new Integer(1))))
    +    val conf = new Configuration()
    +
    +    //No error, non-configurable formats still work
    +    pairs.saveAsNewAPIHadoopFile[FakeFormat]("ignored")
    +
    +    //Configurable intercepts get configured
    +    //ConfigTestFormat throws an exception if we try to write to it
    +    //when setConf hasn't been thrown first.
    +    //Assertion is in ConfigTestFormat.getRecordWriter
    +    pairs.saveAsNewAPIHadoopFile[ConfigTestFormat]("ignored")
    +  }
    +}
    +
    +// These classes are fakes for testing
    +// "saveNewAPIHadoopFile should call setConf if format is configurable".
    +// Unfortunately, they have to be top level classes, and not defined in
    +// the test method, because otherwise Scala won't generate no-args constructors
    +// and the test will therefore throw InstantiationException when saveAsNewAPIHadoopFile
    +// tries to instantiate them with Class.newInstance.
    +class FakeWriter extends RecordWriter[Integer,Integer] {
    --- End diff --
    
    `Integer, Integer`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

Mime
View raw message