spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tdas <...@git.apache.org>
Subject [GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...
Date Thu, 12 Apr 2018 00:08:24 GMT
Github user tdas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20828#discussion_r180933380
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
---
    @@ -53,32 +53,24 @@ class ContinuousSuiteBase extends StreamTest {
       // A continuous trigger that will only fire the initial time for the duration of a
test.
       // This allows clean testing with manual epoch advancement.
       protected val longContinuousTrigger = Trigger.Continuous("1 hour")
    +
    +  override protected implicit val defaultTrigger = Trigger.Continuous(100)
    +  override protected val defaultUseV2Sink = true
     }
     
     class ContinuousSuite extends ContinuousSuiteBase {
       import testImplicits._
     
    -  test("basic rate source") {
    -    val df = spark.readStream
    -      .format("rate")
    -      .option("numPartitions", "5")
    -      .option("rowsPerSecond", "5")
    -      .load()
    -      .select('value)
    +  test("basic") {
    +    val input = MemoryStream[Int]
    --- End diff --
    
    I think that's too much to hope for right now. We can do that later. For now, let's make
everything explicitly ContinuousMemoryStream.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message