spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwend...@apache.org
Subject [9/9] git commit: Merge pull request #400 from tdas/dstream-move
Date Mon, 13 Jan 2014 20:18:26 GMT
Merge pull request #400 from tdas/dstream-move

Moved DStream and PairDSream to org.apache.spark.streaming.dstream

Similar to the package location of `org.apache.spark.rdd.RDD`, `DStream` has been moved from
`org.apache.spark.streaming.DStream` to `org.apache.spark.streaming.dstream.DStream`. I know
that the package name is a little long, but I think its better to keep it consistent with
Spark's structure.

Also fixed persistence of windowed DStream. The RDDs generated generated by windowed DStream
are essentially unions of underlying RDDs, and persistent these union RDDs would store numerous
copies of the underlying data. Instead setting the persistence level on the windowed DStream
is made to set the persistence level of the underlying DStream.


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b93f9d42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b93f9d42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b93f9d42

Branch: refs/heads/master
Commit: b93f9d42f21f03163734ef97b2871db945e166da
Parents: e6ed13f ffa1d38
Author: Patrick Wendell <pwendell@gmail.com>
Authored: Mon Jan 13 12:18:05 2014 -0800
Committer: Patrick Wendell <pwendell@gmail.com>
Committed: Mon Jan 13 12:18:05 2014 -0800

----------------------------------------------------------------------
 docs/streaming-programming-guide.md             |   2 +-
 .../examples/RecoverableNetworkWordCount.scala  |   2 +-
 .../spark/streaming/flume/FlumeUtils.scala      |   4 +-
 .../spark/streaming/kafka/KafkaUtils.scala      |   6 +-
 .../apache/spark/streaming/mqtt/MQTTUtils.scala |   4 +-
 .../spark/streaming/twitter/TwitterUtils.scala  |   7 +-
 .../spark/streaming/zeromq/ZeroMQUtils.scala    |   3 +-
 .../org/apache/spark/streaming/DStream.scala    | 756 ------------------
 .../spark/streaming/DStreamCheckpointData.scala | 128 ----
 .../apache/spark/streaming/DStreamGraph.scala   |   6 +-
 .../spark/streaming/PairDStreamFunctions.scala  | 621 ---------------
 .../spark/streaming/StreamingContext.scala      |   5 +-
 .../spark/streaming/api/java/JavaDStream.scala  |   3 +-
 .../streaming/api/java/JavaDStreamLike.scala    |   1 +
 .../streaming/api/java/JavaPairDStream.scala    |   1 +
 .../api/java/JavaStreamingContext.scala         |   5 +-
 .../spark/streaming/dstream/DStream.scala       | 757 +++++++++++++++++++
 .../dstream/DStreamCheckpointData.scala         | 126 +++
 .../streaming/dstream/FileInputDStream.scala    |   2 +-
 .../streaming/dstream/FilteredDStream.scala     |   2 +-
 .../dstream/FlatMapValuedDStream.scala          |   2 +-
 .../streaming/dstream/FlatMappedDStream.scala   |   2 +-
 .../streaming/dstream/ForEachDStream.scala      |   2 +-
 .../streaming/dstream/GlommedDStream.scala      |   2 +-
 .../spark/streaming/dstream/InputDStream.scala  |   2 +-
 .../dstream/MapPartitionedDStream.scala         |   2 +-
 .../streaming/dstream/MapValuedDStream.scala    |   2 +-
 .../spark/streaming/dstream/MappedDStream.scala |   2 +-
 .../dstream/PairDStreamFunctions.scala          | 622 +++++++++++++++
 .../dstream/ReducedWindowedDStream.scala        |   2 +-
 .../streaming/dstream/ShuffledDStream.scala     |   2 +-
 .../spark/streaming/dstream/StateDStream.scala  |   2 +-
 .../streaming/dstream/TransformedDStream.scala  |   2 +-
 .../spark/streaming/dstream/UnionDStream.scala  |   3 +-
 .../streaming/dstream/WindowedDStream.scala     |  17 +-
 .../streaming/util/MasterFailureTest.scala      |   2 +-
 .../spark/streaming/BasicOperationsSuite.scala  |   1 +
 .../spark/streaming/CheckpointSuite.scala       |   2 +-
 .../spark/streaming/StreamingContextSuite.scala |   3 +-
 .../streaming/StreamingListenerSuite.scala      |   1 +
 .../apache/spark/streaming/TestSuiteBase.scala  |   2 +-
 .../spark/streaming/WindowOperationsSuite.scala |  15 +
 .../tools/JavaAPICompletenessChecker.scala      |  50 +-
 43 files changed, 1613 insertions(+), 1570 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/dstream/StateDStream.scala
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/util/MasterFailureTest.scala
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
----------------------------------------------------------------------


Mime
View raw message