spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ma...@apache.org
Subject [9/9] git commit: Merge pull request #102 from tdas/transform
Date Sat, 26 Oct 2013 00:26:27 GMT
Merge pull request #102 from tdas/transform

Added new Spark Streaming operations

New operations
- transformWith which allows arbitrary 2-to-1 DStream transform, added to Scala and Java API
- StreamingContext.transform to allow arbitrary n-to-1 DStream
- leftOuterJoin and rightOuterJoin between 2 DStreams, added to Scala and Java API
- missing variations of join and cogroup added to Scala Java API
- missing JavaStreamingContext.union

Updated a number of Java and Scala API docs


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d307db6e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d307db6e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d307db6e

Branch: refs/heads/master
Commit: d307db6e554885d33e22ac09b567c66a0e702fd1
Parents: 85e2cab dc95707
Author: Matei Zaharia <matei@eecs.berkeley.edu>
Authored: Fri Oct 25 17:26:06 2013 -0700
Committer: Matei Zaharia <matei@eecs.berkeley.edu>
Committed: Fri Oct 25 17:26:06 2013 -0700

----------------------------------------------------------------------
 .../org/apache/spark/api/java/JavaPairRDD.scala |  11 +
 .../spark/api/java/function/Function3.java      |  36 ++
 .../api/java/function/WrappedFunction3.scala    |  34 ++
 .../org/apache/spark/streaming/DStream.scala    |  46 ++-
 .../spark/streaming/PairDStreamFunctions.scala  | 152 ++++++--
 .../spark/streaming/StreamingContext.scala      |  13 +-
 .../streaming/api/java/JavaDStreamLike.scala    |  97 ++++-
 .../streaming/api/java/JavaPairDStream.scala    | 178 +++++++--
 .../api/java/JavaStreamingContext.scala         |  75 +++-
 .../streaming/dstream/CoGroupedDStream.scala    |  58 ---
 .../streaming/dstream/TransformedDStream.scala  |  20 +-
 .../apache/spark/streaming/JavaAPISuite.java    | 376 ++++++++++++++++++-
 .../spark/streaming/BasicOperationsSuite.scala  | 103 ++++-
 13 files changed, 1037 insertions(+), 162 deletions(-)
----------------------------------------------------------------------



Mime
View raw message