flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From u..@apache.org
Subject [1/2] flink git commit: [FLINK-3469] [docs] Improve documentation for grouping keys
Date Mon, 11 Apr 2016 12:21:03 GMT
Repository: flink
Updated Branches:
  refs/heads/release-1.0 1554c9b42 -> 0e43ce977


[FLINK-3469] [docs] Improve documentation for grouping keys

This closes #1858.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/0e43ce97
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/0e43ce97
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/0e43ce97

Branch: refs/heads/release-1.0
Commit: 0e43ce977575fbbb345cb1350aa219b8a8f4297c
Parents: 62e811f
Author: Greg Hogan <code@greghogan.com>
Authored: Wed Apr 6 15:44:47 2016 -0400
Committer: Ufuk Celebi <uce@apache.org>
Committed: Mon Apr 11 14:20:55 2016 +0200

----------------------------------------------------------------------
 docs/apis/batch/dataset_transformations.md | 120 +++++++++++++++++++++---
 1 file changed, 105 insertions(+), 15 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/0e43ce97/docs/apis/batch/dataset_transformations.md
----------------------------------------------------------------------
diff --git a/docs/apis/batch/dataset_transformations.md b/docs/apis/batch/dataset_transformations.md
index 31a1dfa..be9691c 100644
--- a/docs/apis/batch/dataset_transformations.md
+++ b/docs/apis/batch/dataset_transformations.md
@@ -275,6 +275,69 @@ element using a user-defined reduce function.
 For each group of input elements, a reduce function successively combines pairs of elements
into one
 element until only a single element for each group remains.
 
+#### Reduce on DataSet Grouped by Key Expression
+
+Key expressions specify one or more fields of each element of a DataSet. Each key expression
is
+either the name of a public field or a getter method. A dot can be used to drill down into
objects.
+The key expression "*" selects all fields.
+The following code shows how to group a POJO DataSet using key expressions and to reduce
it
+with a reduce function.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+~~~java
+// some ordinary POJO
+public class WC {
+  public String word;
+  public int count;
+  // [...]
+}
+
+// ReduceFunction that sums Integer attributes of a POJO
+public class WordCounter implements ReduceFunction<WC> {
+  @Override
+  public WC reduce(WC in1, WC in2) {
+    return new WC(in1.word, in1.count + in2.count);
+  }
+}
+
+// [...]
+DataSet<WC> words = // [...]
+DataSet<WC> wordCounts = words
+                         // DataSet grouping on field "word"
+                         .groupBy("word")
+                         // apply ReduceFunction on grouped DataSet
+                         .reduce(new WordCounter());
+~~~
+
+</div>
+<div data-lang="scala" markdown="1">
+
+~~~scala
+// some ordinary POJO
+class WC(val word: String, val count: Int) {
+  def this() {
+    this(null, -1)
+  }
+  // [...]
+}
+
+val words: DataSet[WC] = // [...]
+val wordCounts = words.groupBy("word").reduce {
+  (w1, w2) => new WC(w1.word, w1.count + w2.count)
+}
+~~~
+
+</div>
+<div data-lang="python" markdown="1">
+
+~~~python
+Not supported.
+~~~
+</div>
+</div>
+
 #### Reduce on DataSet Grouped by KeySelector Function
 
 A key-selector function extracts a key value from each element of a DataSet. The extracted
key
@@ -305,9 +368,16 @@ public class WordCounter implements ReduceFunction<WC> {
 DataSet<WC> words = // [...]
 DataSet<WC> wordCounts = words
                          // DataSet grouping on field "word"
-                         .groupBy("word")
+                         .groupBy(new SelectWord())
                          // apply ReduceFunction on grouped DataSet
                          .reduce(new WordCounter());
+
+public class SelectWord implements KeySelector<WC, String> {
+  @Override
+  public String getKey(Word w) {
+    return w.word;
+  }
+}
 ~~~
 
 </div>
@@ -332,7 +402,14 @@ val wordCounts = words.groupBy { _.word } reduce {
 <div data-lang="python" markdown="1">
 
 ~~~python
-Not supported.
+class WordCounter(ReduceFunction):
+    def reduce(self, in1, in2):
+        return (in1[0], in1[1] + in2[1])
+
+words = // [...]
+wordCounts = words \
+    .group_by(lambda x: x[0]) \
+    .reduce(WordCounter())
 ~~~
 </div>
 </div>
@@ -347,10 +424,9 @@ The following code shows how to use field position keys and apply a reduce
funct
 
 ~~~java
 DataSet<Tuple3<String, Integer, Double>> tuples = // [...]
-DataSet<Tuple3<String, Integer, Double>> reducedTuples =
-                                         tuples
+DataSet<Tuple3<String, Integer, Double>> reducedTuples = tuples
                                          // group DataSet on first and second field of Tuple
-                                         .groupBy(0,1)
+                                         .groupBy(0, 1)
                                          // apply ReduceFunction on grouped DataSet
                                          .reduce(new MyTupleReducer());
 ~~~
@@ -364,11 +440,29 @@ val tuples = DataSet[(String, Int, Double)] = // [...]
 val reducedTuples = tuples.groupBy(0, 1).reduce { ... }
 ~~~
 
+</div>
+<div data-lang="python" markdown="1">
+
+~~~python
+ reducedTuples = tuples.group_by(0, 1).reduce( ... )
+~~~
+
+</div>
+</div>
 
 #### Reduce on DataSet grouped by Case Class Fields
 
 When using Case Classes you can also specify the grouping key using the names of the fields:
 
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+~~~java
+Not supported.
+~~~
+</div>
+<div data-lang="scala" markdown="1">
+
 ~~~scala
 case class MyClass(val a: String, b: Int, c: Double)
 val tuples = DataSet[MyClass] = // [...]
@@ -380,9 +474,8 @@ val reducedTuples = tuples.groupBy("a", "b").reduce { ... }
 <div data-lang="python" markdown="1">
 
 ~~~python
- reducedTuples = tuples.group_by(0, 1).reduce( ... )
+Not supported.
 ~~~
-
 </div>
 </div>
 
@@ -442,11 +535,6 @@ val output = input.groupBy(0).reduceGroup {
     }
 ~~~
 
-#### GroupReduce on DataSet Grouped by Case Class Fields
-
-Works analogous to grouping by Case Class fields in *Reduce* transformations.
-
-
 </div>
 <div data-lang="python" markdown="1">
 
@@ -462,13 +550,15 @@ Works analogous to grouping by Case Class fields in *Reduce* transformations.
  output = data.group_by(0).reduce_group(DistinctReduce())
 ~~~
 
-
 </div>
 </div>
 
-#### GroupReduce on DataSet Grouped by KeySelector Function
+#### GroupReduce on DataSet Grouped by Key Expression, KeySelector Function, or Case Class
Fields
+
+Work analogous to [key expressions](#reduce-on-dataset-grouped-by-key-expression),
+[key-selector functions](#reduce-on-dataset-grouped-by-keyselector-function),
+and [case class fields](#reduce-on-dataset-grouped-by-case-class-fields) in *Reduce* transformations.
 
-Works analogous to key-selector functions in *Reduce* transformations.
 
 #### GroupReduce on sorted groups
 


Mime
View raw message