nemo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] taegeonum commented on a change in pull request #123: [NEMO-129] Support Beam's WindowedWordCount example
Date Wed, 17 Oct 2018 01:27:48 GMT
taegeonum commented on a change in pull request #123: [NEMO-129] Support Beam's WindowedWordCount
example
URL: https://github.com/apache/incubator-nemo/pull/123#discussion_r225756889
 
 

 ##########
 File path: compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam/PipelineTranslator.java
 ##########
 @@ -198,11 +196,29 @@ private static void parDoMultiOutputTranslator(final TranslationContext
ctx,
         pValueWithTupleTag.getKey()));
   }
 
+  private static GroupByKeyTransform createGBKTransform(
+    final TranslationContext ctx,
+    final TransformVertex transformVertex) {
+    final AppliedPTransform pTransform = transformVertex.getNode().toAppliedPTransform(PIPELINE.get());
+    final PCollection<?> mainInput = (PCollection<?>)
+      Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(pTransform));
+    final TupleTag mainOutputTag = new TupleTag<>("main output");
+
+    return new GroupByKeyTransform(
+      getOutputCoders(pTransform),
+      mainOutputTag,
+      Collections.emptyList(),
+      mainInput.getWindowingStrategy(),
+      Collections.emptyList(), /* side inputs */
+      ctx.pipelineOptions,
+      SystemReduceFn.buffering(mainInput.getCoder()));
 
 Review comment:
   Nemo currently does not perform incremental combining. 
   It groups inputs according to the window and key and combine the collected inputs after
grouping is done for all bounded inputs. Using `System.ReduceFn.combining` can enable incremental
combining, and I think this is a separate issue. I've filed an issue for this https://issues.apache.org/jira/browse/NEMO-222

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message