beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From k...@apache.org
Subject [36/36] beam git commit: This closes #3705: [BEAM-165] Initial implementation of the MapReduce runner
Date Thu, 07 Sep 2017 18:39:45 GMT
This closes #3705: [BEAM-165] Initial implementation of the MapReduce runner

  mr-runner: Removes WordCountTest, fixes checkstyle, findbugs, and addressed comments.
  mr-runner-hack: disable unrelated modules to shorten build time during development.
  mr-runner: support SourceMetrics, this fixes MetricsTest.testBoundedSourceMetrics().
  mr-runner: introduces duplicateFactor in FlattenOperation, this fixes testFlattenInputMultipleCopies().
  mr-runner: translate empty flatten into EmptySource, this fixes few empty FalttenTests.
  mr-runner: ensure Operation only start/finish once for diamond shaped DAG, this fixes ParDoLifecycleTest.
  mr-runner: Graph.getSteps() to return with topological order, this fixes few CombineTests.
  mr-runner: fail early in the runner when MapReduce job fails.
  mr-runner: use InMemoryStateInternals in ParDoOperation, this fixed ParDoTest that uses
state.
  mr-runner: use the correct step name in ParDoTranslator, this fixes MetricsTest.testAttemptedCounterMetrics().
  mr-runner: remove the hard-coded GlobalWindow coder, and fixes WindowingTest.
  mr-runner: handle no files case in FileSideInputReader for empty views.
  mr-runner: fix NPE in PipelineTest.testIdentityTransform().
  mr-runner: filter out unsupported features in ValidatesRunner tests.
  mr-runner: setMetricsSupported to run ValidatesRunner tests with TestPipeline.
  mr-runner: fix the bug that steps are attached multiple times in diamond shaped DAG.
  [BEAM-2783] support metrics in MapReduceRunner.
  mr-runner: setup file paths for read and write sides of materialization.
  mr-runner: support side inputs by reading in all views contents.
  mr-runner: support multiple SourceOperations by composing and partitioning.
  mr-runner: support PCollections materialization with multiple MR jobs.
  mr-runner: hack to get around that ViewAsXXX.expand() return wrong output PValue.
  mr-runner: support graph visualization with dotfiles.
  mr-runner: refactors and creates Graph data structures to handle general Beam pipelines.
  mr-runner: add JarClassInstanceFactory to run ValidatesRunner tests.
  mr-runner: support reduce side ParDos and WordCount.
  core-java: InMemoryTimerInternals expose getTimers() for timer firings in mr-runner.
  mr-runner: add BeamReducer and support GroupByKey.
  mr-runner: add ParDoOperation and support ParDos chaining.
  mr-runner: add JobPrototype and translate it to a MR job.
  mr-runner: support BoundedSource with BeamInputFormat.
  MapReduceRunner: add unit tests for GraphConverter and GraphPlanner.
  MapReduceRunner: add Graph and its visitors.
  Initial commit for MapReduceRunner.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5fa0b14d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5fa0b14d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5fa0b14d

Branch: refs/heads/mr-runner
Commit: 5fa0b14d20fab007d9e2d954eb4a34155a6f199f
Parents: 2fa4fde 32aeb7a
Author: Kenneth Knowles <klk@google.com>
Authored: Thu Sep 7 11:38:22 2017 -0700
Committer: Kenneth Knowles <klk@google.com>
Committed: Thu Sep 7 11:38:22 2017 -0700

----------------------------------------------------------------------
 pom.xml                                         |   2 +-
 .../runners/core/InMemoryTimerInternals.java    |   5 +
 runners/map-reduce/pom.xml                      | 197 ++++++++++++++
 .../mapreduce/MapReducePipelineOptions.java     |  89 +++++++
 .../mapreduce/MapReducePipelineResult.java      |  67 +++++
 .../runners/mapreduce/MapReduceRegistrar.java   |  55 ++++
 .../beam/runners/mapreduce/MapReduceRunner.java | 113 ++++++++
 .../beam/runners/mapreduce/package-info.java    |  21 ++
 .../mapreduce/translation/BeamInputFormat.java  | 257 ++++++++++++++++++
 .../mapreduce/translation/BeamMapper.java       |  71 +++++
 .../mapreduce/translation/BeamReducer.java      | 104 ++++++++
 .../translation/ConfigurationUtils.java         |  62 +++++
 .../mapreduce/translation/DotfileWriter.java    |  93 +++++++
 .../translation/FileReadOperation.java          | 181 +++++++++++++
 .../translation/FileSideInputReader.java        | 122 +++++++++
 .../translation/FileWriteOperation.java         |  73 +++++
 .../mapreduce/translation/FlattenOperation.java |  42 +++
 .../translation/FlattenTranslator.java          | 154 +++++++++++
 .../runners/mapreduce/translation/Graph.java    | 218 +++++++++++++++
 .../mapreduce/translation/GraphConverter.java   | 178 +++++++++++++
 .../mapreduce/translation/GraphPlanner.java     | 144 ++++++++++
 .../runners/mapreduce/translation/Graphs.java   | 267 +++++++++++++++++++
 .../GroupAlsoByWindowsParDoOperation.java       |  56 ++++
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  | 142 ++++++++++
 .../translation/GroupByKeyOperation.java        |  54 ++++
 .../translation/GroupByKeyTranslator.java       |  45 ++++
 .../mapreduce/translation/JobPrototype.java     | 211 +++++++++++++++
 .../translation/MapReduceMetricResults.java     | 106 ++++++++
 .../mapreduce/translation/MetricsReporter.java  |  97 +++++++
 .../translation/NormalParDoOperation.java       |  51 ++++
 .../mapreduce/translation/Operation.java        | 113 ++++++++
 .../mapreduce/translation/OutputReceiver.java   |  60 +++++
 .../mapreduce/translation/ParDoOperation.java   | 209 +++++++++++++++
 .../mapreduce/translation/ParDoTranslator.java  |  46 ++++
 .../translation/PartitionOperation.java         |  64 +++++
 .../translation/ReadBoundedTranslator.java      |  38 +++
 .../mapreduce/translation/ReadOperation.java    |  58 ++++
 .../ReifyTimestampAndWindowsParDoOperation.java |  64 +++++
 .../translation/SerializableConfiguration.java  |  52 ++++
 .../translation/SerializedPipelineOptions.java  |  76 ++++++
 .../translation/ShuffleWriteOperation.java      |  61 +++++
 .../translation/SourceReadOperation.java        |  44 +++
 .../translation/TransformTranslator.java        |  48 ++++
 .../translation/TranslationContext.java         | 199 ++++++++++++++
 .../translation/TranslatorRegistry.java         |  49 ++++
 .../mapreduce/translation/ViewTranslator.java   |  51 ++++
 .../translation/WindowAssignOperation.java      |  88 ++++++
 .../translation/WindowAssignTranslator.java     |  37 +++
 .../mapreduce/translation/package-info.java     |  22 ++
 .../translation/GraphConverterTest.java         |  60 +++++
 .../mapreduce/translation/GraphPlannerTest.java |  64 +++++
 runners/pom.xml                                 |   1 +
 sdks/java/pom.xml                               |   3 +-
 sdks/pom.xml                                    |   2 +-
 54 files changed, 4783 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/beam/blob/5fa0b14d/pom.xml
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/beam/blob/5fa0b14d/runners/pom.xml
----------------------------------------------------------------------


Mime
View raw message