beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From k...@apache.org
Subject [50/50] incubator-beam git commit: This closes #1663: Merge master (b3de17b) into gearpump-runner
Date Wed, 21 Dec 2016 20:23:58 GMT
This closes #1663: Merge master (b3de17b) into gearpump-runner

Adjustments in gearpump-runner:

  [BEAM-79] Upgrade to beam-0.5.0-incubating-SNAPSHOT
  [BEAM-79] Update to latest Gearpump API

>From master:

  Disable automatic archiving of Maven builds
  [BEAM-59] initial interfaces and classes of Beam FileSystem.
  Change counter name in TestDataflowRunner
  More escaping in Jenkins timestamp spec
  Add RunnableOnService test for Metrics
  Fix seed job fetch spec
  Show timestamps on log lines in Jenkins
  [BEAM-1165] Fix unexpected file creation when checking dependencies
  [BEAM-1178] Make naming of logger objects consistent
  [BEAM-716] Fix javadoc on with* methods [BEAM-959] Improve check preconditions in JmsIO
  [BEAM-716] Use AutoValue in JmsIO
  Fix grammar error (repeated for)
  Empty TestPipeline need not be run
  [BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage
  Change counter name in TestDataflowRunner
  BigQueryIO: fix streaming write, typo in API
  [BEAM-853] Force streaming execution on batch pipelines for testing. Expose the adapted
source.
  Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest
  Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle
  Add timer support to DoFnRunner(s)
  Make TimerSpec and StateSpec fields accessible
  View.asMap: minor javadoc fixes
  Revert "Move InMemoryTimerInternals to runners-core"
  Revert "Moves DoFnAdapters to runners-core"
  Revert "Removes ArgumentProvider.windowingInternals"
  Revert "Removes code for wrapping DoFn as an OldDoFn"
  checkstyle: missed newline in DistributionCell
  Make {Metric,Counter,Distribution}Cell public
  Add PTransformOverrideFactory to the Core SDK
  Move ActiveWindowSet and implementations to runners-core
  Update Dataflow worker to beam-master-20161216
  [BEAM-1108] Remove outdated language about experimental autoscaling
  [BEAM-450] Shade modules to separate paths
  [BEAM-362] Port runners to runners-core AggregatoryFactory
  Move InMemoryTimerInternals to runners-core
  Delete deprecated TimerCallback
  Remove deprecated methods of InMemoryTimerInternals
  Don't incorrectly log error in MetricsEnvironment
  Renames ParDo.getNewFn to getFn
  Moves DoFnAdapters to runners-core
  Removes unused code from NoOpOldDoFn
  Removes ArgumentProvider.windowingInternals
  Removes code for wrapping DoFn as an OldDoFn
  Removes OldDoFn from ParDo
  Pushes uses of OldDoFn deeper inside Flink runner
  Remove ParDo.of(OldDoFn) from Apex runner
  Converts all easy OldDoFns to DoFn
  [BEAM-1022] Add testing coverage for BigQuery streaming writes
  Fix mvn command args in Apex postcommit Jenkins job
  [BEAM-932] Enable findbugs validation (and fix existing issues)
  Fail to split in FileBasedSource if filePattern expands to empty.
  [BEAM-1154] Get side input from proper window in ReduceFn
  [BEAM-1153] GcsUtil: use non-batch API for single file size requests.
  Fix NPE in StatefulParDoEvaluatorFactoryTest mocking
  [BEAM-1033] Retry Bigquery Verifier when Query Fails
  Implement GetDefaultOutputCoder in DirectGroupByKey
  SimpleDoFnRunner observes window if SideInputReader is nonempty
  Better comments and cleanup
  Allow empty string value for ValueProvider types.
  starter: fix typo in pom.xml
  Revert "Allow stateful DoFn in DataflowRunner"
  Re-exclude UsesStatefulParDo tests for Dataflow
  Some minor changes and fixes for sorter module
  [BEAM-1149] Explode windows when fn uses side inputs
  Add Jenkins postcommit for RunnableOnService in Apex runner
  Update version from 0.5.0-SNAPSHOT to 0.5.0-incubating-SNAPSHOT
  Update Maven Archetype versions after cutting the release branch
  Move PerKeyCombineFnRunner to runners-core
  Update Dataflow worker to beam-master-20161212
  [maven-release-plugin] prepare for next development iteration
  [maven-release-plugin] prepare branch release-0.4.0-incubating
  Fix version of Kryo in examples/java jenkins-precommit profile
  Revert 91cc606 "This closes #1586": Kryo + UBRFBS
  [BEAM-909] improve starter archetype
  Fix JDom malformed comment in Apex runner.
  [BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO
  [BEAM-807] Replace OldDoFn with DoFn.
  [BEAM-757] Use DoFnRunner in the implementation of DoFn via FlatMapFunction.
  FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().
  FileBasedSink: ignore exceptions when removing temp output files for issues in Windows OS.
  [BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.
  Add Tests for Kryo Serialization of URFBS
  Add no-arg constructor for UnboundedReadFromBoundedSource
  Revise WindowedWordCount for runner and execution mode portability
  Factor out ShardedFile from FileChecksumMatcher
  Add IntervalWindow coder to the standard registry
  Stop expanding PValues in DirectRunner visitors
  Migrate AppliedPTransform to use AutoValue
  Enable and fix DirectRunnerTest case missing @Test
  [BEAM-1130] SparkRunner ResumeFromCheckpointStreamingTest Failing.
  [BEAM-1133] Add maxNumRecords per micro-batch for Spark runner options.
  BigQueryIO.Write: support runtime schema and table
  Fix handling of null ValueProviders in DisplayData
  [BEAM-551] Fix handling of default for VP
  [BEAM-1120] Move some DataflowRunner configurations from code to properties
  [BEAM-551] Fix toString for FileBasedSource
  [BEAM-921] spark-runner: register sources and coders to serialize with java serializer
  [BEAM-551] Fix handling of TextIO.Sink
  ...


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4c445dd0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4c445dd0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4c445dd0

Branch: refs/heads/gearpump-runner
Commit: 4c445dd0b6de0f5045c02579cb432da4fbc5d486
Parents: b6e7bb6 647034c
Author: Kenneth Knowles <klk@google.com>
Authored: Wed Dec 21 12:20:11 2016 -0800
Committer: Kenneth Knowles <klk@google.com>
Committed: Wed Dec 21 12:20:11 2016 -0800

----------------------------------------------------------------------
 .jenkins/common_job_properties.groovy           |   7 +
 ...ostCommit_Java_RunnableOnService_Apex.groovy |  41 ++
 README.md                                       |   2 +-
 examples/java/pom.xml                           |  20 +-
 .../apache/beam/examples/WindowedWordCount.java | 177 +++---
 .../org/apache/beam/examples/WordCount.java     |   2 +-
 .../examples/common/WriteWindowedFilesDoFn.java |  77 +++
 .../beam/examples/complete/AutoComplete.java    |   6 +-
 .../apache/beam/examples/complete/TfIdf.java    |   6 +-
 .../examples/complete/TopWikipediaSessions.java |  15 +-
 .../examples/complete/TrafficMaxLaneFlow.java   |   4 +-
 .../beam/examples/complete/TrafficRoutes.java   |   4 +-
 .../examples/cookbook/BigQueryTornadoes.java    |   2 +-
 .../cookbook/CombinePerKeyExamples.java         |   2 +-
 .../beam/examples/cookbook/FilterExamples.java  |   2 +-
 .../examples/cookbook/MaxPerKeyExamples.java    |   2 +-
 .../beam/examples/cookbook/TriggerExample.java  |   4 +-
 .../beam/examples/WindowedWordCountIT.java      | 182 +++++-
 .../examples/complete/AutoCompleteTest.java     |   2 +-
 examples/java8/pom.xml                          |   2 +-
 .../beam/examples/complete/game/GameStats.java  |   2 +-
 .../examples/complete/game/LeaderBoard.java     |   4 +-
 .../beam/examples/complete/game/UserScore.java  |   2 +-
 .../complete/game/utils/WriteToBigQuery.java    |   2 +-
 .../game/utils/WriteWindowedToBigQuery.java     |   7 +-
 examples/pom.xml                                |   2 +-
 pom.xml                                         |   5 +-
 runners/apex/pom.xml                            |  66 ++-
 .../apache/beam/runners/apex/ApexRunner.java    | 111 ++--
 .../beam/runners/apex/ApexRunnerResult.java     |  50 +-
 .../beam/runners/apex/ApexYarnLauncher.java     | 395 +++++++++++++
 .../translation/ApexPipelineTranslator.java     |   2 +
 .../translation/ParDoBoundMultiTranslator.java  |  22 +-
 .../apex/translation/ParDoBoundTranslator.java  |  22 +-
 .../apex/translation/TranslationContext.java    |   4 +-
 .../apex/translation/WindowBoundTranslator.java |  78 +++
 .../operators/ApexGroupByKeyOperator.java       |   4 +-
 .../operators/ApexParDoOperator.java            |  27 +-
 .../beam/runners/apex/ApexYarnLauncherTest.java | 138 +++++
 .../FlattenPCollectionTranslatorTest.java       |  15 +-
 .../translation/GroupByKeyTranslatorTest.java   |  21 +-
 .../translation/ParDoBoundTranslatorTest.java   |  38 +-
 .../translation/ReadUnboundTranslatorTest.java  |  15 +-
 runners/core-java/pom.xml                       |   6 +-
 .../beam/runners/core/ActiveWindowSet.java      | 173 ++++++
 .../apache/beam/runners/core/AssignWindows.java |  46 --
 .../apache/beam/runners/core/DoFnRunner.java    |  24 +-
 .../apache/beam/runners/core/DoFnRunners.java   |   2 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  12 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |  48 +-
 .../core/GroupByKeyViaGroupByKeyOnly.java       |   8 +-
 .../apache/beam/runners/core/KeyedWorkItem.java |  44 ++
 .../beam/runners/core/KeyedWorkItemCoder.java   | 130 +++++
 .../beam/runners/core/KeyedWorkItems.java       | 122 ++++
 .../core/LateDataDroppingDoFnRunner.java        |  34 +-
 .../runners/core/MergingActiveWindowSet.java    | 428 ++++++++++++++
 .../runners/core/NonMergingActiveWindowSet.java |  91 +++
 .../beam/runners/core/PaneInfoTracker.java      |   4 +
 .../runners/core/PerKeyCombineFnRunner.java     | 149 +++++
 .../runners/core/PerKeyCombineFnRunners.java    |   1 -
 .../core/PushbackSideInputDoFnRunner.java       |  31 +-
 .../runners/core/ReduceFnContextFactory.java    |  26 +-
 .../beam/runners/core/ReduceFnRunner.java       | 496 ++++++++++------
 .../beam/runners/core/SimpleDoFnRunner.java     | 252 ++++++++-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   9 +-
 .../beam/runners/core/SplittableParDo.java      |  28 +-
 .../core/UnboundedReadFromBoundedSource.java    |   2 +-
 .../apache/beam/runners/core/WatermarkHold.java |   5 +
 .../TriggerStateMachineContextFactory.java      |   2 +-
 .../triggers/TriggerStateMachineRunner.java     |  14 +-
 .../runners/core/KeyedWorkItemCoderTest.java    |  64 +++
 .../core/LateDataDroppingDoFnRunnerTest.java    |   3 +
 .../core/MergingActiveWindowSetTest.java        | 262 +++++++++
 .../core/PushbackSideInputDoFnRunnerTest.java   |  55 +-
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++---
 .../beam/runners/core/ReduceFnTester.java       |  77 ++-
 .../beam/runners/core/SimpleDoFnRunnerTest.java | 301 ++++++++++
 .../beam/runners/core/SplittableParDoTest.java  |   2 -
 .../triggers/TriggerStateMachineTester.java     |  25 +-
 runners/direct-java/pom.xml                     |   3 +-
 .../runners/direct/AggregatorContainer.java     |   2 +-
 .../CopyOnAccessInMemoryStateInternals.java     | 467 +++++++++++++++
 .../runners/direct/DirectExecutionContext.java  |   1 -
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |   9 +-
 .../beam/runners/direct/DirectGraphVisitor.java |  23 +-
 .../beam/runners/direct/DirectGroupByKey.java   |  46 +-
 .../direct/DirectGroupByKeyOverrideFactory.java |   3 +-
 .../beam/runners/direct/DirectRunner.java       |   7 +-
 .../beam/runners/direct/EvaluationContext.java  |   1 -
 .../direct/ExecutorServiceParallelExecutor.java |   4 +-
 .../runners/direct/ForwardingPTransform.java    |   6 +-
 .../GroupAlsoByWindowEvaluatorFactory.java      |   8 +-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   4 +-
 .../direct/KeyedPValueTrackingVisitor.java      |   4 +-
 .../direct/PTransformOverrideFactory.java       |  35 --
 .../beam/runners/direct/ParDoEvaluator.java     |   1 -
 .../runners/direct/ParDoEvaluatorFactory.java   |   2 +-
 .../direct/ParDoMultiOverrideFactory.java       |  29 +-
 .../ParDoSingleViaMultiOverrideFactory.java     |  17 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   4 +-
 .../runners/direct/StepTransformResult.java     |   1 -
 .../direct/TestStreamEvaluatorFactory.java      |   7 +-
 .../beam/runners/direct/TransformResult.java    |   1 -
 .../runners/direct/ViewEvaluatorFactory.java    |   8 +-
 .../direct/WriteWithShardingFactory.java        |  10 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java |  18 +-
 .../runners/direct/CommittedResultTest.java     |   2 +-
 .../CopyOnAccessInMemoryStateInternalsTest.java | 562 +++++++++++++++++++
 .../runners/direct/DirectGraphVisitorTest.java  |  71 ++-
 .../beam/runners/direct/DirectGraphs.java       |  35 ++
 .../beam/runners/direct/DirectRunnerTest.java   |  36 --
 .../runners/direct/EvaluationContextTest.java   |  83 +--
 .../direct/FlattenEvaluatorFactoryTest.java     |  15 +-
 .../direct/ForwardingPTransformTest.java        |   4 +-
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |   6 +-
 .../GroupByKeyOnlyEvaluatorFactoryTest.java     |   7 +-
 .../ImmutabilityEnforcementFactoryTest.java     |   2 +-
 .../direct/KeyedPValueTrackingVisitorTest.java  |   4 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   3 +-
 .../StatefulParDoEvaluatorFactoryTest.java      |  11 +-
 .../runners/direct/StepTransformResultTest.java |   2 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |  14 +-
 .../runners/direct/TransformExecutorTest.java   |   9 +-
 .../UnboundedReadEvaluatorFactoryTest.java      |  24 +-
 .../direct/ViewEvaluatorFactoryTest.java        |   4 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   6 +-
 .../runners/direct/WatermarkManagerTest.java    | 237 ++++----
 .../direct/WriteWithShardingFactoryTest.java    |   4 +-
 runners/flink/examples/pom.xml                  |  14 +-
 .../beam/runners/flink/examples/TFIDF.java      |   6 +-
 .../beam/runners/flink/examples/WordCount.java  |   2 +-
 .../flink/examples/streaming/AutoComplete.java  |   6 +-
 runners/flink/pom.xml                           |   2 +-
 runners/flink/runner/pom.xml                    |  10 +-
 .../FlinkPipelineExecutionEnvironment.java      |  12 +
 .../runners/flink/FlinkPipelineOptions.java     |   5 +
 .../apache/beam/runners/flink/FlinkRunner.java  |  24 +-
 .../FlinkBatchPipelineTranslator.java           |   4 +-
 .../FlinkBatchTransformTranslators.java         |  59 +-
 .../FlinkStreamingPipelineTranslator.java       |   7 +-
 .../FlinkStreamingTransformTranslators.java     |  77 ++-
 .../functions/FlinkDoFnFunction.java            |  10 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   2 +-
 .../FlinkMergingPartialReduceFunction.java      |   2 +-
 .../functions/FlinkMergingReduceFunction.java   |   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  10 +-
 .../functions/FlinkPartialReduceFunction.java   |   2 +-
 .../functions/FlinkProcessContextBase.java      |  21 +-
 .../functions/FlinkReduceFunction.java          |   2 +-
 .../wrappers/streaming/DoFnOperator.java        |  46 +-
 .../streaming/SingletonKeyedWorkItem.java       |   2 +-
 .../streaming/SingletonKeyedWorkItemCoder.java  |   4 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |  14 +-
 .../wrappers/streaming/WorkItemKeySelector.java |   2 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   6 +-
 .../flink/streaming/DoFnOperatorTest.java       |  13 +-
 .../flink/streaming/GroupByNullKeyTest.java     |  18 +-
 .../streaming/TopWikipediaSessionsITCase.java   |  10 +-
 runners/gearpump/pom.xml                        |   2 +-
 .../beam/runners/gearpump/GearpumpRunner.java   |   2 +-
 .../gearpump/examples/StreamingWordCount.java   |  14 +-
 .../translators/ParDoBoundMultiTranslator.java  |   2 +-
 .../translators/ParDoBoundTranslator.java       |   2 +-
 .../translators/TranslationContext.java         |   3 +-
 .../translators/utils/DoFnRunnerFactory.java    |   2 +-
 .../utils/NoOpAggregatorFactory.java            |   2 +-
 runners/google-cloud-dataflow-java/pom.xml      |  12 +-
 .../beam/runners/dataflow/DataflowClient.java   | 140 +++++
 .../runners/dataflow/DataflowPipelineJob.java   |  31 +-
 .../dataflow/DataflowPipelineTranslator.java    |  15 +-
 .../beam/runners/dataflow/DataflowRunner.java   | 136 +++--
 .../runners/dataflow/DataflowRunnerInfo.java    |  92 +++
 .../dataflow/internal/AssignWindows.java        |  10 +-
 .../DataflowUnboundedReadFromBoundedSource.java |   2 +-
 .../DataflowPipelineWorkerPoolOptions.java      |  57 +-
 .../options/DataflowProfilingOptions.java       |   8 +-
 .../dataflow/testing/TestDataflowRunner.java    |  35 +-
 .../dataflow/util/DataflowTemplateJob.java      |   2 +-
 .../runners/dataflow/util/MonitoringUtil.java   |  22 +-
 .../beam/runners/dataflow/dataflow.properties   |  23 +
 .../dataflow/DataflowPipelineJobTest.java       |  49 +-
 .../DataflowPipelineTranslatorTest.java         |  61 +-
 .../dataflow/DataflowRunnerInfoTest.java        |  51 ++
 .../runners/dataflow/DataflowRunnerTest.java    | 285 ++++++----
 .../options/DataflowProfilingOptionsTest.java   |   6 +-
 .../testing/TestDataflowRunnerTest.java         |  52 +-
 .../transforms/DataflowGroupByKeyTest.java      |  18 +-
 .../dataflow/transforms/DataflowViewTest.java   |  20 +-
 .../dataflow/util/MonitoringUtilTest.java       |  21 +-
 runners/pom.xml                                 |   2 +-
 runners/spark/README.md                         |  59 +-
 runners/spark/pom.xml                           |  55 +-
 .../beam/runners/spark/EvaluationResult.java    |  67 ---
 .../runners/spark/SparkPipelineOptions.java     |  10 +
 .../beam/runners/spark/SparkPipelineResult.java | 193 +++++++
 .../apache/beam/runners/spark/SparkRunner.java  | 130 +++--
 .../beam/runners/spark/TestSparkRunner.java     |  91 ++-
 .../spark/aggregators/AccumulatorSingleton.java |   6 +-
 .../spark/aggregators/SparkAggregators.java     | 126 +++++
 .../metrics/WithNamedAggregatorsSupport.java    |   4 +-
 .../coders/BeamSparkRunnerRegistrator.java      |  60 +-
 .../beam/runners/spark/examples/WordCount.java  |  15 +-
 .../apache/beam/runners/spark/io/ConsoleIO.java |   2 +-
 .../beam/runners/spark/io/CreateStream.java     |   2 +-
 .../beam/runners/spark/io/SourceDStream.java    |  21 +-
 .../runners/spark/io/SparkUnboundedSource.java  |  17 +-
 .../beam/runners/spark/io/hadoop/HadoopIO.java  |   4 +-
 .../runners/spark/translation/DoFnFunction.java | 110 ++--
 .../spark/translation/EvaluationContext.java    | 131 +----
 .../translation/GroupCombineFunctions.java      |  23 +-
 .../spark/translation/MultiDoFnFunction.java    | 135 ++---
 .../spark/translation/SparkAssignWindowFn.java  |  69 +++
 .../spark/translation/SparkContextFactory.java  |   2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   | 214 +++++++
 .../spark/translation/SparkProcessContext.java  | 385 +++----------
 .../spark/translation/SparkRuntimeContext.java  |  62 +-
 .../translation/StorageLevelPTransform.java     |   2 +-
 .../spark/translation/TransformTranslator.java  |  67 +--
 .../spark/translation/TranslationUtils.java     |  33 ++
 .../spark/translation/WindowingHelpers.java     |   3 +-
 .../streaming/StreamingTransformTranslator.java |  45 +-
 .../util/SinglePrimitiveOutputPTransform.java   |   2 +-
 .../beam/runners/spark/ForceStreamingTest.java  | 123 ++++
 .../runners/spark/ProvidedSparkContextTest.java |   6 +-
 .../runners/spark/SparkPipelineStateTest.java   | 217 +++++++
 .../spark/aggregators/ClearAggregatorsRule.java |  37 ++
 .../metrics/sink/ClearAggregatorsRule.java      |  33 --
 .../metrics/sink/NamedAggregatorsTest.java      |   3 +-
 .../coders/BeamSparkRunnerRegistratorTest.java  |  57 ++
 .../beam/runners/spark/io/AvroPipelineTest.java |   2 +-
 .../beam/runners/spark/io/NumShardsTest.java    |   2 +-
 .../io/hadoop/HadoopFileFormatPipelineTest.java |   2 +-
 .../spark/translation/SideEffectsTest.java      |  59 --
 .../streaming/EmptyStreamAssertionTest.java     |   4 +
 .../streaming/KafkaStreamingTest.java           |  34 +-
 .../ResumeFromCheckpointStreamingTest.java      |  32 +-
 .../streaming/utils/PAssertStreaming.java       |  31 +-
 sdks/java/build-tools/pom.xml                   |   2 +-
 .../src/main/resources/beam/findbugs-filter.xml |  68 ++-
 sdks/java/core/pom.xml                          |   2 +-
 .../beam/sdk/AggregatorPipelineExtractor.java   |   3 +-
 .../beam/sdk/annotations/Experimental.java      |   5 +-
 .../apache/beam/sdk/coders/CoderRegistry.java   |   2 +
 .../java/org/apache/beam/sdk/io/AvroIO.java     |   4 +-
 .../sdk/io/BoundedReadFromUnboundedSource.java  |  16 +-
 .../org/apache/beam/sdk/io/CountingInput.java   |  52 +-
 .../org/apache/beam/sdk/io/CountingSource.java  |  17 +-
 .../org/apache/beam/sdk/io/FileBasedSink.java   |  77 ++-
 .../org/apache/beam/sdk/io/FileBasedSource.java |  12 +-
 .../java/org/apache/beam/sdk/io/FileSystem.java |  29 +
 .../apache/beam/sdk/io/FileSystemRegistrar.java |  49 ++
 .../org/apache/beam/sdk/io/FileSystems.java     | 155 +++++
 .../org/apache/beam/sdk/io/LocalFileSystem.java |  27 +
 .../beam/sdk/io/LocalFileSystemRegistrar.java   |  41 ++
 .../java/org/apache/beam/sdk/io/PubsubIO.java   |   4 +-
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |   6 +-
 .../beam/sdk/io/PubsubUnboundedSource.java      |  12 +-
 .../main/java/org/apache/beam/sdk/io/Read.java  |   4 +-
 .../java/org/apache/beam/sdk/io/TextIO.java     |   4 +-
 .../main/java/org/apache/beam/sdk/io/Write.java |   2 +-
 .../apache/beam/sdk/io/range/ByteKeyRange.java  |   4 +-
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  |   8 +-
 .../apache/beam/sdk/metrics/CounterCell.java    |  10 +-
 .../beam/sdk/metrics/DistributionCell.java      |  14 +-
 .../org/apache/beam/sdk/metrics/MetricCell.java |   2 +-
 .../org/apache/beam/sdk/metrics/MetricName.java |   3 +-
 .../org/apache/beam/sdk/metrics/Metrics.java    |   5 +-
 .../beam/sdk/metrics/MetricsEnvironment.java    |   8 +-
 .../sdk/options/PipelineOptionsFactory.java     | 110 +++-
 .../beam/sdk/options/StreamingOptions.java      |   2 +-
 .../apache/beam/sdk/options/ValueProvider.java  |  27 +-
 .../sdk/runners/PTransformOverrideFactory.java  |  41 ++
 .../apache/beam/sdk/runners/PipelineRunner.java |   2 +-
 .../beam/sdk/runners/TransformHierarchy.java    |  26 +-
 .../beam/sdk/testing/BigqueryMatcher.java       |  48 +-
 .../beam/sdk/testing/FileChecksumMatcher.java   | 114 +---
 .../apache/beam/sdk/testing/GatherAllPanes.java |   2 +-
 .../org/apache/beam/sdk/testing/PAssert.java    |  12 +-
 .../apache/beam/sdk/testing/TestPipeline.java   | 225 +++++++-
 .../org/apache/beam/sdk/testing/TestStream.java |   2 +-
 .../apache/beam/sdk/testing/UsesMetrics.java    |  24 +
 .../beam/sdk/testing/UsesTimersInParDo.java     |  25 +
 .../sdk/transforms/AggregatorRetriever.java     |   2 +-
 .../beam/sdk/transforms/AppliedPTransform.java  |  75 +--
 .../beam/sdk/transforms/ApproximateUnique.java  |   4 +-
 .../org/apache/beam/sdk/transforms/Combine.java |  14 +-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 .../org/apache/beam/sdk/transforms/Count.java   |   2 +-
 .../org/apache/beam/sdk/transforms/Create.java  |   4 +-
 .../apache/beam/sdk/transforms/Distinct.java    |   4 +-
 .../org/apache/beam/sdk/transforms/DoFn.java    |  22 +
 .../beam/sdk/transforms/DoFnAdapters.java       |  12 +
 .../apache/beam/sdk/transforms/DoFnTester.java  |  29 +-
 .../org/apache/beam/sdk/transforms/Filter.java  |   2 +-
 .../beam/sdk/transforms/FlatMapElements.java    |   2 +-
 .../org/apache/beam/sdk/transforms/Flatten.java |   4 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |   2 +-
 .../org/apache/beam/sdk/transforms/Keys.java    |   2 +-
 .../org/apache/beam/sdk/transforms/KvSwap.java  |   2 +-
 .../org/apache/beam/sdk/transforms/Latest.java  |   4 +-
 .../apache/beam/sdk/transforms/MapElements.java |   2 +-
 .../apache/beam/sdk/transforms/PTransform.java  |   4 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   | 191 +------
 .../apache/beam/sdk/transforms/Partition.java   |   2 +-
 .../org/apache/beam/sdk/transforms/Regex.java   |  14 +-
 .../org/apache/beam/sdk/transforms/Sample.java  |   2 +-
 .../org/apache/beam/sdk/transforms/Values.java  |   2 +-
 .../org/apache/beam/sdk/transforms/View.java    |  16 +-
 .../apache/beam/sdk/transforms/WithKeys.java    |   2 +-
 .../beam/sdk/transforms/WithTimestamps.java     |   2 +-
 .../sdk/transforms/display/DisplayData.java     |  22 +
 .../beam/sdk/transforms/join/CoGroupByKey.java  |   2 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java    |  11 +
 .../sdk/transforms/reflect/DoFnInvoker.java     |   8 +
 .../sdk/transforms/reflect/DoFnSignature.java   |  26 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |  96 +++-
 .../beam/sdk/transforms/windowing/Window.java   |   4 +-
 .../apache/beam/sdk/util/ActiveWindowSet.java   | 173 ------
 .../org/apache/beam/sdk/util/ApiSurface.java    |  52 +-
 .../apache/beam/sdk/util/ExecutionContext.java  |   8 +-
 .../beam/sdk/util/ExplicitShardedFile.java      | 120 ++++
 .../java/org/apache/beam/sdk/util/GcsUtil.java  | 107 +++-
 .../org/apache/beam/sdk/util/KeyedWorkItem.java |  43 --
 .../beam/sdk/util/KeyedWorkItemCoder.java       | 128 -----
 .../apache/beam/sdk/util/KeyedWorkItems.java    | 121 ----
 .../beam/sdk/util/MergingActiveWindowSet.java   | 428 --------------
 .../sdk/util/NonMergingActiveWindowSet.java     |  91 ---
 .../beam/sdk/util/NumberedShardedFile.java      | 220 ++++++++
 .../beam/sdk/util/PerKeyCombineFnRunner.java    | 148 -----
 .../org/apache/beam/sdk/util/PropertyNames.java |   2 +
 .../org/apache/beam/sdk/util/ReleaseInfo.java   |   4 -
 .../org/apache/beam/sdk/util/Reshuffle.java     |   2 +-
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 ++
 .../CopyOnAccessInMemoryStateInternals.java     | 453 ---------------
 .../sdk/util/state/InMemoryStateInternals.java  |  33 +-
 .../sdk/util/state/InMemoryTimerInternals.java  | 126 +++--
 .../beam/sdk/util/state/StateInternals.java     |   2 +-
 .../beam/sdk/util/state/TimerCallback.java      |  35 --
 .../org/apache/beam/sdk/values/PCollection.java |   4 +-
 .../beam/sdk/values/PCollectionTuple.java       |   2 +-
 .../org/apache/beam/sdk/values/TypedPValue.java |   2 +-
 .../sdk/AggregatorPipelineExtractorTest.java    |   8 +-
 .../java/org/apache/beam/sdk/PipelineTest.java  |   6 +-
 .../beam/sdk/coders/CoderRegistryTest.java      |   4 +-
 .../apache/beam/sdk/io/CountingInputTest.java   |  63 ++-
 .../apache/beam/sdk/io/CountingSourceTest.java  |  10 +
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |  11 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java |  26 +
 .../org/apache/beam/sdk/io/FileSystemsTest.java | 104 ++++
 .../sdk/io/LocalFileSystemRegistrarTest.java    |  44 ++
 .../java/org/apache/beam/sdk/io/TextIOTest.java |  19 +
 .../java/org/apache/beam/sdk/io/WriteTest.java  |   2 +-
 .../org/apache/beam/sdk/io/XmlSinkTest.java     |   4 +-
 .../apache/beam/sdk/metrics/MetricMatchers.java |  47 +-
 .../apache/beam/sdk/metrics/MetricsTest.java    |  63 ++-
 .../sdk/options/PipelineOptionsFactoryTest.java | 222 +++++++-
 .../beam/sdk/options/ValueProviderTest.java     |  21 +
 .../sdk/runners/TransformHierarchyTest.java     |  17 +-
 .../beam/sdk/runners/TransformTreeTest.java     |   4 +-
 .../beam/sdk/testing/BigqueryMatcherTest.java   |  82 +--
 .../sdk/testing/FileChecksumMatcherTest.java    |  77 ---
 .../beam/sdk/testing/TestPipelineTest.java      | 186 +++++-
 .../beam/sdk/transforms/DoFnTesterTest.java     |   2 +-
 .../sdk/transforms/FlatMapElementsTest.java     |   2 +-
 .../beam/sdk/transforms/GroupByKeyTest.java     |   4 +-
 .../beam/sdk/transforms/MapElementsTest.java    |   2 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |  72 ---
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++---
 .../beam/sdk/transforms/PTransformTest.java     |   2 +-
 .../beam/sdk/transforms/ParDoLifecycleTest.java |  28 +-
 .../apache/beam/sdk/transforms/ParDoTest.java   | 342 ++++++-----
 .../beam/sdk/transforms/SplittableDoFnTest.java |  58 +-
 .../apache/beam/sdk/transforms/ViewTest.java    |   4 +-
 .../display/DisplayDataEvaluatorTest.java       |   2 +-
 .../display/DisplayDataMatchersTest.java        |   6 +-
 .../sdk/transforms/display/DisplayDataTest.java |  90 ++-
 .../DoFnSignaturesSplittableDoFnTest.java       |   3 +-
 .../transforms/reflect/DoFnSignaturesTest.java  |  47 ++
 .../sdk/transforms/windowing/WindowingTest.java |   2 +-
 .../org/apache/beam/sdk/util/GcsUtilTest.java   |  85 ++-
 .../beam/sdk/util/KeyedWorkItemCoderTest.java   |  62 --
 .../sdk/util/MergingActiveWindowSetTest.java    | 262 ---------
 .../beam/sdk/util/NumberedShardedFileTest.java  | 181 ++++++
 .../apache/beam/sdk/util/StringUtilsTest.java   |   6 +-
 .../CopyOnAccessInMemoryStateInternalsTest.java | 552 ------------------
 .../util/state/InMemoryTimerInternalsTest.java  | 125 +++--
 .../org/apache/beam/sdk/values/PDoneTest.java   |   4 +-
 sdks/java/extensions/join-library/pom.xml       |   2 +-
 sdks/java/extensions/pom.xml                    |   2 +-
 sdks/java/extensions/sorter/README.md           |   2 +-
 sdks/java/extensions/sorter/pom.xml             |  10 +-
 .../sorter/BufferedExternalSorter.java          |   6 +-
 .../sdk/extensions/sorter/ExternalSorter.java   |  15 +-
 .../sdk/extensions/sorter/InMemorySorter.java   |  26 +-
 .../beam/sdk/extensions/sorter/SortValues.java  |   2 +-
 .../sorter/BufferedExternalSorterTest.java      |  16 +
 .../extensions/sorter/ExternalSorterTest.java   |  16 +
 .../extensions/sorter/InMemorySorterTest.java   |   8 +
 sdks/java/io/google-cloud-platform/pom.xml      |   8 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java    | 453 +++++++++++----
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 +++-
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -------
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java    |  28 +-
 .../io/gcp/bigtable/BigtableServiceImpl.java    |   6 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |   4 +-
 .../beam/sdk/io/gcp/storage/GcsFileSystem.java  |  34 ++
 .../io/gcp/storage/GcsFileSystemRegistrar.java  |  42 ++
 .../beam/sdk/io/gcp/storage/package-info.java   |  21 +
 .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java  |   2 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java     | 543 ++++++++++++++----
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 ++++-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --------
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 .../gcp/storage/GcsFileSystemRegistrarTest.java |  51 ++
 sdks/java/io/hdfs/pom.xml                       |   8 +-
 .../beam/sdk/io/hdfs/HadoopFileSystem.java      |  29 +
 .../sdk/io/hdfs/HadoopFileSystemRegistrar.java  |  42 ++
 .../io/hdfs/HadoopFileSystemRegistrarTest.java  |  52 ++
 sdks/java/io/jdbc/pom.xml                       |   2 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java     |   4 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java |   4 +-
 sdks/java/io/jms/pom.xml                        |  22 +-
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  | 342 +++++++----
 .../org/apache/beam/sdk/io/jms/JmsRecord.java   |   8 +-
 sdks/java/io/kafka/pom.xml                      |   2 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  20 +-
 sdks/java/io/kinesis/pom.xml                    |   2 +-
 sdks/java/io/mongodb/pom.xml                    |   2 +-
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java    |   4 +-
 .../apache/beam/sdk/io/mongodb/MongoDbIO.java   |  16 +-
 .../sdk/io/mongodb/MongoDBGridFSIOTest.java     |   8 +-
 .../beam/sdk/io/mongodb/MongoDbIOTest.java      |   8 +-
 sdks/java/io/pom.xml                            |   2 +-
 sdks/java/java8tests/pom.xml                    |   2 +-
 .../maven-archetypes/examples-java8/pom.xml     |  13 +-
 .../main/resources/archetype-resources/pom.xml  | 122 +++-
 sdks/java/maven-archetypes/examples/pom.xml     |  13 +-
 .../main/resources/archetype-resources/pom.xml  |  47 +-
 sdks/java/maven-archetypes/pom.xml              |   2 +-
 sdks/java/maven-archetypes/starter/pom.xml      |   9 +-
 .../main/resources/archetype-resources/pom.xml  |  16 +-
 .../resources/projects/basic/reference/pom.xml  |  16 +-
 sdks/java/pom.xml                               |   2 +-
 sdks/pom.xml                                    |   2 +-
 446 files changed, 13550 insertions(+), 7606 deletions(-)
----------------------------------------------------------------------



Mime
View raw message