Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9E1AB200C78 for ; Thu, 18 May 2017 22:08:09 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9CE6D160BD1; Thu, 18 May 2017 20:08:09 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 16F4B160BD8 for ; Thu, 18 May 2017 22:08:05 +0200 (CEST) Received: (qmail 45371 invoked by uid 500); 18 May 2017 20:08:04 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 43311 invoked by uid 99); 18 May 2017 20:08:03 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 May 2017 20:08:03 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 87B5BF4A57; Thu, 18 May 2017 20:08:02 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: kenn@apache.org To: commits@beam.apache.org Date: Thu, 18 May 2017 20:08:50 -0000 Message-Id: <01ac8f823a04435697c795b62559f54d@git.apache.org> In-Reply-To: <6a686ce935c6441392350bbbfd158514@git.apache.org> References: <6a686ce935c6441392350bbbfd158514@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [50/50] [abbrv] beam git commit: This closes #3172: Sync gearpump-runner with master archived-at: Thu, 18 May 2017 20:08:09 -0000 This closes #3172: Sync gearpump-runner with master Update gearpump-runner against master changes Use stable naming strategy for ByteBuddy invokers Translate a Pipeline in SdkComponents [TRIVIAL] runners-core: delete placeholder Fixes an accidentally found bug in SimpleDoFnRunner Removes OldDoFn and its kin from runners-core Bump Dataflow containers to 0512 Improve Pruning performed by the DirectRunnerApiSurfaceTest Adding support for subnetwork in Python Pipelineoptions Use built-in cmp python function in comparing datastore paths ApexRunner SDF support Fix documentation for the shard_template_name [BEAM-2299] Run maven install on Windows machine for build/test coverage on Windows Remove "Dataflow" from apache_beam __init__.py file Moving the data file for trigger tests to testing/data Fix GcsResourceIdTest in postcommits readAvros should't have proto Message upper bound Reduce Log Level of PubsubUnboundedSource [BEAM-2290] Fix issue where timestamps weren't set when using CompressedSource [BEAM-2279] Fix archetype breakages internal comments Fix shading of guava testlib Rename FileSystems.setDefaultConfigInWorkers [BEAM-2277] HadoopFileSystem: normalize implementation Mark FileSystem and related as Experimental [BEAM-2277] Add ResourceIdTester and test existing ResourceId implementations Remove '/' entirely from determining FileSystem scheme [BEAM-2279] Add HDFS support to Spark runner profiles in archetypes and examples [BEAM-2277] Fix URI_SCHEME_PATTERN in FileSystems BigtableIO should use AutoValue for read and write [BEAM-2153] Move connection management in JmsIO.write() to setup/teardown methods Mark More values methods Internal Rename filesink to filebasedsink Enable SerializableCoder to Serialize with Generic Types Remove unused test data Fix due to GBKO name change. Don't deploy jdk1.8-tests module Remove some internal details from the public API. Move assert_that, equal_to, is_empty to apache_beam.testing.util [BEAM-1345] Clearly delineate public api in apache_beam/typehints. [BEAM-1345] Mark apache_beam/internal as internal. [BEAM-1345] Annotate public members of pvalue. Add internal comments to metrics [BEAM-1340] Add __all__ tags to modules in package apache_beam/transforms [BEAM-2256] Add the last previous range filter Use a consistent calculation for GC Time fix lint error in fake_datastore.py Add __all__ tags to modules in package apache_beam/testing [BEAM-1340] Adds __all__ tags to classes in package apache_beam/io. [BEAM-1345] Clearly delineate public api in apache_beam/coders. [BEAM-1345] Clearly delineate public api in runners package. [BEAM-1345] Mark Pipeline as public. [BEAM-1345] Clearly delineate public API in apache_beam/options Mark internal modules in python datastoreio [BEAM-2260] Improve construction-time errors for Text and AvroIO [BEAM-2179] Archetype generate-sources.sh cleanup the existing sources before rsync [BEAM-1345] Mark windowed value as experimental Add internal usage only comments to util/ Remove protobuf and http-client dependency from runners/google-cloud-dataflow minor typo fix in comment Add support for local execution to PubsubIO using the google cloud emulator [BEAM-2150] Relax regex to support wildcard globbing for GCS bump time of precommits [BEAM-2244] Move details of Metrics to Runners Core Correct javadoc for mobile gaming examples Update SDK Coders to return the Empty List from getCoderArguments Skip generating empty jars for parent poms Fix a typo in TestDataflowRunnerTest Re-enable UsesTimersInParDo tests in Dataflow runner TestDataflowRunner: throw AssertionError only when assertion known failed Allow any throwable in PAssert to constitute adequate failure [BEAM-2242] Ensure that jars are shaded correctly by running the jar plugin before the shade plugin [BEAM-2240] Always augment exception with step name. Adds dependency on findbugs to examples/java Splits WriteBundles into windowed/unwindowed versions Simpler code for setting shard numbers on results in FileBasedSink Implement dynamic-sharding for windowed file outputs, and add an integration test. Renames FileBasedSink inner classes [BEAM-2250] remove experimental and internal things from pydoc [BEAM-2249] Correctly handle partial reads in AvroSource Use text output for first two mobile gaming examples Remove verifyDeterministic from StructuredCoder Update Coder Documentation Improve DirectRunner Javadoc [BEAM-2211] Delete deprecated NoopPathValidator Remove Timer.cancel() from user-facing API Remove Readme files. Renames some python classes and functions that were unnecessarily public. Mark PipelineVisitor and AppliedPTransform as internal. Mark PValue and PValueBase Internal [BEAM-2236] Move test utilities out of python core Include 'sun.reflect' in GcpCoreApiSurfaceTest Fix checkstyle error Shade dependencies in sdks/core Remove trailing whitespace Add per-runner profile to Java 8 examples Register TestSparkPipelineOptions only in src/test to avoid hard hamcrest dep Update Apache Beam Python version to 2.1.0.dev Shade JSR305 in the DirectRunner Remove hadoop io readme ... Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/99221e73 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/99221e73 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/99221e73 Branch: refs/heads/gearpump-runner Commit: 99221e73991002b8ed95cd5649fadee1d14bba4d Parents: 58546ac 6c06967 Author: Kenneth Knowles Authored: Thu May 18 13:07:02 2017 -0700 Committer: Kenneth Knowles Committed: Thu May 18 13:07:02 2017 -0700 ---------------------------------------------------------------------- .gitignore | 1 + .../jenkins/common_job_properties.groovy | 29 +- .../job_beam_PerformanceTests_Dataflow.groovy | 3 + ...job_beam_PostCommit_Java_MavenInstall.groovy | 8 +- ..._PostCommit_Java_MavenInstall_Windows.groovy | 45 ++ ...tCommit_Java_ValidatesRunner_Dataflow.groovy | 2 +- .../job_beam_PreCommit_Java_MavenInstall.groovy | 5 +- .../job_beam_Release_NightlySnapshot.groovy | 2 +- README.md | 4 +- examples/java/README.md | 64 +- examples/java/pom.xml | 53 +- .../apache/beam/examples/WindowedWordCount.java | 9 +- .../beam/examples/common/ExampleUtils.java | 6 +- .../examples/common/WriteOneFilePerWindow.java | 72 +- .../org/apache/beam/examples/complete/README.md | 2 +- .../apache/beam/examples/complete/TfIdf.java | 2 +- .../org/apache/beam/examples/cookbook/README.md | 2 +- .../beam/examples/WindowedWordCountIT.java | 26 +- .../beam/examples/complete/TfIdfTest.java | 2 +- examples/java8/pom.xml | 124 ++- .../beam/examples/complete/game/GameStats.java | 15 +- .../examples/complete/game/HourlyTeamScore.java | 58 +- .../examples/complete/game/LeaderBoard.java | 62 +- .../beam/examples/complete/game/README.md | 131 --- .../beam/examples/complete/game/UserScore.java | 74 +- .../complete/game/utils/WriteToBigQuery.java | 32 +- .../complete/game/utils/WriteToText.java | 184 +++++ .../game/utils/WriteWindowedToBigQuery.java | 9 +- examples/pom.xml | 2 +- pom.xml | 71 +- runners/apex/README.md | 76 -- runners/apex/pom.xml | 10 +- .../beam/runners/apex/ApexPipelineOptions.java | 5 + .../apache/beam/runners/apex/ApexRunner.java | 51 +- .../beam/runners/apex/ApexRunnerRegistrar.java | 3 +- .../beam/runners/apex/ApexRunnerResult.java | 2 - .../beam/runners/apex/ApexYarnLauncher.java | 2 - .../beam/runners/apex/TestApexRunner.java | 10 +- .../translation/ApexPipelineTranslator.java | 23 + .../apex/translation/ParDoTranslator.java | 68 +- .../translation/ReadUnboundedTranslator.java | 1 - .../apex/translation/TransformTranslator.java | 2 - .../apex/translation/TranslationContext.java | 97 ++- .../operators/ApexGroupByKeyOperator.java | 10 +- .../operators/ApexParDoOperator.java | 131 ++- .../operators/ApexProcessFnOperator.java | 8 +- .../ApexReadUnboundedInputOperator.java | 2 - .../operators/ApexTimerInternals.java | 23 +- .../translation/utils/ApexStateInternals.java | 24 +- .../apex/translation/utils/ApexStreamTuple.java | 16 +- .../utils/CoderAdapterStreamCodec.java | 2 - .../utils/SerializablePipelineOptions.java | 19 +- .../translation/utils/StateInternalsProxy.java | 11 +- .../utils/ValueAndCoderKryoSerializable.java | 2 - .../apex/translation/utils/ValuesSource.java | 2 - .../beam/runners/apex/ApexRunnerTest.java | 49 +- .../beam/runners/apex/ApexYarnLauncherTest.java | 2 - .../apex/examples/UnboundedTextSource.java | 2 - .../runners/apex/examples/WordCountTest.java | 2 - .../translation/ApexGroupByKeyOperatorTest.java | 4 +- .../FlattenPCollectionTranslatorTest.java | 13 +- .../apex/translation/ParDoTranslatorTest.java | 2 +- .../operators/ApexTimerInternalsTest.java | 10 +- .../utils/ApexStateInternalsTest.java | 12 +- .../translation/utils/CollectionSource.java | 2 - .../translation/utils/PipelineOptionsTest.java | 98 ++- .../test/resources/beam-runners-apex.properties | 2 +- runners/core-construction-java/pom.xml | 2 +- .../DeduplicatedFlattenFactory.java | 2 +- .../EmptyFlattenAsCreateFactory.java | 2 +- .../core/construction/ForwardingPTransform.java | 6 +- .../runners/core/construction/PCollections.java | 2 +- .../core/construction/PTransformMatchers.java | 2 +- .../construction/PTransformReplacements.java | 2 +- .../runners/core/construction/PTransforms.java | 19 +- .../core/construction/PrimitiveCreate.java | 4 +- .../core/construction/SdkComponents.java | 54 +- .../runners/core/construction/Triggers.java | 4 +- .../UnboundedReadFromBoundedSource.java | 16 +- .../UnsupportedOverrideFactory.java | 2 +- .../core/construction/WindowingStrategies.java | 4 +- .../runners/core/metrics/MetricFiltering.java | 102 +++ .../beam/runners/core/metrics/MetricKey.java | 43 + .../beam/runners/core/metrics/package-info.java | 22 + .../runners/core/construction/CodersTest.java | 18 +- .../EmptyFlattenAsCreateFactoryTest.java | 2 +- .../core/construction/PCollectionsTest.java | 17 +- .../construction/PTransformMatchersTest.java | 34 +- .../PTransformReplacementsTest.java | 2 +- .../core/construction/PTransformsTest.java | 2 +- .../construction/ReplacementOutputsTest.java | 2 +- .../core/construction/SdkComponentsTest.java | 106 ++- .../SingleInputOutputOverrideFactoryTest.java | 2 +- .../construction/WindowingStrategiesTest.java | 4 +- .../core/metrics/MetricFilteringTest.java | 148 ++++ runners/core-java/pom.xml | 2 +- .../beam/runners/core/AssignWindowsDoFn.java | 78 -- .../beam/runners/core/BaseExecutionContext.java | 13 +- .../apache/beam/runners/core/DoFnAdapters.java | 310 ------- .../apache/beam/runners/core/DoFnRunner.java | 2 +- .../apache/beam/runners/core/DoFnRunners.java | 52 +- .../core/ElementAndRestrictionCoder.java | 29 +- .../beam/runners/core/ExecutionContext.java | 10 +- .../runners/core/GlobalCombineFnRunner.java | 1 - .../runners/core/GlobalCombineFnRunners.java | 41 +- .../GroupAlsoByWindowViaOutputBufferDoFn.java | 113 --- .../core/GroupAlsoByWindowViaWindowSetDoFn.java | 94 --- .../GroupAlsoByWindowViaWindowSetNewDoFn.java | 9 +- .../core/GroupAlsoByWindowsAggregators.java | 26 + .../runners/core/GroupAlsoByWindowsDoFn.java | 39 - .../core/GroupByKeyViaGroupByKeyOnly.java | 2 +- .../runners/core/InMemoryStateInternals.java | 22 +- .../runners/core/InMemoryTimerInternals.java | 2 +- .../beam/runners/core/KeyedWorkItemCoder.java | 22 +- .../core/LateDataDroppingDoFnRunner.java | 9 +- .../apache/beam/runners/core/LateDataUtils.java | 37 +- .../runners/core/MergingActiveWindowSet.java | 2 +- .../beam/runners/core/MergingStateAccessor.java | 2 +- .../apache/beam/runners/core/NonEmptyPanes.java | 8 +- .../beam/runners/core/NullSideInputReader.java | 61 ++ .../org/apache/beam/runners/core/OldDoFn.java | 335 -------- ...eBoundedSplittableProcessElementInvoker.java | 18 +- .../beam/runners/core/PaneInfoTracker.java | 4 +- .../beam/runners/core/ProcessFnRunner.java | 3 +- .../core/PushbackSideInputDoFnRunner.java | 2 +- .../core/ReadyCheckingSideInputReader.java | 34 + .../org/apache/beam/runners/core/ReduceFn.java | 6 +- .../runners/core/ReduceFnContextFactory.java | 13 +- .../beam/runners/core/ReduceFnRunner.java | 50 +- .../beam/runners/core/SideInputHandler.java | 15 +- .../beam/runners/core/SideInputReader.java | 47 ++ .../beam/runners/core/SimpleDoFnRunner.java | 223 ++++- .../beam/runners/core/SimpleOldDoFnRunner.java | 500 ------------ .../core/SimplePushbackSideInputDoFnRunner.java | 3 +- .../beam/runners/core/SplittableParDo.java | 50 +- .../apache/beam/runners/core/StateAccessor.java | 2 +- .../beam/runners/core/StateInternals.java | 4 +- .../apache/beam/runners/core/StateMerging.java | 14 +- .../apache/beam/runners/core/StateTable.java | 4 +- .../org/apache/beam/runners/core/StateTag.java | 16 +- .../org/apache/beam/runners/core/StateTags.java | 20 +- .../beam/runners/core/StatefulDoFnRunner.java | 14 +- .../beam/runners/core/SystemReduceFn.java | 8 +- .../core/TestInMemoryStateInternals.java | 4 +- .../beam/runners/core/TimerInternals.java | 30 +- .../core/UnsupportedSideInputReader.java | 1 - .../apache/beam/runners/core/WatermarkHold.java | 15 +- .../core/WindowingInternalsAdapters.java | 75 -- .../beam/runners/core/metrics/CounterCell.java | 86 ++ .../beam/runners/core/metrics/DirtyState.java | 99 +++ .../runners/core/metrics/DistributionCell.java | 80 ++ .../runners/core/metrics/DistributionData.java | 62 ++ .../beam/runners/core/metrics/GaugeCell.java | 78 ++ .../beam/runners/core/metrics/GaugeData.java | 83 ++ .../beam/runners/core/metrics/MetricCell.java | 42 + .../runners/core/metrics/MetricUpdates.java | 79 ++ .../core/metrics/MetricsContainerImpl.java | 188 +++++ .../core/metrics/MetricsContainerStepMap.java | 495 ++++++++++++ .../beam/runners/core/metrics/MetricsMap.java | 88 ++ .../beam/runners/core/metrics/package-info.java | 22 + .../AfterDelayFromFirstElementStateMachine.java | 6 +- .../core/triggers/AfterPaneStateMachine.java | 2 +- .../AfterProcessingTimeStateMachine.java | 2 +- ...rSynchronizedProcessingTimeStateMachine.java | 2 +- .../triggers/AfterWatermarkStateMachine.java | 2 +- .../triggers/DefaultTriggerStateMachine.java | 2 +- .../triggers/ReshuffleTriggerStateMachine.java | 2 +- .../core/triggers/TriggerStateMachine.java | 2 +- .../TriggerStateMachineContextFactory.java | 6 +- .../triggers/TriggerStateMachineRunner.java | 4 +- runners/core-java/src/test/java/.placeholder | 0 ...roupAlsoByWindowViaOutputBufferDoFnTest.java | 109 --- .../core/GroupAlsoByWindowsProperties.java | 744 ----------------- .../core/InMemoryStateInternalsTest.java | 16 +- .../core/InMemoryTimerInternalsTest.java | 2 +- .../runners/core/KeyedWorkItemCoderTest.java | 2 +- .../core/LateDataDroppingDoFnRunnerTest.java | 11 +- .../beam/runners/core/LateDataUtilsTest.java | 90 +++ .../apache/beam/runners/core/NoOpOldDoFn.java | 65 -- .../apache/beam/runners/core/OldDoFnTest.java | 51 -- ...ndedSplittableProcessElementInvokerTest.java | 1 - .../beam/runners/core/ReduceFnRunnerTest.java | 41 +- .../beam/runners/core/ReduceFnTester.java | 8 +- .../beam/runners/core/SideInputHandlerTest.java | 2 +- .../beam/runners/core/SimpleDoFnRunnerTest.java | 17 +- .../runners/core/SimpleOldDoFnRunnerTest.java | 86 -- .../SimplePushbackSideInputDoFnRunnerTest.java | 3 +- .../beam/runners/core/SplittableParDoTest.java | 7 +- .../runners/core/StatefulDoFnRunnerTest.java | 19 +- .../beam/runners/core/TimerInternalsTest.java | 2 +- .../runners/core/metrics/CounterCellTest.java | 56 ++ .../runners/core/metrics/DirtyStateTest.java | 56 ++ .../core/metrics/DistributionCellTest.java | 54 ++ .../runners/core/metrics/GaugeCellTest.java | 51 ++ .../core/metrics/MetricUpdateMatchers.java | 82 ++ .../core/metrics/MetricsContainerImplTest.java | 130 +++ .../metrics/MetricsContainerStepMapTest.java | 272 +++++++ .../runners/core/metrics/MetricsMapTest.java | 103 +++ .../triggers/TriggerStateMachineTester.java | 6 +- .../core/triggers/TriggerStateMachinesTest.java | 2 +- runners/direct-java/pom.xml | 100 ++- .../direct/AbstractModelEnforcement.java | 1 - .../direct/BoundedReadEvaluatorFactory.java | 17 +- .../beam/runners/direct/BundleFactory.java | 3 +- .../org/apache/beam/runners/direct/Clock.java | 2 +- .../runners/direct/CloningBundleFactory.java | 2 - .../beam/runners/direct/CommittedBundle.java | 82 ++ .../beam/runners/direct/CommittedResult.java | 3 +- .../beam/runners/direct/CompletionCallback.java | 9 +- .../CopyOnAccessInMemoryStateInternals.java | 20 +- ...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 2 +- .../apache/beam/runners/direct/DirectGraph.java | 14 +- .../beam/runners/direct/DirectGraphVisitor.java | 2 +- .../beam/runners/direct/DirectGroupByKey.java | 2 +- .../direct/DirectGroupByKeyOverrideFactory.java | 2 +- .../beam/runners/direct/DirectMetrics.java | 25 +- .../beam/runners/direct/DirectOptions.java | 2 +- .../beam/runners/direct/DirectRegistrar.java | 12 +- .../beam/runners/direct/DirectRunner.java | 158 +--- .../runners/direct/DirectTimerInternals.java | 2 +- .../runners/direct/DisplayDataValidator.java | 6 +- .../runners/direct/DoFnLifecycleManager.java | 2 +- .../beam/runners/direct/EmptyInputProvider.java | 3 +- .../runners/direct/EmptyTransformEvaluator.java | 50 -- .../beam/runners/direct/EvaluationContext.java | 14 +- .../runners/direct/ExecutorServiceFactory.java | 2 +- .../direct/ExecutorServiceParallelExecutor.java | 42 +- .../runners/direct/FlattenEvaluatorFactory.java | 4 +- .../GroupAlsoByWindowEvaluatorFactory.java | 18 +- .../direct/GroupByKeyOnlyEvaluatorFactory.java | 4 +- .../ImmutabilityCheckingBundleFactory.java | 2 - .../direct/ImmutabilityEnforcementFactory.java | 3 +- .../direct/ImmutableListBundleFactory.java | 2 - .../beam/runners/direct/ModelEnforcement.java | 12 +- .../runners/direct/ModelEnforcementFactory.java | 5 +- .../beam/runners/direct/NanosOffsetClock.java | 2 +- .../runners/direct/PCollectionViewWindow.java | 67 ++ .../runners/direct/PCollectionViewWriter.java | 34 + .../beam/runners/direct/ParDoEvaluator.java | 7 +- .../runners/direct/ParDoEvaluatorFactory.java | 3 +- .../direct/ParDoMultiOverrideFactory.java | 7 +- .../direct/PassthroughTransformEvaluator.java | 3 +- .../beam/runners/direct/PipelineExecutor.java | 3 +- .../beam/runners/direct/RootInputProvider.java | 3 +- .../runners/direct/RootProviderRegistry.java | 3 +- .../beam/runners/direct/SideInputContainer.java | 7 +- ...littableProcessElementsEvaluatorFactory.java | 18 +- .../direct/StatefulParDoEvaluatorFactory.java | 7 +- .../apache/beam/runners/direct/StepAndKey.java | 2 +- .../runners/direct/StepTransformResult.java | 7 +- .../direct/TestStreamEvaluatorFactory.java | 6 +- .../beam/runners/direct/TransformEvaluator.java | 3 +- .../direct/TransformEvaluatorFactory.java | 8 +- .../direct/TransformEvaluatorRegistry.java | 3 +- .../beam/runners/direct/TransformExecutor.java | 25 +- .../beam/runners/direct/TransformResult.java | 7 +- .../direct/UnboundedReadEvaluatorFactory.java | 4 +- .../beam/runners/direct/UncommittedBundle.java | 57 ++ .../runners/direct/ViewEvaluatorFactory.java | 5 +- .../runners/direct/ViewOverrideFactory.java | 2 +- .../direct/WatermarkCallbackExecutor.java | 4 +- .../beam/runners/direct/WatermarkManager.java | 7 +- .../runners/direct/WindowEvaluatorFactory.java | 4 +- .../direct/WriteWithShardingFactory.java | 2 +- .../direct/BoundedReadEvaluatorFactoryTest.java | 4 +- .../direct/CloningBundleFactoryTest.java | 32 +- .../runners/direct/CommittedResultTest.java | 16 +- .../CopyOnAccessInMemoryStateInternalsTest.java | 16 +- .../runners/direct/DirectGraphVisitorTest.java | 2 +- .../beam/runners/direct/DirectGraphs.java | 2 +- .../DirectGroupByKeyOverrideFactoryTest.java | 2 +- .../beam/runners/direct/DirectMetricsTest.java | 15 +- .../direct/DirectRunnerApiSurfaceTest.java | 68 ++ .../beam/runners/direct/DirectRunnerTest.java | 11 +- .../direct/DirectTimerInternalsTest.java | 2 +- ...leManagerRemovingTransformEvaluatorTest.java | 2 +- .../runners/direct/EvaluationContextTest.java | 13 +- .../direct/FlattenEvaluatorFactoryTest.java | 4 +- .../direct/GroupByKeyEvaluatorFactoryTest.java | 2 - .../GroupByKeyOnlyEvaluatorFactoryTest.java | 2 - .../ImmutabilityCheckingBundleFactoryTest.java | 2 - .../ImmutabilityEnforcementFactoryTest.java | 3 +- .../direct/ImmutableListBundleFactoryTest.java | 2 - .../direct/KeyedPValueTrackingVisitorTest.java | 2 +- .../apache/beam/runners/direct/MockClock.java | 2 +- .../beam/runners/direct/ParDoEvaluatorTest.java | 5 +- .../runners/direct/SideInputContainerTest.java | 8 +- .../StatefulParDoEvaluatorFactoryTest.java | 14 +- .../runners/direct/StepTransformResultTest.java | 3 +- .../direct/TestStreamEvaluatorFactoryTest.java | 3 +- .../runners/direct/TransformExecutorTest.java | 8 +- .../UnboundedReadEvaluatorFactoryTest.java | 13 +- .../direct/ViewEvaluatorFactoryTest.java | 6 +- .../runners/direct/ViewOverrideFactoryTest.java | 6 +- .../direct/WatermarkCallbackExecutorTest.java | 4 +- .../runners/direct/WatermarkManagerTest.java | 6 +- .../direct/WindowEvaluatorFactoryTest.java | 2 - .../direct/WriteWithShardingFactoryTest.java | 61 +- runners/flink/pom.xml | 7 +- .../flink/FlinkBatchTransformTranslators.java | 4 +- .../flink/FlinkBatchTranslationContext.java | 4 +- .../flink/FlinkDetachedRunnerResult.java | 1 - .../apache/beam/runners/flink/FlinkRunner.java | 5 +- .../runners/flink/FlinkRunnerRegistrar.java | 2 +- .../beam/runners/flink/FlinkRunnerResult.java | 8 +- .../flink/FlinkStreamingPipelineTranslator.java | 2 +- .../FlinkStreamingTransformTranslators.java | 84 +- .../flink/FlinkStreamingTranslationContext.java | 2 +- .../flink/FlinkStreamingViewOverrides.java | 2 +- .../beam/runners/flink/TestFlinkRunner.java | 2 +- .../metrics/DoFnRunnerWithMetricsUpdate.java | 17 +- .../flink/metrics/FlinkMetricContainer.java | 276 ++----- .../flink/metrics/FlinkMetricResults.java | 146 ---- .../flink/metrics/MetricsAccumulator.java | 60 ++ .../flink/metrics/ReaderInvocationUtil.java | 12 +- .../functions/AbstractFlinkCombineRunner.java | 4 +- .../functions/FlinkDoFnFunction.java | 3 +- .../FlinkMergingNonShuffleReduceFunction.java | 2 +- .../functions/FlinkPartialReduceFunction.java | 2 +- .../functions/FlinkReduceFunction.java | 2 +- .../functions/FlinkSideInputReader.java | 4 +- .../functions/FlinkStatefulDoFnFunction.java | 3 +- .../functions/HashingFlinkCombineRunner.java | 4 +- .../functions/SortingFlinkCombineRunner.java | 4 +- .../translation/types/CoderTypeSerializer.java | 4 +- .../types/EncodedValueSerializer.java | 2 - .../utils/SerializedPipelineOptions.java | 20 +- .../translation/wrappers/SourceInputFormat.java | 8 +- .../wrappers/streaming/DoFnOperator.java | 58 +- .../streaming/SingletonKeyedWorkItemCoder.java | 20 +- .../streaming/SplittableDoFnOperator.java | 30 +- .../wrappers/streaming/WindowDoFnOperator.java | 2 +- .../streaming/io/BoundedSourceWrapper.java | 8 +- .../wrappers/streaming/io/DedupingOperator.java | 187 +++++ .../streaming/io/UnboundedSourceWrapper.java | 24 +- .../state/FlinkBroadcastStateInternals.java | 20 +- .../state/FlinkKeyGroupStateInternals.java | 29 +- .../state/FlinkSplitStateInternals.java | 20 +- .../streaming/state/FlinkStateInternals.java | 22 +- .../beam/runners/flink/PipelineOptionsTest.java | 89 +- .../flink/streaming/DedupingOperatorTest.java | 131 +++ .../flink/streaming/DoFnOperatorTest.java | 16 +- .../FlinkBroadcastStateInternalsTest.java | 10 +- .../FlinkKeyGroupStateInternalsTest.java | 4 +- .../streaming/FlinkSplitStateInternalsTest.java | 4 +- .../streaming/FlinkStateInternalsTest.java | 12 +- .../streaming/UnboundedSourceWrapperTest.java | 29 +- runners/gearpump/pom.xml | 2 +- .../gearpump/GearpumpPipelineTranslator.java | 4 +- .../beam/runners/gearpump/GearpumpRunner.java | 2 +- .../gearpump/GearpumpRunnerRegistrar.java | 2 +- .../runners/gearpump/TestGearpumpRunner.java | 2 +- .../ParDoSingleOutputTranslator.java | 2 +- .../translators/TranslationContext.java | 2 +- .../translators/WindowAssignTranslator.java | 2 +- .../translators/functions/DoFnFunction.java | 2 +- .../translators/utils/DoFnRunnerFactory.java | 4 +- runners/google-cloud-dataflow-java/pom.xml | 76 +- .../beam/runners/dataflow/AssignWindows.java | 2 +- .../dataflow/BatchStatefulParDoOverrides.java | 4 +- .../runners/dataflow/BatchViewOverrides.java | 32 +- .../beam/runners/dataflow/DataflowMetrics.java | 4 +- .../runners/dataflow/DataflowPipelineJob.java | 20 +- .../dataflow/DataflowPipelineRegistrar.java | 4 +- .../dataflow/DataflowPipelineTranslator.java | 40 +- .../beam/runners/dataflow/DataflowRunner.java | 110 +-- .../dataflow/PrimitiveParDoSingleFactory.java | 2 +- .../beam/runners/dataflow/ReadTranslator.java | 22 +- .../dataflow/ReshuffleOverrideFactory.java | 8 +- .../dataflow/StreamingViewOverrides.java | 2 +- .../dataflow/TestDataflowPipelineOptions.java | 28 + .../runners/dataflow/TestDataflowRunner.java | 388 +++++++++ .../runners/dataflow/TransformTranslator.java | 2 +- .../dataflow/internal/CustomSources.java | 18 +- .../runners/dataflow/internal/IsmFormat.java | 99 ++- .../options/DataflowPipelineOptions.java | 18 +- .../testing/TestDataflowPipelineOptions.java | 28 - .../dataflow/testing/TestDataflowRunner.java | 326 -------- .../runners/dataflow/testing/package-info.java | 24 - .../util/AvroCoderCloudObjectTranslator.java | 2 - .../beam/runners/dataflow/util/CloudObject.java | 1 - .../runners/dataflow/util/CloudObjectKinds.java | 2 - .../dataflow/util/CloudObjectTranslator.java | 2 - .../dataflow/util/CloudObjectTranslators.java | 23 +- .../runners/dataflow/util/CloudObjects.java | 1 - .../CoderCloudObjectTranslatorRegistrar.java | 1 - .../dataflow/util/DataflowTemplateJob.java | 1 - ...aultCoderCloudObjectTranslatorRegistrar.java | 2 - .../beam/runners/dataflow/util/DoFnInfo.java | 2 +- .../beam/runners/dataflow/util/GcsStager.java | 26 +- .../beam/runners/dataflow/util/PackageUtil.java | 41 +- .../runners/dataflow/util/PropertyNames.java | 112 +++ .../runners/dataflow/util/RandomAccessData.java | 26 +- .../SerializableCoderCloudObjectTranslator.java | 2 - .../beam/runners/dataflow/util/Serializer.java | 262 ------ .../beam/runners/dataflow/util/Structs.java | 372 +++++++++ .../BatchStatefulParDoOverridesTest.java | 6 +- .../runners/dataflow/DataflowMetricsTest.java | 7 +- .../dataflow/DataflowPipelineJobTest.java | 18 +- .../dataflow/DataflowPipelineRegistrarTest.java | 2 +- .../DataflowPipelineTranslatorTest.java | 133 ++- .../runners/dataflow/DataflowRunnerTest.java | 90 +-- .../PrimitiveParDoSingleFactoryTest.java | 2 +- .../dataflow/TestDataflowRunnerTest.java | 634 +++++++++++++++ .../options/DataflowPipelineOptionsTest.java | 21 +- .../options/DataflowProfilingOptionsTest.java | 4 +- .../DataflowWorkerLoggingOptionsTest.java | 4 +- .../testing/TestDataflowRunnerTest.java | 661 --------------- .../transforms/DataflowGroupByKeyTest.java | 4 +- .../dataflow/transforms/DataflowViewTest.java | 4 +- .../runners/dataflow/util/CloudObjectsTest.java | 9 +- .../runners/dataflow/util/PackageUtilTest.java | 145 ++-- .../dataflow/util/RandomAccessDataTest.java | 6 +- .../beam/runners/dataflow/util/StructsTest.java | 206 +++++ runners/pom.xml | 2 +- runners/spark/README.md | 124 --- runners/spark/pom.xml | 54 +- .../beam/runners/spark/SparkPipelineResult.java | 8 +- .../apache/beam/runners/spark/SparkRunner.java | 8 +- .../beam/runners/spark/SparkRunnerDebugger.java | 7 +- .../runners/spark/SparkRunnerRegistrar.java | 2 +- .../beam/runners/spark/TestSparkRunner.java | 83 +- .../spark/aggregators/NamedAggregators.java | 6 +- .../beam/runners/spark/coders/CoderHelpers.java | 1 - .../spark/coders/StatelessJavaSerializer.java | 1 - .../beam/runners/spark/io/CreateStream.java | 2 +- .../beam/runners/spark/io/MicrobatchSource.java | 2 +- .../apache/beam/runners/spark/io/SourceRDD.java | 177 ++-- .../runners/spark/io/SparkUnboundedSource.java | 20 +- .../runners/spark/metrics/AggregatorMetric.java | 1 - .../spark/metrics/MetricsAccumulator.java | 20 +- .../spark/metrics/MetricsAccumulatorParam.java | 20 +- .../runners/spark/metrics/SparkBeamMetric.java | 11 +- .../spark/metrics/SparkBeamMetricSource.java | 3 +- .../spark/metrics/SparkMetricResults.java | 172 ---- .../spark/metrics/SparkMetricsContainer.java | 174 ---- .../SparkGroupAlsoByWindowViaWindowSet.java | 17 +- .../spark/stateful/SparkStateInternals.java | 20 +- .../spark/stateful/SparkTimerInternals.java | 2 +- .../spark/stateful/StateSpecFunctions.java | 9 +- .../translation/DoFnRunnerWithMetrics.java | 11 +- .../spark/translation/EvaluationContext.java | 18 +- .../translation/GroupCombineFunctions.java | 5 +- .../spark/translation/MultiDoFnFunction.java | 8 +- .../translation/SparkAbstractCombineFn.java | 4 +- .../spark/translation/SparkAssignWindowFn.java | 4 +- .../spark/translation/SparkGlobalCombineFn.java | 2 +- ...SparkGroupAlsoByWindowViaOutputBufferFn.java | 6 +- .../spark/translation/SparkKeyedCombineFn.java | 2 +- .../spark/translation/SparkRuntimeContext.java | 26 +- .../translation/StorageLevelPTransform.java | 2 +- .../spark/translation/TransformTranslator.java | 8 +- .../spark/translation/TranslationUtils.java | 3 +- .../SparkRunnerStreamingContextFactory.java | 2 +- .../streaming/StreamingTransformTranslator.java | 8 +- .../spark/util/GlobalWatermarkHolder.java | 1 - .../util/SinglePrimitiveOutputPTransform.java | 2 +- .../spark/util/SparkSideInputReader.java | 4 +- .../apache/beam/runners/spark/CacheTest.java | 14 +- .../beam/runners/spark/ForceStreamingTest.java | 18 +- .../apache/beam/runners/spark/PipelineRule.java | 109 --- .../runners/spark/ProvidedSparkContextTest.java | 10 +- .../runners/spark/SparkRunnerDebuggerTest.java | 15 +- .../beam/runners/spark/StreamingTest.java | 23 + .../TestSparkPipelineOptionsRegistrar.java | 36 + .../metrics/sink/SparkMetricsSinkTest.java | 12 +- .../beam/runners/spark/io/AvroPipelineTest.java | 10 +- .../beam/runners/spark/io/NumShardsTest.java | 6 +- .../spark/io/ReaderToIteratorAdapterTest.java | 145 ++++ .../translation/SparkRuntimeContextTest.java | 122 +++ .../spark/translation/StorageLevelTest.java | 31 +- .../translation/streaming/CreateStreamTest.java | 53 +- .../ResumeFromCheckpointStreamingTest.java | 64 +- .../streaming/StreamingSourceMetricsTest.java | 16 +- .../streaming/TrackStreamingSourcesTest.java | 4 +- sdks/common/fn-api/pom.xml | 2 +- sdks/common/pom.xml | 2 +- sdks/common/runner-api/pom.xml | 5 +- sdks/java/build-tools/pom.xml | 2 +- .../src/main/resources/beam/findbugs-filter.xml | 12 +- sdks/java/core/pom.xml | 93 ++- .../main/java/org/apache/beam/sdk/Pipeline.java | 101 ++- .../org/apache/beam/sdk/PipelineResult.java | 11 +- .../org/apache/beam/sdk/PipelineRunner.java | 61 ++ .../beam/sdk/annotations/Experimental.java | 35 +- .../apache/beam/sdk/annotations/Internal.java | 6 +- .../org/apache/beam/sdk/coders/AtomicCoder.java | 85 ++ .../org/apache/beam/sdk/coders/AvroCoder.java | 70 +- .../apache/beam/sdk/coders/BigDecimalCoder.java | 25 +- .../beam/sdk/coders/BigEndianIntegerCoder.java | 10 +- .../beam/sdk/coders/BigEndianLongCoder.java | 12 +- .../apache/beam/sdk/coders/BigIntegerCoder.java | 21 +- .../org/apache/beam/sdk/coders/BitSetCoder.java | 15 +- .../apache/beam/sdk/coders/ByteArrayCoder.java | 41 +- .../org/apache/beam/sdk/coders/ByteCoder.java | 10 +- .../sdk/coders/CannotProvideCoderException.java | 2 +- .../java/org/apache/beam/sdk/coders/Coder.java | 178 +++- .../apache/beam/sdk/coders/CoderFactories.java | 290 ------- .../apache/beam/sdk/coders/CoderFactory.java | 44 - .../apache/beam/sdk/coders/CoderProvider.java | 19 +- .../beam/sdk/coders/CoderProviderRegistrar.java | 42 + .../apache/beam/sdk/coders/CoderProviders.java | 240 +++--- .../apache/beam/sdk/coders/CoderRegistrar.java | 45 -- .../apache/beam/sdk/coders/CoderRegistry.java | 618 +++++--------- .../apache/beam/sdk/coders/CollectionCoder.java | 22 - .../org/apache/beam/sdk/coders/CustomCoder.java | 26 +- .../apache/beam/sdk/coders/DefaultCoder.java | 119 ++- .../apache/beam/sdk/coders/DelegateCoder.java | 11 + .../org/apache/beam/sdk/coders/DoubleCoder.java | 10 +- .../apache/beam/sdk/coders/DurationCoder.java | 18 +- .../apache/beam/sdk/coders/InstantCoder.java | 18 +- .../apache/beam/sdk/coders/IterableCoder.java | 22 - .../beam/sdk/coders/IterableLikeCoder.java | 35 +- .../org/apache/beam/sdk/coders/KvCoder.java | 51 +- .../beam/sdk/coders/LengthPrefixCoder.java | 28 +- .../org/apache/beam/sdk/coders/ListCoder.java | 24 +- .../org/apache/beam/sdk/coders/MapCoder.java | 47 +- .../apache/beam/sdk/coders/NullableCoder.java | 31 +- .../beam/sdk/coders/SerializableCoder.java | 66 +- .../org/apache/beam/sdk/coders/SetCoder.java | 25 - .../beam/sdk/coders/StringDelegateCoder.java | 12 +- .../apache/beam/sdk/coders/StringUtf8Coder.java | 29 +- .../beam/sdk/coders/StructuralByteArray.java | 4 +- .../apache/beam/sdk/coders/StructuredCoder.java | 74 +- .../beam/sdk/coders/TextualIntegerCoder.java | 17 +- .../org/apache/beam/sdk/coders/VarIntCoder.java | 10 +- .../apache/beam/sdk/coders/VarLongCoder.java | 15 +- .../org/apache/beam/sdk/coders/VoidCoder.java | 12 +- .../apache/beam/sdk/coders/package-info.java | 2 +- .../java/org/apache/beam/sdk/io/AvroIO.java | 219 +++-- .../java/org/apache/beam/sdk/io/AvroSink.java | 34 +- .../java/org/apache/beam/sdk/io/AvroSource.java | 23 +- .../sdk/io/BoundedReadFromUnboundedSource.java | 4 +- .../apache/beam/sdk/io/CompressedSource.java | 6 + .../beam/sdk/io/DefaultFilenamePolicy.java | 169 ++++ .../org/apache/beam/sdk/io/FileBasedSink.java | 803 ++++++++++--------- .../org/apache/beam/sdk/io/FileBasedSource.java | 39 +- .../java/org/apache/beam/sdk/io/FileSystem.java | 3 + .../apache/beam/sdk/io/FileSystemRegistrar.java | 3 + .../org/apache/beam/sdk/io/FileSystems.java | 65 +- .../beam/sdk/io/LocalFileSystemRegistrar.java | 3 + .../org/apache/beam/sdk/io/LocalResources.java | 3 + .../main/java/org/apache/beam/sdk/io/Read.java | 8 +- .../java/org/apache/beam/sdk/io/TFRecordIO.java | 153 ++-- .../java/org/apache/beam/sdk/io/TextIO.java | 255 +++--- .../java/org/apache/beam/sdk/io/TextSink.java | 32 +- .../org/apache/beam/sdk/io/UnboundedSource.java | 23 +- .../java/org/apache/beam/sdk/io/WriteFiles.java | 285 ++++--- .../org/apache/beam/sdk/io/fs/ResourceId.java | 3 + .../apache/beam/sdk/metrics/CounterCell.java | 67 -- .../org/apache/beam/sdk/metrics/DirtyState.java | 98 --- .../beam/sdk/metrics/DistributionCell.java | 64 -- .../beam/sdk/metrics/DistributionData.java | 60 -- .../beam/sdk/metrics/DistributionResult.java | 3 + .../org/apache/beam/sdk/metrics/GaugeCell.java | 54 -- .../org/apache/beam/sdk/metrics/GaugeData.java | 81 -- .../apache/beam/sdk/metrics/GaugeResult.java | 3 + .../org/apache/beam/sdk/metrics/Metric.java | 7 +- .../org/apache/beam/sdk/metrics/MetricCell.java | 41 - .../beam/sdk/metrics/MetricFiltering.java | 99 --- .../org/apache/beam/sdk/metrics/MetricKey.java | 41 - .../apache/beam/sdk/metrics/MetricUpdates.java | 78 -- .../org/apache/beam/sdk/metrics/Metrics.java | 6 +- .../beam/sdk/metrics/MetricsContainer.java | 129 +-- .../beam/sdk/metrics/MetricsEnvironment.java | 14 +- .../org/apache/beam/sdk/metrics/MetricsMap.java | 86 -- .../apache/beam/sdk/metrics/SinkMetrics.java | 4 + .../apache/beam/sdk/metrics/SourceMetrics.java | 3 + .../DefaultPipelineOptionsRegistrar.java | 3 +- .../beam/sdk/options/PipelineOptionSpec.java | 4 +- .../beam/sdk/options/PipelineOptions.java | 5 +- .../sdk/options/PipelineOptionsFactory.java | 2 +- .../apache/beam/sdk/options/ValueProvider.java | 15 +- .../beam/sdk/options/ValueProviderUtils.java | 60 -- .../apache/beam/sdk/options/ValueProviders.java | 61 ++ .../java/org/apache/beam/sdk/package-info.java | 2 +- .../beam/sdk/runners/AppliedPTransform.java | 73 ++ .../beam/sdk/runners/PTransformMatcher.java | 7 +- .../beam/sdk/runners/PTransformOverride.java | 6 +- .../sdk/runners/PTransformOverrideFactory.java | 8 +- .../apache/beam/sdk/runners/PipelineRunner.java | 65 -- .../sdk/runners/PipelineRunnerRegistrar.java | 5 + .../beam/sdk/runners/TransformHierarchy.java | 15 +- .../apache/beam/sdk/runners/package-info.java | 4 +- .../org/apache/beam/sdk/state/BagState.java | 36 + .../apache/beam/sdk/state/CombiningState.java | 55 ++ .../apache/beam/sdk/state/GroupingState.java | 44 + .../org/apache/beam/sdk/state/MapState.java | 73 ++ .../apache/beam/sdk/state/ReadableState.java | 51 ++ .../apache/beam/sdk/state/ReadableStates.java | 44 + .../org/apache/beam/sdk/state/SetState.java | 47 ++ .../java/org/apache/beam/sdk/state/State.java | 36 + .../org/apache/beam/sdk/state/StateBinder.java | 70 ++ .../org/apache/beam/sdk/state/StateContext.java | 46 ++ .../apache/beam/sdk/state/StateContexts.java | 51 ++ .../org/apache/beam/sdk/state/StateSpec.java | 63 ++ .../org/apache/beam/sdk/state/StateSpecs.java | 683 ++++++++++++++++ .../org/apache/beam/sdk/state/TimeDomain.java | 49 ++ .../java/org/apache/beam/sdk/state/Timer.java | 70 ++ .../org/apache/beam/sdk/state/TimerSpec.java | 30 + .../org/apache/beam/sdk/state/TimerSpecs.java | 39 + .../java/org/apache/beam/sdk/state/Timers.java | 58 ++ .../org/apache/beam/sdk/state/ValueState.java | 35 + .../beam/sdk/state/WatermarkHoldState.java | 41 + .../org/apache/beam/sdk/state/package-info.java | 22 + .../beam/sdk/testing/CoderProperties.java | 6 +- .../apache/beam/sdk/testing/CrashingRunner.java | 2 +- .../beam/sdk/testing/FileChecksumMatcher.java | 2 +- .../beam/sdk/testing/MatcherDeserializer.java | 4 +- .../beam/sdk/testing/MatcherSerializer.java | 4 +- .../apache/beam/sdk/testing/NeedsRunner.java | 2 +- .../org/apache/beam/sdk/testing/PAssert.java | 4 +- .../apache/beam/sdk/testing/TestPipeline.java | 77 +- .../beam/sdk/testing/TestPipelineOptions.java | 2 - .../org/apache/beam/sdk/testing/TestStream.java | 4 +- .../apache/beam/sdk/testing/UsesMapState.java | 2 +- .../apache/beam/sdk/testing/UsesSetState.java | 2 +- .../apache/beam/sdk/transforms/Aggregator.java | 34 - .../beam/sdk/transforms/AppliedPTransform.java | 73 -- .../sdk/transforms/ApproximateQuantiles.java | 90 ++- .../org/apache/beam/sdk/transforms/Combine.java | 44 +- .../beam/sdk/transforms/CombineFnBase.java | 26 +- .../apache/beam/sdk/transforms/CombineFns.java | 25 +- .../beam/sdk/transforms/CombineWithContext.java | 3 +- .../org/apache/beam/sdk/transforms/Count.java | 12 +- .../org/apache/beam/sdk/transforms/Create.java | 76 +- .../org/apache/beam/sdk/transforms/DoFn.java | 91 ++- .../apache/beam/sdk/transforms/DoFnTester.java | 66 +- .../org/apache/beam/sdk/transforms/Flatten.java | 2 +- .../apache/beam/sdk/transforms/GroupByKey.java | 2 +- .../beam/sdk/transforms/GroupIntoBatches.java | 71 +- .../org/apache/beam/sdk/transforms/Latest.java | 15 +- .../beam/sdk/transforms/Materialization.java | 8 +- .../beam/sdk/transforms/Materializations.java | 11 +- .../org/apache/beam/sdk/transforms/Mean.java | 19 +- .../apache/beam/sdk/transforms/PTransform.java | 10 +- .../org/apache/beam/sdk/transforms/ParDo.java | 20 +- .../org/apache/beam/sdk/transforms/Regex.java | 1 - .../beam/sdk/transforms/ReifyTimestamps.java | 79 ++ .../apache/beam/sdk/transforms/Reshuffle.java | 97 +++ .../org/apache/beam/sdk/transforms/Top.java | 46 +- .../org/apache/beam/sdk/transforms/View.java | 40 +- .../org/apache/beam/sdk/transforms/ViewFn.java | 6 +- .../apache/beam/sdk/transforms/WithKeys.java | 4 +- .../beam/sdk/transforms/join/CoGbkResult.java | 20 +- .../beam/sdk/transforms/join/UnionCoder.java | 29 +- .../reflect/ByteBuddyDoFnInvokerFactory.java | 31 +- .../reflect/ByteBuddyOnTimerInvokerFactory.java | 20 +- .../sdk/transforms/reflect/DoFnInvoker.java | 23 +- .../sdk/transforms/reflect/DoFnSignature.java | 50 +- .../sdk/transforms/reflect/DoFnSignatures.java | 66 +- .../reflect/StableInvokerNamingStrategy.java | 54 ++ .../sdk/transforms/reflect/package-info.java | 5 +- .../beam/sdk/transforms/windowing/AfterAll.java | 8 +- .../sdk/transforms/windowing/AfterEach.java | 2 +- .../sdk/transforms/windowing/AfterFirst.java | 5 +- .../sdk/transforms/windowing/AfterPane.java | 5 +- .../windowing/AfterProcessingTime.java | 6 +- .../transforms/windowing/AfterWatermark.java | 6 +- .../transforms/windowing/DefaultTrigger.java | 10 +- .../sdk/transforms/windowing/GlobalWindow.java | 10 +- .../sdk/transforms/windowing/GlobalWindows.java | 5 +- .../transforms/windowing/IntervalWindow.java | 19 +- .../transforms/windowing/InvalidWindows.java | 2 +- .../MergeOverlappingIntervalWindows.java | 6 +- .../beam/sdk/transforms/windowing/Never.java | 4 +- .../windowing/NonMergingWindowFn.java | 2 +- .../transforms/windowing/OrFinallyTrigger.java | 7 +- .../beam/sdk/transforms/windowing/PaneInfo.java | 18 +- .../sdk/transforms/windowing/Repeatedly.java | 6 +- .../transforms/windowing/ReshuffleTrigger.java | 58 ++ .../beam/sdk/transforms/windowing/Sessions.java | 13 +- .../transforms/windowing/TimestampCombiner.java | 5 + .../windowing/TimestampTransform.java | 41 +- .../beam/sdk/transforms/windowing/Trigger.java | 18 +- .../beam/sdk/transforms/windowing/Window.java | 16 +- .../beam/sdk/transforms/windowing/WindowFn.java | 10 +- .../transforms/windowing/WindowMappingFn.java | 12 +- .../org/apache/beam/sdk/util/ApiSurface.java | 10 +- .../apache/beam/sdk/util/AppliedCombineFn.java | 1 + .../java/org/apache/beam/sdk/util/BackOff.java | 81 ++ .../org/apache/beam/sdk/util/BackOffUtils.java | 57 ++ .../org/apache/beam/sdk/util/BitSetCoder.java | 21 +- .../org/apache/beam/sdk/util/ClassPath.java | 1 + .../apache/beam/sdk/util/CloudKnownType.java | 143 ---- .../org/apache/beam/sdk/util/CloudObject.java | 187 ----- .../org/apache/beam/sdk/util/CoderUtils.java | 127 +-- .../beam/sdk/util/CombineContextFactory.java | 27 +- .../org/apache/beam/sdk/util/CombineFnUtil.java | 2 +- .../beam/sdk/util/ExplicitShardedFile.java | 3 - .../beam/sdk/util/FileIOChannelFactory.java | 223 ----- .../sdk/util/FileIOChannelFactoryRegistrar.java | 38 - .../org/apache/beam/sdk/util/FluentBackoff.java | 1 - .../apache/beam/sdk/util/IOChannelFactory.java | 127 --- .../sdk/util/IOChannelFactoryRegistrar.java | 48 -- .../apache/beam/sdk/util/IOChannelUtils.java | 280 ------- .../apache/beam/sdk/util/IdentityWindowFn.java | 1 + .../apache/beam/sdk/util/NoopPathValidator.java | 49 -- .../beam/sdk/util/NullSideInputReader.java | 61 -- .../beam/sdk/util/NumberedShardedFile.java | 7 +- .../beam/sdk/util/PCollectionViewWindow.java | 67 -- .../apache/beam/sdk/util/PCollectionViews.java | 496 ------------ .../org/apache/beam/sdk/util/PathValidator.java | 48 -- .../org/apache/beam/sdk/util/PropertyNames.java | 112 --- .../sdk/util/ReadyCheckingSideInputReader.java | 34 - .../apache/beam/sdk/util/ReifyTimestamps.java | 76 -- .../org/apache/beam/sdk/util/ReleaseInfo.java | 70 +- .../org/apache/beam/sdk/util/Reshuffle.java | 88 -- .../apache/beam/sdk/util/ReshuffleTrigger.java | 52 -- .../org/apache/beam/sdk/util/Serializer.java | 147 ---- .../org/apache/beam/sdk/util/ShardedFile.java | 2 - .../apache/beam/sdk/util/SideInputReader.java | 47 -- .../java/org/apache/beam/sdk/util/Sleeper.java | 48 ++ .../java/org/apache/beam/sdk/util/Structs.java | 371 --------- .../beam/sdk/util/SystemDoFnInternal.java | 3 - .../org/apache/beam/sdk/util/TimeDomain.java | 43 - .../java/org/apache/beam/sdk/util/Timer.java | 67 -- .../org/apache/beam/sdk/util/TimerSpec.java | 30 - .../org/apache/beam/sdk/util/TimerSpecs.java | 41 - .../java/org/apache/beam/sdk/util/Timers.java | 58 -- .../sdk/util/UploadIdResponseInterceptor.java | 60 -- .../apache/beam/sdk/util/ValueWithRecordId.java | 134 ---- .../java/org/apache/beam/sdk/util/Values.java | 88 -- .../org/apache/beam/sdk/util/WindowedValue.java | 93 +-- .../apache/beam/sdk/util/WindowingStrategy.java | 283 ------- .../apache/beam/sdk/util/state/BagState.java | 28 - .../beam/sdk/util/state/CombiningState.java | 53 -- .../beam/sdk/util/state/GroupingState.java | 42 - .../apache/beam/sdk/util/state/MapState.java | 79 -- .../beam/sdk/util/state/ReadableState.java | 51 -- .../beam/sdk/util/state/ReadableStates.java | 45 -- .../apache/beam/sdk/util/state/SetState.java | 45 -- .../org/apache/beam/sdk/util/state/State.java | 32 - .../apache/beam/sdk/util/state/StateBinder.java | 66 -- .../beam/sdk/util/state/StateContext.java | 43 - .../beam/sdk/util/state/StateContexts.java | 51 -- .../apache/beam/sdk/util/state/StateSpec.java | 53 -- .../apache/beam/sdk/util/state/StateSpecs.java | 629 --------------- .../apache/beam/sdk/util/state/ValueState.java | 37 - .../beam/sdk/util/state/WatermarkHoldState.java | 42 - .../beam/sdk/util/state/package-info.java | 22 - .../java/org/apache/beam/sdk/values/PBegin.java | 26 +- .../org/apache/beam/sdk/values/PCollection.java | 162 +++- .../apache/beam/sdk/values/PCollectionList.java | 14 +- .../beam/sdk/values/PCollectionTuple.java | 24 +- .../apache/beam/sdk/values/PCollectionView.java | 33 +- .../beam/sdk/values/PCollectionViews.java | 495 ++++++++++++ .../java/org/apache/beam/sdk/values/PDone.java | 12 +- .../org/apache/beam/sdk/values/POutput.java | 25 +- .../java/org/apache/beam/sdk/values/PValue.java | 12 +- .../org/apache/beam/sdk/values/PValueBase.java | 61 +- .../apache/beam/sdk/values/TaggedPValue.java | 6 +- .../beam/sdk/values/TimestampedValue.java | 30 +- .../org/apache/beam/sdk/values/TupleTag.java | 22 - .../org/apache/beam/sdk/values/TypedPValue.java | 208 ----- .../beam/sdk/values/ValueInSingleWindow.java | 29 +- .../beam/sdk/values/ValueWithRecordId.java | 149 ++++ .../beam/sdk/values/WindowingStrategy.java | 283 +++++++ .../org/apache/beam/SdkCoreApiSurfaceTest.java | 1 - .../java/org/apache/beam/sdk/PipelineTest.java | 3 +- .../apache/beam/sdk/coders/AvroCoderTest.java | 11 +- .../beam/sdk/coders/BigDecimalCoderTest.java | 5 +- .../beam/sdk/coders/BigIntegerCoderTest.java | 5 +- .../beam/sdk/coders/CoderFactoriesTest.java | 100 --- .../beam/sdk/coders/CoderProvidersTest.java | 82 +- .../beam/sdk/coders/CoderRegistryTest.java | 191 ++--- .../apache/beam/sdk/coders/CustomCoderTest.java | 4 +- .../beam/sdk/coders/DefaultCoderTest.java | 65 +- .../beam/sdk/coders/DelegateCoderTest.java | 25 - .../beam/sdk/coders/IterableCoderTest.java | 17 - .../beam/sdk/coders/LengthPrefixCoderTest.java | 24 +- .../apache/beam/sdk/coders/ListCoderTest.java | 17 - .../apache/beam/sdk/coders/MapCoderTest.java | 20 - .../beam/sdk/coders/NullableCoderTest.java | 31 +- .../beam/sdk/coders/SerializableCoderTest.java | 47 +- .../beam/sdk/coders/StructuredCoderTest.java | 12 +- .../java/org/apache/beam/sdk/io/AvroIOTest.java | 101 ++- .../beam/sdk/io/CompressedSourceTest.java | 25 +- .../beam/sdk/io/DefaultFilenamePolicyTest.java | 55 ++ .../sdk/io/DrunkWritableByteChannelFactory.java | 1 - .../apache/beam/sdk/io/FileBasedSinkTest.java | 460 +++++------ .../apache/beam/sdk/io/LocalResourceIdTest.java | 6 + .../java/org/apache/beam/sdk/io/SimpleSink.java | 31 +- .../org/apache/beam/sdk/io/TFRecordIOTest.java | 2 - .../java/org/apache/beam/sdk/io/TextIOTest.java | 57 +- .../org/apache/beam/sdk/io/WriteFilesTest.java | 41 +- .../apache/beam/sdk/io/fs/ResourceIdTester.java | 150 ++++ .../beam/sdk/metrics/CounterCellTest.java | 55 -- .../apache/beam/sdk/metrics/DirtyStateTest.java | 56 -- .../beam/sdk/metrics/DistributionCellTest.java | 53 -- .../apache/beam/sdk/metrics/GaugeCellTest.java | 48 -- .../beam/sdk/metrics/MetricFilteringTest.java | 145 ---- .../apache/beam/sdk/metrics/MetricMatchers.java | 242 ------ .../beam/sdk/metrics/MetricResultsMatchers.java | 190 +++++ .../beam/sdk/metrics/MetricsContainerTest.java | 129 --- .../sdk/metrics/MetricsEnvironmentTest.java | 23 +- .../apache/beam/sdk/metrics/MetricsMapTest.java | 103 --- .../apache/beam/sdk/metrics/MetricsTest.java | 47 +- .../sdk/options/PipelineOptionsFactoryTest.java | 4 +- .../beam/sdk/options/PipelineOptionsTest.java | 1 - .../sdk/options/ProxyInvocationHandlerTest.java | 4 +- .../beam/sdk/options/ValueProviderTest.java | 23 +- .../sdk/options/ValueProviderUtilsTest.java | 78 -- .../beam/sdk/options/ValueProvidersTest.java | 79 ++ .../beam/sdk/runners/PipelineRunnerTest.java | 1 + .../sdk/runners/TransformHierarchyTest.java | 5 +- .../beam/sdk/runners/TransformTreeTest.java | 2 +- .../beam/sdk/testing/CoderPropertiesTest.java | 73 +- .../beam/sdk/testing/CrashingRunnerTest.java | 2 +- .../beam/sdk/testing/ExpectedLogsTest.java | 2 +- .../sdk/testing/FastNanoClockAndSleeper.java | 47 -- .../testing/FastNanoClockAndSleeperTest.java | 47 -- .../sdk/testing/FileChecksumMatcherTest.java | 12 +- .../apache/beam/sdk/testing/PAssertTest.java | 14 +- .../sdk/testing/PCollectionViewTesting.java | 2 +- .../sdk/testing/SerializableMatchersTest.java | 9 +- .../beam/sdk/testing/SystemNanoTimeSleeper.java | 2 +- .../beam/sdk/testing/TestPipelineTest.java | 6 +- .../beam/sdk/testing/WindowSupplierTest.java | 10 +- .../sdk/transforms/ApproximateUniqueTest.java | 2 +- .../beam/sdk/transforms/CombineFnsTest.java | 30 +- .../apache/beam/sdk/transforms/CombineTest.java | 39 +- .../apache/beam/sdk/transforms/CreateTest.java | 25 +- .../beam/sdk/transforms/DoFnTesterTest.java | 14 +- .../sdk/transforms/FlatMapElementsTest.java | 4 +- .../beam/sdk/transforms/GroupByKeyTest.java | 15 +- .../beam/sdk/transforms/LatestFnTest.java | 2 +- .../beam/sdk/transforms/MapElementsTest.java | 4 +- .../beam/sdk/transforms/ParDoLifecycleTest.java | 16 +- .../apache/beam/sdk/transforms/ParDoTest.java | 258 +++--- .../apache/beam/sdk/transforms/RegexTest.java | 2 - .../sdk/transforms/ReifyTimestampsTest.java | 140 ++++ .../beam/sdk/transforms/ReshuffleTest.java | 293 +++++++ .../beam/sdk/transforms/SplittableDoFnTest.java | 4 +- .../apache/beam/sdk/transforms/ViewTest.java | 17 +- .../display/DisplayDataEvaluator.java | 2 +- .../transforms/reflect/DoFnInvokersTest.java | 73 +- .../DoFnSignaturesSplittableDoFnTest.java | 4 +- .../transforms/reflect/DoFnSignaturesTest.java | 35 +- .../transforms/reflect/OnTimerInvokersTest.java | 38 +- .../testhelper/DoFnInvokersTestHelper.java | 6 +- .../transforms/windowing/GlobalWindowTest.java | 2 +- .../windowing/ReshuffleTriggerTest.java | 42 + .../sdk/transforms/windowing/WindowTest.java | 4 +- ...BufferedElementCountingOutputStreamTest.java | 5 +- .../apache/beam/sdk/util/CoderUtilsTest.java | 8 +- .../apache/beam/sdk/util/CombineFnUtilTest.java | 2 +- .../util/FileIOChannelFactoryRegistrarTest.java | 44 - .../beam/sdk/util/FileIOChannelFactoryTest.java | 242 ------ .../apache/beam/sdk/util/FluentBackoffTest.java | 1 - .../beam/sdk/util/IOChannelUtilsTest.java | 118 --- .../beam/sdk/util/NumberedShardedFileTest.java | 14 +- .../beam/sdk/util/ReifyTimestampsTest.java | 109 --- .../org/apache/beam/sdk/util/ReshuffleTest.java | 270 ------- .../beam/sdk/util/ReshuffleTriggerTest.java | 44 - .../beam/sdk/util/SerializableUtilsTest.java | 8 +- .../apache/beam/sdk/util/SerializerTest.java | 162 ---- .../org/apache/beam/sdk/util/StructsTest.java | 206 ----- .../util/UploadIdResponseInterceptorTest.java | 98 --- .../beam/sdk/util/ValueWithRecordIdTest.java | 34 - .../beam/sdk/values/PCollectionTupleTest.java | 1 - .../apache/beam/sdk/values/TypedPValueTest.java | 28 +- .../beam/sdk/values/ValueWithRecordIdTest.java | 34 + .../google-cloud-platform-core/pom.xml | 8 +- .../sdk/extensions/gcp/options/GcpOptions.java | 9 +- .../sdk/extensions/gcp/options/GcsOptions.java | 4 +- .../gcp/storage/GcsCreateOptions.java | 56 ++ .../extensions/gcp/storage/GcsFileSystem.java | 20 +- .../gcp/storage/GcsFileSystemRegistrar.java | 5 +- .../gcp/storage/GcsPathValidator.java | 105 +++ .../gcp/storage/NoopPathValidator.java | 53 ++ .../extensions/gcp/storage/PathValidator.java | 59 ++ .../apache/beam/sdk/util/BackOffAdapter.java | 43 + .../beam/sdk/util/GcsIOChannelFactory.java | 111 --- .../sdk/util/GcsIOChannelFactoryRegistrar.java | 38 - .../apache/beam/sdk/util/GcsPathValidator.java | 98 --- .../java/org/apache/beam/sdk/util/GcsUtil.java | 71 +- .../sdk/util/UploadIdResponseInterceptor.java | 60 ++ .../extensions/gcp/GcpCoreApiSurfaceTest.java | 3 +- .../extensions/gcp/options/GcpOptionsTest.java | 4 +- .../gcp/options/GoogleApiDebugOptionsTest.java | 8 +- .../gcp/storage/GcsFileSystemTest.java | 30 +- .../gcp/storage/GcsPathValidatorTest.java | 107 +++ .../gcp/storage/GcsResourceIdTest.java | 9 + .../beam/sdk/util/FastNanoClockAndSleeper.java | 47 ++ .../sdk/util/FastNanoClockAndSleeperTest.java | 47 ++ .../util/GcsIOChannelFactoryRegistrarTest.java | 44 - .../beam/sdk/util/GcsIOChannelFactoryTest.java | 43 - .../beam/sdk/util/GcsPathValidatorTest.java | 106 --- .../org/apache/beam/sdk/util/GcsUtilTest.java | 80 +- .../util/UploadIdResponseInterceptorTest.java | 98 +++ sdks/java/extensions/jackson/pom.xml | 2 +- sdks/java/extensions/join-library/README.md | 42 - sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/protobuf/pom.xml | 2 +- .../extensions/protobuf/ByteStringCoder.java | 24 +- .../sdk/extensions/protobuf/ProtoCoder.java | 85 +- .../ProtobufCoderProviderRegistrar.java | 41 + .../protobuf/ProtobufCoderRegistrar.java | 39 - .../protobuf/ByteStringCoderTest.java | 10 +- .../sdk/extensions/protobuf/ProtoCoderTest.java | 7 +- sdks/java/extensions/sorter/README.md | 42 - sdks/java/extensions/sorter/pom.xml | 2 +- .../extensions/sorter/ExternalSorterTest.java | 1 - sdks/java/harness/pom.xml | 12 +- .../org/apache/beam/fn/harness/FnHarness.java | 8 +- .../harness/control/ProcessBundleHandler.java | 4 +- .../BeamFnDataBufferingOutboundObserver.java | 3 +- .../harness/data/BeamFnDataInboundObserver.java | 3 +- .../beam/runners/core/BeamFnDataReadRunner.java | 19 +- .../runners/core/BeamFnDataWriteRunner.java | 20 +- .../control/ProcessBundleHandlerTest.java | 55 +- ...BeamFnDataBufferingOutboundObserverTest.java | 3 +- .../data/BeamFnDataInboundObserverTest.java | 3 +- .../runners/core/BeamFnDataReadRunnerTest.java | 14 +- .../runners/core/BeamFnDataWriteRunnerTest.java | 14 +- sdks/java/io/common/pom.xml | 2 +- .../apache/beam/sdk/io/common/HashingFn.java | 2 - sdks/java/io/elasticsearch/pom.xml | 2 +- .../sdk/io/elasticsearch/ElasticsearchIO.java | 12 +- sdks/java/io/google-cloud-platform/pom.xml | 24 +- .../beam/sdk/io/gcp/bigquery/BatchLoads.java | 253 ++++-- .../BigQueryCoderProviderRegistrar.java | 40 + .../io/gcp/bigquery/BigQueryCoderRegistrar.java | 39 - .../sdk/io/gcp/bigquery/BigQueryHelpers.java | 141 ++-- .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 562 +++++++------ .../io/gcp/bigquery/BigQueryQuerySource.java | 44 +- .../io/gcp/bigquery/BigQueryServicesImpl.java | 101 +-- .../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 46 +- .../gcp/bigquery/BigQueryTableRowIterator.java | 7 +- .../io/gcp/bigquery/BigQueryTableSource.java | 17 +- .../sdk/io/gcp/bigquery/CalculateSchemas.java | 78 ++ .../beam/sdk/io/gcp/bigquery/CreateTables.java | 66 +- .../io/gcp/bigquery/DynamicDestinations.java | 178 ++++ .../bigquery/DynamicDestinationsHelpers.java | 193 +++++ .../beam/sdk/io/gcp/bigquery/PrepareWrite.java | 36 +- .../sdk/io/gcp/bigquery/ShardedKeyCoder.java | 22 +- .../sdk/io/gcp/bigquery/StreamingInserts.java | 59 +- .../sdk/io/gcp/bigquery/StreamingWriteFn.java | 4 +- .../io/gcp/bigquery/StreamingWriteTables.java | 10 +- .../sdk/io/gcp/bigquery/TableDestination.java | 14 +- .../io/gcp/bigquery/TableDestinationCoder.java | 27 +- .../sdk/io/gcp/bigquery/TableRowInfoCoder.java | 19 +- .../sdk/io/gcp/bigquery/TableRowJsonCoder.java | 19 +- .../sdk/io/gcp/bigquery/TableRowWriter.java | 59 +- .../sdk/io/gcp/bigquery/TagWithUniqueIds.java | 2 +- .../io/gcp/bigquery/WriteBundlesToFiles.java | 190 +++-- .../bigquery/WriteGroupedRecordsToFiles.java | 68 ++ .../sdk/io/gcp/bigquery/WritePartition.java | 86 +- .../beam/sdk/io/gcp/bigquery/WriteRename.java | 10 +- .../beam/sdk/io/gcp/bigquery/WriteResult.java | 23 +- .../beam/sdk/io/gcp/bigquery/WriteTables.java | 160 ++-- .../beam/sdk/io/gcp/bigtable/BigtableIO.java | 245 +++--- .../io/gcp/bigtable/BigtableServiceImpl.java | 9 - .../beam/sdk/io/gcp/datastore/DatastoreV1.java | 19 +- .../pubsub/PubsubCoderProviderRegistrar.java | 37 + .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 78 +- .../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 61 ++ .../pubsub/PubsubMessagePayloadOnlyCoder.java | 32 +- .../PubsubMessageWithAttributesCoder.java | 40 +- .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 40 +- .../io/gcp/pubsub/PubsubUnboundedSource.java | 51 +- .../BigQueryCoderProviderRegistrarTest.java | 40 + .../bigquery/BigQueryCoderRegistrarTest.java | 40 - .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 561 ++++++++----- .../gcp/bigquery/BigQueryServicesImplTest.java | 24 +- .../io/gcp/bigquery/FakeBigQueryServices.java | 2 - .../sdk/io/gcp/bigquery/FakeJobService.java | 55 +- .../sdk/io/gcp/bigquery/TableContainer.java | 1 - .../beam/sdk/io/gcp/datastore/V1TestUtil.java | 6 +- .../io/gcp/pubsub/PubsubUnboundedSinkTest.java | 41 +- .../gcp/pubsub/PubsubUnboundedSourceTest.java | 4 +- .../sdk/io/gcp/testing/BigqueryMatcher.java | 4 +- .../sdk/io/gcp/testing/BigqueryMatcherTest.java | 7 +- sdks/java/io/hadoop-common/pom.xml | 9 +- .../beam/sdk/io/hadoop/WritableCoder.java | 78 +- .../beam/sdk/io/hadoop/WritableCoderTest.java | 10 + sdks/java/io/hadoop-file-system/README.md | 43 - sdks/java/io/hadoop-file-system/pom.xml | 45 +- .../apache/beam/sdk/io/hdfs/HDFSFileSink.java | 478 ----------- .../apache/beam/sdk/io/hdfs/HDFSFileSource.java | 625 --------------- .../beam/sdk/io/hdfs/HadoopFileSystem.java | 32 +- .../sdk/io/hdfs/HadoopFileSystemOptions.java | 76 +- .../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 3 + .../beam/sdk/io/hdfs/HadoopResourceId.java | 16 +- .../java/org/apache/beam/sdk/io/hdfs/Sink.java | 195 ----- .../org/apache/beam/sdk/io/hdfs/UGIHelper.java | 38 - .../java/org/apache/beam/sdk/io/hdfs/Write.java | 585 -------------- .../apache/beam/sdk/io/hdfs/package-info.java | 3 +- .../beam/sdk/io/hdfs/HDFSFileSinkTest.java | 172 ---- .../beam/sdk/io/hdfs/HDFSFileSourceTest.java | 231 ------ .../sdk/io/hdfs/HadoopFileSystemModuleTest.java | 2 +- .../io/hdfs/HadoopFileSystemOptionsTest.java | 125 +++ .../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 5 +- .../beam/sdk/io/hdfs/HadoopResourceIdTest.java | 71 ++ sdks/java/io/hadoop/README.md | 167 ---- sdks/java/io/hadoop/input-format/pom.xml | 2 +- .../ConfigurableEmployeeInputFormat.java | 1 - .../hadoop/inputformat/EmployeeInputFormat.java | 1 - .../ReuseObjectsEmployeeInputFormat.java | 1 - .../hadoop/inputformat/TestEmployeeDataSet.java | 2 - sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 30 +- .../inputformat/HIFIOWithElasticTest.java | 1 - .../HIFIOWithEmbeddedCassandraTest.java | 2 - .../integration/tests/HIFIOCassandraIT.java | 2 - .../integration/tests/HIFIOElasticIT.java | 1 - sdks/java/io/hadoop/pom.xml | 2 +- sdks/java/io/hbase/pom.xml | 27 +- .../org/apache/beam/sdk/io/hbase/HBaseIO.java | 7 +- .../beam/sdk/io/hbase/HBaseMutationCoder.java | 10 +- .../beam/sdk/io/hbase/HBaseResultCoder.java | 8 +- .../apache/beam/sdk/io/hbase/HBaseIOTest.java | 4 +- sdks/java/io/jdbc/pom.xml | 2 +- .../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 13 +- sdks/java/io/jdbc/src/test/README.md | 32 - .../org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 1 - sdks/java/io/jms/pom.xml | 2 +- .../java/org/apache/beam/sdk/io/jms/JmsIO.java | 17 +- .../org/apache/beam/sdk/io/jms/JmsRecord.java | 1 - sdks/java/io/kafka/pom.xml | 2 +- .../beam/sdk/io/kafka/KafkaCheckpointMark.java | 2 - .../org/apache/beam/sdk/io/kafka/KafkaIO.java | 348 +++----- .../apache/beam/sdk/io/kafka/KafkaRecord.java | 1 - .../beam/sdk/io/kafka/KafkaRecordCoder.java | 19 +- .../CoderBasedKafkaDeserializer.java | 71 -- .../CoderBasedKafkaSerializer.java | 73 -- .../serialization/InstantDeserializer.java | 1 - .../kafka/serialization/InstantSerializer.java | 1 - .../apache/beam/sdk/io/kafka/KafkaIOTest.java | 166 +--- sdks/java/io/kinesis/pom.xml | 2 +- .../beam/sdk/io/kinesis/KinesisRecordCoder.java | 42 +- sdks/java/io/mongodb/pom.xml | 2 +- .../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 7 +- .../apache/beam/sdk/io/mongodb/MongoDbIO.java | 57 +- .../sdk/io/mongodb/MongoDBGridFSIOTest.java | 3 - .../beam/sdk/io/mongodb/MongoDbIOTest.java | 20 +- sdks/java/io/mqtt/pom.xml | 2 +- .../org/apache/beam/sdk/io/mqtt/MqttIO.java | 3 - sdks/java/io/pom.xml | 2 +- sdks/java/io/xml/pom.xml | 2 +- .../org/apache/beam/sdk/io/xml/JAXBCoder.java | 63 +- .../java/org/apache/beam/sdk/io/xml/XmlIO.java | 19 +- .../org/apache/beam/sdk/io/xml/XmlSink.java | 24 +- .../apache/beam/sdk/io/xml/JAXBCoderTest.java | 25 +- .../org/apache/beam/sdk/io/xml/XmlSinkTest.java | 52 +- sdks/java/java8tests/pom.xml | 2 +- .../beam/sdk/transforms/DistinctJava8Test.java | 8 - .../beam/sdk/transforms/FilterJava8Test.java | 2 +- .../beam/sdk/transforms/PartitionJava8Test.java | 2 +- .../beam/sdk/transforms/WithKeysJava8Test.java | 9 - sdks/java/javadoc/README.md | 23 - sdks/java/javadoc/ant.xml | 5 +- sdks/java/javadoc/pom.xml | 2 +- .../examples-java8/generate-sources.sh | 12 +- .../maven-archetypes/examples-java8/pom.xml | 108 +-- .../main/resources/archetype-resources/pom.xml | 23 +- .../examples/generate-sources.sh | 8 +- sdks/java/maven-archetypes/examples/pom.xml | 67 +- .../main/resources/archetype-resources/pom.xml | 16 +- sdks/java/maven-archetypes/pom.xml | 22 +- sdks/java/maven-archetypes/starter/pom.xml | 19 +- .../resources/projects/basic/reference/pom.xml | 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 20 +- sdks/python/README.md | 298 ------- sdks/python/apache_beam/__init__.py | 2 +- sdks/python/apache_beam/coders/coder_impl.py | 59 +- sdks/python/apache_beam/coders/coders.py | 32 +- sdks/python/apache_beam/coders/coders_test.py | 11 +- .../apache_beam/coders/coders_test_common.py | 2 +- sdks/python/apache_beam/coders/observable.py | 9 +- sdks/python/apache_beam/coders/slow_stream.py | 21 +- .../apache_beam/coders/standard_coders_test.py | 4 +- sdks/python/apache_beam/coders/stream.pyx | 5 + sdks/python/apache_beam/coders/typecoders.py | 7 +- .../examples/complete/autocomplete.py | 4 +- .../examples/complete/autocomplete_test.py | 6 +- .../examples/complete/estimate_pi.py | 4 +- .../examples/complete/estimate_pi_test.py | 6 +- .../examples/complete/game/README.md | 69 -- .../examples/complete/game/hourly_team_score.py | 6 +- .../complete/game/hourly_team_score_test.py | 6 +- .../examples/complete/game/user_score.py | 4 +- .../examples/complete/game/user_score_test.py | 6 +- .../apache_beam/examples/complete/tfidf.py | 4 +- .../apache_beam/examples/complete/tfidf_test.py | 6 +- .../examples/complete/top_wikipedia_sessions.py | 4 +- .../complete/top_wikipedia_sessions_test.py | 6 +- .../examples/cookbook/bigquery_side_input.py | 4 +- .../cookbook/bigquery_side_input_test.py | 6 +- .../cookbook/bigquery_tornadoes_it_test.py | 4 +- .../cookbook/bigquery_tornadoes_test.py | 8 +- .../apache_beam/examples/cookbook/coders.py | 4 +- .../examples/cookbook/coders_test.py | 6 +- .../examples/cookbook/combiners_test.py | 8 +- .../examples/cookbook/custom_ptransform.py | 2 +- .../examples/cookbook/custom_ptransform_test.py | 6 +- .../examples/cookbook/datastore_wordcount.py | 19 +- .../examples/cookbook/filters_test.py | 14 +- .../examples/cookbook/group_with_coder.py | 4 +- .../examples/cookbook/mergecontacts.py | 18 +- .../examples/cookbook/multiple_output_pardo.py | 4 +- .../apache_beam/examples/snippets/snippets.py | 53 +- .../examples/snippets/snippets_test.py | 49 +- sdks/python/apache_beam/examples/wordcount.py | 42 +- .../apache_beam/examples/wordcount_debugging.py | 10 +- .../apache_beam/examples/wordcount_it_test.py | 6 +- .../apache_beam/examples/wordcount_minimal.py | 4 +- sdks/python/apache_beam/internal/__init__.py | 2 + .../python/apache_beam/internal/gcp/__init__.py | 2 + sdks/python/apache_beam/internal/gcp/auth.py | 50 +- .../apache_beam/internal/gcp/json_value.py | 14 +- .../apache_beam/internal/gcp/json_value_test.py | 4 +- sdks/python/apache_beam/internal/pickler.py | 28 +- sdks/python/apache_beam/internal/util.py | 21 +- sdks/python/apache_beam/io/__init__.py | 2 +- sdks/python/apache_beam/io/avroio.py | 17 +- sdks/python/apache_beam/io/avroio_test.py | 6 +- sdks/python/apache_beam/io/concat_source.py | 12 +- .../python/apache_beam/io/concat_source_test.py | 6 +- sdks/python/apache_beam/io/filebasedsink.py | 299 +++++++ .../python/apache_beam/io/filebasedsink_test.py | 303 +++++++ sdks/python/apache_beam/io/filebasedsource.py | 8 +- .../apache_beam/io/filebasedsource_test.py | 10 +- sdks/python/apache_beam/io/fileio.py | 297 ------- sdks/python/apache_beam/io/fileio_test.py | 304 ------- sdks/python/apache_beam/io/filesystem.py | 19 + sdks/python/apache_beam/io/filesystems.py | 38 +- sdks/python/apache_beam/io/filesystems_test.py | 15 +- sdks/python/apache_beam/io/filesystems_util.py | 36 - sdks/python/apache_beam/io/gcp/bigquery.py | 10 +- sdks/python/apache_beam/io/gcp/bigquery_test.py | 2 +- .../io/gcp/datastore/v1/fake_datastore.py | 14 +- .../apache_beam/io/gcp/datastore/v1/helper.py | 35 +- .../io/gcp/datastore/v1/helper_test.py | 26 +- .../io/gcp/datastore/v1/query_splitter_test.py | 37 +- sdks/python/apache_beam/io/gcp/gcsfilesystem.py | 8 + .../apache_beam/io/gcp/gcsfilesystem_test.py | 5 + sdks/python/apache_beam/io/gcp/gcsio.py | 9 +- sdks/python/apache_beam/io/gcp/pubsub.py | 2 + .../io/gcp/tests/bigquery_matcher.py | 5 +- .../io/gcp/tests/bigquery_matcher_test.py | 2 +- sdks/python/apache_beam/io/iobase.py | 23 +- sdks/python/apache_beam/io/localfilesystem.py | 7 + .../apache_beam/io/localfilesystem_test.py | 4 + sdks/python/apache_beam/io/range_trackers.py | 7 +- sdks/python/apache_beam/io/source_test_utils.py | 8 + sdks/python/apache_beam/io/sources_test.py | 6 +- sdks/python/apache_beam/io/textio.py | 19 +- sdks/python/apache_beam/io/textio_test.py | 7 +- sdks/python/apache_beam/io/tfrecordio.py | 19 +- sdks/python/apache_beam/io/tfrecordio_test.py | 26 +- sdks/python/apache_beam/metrics/__init__.py | 1 + sdks/python/apache_beam/metrics/cells.py | 50 +- sdks/python/apache_beam/metrics/execution.py | 7 +- sdks/python/apache_beam/metrics/metric.py | 4 + sdks/python/apache_beam/metrics/metricbase.py | 2 + sdks/python/apache_beam/options/__init__.py | 16 + .../apache_beam/options/pipeline_options.py | 650 +++++++++++++++ .../options/pipeline_options_test.py | 240 ++++++ .../options/pipeline_options_validator.py | 201 +++++ .../options/pipeline_options_validator_test.py | 343 ++++++++ .../apache_beam/options/value_provider.py | 111 +++ .../apache_beam/options/value_provider_test.py | 145 ++++ sdks/python/apache_beam/pipeline.py | 33 +- sdks/python/apache_beam/pipeline_test.py | 8 +- sdks/python/apache_beam/pvalue.py | 18 +- sdks/python/apache_beam/pvalue_test.py | 2 +- sdks/python/apache_beam/runners/api/__init__.py | 4 +- .../apache_beam/runners/api/beam_fn_api_pb2.py | 2 + .../runners/api/beam_fn_api_pb2_grpc.py | 2 + .../runners/api/beam_runner_api_pb2.py | 2 + sdks/python/apache_beam/runners/common.pxd | 6 +- sdks/python/apache_beam/runners/common.py | 99 ++- sdks/python/apache_beam/runners/common_test.py | 58 ++ .../apache_beam/runners/dataflow/__init__.py | 9 + .../runners/dataflow/dataflow_metrics.py | 114 ++- .../runners/dataflow/dataflow_metrics_test.py | 197 +++-- .../runners/dataflow/dataflow_runner.py | 11 +- .../runners/dataflow/dataflow_runner_test.py | 9 +- .../runners/dataflow/internal/apiclient.py | 17 +- .../runners/dataflow/internal/apiclient_test.py | 23 +- .../runners/dataflow/internal/dependency.py | 22 +- .../dataflow/internal/dependency_test.py | 6 +- .../runners/dataflow/internal/names.py | 8 +- .../runners/dataflow/native_io/iobase.py | 2 + .../runners/dataflow/template_runner_test.py | 2 +- .../runners/dataflow/test_dataflow_runner.py | 5 +- .../apache_beam/runners/direct/__init__.py | 6 +- .../runners/direct/bundle_factory.py | 16 +- sdks/python/apache_beam/runners/direct/clock.py | 5 +- .../consumer_tracking_pipeline_visitor.py | 4 +- .../apache_beam/runners/direct/direct_runner.py | 7 +- .../apache_beam/runners/direct/executor.py | 67 +- .../runners/direct/transform_evaluator.py | 20 +- .../runners/direct/transform_result.py | 4 +- .../runners/direct/watermark_manager.py | 14 +- .../apache_beam/runners/pipeline_context.py | 10 +- .../apache_beam/runners/portability/__init__.py | 2 + .../runners/portability/fn_api_runner.py | 2 + .../portability/maptask_executor_runner.py | 9 +- .../portability/maptask_executor_runner_test.py | 35 +- sdks/python/apache_beam/runners/runner.py | 11 +- sdks/python/apache_beam/runners/runner_test.py | 6 +- .../apache_beam/runners/worker/__init__.py | 2 + .../apache_beam/runners/worker/data_plane.py | 2 + .../apache_beam/runners/worker/log_handler.py | 2 + .../python/apache_beam/runners/worker/logger.py | 2 + .../apache_beam/runners/worker/opcounters.py | 2 + .../runners/worker/operation_specs.py | 2 + .../apache_beam/runners/worker/sdk_worker.py | 4 + .../runners/worker/sdk_worker_main.py | 2 + .../apache_beam/runners/worker/sideinputs.py | 2 + .../runners/worker/statesampler_fake.py | 2 + sdks/python/apache_beam/test_pipeline.py | 163 ---- sdks/python/apache_beam/test_pipeline_test.py | 112 --- sdks/python/apache_beam/testing/__init__.py | 16 + .../testing/data/standard_coders.yaml | 196 +++++ .../testing/data/trigger_transcripts.yaml | 224 ++++++ .../apache_beam/testing/pipeline_verifiers.py | 154 ++++ .../testing/pipeline_verifiers_test.py | 148 ++++ .../python/apache_beam/testing/test_pipeline.py | 168 ++++ .../apache_beam/testing/test_pipeline_test.py | 112 +++ sdks/python/apache_beam/testing/test_stream.py | 175 ++++ .../apache_beam/testing/test_stream_test.py | 83 ++ sdks/python/apache_beam/testing/test_utils.py | 73 ++ sdks/python/apache_beam/testing/util.py | 107 +++ sdks/python/apache_beam/testing/util_test.py | 50 ++ sdks/python/apache_beam/tests/__init__.py | 16 - sdks/python/apache_beam/tests/data/README.md | 20 - .../apache_beam/tests/data/privatekey.p12 | Bin 2452 -> 0 bytes .../apache_beam/tests/data/standard_coders.yaml | 196 ----- .../apache_beam/tests/pipeline_verifiers.py | 146 ---- .../tests/pipeline_verifiers_test.py | 148 ---- sdks/python/apache_beam/tests/test_utils.py | 69 -- sdks/python/apache_beam/transforms/__init__.py | 2 +- .../apache_beam/transforms/combiners_test.py | 4 +- sdks/python/apache_beam/transforms/core.py | 96 ++- .../apache_beam/transforms/create_test.py | 5 +- .../apache_beam/transforms/cy_combiners.py | 5 +- sdks/python/apache_beam/transforms/display.py | 2 +- .../apache_beam/transforms/display_test.py | 2 +- .../python/apache_beam/transforms/ptransform.py | 45 +- .../apache_beam/transforms/ptransform_test.py | 22 +- .../python/apache_beam/transforms/sideinputs.py | 2 + .../apache_beam/transforms/sideinputs_test.py | 4 +- sdks/python/apache_beam/transforms/timeutil.py | 33 +- sdks/python/apache_beam/transforms/trigger.py | 118 +-- .../apache_beam/transforms/trigger_test.py | 50 +- .../transforms/trigger_transcripts.yaml | 224 ------ sdks/python/apache_beam/transforms/util.py | 79 -- sdks/python/apache_beam/transforms/util_test.py | 50 -- sdks/python/apache_beam/transforms/window.py | 51 +- .../apache_beam/transforms/window_test.py | 12 +- .../transforms/write_ptransform_test.py | 4 +- sdks/python/apache_beam/typehints/decorators.py | 17 +- sdks/python/apache_beam/typehints/opcodes.py | 2 + .../apache_beam/typehints/trivial_inference.py | 2 + sdks/python/apache_beam/typehints/typecheck.py | 17 +- .../typehints/typed_pipeline_test.py | 6 +- sdks/python/apache_beam/typehints/typehints.py | 18 + .../apache_beam/typehints/typehints_test.py | 19 +- sdks/python/apache_beam/utils/__init__.py | 5 +- sdks/python/apache_beam/utils/annotations.py | 4 +- sdks/python/apache_beam/utils/counters.py | 5 +- .../apache_beam/utils/pipeline_options.py | 627 --------------- .../apache_beam/utils/pipeline_options_test.py | 240 ------ .../utils/pipeline_options_validator.py | 199 ----- .../utils/pipeline_options_validator_test.py | 343 -------- sdks/python/apache_beam/utils/processes.py | 6 +- sdks/python/apache_beam/utils/profiler.py | 5 +- sdks/python/apache_beam/utils/proto_utils.py | 2 + sdks/python/apache_beam/utils/retry.py | 2 + sdks/python/apache_beam/utils/test_stream.py | 163 ---- .../apache_beam/utils/test_stream_test.py | 83 -- sdks/python/apache_beam/utils/timestamp.py | 5 +- sdks/python/apache_beam/utils/urns.py | 2 + sdks/python/apache_beam/utils/value_provider.py | 103 --- .../apache_beam/utils/value_provider_test.py | 145 ---- sdks/python/apache_beam/utils/windowed_value.py | 4 +- .../apache_beam/utils/windowed_value_test.py | 2 +- sdks/python/apache_beam/version.py | 2 +- sdks/python/generate_pydoc.sh | 15 +- sdks/python/pom.xml | 2 +- sdks/python/setup.py | 4 +- sdks/python/tox.ini | 5 + 1287 files changed, 28743 insertions(+), 31376 deletions(-) ----------------------------------------------------------------------