beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From k...@apache.org
Subject [50/50] [abbrv] beam git commit: This closes #2672: Merge branch 'master' upto commit 686b774 into jstorm-runner
Date Tue, 25 Apr 2017 17:30:45 GMT
This closes #2672: Merge branch 'master' upto commit 686b774 into jstorm-runner

  [BEAM-1993] Remove special unbounded Flink source/sink
  Remove flink-annotations dependency
  Fix Javadoc warnings on Flink Runner
  Enable flink dependency enforcement and make dependencies explicit
  [BEAM-59] Register standard FileSystems wherever we register IOChannelFactories
  [BEAM-1991] Sum.SumDoubleFn => Sum.ofDoubles
  clean up description for sdk_location
  Set the Project of a Table Reference at Runtime
  Only compile HIFIO ITs when compiling with java 8.
  Update assertions of source_test_utils from camelcase to underscore-separated.
  Add no-else return to pylintrc
  Remove getSideInputWindow
  Remove reference to the isStreaming flag
  Javadoc fixups after style guide changes
  Update Dataflow Worker Version
  [BEAM-1922] Close datasource in JdbcIO when possible
  Fix javadoc warnings
  Add javadoc to getCheckpointMark in UnboundedSource
  Removes final minor usages of OldDoFn outside OldDoFn itself
  [BEAM-1915] Removes use of OldDoFn from Apex
  Update Signature of PTransformOverrideFactory
  [BEAM-1964] Fix lint issues and pylint upgrade
  Rename DoFn.Context#sideOutput to output
  [BEAM-1964] Fix lint issues for linter upgrade -3
  [BEAM-1964] Fix lint issues for linter upgrade -2
  Avoi repackaging bigtable classes in dataflow runner.
  ApexRunner: register standard IOs when deserializing pipeline options
  Add PCollections Utilities
  Free PTransform Names if they are being Replaced
  [BEAM-1347] Update protos related to State API for prototyping purposes.
  Update java8 examples pom files to include maven-shade-plugin.
  fix the simplest typo
  [BEAM-1964] Fix lint issues for linter upgrade
  Merge PR#2423: Add Kubernetes scripts for clusters for Performance and Integration tests
of Cassandra and ES for Hadoop Input Format IO
  Remove Triggers.java from SDK entirely
  [BEAM-1708] Improve error message when GCP not installed
  Improve gcloud logging message
  [BEAM-1101, BEAM-1068] Remove service account name credential pipeline options
  Update user_score.py
  Pin versions in tox script
  Improve Empty Create Default Coder Error Message
  Represent a Pipeline via a list of Top-level Transforms
  Test all Known Coders to ensure they Serialize via URN
  [BEAM-1950] Add missing 'static' keyword to MicrobatchSource#initReaderCache
  Move Triggers from sdk-core to runners-core-construction
  [BEAM-1222] Chunk size should be FS dependent
  Move HIFIO k8s scripts into shared dir
  Move jdbc's postgres k8s scripts into shared k8s dir
  Move travis/jenkins folders in a test-infra folder
  [BEAM-911] Mark IO APIs as @Experimental
  Revert "Revert "Revert "Add ValueProvider class for FileBasedSource I/O Transforms"""
  Revert "Throw specialized exception in value providers"
  Removes FlatMapElements.MissingOutputTypeDescriptor
  Removes MapElements.MissingOutputTypeDescriptor
  [BEAM-1882] Update postgres k8 scripts & add scripts for running local dev test
  [BEAM-115] Update timer/state fields on ParDoPayload to use a map field for consistent tag
usage
  Use SdkComponents in WindowingStrategy.toProto
  [BEAM-1722] Move PubsubIO into the google-cloud-platform module
  Triggers: handle missing case
  Clean HFIOWithEmbeddedCassandraTest before Execution
  DataflowRunner: remove dead code
  Throw specialized exception in value providers
  DataflowRunner: send windowing strategy using Runner API proto
  DataflowRunner misc cleanups
  Improve Work Rejection handling
  Remove Orderdness of Input, Output expansions
  Ignore more python build artifacts.
  Fix build breaks caused by overlaps between b615013 and c08b7b1
  Remove Jdk1.8-tests/.toDelete
  Improve HadoopInputFormatIO DisplayData and Cassandra tests
  Add Coder utilities for Proto conversions
  Flip dependency edge between Dataflow runner and IO-GCP
  Move HashingFn to io/common, switch to better hash
  PubsubIO: remove support for BoundedReader
  Bump Dataflow worker to 20170410
  Removes DoFn.ProcessContinuation completely
  Move WindowingStrategies to runners-core-construction
  Fix GroupByKeyInputVisitor for Direct Runner
  Skip query metrics when creating a template
  Upgrade dependencies.
  Add SdkComponents
  Create as custom source
  BEAM-1053 ApexGroupByKeyOperator serialization issues
  enable test_multi_valued_singleton_side_input test
  [BEAM-386] Move UnboundedReadFromBoundedSource to core-construction-java
  BEAM-1390 Update top level README.md to include Apex Runner
  better log message for bigquery temp tables
  [BEAM-1921] Expose connection properties in JdbcIO
  [BEAM-1294] Long running UnboundedSource Readers
  [BEAM-1737] Implement a Single-output ParDo as a Multi-output ParDo with a single output.
  Fix for potentially unclosed streams in ApexYarnLauncher
  TestDataflowRunner: better error handling
  BEAM-1887 Switch Apex ParDo to new DoFn.
  Adds tests for the watermark hold (previously untested)
  Fixes SDF issues re: watermarks and stop/resume
  Clarifies doc of ProcessElement re: HasDefaultTracker
  [BEAM-65] Adds HasDefaultTracker for RestrictionTracker inference
  Cleanup: removes two unused constants
  [BEAM-1823] Improve ValidatesRunner Test Log
  Clean up in textio and tfrecordio
  ...


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/58d4b97c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/58d4b97c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/58d4b97c

Branch: refs/heads/jstorm-runner
Commit: 58d4b97c0a218d01e1b64d5fced693b15d941074
Parents: f6a89b0 f1e170a
Author: Kenneth Knowles <klk@google.com>
Authored: Tue Apr 25 10:29:18 2017 -0700
Committer: Kenneth Knowles <klk@google.com>
Committed: Tue Apr 25 10:29:18 2017 -0700

----------------------------------------------------------------------
 .gitignore                                      |    3 +
 .jenkins/common_job_properties.groovy           |  261 ----
 .../job_beam_PerformanceTests_Dataflow.groovy   |   43 -
 .jenkins/job_beam_PerformanceTests_JDBC.groovy  |   60 -
 .jenkins/job_beam_PerformanceTests_Spark.groovy |   44 -
 ...job_beam_PostCommit_Java_MavenInstall.groovy |   42 -
 ..._PostCommit_Java_ValidatesRunner_Apex.groovy |   48 -
 ...tCommit_Java_ValidatesRunner_Dataflow.groovy |   45 -
 ...PostCommit_Java_ValidatesRunner_Flink.groovy |   43 -
 ...tCommit_Java_ValidatesRunner_Gearpump.groovy |   49 -
 ...PostCommit_Java_ValidatesRunner_Spark.groovy |   44 -
 .../job_beam_PostCommit_Python_Verify.groovy    |   55 -
 .../job_beam_PreCommit_Java_MavenInstall.groovy |   42 -
 .../job_beam_PreCommit_Website_Stage.groovy     |   80 -
 .jenkins/job_beam_PreCommit_Website_Test.groovy |   65 -
 .../job_beam_Release_NightlySnapshot.groovy     |   45 -
 .jenkins/job_seed.groovy                        |   53 -
 .../jenkins/common_job_properties.groovy        |  261 ++++
 .../job_beam_PerformanceTests_Dataflow.groovy   |   43 +
 .../job_beam_PerformanceTests_JDBC.groovy       |   60 +
 .../job_beam_PerformanceTests_Spark.groovy      |   44 +
 ...job_beam_PostCommit_Java_MavenInstall.groovy |   42 +
 ..._PostCommit_Java_ValidatesRunner_Apex.groovy |   48 +
 ...tCommit_Java_ValidatesRunner_Dataflow.groovy |   45 +
 ...PostCommit_Java_ValidatesRunner_Flink.groovy |   43 +
 ...tCommit_Java_ValidatesRunner_Gearpump.groovy |   49 +
 ...PostCommit_Java_ValidatesRunner_Spark.groovy |   44 +
 .../job_beam_PostCommit_Python_Verify.groovy    |   55 +
 .../job_beam_PreCommit_Java_MavenInstall.groovy |   42 +
 .../job_beam_PreCommit_Website_Stage.groovy     |   80 +
 .../job_beam_PreCommit_Website_Test.groovy      |   65 +
 .../job_beam_Release_NightlySnapshot.groovy     |   45 +
 .test-infra/jenkins/job_seed.groovy             |   53 +
 .../cassandra-service-for-local-dev.yaml        |   28 +
 .../cassandra-svc-statefulset.yaml              |  114 ++
 .../LargeITCluster/cassandra-svc-temp.yaml      |   74 +
 .../cassandra/LargeITCluster/data-load.sh       |  122 ++
 .../cassandra/LargeITCluster/show_health.sh     |   47 +
 .../cassandra/LargeITCluster/start-up.sh        |   22 +
 .../cassandra/LargeITCluster/teardown.sh        |   25 +
 .../cassandra-service-for-local-dev.yaml        |   30 +
 .../SmallITCluster/cassandra-svc-rc.yaml        |   74 +
 .../cassandra/SmallITCluster/data-load.sh       |   86 +
 .../cassandra/SmallITCluster/show_health.sh     |   47 +
 .../cassandra/SmallITCluster/start-up.sh        |   23 +
 .../cassandra/SmallITCluster/teardown.sh        |   22 +
 .../kubernetes/cassandra/data-load-setup.sh     |   29 +
 .../elasticsearch-service-for-local-dev.yaml    |   33 +
 .../es-services-deployments.yaml                |  258 +++
 .../LargeProductionCluster/start-up.sh          |   22 +
 .../LargeProductionCluster/teardown.sh          |   21 +
 .../elasticsearch-service-for-local-dev.yaml    |   34 +
 .../SmallITCluster/elasticsearch-svc-rc.yaml    |   96 ++
 .../elasticsearch/SmallITCluster/start-up.sh    |   23 +
 .../elasticsearch/SmallITCluster/teardown.sh    |   21 +
 .../kubernetes/elasticsearch/data-load-setup.sh |   26 +
 .../kubernetes/elasticsearch/data-load.sh       |   33 +
 .../kubernetes/elasticsearch/es_test_data.py    |  299 ++++
 .../kubernetes/elasticsearch/show-health.sh     |   33 +
 .../postgres/postgres-service-for-local-dev.yml |   28 +
 .test-infra/kubernetes/postgres/postgres.yml    |   56 +
 .test-infra/travis/README.md                    |   23 +
 .test-infra/travis/settings.xml                 |   33 +
 .test-infra/travis/test_wordcount.sh            |  125 ++
 .travis.yml                                     |    4 +-
 .travis/README.md                               |   23 -
 .travis/settings.xml                            |   33 -
 .travis/test_wordcount.sh                       |  125 --
 README.md                                       |    1 +
 examples/java8/pom.xml                          |   34 +
 .../beam/examples/MinimalWordCountJava8.java    |    9 +-
 .../beam/examples/complete/game/GameStats.java  |    8 +-
 .../examples/complete/game/LeaderBoard.java     |    2 +-
 .../beam/examples/complete/game/UserScore.java  |    5 +-
 .../examples/MinimalWordCountJava8Test.java     |    9 +-
 .../complete/game/HourlyTeamScoreTest.java      |    5 +-
 .../examples/complete/game/UserScoreTest.java   |    6 +-
 .../apache/beam/runners/apex/ApexRunner.java    |   34 +-
 .../beam/runners/apex/ApexYarnLauncher.java     |  111 +-
 .../translation/ApexPipelineTranslator.java     |    2 +-
 .../FlattenPCollectionTranslator.java           |   13 +-
 .../apex/translation/GroupByKeyTranslator.java  |    4 +-
 .../apex/translation/ParDoTranslator.java       |   30 +-
 .../apex/translation/TranslationContext.java    |   22 +-
 .../translation/WindowAssignTranslator.java     |   58 +-
 .../operators/ApexGroupByKeyOperator.java       |  273 +---
 .../operators/ApexParDoOperator.java            |  235 ++-
 .../operators/ApexProcessFnOperator.java        |  184 +++
 .../translation/utils/ApexStateInternals.java   |   73 +-
 .../apex/translation/utils/NoOpStepContext.java |    2 +-
 .../utils/SerializablePipelineOptions.java      |   13 +-
 .../translation/utils/StateInternalsProxy.java  |   67 +
 .../translation/ApexGroupByKeyOperatorTest.java |    2 +-
 .../apex/translation/ParDoTranslatorTest.java   |   20 +-
 .../utils/ApexStateInternalsTest.java           |   25 +-
 runners/core-construction-java/pom.xml          |   36 +
 .../beam/runners/core/construction/Coders.java  |  174 +++
 .../DeduplicatedFlattenFactory.java             |   79 +-
 .../EmptyFlattenAsCreateFactory.java            |   25 +-
 .../runners/core/construction/PCollections.java |   97 ++
 .../core/construction/PTransformMatchers.java   |    7 +-
 .../construction/PTransformReplacements.java    |   69 +
 .../core/construction/PrimitiveCreate.java      |   18 +-
 .../core/construction/ReplacementOutputs.java   |   63 +-
 .../core/construction/SdkComponents.java        |  159 ++
 .../SingleInputOutputOverrideFactory.java       |   14 +-
 .../runners/core/construction/Triggers.java     |  336 ++++
 .../UnboundedReadFromBoundedSource.java         |  542 +++++++
 .../UnsupportedOverrideFactory.java             |   20 +-
 .../core/construction/WindowingStrategies.java  |  245 +++
 .../runners/core/construction/CodersTest.java   |  163 ++
 .../DeduplicatedFlattenFactoryTest.java         |   24 +-
 .../EmptyFlattenAsCreateFactoryTest.java        |   42 +-
 .../core/construction/PCollectionsTest.java     |  188 +++
 .../construction/PTransformMatchersTest.java    |  132 +-
 .../PTransformReplacementsTest.java             |  131 ++
 .../construction/ReplacementOutputsTest.java    |  109 +-
 .../core/construction/SdkComponentsTest.java    |  157 ++
 .../SingleInputOutputOverrideFactoryTest.java   |   37 +-
 .../runners/core/construction/TriggersTest.java |  111 ++
 .../UnboundedReadFromBoundedSourceTest.java     |  373 +++++
 .../UnsupportedOverrideFactoryTest.java         |   16 +-
 .../construction/WindowingStrategiesTest.java   |  110 ++
 runners/core-java/pom.xml                       |   10 +-
 .../beam/runners/core/AssignWindowsDoFn.java    |   78 -
 .../beam/runners/core/BaseExecutionContext.java |   13 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |  323 ----
 .../apache/beam/runners/core/DoFnRunners.java   |   10 +-
 .../beam/runners/core/ExecutionContext.java     |   13 +-
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |   19 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |    9 +-
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   19 +-
 .../core/GroupAlsoByWindowsAggregators.java     |   28 +
 .../runners/core/GroupAlsoByWindowsDoFn.java    |   46 -
 .../core/LateDataDroppingDoFnRunner.java        |    3 +-
 .../org/apache/beam/runners/core/OldDoFn.java   |   41 +-
 ...eBoundedSplittableProcessElementInvoker.java |  133 +-
 .../beam/runners/core/OutputWindowedValue.java  |   10 +-
 .../beam/runners/core/SimpleDoFnRunner.java     |   59 +-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   63 +-
 .../beam/runners/core/SplittableParDo.java      |   36 +-
 .../core/SplittableProcessElementInvoker.java   |   22 +-
 .../core/UnboundedReadFromBoundedSource.java    |  542 -------
 .../beam/runners/core/WindowingInternals.java   |    8 +-
 .../core/WindowingInternalsAdapters.java        |    8 +-
 .../triggers/AfterWatermarkStateMachine.java    |   14 +-
 ...roupAlsoByWindowViaOutputBufferDoFnTest.java |    4 +-
 .../core/GroupAlsoByWindowsProperties.java      |   37 +-
 .../apache/beam/runners/core/NoOpOldDoFn.java   |    4 +-
 .../apache/beam/runners/core/OldDoFnTest.java   |    4 +-
 ...ndedSplittableProcessElementInvokerTest.java |   27 +-
 .../beam/runners/core/ReduceFnTester.java       |   13 +-
 .../runners/core/SimpleOldDoFnRunnerTest.java   |    4 +-
 .../beam/runners/core/SplittableParDoTest.java  |  238 +--
 .../UnboundedReadFromBoundedSourceTest.java     |  373 -----
 .../direct/BoundedReadEvaluatorFactory.java     |    2 +-
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |   16 +-
 .../beam/runners/direct/DirectGraphVisitor.java |    5 +-
 .../direct/DirectGroupByKeyOverrideFactory.java |   14 +-
 .../direct/ExecutorServiceParallelExecutor.java |    4 +-
 .../runners/direct/FlattenEvaluatorFactory.java |    2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java      |   18 +-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |    4 +-
 .../direct/KeyedPValueTrackingVisitor.java      |   14 +-
 .../beam/runners/direct/ModelEnforcement.java   |   13 +-
 .../beam/runners/direct/ParDoEvaluator.java     |    4 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   20 +-
 .../direct/ParDoMultiOverrideFactory.java       |   29 +-
 ...littableProcessElementsEvaluatorFactory.java |    8 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   10 +-
 .../direct/TestStreamEvaluatorFactory.java      |   20 +-
 .../direct/TransformEvaluatorFactory.java       |   10 +-
 .../direct/TransformExecutorServices.java       |   37 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |    4 +-
 .../runners/direct/ViewEvaluatorFactory.java    |    4 +-
 .../runners/direct/ViewOverrideFactory.java     |   23 +-
 .../beam/runners/direct/WatermarkManager.java   |   19 +-
 .../runners/direct/WindowEvaluatorFactory.java  |    2 +-
 .../direct/WriteWithShardingFactory.java        |   22 +-
 .../runners/direct/DirectGraphVisitorTest.java  |    7 +-
 .../DirectGroupByKeyOverrideFactoryTest.java    |   12 +-
 .../beam/runners/direct/DirectRunnerTest.java   |    5 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |    8 +-
 .../direct/ParDoMultiOverrideFactoryTest.java   |   45 -
 .../StatefulParDoEvaluatorFactoryTest.java      |    2 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |   11 -
 .../direct/TransformExecutorServicesTest.java   |   48 +
 .../runners/direct/ViewOverrideFactoryTest.java |   44 +-
 .../direct/WriteWithShardingFactoryTest.java    |   23 +-
 .../examples/streaming/KafkaIOExamples.java     |  338 ----
 .../KafkaWindowedWordCountExample.java          |  164 --
 runners/flink/pom.xml                           |   45 +-
 runners/flink/runner/pom.xml                    |  109 +-
 .../flink/FlinkBatchTransformTranslators.java   |   36 +-
 .../flink/FlinkBatchTranslationContext.java     |   11 +-
 .../flink/FlinkDetachedRunnerResult.java        |    3 +-
 .../flink/FlinkStreamingPipelineTranslator.java |   61 +-
 .../FlinkStreamingTransformTranslators.java     |  139 +-
 .../flink/FlinkStreamingTranslationContext.java |   12 +-
 .../functions/FlinkDoFnFunction.java            |    4 +-
 .../functions/FlinkNoOpStepContext.java         |    2 +-
 .../functions/FlinkStatefulDoFnFunction.java    |    4 +-
 .../types/EncodedValueTypeInformation.java      |    9 -
 .../flink/translation/types/FlinkCoder.java     |   63 -
 .../utils/SerializedPipelineOptions.java        |    2 +
 .../wrappers/streaming/DoFnOperator.java        |   14 +-
 .../streaming/SingletonKeyedWorkItem.java       |    2 -
 .../streaming/SingletonKeyedWorkItemCoder.java  |    4 +-
 .../streaming/SplittableDoFnOperator.java       |   10 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |    7 +-
 .../streaming/io/UnboundedFlinkSink.java        |  200 ---
 .../streaming/io/UnboundedFlinkSource.java      |  120 --
 .../beam/runners/flink/PipelineOptionsTest.java |    2 +-
 .../flink/streaming/DoFnOperatorTest.java       |   34 +-
 runners/google-cloud-dataflow-java/pom.xml      |   25 +-
 .../dataflow/BatchStatefulParDoOverrides.java   |   51 +-
 .../runners/dataflow/BatchViewOverrides.java    |   23 +-
 .../dataflow/DataflowPipelineTranslator.java    |   37 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  146 +-
 .../DataflowUnboundedReadFromBoundedSource.java |  547 -------
 .../dataflow/PrimitiveParDoSingleFactory.java   |   15 +-
 .../dataflow/ReshuffleOverrideFactory.java      |   12 +-
 .../dataflow/StreamingViewOverrides.java        |   14 +-
 .../runners/dataflow/TransformTranslator.java   |    6 +-
 .../dataflow/testing/TestDataflowRunner.java    |   34 +-
 .../dataflow/BatchViewOverridesTest.java        |    4 +-
 .../dataflow/DataflowPipelineJobTest.java       |    7 +-
 ...aflowUnboundedReadFromBoundedSourceTest.java |   79 -
 .../PrimitiveParDoSingleFactoryTest.java        |   59 +-
 .../testing/TestDataflowRunnerTest.java         |    3 +-
 .../apache/beam/runners/spark/SparkRunner.java  |   20 +-
 .../beam/runners/spark/TestSparkRunner.java     |   21 +-
 .../beam/runners/spark/io/MicrobatchSource.java |  113 +-
 .../beam/runners/spark/io/SourceDStream.java    |   11 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java     |   22 +-
 .../spark/stateful/StateSpecFunctions.java      |    6 +-
 .../runners/spark/translation/DoFnFunction.java |  130 --
 .../spark/translation/EvaluationContext.java    |   11 +-
 .../spark/translation/MultiDoFnFunction.java    |    4 +-
 .../spark/translation/SparkAssignWindowFn.java  |    3 +-
 ...SparkGroupAlsoByWindowViaOutputBufferFn.java |   18 +-
 .../spark/translation/SparkProcessContext.java  |    2 +-
 .../spark/translation/SparkRuntimeContext.java  |    2 +
 .../spark/translation/TransformTranslator.java  |   93 +-
 .../streaming/StreamingTransformTranslator.java |   96 +-
 .../spark/util/SparkSideInputReader.java        |    3 +-
 .../ResumeFromCheckpointStreamingTest.java      |   14 +-
 sdks/common/fn-api/pom.xml                      |    5 -
 .../fn-api/src/main/proto/beam_fn_api.proto     |  174 +--
 .../src/main/proto/beam_runner_api.proto        |   14 +-
 sdks/java/core/pom.xml                          |   44 -
 .../main/java/org/apache/beam/sdk/Pipeline.java |   39 +-
 .../java/org/apache/beam/sdk/io/AvroIO.java     |   13 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 1195 --------------
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  494 ------
 .../beam/sdk/io/PubsubUnboundedSource.java      | 1463 ------------------
 .../main/java/org/apache/beam/sdk/io/Sink.java  |    2 +-
 .../org/apache/beam/sdk/io/UnboundedSource.java |   13 +
 .../beam/sdk/options/PipelineOptions.java       |   12 +-
 .../sdk/runners/PTransformOverrideFactory.java  |   31 +-
 .../apache/beam/sdk/runners/PipelineRunner.java |    7 +-
 .../beam/sdk/runners/TransformHierarchy.java    |   80 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |    2 +
 .../beam/sdk/transforms/AppliedPTransform.java  |   24 +-
 .../org/apache/beam/sdk/transforms/Combine.java |    4 +-
 .../org/apache/beam/sdk/transforms/Create.java  |    7 +-
 .../org/apache/beam/sdk/transforms/DoFn.java    |   96 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   46 +-
 .../beam/sdk/transforms/FlatMapElements.java    |  113 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |    4 +-
 .../apache/beam/sdk/transforms/MapElements.java |   99 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |   82 +-
 .../apache/beam/sdk/transforms/Partition.java   |    2 +-
 .../org/apache/beam/sdk/transforms/Sample.java  |    4 +
 .../transforms/join/KeyedPCollectionTuple.java  |   12 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java    |   47 +-
 .../sdk/transforms/reflect/DoFnInvoker.java     |    4 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |   10 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |   96 +-
 .../splittabledofn/HasDefaultTracker.java       |   30 +
 .../transforms/splittabledofn/OffsetRange.java  |    8 +-
 .../splittabledofn/OffsetRangeTracker.java      |   33 +-
 .../splittabledofn/RestrictionTracker.java      |    8 +
 .../transforms/windowing/AfterWatermark.java    |   14 +-
 .../beam/sdk/transforms/windowing/Triggers.java |  320 ----
 .../beam/sdk/transforms/windowing/WindowFn.java |   13 -
 .../org/apache/beam/sdk/util/PubsubClient.java  |  544 -------
 .../apache/beam/sdk/util/PubsubGrpcClient.java  |  424 -----
 .../apache/beam/sdk/util/PubsubJsonClient.java  |  317 ----
 .../apache/beam/sdk/util/PubsubTestClient.java  |  436 ------
 .../org/apache/beam/sdk/util/Transport.java     |    3 +-
 .../beam/sdk/util/WindowingStrategies.java      |  267 ----
 .../java/org/apache/beam/sdk/values/PBegin.java |    6 +-
 .../apache/beam/sdk/values/PCollectionList.java |   27 +-
 .../beam/sdk/values/PCollectionTuple.java       |   13 +-
 .../java/org/apache/beam/sdk/values/PDone.java  |    6 +-
 .../java/org/apache/beam/sdk/values/PInput.java |    4 +-
 .../org/apache/beam/sdk/values/POutput.java     |    4 +-
 .../java/org/apache/beam/sdk/values/PValue.java |    4 +-
 .../org/apache/beam/sdk/values/PValueBase.java  |    6 +-
 .../apache/beam/sdk/values/TaggedPValue.java    |    5 +
 .../org/apache/beam/sdk/values/TupleTag.java    |   26 +-
 .../apache/beam/sdk/values/TupleTagList.java    |    2 +-
 .../apache/beam/sdk/values/TypeDescriptors.java |   25 +-
 .../org/apache/beam/sdk/values/TypedPValue.java |    4 +-
 .../java/org/apache/beam/sdk/PipelineTest.java  |  108 +-
 .../org/apache/beam/sdk/io/PubsubIOTest.java    |  197 ---
 .../beam/sdk/io/PubsubUnboundedSinkTest.java    |  190 ---
 .../beam/sdk/io/PubsubUnboundedSourceTest.java  |  411 -----
 .../apache/beam/sdk/metrics/MetricsTest.java    |    2 +-
 .../sdk/runners/TransformHierarchyTest.java     |   72 +-
 .../beam/sdk/testing/StaticWindowsTest.java     |   10 +-
 .../apache/beam/sdk/transforms/CreateTest.java  |    8 +-
 .../beam/sdk/transforms/MapElementsTest.java    |   25 +-
 .../apache/beam/sdk/transforms/ParDoTest.java   |  293 ++--
 .../beam/sdk/transforms/SplittableDoFnTest.java |   58 +-
 .../transforms/reflect/DoFnInvokersTest.java    |  153 +-
 .../DoFnSignaturesProcessElementTest.java       |    2 +-
 .../DoFnSignaturesSplittableDoFnTest.java       |  117 +-
 .../splittabledofn/OffsetRangeTrackerTest.java  |   49 +-
 .../sdk/transforms/windowing/TriggersTest.java  |  100 --
 .../apache/beam/sdk/util/PubsubClientTest.java  |  189 ---
 .../beam/sdk/util/PubsubGrpcClientTest.java     |  207 ---
 .../beam/sdk/util/PubsubJsonClientTest.java     |  140 --
 .../beam/sdk/util/PubsubTestClientTest.java     |  114 --
 .../beam/sdk/util/WindowingStrategiesTest.java  |   91 --
 .../beam/sdk/values/PCollectionListTest.java    |   70 +-
 .../beam/sdk/values/PCollectionTupleTest.java   |   13 +-
 .../apache/beam/sdk/values/TypedPValueTest.java |   46 +-
 .../beam/fn/harness/fake/FakeStepContext.java   |    2 +-
 .../control/ProcessBundleHandlerTest.java       |   30 +-
 sdks/java/io/common/pom.xml                     |    4 +
 .../apache/beam/sdk/io/common/HashingFn.java    |  109 ++
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |    2 +
 sdks/java/io/google-cloud-platform/pom.xml      |   75 +-
 .../sdk/io/gcp/bigquery/BatchLoadBigQuery.java  |    7 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java    |   59 +-
 .../io/gcp/bigquery/BigQueryTableSource.java    |   30 +-
 .../sdk/io/gcp/bigquery/WritePartition.java     |    6 +-
 .../beam/sdk/io/gcp/bigquery/WriteResult.java   |   11 +-
 .../beam/sdk/io/gcp/pubsub/PubsubClient.java    |  544 +++++++
 .../sdk/io/gcp/pubsub/PubsubGrpcClient.java     |  424 +++++
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 1014 ++++++++++++
 .../sdk/io/gcp/pubsub/PubsubJsonClient.java     |  319 ++++
 .../sdk/io/gcp/pubsub/PubsubTestClient.java     |  436 ++++++
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |  490 ++++++
 .../io/gcp/pubsub/PubsubUnboundedSource.java    | 1463 ++++++++++++++++++
 .../beam/sdk/io/gcp/pubsub/package-info.java    |   24 +
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java      |    5 +-
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java     |    4 +-
 .../sdk/io/gcp/pubsub/PubsubClientTest.java     |  189 +++
 .../sdk/io/gcp/pubsub/PubsubGrpcClientTest.java |  208 +++
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java    |  189 +++
 .../sdk/io/gcp/pubsub/PubsubJsonClientTest.java |  139 ++
 .../sdk/io/gcp/pubsub/PubsubTestClientTest.java |  114 ++
 .../io/gcp/pubsub/PubsubUnboundedSinkTest.java  |  188 +++
 .../gcp/pubsub/PubsubUnboundedSourceTest.java   |  409 +++++
 .../hadoop/inputformat/HadoopInputFormatIO.java |   36 +-
 .../inputformat/HadoopInputFormatIOTest.java    |   99 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml        |   46 +-
 .../inputformat/HIFIOWithElasticTest.java       |    6 +-
 .../HIFIOWithEmbeddedCassandraTest.java         |  215 +++
 .../hadoop/inputformat/hashing/HashingFn.java   |  109 --
 .../integration/tests/HIFIOCassandraIT.java     |    8 +-
 .../integration/tests/HIFIOElasticIT.java       |    6 +-
 .../SmallITCluster/cassandra-svc-rc.yaml        |   88 --
 .../cassandra/SmallITCluster/start-up.sh        |   21 -
 .../cassandra/SmallITCluster/teardown.sh        |   21 -
 .../kubernetes/cassandra/data-load-setup.sh     |   29 -
 .../resources/kubernetes/cassandra/data-load.sh |   67 -
 .../LargeProductionCluster/es-services.yaml     |  277 ----
 .../LargeProductionCluster/start-up.sh          |   21 -
 .../LargeProductionCluster/teardown.sh          |   20 -
 .../SmallITCluster/elasticsearch-svc-rc.yaml    |   84 -
 .../elasticsearch/SmallITCluster/start-up.sh    |   22 -
 .../elasticsearch/SmallITCluster/teardown.sh    |   20 -
 .../kubernetes/elasticsearch/data-load-setup.sh |   26 -
 .../kubernetes/elasticsearch/data-load.sh       |   33 -
 .../kubernetes/elasticsearch/es_test_data.py    |  299 ----
 .../kubernetes/elasticsearch/show-health.sh     |   25 -
 sdks/java/io/hadoop/pom.xml                     |   12 +-
 .../apache/beam/sdk/io/hdfs/HDFSFileSink.java   |    2 +
 .../apache/beam/sdk/io/hdfs/HDFSFileSource.java |    2 +
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java     |   61 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java |   10 +-
 .../kubernetes/postgres-pod-no-vol.yml          |   32 -
 .../kubernetes/postgres-service-public.yml      |   28 -
 .../jdbc/src/test/resources/kubernetes/setup.sh |   20 -
 .../src/test/resources/kubernetes/teardown.sh   |   20 -
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  |    2 +
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |    2 +
 .../apache/beam/sdk/io/kinesis/KinesisIO.java   |    2 +
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java    |    9 +-
 .../apache/beam/sdk/io/mongodb/MongoDbIO.java   |    2 +
 .../org/apache/beam/sdk/io/mqtt/MqttIO.java     |    2 +
 .../transforms/FlatMapElementsJava8Test.java    |   10 +-
 .../sdk/transforms/MapElementsJava8Test.java    |   10 +-
 sdks/python/.pylintrc                           |    6 +-
 sdks/python/apache_beam/coders/coder_impl.py    |   50 +-
 sdks/python/apache_beam/coders/coders.py        |    6 +-
 .../examples/complete/game/user_score.py        |    8 +-
 .../examples/complete/top_wikipedia_sessions.py |    8 -
 .../examples/cookbook/group_with_coder.py       |    6 +-
 .../examples/snippets/snippets_test.py          |    3 +-
 sdks/python/apache_beam/examples/wordcount.py   |   48 +-
 .../apache_beam/examples/wordcount_debugging.py |    4 -
 sdks/python/apache_beam/internal/gcp/auth.py    |   80 +-
 .../apache_beam/internal/gcp/auth_test.py       |   44 -
 .../apache_beam/internal/gcp/json_value.py      |    6 -
 sdks/python/apache_beam/internal/pickler.py     |   20 +-
 sdks/python/apache_beam/io/avroio_test.py       |   14 +-
 sdks/python/apache_beam/io/concat_source.py     |   74 +-
 .../python/apache_beam/io/concat_source_test.py |   12 +-
 sdks/python/apache_beam/io/filebasedsource.py   |   54 +-
 .../apache_beam/io/filebasedsource_test.py      |   26 +-
 sdks/python/apache_beam/io/fileio.py            |   72 +-
 sdks/python/apache_beam/io/fileio_test.py       |   45 +-
 sdks/python/apache_beam/io/filesystem.py        |    3 +-
 sdks/python/apache_beam/io/filesystem_test.py   |  242 ++-
 sdks/python/apache_beam/io/filesystems_util.py  |   10 +-
 sdks/python/apache_beam/io/gcp/bigquery.py      |   38 +-
 .../io/gcp/datastore/v1/datastoreio.py          |   24 +-
 .../io/gcp/datastore/v1/datastoreio_test.py     |    4 +-
 .../apache_beam/io/gcp/datastore/v1/helper.py   |   16 +-
 .../io/gcp/datastore/v1/query_splitter.py       |    2 +-
 sdks/python/apache_beam/io/gcp/gcsfilesystem.py |    7 +-
 .../io/gcp/tests/bigquery_matcher.py            |    3 +-
 sdks/python/apache_beam/io/iobase.py            |    7 +-
 sdks/python/apache_beam/io/localfilesystem.py   |    3 +-
 sdks/python/apache_beam/io/range_trackers.py    |   19 +-
 sdks/python/apache_beam/io/source_test_utils.py |   79 +-
 .../apache_beam/io/source_test_utils_test.py    |   20 +-
 sdks/python/apache_beam/io/textio.py            |   16 +-
 sdks/python/apache_beam/io/textio_test.py       |   18 +-
 sdks/python/apache_beam/io/tfrecordio.py        |   12 +-
 sdks/python/apache_beam/metrics/cells.py        |   28 +-
 sdks/python/apache_beam/metrics/execution.py    |    3 +-
 sdks/python/apache_beam/metrics/metric.py       |    9 +-
 sdks/python/apache_beam/pipeline.py             |    4 +-
 sdks/python/apache_beam/pipeline_test.py        |   13 +-
 sdks/python/apache_beam/pvalue.py               |   16 +-
 sdks/python/apache_beam/runners/common.py       |    9 +-
 .../runners/dataflow/dataflow_metrics_test.py   |    3 +-
 .../runners/dataflow/dataflow_runner.py         |   39 +-
 .../runners/dataflow/dataflow_runner_test.py    |    2 +-
 .../runners/dataflow/internal/apiclient.py      |    7 +-
 .../runners/dataflow/internal/dependency.py     |    6 +-
 .../runners/dataflow/native_io/iobase_test.py   |    2 +-
 .../runners/dataflow/test_dataflow_runner.py    |   14 +-
 .../runners/direct/bundle_factory.py            |   14 +-
 .../consumer_tracking_pipeline_visitor_test.py  |   22 +-
 .../apache_beam/runners/direct/direct_runner.py |   27 +-
 .../runners/direct/evaluation_context.py        |   10 +-
 .../apache_beam/runners/direct/executor.py      |    9 +-
 .../runners/direct/transform_evaluator.py       |   38 -
 sdks/python/apache_beam/runners/runner.py       |   80 +-
 sdks/python/apache_beam/runners/runner_test.py  |   41 +
 .../apache_beam/tests/pipeline_verifiers.py     |    7 +-
 sdks/python/apache_beam/transforms/combiners.py |   62 +-
 .../apache_beam/transforms/combiners_test.py    |    4 +-
 sdks/python/apache_beam/transforms/core.py      |  192 ++-
 .../apache_beam/transforms/create_test.py       |  121 ++
 sdks/python/apache_beam/transforms/display.py   |    4 +-
 .../apache_beam/transforms/display_test.py      |   36 -
 .../python/apache_beam/transforms/ptransform.py |   38 +-
 .../apache_beam/transforms/ptransform_test.py   |   17 +-
 .../python/apache_beam/transforms/sideinputs.py |   11 +-
 .../apache_beam/transforms/sideinputs_test.py   |    6 +-
 sdks/python/apache_beam/transforms/trigger.py   |   26 +-
 .../apache_beam/transforms/trigger_test.py      |    6 +-
 sdks/python/apache_beam/typehints/decorators.py |   26 +-
 .../apache_beam/typehints/trivial_inference.py  |   26 +-
 .../typehints/trivial_inference_test.py         |    3 +-
 sdks/python/apache_beam/typehints/typecheck.py  |    7 +-
 sdks/python/apache_beam/typehints/typehints.py  |   66 +-
 .../apache_beam/typehints/typehints_test.py     |    7 +-
 .../apache_beam/utils/annotations_test.py       |    2 +-
 sdks/python/apache_beam/utils/path.py           |    3 +-
 .../apache_beam/utils/pipeline_options.py       |  109 +-
 .../apache_beam/utils/pipeline_options_test.py  |   52 +-
 sdks/python/apache_beam/utils/proto_utils.py    |   15 +-
 sdks/python/apache_beam/utils/retry.py          |   14 +-
 sdks/python/apache_beam/utils/timestamp.py      |    6 +-
 sdks/python/apache_beam/utils/value_provider.py |  110 --
 .../apache_beam/utils/value_provider_test.py    |  165 --
 sdks/python/apache_beam/utils/windowed_value.py |   17 +-
 sdks/python/run_postcommit.sh                   |    6 +-
 sdks/python/run_pylint.sh                       |    2 +-
 sdks/python/setup.py                            |    8 +-
 sdks/python/tox.ini                             |   18 +-
 490 files changed, 17307 insertions(+), 17743 deletions(-)
----------------------------------------------------------------------



Mime
View raw message