beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From k...@apache.org
Subject [50/50] [abbrv] beam git commit: This closes #4091: Merge branch 'master' upto commit 269bf89 into mr-runner
Date Tue, 07 Nov 2017 16:55:06 GMT
This closes #4091: Merge branch 'master' upto commit 269bf89 into mr-runner

  mr-runner: re-enable sdks/python module.
  mr-runner: update to 2.3.0-SNAPSHOT.
  Check that bigtableWriter is non-null before calling close().
  Added a preprocessing step to the Cloud Spanner sink.
  Fix Repackaging Configuration in the the DirectRunner
  Migrate shared Fn Execution code to Java7
  update dataflow.version
  [BEAM-3114] Generate text proto config properly in container boot code
  [BEAM-3113] Disable stack trace optimization in java container
  Fix Go package comment for syscallx
  Add all portability protos to Go
  [BEAM-2728] Extension for sketch-based statistics : HyperLogLog
  Add a runners/java-fn-execution module
  Add sdks/java/fn-execution
  [BEAM-3135] Adding futures dependency to python SDK
  Updates BigQueryTableSource to consider data in streaming buffer when determining estimated size.
  Getting AutoValue 1.5.1 working in Beam.
  Add License Header to SqlTypeUtils
  [BEAM-2203] Implement TIMESTAMPADD
  Fix working dir in website precommits
  Do not relocate generated Model Classes
  Updates Python datastore wordcount example to take a dataset parameter.
  Remove obsolete extra parameter
  [BEAM-2482] - CodedValueMutationDetector should use the coders structural value
  Changed the mutation detector to be based on structural value only
  Allows to set a Cloud Spanner host. https://batch-spanner.googleapis.com/ is set as a default host name.
  [BEAM-2468] Reading Kinesis records in the background
  [BEAM-3054] Uses locale-insensitive number formatting in ESIO and tests
  Added VoidSerializer for KafkaIO. Modified KafkaIO.Write.values() to auto add the VoidSerializer for the key.serializer config for kafka producer
  [BEAM-1542] SpannerIO: mutation encoding and size estimation improvements
  [BEAM-2979] Fix a race condition in getWatermark() in KafkaIO.
  [BEAM-3112] Improve error messages in ElasticsearchIO test utils
  [BEAM-3111] Upgrade Elasticsearch to 5.6.3 and clean pom
  Temporarily disable Dataflow pipeline_url metadata
  NonNull by default in sdk/transforms/splittabledofn
  NonNull by default in sdk/transforms/join
  NonNull by default in sdk/transforms/display
  NonNull by default in sdk/transforms/reflect
  NonNull by default in sdk/transforms/windowing
  NonNull by default in sdk/transforms
  Reading spanner schema transform
  Suppress AutoValue warnings in TextIO
  Remove extraneous type arguments in Latest.java
  Remove extraneous type arguments in PAssert
  Remove coveralls invocations from all jobs
  Adds ParseResult.failure()
  Many improvements to TikaIO
  Add missing @RunWith to test.
  [BEAM-2566] Decouple SDK harness from Dataflow runner by elevating experiments and SDK harness configuration to java-sdk-core.
  Add python_requires to setup.py
  CR: [BEAM-3005] Set JVM max heap size in java container
  Update rat exclusion for python and Go protos
  Declare .go and Dockerfile as text in gitattributes
  [BEAM-3005] Set JVM max heap size in java container
  [BEAM-3005] Add resource limits to provision proto
  Stage the pipeline in Python DataflowRunner
  Add zip to the list of accepted extra package file types.
  Fix remaining nullability errors
  Increase Java postcommit timeout to 240
  Manage RAT plugin more centrally; only toggle skipping
  Fix RAT exclusions
  Rearrange .gitignore slightly
  Fix typo in seed job
  Rename seed job so it is first in glob used by prior seed job
  Fix typo in seed jobs
  Increase seed job(s) timeout to 60 minutes
  Increase seed job(s) timeout to 30 minutes
  Make the main seed job standalone
  Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours.
  Implement FnApi side inputs in Python.
  Unit test for label pipeline option
  NonNull by default in metrics
  Ignore findbugs in AutoValue generated classes
  NonNull by default for sdk/testing
  NonNull by default in sdk/state
  NonNull by default in sdk/runners
  NonNull by default in sdk/annotations
  NonNull by default in sdk/coders
  Make Java core SDK root dir NonNull by default
  Add dep on Apache-licensed findbugs-annotations implementation
  [BEAM-2682] Deletes AvroIOTransformTest
  Clone source to a distinguished subdirectory of Jenkins workspace
  Adding lull tracking for python sampler
  Avoids generating proto files for Windows if grpcio-tools is not installed.
  Do not crash when RawPTransform has null spec
  Unit test to repro NPE in PTransformTranslation
  Created Java snippets file
  Add standalone version of seed job
  Pin runner harness also for official BEAM releases.
  Remove duplicate mocking in DataflowRunnerTest
  Add assertion that valid jobs must have staged pipeline
  Stage the pipeline without using a temp file
  Stage the portable pipeline in Dataflow
  Add ability to stage explicit file list
  Improve GcsFileSystem errors messages slightly
  [BEAM-2720] Update kafka client version to 0.11.0.1
  Update PipelineTest.testReplacedNames
  DirectRunner: Replace use of RawPTransform with NotSerializable.forUrn translators
  Wordcount on fnapi pipeline and IT test.
  Clearer getOrDefault style in RehydratedComponents
  ...


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6f22aa7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6f22aa7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6f22aa7

Branch: refs/heads/mr-runner
Commit: b6f22aa76ad217fff6078e9351676895ac793ecc
Parents: 5fa0b14 202a2cb
Author: Kenneth Knowles <kenn@apache.org>
Authored: Tue Nov 7 08:53:31 2017 -0800
Committer: Kenneth Knowles <kenn@apache.org>
Committed: Tue Nov 7 08:53:31 2017 -0800

----------------------------------------------------------------------
 .gitattributes                                  |    2 +
 .gitignore                                      |    9 +-
 .test-infra/jenkins/PreCommit_Pipeline.groovy   |  129 +
 .../jenkins/common_job_properties.groovy        |  205 +-
 .test-infra/jenkins/job_00_seed.groovy          |  114 +
 .test-infra/jenkins/job_beam_Java_Build.groovy  |   82 +
 .../jenkins/job_beam_Java_CodeHealth.groovy     |   39 +
 .../job_beam_Java_IntegrationTest.groovy        |   63 +
 .../jenkins/job_beam_Java_UnitTest.groovy       |   48 +
 ...job_beam_PostCommit_Java_MavenInstall.groovy |   21 +-
 ..._PostCommit_Java_MavenInstall_Windows.groovy |    2 +-
 ...ommit_Python_ValidatesRunner_Dataflow.groovy |   54 +
 .../job_beam_PreCommit_Go_MavenInstall.groovy   |   56 +
 .../job_beam_PreCommit_Java_MavenInstall.groovy |   21 +-
 .../jenkins/job_beam_PreCommit_Pipeline.groovy  |   84 +
 ...ob_beam_PreCommit_Python_MavenInstall.groovy |   56 +
 .../job_beam_PreCommit_Website_Merge.groovy     |    3 +
 .../job_beam_PreCommit_Website_Stage.groovy     |    3 +
 .../job_beam_PreCommit_Website_Test.groovy      |    3 +
 .../jenkins/job_beam_Python_UnitTest.groovy     |   40 +
 .../job_beam_Release_NightlySnapshot.groovy     |   22 +-
 .test-infra/jenkins/job_seed.groovy             |   53 -
 .test-infra/jenkins/job_seed_standalone.groovy  |  114 +
 examples/java/pom.xml                           |    8 +-
 .../apache/beam/examples/WindowedWordCount.java |    1 -
 .../examples/cookbook/BigQueryTornadoes.java    |    2 +-
 .../cookbook/CombinePerKeyExamples.java         |    2 +-
 .../beam/examples/cookbook/FilterExamples.java  |    2 +-
 .../beam/examples/cookbook/JoinExamples.java    |    6 +-
 .../examples/cookbook/MaxPerKeyExamples.java    |    2 +-
 examples/java8/pom.xml                          |   14 +-
 .../complete/game/injector/Injector.java        |    2 +-
 .../complete/game/injector/InjectorUtils.java   |    2 +-
 .../examples/website_snippets/Snippets.java     |   87 +
 .../examples/website_snippets/SnippetsTest.java |  114 +
 examples/pom.xml                                |    2 +-
 model/fn-execution/pom.xml                      |  114 +
 .../src/main/proto/beam_fn_api.proto            |  729 ++++
 .../src/main/proto/beam_provision_api.proto     |   99 +
 .../model/fnexecution/v1/standard_coders.yaml   |  195 +
 model/job-management/pom.xml                    |  114 +
 .../src/main/proto/beam_artifact_api.proto      |  134 +
 .../src/main/proto/beam_job_api.proto           |  174 +
 model/pipeline/pom.xml                          |   89 +
 .../src/main/proto/beam_runner_api.proto        |  843 +++++
 model/pipeline/src/main/proto/endpoints.proto   |   47 +
 .../src/main/proto/standard_window_fns.proto    |   54 +
 model/pom.xml                                   |   40 +
 pom.xml                                         |  368 +-
 runners/apex/pom.xml                            |    9 +-
 .../beam/runners/apex/ApexPipelineOptions.java  |    2 +-
 .../beam/runners/apex/ApexRunnerResult.java     |    4 +-
 .../apex/translation/ParDoTranslator.java       |   24 +-
 .../operators/ApexParDoOperator.java            |   22 +-
 .../translation/utils/ApexStateInternals.java   |   17 +-
 .../apex/translation/utils/ValuesSource.java    |    4 -
 .../beam/runners/apex/ApexYarnLauncherTest.java |    9 +-
 .../apex/examples/UnboundedTextSource.java      |    4 -
 .../FlattenPCollectionTranslatorTest.java       |    1 -
 .../translation/GroupByKeyTranslatorTest.java   |    4 -
 .../apex/translation/ParDoTranslatorTest.java   |    4 +-
 .../translation/utils/CollectionSource.java     |    4 -
 runners/core-construction-java/pom.xml          |   19 +-
 .../construction/ArtifactServiceStager.java     |  244 ++
 .../core/construction/CoderTranslation.java     |   22 +-
 .../core/construction/CombineTranslation.java   |  224 +-
 .../CreatePCollectionViewTranslation.java       |   28 +-
 .../construction/DisplayDataTranslation.java    |    4 +-
 .../core/construction/FlattenTranslator.java    |   12 +-
 .../construction/GroupByKeyTranslation.java     |   17 +-
 .../construction/PCollectionTranslation.java    |   10 +-
 .../construction/PTransformTranslation.java     |  292 +-
 .../core/construction/ParDoTranslation.java     |  392 +-
 .../PipelineOptionsTranslation.java             |   51 +
 .../core/construction/PipelineTranslation.java  |  137 +-
 .../core/construction/ReadTranslation.java      |  113 +-
 .../core/construction/RehydratedComponents.java |    9 +-
 .../construction/RunnerPCollectionView.java     |    2 +-
 .../core/construction/SdkComponents.java        |   17 +-
 .../core/construction/SplittableParDo.java      |    8 +
 .../construction/TestStreamTranslation.java     |  170 +-
 .../TransformPayloadTranslatorRegistrar.java    |    2 +
 .../core/construction/TriggerTranslation.java   |   10 +-
 .../construction/WindowIntoTranslation.java     |   30 +-
 .../WindowingStrategyTranslation.java           |  150 +-
 .../construction/WriteFilesTranslation.java     |  245 +-
 .../construction/ArtifactServiceStagerTest.java |  141 +
 .../core/construction/CoderTranslationTest.java |    4 +-
 .../construction/CombineTranslationTest.java    |   20 +-
 .../CreatePCollectionViewTranslationTest.java   |   10 +-
 .../InMemoryArtifactStagerService.java          |  152 +
 .../PCollectionTranslationTest.java             |    2 +-
 .../construction/PTransformMatchersTest.java    |   15 +-
 .../construction/PTransformTranslationTest.java |   39 +-
 .../core/construction/ParDoTranslationTest.java |    9 +-
 .../PipelineOptionsTranslationTest.java         |  143 +
 .../construction/PipelineTranslationTest.java   |    2 +-
 .../core/construction/ReadTranslationTest.java  |   14 +-
 .../core/construction/SdkComponentsTest.java    |    2 +-
 .../construction/TestStreamTranslationTest.java |   12 +-
 .../construction/WindowIntoTranslationTest.java |    2 +-
 .../WindowingStrategyTranslationTest.java       |    2 +-
 .../construction/WriteFilesTranslationTest.java |   15 +-
 runners/core-java/pom.xml                       |   31 +-
 .../runners/core/InMemoryStateInternals.java    |   39 +-
 .../runners/core/InMemoryTimerInternals.java    |   28 +-
 .../beam/runners/core/SideInputHandler.java     |    2 +-
 .../core/SplittableParDoViaKeyedWorkItems.java  |    7 +
 .../apache/beam/runners/core/StateTable.java    |   40 +-
 .../org/apache/beam/runners/core/StateTags.java |   13 +
 .../core/TestInMemoryStateInternals.java        |    6 +-
 .../apache/beam/runners/core/WatermarkHold.java |    6 +-
 .../runners/core/fn/FnApiControlClient.java     |  148 +
 .../core/fn/FnApiControlClientPoolService.java  |   66 +
 .../beam/runners/core/fn/FnDataReceiver.java    |   33 +
 .../beam/runners/core/fn/FnDataService.java     |   81 +
 .../beam/runners/core/fn/SdkHarnessClient.java  |  172 +
 .../runners/core/fn/SdkHarnessDoFnRunner.java   |  102 +
 .../beam/runners/core/fn/package-info.java      |   22 +
 .../core/metrics/MetricsContainerImpl.java      |   40 +
 .../triggers/AfterWatermarkStateMachine.java    |    8 +-
 .../triggers/TriggerStateMachineRunner.java     |    3 +-
 .../core/triggers/TriggerStateMachines.java     |    2 +-
 .../beam/runners/core/ReduceFnTester.java       |   22 +-
 .../beam/runners/core/StateInternalsTest.java   |   59 +
 .../fn/FnApiControlClientPoolServiceTest.java   |   65 +
 .../runners/core/fn/FnApiControlClientTest.java |  139 +
 .../runners/core/fn/SdkHarnessClientTest.java   |   96 +
 .../core/fn/SdkHarnessDoFnRunnerTest.java       |   73 +
 .../core/metrics/MetricsContainerImplTest.java  |   10 +
 .../AfterWatermarkStateMachineTest.java         |   29 +
 .../triggers/TriggerStateMachineTester.java     |   13 +-
 .../core/triggers/TriggerStateMachinesTest.java |    2 +-
 runners/direct-java/pom.xml                     |   14 +-
 .../CopyOnAccessInMemoryStateInternals.java     |   10 +-
 .../beam/runners/direct/DirectGroupByKey.java   |   16 +-
 .../beam/runners/direct/DirectRunner.java       |    5 +-
 .../GroupAlsoByWindowEvaluatorFactory.java      |    2 +-
 .../beam/runners/direct/MultiStepCombine.java   |   18 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   30 +-
 .../direct/ParDoMultiOverrideFactory.java       |    9 +-
 ...littableProcessElementsEvaluatorFactory.java |    3 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   15 +-
 .../direct/TestStreamEvaluatorFactory.java      |    9 +-
 .../direct/TransformEvaluatorRegistry.java      |   37 +-
 .../runners/direct/ViewOverrideFactory.java     |    8 +-
 .../direct/WriteWithShardingFactory.java        |   37 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java |    4 -
 .../CopyOnAccessInMemoryStateInternalsTest.java |   74 +-
 .../UnboundedReadEvaluatorFactoryTest.java      |    3 -
 .../direct/WriteWithShardingFactoryTest.java    |   11 +-
 runners/flink/pom.xml                           |    4 +-
 .../runners/flink/CreateStreamingFlinkView.java |    3 +
 .../flink/FlinkBatchPipelineTranslator.java     |    3 +-
 .../runners/flink/FlinkPipelineOptions.java     |   11 +
 .../FlinkStreamingTransformTranslators.java     |  244 +-
 .../functions/FlinkAssignContext.java           |   17 +-
 .../translation/types/CoderTypeSerializer.java  |    2 +-
 .../wrappers/streaming/DoFnOperator.java        |  429 ++-
 .../streaming/SplittableDoFnOperator.java       |    4 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |    4 +-
 .../state/FlinkSplitStateInternals.java         |    8 +-
 .../streaming/state/FlinkStateInternals.java    |   13 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   21 +-
 .../flink/streaming/DoFnOperatorTest.java       |  161 +-
 .../flink/streaming/TestCountingSource.java     |    3 -
 .../types/CoderTypeSerializerTest.java          |   79 +
 runners/gcp/gcemd/Dockerfile                    |   30 +
 runners/gcp/gcemd/main.go                       |   85 +
 runners/gcp/gcemd/pom.xml                       |  154 +
 runners/gcp/gcsproxy/Dockerfile                 |   30 +
 runners/gcp/gcsproxy/main.go                    |   91 +
 runners/gcp/gcsproxy/pom.xml                    |  154 +
 runners/gcp/pom.xml                             |   38 +
 runners/gearpump/pom.xml                        |   21 +-
 .../gearpump/GearpumpPipelineResult.java        |    3 +-
 .../translators/GroupByKeyTranslator.java       |    2 +-
 .../gearpump/translators/io/GearpumpSource.java |    7 +-
 .../gearpump/translators/io/ValuesSource.java   |    8 -
 .../translators/utils/DoFnRunnerFactory.java    |    7 +-
 .../translators/utils/TranslatorUtils.java      |   20 -
 .../FlattenPCollectionsTranslatorTest.java      |    6 +
 runners/google-cloud-dataflow-java/pom.xml      |   22 +-
 .../beam/runners/dataflow/AssignWindows.java    |   84 -
 .../dataflow/BatchStatefulParDoOverrides.java   |   71 +-
 .../dataflow/DataflowPipelineTranslator.java    |    5 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   69 +-
 .../runners/dataflow/DataflowRunnerInfo.java    |   91 +-
 .../dataflow/PrimitiveParDoSingleFactory.java   |    2 +-
 .../dataflow/options/CloudDebuggerOptions.java  |    3 +-
 .../options/DataflowPipelineDebugOptions.java   |   23 +-
 .../options/DataflowPipelineOptions.java        |    8 +
 .../options/DataflowProfilingOptions.java       |    3 +-
 .../options/DataflowWorkerLoggingOptions.java   |    6 +
 ...aultCoderCloudObjectTranslatorRegistrar.java |    2 +
 .../beam/runners/dataflow/util/DoFnInfo.java    |  104 -
 .../beam/runners/dataflow/util/GcsStager.java   |   53 +-
 .../runners/dataflow/util/MonitoringUtil.java   |    2 +-
 .../beam/runners/dataflow/util/PackageUtil.java |  581 +--
 .../beam/runners/dataflow/util/Stager.java      |   32 +-
 .../BatchStatefulParDoOverridesTest.java        |   45 +-
 .../DataflowPipelineTranslatorTest.java         |    4 +-
 .../dataflow/DataflowRunnerInfoTest.java        |    9 +
 .../runners/dataflow/DataflowRunnerTest.java    |  107 +-
 .../dataflow/util/MonitoringUtilTest.java       |    4 +-
 .../runners/dataflow/util/PackageUtilTest.java  |  124 +-
 runners/java-fn-execution/pom.xml               |   91 +
 .../beam/runners/fnexecution/ServerFactory.java |  104 +
 .../beam/runners/fnexecution/package-info.java  |   23 +
 .../runners/fnexecution/ServerFactoryTest.java  |  153 +
 runners/local-artifact-service-java/pom.xml     |  116 +
 .../LocalFileSystemArtifactStagerService.java   |  279 ++
 .../beam/artifact/local/package-info.java       |   22 +
 ...ocalFileSystemArtifactStagerServiceTest.java |  301 ++
 runners/map-reduce/pom.xml                      |    2 +-
 runners/pom.xml                                 |    6 +-
 runners/reference/job-server/pom.xml            |   82 +
 .../reference/job/ReferenceRunnerJobServer.java |   77 +
 .../job/ReferenceRunnerJobService.java          |   79 +
 .../runners/reference/job/package-info.java     |   23 +
 .../job/ReferenceRunnerJobServiceTest.java      |   34 +
 runners/reference/pom.xml                       |   39 +
 runners/spark/pom.xml                           |    6 +-
 .../apache/beam/runners/spark/SparkRunner.java  |    5 +-
 .../beam/runners/spark/io/CreateStream.java     |  104 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java     |  819 ++--
 .../spark/stateful/SparkTimerInternals.java     |   21 +-
 .../spark/translation/BoundedDataset.java       |   17 +-
 .../beam/runners/spark/translation/Dataset.java |    3 +-
 .../spark/translation/EvaluationContext.java    |   23 +-
 .../spark/translation/SparkContextFactory.java  |    2 -
 .../translation/StorageLevelPTransform.java     |   37 -
 .../spark/translation/TransformTranslator.java  |   53 +-
 .../spark/translation/TranslationUtils.java     |   78 +
 .../streaming/StreamingTransformTranslator.java |   86 +-
 .../translation/streaming/UnboundedDataset.java |   27 +-
 .../streaming/WatermarkSyncedDStream.java       |  149 +
 .../spark/util/GlobalWatermarkHolder.java       |  302 +-
 .../runners/spark/SparkPipelineStateTest.java   |    4 +-
 .../runners/spark/SparkRunnerDebuggerTest.java  |   11 +-
 .../spark/translation/StorageLevelTest.java     |   75 -
 .../translation/streaming/CreateStreamTest.java |   33 +-
 .../spark/src/test/resources/log4j.properties   |   11 +-
 sdks/CONTAINERS.md                              |  162 +
 sdks/common/fn-api/pom.xml                      |  109 -
 .../fn-api/src/main/proto/beam_fn_api.proto     |  659 ----
 .../org/apache/beam/fn/v1/standard_coders.yaml  |  195 -
 sdks/common/pom.xml                             |   39 -
 sdks/common/runner-api/pom.xml                  |  109 -
 .../src/main/proto/beam_job_api.proto           |  143 -
 .../src/main/proto/beam_runner_api.proto        |  812 ----
 .../src/main/proto/standard_window_fns.proto    |   53 -
 sdks/go/BUILD.md                                |   63 +
 sdks/go/cmd/beamctl/cmd/artifact.go             |   98 +
 sdks/go/cmd/beamctl/cmd/root.go                 |   56 +
 sdks/go/cmd/beamctl/main.go                     |   31 +
 sdks/go/descriptor.xml                          |   29 +
 sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go |  155 +
 sdks/go/pkg/beam/artifact/gcsproxy/staging.go   |  200 +
 sdks/go/pkg/beam/artifact/materialize.go        |  240 ++
 sdks/go/pkg/beam/artifact/materialize_test.go   |  238 ++
 sdks/go/pkg/beam/artifact/server_test.go        |  213 ++
 sdks/go/pkg/beam/artifact/stage.go              |  238 ++
 sdks/go/pkg/beam/artifact/stage_test.go         |   98 +
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go | 2729 ++++++++++++++
 .../fnexecution_v1/beam_provision_api.pb.go     |  306 ++
 sdks/go/pkg/beam/model/gen.go                   |   22 +
 .../jobmanagement_v1/beam_artifact_api.pb.go    |  690 ++++
 .../model/jobmanagement_v1/beam_job_api.pb.go   |  903 +++++
 .../model/pipeline_v1/beam_runner_api.pb.go     | 3491 ++++++++++++++++++
 .../pkg/beam/model/pipeline_v1/endpoints.pb.go  |  160 +
 .../model/pipeline_v1/standard_window_fns.pb.go |  120 +
 sdks/go/pkg/beam/provision/provision_test.go    |   54 +
 sdks/go/pkg/beam/provision/provison.go          |   80 +
 sdks/go/pkg/beam/util/errorx/guarded.go         |   47 +
 sdks/go/pkg/beam/util/execx/exec.go             |   33 +
 sdks/go/pkg/beam/util/gcsx/gcs.go               |   88 +
 sdks/go/pkg/beam/util/grpcx/dial.go             |   37 +
 sdks/go/pkg/beam/util/grpcx/metadata.go         |   55 +
 sdks/go/pkg/beam/util/syscallx/syscall.go       |   27 +
 .../pkg/beam/util/syscallx/syscall_default.go   |   28 +
 sdks/go/pkg/beam/util/syscallx/syscall_linux.go |   38 +
 sdks/go/pom.xml                                 |  163 +
 sdks/java/build-tools/pom.xml                   |    2 +-
 .../src/main/resources/beam/findbugs-filter.xml |   51 +-
 .../resources/docker/file/openjdk7/Dockerfile   |   49 +
 .../docker/file/openjdk7/docker-entrypoint.sh   |   24 +
 .../resources/docker/file/openjdk8/Dockerfile   |   49 +
 .../docker/file/openjdk8/docker-entrypoint.sh   |   24 +
 .../resources/docker/git/openjdk8/Dockerfile    |   53 +
 .../docker/git/openjdk8/docker-entrypoint.sh    |   22 +
 .../resources/docker/release/python2/Dockerfile |   21 +
 sdks/java/container/Dockerfile                  |   28 +
 sdks/java/container/boot.go                     |  134 +
 sdks/java/container/pom.xml                     |  184 +
 sdks/java/core/pom.xml                          |   15 +-
 .../main/java/org/apache/beam/sdk/Pipeline.java |   12 +-
 .../org/apache/beam/sdk/PipelineResult.java     |   14 +-
 .../beam/sdk/annotations/Experimental.java      |    8 +-
 .../beam/sdk/annotations/package-info.java      |    4 +
 .../apache/beam/sdk/coders/BeamRecordCoder.java |  111 +
 .../apache/beam/sdk/coders/CoderRegistry.java   |    6 +
 .../apache/beam/sdk/coders/DefaultCoder.java    |   17 +-
 .../beam/sdk/coders/LengthPrefixCoder.java      |    3 +-
 .../beam/sdk/coders/SerializableCoder.java      |    5 +-
 .../apache/beam/sdk/coders/StructuredCoder.java |   42 +-
 .../org/apache/beam/sdk/coders/VoidCoder.java   |   12 +-
 .../apache/beam/sdk/coders/package-info.java    |    4 +
 .../java/org/apache/beam/sdk/io/AvroIO.java     |  544 ++-
 .../java/org/apache/beam/sdk/io/AvroSink.java   |    2 +-
 .../java/org/apache/beam/sdk/io/AvroSource.java |   32 +-
 .../apache/beam/sdk/io/BlockBasedSource.java    |   27 +-
 .../apache/beam/sdk/io/CompressedSource.java    |  292 +-
 .../org/apache/beam/sdk/io/Compression.java     |  228 ++
 .../org/apache/beam/sdk/io/CountingSource.java  |    3 -
 .../beam/sdk/io/DefaultFilenamePolicy.java      |   25 +-
 .../org/apache/beam/sdk/io/FileBasedSink.java   |  138 +-
 .../org/apache/beam/sdk/io/FileBasedSource.java |   29 +-
 .../java/org/apache/beam/sdk/io/FileIO.java     |  450 +++
 .../apache/beam/sdk/io/GenerateSequence.java    |    3 +-
 .../org/apache/beam/sdk/io/LocalFileSystem.java |    9 +-
 .../main/java/org/apache/beam/sdk/io/Match.java |  156 -
 .../beam/sdk/io/ReadAllViaFileBasedSource.java  |  100 +-
 .../apache/beam/sdk/io/ReadableFileCoder.java   |   50 +
 .../java/org/apache/beam/sdk/io/Source.java     |    2 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |  171 +-
 .../java/org/apache/beam/sdk/io/TextIO.java     |  590 +--
 .../java/org/apache/beam/sdk/io/TextSource.java |  117 +-
 .../java/org/apache/beam/sdk/io/WriteFiles.java |  127 +-
 .../apache/beam/sdk/io/WriteFilesResult.java    |   81 +
 .../org/apache/beam/sdk/io/package-info.java    |    2 +-
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  |    5 +
 .../beam/sdk/io/range/OffsetRangeTracker.java   |   22 +-
 .../org/apache/beam/sdk/metrics/MetricName.java |    7 +
 .../apache/beam/sdk/metrics/package-info.java   |    4 +
 .../DefaultPipelineOptionsRegistrar.java        |    2 +
 .../beam/sdk/options/ExperimentalOptions.java   |   38 +
 .../beam/sdk/options/PipelineOptions.java       |   36 +
 .../sdk/options/PipelineOptionsFactory.java     |   45 +
 .../beam/sdk/options/SdkHarnessOptions.java     |  173 +
 .../apache/beam/sdk/options/ValueProvider.java  |   28 +-
 .../apache/beam/sdk/options/ValueProviders.java |   15 +-
 .../java/org/apache/beam/sdk/package-info.java  |    4 +
 .../beam/sdk/runners/TransformHierarchy.java    |   19 +-
 .../apache/beam/sdk/runners/package-info.java   |    4 +
 .../org/apache/beam/sdk/state/BagState.java     |    6 +
 .../apache/beam/sdk/state/CombiningState.java   |    5 +
 .../apache/beam/sdk/state/GroupingState.java    |   12 +-
 .../org/apache/beam/sdk/state/MapState.java     |   20 +-
 .../apache/beam/sdk/state/ReadableState.java    |    6 +
 .../apache/beam/sdk/state/ReadableStates.java   |    4 +-
 .../org/apache/beam/sdk/state/SetState.java     |   10 +-
 .../org/apache/beam/sdk/state/StateSpecs.java   |    4 +-
 .../org/apache/beam/sdk/state/package-info.java |    4 +
 .../beam/sdk/testing/FileChecksumMatcher.java   |   41 +-
 .../org/apache/beam/sdk/testing/PAssert.java    |   74 +-
 .../apache/beam/sdk/testing/PaneExtractors.java |   25 +-
 .../beam/sdk/testing/SourceTestUtils.java       |   11 +
 .../beam/sdk/testing/SuccessOrFailure.java      |   24 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |   53 +-
 .../beam/sdk/testing/WindowFnTestUtils.java     |  141 +-
 .../apache/beam/sdk/testing/WindowSupplier.java |    4 +-
 .../apache/beam/sdk/testing/package-info.java   |    5 +
 .../sdk/transforms/ApproximateQuantiles.java    |   10 +-
 .../beam/sdk/transforms/ApproximateUnique.java  |    4 +-
 .../org/apache/beam/sdk/transforms/Combine.java |   10 +-
 .../apache/beam/sdk/transforms/CombineFns.java  |    2 +
 .../apache/beam/sdk/transforms/Contextful.java  |  127 +
 .../org/apache/beam/sdk/transforms/Create.java  |    5 +-
 .../apache/beam/sdk/transforms/Distinct.java    |    2 +
 .../apache/beam/sdk/transforms/DoFnTester.java  |   27 +-
 .../beam/sdk/transforms/FlatMapElements.java    |  150 +-
 .../org/apache/beam/sdk/transforms/Latest.java  |    2 +-
 .../apache/beam/sdk/transforms/MapElements.java |   77 +-
 .../org/apache/beam/sdk/transforms/Max.java     |    5 +-
 .../org/apache/beam/sdk/transforms/Min.java     |    8 +-
 .../apache/beam/sdk/transforms/PTransform.java  |    7 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |    8 +-
 .../beam/sdk/transforms/Requirements.java       |   61 +
 .../apache/beam/sdk/transforms/Reshuffle.java   |   47 +
 .../org/apache/beam/sdk/transforms/Top.java     |    5 +-
 .../org/apache/beam/sdk/transforms/View.java    |    9 +-
 .../org/apache/beam/sdk/transforms/Watch.java   |   82 +-
 .../apache/beam/sdk/transforms/WithKeys.java    |   12 +-
 .../sdk/transforms/display/DisplayData.java     |   25 +-
 .../sdk/transforms/display/package-info.java    |    4 +
 .../beam/sdk/transforms/join/CoGbkResult.java   |   15 +-
 .../transforms/join/KeyedPCollectionTuple.java  |   10 +-
 .../beam/sdk/transforms/join/RawUnionValue.java |    8 +-
 .../beam/sdk/transforms/join/package-info.java  |    4 +
 .../beam/sdk/transforms/package-info.java       |    4 +
 .../reflect/ByteBuddyDoFnInvokerFactory.java    |    3 +-
 .../sdk/transforms/reflect/DoFnInvoker.java     |   50 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |    1 +
 .../sdk/transforms/reflect/package-info.java    |    3 +
 .../splittabledofn/OffsetRangeTracker.java      |    5 +-
 .../transforms/splittabledofn/package-info.java |    4 +
 .../windowing/IncompatibleWindowException.java  |    2 +-
 .../MergeOverlappingIntervalWindows.java        |    4 +-
 .../beam/sdk/transforms/windowing/Trigger.java  |   20 +-
 .../beam/sdk/transforms/windowing/Window.java   |    1 +
 .../sdk/transforms/windowing/package-info.java  |    4 +
 .../org/apache/beam/sdk/util/ApiSurface.java    |    2 +
 .../org/apache/beam/sdk/util/CoderUtils.java    |    2 +-
 .../java/org/apache/beam/sdk/util/DoFnInfo.java |  104 +
 .../apache/beam/sdk/util/MutationDetectors.java |   79 +-
 .../beam/sdk/util/SerializableThrowable.java    |   49 +
 .../apache/beam/sdk/util/SerializableUtils.java |   69 +-
 .../org/apache/beam/sdk/values/BeamRecord.java  |  319 ++
 .../apache/beam/sdk/values/BeamRecordType.java  |   96 +
 .../apache/beam/sdk/values/TypeDescriptors.java |   37 +-
 .../java/org/apache/beam/sdk/PipelineTest.java  |   51 +-
 .../apache/beam/sdk/coders/AvroCoderTest.java   |   35 +-
 .../apache/beam/sdk/coders/CommonCoderTest.java |    4 +-
 .../apache/beam/sdk/coders/VoidCoderTest.java   |   11 +-
 .../java/org/apache/beam/sdk/io/AvroIOTest.java |  417 ++-
 .../apache/beam/sdk/io/AvroIOTransformTest.java |  324 --
 .../beam/sdk/io/CompressedSourceTest.java       |   96 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |   43 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java |    3 -
 .../java/org/apache/beam/sdk/io/FileIOTest.java |  313 ++
 .../org/apache/beam/sdk/io/FileSystemsTest.java |   15 +-
 .../beam/sdk/io/OffsetBasedSourceTest.java      |    3 -
 .../java/org/apache/beam/sdk/io/ReadTest.java   |    6 -
 .../java/org/apache/beam/sdk/io/SimpleSink.java |   23 +-
 .../org/apache/beam/sdk/io/TFRecordIOTest.java  |   35 +-
 .../org/apache/beam/sdk/io/TextIOReadTest.java  | 1305 ++++---
 .../org/apache/beam/sdk/io/TextIOWriteTest.java |   40 +-
 .../org/apache/beam/sdk/io/WriteFilesTest.java  |   11 +-
 .../sdk/io/range/ByteKeyRangeTrackerTest.java   |   23 +
 .../apache/beam/sdk/metrics/MetricsTest.java    |   28 +
 .../sdk/options/PipelineOptionsFactoryTest.java |   39 +
 .../beam/sdk/options/PipelineOptionsTest.java   |   11 +
 .../sdk/options/ProxyInvocationHandlerTest.java |    4 +-
 .../beam/sdk/options/SdkHarnessOptionsTest.java |   76 +
 .../beam/sdk/options/ValueProviderTest.java     |   38 +-
 .../runners/dataflow/TestCountingSource.java    |    3 -
 .../sdk/testing/InterceptingUrlClassLoader.java |   57 +
 .../apache/beam/sdk/testing/PAssertTest.java    |   41 +-
 .../beam/sdk/testing/PaneExtractorsTest.java    |    7 +-
 .../beam/sdk/testing/TestPipelineTest.java      |   37 +-
 .../transforms/ApproximateQuantilesTest.java    |  528 +--
 .../apache/beam/sdk/transforms/CreateTest.java  |   22 +-
 .../sdk/transforms/FlatMapElementsTest.java     |   35 +-
 .../apache/beam/sdk/transforms/FlattenTest.java |   35 +
 .../beam/sdk/transforms/MapElementsTest.java    |   42 +-
 .../apache/beam/sdk/transforms/ParDoTest.java   |   44 +-
 .../apache/beam/sdk/transforms/WatchTest.java   |   46 +-
 .../beam/sdk/transforms/WithKeysTest.java       |   30 +-
 .../transforms/reflect/DoFnInvokersTest.java    |   12 +-
 .../sdk/transforms/windowing/WindowTest.java    |  125 +-
 .../beam/sdk/util/MutationDetectorsTest.java    |   56 +
 .../beam/sdk/util/SerializableUtilsTest.java    |   60 +
 .../beam/sdk/values/TypeDescriptorsTest.java    |   17 +-
 .../google-cloud-platform-core/pom.xml          |    2 +-
 .../extensions/gcp/storage/GcsFileSystem.java   |    5 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |    6 +-
 .../org/apache/beam/sdk/util/GcsUtilTest.java   |   45 +
 sdks/java/extensions/jackson/pom.xml            |    2 +-
 sdks/java/extensions/join-library/pom.xml       |    2 +-
 sdks/java/extensions/pom.xml                    |    4 +-
 sdks/java/extensions/protobuf/pom.xml           |    2 +-
 sdks/java/extensions/sketching/pom.xml          |  104 +
 .../sketching/ApproximateDistinct.java          |  573 +++
 .../sdk/extensions/sketching/package-info.java  |   22 +
 .../sketching/ApproximateDistinctTest.java      |  209 ++
 sdks/java/extensions/sorter/pom.xml             |    2 +-
 sdks/java/extensions/sql/NOTICE                 |   45 +
 sdks/java/extensions/sql/pom.xml                |  275 ++
 .../sdk/extensions/sql/BeamRecordSqlType.java   |  186 +
 .../apache/beam/sdk/extensions/sql/BeamSql.java |  250 ++
 .../sdk/extensions/sql/BeamSqlRecordHelper.java |  217 ++
 .../beam/sdk/extensions/sql/BeamSqlUdf.java     |   43 +
 .../extensions/sql/example/BeamSqlExample.java  |  104 +
 .../extensions/sql/example/package-info.java    |   23 +
 .../sdk/extensions/sql/impl/BeamSqlCli.java     |   65 +
 .../sdk/extensions/sql/impl/BeamSqlEnv.java     |  135 +
 .../interpreter/BeamSqlExpressionExecutor.java  |   44 +
 .../sql/impl/interpreter/BeamSqlFnExecutor.java |  458 +++
 .../operator/BeamSqlCaseExpression.java         |   64 +
 .../operator/BeamSqlCastExpression.java         |  132 +
 .../interpreter/operator/BeamSqlExpression.java |   79 +
 .../operator/BeamSqlInputRefExpression.java     |   48 +
 .../interpreter/operator/BeamSqlPrimitive.java  |  157 +
 .../operator/BeamSqlReinterpretExpression.java  |   55 +
 .../operator/BeamSqlUdfExpression.java          |   92 +
 .../operator/BeamSqlWindowEndExpression.java    |   48 +
 .../operator/BeamSqlWindowExpression.java       |   51 +
 .../operator/BeamSqlWindowStartExpression.java  |   49 +
 .../sql/impl/interpreter/operator/UdafImpl.java |   87 +
 .../arithmetic/BeamSqlArithmeticExpression.java |  124 +
 .../arithmetic/BeamSqlDivideExpression.java     |   37 +
 .../arithmetic/BeamSqlMinusExpression.java      |   36 +
 .../arithmetic/BeamSqlModExpression.java        |   36 +
 .../arithmetic/BeamSqlMultiplyExpression.java   |   36 +
 .../arithmetic/BeamSqlPlusExpression.java       |   36 +
 .../operator/arithmetic/package-info.java       |   22 +
 .../comparison/BeamSqlCompareExpression.java    |   97 +
 .../comparison/BeamSqlEqualsExpression.java     |   49 +
 .../BeamSqlGreaterThanExpression.java           |   49 +
 .../BeamSqlGreaterThanOrEqualsExpression.java   |   49 +
 .../comparison/BeamSqlIsNotNullExpression.java  |   54 +
 .../comparison/BeamSqlIsNullExpression.java     |   54 +
 .../comparison/BeamSqlLessThanExpression.java   |   49 +
 .../BeamSqlLessThanOrEqualsExpression.java      |   49 +
 .../comparison/BeamSqlNotEqualsExpression.java  |   49 +
 .../operator/comparison/package-info.java       |   22 +
 .../date/BeamSqlCurrentDateExpression.java      |   45 +
 .../date/BeamSqlCurrentTimeExpression.java      |   53 +
 .../date/BeamSqlCurrentTimestampExpression.java |   49 +
 .../date/BeamSqlDateCeilExpression.java         |   55 +
 .../date/BeamSqlDateFloorExpression.java        |   55 +
 .../date/BeamSqlDatetimePlusExpression.java     |  129 +
 .../operator/date/BeamSqlExtractExpression.java |  102 +
 .../date/BeamSqlIntervalMultiplyExpression.java |  103 +
 .../operator/date/TimeUnitUtils.java            |   54 +
 .../interpreter/operator/date/package-info.java |   22 +
 .../operator/logical/BeamSqlAndExpression.java  |   48 +
 .../logical/BeamSqlLogicalExpression.java       |   46 +
 .../operator/logical/BeamSqlNotExpression.java  |   54 +
 .../operator/logical/BeamSqlOrExpression.java   |   48 +
 .../operator/logical/package-info.java          |   22 +
 .../operator/math/BeamSqlAbsExpression.java     |   74 +
 .../operator/math/BeamSqlAcosExpression.java    |   40 +
 .../operator/math/BeamSqlAsinExpression.java    |   40 +
 .../operator/math/BeamSqlAtan2Expression.java   |   42 +
 .../operator/math/BeamSqlAtanExpression.java    |   40 +
 .../operator/math/BeamSqlCeilExpression.java    |   45 +
 .../operator/math/BeamSqlCosExpression.java     |   40 +
 .../operator/math/BeamSqlCotExpression.java     |   40 +
 .../operator/math/BeamSqlDegreesExpression.java |   40 +
 .../operator/math/BeamSqlExpExpression.java     |   40 +
 .../operator/math/BeamSqlFloorExpression.java   |   45 +
 .../operator/math/BeamSqlLnExpression.java      |   40 +
 .../operator/math/BeamSqlLogExpression.java     |   40 +
 .../math/BeamSqlMathBinaryExpression.java       |   65 +
 .../math/BeamSqlMathUnaryExpression.java        |   60 +
 .../operator/math/BeamSqlPiExpression.java      |   43 +
 .../operator/math/BeamSqlPowerExpression.java   |   44 +
 .../operator/math/BeamSqlRadiansExpression.java |   40 +
 .../operator/math/BeamSqlRandExpression.java    |   55 +
 .../math/BeamSqlRandIntegerExpression.java      |   59 +
 .../operator/math/BeamSqlRoundExpression.java   |  107 +
 .../operator/math/BeamSqlSignExpression.java    |   72 +
 .../operator/math/BeamSqlSinExpression.java     |   40 +
 .../operator/math/BeamSqlTanExpression.java     |   40 +
 .../math/BeamSqlTruncateExpression.java         |   75 +
 .../interpreter/operator/math/package-info.java |   22 +
 .../impl/interpreter/operator/package-info.java |   22 +
 .../string/BeamSqlCharLengthExpression.java     |   40 +
 .../string/BeamSqlConcatExpression.java         |   63 +
 .../string/BeamSqlInitCapExpression.java        |   56 +
 .../operator/string/BeamSqlLowerExpression.java |   40 +
 .../string/BeamSqlOverlayExpression.java        |   77 +
 .../string/BeamSqlPositionExpression.java       |   73 +
 .../string/BeamSqlStringUnaryExpression.java    |   44 +
 .../string/BeamSqlSubstringExpression.java      |   83 +
 .../operator/string/BeamSqlTrimExpression.java  |  102 +
 .../operator/string/BeamSqlUpperExpression.java |   40 +
 .../operator/string/package-info.java           |   22 +
 .../sql/impl/interpreter/package-info.java      |   22 +
 .../sdk/extensions/sql/impl/package-info.java   |   22 +
 .../sql/impl/planner/BeamQueryPlanner.java      |  168 +
 .../sql/impl/planner/BeamRelDataTypeSystem.java |   40 +
 .../sql/impl/planner/BeamRuleSets.java          |   75 +
 .../sql/impl/planner/package-info.java          |   24 +
 .../sql/impl/rel/BeamAggregationRel.java        |  182 +
 .../extensions/sql/impl/rel/BeamFilterRel.java  |   69 +
 .../extensions/sql/impl/rel/BeamIOSinkRel.java  |   75 +
 .../sql/impl/rel/BeamIOSourceRel.java           |   62 +
 .../sql/impl/rel/BeamIntersectRel.java          |   58 +
 .../extensions/sql/impl/rel/BeamJoinRel.java    |  298 ++
 .../sql/impl/rel/BeamLogicalConvention.java     |   72 +
 .../extensions/sql/impl/rel/BeamMinusRel.java   |   56 +
 .../extensions/sql/impl/rel/BeamProjectRel.java |   80 +
 .../extensions/sql/impl/rel/BeamRelNode.java    |   39 +
 .../sql/impl/rel/BeamSetOperatorRelBase.java    |   98 +
 .../extensions/sql/impl/rel/BeamSortRel.java    |  235 ++
 .../sql/impl/rel/BeamSqlRelUtils.java           |   72 +
 .../extensions/sql/impl/rel/BeamUnionRel.java   |   88 +
 .../extensions/sql/impl/rel/BeamValuesRel.java  |   79 +
 .../extensions/sql/impl/rel/package-info.java   |   23 +
 .../sql/impl/rule/BeamAggregationRule.java      |  162 +
 .../sql/impl/rule/BeamFilterRule.java           |   49 +
 .../sql/impl/rule/BeamIOSinkRule.java           |   81 +
 .../sql/impl/rule/BeamIOSourceRule.java         |   49 +
 .../sql/impl/rule/BeamIntersectRule.java        |   50 +
 .../extensions/sql/impl/rule/BeamJoinRule.java  |   53 +
 .../extensions/sql/impl/rule/BeamMinusRule.java |   50 +
 .../sql/impl/rule/BeamProjectRule.java          |   50 +
 .../extensions/sql/impl/rule/BeamSortRule.java  |   51 +
 .../extensions/sql/impl/rule/BeamUnionRule.java |   50 +
 .../sql/impl/rule/BeamValuesRule.java           |   48 +
 .../extensions/sql/impl/rule/package-info.java  |   23 +
 .../sql/impl/schema/BaseBeamTable.java          |   35 +
 .../extensions/sql/impl/schema/BeamIOType.java  |   28 +
 .../sql/impl/schema/BeamPCollectionTable.java   |   63 +
 .../sql/impl/schema/BeamSqlTable.java           |   54 +
 .../sql/impl/schema/BeamTableUtils.java         |  118 +
 .../impl/schema/kafka/BeamKafkaCSVTable.java    |  109 +
 .../sql/impl/schema/kafka/BeamKafkaTable.java   |  109 +
 .../sql/impl/schema/kafka/package-info.java     |   22 +
 .../sql/impl/schema/package-info.java           |   22 +
 .../sql/impl/schema/text/BeamTextCSVTable.java  |   70 +
 .../schema/text/BeamTextCSVTableIOReader.java   |   58 +
 .../schema/text/BeamTextCSVTableIOWriter.java   |   58 +
 .../sql/impl/schema/text/BeamTextTable.java     |   41 +
 .../sql/impl/schema/text/package-info.java      |   22 +
 .../transform/BeamAggregationTransforms.java    |  311 ++
 .../impl/transform/BeamBuiltinAggregations.java |  557 +++
 .../sql/impl/transform/BeamJoinTransforms.java  |  161 +
 .../transform/BeamSetOperatorsTransforms.java   |  111 +
 .../sql/impl/transform/BeamSqlFilterFn.java     |   63 +
 .../transform/BeamSqlOutputToConsoleFn.java     |   41 +
 .../sql/impl/transform/BeamSqlProjectFn.java    |   72 +
 .../sql/impl/transform/package-info.java        |   22 +
 .../extensions/sql/impl/utils/CalciteUtils.java |  113 +
 .../extensions/sql/impl/utils/SqlTypeUtils.java |   63 +
 .../extensions/sql/impl/utils/package-info.java |   22 +
 .../beam/sdk/extensions/sql/package-info.java   |   22 +
 .../sql/src/main/resources/log4j.properties     |   23 +
 ...ged.org.codehaus.commons.compiler.properties |   18 +
 .../extensions/sql/BeamSqlApiSurfaceTest.java   |   57 +
 .../sql/BeamSqlDslAggregationTest.java          |  400 ++
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java |  136 +
 .../extensions/sql/BeamSqlDslFilterTest.java    |  155 +
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java  |  188 +
 .../extensions/sql/BeamSqlDslProjectTest.java   |  227 ++
 .../extensions/sql/BeamSqlDslUdfUdafTest.java   |  139 +
 .../beam/sdk/extensions/sql/TestUtils.java      |  190 +
 .../impl/interpreter/BeamSqlFnExecutorTest.java |  446 +++
 .../interpreter/BeamSqlFnExecutorTestBase.java  |   88 +
 .../operator/BeamNullExperssionTest.java        |   55 +
 .../operator/BeamSqlAndOrExpressionTest.java    |   61 +
 .../operator/BeamSqlCaseExpressionTest.java     |   93 +
 .../operator/BeamSqlCastExpressionTest.java     |  129 +
 .../operator/BeamSqlCompareExpressionTest.java  |  115 +
 .../operator/BeamSqlInputRefExpressionTest.java |   57 +
 .../operator/BeamSqlPrimitiveTest.java          |   59 +
 .../BeamSqlReinterpretExpressionTest.java       |   75 +
 .../operator/BeamSqlUdfExpressionTest.java      |   51 +
 .../BeamSqlArithmeticExpressionTest.java        |  237 ++
 .../date/BeamSqlCurrentDateExpressionTest.java  |   38 +
 .../date/BeamSqlCurrentTimeExpressionTest.java  |   39 +
 .../BeamSqlCurrentTimestampExpressionTest.java  |   39 +
 .../date/BeamSqlDateCeilExpressionTest.java     |   50 +
 .../date/BeamSqlDateExpressionTestBase.java     |   52 +
 .../date/BeamSqlDateFloorExpressionTest.java    |   49 +
 .../date/BeamSqlDatetimePlusExpressionTest.java |  155 +
 .../date/BeamSqlExtractExpressionTest.java      |  103 +
 .../BeamSqlIntervalMultiplyExpressionTest.java  |  107 +
 .../operator/date/TimeUnitUtilsTest.java        |   54 +
 .../logical/BeamSqlNotExpressionTest.java       |   47 +
 .../math/BeamSqlMathBinaryExpressionTest.java   |  215 ++
 .../math/BeamSqlMathUnaryExpressionTest.java    |  312 ++
 .../string/BeamSqlCharLengthExpressionTest.java |   44 +
 .../string/BeamSqlConcatExpressionTest.java     |   66 +
 .../string/BeamSqlInitCapExpressionTest.java    |   54 +
 .../string/BeamSqlLowerExpressionTest.java      |   44 +
 .../string/BeamSqlOverlayExpressionTest.java    |   87 +
 .../string/BeamSqlPositionExpressionTest.java   |   84 +
 .../BeamSqlStringUnaryExpressionTest.java       |   52 +
 .../string/BeamSqlSubstringExpressionTest.java  |  101 +
 .../string/BeamSqlTrimExpressionTest.java       |  103 +
 .../string/BeamSqlUpperExpressionTest.java      |   44 +
 .../extensions/sql/impl/rel/BaseRelTest.java    |   34 +
 .../sql/impl/rel/BeamIntersectRelTest.java      |  118 +
 .../rel/BeamJoinRelBoundedVsBoundedTest.java    |  203 +
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java  |  240 ++
 .../BeamJoinRelUnboundedVsUnboundedTest.java    |  218 ++
 .../sql/impl/rel/BeamMinusRelTest.java          |  117 +
 .../impl/rel/BeamSetOperatorRelBaseTest.java    |  105 +
 .../sql/impl/rel/BeamSortRelTest.java           |  257 ++
 .../sql/impl/rel/BeamUnionRelTest.java          |  103 +
 .../sql/impl/rel/BeamValuesRelTest.java         |  104 +
 .../sdk/extensions/sql/impl/rel/CheckSize.java  |   41 +
 .../sql/impl/schema/BeamSqlRowCoderTest.java    |   77 +
 .../schema/kafka/BeamKafkaCSVTableTest.java     |  107 +
 .../impl/schema/text/BeamTextCSVTableTest.java  |  176 +
 .../transform/BeamAggregationTransformTest.java |  453 +++
 .../schema/transform/BeamTransformBaseTest.java |   97 +
 .../sql/impl/utils/SqlTypeUtilsTest.java        |   76 +
 ...amSqlArithmeticOperatorsIntegrationTest.java |  165 +
 ...mSqlBuiltinFunctionsIntegrationTestBase.java |  168 +
 ...amSqlComparisonOperatorsIntegrationTest.java |  329 ++
 ...mSqlConditionalFunctionsIntegrationTest.java |   60 +
 .../BeamSqlDateFunctionsIntegrationTest.java    |  125 +
 .../BeamSqlLogicalFunctionsIntegrationTest.java |   43 +
 .../BeamSqlMathFunctionsIntegrationTest.java    |  351 ++
 .../BeamSqlStringFunctionsIntegrationTest.java  |   51 +
 .../extensions/sql/mock/MockedBoundedTable.java |  134 +
 .../sdk/extensions/sql/mock/MockedTable.java    |   42 +
 .../sql/mock/MockedUnboundedTable.java          |  110 +
 sdks/java/fn-execution/pom.xml                  |   82 +
 .../harness/channel/ManagedChannelFactory.java  |   82 +
 .../harness/channel/SocketAddressFactory.java   |   64 +
 .../beam/harness/channel/package-info.java      |   22 +
 .../channel/ManagedChannelFactoryTest.java      |   71 +
 .../channel/SocketAddressFactoryTest.java       |   56 +
 .../org/apache/beam/harness/test/Consumer.java  |   26 +
 .../org/apache/beam/harness/test/Supplier.java  |   26 +
 .../apache/beam/harness/test/TestExecutors.java |   93 +
 .../beam/harness/test/TestExecutorsTest.java    |  175 +
 .../apache/beam/harness/test/TestStreams.java   |  185 +
 .../beam/harness/test/TestStreamsTest.java      |  109 +
 sdks/java/harness/pom.xml                       |  128 +-
 .../beam/fn/harness/BeamFnDataReadRunner.java   |   15 +-
 .../beam/fn/harness/BeamFnDataWriteRunner.java  |   15 +-
 .../beam/fn/harness/BoundedSourceRunner.java    |   14 +-
 .../apache/beam/fn/harness/FnApiDoFnRunner.java |  398 +-
 .../org/apache/beam/fn/harness/FnHarness.java   |   64 +-
 .../org/apache/beam/fn/harness/IdGenerator.java |   33 +
 .../fn/harness/PTransformRunnerFactory.java     |    7 +-
 .../harness/channel/ManagedChannelFactory.java  |   86 -
 .../harness/channel/SocketAddressFactory.java   |   64 -
 .../beam/fn/harness/channel/package-info.java   |   22 -
 .../fn/harness/control/BeamFnControlClient.java |   15 +-
 .../harness/control/ProcessBundleHandler.java   |  154 +-
 .../fn/harness/control/RegisterHandler.java     |    8 +-
 .../BeamFnDataBufferingOutboundObserver.java    |    8 +-
 .../beam/fn/harness/data/BeamFnDataClient.java  |    7 +-
 .../fn/harness/data/BeamFnDataGrpcClient.java   |   19 +-
 .../harness/data/BeamFnDataGrpcMultiplexer.java |   24 +-
 .../harness/data/BeamFnDataInboundObserver.java |    4 +-
 .../beam/fn/harness/fake/FakeStepContext.java   |   39 -
 .../beam/fn/harness/fake/package-info.java      |   22 -
 .../beam/fn/harness/fn/ThrowingBiConsumer.java  |   33 +
 .../fn/harness/logging/BeamFnLoggingClient.java |  202 +-
 .../beam/fn/harness/state/BagUserState.java     |  121 +
 .../fn/harness/state/BeamFnStateClient.java     |   39 +
 .../state/BeamFnStateGrpcClientCache.java       |  173 +
 .../state/LazyCachingIteratorToIterable.java    |   72 +
 .../harness/state/StateFetchingIterators.java   |  126 +
 .../beam/fn/harness/state/package-info.java     |   22 +
 .../harness/stream/BufferingStreamObserver.java |   16 +-
 .../beam/fn/harness/stream/DataStreams.java     |   73 +-
 .../harness/stream/StreamObserverFactory.java   |    4 +-
 .../fn/harness/BeamFnDataReadRunnerTest.java    |   19 +-
 .../fn/harness/BeamFnDataWriteRunnerTest.java   |   15 +-
 .../fn/harness/BoundedSourceRunnerTest.java     |   21 +-
 .../beam/fn/harness/FnApiDoFnRunnerTest.java    |  247 +-
 .../apache/beam/fn/harness/FnHarnessTest.java   |   25 +-
 .../apache/beam/fn/harness/IdGeneratorTest.java |   40 +
 .../channel/ManagedChannelFactoryTest.java      |   74 -
 .../channel/SocketAddressFactoryTest.java       |   56 -
 .../control/BeamFnControlClientTest.java        |   13 +-
 .../control/ProcessBundleHandlerTest.java       |  160 +-
 .../fn/harness/control/RegisterHandlerTest.java |   10 +-
 ...BeamFnDataBufferingOutboundObserverTest.java |    4 +-
 .../harness/data/BeamFnDataGrpcClientTest.java  |   30 +-
 .../data/BeamFnDataGrpcMultiplexerTest.java     |    9 +-
 .../data/BeamFnDataInboundObserverTest.java     |    2 +-
 .../logging/BeamFnLoggingClientTest.java        |  136 +-
 .../beam/fn/harness/state/BagUserStateTest.java |  106 +
 .../state/BeamFnStateGrpcClientCacheTest.java   |  234 ++
 .../fn/harness/state/FakeBeamFnStateClient.java |  110 +
 .../LazyCachingIteratorToIterableTest.java      |   76 +
 .../state/StateFetchingIteratorsTest.java       |   99 +
 .../stream/BufferingStreamObserverTest.java     |   12 +-
 .../beam/fn/harness/stream/DataStreamsTest.java |  165 +-
 .../stream/DirectStreamObserverTest.java        |    8 +-
 .../beam/fn/harness/test/TestExecutors.java     |   85 -
 .../beam/fn/harness/test/TestExecutorsTest.java |  160 -
 .../beam/fn/harness/test/TestStreams.java       |  162 -
 .../beam/fn/harness/test/TestStreamsTest.java   |   84 -
 sdks/java/io/amqp/pom.xml                       |   24 +-
 .../org/apache/beam/sdk/io/amqp/AmqpIO.java     |   33 +-
 .../org/apache/beam/sdk/io/amqp/AmqpIOTest.java |  112 +-
 sdks/java/io/cassandra/pom.xml                  |    2 +-
 .../beam/sdk/io/cassandra/CassandraIO.java      |   64 +-
 sdks/java/io/common/pom.xml                     |    2 +-
 .../sdk/io/common/IOTestPipelineOptions.java    |    6 +
 .../elasticsearch-tests-2/pom.xml               |   60 +
 .../src/test/contrib/create_elk_container.sh    |   24 +
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java |  123 +
 .../io/elasticsearch/ElasticsearchIOTest.java   |  185 +
 .../elasticsearch-tests-5/pom.xml               |  124 +
 .../src/test/contrib/create_elk_container.sh    |   24 +
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java |  122 +
 .../io/elasticsearch/ElasticsearchIOTest.java   |  185 +
 .../org/elasticsearch/bootstrap/JarHell.java    |   39 +
 .../elasticsearch-tests-common/pom.xml          |   77 +
 .../elasticsearch/ElasticSearchIOTestUtils.java |  141 +
 .../elasticsearch/ElasticsearchIOITCommon.java  |   92 +
 .../ElasticsearchIOTestCommon.java              |  306 ++
 sdks/java/io/elasticsearch-tests/pom.xml        |  144 +
 sdks/java/io/elasticsearch/pom.xml              |  228 +-
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |  448 ++-
 .../beam/sdk/io/elasticsearch/package-info.java |    1 -
 .../src/test/contrib/create_elk_container.sh    |   24 -
 .../elasticsearch/ElasticSearchIOTestUtils.java |  138 -
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java |  155 -
 .../io/elasticsearch/ElasticsearchIOTest.java   |  355 --
 .../elasticsearch/ElasticsearchTestDataSet.java |   97 -
 sdks/java/io/google-cloud-platform/pom.xml      |    9 +-
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java    |  473 ++-
 .../sdk/io/gcp/bigquery/BigQueryHelpers.java    |   28 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java    |  755 +++-
 .../io/gcp/bigquery/BigQueryQuerySource.java    |   49 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   53 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   |   82 +-
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java |  131 +-
 .../gcp/bigquery/BigQueryTableRowIterator.java  |  501 ---
 .../io/gcp/bigquery/BigQueryTableSource.java    |   40 +-
 .../sdk/io/gcp/bigquery/CalculateSchemas.java   |   78 -
 .../beam/sdk/io/gcp/bigquery/CreateTables.java  |   23 +-
 .../io/gcp/bigquery/DynamicDestinations.java    |   13 +-
 .../bigquery/DynamicDestinationsHelpers.java    |   27 +-
 .../sdk/io/gcp/bigquery/ReifyAsIterable.java    |   51 +
 .../sdk/io/gcp/bigquery/SchemaAndRecord.java    |   43 +
 .../io/gcp/bigquery/StreamingWriteTables.java   |   10 +-
 .../sdk/io/gcp/bigquery/TableDestination.java   |   43 +-
 .../io/gcp/bigquery/TableDestinationCoder.java  |    2 +
 .../gcp/bigquery/TableDestinationCoderV2.java   |   59 +
 .../io/gcp/bigquery/WriteBundlesToFiles.java    |   43 +-
 .../bigquery/WriteGroupedRecordsToFiles.java    |    7 +-
 .../sdk/io/gcp/bigquery/WritePartition.java     |   13 +-
 .../beam/sdk/io/gcp/bigquery/WriteRename.java   |  112 +-
 .../beam/sdk/io/gcp/bigquery/WriteTables.java   |  189 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java    |  135 +-
 .../io/gcp/bigtable/BigtableServiceImpl.java    |   16 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  111 +-
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java |   35 +-
 .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java  |    6 +-
 .../io/gcp/pubsub/PubsubUnboundedSource.java    |   41 +-
 .../sdk/io/gcp/spanner/AbstractSpannerFn.java   |   71 -
 .../sdk/io/gcp/spanner/CreateTransactionFn.java |   22 +-
 .../io/gcp/spanner/MutationGroupEncoder.java    |  660 ++++
 .../io/gcp/spanner/MutationSizeEstimator.java   |   48 +
 .../sdk/io/gcp/spanner/NaiveSpannerReadFn.java  |   19 +-
 .../beam/sdk/io/gcp/spanner/OrderedCode.java    |  764 ++++
 .../sdk/io/gcp/spanner/ReadSpannerSchema.java   |   94 +
 .../sdk/io/gcp/spanner/SerializedMutation.java  |   35 +
 .../io/gcp/spanner/SerializedMutationCoder.java |   60 +
 .../sdk/io/gcp/spanner/SpannerAccessor.java     |   43 +
 .../beam/sdk/io/gcp/spanner/SpannerConfig.java  |   41 +-
 .../beam/sdk/io/gcp/spanner/SpannerIO.java      |  384 +-
 .../beam/sdk/io/gcp/spanner/SpannerSchema.java  |  144 +
 .../sdk/io/gcp/spanner/SpannerWriteGroupFn.java |  125 -
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java      |    2 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java     |  811 ++--
 .../bigquery/BigQueryTableRowIteratorTest.java  |  358 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  187 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   78 -
 .../sdk/io/gcp/bigquery/FakeDatasetService.java |   21 +-
 .../sdk/io/gcp/bigquery/FakeJobService.java     |   46 +-
 .../sdk/io/gcp/bigquery/TableContainer.java     |    2 +
 .../sdk/io/gcp/bigtable/BigtableIOTest.java     |   30 +-
 .../sdk/io/gcp/datastore/DatastoreV1Test.java   |  167 +-
 .../sdk/io/gcp/datastore/SplitQueryFnIT.java    |    5 +-
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java    |   55 +
 .../gcp/spanner/MutationGroupEncoderTest.java   |  636 ++++
 .../sdk/io/gcp/spanner/OrderedCodeTest.java     |  890 +++++
 .../io/gcp/spanner/ReadSpannerSchemaTest.java   |  134 +
 .../sdk/io/gcp/spanner/SpannerIOReadTest.java   |   69 -
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java  |  453 ++-
 .../sdk/io/gcp/spanner/SpannerSchemaTest.java   |   61 +
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java |    5 +-
 sdks/java/io/hadoop-common/pom.xml              |    2 +-
 sdks/java/io/hadoop-file-system/pom.xml         |    2 +-
 .../beam/sdk/io/hdfs/HadoopFileSystem.java      |   20 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemTest.java  |   66 +-
 sdks/java/io/hadoop/input-format/pom.xml        |    7 +-
 .../hadoop/inputformat/HadoopInputFormatIO.java |   48 +-
 .../inputformat/HadoopInputFormatIOTest.java    |   14 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml        |    2 +-
 sdks/java/io/hadoop/pom.xml                     |    2 +-
 sdks/java/io/hbase/pom.xml                      |    2 +-
 .../io/hbase/HBaseCoderProviderRegistrar.java   |    8 +-
 .../org/apache/beam/sdk/io/hbase/HBaseIO.java   | 1001 ++---
 .../beam/sdk/io/hbase/HBaseMutationCoder.java   |   27 +-
 .../beam/sdk/io/hbase/HBaseResultCoder.java     |    6 +-
 .../beam/sdk/io/hbase/SerializableScan.java     |   37 +-
 .../hbase/HBaseCoderProviderRegistrarTest.java  |    4 +-
 .../apache/beam/sdk/io/hbase/HBaseIOTest.java   |  754 ++--
 .../sdk/io/hbase/HBaseMutationCoderTest.java    |    4 +-
 .../beam/sdk/io/hbase/HBaseResultCoderTest.java |    4 +-
 .../beam/sdk/io/hbase/SerializableScanTest.java |    6 +-
 sdks/java/io/hcatalog/pom.xml                   |    2 +-
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java |   23 +-
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java    |   24 +-
 sdks/java/io/jdbc/pom.xml                       |    2 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIO.java     |  313 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java |    6 -
 sdks/java/io/jms/pom.xml                        |    2 +-
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  |   71 +-
 sdks/java/io/kafka/README.md                    |   36 +
 sdks/java/io/kafka/pom.xml                      |   19 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  790 +++-
 .../apache/beam/sdk/io/kafka/ProducerSpEL.java  |  135 +
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   |  233 +-
 sdks/java/io/kinesis/pom.xml                    |    8 +-
 .../beam/sdk/io/kinesis/AWSClientsProvider.java |   36 +
 .../sdk/io/kinesis/GetKinesisRecordsResult.java |    8 +-
 .../sdk/io/kinesis/KinesisClientProvider.java   |   33 -
 .../apache/beam/sdk/io/kinesis/KinesisIO.java   |  125 +-
 .../beam/sdk/io/kinesis/KinesisReader.java      |  146 +-
 .../sdk/io/kinesis/KinesisReaderCheckpoint.java |   18 -
 .../beam/sdk/io/kinesis/KinesisSource.java      |   40 +-
 .../apache/beam/sdk/io/kinesis/RoundRobin.java  |   54 -
 .../beam/sdk/io/kinesis/ShardCheckpoint.java    |    8 +-
 .../beam/sdk/io/kinesis/ShardReadersPool.java   |  162 +
 .../sdk/io/kinesis/ShardRecordsIterator.java    |   88 +-
 .../sdk/io/kinesis/SimplifiedKinesisClient.java |   98 +-
 .../io/kinesis/TransientKinesisException.java   |    4 +-
 .../beam/sdk/io/kinesis/AmazonKinesisMock.java  |   19 +-
 .../sdk/io/kinesis/KinesisMockReadTest.java     |    5 +-
 .../beam/sdk/io/kinesis/KinesisReaderIT.java    |    5 +-
 .../beam/sdk/io/kinesis/KinesisReaderTest.java  |  152 +-
 .../beam/sdk/io/kinesis/RoundRobinTest.java     |   59 -
 .../sdk/io/kinesis/ShardReadersPoolTest.java    |  185 +
 .../io/kinesis/ShardRecordsIteratorTest.java    |   35 +-
 .../io/kinesis/SimplifiedKinesisClientTest.java |  107 +
 sdks/java/io/mongodb/pom.xml                    |    2 +-
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java    |    4 -
 .../apache/beam/sdk/io/mongodb/MongoDbIO.java   |   62 +-
 .../sdk/io/mongodb/MongoDBGridFSIOTest.java     |    2 +-
 .../beam/sdk/io/mongodb/MongoDbIOTest.java      |   11 +
 sdks/java/io/mqtt/pom.xml                       |    2 +-
 .../org/apache/beam/sdk/io/mqtt/MqttIO.java     |   49 +-
 sdks/java/io/pom.xml                            |   12 +-
 sdks/java/io/redis/pom.xml                      |   90 +
 .../io/redis/RedisConnectionConfiguration.java  |  122 +
 .../org/apache/beam/sdk/io/redis/RedisIO.java   |  451 +++
 .../apache/beam/sdk/io/redis/package-info.java  |   22 +
 .../apache/beam/sdk/io/redis/RedisIOTest.java   |  109 +
 sdks/java/io/solr/pom.xml                       |  145 +
 .../beam/sdk/io/solr/AuthorizedSolrClient.java  |   91 +
 .../beam/sdk/io/solr/JavaBinCodecCoder.java     |   98 +
 .../org/apache/beam/sdk/io/solr/SolrIO.java     |  705 ++++
 .../apache/beam/sdk/io/solr/package-info.java   |   20 +
 .../beam/sdk/io/solr/JavaBinCodecCoderTest.java |   81 +
 .../org/apache/beam/sdk/io/solr/SolrIOTest.java |  269 ++
 .../beam/sdk/io/solr/SolrIOTestUtils.java       |  132 +
 .../resources/cloud-minimal/conf/schema.xml     |   29 +
 .../resources/cloud-minimal/conf/solrconfig.xml |   48 +
 sdks/java/io/tika/pom.xml                       |  113 +
 .../apache/beam/sdk/io/tika/ParseResult.java    |  144 +
 .../org/apache/beam/sdk/io/tika/TikaIO.java     |  284 ++
 .../apache/beam/sdk/io/tika/package-info.java   |   22 +
 .../beam/sdk/io/tika/ParseResultTest.java       |   83 +
 .../org/apache/beam/sdk/io/tika/TikaIOTest.java |  149 +
 .../java/io/tika/src/test/resources/damaged.pdf |    2 +
 .../resources/valid/apache-beam-tika-pdf.zip    |  Bin 0 -> 11685 bytes
 .../test/resources/valid/apache-beam-tika.odt   |  Bin 0 -> 12540 bytes
 sdks/java/io/xml/pom.xml                        |    2 +-
 .../java/org/apache/beam/sdk/io/xml/XmlIO.java  |  375 +-
 .../org/apache/beam/sdk/io/xml/XmlSink.java     |   10 -
 .../org/apache/beam/sdk/io/xml/XmlSource.java   |   63 +-
 .../org/apache/beam/sdk/io/xml/XmlSinkTest.java |   12 +-
 .../apache/beam/sdk/io/xml/XmlSourceTest.java   |   74 +-
 sdks/java/java8tests/pom.xml                    |    2 +-
 sdks/java/javadoc/ant.xml                       |    1 +
 sdks/java/javadoc/pom.xml                       |   22 +-
 .../maven-archetypes/examples-java8/pom.xml     |    2 +-
 .../main/resources/archetype-resources/pom.xml  |   13 +
 sdks/java/maven-archetypes/examples/pom.xml     |    2 +-
 sdks/java/maven-archetypes/pom.xml              |    2 +-
 sdks/java/maven-archetypes/starter/pom.xml      |    2 +-
 sdks/java/nexmark/README.md                     |  340 ++
 sdks/java/nexmark/pom.xml                       |  265 ++
 .../java/org/apache/beam/sdk/nexmark/Main.java  |  303 ++
 .../org/apache/beam/sdk/nexmark/Monitor.java    |   78 +
 .../beam/sdk/nexmark/NexmarkConfiguration.java  |  721 ++++
 .../beam/sdk/nexmark/NexmarkLauncher.java       | 1157 ++++++
 .../apache/beam/sdk/nexmark/NexmarkOptions.java |  403 ++
 .../apache/beam/sdk/nexmark/NexmarkPerf.java    |  207 ++
 .../apache/beam/sdk/nexmark/NexmarkSuite.java   |  112 +
 .../apache/beam/sdk/nexmark/NexmarkUtils.java   |  674 ++++
 .../apache/beam/sdk/nexmark/model/Auction.java  |  187 +
 .../beam/sdk/nexmark/model/AuctionBid.java      |   85 +
 .../beam/sdk/nexmark/model/AuctionCount.java    |   84 +
 .../beam/sdk/nexmark/model/AuctionPrice.java    |   88 +
 .../org/apache/beam/sdk/nexmark/model/Bid.java  |  177 +
 .../beam/sdk/nexmark/model/BidsPerSession.java  |   87 +
 .../beam/sdk/nexmark/model/CategoryPrice.java   |   97 +
 .../org/apache/beam/sdk/nexmark/model/Done.java |   80 +
 .../apache/beam/sdk/nexmark/model/Event.java    |  171 +
 .../beam/sdk/nexmark/model/IdNameReserve.java   |   98 +
 .../beam/sdk/nexmark/model/KnownSize.java       |   26 +
 .../beam/sdk/nexmark/model/NameCityStateId.java |  103 +
 .../apache/beam/sdk/nexmark/model/Person.java   |  163 +
 .../beam/sdk/nexmark/model/SellerPrice.java     |   89 +
 .../beam/sdk/nexmark/model/package-info.java    |   22 +
 .../apache/beam/sdk/nexmark/package-info.java   |   21 +
 .../sdk/nexmark/queries/AbstractSimulator.java  |  211 ++
 .../beam/sdk/nexmark/queries/NexmarkQuery.java  |  270 ++
 .../sdk/nexmark/queries/NexmarkQueryModel.java  |  117 +
 .../apache/beam/sdk/nexmark/queries/Query0.java |   70 +
 .../beam/sdk/nexmark/queries/Query0Model.java   |   64 +
 .../apache/beam/sdk/nexmark/queries/Query1.java |   67 +
 .../beam/sdk/nexmark/queries/Query10.java       |  367 ++
 .../beam/sdk/nexmark/queries/Query11.java       |   79 +
 .../beam/sdk/nexmark/queries/Query12.java       |   80 +
 .../beam/sdk/nexmark/queries/Query1Model.java   |   76 +
 .../apache/beam/sdk/nexmark/queries/Query2.java |   79 +
 .../beam/sdk/nexmark/queries/Query2Model.java   |   80 +
 .../apache/beam/sdk/nexmark/queries/Query3.java |  301 ++
 .../beam/sdk/nexmark/queries/Query3Model.java   |  124 +
 .../apache/beam/sdk/nexmark/queries/Query4.java |  116 +
 .../beam/sdk/nexmark/queries/Query4Model.java   |  186 +
 .../apache/beam/sdk/nexmark/queries/Query5.java |  138 +
 .../beam/sdk/nexmark/queries/Query5Model.java   |  176 +
 .../apache/beam/sdk/nexmark/queries/Query6.java |  155 +
 .../beam/sdk/nexmark/queries/Query6Model.java   |  133 +
 .../apache/beam/sdk/nexmark/queries/Query7.java |   90 +
 .../beam/sdk/nexmark/queries/Query7Model.java   |  130 +
 .../apache/beam/sdk/nexmark/queries/Query8.java |   98 +
 .../beam/sdk/nexmark/queries/Query8Model.java   |  148 +
 .../apache/beam/sdk/nexmark/queries/Query9.java |   44 +
 .../beam/sdk/nexmark/queries/Query9Model.java   |   44 +
 .../beam/sdk/nexmark/queries/WinningBids.java   |  418 +++
 .../nexmark/queries/WinningBidsSimulator.java   |  206 ++
 .../beam/sdk/nexmark/queries/package-info.java  |   22 +
 .../sdk/nexmark/sources/BoundedEventSource.java |  190 +
 .../beam/sdk/nexmark/sources/Generator.java     |  609 +++
 .../sdk/nexmark/sources/GeneratorConfig.java    |  298 ++
 .../nexmark/sources/UnboundedEventSource.java   |  329 ++
 .../beam/sdk/nexmark/sources/package-info.java  |   22 +
 .../nexmark/src/main/resources/log4j.properties |   55 +
 .../beam/sdk/nexmark/queries/QueryTest.java     |  185 +
 .../nexmark/sources/BoundedEventSourceTest.java |   70 +
 .../beam/sdk/nexmark/sources/GeneratorTest.java |  110 +
 .../sources/UnboundedEventSourceTest.java       |  105 +
 sdks/java/pom.xml                               |    5 +-
 sdks/pom.xml                                    |    6 +-
 sdks/python/apache_beam/__init__.py             |   47 +-
 sdks/python/apache_beam/coders/coder_impl.py    |   22 +-
 sdks/python/apache_beam/coders/coders.py        |  118 +-
 sdks/python/apache_beam/coders/coders_test.py   |    2 +-
 .../apache_beam/coders/coders_test_common.py    |   17 +-
 .../apache_beam/coders/observable_test.py       |    1 -
 .../coders/proto2_coder_test_messages_pb2.py    |    6 +-
 .../apache_beam/coders/standard_coders_test.py  |   11 +-
 sdks/python/apache_beam/coders/stream_test.py   |    1 -
 sdks/python/apache_beam/coders/typecoders.py    |    1 -
 .../examples/complete/autocomplete.py           |    6 +-
 .../examples/complete/autocomplete_test.py      |    2 +-
 .../examples/complete/estimate_pi.py            |    5 +-
 .../examples/complete/estimate_pi_test.py       |    2 +-
 .../examples/complete/game/game_stats.py        |  393 ++
 .../examples/complete/game/game_stats_test.py   |   81 +
 .../examples/complete/game/hourly_team_score.py |  247 +-
 .../examples/complete/game/leader_board.py      |  349 ++
 .../examples/complete/game/leader_board_test.py |   69 +
 .../examples/complete/game/user_score.py        |  186 +-
 .../complete/juliaset/juliaset/juliaset.py      |    8 +-
 .../complete/juliaset/juliaset/juliaset_test.py |    1 -
 .../examples/complete/juliaset/juliaset_main.py |    2 -
 .../examples/complete/juliaset/setup.py         |    7 +-
 .../apache_beam/examples/complete/tfidf.py      |   17 +-
 .../apache_beam/examples/complete/tfidf_test.py |    7 +-
 .../examples/complete/top_wikipedia_sessions.py |    5 +-
 .../complete/top_wikipedia_sessions_test.py     |    1 -
 .../examples/cookbook/bigquery_side_input.py    |    5 +-
 .../examples/cookbook/bigquery_tornadoes.py     |    2 +-
 .../cookbook/bigquery_tornadoes_it_test.py      |    2 +-
 .../examples/cookbook/custom_ptransform.py      |    2 +-
 .../examples/cookbook/datastore_wordcount.py    |   39 +-
 .../examples/cookbook/group_with_coder.py       |    6 +-
 .../examples/cookbook/group_with_coder_test.py  |    1 -
 .../examples/cookbook/mergecontacts.py          |   36 +-
 .../examples/cookbook/multiple_output_pardo.py  |   14 +-
 .../apache_beam/examples/snippets/snippets.py   |   82 +-
 .../examples/snippets/snippets_test.py          |   70 +-
 .../apache_beam/examples/streaming_wordcount.py |    9 +-
 .../apache_beam/examples/windowed_wordcount.py  |    7 +-
 sdks/python/apache_beam/examples/wordcount.py   |   12 +-
 .../apache_beam/examples/wordcount_debugging.py |   12 +-
 .../apache_beam/examples/wordcount_fnapi.py     |  151 +
 .../apache_beam/examples/wordcount_it_test.py   |   16 +
 .../apache_beam/examples/wordcount_minimal.py   |    6 +-
 sdks/python/apache_beam/internal/gcp/auth.py    |    1 -
 .../apache_beam/internal/gcp/json_value.py      |   46 +-
 .../apache_beam/internal/gcp/json_value_test.py |    3 +-
 sdks/python/apache_beam/internal/pickler.py     |    2 +-
 sdks/python/apache_beam/internal/util.py        |    6 +-
 sdks/python/apache_beam/io/avroio.py            |  164 +-
 sdks/python/apache_beam/io/avroio_test.py       |   54 +-
 .../python/apache_beam/io/concat_source_test.py |    1 -
 sdks/python/apache_beam/io/filebasedsink.py     |   22 +-
 .../python/apache_beam/io/filebasedsink_test.py |    3 +-
 sdks/python/apache_beam/io/filebasedsource.py   |  234 +-
 .../apache_beam/io/filebasedsource_test.py      |    8 +-
 sdks/python/apache_beam/io/filesystem.py        |   31 +-
 sdks/python/apache_beam/io/filesystem_test.py   |    5 +-
 sdks/python/apache_beam/io/filesystems.py       |    1 -
 sdks/python/apache_beam/io/filesystems_test.py  |    4 +-
 sdks/python/apache_beam/io/gcp/bigquery.py      |  264 +-
 sdks/python/apache_beam/io/gcp/bigquery_test.py |    4 +-
 .../io/gcp/datastore/v1/adaptive_throttler.py   |   94 +
 .../gcp/datastore/v1/adaptive_throttler_test.py |   95 +
 .../io/gcp/datastore/v1/datastoreio.py          |   55 +-
 .../io/gcp/datastore/v1/datastoreio_test.py     |   11 +-
 .../apache_beam/io/gcp/datastore/v1/helper.py   |   31 +-
 .../io/gcp/datastore/v1/helper_test.py          |    7 +-
 .../apache_beam/io/gcp/gcsfilesystem_test.py    |    1 +
 sdks/python/apache_beam/io/gcp/gcsio.py         |   17 +-
 .../clients/bigquery/bigquery_v2_messages.py    |    3 -
 .../clients/storage/storage_v1_messages.py      |    3 -
 sdks/python/apache_beam/io/gcp/pubsub.py        |    5 +-
 sdks/python/apache_beam/io/gcp/pubsub_test.py   |    5 +-
 .../io/gcp/tests/bigquery_matcher_test.py       |    3 +-
 sdks/python/apache_beam/io/gcp/tests/utils.py   |    8 +-
 .../apache_beam/io/gcp/tests/utils_test.py      |   70 +-
 sdks/python/apache_beam/io/iobase.py            |   83 +-
 .../apache_beam/io/localfilesystem_test.py      |    4 +-
 sdks/python/apache_beam/io/range_trackers.py    |   54 +-
 .../apache_beam/io/range_trackers_test.py       |   37 +
 sdks/python/apache_beam/io/source_test_utils.py |   92 +-
 .../apache_beam/io/source_test_utils_test.py    |    2 +-
 sdks/python/apache_beam/io/sources_test.py      |    1 -
 sdks/python/apache_beam/io/textio.py            |  178 +-
 sdks/python/apache_beam/io/textio_test.py       |  109 +-
 sdks/python/apache_beam/io/tfrecordio.py        |    5 +-
 sdks/python/apache_beam/io/tfrecordio_test.py   |    8 +-
 sdks/python/apache_beam/metrics/cells_test.py   |    2 +-
 sdks/python/apache_beam/metrics/execution.py    |    5 +-
 .../apache_beam/metrics/execution_test.py       |    8 +-
 sdks/python/apache_beam/metrics/metric.py       |    3 +-
 sdks/python/apache_beam/metrics/metric_test.py  |   18 +-
 sdks/python/apache_beam/metrics/metricbase.py   |    4 +
 .../apache_beam/options/pipeline_options.py     |   32 +-
 .../options/pipeline_options_test.py            |    7 +-
 .../options/pipeline_options_validator_test.py  |    3 +-
 .../apache_beam/options/value_provider.py       |    1 -
 sdks/python/apache_beam/pipeline.py             |  136 +-
 sdks/python/apache_beam/pipeline_test.py        |   92 +-
 sdks/python/apache_beam/pvalue.py               |  111 +-
 sdks/python/apache_beam/pvalue_test.py          |    8 +
 sdks/python/apache_beam/runners/common.py       |    6 +-
 sdks/python/apache_beam/runners/common_test.py  |    2 +-
 .../runners/dataflow/dataflow_metrics.py        |    2 +-
 .../runners/dataflow/dataflow_runner.py         |  122 +-
 .../runners/dataflow/dataflow_runner_test.py    |   89 +-
 .../runners/dataflow/internal/apiclient.py      |  106 +-
 .../runners/dataflow/internal/apiclient_test.py |  112 +-
 .../clients/dataflow/dataflow_v1b3_client.py    |  259 +-
 .../clients/dataflow/dataflow_v1b3_messages.py  |   92 +-
 .../clients/dataflow/message_matchers.py        |    1 -
 .../clients/dataflow/message_matchers_test.py   |    3 +-
 .../runners/dataflow/internal/dependency.py     |   58 +-
 .../dataflow/internal/dependency_test.py        |   10 +-
 .../runners/dataflow/internal/names.py          |    2 +
 .../runners/dataflow/native_io/iobase_test.py   |   22 +-
 .../dataflow/native_io/streaming_create.py      |    2 +-
 .../runners/dataflow/template_runner_test.py    |    5 +-
 .../runners/dataflow/test_dataflow_runner.py    |    5 +-
 .../consumer_tracking_pipeline_visitor_test.py  |    2 +-
 .../runners/direct/direct_metrics.py            |    2 +-
 .../runners/direct/direct_metrics_test.py       |    8 +-
 .../apache_beam/runners/direct/direct_runner.py |   19 +-
 .../runners/direct/direct_runner_test.py        |   41 +
 .../runners/direct/evaluation_context.py        |   31 +-
 .../apache_beam/runners/direct/executor.py      |  127 +-
 .../runners/direct/helper_transforms.py         |    2 +-
 .../runners/direct/transform_evaluator.py       |   67 +-
 sdks/python/apache_beam/runners/direct/util.py  |    7 +-
 .../runners/experimental/__init__.py            |   16 +
 .../experimental/python_rpc_direct/__init__.py  |   22 +
 .../python_rpc_direct_runner.py                 |  110 +
 .../experimental/python_rpc_direct/server.py    |  111 +
 sdks/python/apache_beam/runners/job/__init__.py |   16 +
 sdks/python/apache_beam/runners/job/manager.py  |   52 +
 sdks/python/apache_beam/runners/job/utils.py    |   32 +
 .../apache_beam/runners/pipeline_context.py     |    2 +-
 .../runners/portability/fn_api_runner.py        |  393 +-
 .../runners/portability/fn_api_runner_test.py   |  115 +-
 .../portability/maptask_executor_runner.py      |   19 +-
 .../portability/maptask_executor_runner_test.py |   12 +-
 .../portability/universal_local_runner.py       |  409 ++
 .../portability/universal_local_runner_main.py  |   44 +
 .../portability/universal_local_runner_test.py  |   85 +
 sdks/python/apache_beam/runners/runner.py       |   61 +-
 sdks/python/apache_beam/runners/runner_test.py  |    2 +-
 .../runners/worker/bundle_processor.py          |  235 +-
 .../apache_beam/runners/worker/data_plane.py    |   35 +-
 .../runners/worker/data_plane_test.py           |    7 +-
 .../apache_beam/runners/worker/log_handler.py   |   17 +-
 .../runners/worker/log_handler_test.py          |   20 +-
 .../apache_beam/runners/worker/opcounters.py    |   10 +-
 .../runners/worker/opcounters_test.py           |    1 -
 .../apache_beam/runners/worker/operations.pxd   |    4 +-
 .../apache_beam/runners/worker/operations.py    |   84 +-
 .../apache_beam/runners/worker/sdk_worker.py    |  206 +-
 .../runners/worker/sdk_worker_main.py           |   30 +-
 .../runners/worker/sdk_worker_test.py           |   11 +-
 .../apache_beam/runners/worker/statesampler.pyx |   68 +-
 .../runners/worker/statesampler_fake.py         |   17 +-
 .../runners/worker/statesampler_test.py         |    5 +-
 .../apache_beam/testing/pipeline_verifiers.py   |    1 -
 .../testing/pipeline_verifiers_test.py          |    5 +-
 .../python/apache_beam/testing/test_pipeline.py |   54 +-
 .../apache_beam/testing/test_pipeline_test.py   |    4 +-
 sdks/python/apache_beam/testing/test_stream.py  |    1 -
 .../apache_beam/testing/test_stream_test.py     |    3 +-
 sdks/python/apache_beam/testing/test_utils.py   |   23 +-
 .../apache_beam/testing/test_utils_test.py      |   62 +
 sdks/python/apache_beam/testing/util.py         |    5 +-
 sdks/python/apache_beam/testing/util_test.py    |    4 +-
 sdks/python/apache_beam/transforms/combiners.py |    9 +-
 .../apache_beam/transforms/combiners_test.py    |    5 +-
 sdks/python/apache_beam/transforms/core.py      |  397 +-
 .../apache_beam/transforms/create_test.py       |    6 +-
 sdks/python/apache_beam/transforms/display.py   |   91 +-
 .../apache_beam/transforms/display_test.py      |    6 +-
 .../python/apache_beam/transforms/ptransform.py |  147 +-
 .../apache_beam/transforms/ptransform_test.py   |   78 +-
 .../apache_beam/transforms/sideinputs_test.py   |    5 +-
 sdks/python/apache_beam/transforms/timeutil.py  |    1 -
 sdks/python/apache_beam/transforms/trigger.py   |   32 +-
 .../apache_beam/transforms/trigger_test.py      |   19 +-
 sdks/python/apache_beam/transforms/util.py      |  273 +-
 sdks/python/apache_beam/transforms/util_test.py |  108 +
 sdks/python/apache_beam/transforms/window.py    |    9 +-
 .../apache_beam/transforms/window_test.py       |   17 +-
 .../transforms/write_ptransform_test.py         |    4 +-
 sdks/python/apache_beam/typehints/decorators.py |  129 +-
 .../typehints/native_type_compatibility.py      |  166 +
 .../typehints/native_type_compatibility_test.py |   92 +
 sdks/python/apache_beam/typehints/opcodes.py    |   19 +-
 .../apache_beam/typehints/trivial_inference.py  |   80 +-
 .../typehints/trivial_inference_test.py         |    9 +-
 sdks/python/apache_beam/typehints/typecheck.py  |    6 +-
 .../typehints/typed_pipeline_test.py            |   33 +-
 sdks/python/apache_beam/typehints/typehints.py  |   79 +-
 .../apache_beam/typehints/typehints_test.py     |    3 +-
 .../apache_beam/utils/annotations_test.py       |    1 +
 sdks/python/apache_beam/utils/counters.py       |  107 +-
 sdks/python/apache_beam/utils/counters_test.py  |   78 +
 sdks/python/apache_beam/utils/processes_test.py |    1 -
 sdks/python/apache_beam/utils/proto_utils.py    |   11 +
 sdks/python/apache_beam/utils/retry.py          |    9 +-
 sdks/python/apache_beam/utils/retry_test.py     |    5 +-
 sdks/python/apache_beam/utils/urns.py           |   22 +-
 sdks/python/apache_beam/version.py              |    2 +-
 sdks/python/container/Dockerfile                |   27 +
 sdks/python/container/boot.go                   |  123 +
 sdks/python/container/pom.xml                   |  154 +
 sdks/python/gen_protos.py                       |   30 +-
 sdks/python/generate_pydoc.sh                   |  134 +-
 sdks/python/pom.xml                             |    9 +-
 sdks/python/run_postcommit.sh                   |   20 -
 sdks/python/run_pylint.sh                       |   45 +-
 sdks/python/run_validatesrunner.sh              |   71 +
 sdks/python/setup.py                            |   10 +-
 sdks/python/tox.ini                             |   21 +-
 1247 files changed, 92590 insertions(+), 17747 deletions(-)
----------------------------------------------------------------------



Mime
View raw message