beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-1644) IO ITs: shared directory for kubernetes resources and PipelineOptions?
Date Fri, 07 Apr 2017 23:15:42 GMT


ASF GitHub Bot commented on BEAM-1644:

GitHub user ssisk opened a pull request:

    [BEAM-1644] Move travis & jenkins into shared test-infra dir, and move k8s scripts

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
     - [X] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [X] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [X] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](
    Reasoning is discussed in the bug, but generally, we want to have a language neutral place
for these kubernetes scripts to live, and also don't want a million top level directories
for test-infra
    cc @jasonkuster 

You can merge this pull request into a Git repository by running:

    $ git pull create-k8s-home

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2466
commit 617297662e2649f5e38f5c4ae9047f7e9bcecbe8
Author: Stephen Sisk <>
Date:   2017-04-07T22:57:33Z

    Move travis/jenkins folders in a test-infra folder

commit 4cd666c5026ef91121a5cf7c6786c21c8d39d904
Author: Stephen Sisk <>
Date:   2017-04-07T23:06:15Z

    Move jdbc's postgres k8s scripts into shared k8s dir

commit 3468cec566bd24164922c0c063067d90e1c43849
Author: Stephen Sisk <>
Date:   2017-04-07T23:11:19Z

    Move HIFIO k8s scripts into shared dir


> IO ITs: shared directory for kubernetes resources and PipelineOptions?
> ----------------------------------------------------------------------
>                 Key: BEAM-1644
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Stephen Sisk
>            Assignee: Stephen Sisk
>             Fix For: First stable release
> It is the case that different IOs will be created that connect to the same data stores
- HadoopInputFormat in particular uses ES and cassandra, which are also used in their respective
IOs as well. Jdbc is likely to have the same type of overlap.
> It would be nice to share TestPipelineOption and kubernetes/docker scripts so that we
don't need to repeat them in each module. 
> As such, we should put them in a shared location. It seems like the best bet is to create
a new module for test resources. 
> Proposal: 
> * Create a new module under io, test-common (? I'm not in love with that name) 
> * All the IOs with ITs can put their TestPipelineOption classes there
> * it will have a kubernetes directory with the various kubernetes scripts in sub directories
> Further thoughts:
> If 2 IOs need different configurations of a data store, this would make that harder.
In that case, there's nothing stopping us from having two different sets of the kubernetes
scripts in the shared location

This message was sent by Atlassian JIRA

View raw message