beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré (JIRA) <>
Subject [jira] [Resolved] (BEAM-1644) IO ITs: shared directory for kubernetes resources and PipelineOptions?
Date Mon, 20 Mar 2017 13:29:41 GMT


Jean-Baptiste Onofré resolved BEAM-1644.
       Resolution: Fixed
    Fix Version/s: First stable release

> IO ITs: shared directory for kubernetes resources and PipelineOptions?
> ----------------------------------------------------------------------
>                 Key: BEAM-1644
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Stephen Sisk
>            Assignee: Stephen Sisk
>             Fix For: First stable release
> It is the case that different IOs will be created that connect to the same data stores
- HadoopInputFormat in particular uses ES and cassandra, which are also used in their respective
IOs as well. Jdbc is likely to have the same type of overlap.
> It would be nice to share TestPipelineOption and kubernetes/docker scripts so that we
don't need to repeat them in each module. 
> As such, we should put them in a shared location. It seems like the best bet is to create
a new module for test resources. 
> Proposal: 
> * Create a new module under io, test-common (? I'm not in love with that name) 
> * All the IOs with ITs can put their TestPipelineOption classes there
> * it will have a kubernetes directory with the various kubernetes scripts in sub directories
> Further thoughts:
> If 2 IOs need different configurations of a data store, this would make that harder.
In that case, there's nothing stopping us from having two different sets of the kubernetes
scripts in the shared location

This message was sent by Atlassian JIRA

View raw message