beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Sisk (JIRA)" <>
Subject [jira] [Assigned] (BEAM-1807) IO ITs: shared language neutral directory for kubernetes resources
Date Fri, 07 Apr 2017 22:21:41 GMT


Stephen Sisk reassigned BEAM-1807:

    Assignee: Stephen Sisk  (was: Davor Bonaci)

> IO ITs: shared language neutral directory for kubernetes resources
> ------------------------------------------------------------------
>                 Key: BEAM-1807
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: testing
>            Reporter: Stephen Sisk
>            Assignee: Stephen Sisk
> This is a follow-up to BEAM-1644. As was discussed there: 
> "
> It is the case that different IOs will be created that connect to the same data stores
- HadoopInputFormat in particular uses ES and cassandra, which are also used in their respective
IOs as well. Jdbc is likely to have the same type of overlap.
> It would be nice to share [...] kubernetes/docker scripts so that we don't need to repeat
them in each module. 
> "
> For BEAM-1644, we created a directory for java io-common resources - that's perfect for
the java pipeline options we needed. However, we shouldn't put kubernetes resources in the
newly created sdks/java/io/common because that'd indicate that the scripts are java specific.

> It's also worth noting that we have this problem already for jenkins and travis, and
solved it by creating .jenkins and .travis directories at the top-level.
> Proposal
> =======
> move .jenkins and .travis into a new top level ".test-infra" folder, and put a kubernetes
directory there.
> So the new structure would look like:
> .test-infra
>   jenkins
>   travis
>   kubernetes
> sdks
> runners
> examples
> ...
> I don't know if travis/jenkins must look in .travis/.jenkins directories or if those
are things that we can change. If both do, would lessen my excitement, but if at least one
other thing can share that directory, that would make it worthwhile in my mind.
> Alternate proposal
> ===============
> add a top-level .kubernetes directory alongside .jenkins/.travis. 
> I'm not a huge fan of this since I'd love to not add more top level clutter.
> Alternate proposal
> ===============
> We could create:
> sdks/common/test-infra/kubernetes
> and put the scripts there. 
> I don't like this option as much because it's kind of a just a random directory and is
disconnected from the rest of the test infrastructure scripts that we use. I also prefer the
other option since it reduces the amount of top-level clutter.
> Thoughts?
> cc [~jasonkuster] [~davor] [~iemejia]

This message was sent by Atlassian JIRA

View raw message