airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Walther <georg.walt...@markovian.com>
Subject Re: Integration test env
Date Tue, 20 Dec 2016 18:31:06 GMT
Hi,


why would you want these services to be long-running? Why not write
integration tests against clean instances of each service or clean
instances with data fixtures on top?
Travis offers spinning up a variety of services (
https://docs.travis-ci.com/user/database-setup/) so does wercker (
http://devcenter.wercker.com/docs/services).
These services are docker containers run off of pre-build Docker images.


Best,

Georg


On Tue, Dec 20, 2016 at 5:18 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Hi Chris,
>
> Didn’t they offer a dedicated Travis thing as well? And can we setup long
> running services (ie. Hadoop, KDC, Presto, MySQL, etc) in the Jenkins
> environment, because that would be sufficient I guess? To my knowledge what
> Apache offers is more suited for unit tests (ie. the hadoop-qa runs) than
> integration tests.
>
> What are your thoughts?
>
> Bolke
>
> > Op 19 dec. 2016, om 22:26 heeft Chris Riccomini <criccomini@apache.org>
> het volgende geschreven:
> >
> > Hey Bolke,
> >
> > Thanks for stepping up. The "traditional" Apache infra is Jenkins with a
> > pool of machines that they provide. That might or might not be
> satisfactory
> > for us (it's certainly antiquated technology). If we decide we don't like
> > it, I think that's OK, as long as we move forward knowing that any other
> > testing solution can disappear at any time.
> >
> > My 2c. :)
> >
> > Cheers,
> > Chris
> >
> > On Wed, Dec 14, 2016 at 11:11 AM, Dan Davydov <
> > dan.davydov@airbnb.com.invalid> wrote:
> >
> >> This is extremely generous of you! I do agree with the approach of
> trying
> >> to get funding from Apache and having shared resources (e.g. so that we
> >> don't depend on any one company or individual for the uptime of the
> >> integration environment, plus so we would have public cloud integration
> >> potentially).
> >>
> >> On Wed, Dec 14, 2016 at 1:22 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> I have been thinking about an integration test environment. Aside from
> >> any
> >>> technical requirements we need a place to do it. I am willing to offer
> a
> >>> place in Lab env I am running or to fund an environment in AWS/GCloud
> if
> >>> Apache cannot make these kind of resources available.
> >>>
> >>> If running in our Lab there is virtually no restriction what we could
> do,
> >>> however I will hand select people who have access to this environment.
> I
> >>> will also hold ultimate power to remove access from anyone. I even
> might
> >>> ask for a confirmation that you will behave when using our property
> >> (don’t
> >>> worry won’t cover it with legal wording). This is a IAAS service so we
> >> need
> >>> to cover the things we need ourselves, but the upside is we can and it
> is
> >>> free. We could setup a Gitlab instance that mirrors from Apache a kicks
> >> off
> >>> runners to do testing. Downside 1) it might not be entirely Apache
> like.
> >>> Sorry cant help that. 2) there is no guaranteed up time 3) I might need
> >> to
> >>> remove it in the future e.g. when I change jobs for example :). 4) No
> >>> public cloud integration, it’s a private stack after all.
> >>>
> >>> I can also fund on AWS/GCloud. Again, I probably want to have ultimate
> >>> power on access to this environment - it’s my company’s money on the
> line
> >>> after all. Major downside to this is that it is dependent on and
> limited
> >> by
> >>> the budget I can make available. Upside is that it is not company
> >> property.
> >>> Also I personally have less exposure to public cloud environments due
> to
> >>> company restrictions.
> >>>
> >>> Are there any other options? Any thoughts?
> >>>
> >>> Bolke
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message