airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Xuan Ji <xua...@gmail.com>
Subject Re: Integration test env
Date Tue, 20 Dec 2016 20:13:03 GMT
Chris, is this the authoritative list of services apache provides to
projects? https://www.apache.org/dev/services.html

The builds section / page links to https://ci.apache.org/ and lists the
following build services: buildbot, gump, jenkins (the link to continuum is
broken). From my brief reading, it seems they use a shared pool of machines
(for all projects) are designed for running unit/integration tests that can
be torn down after running, not long-running services.

There's no references to Travis on those pages, there are some tickets
related to travis (
https://www.google.ca/search?sitesearch=*.apache.org&q=travis) but I think
they are referring to integration with the public travis-ci.org service
that we already use.

The page also says that FreeBSD jails and Ubuntu VMs are available,
although I'm not sure what they are normally used for.

One thing I don't really like about the "official" infrastructure is
access, eg we can't ssh into the travis-ci.org builds to debug errors, and
the travis build script we have is quite verbose, presumably for debugging
purposes.

On 20 December 2016 at 13:31, Georg Walther <georg.walther@markovian.com>
wrote:

> Hi,
>
>
> why would you want these services to be long-running? Why not write
> integration tests against clean instances of each service or clean
> instances with data fixtures on top?
> Travis offers spinning up a variety of services (
> https://docs.travis-ci.com/user/database-setup/) so does wercker (
> http://devcenter.wercker.com/docs/services).
> These services are docker containers run off of pre-build Docker images.
>
>
> Best,
>
> Georg
>
>
> On Tue, Dec 20, 2016 at 5:18 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
> > Hi Chris,
> >
> > Didn’t they offer a dedicated Travis thing as well? And can we setup long
> > running services (ie. Hadoop, KDC, Presto, MySQL, etc) in the Jenkins
> > environment, because that would be sufficient I guess? To my knowledge
> what
> > Apache offers is more suited for unit tests (ie. the hadoop-qa runs) than
> > integration tests.
> >
> > What are your thoughts?
> >
> > Bolke
> >
> > > Op 19 dec. 2016, om 22:26 heeft Chris Riccomini <criccomini@apache.org
> >
> > het volgende geschreven:
> > >
> > > Hey Bolke,
> > >
> > > Thanks for stepping up. The "traditional" Apache infra is Jenkins with
> a
> > > pool of machines that they provide. That might or might not be
> > satisfactory
> > > for us (it's certainly antiquated technology). If we decide we don't
> like
> > > it, I think that's OK, as long as we move forward knowing that any
> other
> > > testing solution can disappear at any time.
> > >
> > > My 2c. :)
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Wed, Dec 14, 2016 at 11:11 AM, Dan Davydov <
> > > dan.davydov@airbnb.com.invalid> wrote:
> > >
> > >> This is extremely generous of you! I do agree with the approach of
> > trying
> > >> to get funding from Apache and having shared resources (e.g. so that
> we
> > >> don't depend on any one company or individual for the uptime of the
> > >> integration environment, plus so we would have public cloud
> integration
> > >> potentially).
> > >>
> > >> On Wed, Dec 14, 2016 at 1:22 AM, Bolke de Bruin <bdbruin@gmail.com>
> > wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I have been thinking about an integration test environment. Aside
> from
> > >> any
> > >>> technical requirements we need a place to do it. I am willing to
> offer
> > a
> > >>> place in Lab env I am running or to fund an environment in AWS/GCloud
> > if
> > >>> Apache cannot make these kind of resources available.
> > >>>
> > >>> If running in our Lab there is virtually no restriction what we could
> > do,
> > >>> however I will hand select people who have access to this
> environment.
> > I
> > >>> will also hold ultimate power to remove access from anyone. I even
> > might
> > >>> ask for a confirmation that you will behave when using our property
> > >> (don’t
> > >>> worry won’t cover it with legal wording). This is a IAAS service
so
> we
> > >> need
> > >>> to cover the things we need ourselves, but the upside is we can and
> it
> > is
> > >>> free. We could setup a Gitlab instance that mirrors from Apache a
> kicks
> > >> off
> > >>> runners to do testing. Downside 1) it might not be entirely Apache
> > like.
> > >>> Sorry cant help that. 2) there is no guaranteed up time 3) I might
> need
> > >> to
> > >>> remove it in the future e.g. when I change jobs for example :). 4)
No
> > >>> public cloud integration, it’s a private stack after all.
> > >>>
> > >>> I can also fund on AWS/GCloud. Again, I probably want to have
> ultimate
> > >>> power on access to this environment - it’s my company’s money on
the
> > line
> > >>> after all. Major downside to this is that it is dependent on and
> > limited
> > >> by
> > >>> the budget I can make available. Upside is that it is not company
> > >> property.
> > >>> Also I personally have less exposure to public cloud environments due
> > to
> > >>> company restrictions.
> > >>>
> > >>> Are there any other options? Any thoughts?
> > >>>
> > >>> Bolke
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>



-- 
Im Xuan Ji!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message